Contributor Patrick Stox walks us through how to use a cluster analysis tool to visualize websites and identify opportunities for improving their link structure.

In April of last year, Search Engine Land contributor Paul Shapiro wrote a brilliant post on Calculating Internal PageRank. The post outlined a method to look at a website’s internal linking in order to determine the importance of pages within the website.

This is amazingly powerful, but I think Paul’s concept could be more user-friendly. He used R, which is a language and environment for statistical computing, and the output is basically a bunch of numbers.

I want to show you how to do the same in Gephi with the push of a few buttons instead of a bunch of code — and, with a few more clicks, you can visualize the data in a way you will be proud to show your clients.

I’ll show you how to get this result as an example of how Gephi can be useful in your SEO efforts. You’ll be able to see what pages are the strongest pages on your website, determine how pages can be grouped by topic and identify some common website issues such as crawl errors or poor internal linking. Then I’ll describe some ideas for taking the concept to the next level of geekery.

What is Gephi?

Gephi is free open-source software that is used to graph networks and is commonly used to represent computer networks and social media networks.

It’s a simple, Java-based desktop program that runs on Windows, Mac or Linux. Though the current version of Gephi is 0.9.1, I encourage you to download the previous version, 0.9.0, or the later version, 0.9.2, instead. That way you’ll be able to follow along here, and you’ll avoid the bugs and headaches of the current version. (If you haven’t done it recently, you may need to install Java onto your computer as well.)

1. Start by crawling your website and gathering data

I normally use Screaming Frog for crawling. Since we are interested in pages here and not other files, you’ll need to exclude things from the crawl data.

To do that, those of you with the paid version of the software should implement the settings I’ll describe next. (If you’re using the free version, which limits you to collecting 500 URLs and doesn’t allow you to tweak as many settings, I’ll explain what to do later.)

Go to “Configuration” > “Spider” and you’ll see something like the screen shot below. Make yours match mine for the best results. I also normally add .*(png|jpg|jpeg|gif|bmp)$ to “Configuration” > “Exclude” to get rid of images, which Screaming Frog sometimes leaves in the crawl report.


To start the crawl, put your site’s URL into the space at the top left (pictured below). Then click “Start” and wait for the crawl to finish.


When your crawl is finished, go to “Bulk Export” > “All Inlinks.” You’ll want to change “Files of Type” to “.csv” and save your file.