For many, Google is the internet. It’s the starting point for finding new sites, and is arguably the most important invention since the internet itself. Without search engines, new web content would be inaccessible to the masses.

Every search engine has three main functions: crawling (to discover content), indexing (to track and store content), and retrieval (to fetch relevant content when users query the search engine).

Crawling

Crawling is where it all begins the acquisition of data about a website. This involves scanning sites and collecting details about each page: titles, images, keywords, other linked pages, etc. Different crawlers may also look for different details, like page layouts, where advertisements are placed, whether links are crammed in, etc.

An automated bot (called a “spider”) visits page after page as quickly as possible, using page links to find where to go next. Even in the earliest days, Google’s spiders could read several hundred pages per second. Nowadays, it’s in the thousands. When a web crawler visits a page, it collects every link on the page and adds them to its list of next pages to visit. It goes to the next page in its list, collects the links on that page, and repeats. Web crawlers also revisit past pages once in a while to see if any changes happened.

This means any site that’s linked from an indexed site will eventually be crawled. Some sites are crawled more frequently, and some are crawled to greater depths, but sometimes a crawler may give up if a site’s page hierarchy is too complex.

Indexing

Indexing is when the data from a crawl is processed and placed in a database.

Imagine making a list of all the books you own, their publishers, their authors, their genres, their page counts, etc. Crawling is when you comb through each book while indexing is when you log them to your list.

Retrieval and Ranking

Retrieval is when the search engine processes your search query and returns the most relevant pages that match your query. Most search engines differentiate themselves through their retrieval methods: they use different criteria to pick and choose which pages fit best with what you want to find. That’s why search results vary between Google and Bing.

Originally, search engines ranked sites by how often keywords appeared on a page, which led to “keyword stuffing” — filling pages with keyword-heavy nonsense.

Then came the concept of link importance: search engines valued sites with lots of incoming links because they interpreted site popularity as relevance. But this led to link spamming all over the web. Nowadays, search engines weight links depending on the “authority” of the linking site. Search engines put more value on links from a government agency than links from a link directory.

What’s Next for Search Engine?

Right now, you can search for “gluten-free cookies” but the results may return recipes for gluten-free cookies. Instead, you might find regular cookie recipes that say “This recipe is not gluten-free.” It has the right keywords, but the wrong meaning.

With semantics, you can search for cookie recipes and then remove certain ingredients: flour, nuts, etc. You can also narrow down results to only recipes with prep times less than 30 minutes and review scores of 4/5 or greater. That would be cool, right? That’s where we’re heading!

The post How do Search Engines Work appeared first on SEO Training.