Have you ever wondered how many times a day Google is used to Search the web? 5 times, 10 times, or even more? Try 2 trillion searches per year, that’s 5479452054 times a day (I literally haven’t a clue to what number is)? Its a seriously huge amount!
Google and other Search Engines have become a huge part of our daily lives. We use them as a learning tool, to help us shop, for fun and leisure and we also use it for business. Let’s just say we all depend on search engines for almost anything we do.
The reason, Google has all of the answers to all of our questions and queries. But what really happens behind the scenes when we type a query into a search engine and click on the search button?
If like me, you rely on search engines to make a living online, then you need to understand how a search engine works. Having a clear understanding of how a search engine works, can help you create and optimise a website that search engines can understand. This is called Search Engine Optimisation (SEO).
How search works
Search engines are a bunch of complex computer algorithms, before they even allow you to type in a search term, they have to do a lot of prep work, so that when you do click the search button, you’re presented with a relevant and meaning full list of results that should answer your search term.
The preparation work consists of two main stages. First, is the process of discovering the information, second is organising the information so that it can be used later for search purposes. This process is generally known as Crawling and Indexing.
Crawling
Search engines have a number of computer programmes called web crawlers, their job is to find information that is publicly available on the Internet. Crawlers can the Internet and find the servers that are hosting websites.
Each crawler visits each website to try and find out how many pages they have, whether it is text content, images, videos or any other format (CSS, HTML, javascript, etc). Each crawler also follows any links (pointing within your website or to external websites) to discover more and more pages.
Crawlers do this continuously, so they keep track of changes to your website, like when new pages and content are added or deleted, the links have been updated etc.
Why do I need to care about crawling?
When you’re optimising your website for SEO, the first thing to do is to ensure that your website can be accessed by search engines, otherwise, you will not rank very high within the search engine’s search results.
There are a number of things you can do to make sure that crawlers can discover your website in the fastest possible way. Here are a few of the best ways to get you started.
Use Robots.txt to specify which of your site’s pages you don’t want the crawlers to access. For example, pages like your admin or backend pages and other pages that you don’t want to be on display to the public.
Google has a tool (google search console) that you can use to give Google more information about your website, like the number of pages, website structure etc.
Use an XML sitemap, this will list all of your important web pages and structures them in a way that a crawler can easily understand, the crawler can then go through the sitemap to monitor for changes.
Indexing
Search engines don’t just use crawlers to build their search databases, information that is collected by the crawlers need to be organised, sorted and stored so that It can be processed by the search engine algorithms before they are severed up and made available in their search results.
This process is called indexing. Search engines don’t store all the information that is collected about a website in their index, the information they do keep is; when the site page was created/updated, title and description of the site page, type of content on the page, associated keywords, incoming and outgoing links and a lot of other parameters.
Why should you care about the indexing process?
If your website is not in the Google search index, it will not appear for any searches. The more web pages that you have in the search index, the more chances you have of appearing in the search results when a keyword is searched.
Now when I say “search results” this means on any position and not necessarily on the top positions or pages. If you want to appear in the first 5 positions of the SERP (search engine results pages) then you have to spend some time optimising your website using Search Engine Optimisation (SEO for short)
Remember I said that the more of your web pages that are indexed the more chance there is for it to appear in the search results pages. To get an idea of how many of the pages of your website are listed within the Google search index you can do it in two ways.
The first way is to go to Google and type in site:mywebsite.co.uk – enter your domain name. Now Google will show only the search results for your domain name. From there you will be able to see how many of your website pages are already indexed within the search results.
The second way is to create a free Google search console account and look at the Index Status report.
Search Engine Ranking Algorithms
The third and final setup in the search engine process is to decide which pages to show in the SERPS and in what order when someone types a search query.
Search engines do this through the use of search engine ranking algorithms, Googles algorithm is called PageRank. Simply put, these algorithms are software that contains a number of rules that analyse what the user is looking for and what information will be returned. These rules and decisions are made based on what information is available in their search index.
How do search engine algorithms work?
Google’s PageRank has evolved over the years and has become really, really complex. In the beginning, it was as simple as matching a user’s query with the title of the page, but nowadays this is no longer the case.
Googles ranking algorithm takes into account more than 255 rules before making a decision and nobody knows for the user what these rules are.
And this includes Larry Page and Sergey Brin (the founders of Google), who created the PageRank algorithm.
New technology has allowed for the PageRank algorithm to evolve and is now using machine learning and other computer programmes to help with making decisions based on a number parameters that are outside the boundaries of the content that is found on your website page.
I have created the table below to help you understand the most common factors taking into account by PageRank.
Here is a simplified process of search engine ranking algorithms work;
Step 1: Analyse user query
The first step is to understand what kind of information the user is looking for. To do that they analyse the user’s query by breaking it down into a number of meaningful keywords.
A keyword is a word that has a specific meaning and purpose, for example, when you type “How to make a chocolate cake” Google will know that from the words “how to”, that you’re looking for instructions on how to make a chocolate cake and will return results of websites that are relevant to that search term.
with the updates to PageRank, machine learning has helped google associate related keywords together. For example, they know the meaning of this query “how to change a light bulb” is the same as this “how to replace a light bulb”. Not just that, the PageRank algorithm is now clever enough to interpret spelling mistakes, it understands plurals and can even extract the meaning of a query from natural language (this can be written or even through voice search!)
Step 2: Finding matching pages
The second step is to look into their search index and decide which pages can provide the best answer for a given search query. This is a very important stage of the PageRank algorithm and for website owners.
Google needs to return the best possible results in the latest possible way so that they keep their users happy. Website owners want their websites to be picked up so that they get traffic and visitors.
This is also the stage where good SEO techniques can influence the decision made by Googles PageRank. To give you an idea of how this works, these are the most important factors;
Title and content relevancy
How relevant is the title and content of the website compared with the user’s search query?
Type of content
If the user is asking for images, the returned results will contain images and not text.
Quality of the content
You’re websites content needs to be thorough, useful and informative, unbiased and cover both sides of the story.
Quality of the website
The overall quality of a website matters. Google will not show pages from websites that don’t meet their quality standards.
Date of publication
For news related search queries, Google wants to show the latest results so the date of publication is taken into account.
The popularity of a page
This doesn’t have much to do with how much traffic your website gets, but how other websites perceive your websites page.
A page that has a lot of backlinks, from other websites is considered to be more popular than other pages with no links and thus has more chances of getting picked up by PageRank. This process is referred to as off-page SEO.
The language of the page
Users are served pages in their native language.
Website speed
Websites that load fast, like within 2-3 seconds, have a small advantage over websites that are slow to load.
Device type
Users searching on mobile are served mobile-friendly pages
Location
Users searching for results in their area i.e “Italian restaurants in London” will be shown results related to their location
These factors are just the tip of the iceberg. As mentioned before, Goole uses more than 255 factors in their algorithms to ensure that their users are happy with the results they get.
Why care how search engines work?
in order to get traffic to your website from Google, your website needs to appear in the top positions on the first page of the results.
It is statically proven that the majority of users click on the top 5 results (that’s both desktop and mobile devices).
Appearing in the second or third page of the results will not get you any traffic at all. Traffic is just one of the benefits of SEO, once you get to the top positions for keywords that make sense for your business, the added benefits are much more.
Once search engines can read your website without issues, you then need to ensure that you give them the right signals to help the PageRank algorithms pick your website when a user types a relevant query.
Don’t forget that if you manage to get a tiny share of the overall Google search traffic, that’s enough to build a long-lasting online business.
