How does Google Search work? Crawling, indexing and ranking in plain words

You type “best way to teach children programming” into Google, and within a fraction of a second an ordered list of pages from different sites appears. But how did Google know those pages existed in the first place? And why did one page appear in first place and another on page three? Understanding this process is the first step in learning search engine optimisation (SEO), because you cannot improve something whose workings you do not understand. In this article we explain the three stages every page goes through before it appears in search results – crawling, indexing and ranking – what can go wrong at each stage, and how to check the state of your pages yourself.

The whole picture in three stages

Google explains in its official documentation that Search works in three main stages, and not every page makes it through every stage. That is an important point: having your page on the internet does not mean Google will show it automatically.

Stage one: crawling

There is no central list of every page on the web. So Google uses automated programs called crawlers – the best known is Googlebot – which move between pages through links, much as a person does, discovering new and updated pages. There are a few main ways Google discovers your page: a link to it from a page it already knows, on your site or someone else’s; a sitemap, a file listing your site’s important pages; or your own request through Search Console.

What gets in the way of crawling?

• The robots.txt file: a file at the root of the site that tells crawlers which paths not to visit. One wrong line in it can block a whole site.
• Orphan pages: a page no internal link points to is hard to discover.
• Server errors: if the site is very slow or returns repeated errors, crawlers may visit less often.
• Content behind a login: a crawler does not log in, so it will not see what sits behind a password.

Stage two: indexing

After crawling, Google tries to understand the page: its topic, text, headings, images, image descriptions and videos. It renders the page as a browser would, including running JavaScript, to see the real content. Then it decides whether to store it in the index, a very large database that results are drawn from.

An important task at this stage is dealing with duplicate content. If Google finds several very similar pages, it picks one to be the “canonical” page and usually shows that one in results instead of the others. You can help it choose with a canonical tag on the page.

What prevents indexing?

• A noindex tag: an instruction in the page that explicitly asks not to be indexed. Useful for pages such as a shopping basket, and disastrous if placed by mistake on an important page.
• Thin or duplicated content: Google does not index everything it crawls; pages of little value may be left out.
• Technical rendering problems: content that only appears after a user interacts may not be seen.

Stage three: serving and ranking

When someone types a search, Google’s systems look through the index for matching pages and order them by what they judge most useful to that particular searcher. Google does not publish its full ranking recipe, but its public documentation describes the main groups of factors:

Notice that most of these factors are about serving the searcher, not tricking the search engine. That is why attempts at manipulation, such as keyword stuffing or buying links, fail in the long run and can bring penalties.

The following video from Google sums up these stages in five minutes:

What about multilingual sites?

If your site is in Arabic and English, like Badr Academy Education’s, there is an extra question: how does Google know which version to show each searcher? The first rule is to give each language its own page with its own URL, rather than one page whose text changes at the press of a button. The second rule is to link the versions together with hreflang tags – signals in the page’s code that tell Google “this is the Arabic page, and this is its English translation”. That way an Arabic searcher sees the Arabic version in results, an English searcher sees the English one, and the two translations are not treated as duplicate content. And remember that literal machine translation without review usually produces weak text that does not serve the reader, however correct the technical signals are.

How do you check the state of your pages?

The free Google Search Console is your official window onto your site’s relationship with Google. Once you have verified ownership of the site, you can:

• Use the URL Inspection tool to see whether a page is indexed, when Google last crawled it, and whether a problem is preventing indexing.
• Review the indexing report, which shows how many pages are indexed and why others are excluded.
• Submit your sitemap so Google can discover your pages easily.
• Follow the performance report, which shows the searches your pages appeared for and the number of clicks.

We explain how to use these reports in detail in measuring digital marketing.

Common mistakes on new sites

• Expecting to appear immediately: crawling and indexing a new site can take from days to weeks, and that is normal.
• Blocking crawling by mistake after launch: many sites are developed hidden from search engines, and lifting that block is forgotten at launch.
• Believing that paying Google improves organic ranking: paid ads are entirely separate from organic results.
• Neglecting internal links: links between your pages help crawlers discover them and readers move between them.

A quick checklist for your site

1. Search Google for site: followed by your domain name to see a rough estimate of indexed pages.
2. Open your site’s robots.txt and make sure it does not block important pages.
3. Make sure you have a sitemap and submit it in Search Console.
4. Inspect your five most important pages with the URL Inspection tool.
5. Make sure every important page is reached by at least one internal link.

Once you understand how a search engine works, the next step is understanding what SEO is in all its parts, then learning keyword research. In the digital marketing and SEO course at Badr Academy Education we apply all of this to real sites. For further resources visit the learning library, and for an ordered learning plan try the Find your path tool.