Crawling, indexing and serving: how a page gets into Google

You will be able to explain the three stages of Google Search and say which stage a missing page has failed at.

When someone in Singapore types tuition centre near Bishan into Google, the results appear in well under a second. That speed makes it easy to imagine Google checking every website on the spot. It does not. The results come from a stored copy of the web that Google built in advance, and a page that is not in that copy cannot appear, however good it is. Google's own documentation describes the process in three stages, and knowing them tells you where to look when a page is missing.

The first stage is crawling: finding pages and downloading them. Google uses a program called Googlebot that discovers new addresses mainly by following links from pages it already knows and by reading sitemaps, which are lists of URLs that site owners provide. Googlebot fetches each page and renders it much as a recent version of Chrome would, so it can see content that loads with JavaScript. If no page links to yours and it is not in a sitemap, Google may simply never learn it exists. If your server blocks or errors when Googlebot visits, the page may not be fetched.

The second stage is indexing: understanding what a page is about and storing it. Google reads the text, titles, headings, images and other elements, and works out the language and topic. It also groups pages that are duplicates or near duplicates and picks one of them as the canonical version, the one it may show in results. A tuition centre with the same timetable page at three slightly different addresses will usually see only one of them in Google. Google is clear that it does not index every page it crawls. Pages can be left out because they are low quality, duplicate something else, or are blocked by a noindex rule the site owner set, sometimes by accident.

The third stage is serving: choosing and ordering results when someone searches. Google's systems look through the index for pages that match the query and rank them on many factors. Google says these include what the query means, how relevant and useful the pages are, and the searcher's context, such as location, language and device. This is why the same search gives different results in Singapore and in Kuala Lumpur, and why a hawker stall's listing can appear for someone nearby but not for someone in Woodlands.

Two points from Google's documentation are worth knowing early. First, Google states that it does not accept payment to crawl a site more often or to rank it higher in the organic results. Ads are sold separately and are labelled as sponsored. Anyone offering a guaranteed top organic position for a fee is selling something Google says it does not offer. Second, Google does not guarantee that it will crawl, index or show any page, even one that follows all of its guidelines. SEO is the work of making each stage as easy as possible and the page as useful as possible, not a set of switches.

The three stages also give you a simple way to diagnose problems. If a page never appears in Google at all, start with crawling and indexing. Is it linked from other pages, listed in the sitemap, free of a noindex rule, and reported as indexed in Google Search Console? If the page is indexed but appears far down for the searches you care about, the issue is serving: relevance, quality, competition or intent. Most site owners jump straight to ranking tactics when the real problem is a page Google has not indexed.

You can see a rough version of what Google holds for your site right now. Type site: followed by your domain, with no space, into Google. The count it shows is an estimate and not exact, but if you expect 40 pages and it shows four, something at the crawling or indexing stage needs attention. Later in this module you will use Search Console to get the precise picture.

Your task: run a site: search for your own site or one you look after. Compare the rough number of results with the number of pages you expect, and list three important pages that do not seem to be there. You will check each one properly in lesson 1.5.

Search Google for site: followed by your domain, compare the rough number of results with the number of pages you expect, and list three pages that seem missing.

Course

• SEO and AI search