Sitemaps and getting new pages found

You will be able to create or locate an XML sitemap and submit it in Search Console.

Mei Ling added twelve new cake pages to her site before the festive season, one for each design. A week later she searched for a few of them and found nothing. She had linked them from a single "Gallery" page, which was itself only reachable from the footer. Google would find them eventually by following links, but the season would be over by then.

A sitemap would not have guaranteed anything, but it would have told Google about all twelve pages on the day she published them. This lesson covers what a sitemap is, where to find yours and how to hand it to Google.

What a sitemap does

An XML sitemap is a file that lists the addresses of the pages on your site that you want search engines to crawl. It is written in a simple format that machines read easily, and each entry can include extra details, such as when the page last changed.

Lesson 1.1 explained that Googlebot discovers pages mainly through links and sitemaps. Links remain the main route, and lesson 3.3, Internal links, images and URLs, covered why important pages should be linked properly. A sitemap is the backup: a direct list that helps Google find pages it might otherwise reach slowly or miss.

Google's documentation says sitemaps help most on sites that are new and have few links pointing to them, sites that are very large, sites with many pages that are poorly linked to each other, and sites with lots of rich media such as video. A small, well-linked site may not strictly need one. It costs almost nothing to have one, though, and most small business sites benefit.

On the extra details, Google says it uses the last-modified date if it is consistently accurate, and it ignores the priority and change frequency values that some sitemap generators add. So there is no need to fiddle with those.

Find yours before you make one

Most website builders and content management systems create a sitemap automatically and update it whenever you add or remove a page. Before you generate anything, check whether you already have one.

Try typing your domain followed by /sitemap.xml into a browser. If that does not work, try /sitemap_index.xml, or look at your robots.txt file, where many sites list the sitemap address on a line starting "Sitemap:". Your website builder's help pages will also say where it puts the file. When you find it, you should see a list of addresses, sometimes split into several smaller sitemaps for pages, posts and products.

If your platform truly does not create one, free sitemap generators can crawl your site and produce the file for you to upload. The drawback is that a generated file does not update itself, so you have to repeat the process whenever pages change.

Include only pages you want indexed

A sitemap should list only canonical, indexable pages: the versions you want to appear in Google. Leave out pages with a noindex rule, pages that redirect elsewhere, duplicates that point to a different canonical and pages that return errors. Listing them sends Google mixed signals. Lesson 4.1, Robots.txt, noindex and canonicals do different jobs, explained why consistent signals matter.

Automatically generated sitemaps are usually good at this, but not always. Some include tag pages, author archives or test pages you would rather leave out. Open yours and scan the list. If you see pages you have set to noindex, or pages you did not know existed, check your builder's settings for which page types go into the sitemap.

Being listed is no promise of being indexed. Google says a sitemap helps it discover pages but does not guarantee every listed page will be crawled or indexed. Google still decides based on the page itself, as lesson 1.1 described.

Submit it to Google

There are two ways to tell Google where your sitemap is. The first is the Sitemaps report in Google Search Console. You enter the sitemap address and submit it. Search Console then shows whether Google could read the file, when it last did, and how many addresses it discovered. If there is a problem, such as a file that cannot be fetched or has formatting errors, the report says so.

The second is to add a line to your robots.txt file that reads "Sitemap:" followed by the full sitemap address. Any crawler that reads your robots.txt, including other search engines, can then find it.

Many sites do both. Submitting in Search Console has the advantage of the status report, which you will come back to during the technical health check in lesson 4.5.

After Mei Ling submitted her sitemap, Search Console showed it had been read and listed the number of pages discovered. A few of her cake pages appeared in Google within days and others took longer. She also added a "Cakes" link to her main menu, so the pages were no longer only reachable from the footer.

Find your own sitemap now, open it and read through the list before you submit anything, as the activity below asks.

Find or generate your sitemap, check it lists only pages you want indexed, and submit it in Search Console.

Course

Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).