Skip to content

Frédéric Matty

Web, logiciels et culture numérique

How to Explore a Website Efficiently with a Well-Organized Sitemap

An XML sitemap tells search engines about URLs, but a URL listed in the sitemap isn’t automatically crawled or indexed. Google makes this clear…

Femme consultant un plan de site imprimé sur un bureau en bois dans un bureau à domicile organisé

An XML sitemap declares URLs to search engines, but a declared URL is not automatically crawled or indexed. Google makes this clear in its documentation: the file is a signal, not a directive. Starting from this understanding changes how you design and manage a sitemap.

Cross-reference your sitemap with crawl reports to find orphan pages

An orphan page is a URL listed in the sitemap with no internal links pointing to it. Crawlers discover it in the XML file and may visit it, but then abandon it for lack of relevance signals within the site architecture.

To find these pages, we export the full sitemap URL list and cross-reference it with the internal links report from a crawler (Screaming Frog, Sitebulb, or an equivalent tool). Any URL listed in the sitemap that does not appear as the destination of at least one internal link is effectively an orphan.

The Search Console crawl report provides another layer of insight. A URL marked “Discovered – currently not indexed” or “Crawled – currently not indexed” despite appearing in the sitemap points to a problem with internal linking or perceived value. If the Finance Technique sitemap lists hundreds of URLs, this kind of cross-referencing quickly identifies those that remain invisible despite being declared.

The solution is not to resubmit the sitemap. It is to add contextual internal links from pages that have already been crawled and indexed.

Man analyzing a hierarchical sitemap on a large screen in a modern open-plan office

Structure an XML sitemap around URLs that are genuinely useful

Including every URL on a site in the sitemap is a common mistake. Pagination pages, faceted filters, internal search results, and low-content tag pages dilute the signal sent to crawlers.

URL selection criteria

  • The page returns an HTTP 200 status code and is not blocked by robots.txt or a noindex tag. A URL returning a 301 or 404 in the sitemap wastes crawl budget.
  • The page has unique content that does not duplicate content on another URL on the site. Sorting and filtering variations should be excluded.
  • The page receives or should receive organic traffic. If it has no target queries and plays no role in internal linking, it does not belong in the file.

A shorter sitemap containing only strategic URLs delivers better results than an exhaustive file. Crawlers can spend their crawl budget on the pages that matter.

The lastmod field: reliable or ignored?

Google may use the lastmod tag if the date reflects an actual content change. Updating every date whenever the file is automatically generated nullifies this signal. Best practice is to update lastmod only when the page’s textual or structural content has changed significantly.

Google ignores the changefreq and priority tags. Bing takes the same position. Spending time configuring them achieves nothing. The only useful metadata is lastmod, provided it is accurate.

Sitemaps and IndexNow: complementary mechanisms for Bing

A recommendation published by Bing on July 31, 2025, clarifies how sitemaps and IndexNow work together. A sitemap serves as a permanent inventory of URLs. IndexNow provides real-time notifications when a page is created or updated.

The two protocols do not replace each other. A site that publishes several pieces of content a day should consider implementing IndexNow so Bing (and compatible search engines) can process new pages without waiting for the next sitemap crawl. The sitemap remains the baseline reference, checked periodically to verify the overall inventory.

IndexNow reduces the time between publication and discovery by the search engine, whereas a sitemap alone means waiting for the crawler’s chosen crawl schedule.

Two colleagues collaborating on a website sitemap on a laptop in a meeting room

HTML sitemap: its role in internal linking and crawling

An HTML sitemap (a page on the site listing links to its main sections) serves a different purpose from an XML file. It is intended for both users and crawlers.

Its technical value lies in internal linking. An HTML sitemap page, accessible from the footer or menu, provides an internal link to every top-level section. For crawlers, it offers an alternative entry point to areas of the site that may be poorly connected through the main navigation.

On a small site with strong internal linking, Google says it can discover pages without a sitemap. In that case, the HTML sitemap is primarily a navigation tool for visitors. On a large site or one with deep silos in its architecture, it complements the XML sitemap by providing an additional layer of internal links.

Regularly audit the consistency between your sitemap and the actual state of your site

A sitemap is not a file to generate once and then forget. Every page deletion, redirect, and content addition changes the gap between the file and the reality of the site.

  • Check monthly that the sitemap contains no URLs returning a 404 or 301 redirect. These entries consume crawl budget without delivering results.
  • Compare the number of URLs in the sitemap with the number indexed in Search Console. A significant gap (more URLs in the sitemap than in the index) points to a quality or internal linking issue.
  • Check that lastmod dates correspond to actual changes. A script that compares the lastmod date with the date the content was last modified in the database can automate this check.

The gap between declared URLs and indexed URLs is the first indicator of sitemap health. If most of the declared URLs are not indexed, the problem lies not with the file but with the site architecture or content.

A well-maintained sitemap remains an underused technical asset. Its value lies not in its mere existence, but in keeping it rigorously up to date and consistent with internal linking. Without this discipline, the XML file becomes an inventory disconnected from the reality of the site, and search engines eventually treat it as such.

How to Explore a Website Efficiently with a Well-Organized Sitemap