XML sitemap

An XML sitemap is a file listing the URLs you want search engines to know about, optionally with metadata like last-modified date. It does not force indexing, but it helps discovery, especially for large sites, new sites with few backlinks, sites with deep or poorly linked pages, and sites with content that changes often. You submit it in Search Console and reference it in robots.txt.

The value is in accuracy. A sitemap should list canonical, indexable, 200-status URLs only. Common problems are including non-canonical URLs, listing pages that are blocked by robots or marked noindex (a direct conflict), including redirected or 404 URLs, and stale last-modified dates that lie about freshness. Each of those wastes the engine's trust in the sitemap.

Limits matter at scale: a single sitemap caps at 50,000 URLs and 50 MB uncompressed, beyond which you use a sitemap index that points to multiple sitemap files. The cleanest approach is to generate the sitemap automatically on every deploy from the same source of truth as your routing, so it never drifts from reality. We check for orphans-in-sitemap, blocked-but-listed conflicts, and canonical mismatches.