The encyclopedia · Software & IT · Strategic decision · 2005–2006
Sitemaps let sites hand search engines their own URL list and won crawling
Google's 2005 XML sitemap let webmasters list their URLs for crawlers; Yahoo! and Microsoft joined in 2006 and it became universal.
the move
Large websites hide pages behind forms, JavaScript and weak linking, so crawlers following links alone miss them. In June 2005 Google introduced Sitemaps 0.84 so web developers could publish lists of links from across their sites — a simple XML file describing which URLs exist.
The protocol's design is deliberately minimal: the file must begin with <urlset>, contain one <url> entry per page, and each entry must include a <loc> URL, with optional <lastmod>, <changefreq> and <priority> tags. That XML is trivial for a content management system to generate, and auto-discovery through robots.txt means no extra configuration.
The adoption move was joint: in November 2006 Google, Yahoo! and Microsoft announced combined support for the protocol, and Ask.com and IBM followed in 2007. One file format satisfied every major crawler, so sitemaps became standard practice across the web.
why it works
- Webmasters know their URLs better than crawlers can discover them
- XML is trivial to generate from a content management system
- Google, Yahoo! and Microsoft agreed on one protocol in 2006
- Auto-discovery through robots.txt meant no extra setup
what transfers
Instead of asking crawlers to discover everything, let sites hand search engines their own index — the publisher has the best map of its content.
what came after
sitemaps.org hosts the protocol every major search engine reads; sitemap index files, image and video sitemaps and gz compression extended it, and it remains the standard way sites tell crawlers what to index.
references
spotted an error? The archive wants to know.