robots.txt ran the web's politeness layer with no enforcement at all

In 1993-94 the young web developed a nuisance: web robots — wanderers, spiders — traversed pages at their own pace, hammering servers that had not agreed to be crawled, sometimes trapping servers in loops or swamping logs. The response was not a law or a protocol with enforcement, but a file: any server could publish /robots.txt, listing which robots were welcome on which paths, and crawler authors would honor it.

The format's own documentation is the clearest statement of its design. The June 30, 1994 consensus of the robots mailing list was explicitly 'not an official standard backed by a standards body, or owned by any commercial organisation. It is not enforced by anybody, and there is no guarantee that all current and future robots will use it.' It asked to be considered 'a common facility the majority of robot authors offer the WWW community to protect WWW servers against unwanted accesses.'

Voluntariness turned out to be the enforcement mechanism. Search engines — the crawlers with the most to lose from a reputation as a bad citizen — complied scrupulously, and compliance became the price of being a legitimate crawler at all. The convention ran for twenty-eight years before the IETF gave it a formal home: RFC 9309, 'The Robots Exclusion Protocol', published in September 2022, standardized what the file had to say and how it had to be parsed.

A voluntary file beat any enforced rule

A well-known path plus a trivial text format made compliance nearly free for both sides — no registration, no negotiation, no software for site owners.

The parties whose behavior mattered (crawler vendors) policed the norm themselves, because their businesses depended on being welcomed rather than blocked.

Voluntary rules scaled with goodwill: when a new crawler appeared, honoring robots.txt was how it announced it was not malware.

Leaving the standard unowned meant no one could weaponize it, which kept every search engine and publisher at the table.

A standard can win on alignment alone: when every player needs the same thing — predictable crawling, unbothered servers — a zero-cost convention at a well-known path outlives anything with teeth.

The convention held for decades as the web's de facto access-control layer for crawlers, and the IETF formally adopted it as RFC 9309, the Robots Exclusion Protocol, in September 2022 — a specification that ratified, almost word for word, what a mailing-list consensus had settled in 1994.