The encyclopedia · Software & IT · Technical decision · 1994–2022
Koster's robots.txt let websites tell crawlers what to skip, and it became the web's rule
After crawlers crashed his server, Martijn Koster proposed a plain-text opt-out file; crawler writers honored it and the web got a voluntary rulebook.
Nexor · Martijn Koster
the move
By 1993 the young web had crawlers that fetched pages automatically, and some hammered the servers they visited. Martijn Koster, running a server at Nexor, found his machine being downloaded at roughly a document per second by a web wanderer, and was complaining about repeated automated downloads as early as September 1993.
The obvious fix was to police crawlers one at a time: find the operator, call or write, and ask them to stop. That did not scale, and robots were multiplying. So on 25 February 1994 Koster posted 'Important: Spiders, Robots and Web Wanderers' to the www-talk mailing list, proposing a standard that 'doesn't require any server/client/protocol changes': a server could publish which parts of its URL space robots should not visit.
The file began life as /RobotsNotWanted.txt, then became robots.txt — the short name won because DOS file systems limited filename length. On 3 July 1994 Koster announced 'A Standard for Robot Exclusion'; most robots in operation already used it or had promised support. Each crawler simply fetched the file before crawling and obeyed its rules.
What made it stick was the design: a fixed, well-known URL, plain text, no central registry and no enforcement — the file works only because crawler writers agree to check it. Twenty-eight years later the IETF formalized the convention as RFC 9309 (2022), still the web's standard for how sites and bots negotiate access.
why it works
- Required no protocol, server or browser changes, so adoption was a single small change on the crawler side.
- A fixed well-known path gave every site owner one place to declare rules, and every crawler one place to look.
- Voluntary compliance aligned incentives: crawlers that ignored the file lost the goodwill and access that made them useful.
- Koster brought the major robot writers into the 1994 mailing-list discussion, so the de facto standard was born already adopted.
what transfers
When a rule cannot be enforced, design it so compliance is nearly free and the parties with power have an incentive to follow it; then the standard enforces itself.
what came after
robots.txt became the web's default access contract, honored by search engines and crawlers for three decades. In 2019 it turned 25; in September 2022 the IETF published RFC 9309, 'Robots Exclusion Protocol', written by Koster with Google's Gary Illyes and others, adding formal error-handling and caching rules. It remains central in the 2020s debate over AI crawlers and content licensing — a format that outlived every attempt to replace it.
references
spotted an error? The archive wants to know.