The encyclopedia · Software & IT · Technical decision · 1992–2003
UTF-8 won by being a drop-in superset of ASCII, so no one had to upgrade all at once.
Ken Thompson and Rob Pike designed UTF-8 in one evening so any ASCII byte stayed untouched, letting existing systems adopt Unicode without rewriting files.
Ken Thompson · Rob Pike · X/Open Consortium
the move
By 1992 the industry had two parallel standards for encoding the world's scripts, ISO 10646's UCS and Unicode, but no clean way to put them into the byte-oriented files the Unix world already used. The emerging UTF from ISO 10646 could produce null bytes and the ASCII slash, which made it incompatible with existing Unix file systems and programs.
Late one afternoon Ken Thompson and Rob Pike were asked to vet an X/Open committee's FSS/UTF design. They saw a chance to design a better encoding and have the committee push it out instead. They went to dinner, Thompson worked out the bit-packing on a placemat, and by Wednesday night the scheme was agreed; Plan 9 was converted to it within days.
The trick was a set of criteria that made the encoding a true ASCII superset: no ASCII code value appeared inside the multibyte representation of another character, and the byte stream could be synchronized from the middle. That meant historical file systems and existing programs kept working unchanged, so nobody had a reason to resist.
They mailed an outline to X/Open, which adopted the scheme and steered it toward what became RFC 2279 and later RFC 3629, the standard that carried UTF-8 across the whole web and into nearly every modern system.
why it works
- ASCII bytes map to themselves, so old software and file systems read the same text bit-for-bit.
- Non-ASCII characters never contain an ASCII byte, so a slash or NUL a program depends on cannot hide inside a name.
- A decoder can find character boundaries from the middle of a stream, so a truncated or corrupted file stays recoverable.
- Thompson and Pike got X/Open to adopt and push the scheme, giving it an institutional runway into the RFCs.
what transfers
A new format wins fastest when it is a strict superset of what everyone already has, not a replacement that forces hundreds of systems to flip at once.
what came after
UTF-8 became the dominant text encoding on the web and in modern operating systems, and its multi-byte safety was tightened with strict validation in RFC 3629. The single backward-compatible design decision is why it displaced every rival byte encoding without a coordinated migration.
references
- UTF-8 history: original criteria for a transformation format of UCS (1992)
- The history of UTF-8 as told by Rob Pike
spotted an error? The archive wants to know.