#893 1996 · The Unicode Consortium · Standards / computing

Unicode beat the endian deadlock: a marker that makes either byte order announce itself

问题

A text standard had to choose byte order, but neither endian camp would yield -- one choice would strand half the machines.

背景

A computer writes a multi-byte value either with its most significant byte first (big-endian, how most early workstations and the Internet's own machines stored them) or with its least significant byte first (little-endian, how Intel and the mass PC market stored them). Any text encoding standard that emits 16- or 32-bit code units had to serialize bytes in one of the two orders. The trouble was that the choice was not technical -- it was tribal. Big-endian machines and little-endian machines each read files effortlessly in their own order and saw the other's as scrambled, and neither camp had a reason to surrender its architecture, because the standard was meant to let all the world's text move between them.

Picking one order by vote would have been a defeat for the camp that lost, and worse, it would have locked every conformant file to an order that a large, fast-growing slice of the world's machines read backwards. The standard risked either enshrining one faction or fragmenting into two incompatible dialects that recreated the very problem it existed to solve.

换别人会怎么做

The obvious fix was to vote: the standard's committee picks the byte order with wider support and every conformant implementation converts on write or read. It fails on the same tripwire that created the problem -- the losing architecture now reverse-reads every standard file by default, so the standard either disenfranchises half the installed base or is ignored by them.

他们看到了什么

The committee stopped picking a winner and put the choice in the data: a leading marker any reader checks first, so either endian order is valid and self-declaring. It needn't win the argument -- only agree where the signal lives.

那一手

Unicode added a single codepoint, U+FEFF, the byte order mark (BOM), written first into the stream: the first two bytes of any UTF-16 or UTF-32 file signal which endian order the rest of the file uses. Big-endian starts with FE FF, little-endian with FF FE. No camp had to lose -- either output order is now a legal, interchangeable choice, and the file carries the proof of which one it is on its own front byte.

为什么管用

The mechanism moves the decision out of the committee room and into the artifact, where it can no longer split anyone. Because both orders are legal and each declares itself, every machine reads every file correctly in its own native order with one cheap read of the first two bytes -- so there is no losing side and no reason to defect from the standard. The BOM is read exactly once, at the front of the stream, before any of the payload, so its cost is fixed and negligible compared with re-architecting one camp's machines. The barrier every standards body dreads, a binary that makes a faction a permanent loser, simply disappears, because the standard no longer commits to an answer.

值了多少

The mechanism stuck: files may be either endian with the order self-announced, and both camps' machines interoperate without either surrendering.

什么时候会失灵

It fails when the two sides genuinely cannot interoperate because of the difference, not just disagree about it -- a marker declares which variant is in play but does nothing if the variants are semantically incompatible underneath. It also fails when uninterested parties stop reading the signal: a legacy contract that assumes one order ignores the BOM and misreads files of the other endianness, and some formats (like UTF-8, which has no byte-order problem) cannot use the marker to say anything. And it assumes every reader is willing to check the front byte every time -- a closed, single-vendor chain that has already settled on one order treats the marker as noise.

后来呢

The byte order mark still leads every UTF-16 and UTF-32 stream (and doubles as the leading signature of modern text files) decades on. Endianness became a colonial detail serialized into the data rather than a standards war -- a precedent for letting incompatible implementations coexist when the architecture between them cannot be settled by argument.

资料来源

[1]The Unicode Standard FAQ - Byte Order Mark (BOM), U+FEFFUnicode Consortium, 2024unicode.org
[2]RFC 2781 - UTF-16, an encoding of ISO 10646 (section 3.2, Byte order mark)IETF / RFC Editor, 2000rfc-editor.org