The encyclopedia · Software & IT · Technical decision · 2013–2016
Facebook's Zstandard compressed like zlib but ran several times faster
Zstandard paired finite-state entropy coding with a performance-first engine, beating zlib on speed, ratio and decompression at once.
Facebook (Meta)
the move
For two decades Deflate, the algorithm in zip, gzip and zlib, set the standard balance between compression speed and size. Rivals usually offered either a better ratio or better speed, rarely both, and Facebook needed both for its growing data centers.
The theoretical fix was asymmetric numeral systems (ANS), introduced by Jarek Duda: unlike Huffman coding, which rounds probabilities to powers of two, or arithmetic coding, which tracks a whole range, ANS uses a single natural number as its state. The 2014 paper reported decoding roughly 50% faster than Huffman with arithmetic-coding-class ratios, using only a few kilobytes of tables.
Facebook's Yann Collet turned the idea into Zstandard 1.0 in 2016, released under the BSD license. Benchmarks on the Silesia corpus showed that at the same compression ratio it compressed about 3 to 5 times faster than zlib, at the same speed produced files 10-15% smaller, and decompressed nearly twice as fast, with 22 tuning levels.
why it works
- A single-number state makes coding simple enough to be fast
- Exact probabilities keep compression close to the entropy limit
- Levels let one codec cover speed-critical and size-critical uses
- An open license spread the format across the industry
what transfers
A new theoretical idea pays off when someone turns it into a tuned engine: the winning move was combining ANS coding with careful modern-CPU implementation, not the algorithm alone.
what came after
Zstandard became a widely used open-source format for lossless compression, designed by its authors to fit nearly every lossless compression scenario; the same ANS family of ideas also fed later codecs and is now a standard tool in storage, logs and network transfer.
references
- Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding
- Smaller and faster data compression with Zstandard
spotted an error? The archive wants to know.