Facebook built Haystack to store billions of photos with metadata in RAM

By 2010 Facebook had over 65 billion uploaded photos — 260 billion images once all sizes were counted — growing by a billion new photos each week. On the old network-attached storage, each read needed several disk operations to translate a filename to an inode and load metadata, so metadata access became the throughput bottleneck.

Haystack's design inverts that: photos are appended sequentially into large files (the haystack), each photo a 'needle' with only a few bytes of metadata — enough to locate it. All metadata is loaded into main memory, so a read touches disk at most once. Because photos are written once, read often, never modified and rarely deleted, the append-only layout fits the workload perfectly.

Haystack also replicated photos across storage machines for fault tolerance and cut reliance on expensive CDN reads. The OSDI 2010 paper reported it as Facebook's production photo store handling over 20 petabytes on inexpensive commodity hardware.