Twitter and Cloudera's Parquet stored data by column and became big data's default

Analytics over Hadoop read records row by row, so a query that needs one salary still pulls each employee's whole record, and heterogeneous records compress poorly. Twitter's own datasets nested seven levels deep with over 80 leaf nodes.

Twitter and Cloudera built Parquet, an open-source columnar storage format for Hadoop. Instead of rows it writes columns: all the names, then all the ages, then all the salaries. Its big feat was slicing nested data into columns using Google's Dremel record-shredding algorithm and a shared encoding spec.

Because a column holds one type, generic and type-specific compression both improve, and a query engine skips loading columns it does not need. Automatic dictionary encoding plus bit-packing and run-length encoding shrink the rest. Parquet ties users to no serialization library, so many engines adopt it.

Parquet 1.0 launched in July 2013 from contributors across Twitter, Cloudera, Criteo, Berkeley AMPLab and Stripe, and it was already running in production at Twitter.

Store by column so analytics skip what they don't need

Homogeneous column values compress far better than mixed rows, so files get much smaller.

Queries load only the columns they need, so I/O and cost drop sharply.

Slicing nested data via Dremel means a complex schema still fits columns.

Being framework-agnostic let every engine adopt it without rewriting.

When the bottleneck is reading and compressing, change the physical layout, not the logic: storing homogeneous values together makes the expensive I/O shrink and the file smaller for free.

Parquet 1.0 shipped in July 2013 and quickly became the default columnar format across the Hadoop and Spark ecosystem, then inside data lakes, lakehouses and cloud warehouses. It is now one of the most broadly supported open storage formats, used wherever analytics need fast, cheap scans of wide tables.