The encyclopedia · Software & IT · Technical decision · 2013
Twitter and Cloudera's Parquet stored data by column and became big data's default
Parquet stores tables column-by-column instead of row-by-row, so analytics read only what they need and files compress far better.
Twitter · Cloudera
the move
Analytics over Hadoop read records row by row, so a query that needs one salary still pulls each employee's whole record, and heterogeneous records compress poorly. Twitter's own datasets nested seven levels deep with over 80 leaf nodes.
Twitter and Cloudera built Parquet, an open-source columnar storage format for Hadoop. Instead of rows it writes columns: all the names, then all the ages, then all the salaries. Its big feat was slicing nested data into columns using Google's Dremel record-shredding algorithm and a shared encoding spec.
Because a column holds one type, generic and type-specific compression both improve, and a query engine skips loading columns it does not need. Automatic dictionary encoding plus bit-packing and run-length encoding shrink the rest. Parquet ties users to no serialization library, so many engines adopt it.
Parquet 1.0 launched in July 2013 from contributors across Twitter, Cloudera, Criteo, Berkeley AMPLab and Stripe, and it was already running in production at Twitter.
why it works
- Homogeneous column values compress far better than mixed rows, so files get much smaller.
- Queries load only the columns they need, so I/O and cost drop sharply.
- Slicing nested data via Dremel means a complex schema still fits columns.
- Being framework-agnostic let every engine adopt it without rewriting.
what transfers
When the bottleneck is reading and compressing, change the physical layout, not the logic: storing homogeneous values together makes the expensive I/O shrink and the file smaller for free.
what came after
Parquet 1.0 shipped in July 2013 and quickly became the default columnar format across the Hadoop and Spark ecosystem, then inside data lakes, lakehouses and cloud warehouses. It is now one of the most broadly supported open storage formats, used wherever analytics need fast, cheap scans of wide tables.
references
spotted an error? The archive wants to know.