The encyclopedia · Engineering & Operations · Technical decision · 2006–2012
Google's Dremel stored data in columns, so petabyte queries answer in seconds
Dremel's column-striped storage plus a multi-level execution tree made trillion-row queries interactive; it became BigQuery.
The solution
In the mid-2000s Google's analysts could run huge computations with MapReduce, but each job took minutes to hours — too slow for exploring data interactively. Engineers wanted web-search speed for database-style questions over petabytes.
Dremel, in production since 2006, combined two ideas: column-striped storage that encodes nested records so a query reads only the fields it needs, and a multi-level serving tree that fans a query out to thousands of nodes and aggregates replies. The 2010 VLDB paper reports scans of 85-billion-record tables on 3,000-node clusters, with most queries finishing in under 10 seconds.
Google launched the technology as BigQuery at its I/O conference in May 2010, added a GUI and REST API in November 2011, and made it generally available in May 2012 — billed as letting users analyze terabytes of data without buying hardware.
Why it worked
- Columnar layout cuts bytes read: a query touches only needed fields.
- Tree execution parallelizes across thousands of commodity machines.
- Native SQL-like queries replace multi-step MapReduce pipelines.
- In-situ access reads data straight from GFS or Bigtable, no loading step.
What can be applied
Organize data around the questions you will ask, not the way it was written: reading only needed columns and pushing work down a tree turns jobs that took a batch into queries that fit a meeting.
Aftermath
BigQuery became one of the mainstream cloud data warehouses, used by thousands of companies, and Dremel's columnar format influenced later systems such as Apache Parquet. The paper remains one of the most-cited works in big-data systems.
Sources
- Dremel: Interactive Analysis of Web-Scale Datasets
- Google opens BigQuery cloud analytics to all
- Big Data Analysis Field Expands with Launch of Google BigQuery
spotted an error? The archive wants to know.