Google's Dremel stored data in columns, so petabyte queries answer in seconds

In the mid-2000s Google's analysts could run huge computations with MapReduce, but each job took minutes to hours — too slow for exploring data interactively. Engineers wanted web-search speed for database-style questions over petabytes.

Dremel, in production since 2006, combined two ideas: column-striped storage that encodes nested records so a query reads only the fields it needs, and a multi-level serving tree that fans a query out to thousands of nodes and aggregates replies. The 2010 VLDB paper reports scans of 85-billion-record tables on 3,000-node clusters, with most queries finishing in under 10 seconds.

Google launched the technology as BigQuery at its I/O conference in May 2010, added a GUI and REST API in November 2011, and made it generally available in May 2012 — billed as letting users analyze terabytes of data without buying hardware.

Store data by column; answer in seconds

Columnar layout cuts bytes read: a query touches only needed fields.

Tree execution parallelizes across thousands of commodity machines.

Native SQL-like queries replace multi-step MapReduce pipelines.

In-situ access reads data straight from GFS or Bigtable, no loading step.

Organize data around the questions you will ask, not the way it was written: reading only needed columns and pushing work down a tree turns jobs that took a batch into queries that fit a meeting.

BigQuery became one of the mainstream cloud data warehouses, used by thousands of companies, and Dremel's columnar format influenced later systems such as Apache Parquet. The paper remains one of the most-cited works in big-data systems.