The encyclopedia · Software & IT · Technical decision · 2004–2008
Google ran 60+ products on one Bigtable instead of a database per product
One sparse sorted map on thousands of commodity servers let every Google product tune its own schema and still share the platform.
the move
By 2004 Google's products each faced the same problem: storing huge amounts of structured data, from web crawls to satellite imagery to finance feeds. Building a custom storage system per product was expensive, and off-the-shelf databases could not reach the required scale, so the company designed one shared system.
Bigtable presents a simple model: a sparse, sorted map keyed by row, column and timestamp, with values stored as uninterpreted bytes. Applications decide their own layout, control data locality, and even choose whether data is served from memory or disk, while the platform handles partitioning, replication and failover on thousands of commodity servers.
By the time of the 2006 paper, more than sixty Google products, including Google Analytics, Google Finance, Personalized Search and Google Earth, used Bigtable. Clusters ranged from a handful to thousands of servers, storing up to several hundred terabytes each.
why it works
- One platform spreads engineering and operations cost across many products
- Applications control layout, so the model stays simple while uses differ
- Commodity servers plus automatic partitioning scale to petabytes
- A shared system means one team can harden it for everyone
what transfers
Before scaling a product, build the shared substrate once: a system that many teams tune beats many bespoke systems nobody can improve together.
what came after
Bigtable remained the backbone of Google's own services and later became a Google Cloud offering, 'the NoSQL pioneer': a managed key-value and wide-column store for latency-sensitive personalization, clickstream and IoT workloads, with HBase- and Cassandra-compatible APIs. Its ideas also shaped open-source clones such as HBase.
references
spotted an error? The archive wants to know.