The encyclopedia · Software & IT · Technical decision · 2012–2018
Presto gave Facebook's analysts interactive SQL over petabytes of data
A distributed SQL engine streams results between workers in memory, turning multi-hour warehouse queries into seconds-to-minutes interactivity.
Facebook (Presto)
the move
Facebook's data warehouse ran on Hive over Hadoop, where a typical query could take hours: fine for batch, useless for an analyst trying to iterate. In 2012 a team led by Martin Traverso and others began building Presto, a distributed SQL engine designed for interactive queries on the same warehouse.
Presto's design avoids MapReduce's materialize-to-disk round trips. A coordinator plans queries and streams them through a tree of workers, where in-memory operators process data read directly from sources through the Connector API: HDFS and S3 today, RDBMS and NoSQL systems through the same interface, even across multiple systems in one query. Code generation and vectorized execution keep per-row costs low.
In production at Facebook since 2013, Presto grew to support user-facing analytics tools, dashboards and A/B-testing infrastructure, processing hundreds of petabytes and quadrillions of rows per day by late 2018, across clusters with thousands of workers.
why it works
- Streaming operators avoid costly disk round trips between stages
- One SQL engine federates HDFS, S3, RDBMS and NoSQL sources
- In-memory pipelining makes interactive latencies possible
- Code generation and vectorization keep CPU overhead low
what transfers
For analytics, interactivity is a product: an engine built to stream results between memory and CPUs can turn a batch culture into a query culture without changing the warehouse underneath.
what came after
Presto was open-sourced and became one of the most widely deployed query engines: by 2019 it ran at Uber, Netflix, Airbnb, Bloomberg and LinkedIn, powered Amazon Athena, and supported commercial distributions from Starburst, Qubole and Treasure Data. Its GitHub repo still introduces it as a distributed SQL query engine for big data.
references
- Presto: SQL on Everything
- prestodb/presto: The official home of the Presto distributed SQL query engine
spotted an error? The archive wants to know.