The encyclopedia · Software & IT · Technical decision · 2010–2013
Spark kept data in memory and recovered failures by replaying lineage, not replicating
RDDs persist intermediate results in memory and recover faults by recomputing from logged transformations, roughly an order of magnitude faster.
Apache Spark (UC Berkeley AMPLab)
the move
MapReduce made cluster programming easy, but its only way to reuse data between jobs was writing it back to a distributed file system, a heavy cost for iterative machine learning and interactive data mining. Two classes of applications, iterative algorithms and ad-hoc queries over the same dataset, suffered most.
RDDs gave programmers a restricted form of distributed shared memory: coarse-grained transformations applied to many items at once. Each RDD logs its lineage, the operations that built it, so if a partition is lost the system simply recomputes it instead of replicating it or logging fine-grained updates like earlier distributed memory systems.
Implemented in Spark at UC Berkeley's AMPLab, the abstraction handled PageRank, K-means and logistic regression, and the paper reported that keeping data in memory could improve performance by an order of magnitude for reuse-heavy workloads.
why it works
- In-memory reuse removes disk round trips between iterations
- Lineage recomputation avoids copying data across the network
- Coarse-grained operators keep the programming model simple
- One abstraction covers iterative, interactive and batch workloads
what transfers
Design the failure model into the abstraction: when rebuilding is cheap and copying is not, make the system remember how data was made rather than duplicate what it holds.
what came after
Spark became an Apache project and one of the industry's standard big-data engines: Apache Spark's site today describes a multi-language engine for data engineering, data science and machine learning, from batch and streaming to SQL and petabyte-scale exploratory analysis, on laptops or fault-tolerant clusters of thousands of machines.
references
- Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
- RDD Programming Guide: resilient distributed datasets
spotted an error? The archive wants to know.