The encyclopedia · Software & IT · Technical decision · 2009–2010
Spark kept data in memory between jobs and beat Hadoop 10x on iterative work
UC Berkeley's Spark introduced RDDs that cache working sets in memory; it ran iterative ML jobs 10x faster than Hadoop and queried 39 GB in sub-second time.
UC Berkeley AMPLab · Apache Spark
The solution
MapReduce succeeded at large-scale batch jobs, but its acyclic model forced every job to reload data from disk — painful for iterative machine learning and interactive queries.
Spark's RDDs let users cache a working set in memory and reuse it across operations, with lineage-based rebuilds for fault tolerance.
The paper reported 10x speedups over Hadoop on iterative machine learning jobs and sub-second interactive queries over a 39 GB dataset.
Why it worked
- Memory reuse removes the dominant I/O cost in iterative workloads.
- Lineage gives fault tolerance without replication overhead.
- One abstraction covered both batch and interactive use.
- Cache decisions stayed explicit, so users controlled the tradeoff.
What can be applied
When the same data is used repeatedly, the bottleneck isn't compute — it's re-reading what you already loaded; cache what's hot and rebuild only what breaks.
Aftermath
Spark became Apache's widely used cluster-computing engine, and the RDD abstraction shaped a generation of data systems and the Databricks company that grew around it.
Sources
spotted an error? The archive wants to know.