案例库 · 软件与 IT · 技术决策 · 2003–2008
这条还没译成中文，下面是英文原文。
Google made massive data processing into a two-function map-and-reduce chore.
Google's MapReduce let programmers crunch huge datasets by writing two functions; the runtime split the work across cheap servers and retried failures.
那一手
Google's crawl and log data grew past what any single server could hold or process, and the natural answer was a larger, more expensive server or a bespoke parallel program no one wanted to write.
Dean and Ghemawat's library, first written in February 2003, let a programmer supply just two functions; the runtime split the input, shipped it out, sorted by key, and retried anything that failed.
By late 2004 Google was running hundreds of distinct MapReduce programs on up to tens of thousands of machines for tasks from Google News and Zeitgeist to large graph and machine-learning jobs.
The paper, published in OSDI 2004 and later in Communications of the ACM, became the template for big-data processing and was re-implemented as the open-source Hadoop, which made the model an industry default.
为什么管用
- Splitting a job into map and reduce lets ordinary engineers parallelize without writing threads or message passing.
- Keeping data on the same machine as its computation avoids shipping terabytes across the network.
- Retrying failed map tasks makes a farm of unreliable PCs behave like one dependable machine.
可以搬走什么
When data is too large for one box, decompose the job into embarrassingly parallel pieces and let a scheduler hide hardware failures, instead of buying a bigger, costlier machine.
后来呢
MapReduce became Google's standard batch model and, through Hadoop, the engine that established the big-data era across industry. Google later added higher-level systems (FlumeJava, Dataflow) as interactive SQL replaced the two-step batch model, and MapReduce itself was eclipsed, but its core idea that a large job can be two functions became permanent.
资料来源
- MapReduce: Simplified Data Processing on Large Clusters (OSDI 2004)
- Sorting the World: Google Invents New Way to Manage Data
发现哪里写错了？告诉我们。