#407 2006 · Netflix · Streaming media / softwarewrong-door
Netflix couldn't out-hire the world's data scientists to fix its recommendation engine, so it paid the world's data scientists to compete against each other instead.
the problem
an internal engineering team has plateaued on a hard optimization problem and more in-house hiring keeps producing diminishing returns
background
By 2006, Netflix's DVD-by-mail recommendation engine, Cinematch, was central to the business — better recommendations meant customers kept finding movies they liked and kept subscribing — but the algorithm's accuracy had plateaued despite Netflix's own data science team continuing to work on it. Hiring more machine-learning engineers to compete for the same marginal accuracy gains was expensive and offered no guarantee of a breakthrough; the underlying prediction problem, estimating how a given user would rate a movie they hadn't seen, was hard enough that specialists across academia and industry had never solved it much better than Netflix already had.
Rather than continue scaling its internal team, Netflix considered a different lever: put its actual training data — real, anonymized user ratings — in front of far more researchers than it could ever employ, and pay only for a result that beat a clearly defined bar, with no obligation to pay anything if no one succeeded.
the move
In October 2006, Netflix published roughly 100 million anonymized movie ratings from its own users and offered $1,000,000 to the first team that could beat Cinematch's prediction accuracy by 10%, measured by a fixed error metric, with an interim $50,000 Progress Prize each year for the best improvement so far to keep momentum going during the multi-year contest.
the payoff
More than 40,000 teams from over 180 countries entered over three years, and Netflix folded incremental gains from the annual Progress Prize leaderboard into its production algorithm well before any final winner was declared. In 2009, a merged team called BellKor's Pragmatic Chaos crossed the 10% threshold and won the $1,000,000 prize — a fraction of what recruiting an equivalent concentration of machine-learning talent in-house for three years would have cost, and Netflix had already captured much of the accuracy improvement for free during the competition itself.
what came after
The Netflix Prize became the reference case for open-data crowdsourced R&D competitions, directly inspiring platforms like Kaggle and government and pharmaceutical open-challenge programs — cited alongside the Goldcorp Challenge as proof that publishing a hard problem's real data to outside strangers can out-produce solving it behind closed doors, even though Netflix itself later judged the final winning ensemble too complex to fully deploy and instead used it as a foundation the production system evolved from.
filed under
references
- [1]History of Crowdsourcing: The Netflix PrizeHeroX, 2020herox.com
- [2]Netflix Never Used its $1m Algorithm. Here's Why.The Next Web, 2012thenextweb.com