Netflix paid outsiders to beat its own algorithm

By 2006, Netflix's DVD-by-mail recommendation engine, Cinematch, was central to the business — better recommendations meant customers kept finding movies they liked and kept subscribing — but the algorithm's accuracy had plateaued despite Netflix's own data science team continuing to work on it.

Rather than continue scaling its internal team, Netflix considered a different lever: put its actual training data — real, anonymized user ratings — in front of far more researchers than it could ever employ, and pay only for a result that beat a clearly defined bar, with no obligation to pay anything if no one succeeded.

Published real anonymized ratings and offered $1M for a 10% accuracy gain, letting thousands of outside teams compete and paying only on success.

Publishing real data attracted 40,000+ teams, far more diverse than any internal team.

Prize was contingent on beating a clear benchmark, so Netflix paid only for success.

Annual progress prizes surfaced incremental gains, letting Netflix deploy improvements early.

Cost of prize was far less than hiring equivalent talent in-house for three years.

When internal R&D plateaus, open your data and pay only for results that beat your benchmark, turning fixed costs into contingent ones.

More than 40,000 teams from over 180 countries entered over three years, and Netflix folded incremental gains from the annual Progress Prize leaderboard into its production algorithm well before any final winner was declared. In 2009, a merged team called BellKor's Pragmatic Chaos crossed the 10% threshold and won the $1,000,000 prize — a fraction of what recruiting an equivalent concentration of machine-learning talent in-house for three years would have cost, and Netflix had already captured much of the accuracy improvement for free during the competition itself.