#1374 2008 · Billion Prices Project (MIT/Harvard; PriceStats) · Economic measurement / data
Economists measured inflation daily by scraping prices retailers already post
the problem
Official inflation takes months and field agents — and in some countries it is lied about
background
Consumer price indices are built the way they were built a century ago: statistical offices send agents to stores to record prices monthly, then publish with a lag. The method is slow and, where a government manipulates the statistic, unverifiable from outside — Argentina's official index after 2007 famously understated inflation, and publishing alternatives risked fines.
In 2008 Alberto Cavallo and Roberto Rigobon of MIT founded the Billion Prices Project: software collecting the posted prices of large online retailers daily — hundreds of retailers in more than 50 countries — with the same data commercialized through PriceStats (now part of State Street) as real-time inflation indicators.
what everyone would do
Commission a larger survey or model the official series harder — still slow, still dependent on the very agency whose numbers you need to check.
what they saw
Inflation statistics needed months and a government's honesty. The prices were already online, posted daily by retailers — scrape them, and the index becomes both faster and independent of whoever it measures.
the move
The price data already existed; retailers post every price online for their own sales. BPP replaced clipboards with scrapers: nightly reads of entire catalogs, every variety, every day. The project published its first daily index for Argentina in 2008 (where the gap against the official series became itself measurable), a daily US index in 2010, and from 2011 PriceStats has published daily inflation indices for 22 countries on a three-day lag.
why it works
Scraping removes the two binding constraints of statistical collection: labor (agents visiting stores) and time (monthly cycles). A bot reads entire catalogs nightly at near-zero marginal cost, so the index can be daily and cover every variety sold. Because posted online prices co-move with offline levels in most sectors, the series tracks official CPI where that is honest — and where it is manipulated, the divergence becomes visible in real time rather than argued about annually. Commercialization through PriceStats funds the collection, and academic publication keeps the methodology open.
the payoff
Daily prices from hundreds of retailers in 50+ countries since 2008; PriceStats publishes real-time inflation for 22 countries at 3-day lag
where it breaks
Online prices skew toward sectors that sell online, so the basket differs from a national CPI; quality adjustment is cruder than statistical offices' hedonic methods; retailers redesign sites and scrapers break, so maintenance never ends; and in economies with thin e-commerce or large informal sectors, scraping sees almost nothing. It also measures posted prices, not the negotiated and couponed prices people actually pay.
what came after
Scraped-price indices now supplement central-bank statistics; the Argentina daily series (Inflacion Verdadera, 2007-15) became the standard exposure of official understatement; the work seeded big-data measurement across economics.
references
- [1]The Billion Prices Project: Using Online Prices for Measurement and Research (NBER Working Paper 22111; JEP 30(2))National Bureau of Economic Research, 2016nber.org
- [2]The Billion Prices Project — data and design (Cavallo, Deutsche Bundesbank Statistics Forum slides)Deutsche Bundesbank, 2015bundesbank.de