NYC ranked 900,000 lots against fire data — inspection hit rates jumped fivefold

Mike Flowers, a former Manhattan prosecutor who had watched US military intelligence predict safe Baghdad routes by mining IED data, became New York City's first 'director of analytics' under Mayor Bloomberg. His first big target: illegal conversions, dwellings chopped into units housing up to ten times their design load — fire traps where packed residents regularly died, and where two firefighters died in 2005.

His five-person team of recent economics graduates — Flowers deliberately avoided veteran statisticians, wanting people who would not protest the unorthodox method — started from the city's 900,000 property lots and poured in data from 19 agencies: property-tax delinquency, foreclosure proceedings, utility anomalies and cuts, building type and age, ambulance visits, crime rates, rodent complaints. They matched it against five years of fire data ranked by severity to predict which complaints deserved urgent inspection.

The data was a mess: every agency identified buildings differently — building numbers, borough-block-lot, police Cartesian coordinates, fire-department call-box proximity. The team embraced the mess, matching buildings by a small Cartesian area at the front of the property and pulling in each agency's data loosely. Field work sharpened the model: brickwork-permit records, for instance, indicated some suspect properties were probably not major risks.

The system went live in June 2011, forwarding the top 5% of ranked complaints weekly. Previously inspectors found conditions bad enough to warrant a vacate order in 13% of follow-ups; after ranking, more than 70% of inspected buildings got vacate orders — a fivefold efficiency gain.

Rank all 900,000 lots against five years of fire data and send inspectors only to the top 5% of predicted risk.

The city had abundant fire outcome data that no one had ever joined to complaint data.

Veteran intuition proxies were misleading: a rat on the Upper East Side drew 30 calls in an hour, while Bronx residents needed a battalion of rodents before dialing 311.

Perfect record-matching was impossible across 19 agencies, so the team matched by geography and let volume compensate.

Ranking was tested continuously with veteran inspectors rather than imposed as a black box.

Let data volume swamp imprecision instead of chasing perfect entity resolution — and drop domain intuitions, like 311 call volume, the moment the data contradicts them.

By 2013, when the account was published as an excerpt of 'Big Data', the method was held up as the model for predictive city inspections, and Flowers' office kept expanding the approach across city agencies.