The solution
By 2013 Hacker News had published its ranking formula and even the Arc source code behind it, yet the front page behaved as if other forces were at work: popular stories suddenly vanished, NSA-titled items seemed to sink fast, and a thread could kill a submission once it grew argumentative. Users traded theories about shadow penalties, but nobody had proven that invisible factors existed, let alone measured them.
Rather than speculate or ask the moderators, Ken Shirriff treated the front page as the measuring instrument. He crawled /news and /news2 once a minute, within the site's two-pages-per-minute guideline, parsed the HTML with Beautiful Soup, and recomputed every story's raw score with the published formula. Wherever an article ranked lower than its score predicted, he inferred a hidden multiplicative penalty; since neighboring stories bracket each gap, repeated snapshots turned a noisy range into a bounded factor.
The measurements showed the published code was mostly accurate but far from the whole story: about 20% of front-page stories and 38% of second-page stories carried penalties. A .4 factor made each vote count as about .3 and dropped a story 66% faster; an automatic title penalty hit anything with NSA; and a controversy penalty that kicked in at 40 comments, not the 20 in the code, could push a #5 story to #22 within a minute.
The demonstration arrived when the system penalized the article describing the system: minutes after reaching the front page the post took a .2 penalty that would have left it near #5 and instead kept it off the front page entirely; on November 26 Shirriff reported that HN told him the voting-ring detector had fired erroneously. The thread grew to 920 points and 190 comments, the author logged roughly 25,000 page views, and a later editor's note recorded that HN no longer penalized NSA-titled stories.
What it achieved
Score every story with the public formula, then read rank gaps as measurable hidden penalties — the system's own output becomes the probe
Why it worked
A deterministic rank function turns any hidden penalty into a visible gap, so anomalies are measurements, not anecdotes.
Minute-by-minute crawling catches penalties the moment they land and brackets each factor's size between neighboring scores.
Concrete case studies (Snapchat vanishing, MongoDB stuck at #7) made an invisible system auditable by outsiders.
The stakes were real: about 20% of front-page stories carried a penalty, and a .4 factor left each vote counting as .3.
What can be applied
When a platform's behavior diverges from its published spec, don't argue or ask — measure the gap continuously; a deterministic formula turns invisible decisions into numbers anyone can audit.
Aftermath
Shirriff's own post became the live demonstration: minutes after reaching the front page a .2 penalty knocked it down, although without it the post would have ranked about #5; on November 26 he reported that HN told him the voting-ring detector had fired erroneously. The Hacker News thread grew to 920 points and 190 comments, with traffic around 25,000 page views. An editor's note later recorded that HN no longer penalized NSA-titled stories, and the finding that about a fifth of front-page stories carry penalties became the standard explanation for why popular items quietly disappear.
FOLLOW THE EVIDENCE