MarginLab's daily Claude Code benchmarks made degradation measurable

In September 2025 Anthropic published a postmortem of three infrastructure bugs - misrouted context-window requests, corrupted token output on TPU servers and a compiler fault in top-k sampling - that had intermittently degraded Claude for weeks. The report admitted its own evaluations never captured the degradation users were reporting, and that noisy evals and privacy controls made the incidents hard to find. Anyone building on Claude Code was left with a question the provider could not answer: is the product quietly getting worse?

The obvious answer is to trust the provider's dashboards or wait for occasional third-party model benchmarks. Margin Eval, the independent operation behind MarginLab, instead runs the real product: every day it drives the current Claude Code CLI on a curated, contamination-resistant subset of SWE-Bench-Pro, 50 tasks per run, and publishes pass rates with 95% confidence intervals, seven- and thirty-day rolling aggregates, significance thresholds and email alerts. Its stated rule is "what you see is what you get": no custom harness, so results reflect what an actual user would experience.

That design is what made the numbers usable as evidence. Because the harness is the shipped CLI, a measured drop means something a paying user would hit; because the statistics are explicit, the page can flag a real regression while ignoring daily noise. The launch on January 29, 2026 drew 760 points and 354 comments on Hacker News, and the submitter's first comment pointed to the tracker's own statistically significant ~4% drop over the previous month.

Measure the shipped product daily, not the model in a harness

Measuring the shipped CLI rather than the model API means any detected drop is one users would actually experience, which is what makes the number actionable.

Confidence intervals and declared thresholds keep a noisy 50-task day from being read as a regression, while sustained drops still clear the bar and get flagged.

Public methodology, declared independence and daily updates let the page serve as shared evidence in a debate otherwise made of anecdotes.

A continuous cadence with email alerts turns a one-off benchmark into a standing watchdog that outlives any single incident.

When an incumbent admits its own instruments were blind, build the independent instrument that measures the product as shipped; transparency earns trust, and trust is what gets a benchmark adopted.

The launch drew 760 points and 354 comments; the submitter opened the thread by citing a statistically significant ~4% monthly drop, and the debate over thresholds, sample sizes and multiple-comparison corrections treated the page as the shared reference. By September 2026 the tracker still ran daily, collecting a fresh Opus 5.0 baseline with detection paused and pass rates of 86%, 85% and 84% over one-, seven- and thirty-day windows from 50, 350 and 1,350 evaluations. A separate page preserves the historical series, and email alerts still fire on statistically significant drops.