Ramp cut its 45-minute test suite to under 5 minutes by chasing asymmetric bets

In July 2023, Ramp's Python monolith test suite took 45 minutes to run, and slow tests were backing up the merge queue across the company — engineers were blocking off half a day just to babysit pending changes. Two years earlier, an internal post had already flagged that slow tests hurt engineering productivity, but nothing had been done since.

Instead of committing to a big rewrite, the team first measured: about 38 of the 45 minutes came from the pytest invocation itself. Profiling showed the test run was CPU-bound, so the single first fix — faster CPUs — cut run time roughly 50% on its own, the largest single gain of the whole project and the one that took the least effort to find.

From there, engineers split up and ran cheap, parallel experiments scoped to whatever each already had context on: docker layer caching on self-hosted runners saved over a minute; a rewritten pytest-xdist work-splitting algorithm saved 2-3 minutes; more test databases and tuned Postgres connection settings fixed lock contention once over 20 workers hit the same database. Not every experiment paid off — mounting test data in RAM showed no gain and was dropped — but each was cheap enough to be worth the shot.

Chase cheap experiments with high potential upside, and stop the moment gains diminish

Measuring first (38 of 45 minutes was the pytest call itself, CPU-bound) meant the team fixed the actual bottleneck instead of guessing.

Doing the highest-leverage, lowest-effort fix first (faster CPUs, ~50% gain) built momentum and freed time to explore riskier ideas.

Splitting into parallel, low-cost experiments let the team test many hypotheses fast instead of betting everything on one big change.

Graphing time saved per change made the diminishing-returns curve visible, giving the team a concrete signal to stop rather than keep polishing for marginal gains.

Measure before you tune, fix the highest-leverage item first, then run a portfolio of cheap experiments in parallel — and use the shrinking size of each gain as your signal to stop.

After a week, the suite ran in under 5 minutes, roughly a 70%+ reduction from the original 45. New improvements by the end of the week were shaving only seconds (under 5%) off run time, which the team used as the signal to stop and move the remaining ideas into a long-tail backlog rather than keep tuning.