Ramp cut Python CI time by tracing which classes a test actually touches

Ramp's Python monolith test suite was creeping toward 20 minutes as the team scaled past 5,000 tests, mostly functional/integration style. Rather than rewrite the suite, the infrastructure team attacked runtime directly: raise CI concurrency by fixing a database-setup bottleneck that limited parallel workers to four, stop tearing down and rebuilding the test database from scratch between local runs, and build an incremental test runner that only executes tests relevant to a given code change.

The incremental runner's first version used coverage.py to map, for each test, which lines of code it executed, then on a new commit diffed changed files against that map. It worked, but had a specific gap: Ramp's codebase uses Flask-SQLAlchemy model classes, and coverage measurement attributes almost all execution to SQLAlchemy's internals rather than to the model class declaration itself. Adding a column to a model didn't register as 'this class changed' under pure line coverage, so the incremental runner could skip tests that should have caught a schema-changing bug.

To fix it, the team replaced coverage.py with Python's built-in sys.settrace, inspecting `call` events for a `self` or `cls` argument, resolving that object to its class, and doing a breadth-first walk up `__bases__` to map every ancestor class to the file where it was defined. That let the snapshot record not just which lines a test executed, but which classes in its full inheritance chain it depended on — so a change to a base class's declaration correctly invalidated tests for every subclass, even when no subclass code path was directly exercised.

Combined with the concurrency fix and skipping full database rebuilds, the incremental runner covered 30-40% of the suite on a typical commit and caught roughly 90% of the failures a full run would catch, giving engineers actionable signal in a few minutes instead of the full 10-20.

Trace class inheritance at runtime so an unexercised base-class edit still triggers its tests

Coverage.py records executed lines, not the classes they belong to, so execution inside SQLAlchemy's ORM internals doesn't get attributed to the user's model class.

sys.settrace exposes call frames directly, so inspecting `self`/`cls` recovers 'which class is this' independent of where the line-level execution happened.

Walking `__bases__` from that class turns one call event into a full ancestor list, so a base-class change is attributed to every file in the chain.

Running the incremental runner alongside the full suite, not instead of it, meant the ~10% detection gap never became a production risk.

A coverage-based dependency shortcut can miss real dependencies, like inheritance, that coverage tools don't attribute to the changed code — check for that blind spot before trusting the fast path.

Ramp judged the incremental runner not worth further investment past this point — closing the remaining ~10% detection gap would need restructuring around something like Bazel, not the highest priority then. The concurrency fix (doubling machine size after removing a redundant per-worker database reset) alone halved CI time, and skipping full database rebuilds locally shaved 10-30 seconds per run. The team published the sys.settrace-based inheritance traversal as an alternative to open-source options like pytest-testmon, which lacked Ramp's parallelization support.