SoundCloud tested every JVMKit release against all its dependent services first

SoundCloud's JVMKit library provided shared JVM patterns — service scaffolding, database and Kafka integration, metrics — to hundreds of internal Scala, Java, Clojure, and JRuby services. As adoption grew, the platform team lost visibility into which services depended on which parts of JVMKit's API, making every release a gamble: a routine update could silently break services the team didn't even know were affected.

Because dependencies at SoundCloud were declared explicitly in each repository's build file (sbt for JVM, Gradle for Android, npm for web), the team indexed every repository with Zoekt, a regex-based code search engine, refreshed every five minutes. A simple query against build files listing JVMKit as a dependency produced a live reverse dependency graph — every service that would be affected by a JVMKit change, generated automatically rather than tracked by hand.

With that graph in hand, the team built a verification pipeline: for each dependent service, clone the repo, compile and test it unmodified to establish a baseline, then swap in the candidate JVMKit version, apply any scripted fixes for known breaking API changes, and compile and test again. If the baseline passed but the candidate failed, the JVMKit change was the cause, not a pre-existing break in that service. The check ran nightly across the full dependent set and on every pull request against a faster, allowlisted subset.

Compile and test every caller before releasing the library

It replaced a manual, error-prone process (release, wait for teams to complain) with an automatic, falsifiable check that runs before the release ships.

Running an unmodified baseline first isolated whether a failure came from the JVMKit change or from a service that was already broken that day.

Splitting verification into a nightly full run and a fast PR-scoped run kept feedback quick without skipping full coverage.

The reverse dependency graph was generated from code, not maintained by hand, so it stayed accurate as services were added or removed.

When one internal library sits under hundreds of services, don't ship on hope — make the dependency graph queryable and let CI test every downstream caller first.

JVMKit continued to power hundreds of services and thousands of production instances at SoundCloud. With the verification pipeline in place, the platform team reported releasing JVMKit more often, with failures traceable to a specific step and a specific cause rather than surfacing as unexplained breakage downstream.