SoundCloud shadow-compared a new RPC protocol against production before ever serving it

SoundCloud's backend ran on microservices with hand-built REST/JSON interfaces, easy to debug but leaving every endpoint's conventions up to the team that built it — HTTP method, JSON field names, timestamp formats, error shapes. Internal guidelines only helped so much: a client once sent camelCase fields to a server expecting snake_case and caused an outage, and understanding any endpoint meant reading application code rather than a contract.

SoundCloud built Twinagle, a Scala/Finagle implementation of Twitch's Twirp RPC protocol, so engineers could define an API once in protobuf and generate server and client code from it. They migrated their high-traffic track authorization service first. Rather than cutting over directly, they ran REST and Twirp side by side behind a percentage-based flag: at low rollout percentages, both endpoints were called and compared, with only the REST response returned, so mismatches surfaced as logged discrepancies, not user-facing bugs.

Mirror traffic to the new endpoint, diff it, then cut over

Generating server and client code from one protobuf contract removed the class of bugs caused by teams inventing their own JSON field-naming and HTTP conventions.

Running both endpoints in parallel at a small traffic percentage before switching over let the team catch response mismatches without any user seeing a wrong answer.

Starting with a single-client, high-traffic service kept the first migration's integration surface small while still proving Twinagle under real load.

Keeping existing business logic untouched and confining new code to a thin serialization layer limited where new bugs could hide.

Before retiring a working system, make the replacement answer every request in parallel and diff it against the incumbent — the bugs surface while the old system still serves the response.

The shadow-comparison rollout caught several critical bugs before they reached users, and SoundCloud ramped the track authorization service to 100% Twirp traffic with no outages and no bugs on the first attempt. Bandwidth between the two services dropped more than 40% from switching JSON to binary protobuf, and the team deleted all of its custom JSON serialization code for the service. SoundCloud went on to convert its other services to Twinagle, and made it the default for all new services going forward.