SoundCloud replaced 100 undocumented ETLs with six-principle data governance

By mid-2019, SoundCloud's data warehouse was six years of decisions that no longer made sense: a single small team owned more than 100 ETL pipelines on Amazon Redshift, most with little documentation and inconsistencies between them, so business and product teams couldn't reliably self-serve data. A contract to move to Google BigQuery gave the team a reason to start over, but an earlier 'lift and shift' migration attempt in early 2018 had already been aborted once.

The new Data Corpus team wrote a mission statement and six governance principles before rebuilding tables: quality, compliance, timeliness, usability, efficiency, maintainability, adapted from an ISO standard. Quality meant a tool that diffs a new ETL version against the deployed one so only the intended change shows up. Timeliness meant mapping landing times of every ETL and its dependencies; decoupling a needed table from an unrelated slower one removed four hours from the pipeline. Maintainability meant an '80 percent principle': build for the bulk of requests, not edge cases.

The rebuild happened in stages: an intermediate Corpus release in December 2019 still read from the old warehouse, Corpus Alpha in April 2020 was the first version fully independent of it, and Corpus 1.0 in July 2020 was the first release satisfying all six principles. By Corpus 2.2 in April 2021, the resulting BigQuery project used about 90 percent less storage than the raw data it read from, while stakeholders with basic SQL could self-serve most questions through documented, described tables.

Write governance principles before writing a single table

Automated version diffing turns 'did this change break something downstream' from a code-review judgment call into a mechanical check run on every ETL update.

The 80 percent principle gave the team a standing answer to scope-creep debates, so tables stayed simple enough for non-experts to query instead of accumulating edge-case complexity.

Mapping ETL dependencies and landing times turned a vague 'data is sometimes late' complaint into a specific fix, decoupling two tables, with a measured four-hour improvement.

Releasing in staged versions over roughly 18 months, each judged against the same six principles, let the team learn BigQuery's features before design decisions that were hardest to reverse.

When you migrate a system that grew undocumented, write down the principles that keep the new one from rotting the same way, then turn each into an automated check, not a guideline nobody enforces.

SoundCloud continued releasing further Corpus versions after 2.2, extending table coverage as the product evolved rather than treating the governance model as a one-time migration deliverable. The team frames Corpus as a continuously evolving project and explicitly expects the current design to eventually look as outdated as the pre-migration warehouse did, keeping the same principles as the standing framework for deciding when to change it.