#419 2017 · GitLab · Software / DevOps infrastructurecostly-signal
GitLab lost 300GB of customer data and every one of its backup systems at once — then livestreamed the recovery to thousands of strangers instead of hiding it.
the problem
a company suffers a severe, embarrassing operational failure that would normally be managed through careful, delayed, lawyer-reviewed public communication
background
On January 31, 2017, a tired GitLab engineer, trying to clear a replication issue, accidentally deleted the company's primary production database directory instead of a secondary replica, wiping roughly 300GB of live data including projects, comments, issues and user accounts. As the team scrambled to restore service, they discovered a second, far worse problem: all five of GitLab's official backup mechanisms — scheduled dumps, disk snapshots, replication — had been silently failing for weeks, some due to version mismatches, without ever triggering an alert.
The standard corporate response to a disaster this severe — real data loss, compounded by a total backup failure the company hadn't even known about — is to go quiet, manage the story carefully, and issue a controlled statement once the facts and legal exposure are fully understood. GitLab's team, mid-crisis, made the opposite call: rather than retreat behind closed doors while they figured out what happened, they kept the incident visible to the public in real time, while it was still unresolved and the outcome uncertain.
the move
GitLab opened a live, publicly editable Google Doc tracking the incident as engineers worked it, live-tweeted updates, and streamed the entire database recovery process on YouTube — including the discovery that all backups had failed and the eventual recovery relying on a six-hour-old disk snapshot one engineer happened to have taken for an unrelated load-balancing test, not any of the systems designed for exactly this scenario. GitLab followed up with an extraordinarily detailed public postmortem naming every failure point without obscuring its own errors.
the payoff
The recovery livestream drew over 5,000 concurrent viewers at its peak, and rather than damaging GitLab's reputation, the transparent handling drew broad public sympathy and respect across the tech industry — commentary at the time and since consistently describes GitLab's reputation as effectively unharmed by an incident that, handled opaquely, would likely have been a serious trust crisis. GitLab followed the postmortem with dozens of public engineering issues tracking concrete fixes, giving customers direct visibility into exactly what was changing.
what came after
The GitLab 2017 postmortem is now a standard reference in site-reliability engineering and incident-response training for the principle that radical, real-time transparency during a failure can outperform narrative control — cited alongside blameless postmortem culture as a template other engineering organizations, including AWS, Cloudflare and Slack, have explicitly cited when designing their own public incident communication practices.
references
- [1]Postmortem of database outage of January 31GitLab, 2017about.gitlab.com
- [2]The benefits of transparency: Interview with Sytse "Sid" Sijbrandij, CEO of GitLabIncrement, 2019increment.com