The encyclopedia · Engineering & Operations · Technical decision · 2008–2010
Google's Dapper traced one request across thousands of machines with sampling
Dapper gave every Google request a trace id passed between services, then sampled traces cheaply, turning guesswork about latency into a monitoring platform.
The solution
A single Google search or request could fan out across thousands of machines, dozens of services and several data centers, so a slow response was nearly impossible to explain from per-machine logs. Research tools existed (Magpie, X-Trace), but none met Google's need for low overhead and universal deployment.
Dapper's design made tracing nearly free: trace and span ids travel inside RPC metadata, a few common libraries do the instrumentation (so applications need no changes), and a sampling policy records only a fraction of traces — about one in 1024 by default. The result was application-transparent tracing at scale.
After two years in production, Dapper had evolved from a debugging tool into a monitoring platform: teams used it to find latency outliers, measure dependencies, and optimize storage and search paths. Its ideas directly inspired open-source successors like Zipkin and Jaeger, which brought the same approach to the wider industry.
Why it worked
- Sampling keeps overhead negligible at Google scale
- Span ids in RPC metadata make tracing application-transparent
- Tree reassembly shows the whole call path, not one server
- Tracing data became a platform for new monitoring tools
What can be applied
To debug a system too big to watch, watch a statistical sample and make identification cheap: a trace id in every RPC turns distributed latency into a queryable tree.
Aftermath
Dapper's design became the blueprint for distributed tracing — Zipkin and Jaeger were built on its ideas and became CNCF standards. Tracing is now a core pillar of observability in modern cloud systems.
Sources
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure
- Dapper, a Large-Scale Distributed Systems Tracing Infrastructure (paper PDF)
spotted an error? The archive wants to know.