The encyclopedia · Engineering & Operations · Technical decision · 2011
Kafka became LinkedIn's data backbone by storing every message in a replayable log
LinkedIn's Kafka kept a durable append-only log that any consumer could rewind, instead of a queue that dropped messages after delivery.
LinkedIn · Apache Kafka · Confluent
the move
LinkedIn had an O(N²) data integration problem: many systems needed each other's data, and point-to-point connections were a maintenance and latency mess. A queue that deleted each message after delivery also couldn't let a new pipeline read history.
Kreps, Narkhede and Rao built Kafka as a distributed messaging system for log processing, published at NetDB '11. It used a partitioned, append-only commit log where each consumer advances its own offset, so messages can be replayed instead of being consumed once.
This made it the central nervous system for LinkedIn's activity streams, analytics and operational pipelines, and it opened the way to event sourcing and building state from a stream.
why it works
- An append-only log underlies the internal change log a database uses, and applying it to the message stream makes replay natural.
- Consumers keep their own offset, so a slow consumer doesn't make others lose messages.
- Storing records sequentially in partitions lets Kafka exploit sequential disk access and page cache.
- Because the log is durable, you can rebuild downstream state from it, not just push events onward.
what transfers
When lots of services need the same events and some lag, store the events once in a rewindable log and let every consumer keep its own position, instead of copying the data between each pair.
what came after
LinkedIn open-sourced Kafka in early 2011, it became a top Apache project, and Kreps, Narkhede and Rao founded Confluent to commercialize it. Kafka became the standard backbone for real-time data pipelines and event-driven architecture.
references
spotted an error? The archive wants to know.