The encyclopedia · R&D & Science · Technical decision · 2017
Google Brain dropped recurrence so every word could attend to every word
The Transformer replaced recurrence with self-attention, training far faster and beating prior translation models on a fraction of the compute.
Google Brain
the move
Sequence models in 2017 were built on recurrence or convolution, which process a sequence one piece at a time. Recurrence forces the model to wait for step one before it can do step two, so long-range context is slow to learn and hard to parallelize across hardware.
The Transformer drew on an idea called attention and pushed it to its logical end: drop the recurrence entirely and let every position compare itself against every other position in one operation. Each word can weigh how much every other word matters before it takes the next step.
Because the whole sequence can be processed in parallel, training scales across many GPUs far more easily. On the WMT 2014 English-to-German task the model beat the previous best by over 2 BLEU points, and on English-to-French it set a single-model record after 3.5 days on eight GPUs — a fraction of the training cost of the then-leading models.
The insight was a simplification, not a bigger model: by removing the sequential bottleneck the network could learn long-distance relationships cheaply and at scale.
why it works
- Attention allows every token to contextually relate to every other token in a single parallel pass, no matter how far apart they are.
- Parallel processing means the model trains faster and uses existing hardware far more efficiently.
- The architecture generalizes, reaching strong results on translation and constituency parsing with no task-specific changes.
- Dropping recurrence removed a structural bottleneck rather than tuning a hyperparameter, an architectural change that paid off everywhere.
what transfers
Replacing a serial process with a parallel one is a structural win — architecture, not just data, sets the ceiling on what you can learn.
what came after
The Transformer became the architecture beneath nearly all modern language models, including the large chat models of the 2020s, because it scales with data and compute in a way recurrent networks could not. Its replacement of recurrence with parallel attention is one of the most consequential single architectural decisions in modern software.
references
spotted an error? The archive wants to know.