FlashAttention made exact attention faster by moving less data

Long-sequence Transformers were slow and memory-hungry because standard self-attention materialized large intermediate matrices and moved them repeatedly between GPU high-bandwidth memory and compute units.

The tempting fix was approximate attention: change the model's math to reduce complexity. FlashAttention took the opposite route. It kept exact attention, but processed blocks in SRAM and recomputed the running softmax statistics so the full attention matrix never had to be stored.

The paper reported a 15% end-to-end wall-clock speedup on BERT-large versus the MLPerf 1.1 training record, 3x speedup on GPT-2 at sequence length 1K, and 2.4x speedup on Long Range Arena tasks.

Optimize memory traffic, not math

The core constraint was IO between HBM and SRAM, not only floating-point operations.

Tiling let each block do useful work while the data was already on chip.

Keeping the algorithm exact avoided the quality regressions that made many approximate methods unattractive.

The same primitive unlocked longer contexts and new sequence tasks without changing the Transformer abstraction.

When hardware movement is the bottleneck, a better algorithm can be the one that computes the same answer while crossing the memory hierarchy fewer times.

FlashAttention became a standard building block for efficient Transformer training and inference. Its larger lesson spread beyond attention: modern accelerator algorithms often win by designing around memory hierarchy before inventing new model math.