NVIDIA Tensor Cores made low precision accurate enough for AI and HPC

Deep learning made dense matrix multiplication the hot loop. The obvious answer was more ordinary floating-point units, but that would spend silicon and power on precision many models did not need.

With Volta in 2017, NVIDIA added Tensor Cores: 4 by 4 matrix multiply-accumulate units taking FP16 inputs while accumulating in FP32. Later numerical studies showed why the design mattered: it was not simply lower precision, but a dedicated mixed-precision operation.

Researchers then used iterative refinement to push the trick beyond neural nets. A Royal Society paper reported 4x to 5x speedups and about 5x better energy efficiency on a Volta GPU while maintaining FP64-level stability for linear solves.

Compute cheap, accumulate carefully

Matrix multiply is dense enough to justify a hardwired unit instead of only general CUDA cores.

FP16 inputs cut bandwidth and multiplier cost, while FP32 accumulation keeps the sum from collapsing immediately.

Iterative refinement turns approximate fast solves into accurate final answers.

Putting the path in CUDA libraries let ordinary software inherit the hardware trick.

When full precision is too expensive, split the job. Use low precision where errors can be repaired, and reserve high precision for the steps that police the answer.

Tensor Cores became a central NVIDIA differentiator across Volta, Turing and Ampere. The idea also escaped AI: mixed-precision solvers, scientific kernels and later TF32 workflows used the same pattern of fast approximate arithmetic plus accuracy repair.