The encyclopedia · R&D & Science · Technical decision · 1977
Dempster-Laird-Rubin fit models with hidden data by alternating guess and maximize
Dempster, Laird and Rubin's 1977 EM algorithm fit maximum-likelihood models when data is missing by alternating between estimating hidden values and maximizing.
Harvard University
the move
Many models have variables that can't be observed — which cluster a point belongs to, the true value behind a censored measurement. Maximum likelihood is impossible when the data is incomplete.
The 1977 paper by Dempster, Laird and Rubin gave the field an algorithm: iteratively fill in the expected values of the hidden data (E-step), then maximize the parameters assuming those values are real (M-step). The likelihood is provably non-decreasing each round.
That guarantee is the point: EM needs only the current estimate, never the hidden truth, and is guaranteed to improve until it stops. It turned a hard, often unsolvable estimation problem into a loop anyone can run.
why it works
- Filling in hidden values lets maximization proceed even with missing data.
- The likelihood never falls, so the loop always makes progress.
- It needs no gradient over the latent variables, only a tractable conditional expectation.
- It handles clustering, mixture models, and imputation with one machinery.
what transfers
When unobserved data blocks estimation, don't discard it. Repeatedly impute the hidden part from a current guess and re-estimate, since each round only improves fit.
what came after
EM became a standard tool in statistics and machine learning, behind Gaussian-mixture clustering, latent variable models, hidden Markov model training and recommendation systems. Its guarantee and simple loop make it the default for incomplete-data maximum likelihood.
references
spotted an error? The archive wants to know.