The encyclopedia · Software & IT · Technical decision · 2002
Paul Graham ended spam with a probability for every word, not a rulebook
Graham's Bayesian filter grades each message on the combined probabilities of its words, catching spam without a lawyer-maintained blacklist.
Viaweb
the move
By 2002 spam had outrun hand-maintained keyword filters: spammers changed wording, and hosts blocked anything too strict. Paul Graham published 'A Plan for Spam' proposing a statistical classifier.
The filter learns a per-word spam probability from the user's own mailbox. A message's score is the combination of the word probabilities it contains, so a handful of strongly spammy words outweighs a benign subject line.
Graham reported the approach missed under 5 of every 1,000 spams with zero false positives in his own test — a result no rules-based filter could match.
why it works
- Statistics generalize to wording a technical expert never thought to block.
- Training on the user's own mail tailors thresholds to what that person calls spam.
- There is nothing for the spammer to reword around, because the filter keys on the statistical weight of many words.
what transfers
Instead of writing rules for an adversary who reads your rules, use statistics that update as the data does. A classifier that learns your mail is always current; a blacklist is slowly obsolete.
what came after
Bayesian filtering became the standard for email providers and shaped how mail, and later much of text classification, was done. The same idea — learn from data rather than enforce a list — moved into search, recommendation and every modern classifier.
references
spotted an error? The archive wants to know.