The encyclopedia · R&D & Science · Technical decision · 2002
Bayesian spam filtering scored messages by word probabilities learned from your own mail
Paul Graham's spam filter graded each word's odds of being spam and let the whole message's score decide, so it adapted per user.
Paul Graham
the move
Keyword and blacklist filters tried to catch spam with specific words, but spammers just changed the words and the rules lagged behind. Paul Graham's 2002 essay "A Plan for Spam" proposed a different approach.
The filter assigns each word a spam probability based on how often it appears in a user's spam versus ham, then combines those word probabilities with Bayes' rule to score the whole message. A word like "click" scores as spammy, while first-person words like "I" score as not, so a legitimate note containing the word "viagra" among many neutral words can pass.
Because the filter is trained on the user's own mail, it is personalized and adapts as the words spammers use change, which was its main advantage over static rules.
why it works
- It looks at the whole message, not a single banned word, so false positives fall.
- Probabilities give a smooth ranking of spamminess rather than a yes/no rule.
- Trained per user, it uses that user's definition of spam and ham.
- It adapts automatically because the word probabilities are learned and can be retrained.
what transfers
Instead of writing rules for every known trick, learn a model of the difference between good and bad from the user's own data and let it adapt as the adversary shifts.
what came after
Bayesian filtering became the mainstream anti-spam tool of the mid-2000s, and it was the subject of academic evaluation and subsequent variations, though spammers later tried to defeat it by padding mail with neutral words, and it was complemented by other methods.
references
spotted an error? The archive wants to know.