The encyclopedia · R&D & Science · Technical decision · 2013
word2vec turned word meaning into vectors you can add and subtract
Mikolov's skip-gram model learned dense word vectors from raw text, so king − man + woman lands on queen.
The solution
In January 2013, Tomas Mikolov and colleagues at Google proposed two architectures — CBOW and skip-gram — for learning continuous word vectors from very large text. Their aim was to replace one-hot encodings, which treat every word as an unrelated dimension.
The models are startlingly simple: a single hidden layer learns to predict either a word from its neighbours or the neighbours from the word. Because training on billions of words forces each word into a small dense vector, distributional similarity becomes geometric proximity.
The paper reported large accuracy gains on word-similarity tests at far lower computational cost, and demonstrated that vector arithmetic captures relations: the vector for king minus man plus woman lands closest to queen. Training on a 1.6-billion-word dataset took less than a day.
Why it worked
- Prediction forces meaning into a compact vector
- Similar contexts converge to nearby points
- Vector offsets encode analogies like king to queen
- Training on 1.6 billion words took under a day
What can be applied
Don't hand-engineer features for a million words — set a prediction task whose solution must compress meaning, then let gradient descent find the geometry.
Aftermath
Word vectors became the standard input representation for natural-language models, and word2vec's tricks — subsampling, negative sampling, phrase vectors — were extended in the follow-up paper 'Distributed Representations of Words and Phrases'. Embedding ideas later fed GloVe, BERT and the transformer era.
Sources
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
spotted an error? The archive wants to know.