Meaning from context
Words do not overlap, meanings do. The fix is seventy years old and fits in one sentence.
1The problem with keywords
Someone types “how do I fix a flat”. The perfect document is titled “Repairing a punctured bicycle tyre”. They share zero words.
Classic search (Salton’s vector space model and its descendant BM25, which still powers Lucene and Elasticsearch) represents a text as a vector with one dimension per vocabulary word. Different words sit on different axes, so “flat” and “punctured” are perpendicular: similarity zero.
What we want instead is a space with a few hundred to a few thousand axes that do not correspond to words at all, where texts about the same thing land near each other however they are worded. Such a vector is a dense embedding: dense because every coordinate is non-zero, unlike the sparse one-slot-per-word vectors.
2The founding idea
You shall know a word by the company it keeps.
J. R. Firth, 1957. This is the distributional hypothesis: words used in similar contexts have similar meanings.
“Coffee” and “tea” both appear near “cup”, “hot”, “drink”, “morning”. Describe each word by the contexts it appears in, and similar words get similar descriptions automatically. Nobody has to tell the machine what tea is.
The first practical version just counted:
- Build a giant matrix of how often each word appears with each context word.
- Squeeze it down to a few hundred dimensions with an SVD, the matrix version of “find the main directions of variation”.
That is Latent Semantic Analysis (1990). The squeeze forces words that behave alike to share coordinates.
3word2vec: predict instead of count
In 2013 Tomas Mikolov’s team at Google flipped it around. Keep a table with one learned vector per word and train it on a simple game: given a word, can you tell its real neighbours in a sentence from random words? The only way to win is to give words that share contexts similar vectors.
# Two tables of shape (vocab_size, d), random at the start.
# For every (center, context) pair in real text, plus k random "negative" words:
pos = sigmoid(W_in[center] @ W_out[context]) # want this near 1
neg = sigmoid(-W_in[center] @ W_out[negatives].T) # want each near 1 (dot product low)
loss = -log(pos) - log(neg).sum()
# Gradient descent nudges the vectors. Afterwards keep W_in: those are "the embeddings".
Two things made it famous:
- Speed. It trained on billions of words on one machine.
- Directions that mean something. The step from “man” to “woman” is roughly the same vector as from “king” to “queen”.
Drag the star, or tap a word A hand-placed two-dimensional toy, not a real model. Real embeddings have hundreds of dimensions but the logic is the same: similar items cluster, and consistent offsets (gender, royalty, age) act like directions you can add. Try the analogy button, then switch the distance to cosine and see which neighbours change.
4Where it runs out
The limits of word2vec point straight at the next page:
- One vector per word. “Jaguar”, the cat and the car, gets one compromise vector stuck between the two meanings. Tap Query “jaguar” in the playground to see where it lands.
- Words, not sentences. Averaging word vectors throws away word order and negation: “dog bites man” and “man bites dog” come out identical.