Close Enough 0% ◫Browse AaGlossary
1.3
Stage 1 · Where vectors come from

Context arrives

A transformer gives every word a different vector in every sentence. Turning that into one vector per passage takes two more steps, and one of them is routinely misunderstood.

~2 min

1One word, many vectors

ELMo and then BERT (2018) produced a different vector for each word in each sentence. “Bank” in “river bank” and in “bank account” now get different vectors, because every layer of the transformer mixes in information from the surrounding tokens.

That solves the jaguar problem. It does not yet give us one vector per passage.

2From token vectors to one embedding

text "river bank" token ids [8613, 4314] lookup table vocab × d attention + MLP attention + MLP … × N layers transformer hidden states one per token pool + normalize embedding 768 floats word2vec-era vectors live here: one fixed row per token, no context. Not used for search. contextual: "bank" here differs from "bank" in "bank account" what an embedding API returns pretrained backbone (an LLM or a BERT-style encoder) Then the whole stack is fine-tuned contrastively so the final vectors are good for "is this relevant to that?"
Fig 1.3.1Inside a text embedding model

The pipeline A text embedding model reuses an LLM-style backbone end to end, not just its first layer. The blue box is where word2vec-era vectors live; the yellow box is what an embedding API returns.

Read the figure left to right:

  1. Tokenize the text into integer ids.
  2. Look up each id in a table. This is a fixed vector per token, context-free.
  3. Run all the layers. Attention lets each position absorb its neighbours.
  4. Take the last layer’s hidden states: one contextual vector per token.
  5. Pool them into one vector, usually by averaging (mean pooling) or by taking one special token’s state.
  6. Normalize to length 1.

So is a text embedding just the LLM’s first layer, the token embedding table?

No, and this is the single most common confusion. The first layer (wte in GPT code) is a lookup table: token id in, fixed vector out. It is exactly word2vec’s kind of table. It knows nothing about context, and it gives one vector per token, not one per passage.

A modern text embedding runs the whole text through all the layers, pools the last layer, and then, crucially, the model is fine-tuned so that distances mean “relevant to each other”. Without that fine-tuning, raw LLM hidden states make mediocre search vectors.

3Why the raw states are not enough

A pretrained language model was trained to predict tokens. Nothing in that goal asks “are these two texts about the same thing?” So its raw vectors are organised around whatever helps prediction:

  • formatting and punctuation,
  • how frequent the words are,
  • position in the text.

Two passages about the same topic can sit far apart, and two unrelated passages with the same layout can sit close. The next page is the fix.