Close Enough 0% ◫Browse AaGlossary
1.4
Stage 1 · Where vectors come from

The secret ingredient

Train directly for the thing you want. Show the model pairs that should match, and make everything else in the batch the wrong answer.

~3 min

1Pull together, push apart

Sentence-BERT (2019) showed the fix for the problem on the previous page. Give the model pairs that should match (a question and its answer, a title and its article, a caption and its image), make their vectors close, and push everything else away.

query: "how do I reset my router" positive: "restarting your wifi modem" pull hard negative: "router bits for woodworking" "banana bread recipe" "quarterly earnings call" every other item in the batch is a free negative
Fig 1.4.1Contrastive training

Contrastive training The query pulls its positive closer and pushes negatives away. The woodworking passage is a hard negative: it shares the word “router” but answers a different question.

2The loss, in four lines

InfoNCE, the loss behind nearly every modern embedding modelnumpy
# q, p: (B, d) normalized embeddings of B queries and their matching passages
logits = (q @ p.T) / 0.05            # B x B similarity matrix, sharpened by a temperature
labels = np.arange(B)                # the right answer for row i is column i (the diagonal)
loss = cross_entropy(logits, labels) # classify "which passage is mine?" among B candidates

It is literally a classification over the batch: which of these B passages belongs to my query?

L=−1B∑i=1Blog⁡exp⁡(qi⋅pi/τ)∑j=1Bexp⁡(qi⋅pj/τ) \mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp(\kq{q_i}\cdot \kvec{p_i}/\tau)}{\sum_{j=1}^{B}\exp(\kq{q_i}\cdot \kvec{p_j}/\tau)}

Three knobs follow directly from the formula:

  • Batch size BB. More candidates per question make a harder, more informative game.
  • Temperature τ\tau. Small values make the model care about near-misses.
  • Hard negatives. Passages that look relevant but are not teach the fine distinctions that random negatives never test.

The same recipe trains image–text models (CLIP), face recognition (FaceNet) and recommender towers.

3What the big labs ship today

Since about 2023 the recipe has converged:

  • Google describes gemini-embedding-001 as initialized from Gemini, with mean pooling, a projection, and contrastive training on large mixes that include synthetic data.
  • EmbeddingGemma (2025, about 300 M parameters) starts from a Gemma 3 backbone switched to bidirectional attention, trained contrastively and by distillation from a larger embedder.
  • Open models such as E5-Mistral, NV-Embed and Qwen3-Embedding follow similar recipes. Some pool the last token instead of the mean.

Model cards change between releases, so check the card for exact details.

4Four terms from every model card

  • Task prefixes "query: …" versus "passage: …". A search query and the document that answers it look different, so the model is told which side it is embedding. This is asymmetric retrieval.
  • Matryoshka dimensions Trained so that the first 256 of 3,072 numbers already carry most of the meaning. Truncate to save storage. It is a learned version of the idea on 3.4 From spectrum to bit budget.
  • Bi-encoder versus cross-encoder Embedders encode query and document separately, which is what makes precomputing and indexing possible. A cross-encoder reads both together: far more accurate, far too slow for a million documents, so it re-ranks the top 50 to 100.
  • MTEB The standard leaderboard, comparing models across retrieval, clustering and classification.