Close Enough 0% ◫Browse AaGlossary
7.5
Wrap-up

The whole thing, and where to go next

Seven sentences, the measured numbers on one card, a learning path that adds detail without losing the map, and the papers worth your time.

~4 min

1Seven sentences

Meaning can be learned from context, and a neural network trained contrastively turns any item into a point where nearness means relevance.

Searching those points is the nearest-neighbour problem, which high dimensions make hard, so we trade a little accuracy for a lot of speed.

The tricks are to partition the space (hashing, clustering), compress the points (quantization), or navigate between them (graphs), and production systems combine all three.

Compression is a stack: a transform says which directions matter, a quantizer turns them into bits, prediction uses what the decoder already has, and an entropy coder prices what is left.

At a fraction of a bit per dimension, sums of small codebooks give a huge implicit codebook for the price of a hard encoding problem; at a bit per dimension and above, a rotation and a rounding do as well.

When every bit must come back, the job is a probability model: for embeddings, a bell curve per coordinate, a triangular predictor and the previous vector get within a rounding error of what linear methods can do.

And the same query–key similarity sits at the heart of attention, which is why search, compression and language models keep borrowing each other’s tools.

2The numbers on one card

All on 384-dimensional MiniLM embeddings of held-out books.

  • 68 effective dimensions out of 384
  • 0.70 cosine between consecutive windows
  • 0.71 → 0.54 → 0.48 relative error at 64 bits: PQ → RQ → LSQ
  • 0.99 recall@10 with plain int8
  • 0.80 recall@10 from 48-byte sign sketches, re-ranking 100
  • 66% vs 29% recall of IVF vs LSH at equal work
  • 16% the most a lossless coder saves on float32
  • 29% saved on exact int8 with the full model

3A learning path

  1. Embed 1,000 sentences with sentence-transformers, search by brute-force NumPy, and look at the failures. You will feel what the model thinks “similar” means.
  2. Train a tiny word2vec on a small corpus. Watch the analogy directions appear.
  3. Write the InfoNCE loss and fine-tune a small encoder on question–answer pairs. Compare retrieval before and after.
  4. In FAISS, index a million vectors as Flat, IVF, PQ, IVF-PQ and HNSW. Plot recall@10 against queries per second. Then browse ann-benchmarks.com.
  5. Compute the spectrum of your own embeddings and run the water-filling allocation of 3.4 From spectrum to bit budget on it.
  6. Write an int8 lossless coder: a Gaussian per coordinate and an off-the-shelf range coder. Add triangular prediction. Check your numbers against 6.1 Every bit must come back.
  7. Revisit attention as key–value retrieval: kNN-LM, Memorizing Transformers, KV-cache compression.

4Further reading

Foundations

  • Gray and Neuhoff, “Quantization”, IEEE Trans. Information Theory, 1998. The survey.
  • Goyal, “Theoretical Foundations of Transform Coding”, 2001.
  • Cover and Thomas, Elements of Information Theory: rate–distortion.
  • Sayood, Introduction to Data Compression: all four layers, readable.
  • Gersho and Gray, Vector Quantization and Signal Compression.

Embeddings

Search and quantization

Hashing

Entropy and lossless coding

Systems