Compression is a stack
"PCA or vector quantization?" is like asking "a map or a car?" They do different jobs. Almost every practical compressor is the same four layers, and each layer exploits a different kind of structure.
1A question that dissolves
A natural first question about compressing embeddings: is PCA better than vector quantization, or the other way round?
PCA compresses nothing. Rotating the axes still leaves you storing 384 floats.
It saves space only once you throw away the small directions or give them fewer bits. That second step is quantization.
So the question is never “PCA or VQ?” It is “which transform, and which quantizer after it?”
2Four layers
From JPEG and MP3 to Faiss indexes and neural codecs, practical compressors are built from the same four layers.
| Layer | Question it answers | Tools | Structure it exploits | In this book |
|---|---|---|---|---|
| 1 · Transform | Which directions matter? | PCA / SVD, rotations (OPQ), DCT, learned encoders | Correlation between coordinates | 3.4 From spectrum to bit budget, 6.4 The transform zoo |
| 2 · Quantize | How do I turn numbers into a few bits? | Scalar (int8, int4, binary) or vector (k-means, PQ, RQ, additive) | Limited precision that matters; clusters | 3.5 Scalar quantization, Part 4, Part 5 |
| 3 · Predict | Can I guess this from what I already have? | Previous vector, earlier coordinates, a coarse code | Order: sequences, neighbours | 6.5 The prediction zoo |
| 4 · Entropy-code | Common symbols deserve shorter codes | Huffman, arithmetic coding, ANS | Uneven probabilities | 6.6 The entropy-coder zoo |
3Lossy and lossless are the same stack
The difference is only where information is allowed to leak:
- Lossy: layer 2 discards information. Layers 1, 3 and 4 are there to make that loss land where it hurts least and to make the surviving symbols cheap.
- Lossless: nothing may be discarded, so layer 2 changes role. A quantizer’s output becomes a prediction, and the exact difference is entropy-coded. Part 6 is about this.
4What each family is good at
- Transform + scalar Transform coding: rotate, give each coordinate its own number of bits, round. Near-optimal for bell-shaped data at about one bit per dimension and above. No search to encode.
- Vector quantizers Spend bits on whole vectors, so they can spend a fraction of a bit per coordinate, and they can sit on clusters. They win at very low rates.
- Hybrids OPQ is a rotation followed by product quantization. IVF with residual codes is a coarse vector quantizer followed by a fine one. Real systems are almost always hybrids.