Napkin math
Before choosing an index, multiply three numbers. At a billion vectors memory is the bill, and the vector is far bigger than the information in it.
1Orders of magnitude
| Vectors | Typical use |
|---|---|
| 10⁴–10⁷ | a company’s documents chunked for RAG |
| 10⁸ | a large e-commerce catalogue, a news archive |
| 10⁹–10¹¹ | web search, social posts, ad inventory, deduplicating an LLM training set |
Slide the numbers yourself The vertical ticks are a 16 GB laptop, an 80 GB GPU, 1 TB of server RAM and a 100 TB cluster, on a log scale: each step to the right is ten times more. HNSW adds roughly 2 × M × 4 bytes of links per vector on top (about 256 bytes at M = 32).
2Where is the redundancy?
A 768-dimensional float32 vector spends 24,576 bits. The meaning it carries is far smaller, for three reasons. Each one becomes a chapter.
-
Precision is wasted float32 gives about 7 significant digits. The model’s own noise is far larger. Rounding each number to 8 bits barely changes rankings; even keeping only the sign preserves a surprising amount.
-
Coordinates are correlated The data lives near a lower-dimensional surface, so many coordinates are partly predictable from others.
→ 3.4 From spectrum to bit budget, Part 4, 6.4 The transform zoo
-
Vectors are correlated with each other They come in clusters, and in sequences. A cluster id is already a few bits of information.
Why not just zip them?
General-purpose compressors such as gzip or zstd look for exact repeated byte patterns. The low bits of a float are essentially noise, so there is nothing to repeat. On this book’s embeddings zstd at its highest setting saves 7% (6.3 Floats are stubborn).
The big wins come from lossy, task-aware compression: throw away information that does not change which neighbours come back. In information-theory terms this is rate–distortion, with distortion measured in lost recall, not lost bits.
3The cheap tricks, in three lines
x = X.astype(np.float32) # 768 dims: 3,072 bytes each
int8 = np.round(x / np.abs(x).max(0) * 127).astype(np.int8) # 768 bytes, 4x smaller
bits = np.packbits(x > 0, axis=1) # 96 bytes, 32x smaller
# Search the bits with Hamming distance (XOR + popcount), take the top 200,
# then re-rank those with the float vectors kept on disk.
4The standard production pattern
A compressed index in fast memory, full-precision vectors on cheap storage, and a re-ranked shortlist in between.
Newer systems (DiskANN, turbopuffer, LanceDB, S3 Vectors) push ever more onto cheap storage and accept a few extra milliseconds, because past a billion vectors RAM is the dominant cost.