Two codecs · Close Enough

1The archive codec

Sequential, per book, written once. It can use everything.

Round to the target. int8 with one global step. This is the only lossy act, and it is a decision, not a side effect.
Subtract the mean (one vector of 384 integers in the header).
Predict from the previous window: $\beta \cdot x_{t-1}$ with $\beta = 0.63$ , rounded. Every 64th window is a keyframe and skips this step.
Predict each coordinate from the earlier ones with a triangular matrix (a 384 × 384 fixed-point table in the header).
Entropy-code each prediction error with a quantized bell curve of that coordinate’s spread, using rANS.
Chunk the stream at keyframes and write an index of chunk offsets.

3,072 bits per vector, raw int8
2,171 bits with the full model
1.41× smaller
81 GB instead of 115 GB for 300 M vectors

2The wire codec

A response holds 50 to 200 vectors from all over the corpus. There is no previous window: step 3 is unavailable. Everything else applies to a single vector on its own.

Fig 6.9.1Compressing a search response

A search response, measured 300 queries, the top 50 vectors of each. General-purpose compression against a model that both ends hold.

Three observations:

zstd barely helps (17%), even with a dictionary trained on similar vectors. There are no repeated byte strings in a batch of embeddings.
A shipped model helps a lot. A bell curve per coordinate saves 20%; adding triangular prediction saves 28%. The client needs the same two tables, loaded once.
Use what the client already has. If the client holds the search index’s RQ code for each result, the code is free side information, and the remaining bits drop to 2,171. The economics of 6.7 Where should the bits go? flip when the code costs nothing.

For a response of 100 vectors that is 28 kB instead of 38 kB.

3Side by side

	Archive	Wire
Access pattern	sequential, a book at a time	random, a vector at a time
Context available	previous window + earlier coordinates	earlier coordinates (+ the index code, if held)
Encode speed	irrelevant	matters: per request
Decode speed	moderate	critical: adds to latency
Model	global tables in the file header	the same tables, versioned, shipped once
Bits per vector (int8)	2,171	2,223, or 2,171 given the code
Framing	chunks at keyframes + offset index	length-prefixed, one vector per record

4The format checklist

The compression ratio is the easy part. These are what make a format last.

Determinism All prediction arithmetic in integers, with a specified rounding. Test by decoding on a different CPU architecture.
Version the model The tables are the format. A hash of the model in every chunk header; refuse to decode on mismatch.
Bound the damage A checksum per chunk. A flipped bit should cost one chunk, not one book.
Escape hatch An escape symbol for out-of-range values and a raw mode for chunks where the model does worse than 8 bits. Then the worst case is raw + a few bytes.
Fit on what you store Spreads and tables fitted on a sample of the corpus itself, not on some other corpus (6.8 Squeezing the codes themselves shows what a mismatched model costs).
Measure decode time first The triangular solve is 73,536 multiply-adds per vector. On the wire, check that it costs less than the bytes it saves.

5The dial: near-lossless

Between “bit-exact” and “8 bytes, lossy” is a continuum. Near-lossless coding guarantees every coordinate within $\pm\epsilon$ steps: quantize the prediction error with step $2\epsilon + 1$ instead of 1 and keep the rest of the machine.

Each doubling of the step saves about one bit per coordinate, as the $\log_2(\sigma/\Delta)$ formula promises.

Guarantee per coordinate	Bits per vector	versus exact
exact (ε = 0)	2,397	—
within ±1 step	1,793	−604
within ±2 steps	1,513
within ±5 steps	1,081
within ±8 steps	848

(Measured with the RQ-8 predictor and per-coordinate model; the same offsets apply to any row of the ladder.)

6What Marginalia ships

PPriyaWEEK 12 · THE ARCHITECTURE

“Three tiers, each with the compressor that fits its job. And one honest surprise: the fancy codebooks we built for search turned out not to help the archive. A matrix and a bell curve did.”

Fig 6.9.2One product, three compressors

One product, three compressors The RAM tier is encoder-heavy and lossy. The SSD tier is cheap both ways. The archive is lossless and sequential, and is what lets Marginalia retrain codebooks or change quantizers later without paying to re-embed.