Everything is a probability model

1What you pay

If values really follow $p$ and the coder assumes $q$ , the average cost per value is the cross-entropy:

\underbrace{-\sum_x p(x)\log_2 \kbits{q}(x)}_{\text{what you pay}} \;=\; \underbrace{H(p)}_{\text{what you must pay}} \;+\; \underbrace{D(p\,\Vert\, \kbits{q})}_{\text{what a wrong model costs}}

The KL divergence is pure waste, and it is always positive. A better model is the only way to shrink it.

2From a density to bits

Embedding coordinates are continuous numbers on a grid of step $\kbook{\Delta}$ . If the underlying density is $f$ , the probability of the grid cell at $x$ is about $f(x)\,\Delta$ , so

\text{bits per coordinate} \;\approx\; \underbrace{h(f)}_{\text{differential entropy}} \;-\; \log_2 \kbook{\Delta}

For a bell curve with spread $\sigma$ , the differential entropy is $\tfrac12\log_2(2\pi e\,\sigma^2)$ , which gives the formula this whole part leans on:

Everything a model can do is on the left of that formula: make σ smaller. Everything the choice of target does is on the right.

3Look at the actual distribution

Fig 6.2.1Probability models for a coordinate

Choose what is being coded and how it is modelled Bars are the measured histogram, pooled over every coordinate of every held-out vector. The line is the model. The readout compares what this model charges with the entropy of the histogram itself; the difference is the KL waste. Try Laplace, then flat, then drag the width.

4Measured: what the model’s shape costs

The same int8 values under five models, each fitted on the training books and charged on held-out books.

Model $q$ for one coordinate	Parameters	Bits per vector	versus raw
Flat: every value equally likely	0	3,072	—
One bell curve for all coordinates	2	2,499	−19%
A bell curve per coordinate	2 × 384	2,477	−19%
A Laplace curve per coordinate	2 × 384	2,505
A full histogram per coordinate	513 × 384	2,479

Three readings:

The shape is worth 595 bits per vector. That is more than every other trick in this part put together.
Gaussian beats Laplace by 28 bits. Audio and image residuals are Laplace-shaped, which is why FLAC and JPEG-LS use codes built for that. Embedding coordinates are not: their excess kurtosis is -0.02.
A free-form histogram buys nothing. With 2,479 against 2,477 bits, 200 times the parameters make no difference. Two numbers per coordinate capture the whole marginal distribution.

5What is still on the table

A per-coordinate model treats the 384 coordinates as independent and each vector as unrelated to the last. Both are false:

coordinates are correlated with each other → 6.4 The transform zoo;
each window resembles the previous one → 6.5 The prediction zoo.

The benchmark to beat is 2,477 bits.