Know your data · Close Enough

1Unit length: everything is in the direction

Normalized embeddings live on a sphere, so cosine equals the inner product and equals $1 - \tfrac12\lVert x - y\rVert^2$ (1.7 What does “close” mean?). That is why compression error can be measured as plain squared distance.

2Centre first

The mean vector of this data has length 0.44, and two random windows have cosine 0.19 instead of 0. This is anisotropy: the cloud sits in a cone off to one side.

For compression the fix is one subtraction. Remove the mean before quantizing, so that codebooks do not spend bits re-describing it.

3Variance is concentrated in a few directions

Take the covariance of the centred vectors and look at its eigenvalues: the variance along each principal direction. Sorted from largest to smallest, this is the eigenvalue spectrum.

Fig 3.3.1Variance per principal direction

Hover the bars The first 8 directions (yellow) hold 24% of the variance, the first 64 hold 61%, and the first 128 hold 80%.

Read it both ways:

A few directions carry a lot. 82 of 384 directions (21% of the axes) hold 67% of the signal. Any good compressor will spend most of its bits there.
The tail is long. The remaining 61%-to-100% is spread thinly over three hundred directions. Throwing them all away loses the fine distinctions that separate near neighbours, as 4.8 Measured on real embeddings will show.

Measure it yourselfnumpy

Xc = X - X.mean(0)
ev = np.linalg.svd(Xc, compute_uv=False) ** 2
ev /= ev.sum()
print("variance in the top 64 directions:", ev[:64].sum())     # 0.61
print("participation ratio:", 1 / (ev ** 2).sum())             # about 68

4Neighbours in the text are neighbours in space

These are not independent vectors. They are a sequence. You met this experiment in 1.6 The shape of the cloud; here it is again, read for what it offers a compressor.

Experiment 1.6CA walk through Aliceanother look

Same experiment as in 1.6 The shape of the cloud: show what was done and how to read it

Read a book passage by passage. Does the embedding jump around at random, or walk?

What was done

Take the first three chapters of Alice in Wonderland: “Down the Rabbit-Hole”, “The Pool of Tears” and “A Caucus-Race and a Long Tale”.
Cut them into passages of 48 words, one every 24 words, as everywhere in this book: 249 passages, in reading order.
Embed each: 384 numbers, length 1.
Grid. Compute the cosine of every passage with every other: a 249 × 249 table.
Path. Subtract the average of these passages, find their first two principal directions, place each passage by its reach along them, and join the dots in reading order.
Fading. Over the whole book, average the cosine between passages that are 1, 2, 3 … 64 passages apart.

How to read it

One cell of the grid compares passage i (its row) with passage j (its column). Brighter is a higher cosine; the colour bar under the grid gives the scale. Coloured bars along the edges mark the chapters.
One dot of the path is one passage. !thick The thick line follows the average position of neighbouring passages; the slider sets how many.
Bottom chart: average cosine against distance in the text. The dashed line is two passages of Alice picked at random.

Read it again

The previous passage is a good guess. The bright diagonal says each vector is close to the one before it. Store the difference from a prediction, not the vector itself.
The guess decays slowly. One passage back is much better than two, and two is still better than a random passage. A predictor should lean on the most recent vector and on the book’s own average.

0.70 cosine between a window and the next (they share half their words)
0.47 two windows apart (no shared words)
0.37 sixteen apart
0.19 two random windows from anywhere

Embeddings of a sequence are not a bag of points. They are a path, and paths are predictable.

Video codecs, audio codecs and DPCM all live off that one fact. 6.5 The prediction zoo puts it to work.

5And the numbers themselves

Each coordinate is a float32 with a typical size of 0.051 (the largest in the whole set is 0.28). Their distribution is a nearly perfect bell curve: the excess kurtosis is -0.02, where 0 is exactly Gaussian. That fact will matter twice: it makes scalar quantization theory apply cleanly (3.5 Scalar quantization), and it picks the probability model in Part 6.