The shape of cached values

If the values form a thin cloud, perhaps we can describe that cloud with fewer numbers. But a thin cloud and a safely discardable tail are different things. Let’s try the idea, then ask whether the same description works on another sequence.

1One token, one vector

Take a math prompt and the model’s answer. We call that whole sequence a thread. At each token position, each cached attention layer and KV head contributes a value vector: 256 coordinates. This is the value half of the KV cache from 7.3 The same math in other places.

Fixing those three axes gives a [tokens, 256] matrix: one block. Its 341–2,548 rows are token positions, not separate PCA directions. Five threads × six cached layers × two heads give 60 blocks.

Within a block, how do the 256 numbers vary across rows? We subtract the block’s mean vector and use PCA to find directions in coordinate space that capture that variation. Each direction combines coordinates. It isn’t a token, a layer or a head.

2Try a smaller world

Imagine V had two coordinates. Each token would be a point on a plane. A round cloud needs both directions. A thin cloud mostly changes along one. Which line should we keep?

When the line follows the cloud, little variation is lost. Turn it across the cloud and the error jumps. Add an offset and the cloud travels, while its centred shape stays the same. This is the geometry of transform coding in miniature.

So let’s order the real directions from strongest to weakest. But what if each weak direction is small, and there are hundreds of them?

3A small dimension number can hide a long tail

effective dimension measures concentration. It doesn’t say how many coordinates you can keep without much error. These three toy spectra make the distinction visible.

The formula names what we just saw. If direction $i$ has variance $\lambda_i$, the participation ratio is

Equal variance in $d$ directions gives PR $=d$. A long tail can contribute a lot to the numerator while contributing little to the sum of squares. The same distinction matters for the embedding cloud in 3.3 Know your data.

Now try the actual values

Choose a layer and head below. We pool all five sequences’ token rows after centring each sequence separately. That leaves one 4,719 × 256 matrix for the selected layer and head.

Experiment 9.2AThe measured V spectrumfirst look

How much variation survives when we keep only the strongest directions?

What was done

1. Select one of six cached layers and one of two KV heads.
2. Subtract each sequence’s own mean and concatenate its token rows.
3. Fit PCA to all 4,719 rows; sort the 256 variances and add their shares.

How to read it

• Blue bars are retained directions; red bars are discarded.
• The cumulative curve adds variance shares from strongest to weakest.

Before you read on, look for

• Move the rank: does the tail end cleanly anywhere?
• This is an in-sample spectrum. The longest sequence supplies over half the rows.

The weak directions are collectively expensive to discard.

Keeping only 32 leaves 44% of held-out squared variation behind. This points toward fewer bits on weak directions, rather than deleting them. It is reconstruction error, not a measured change in model accuracy.

4Could we learn the directions once?

A tailored basis for every sequence costs fitting time and storage. Reusing a basis would be more useful. Where could we borrow it from?

The other-thread basis works much better than a random orientation here. The other head and next layer stay near random in their stored coordinate systems. A learned alignment could reveal related information; this plot doesn’t rule that out.

Promising—but the competitors didn’t get the same fitting sample sizes. Own-thread fits use chunk cross-validation; other-thread fits use all remaining sequences. What happens if both get the same budget?

5Give both bases 160 fitting tokens

For each target, its own basis now uses 160 tokens from its first half. The borrowed basis uses 160 tokens from other threads, at the same layer and head. Both retain 32 directions and describe the target’s second half.

The gap is small in several threads. But thread 104 pushes back: its own past captures 57%, other threads 42%. Its answer repeatedly revisits the same arithmetic; only 58% of token 4-grams are distinct. Repetition is a plausible contributor, not a demonstrated cause. Thread 286 has an unexplained gap too.

Another check: pool more axes

Start with one block. Adding threads changes sequence only; adding the other head changes head only; adding layers changes layer only. Every source block is centred separately before concatenation.

Thread	Tokens	Distinct 4-grams	Naive PR	Corrected PR	Own past, k=32	Other threads, k=32
12	740	86%	42.8	50.5	44%	42%
104	2,548	58%	37.2	38.6	57%	42%
286	341	87%	33.2	45.0	58%	47%
474	457	86%	35.3	44.3	46%	45%
493	633	81%	39.0	47.5	48%	44%

PR is averaged over layers and heads. The last two columns use the equal-size temporal test, not the chunk-cross-validation curves.

6What if most of the vector is an offset?

Remember moving the toy cloud without changing its shape? A large raw norm can likewise come from a constant component. We need to separate the shape of the centred cloud, the energy in its mean, and the size of the raw vectors.

The two energies in the toy

For a mean $\mu$ and centred token vector $z$, $x=\mu+z$. Averaging over the block gives $\mathbb E\|x\|^2=\|\mu\|^2+\mathbb E\|z\|^2$. The projection toy changes the first term with its offset control and the second with its thickness control. Centred reconstruction error is measured against the second term.

7Are short sequences fooling us?

With fewer rows, PCA can mistake sampling noise for a strong pattern. Neighbouring tokens are correlated too. If short sequences look simpler, is that the data—or the estimator?

The correction reduces short-sequence bias. It doesn’t make time-correlated samples independent. These five math answers, with prompts and responses combined, are a pilot—not a population-wide statement about V.

Why split halves help, and where they still fail

Separate sets of 32-token chunks give covariance estimates $S_A$ and $S_B$. The corrected participation ratio is

$$\widehat{\mathrm{PR}} = \frac{\operatorname{tr}(S_A)\operatorname{tr}(S_B)}{\operatorname{tr}(S_A S_B)}.$$

The estimate is averaged over random chunk splits. Independent halves reduce the sampling-noise contribution to the second covariance moment. Temporal dependence, estimated means and ratio bias remain. The temporal transfer test separates past and future, but long-range dependence can remain there too.

8So where might the savings come from?

The tail makes truncation expensive. A more promising first codec would keep a basis per layer and head, then spend fewer bits on weaker directions. That’s the reverse water-filling idea applied inside a transformer.

At 5% squared error relative to centred variance, that Gaussian calculation gives 390 bits per vector, against 4,096 bits for 256 bf16 numbers. This is room to investigate, not a measured compression ratio. Non-Gaussian structure could help; practical coding costs could hurt. The metric weights each coordinate equally; model sensitivity may not.

The bit-allocation calculation

Choose a water level $\theta$ so that $\sum_i\min(\lambda_i,\theta)=D$, where $D$ is the allowed squared error in absolute variance units. Then

$$\kbits{R}=\tfrac12\sum_{\lambda_i>\theta}\log_2(\lambda_i/\theta).$$

The figure divides $D$ by total centred variance to report relative error. This is the ideal asymptotic rate for a Gaussian source, not a universal lower bound on codecs for real V.

9What would we try next?

10Reproduce and trace the results

The original analysis lives in the kvdiff project on this machine. From that checkout:

uv run marimo edit notebooks/03_v_spectrum.py

uv run python overview_src/build.py 03_v_spectrum

From the Close Enough checkout, import the rebuilt overview and rebuild the book:

uv run experiments/import_kvdiff.py /attic/d/kvdiff/overviews/03_v_spectrum.html
uv run build.py