Our experiments
Overviews of our own experiments, with toys to explain the mechanism, measured results to test it, and enough detail to see what remains uncertain.
A vector has 256 coordinates. Does it need 256 numbers in storage? Before building a codec, we can look at the vectors themselves: where they vary, what carries over to another sequence, and how much error a shortcut leaves behind.
This part collects experiment overviews from kvdiff, our project for studying and compressing transformer KV caches. The earlier chapters supply the tools; here we use them on a model’s internal vectors and follow the measurements where they lead.
- The question Start with one concrete thing we want to know about the vectors.
- The mechanism Try a small live toy before looking at the model’s data.
- The measurement Name the model, samples, splits and averaging rule behind each result.
- The catch Separate an observation, a model calculation and a demonstrated codec.
1From kvdiff
- The shape of cached values Experiment 03 · Qwen 3.5 2B · five GSM8K sequences. A concentrated spectrum, a stubborn tail, and a first test of whether PCA directions transfer between sequences.
2What comes next
As the project grows, this section will collect more overviews: compression at equal storage budgets, the effect on attention outputs and next-token loss, and transfer to new sequences and tasks. Each will connect back to the relevant geometry, quantization and information-theory chapters.