Scalar quantization · Close Enough

1Rounding with a step

Pick a step size $\kbook{\Delta}$ and round each coordinate to the nearest multiple of it. With $b$ bits you get $2^b$ levels.

If the step is small compared with the spread of the data, the rounding error is spread evenly between $-\Delta/2$ and $+\Delta/2$ . The variance of a uniform distribution of width $\Delta$ is

\kres{D} \;=\; \frac{\kbook{\Delta}^2}{12}

Fig 3.5.1Scalar quantization: bits and clipping

Slide the bits, then the clip range With few bits you must choose: a wide range wastes levels on values that almost never occur, a narrow one clips the tails (red). Press best c to see the compromise move outward as bits are added.

Two more things to notice in the readout:

The output’s entropy is below $b$ . The middle levels are used far more than the outer ones. An entropy coder could store the same indices in fewer bits, which is layer 4 of the stack.
$\Delta^2/12$ is accurate from about 3 bits up. Below that the “small step” assumption fails.

2The three you will meet

Name	Bits per coordinate	Size of a 384-d vector	Training	Typical use
float32	32	1,536 B	none	the model’s output
float16	16	768 B	none	storage; almost no loss
int8	8	384 B	a min and max	the default “quantized” option in vector DBs
int4	4	192 B	per-coordinate ranges	when RAM is tight
binary	1	48 B	none	first-pass filter with re-ranking

On Marginalia’s vectors

Per-coordinate quantizers fitted on the training books, scored on held-out books, without re-ranking:

Bits per coordinate	Vector size	Relative squared error	10-recall@10
1	48 B	0.359	0.42
2	96 B	0.116	0.67
4	192 B	0.0133	0.88
8	384 B	0.00008	0.99

At 8 bits the top-10 lists are almost untouched. At 1 bit, recall without re-ranking is poor, yet 5.4 Bits as a compressed sketch shows the same bits are an excellent shortlist generator.

3Where it stops

Scalar quantization treats every coordinate alone, so it is blind to two things:

Correlation between coordinates. Each coordinate’s range is coded as if the others told you nothing. A rotation first (3.4 From spectrum to bit budget) fixes the linear part of that.
The floor of one bit. There is no way to give a coordinate a third of a bit. Marginalia’s index budget is 16 bytes for 384 dimensions: 0.33 bits per dimension.

SCALAR / BINARYdistortion at 16 bytes not reachableencode one rounding per coordinatesearch SIMD integer dot productstraining none

To spend less than one bit per coordinate, you have to stop quantizing coordinates and start quantizing vectors.