Floats are stubborn · Close Enough

1Where the bits go

Here is the measured entropy of each of the 32 bit positions, across all 9,020,160 float32 values in the database books.

Fig 6.3.1The bits of a float32

Where the bits go Top: six real values from the first window of Alice in Wonderland, bit by bit: sign, exponent, mantissa. Bottom: the entropy of each bit position over the whole dataset. A bar at full height is a fair coin.

The sign is a coin flip: 1 bit.
The exponent is mostly predictable: every value is small, so its top bits never change.
The mantissa is a coin flip from about its fourth bit onward.

The per-bit entropies sum to 27.2 of 32 bits. A coder that treats bits independently cannot beat that, and a good predictor barely changes it.

2Measured

Method	float32, bits per value	float16, bits per value
raw	32.0	16.0
zstd at level 19	29.6	14.7
byte shuffle + zstd	27.0	13.6
XOR with the previous window + shuffle + zstd	27.1	14.1
16-stage RQ prediction + shuffle + zstd	27.2	13.8

zstd alone saves 7%.
Shuffling the bytes first, so that all the exponent bytes sit together, saves 16%. This is what Blosc does, and it is the entire win.
Predicting from the previous window or from a 16-stage quantizer adds nothing. In the float domain the prediction only rearranges noise.

3The lesson

Decide how many bits you actually trust. Then compress exactly at that precision.

A model’s outputs are not accurate to 24 bits. The low mantissa bits differ between GPU models, batch sizes and library versions.

Rounding to a coarser grid is not a loss of information you had. It is a refusal to store noise you never wanted.

Target	Raw	Best lossless found	Saving
float32	12,288 bits	about 384 × 27.0	16%
16-bit fixed point	6,144	4,697	modelled as in the next pages
int8	3,072	2,171	29%

The int8 copy is 4× smaller than float32 before any coder runs, and it is the only one of the three where a model has something to grab.