Ternary training and post-training quantization
Comparing ternary models with 2-, 3- and 4-bit quantization, and measuring where the quality gap falls.
Notes
The ternary-float gap during training
A matched small-scale experiment finds that the overall loss gap grows as training data increases.
Token frequency and the ternary gap
Released models show larger degradation on low-frequency tokens, while the rare-fact comparison remains inconclusive.
In progress
A wider view of the ternary gap
Extending the matched experiments across a wider training range, with additional seeds and English, Portuguese and Spanish probes.
Coming soon