Mixed precision
Training with lower-precision numbers for most calculations to save memory and time, while keeping key values precise.
Also: bfloat16, BF16, FP8
Computers can store numbers with more or fewer bits. Full 32-bit precision is accurate but slow and memory-hungry. Most models are trained in 16-bit formats, especially bfloat16, a format devised at Google Brain that keeps the range of 32-bit numbers with less detail. Sensitive values such as the master copy of the weights stay in higher precision. DeepSeek-V3 in 2024 was one of the first large open models trained largely in 8-bit FP8. Lower precision at inference time is called quantisation.