Quantisation
Storing a model's weights with fewer bits, such as 8 or 4 instead of 16, to cut memory and speed up inference.
Also: quantization, 4-bit, GPTQ
A model trained in 16-bit numbers can often be converted to 8-bit or 4-bit numbers with only a small loss in quality. A 70-billion-parameter model shrinks from about 140 gigabytes to about 35 at 4 bits, small enough for a single high-end graphics card or a well-equipped laptop. Methods such as GPTQ and AWQ choose the rounding carefully, and formats such as GGUF package quantised models for local use. Very low precision, below about 4 bits, usually costs noticeable accuracy.