Speculative decoding
Speeding up generation by letting a small fast model draft several tokens and the big model check them in one pass.
Also: draft model, multi-token prediction
Because a large model must generate one token at a time, much of its hardware sits idle. In speculative decoding, described by Google and DeepMind researchers in 2022 and 2023, a small draft model guesses the next few tokens and the large model verifies all of them at once, keeping the ones it agrees with. The output is exactly what the large model alone would have produced, only faster. Some models, such as DeepSeek-V3, are trained to predict several tokens ahead themselves, which serves the same purpose.