Next-token prediction
The training task of guessing the next token of a text from all the tokens before it.
Also: causal language modelling
Next-token prediction is the single task behind pre-training a decoder model. The model sees a passage, predicts a probability for every possible next token, and is scored on how much probability it gave the token that actually came next. Repeated over trillions of tokens, this simple goal pushes the model to learn spelling, grammar, facts, the structure of code and arguments, and patterns of reasoning, because all of them help predict text. Critics note that a model trained this way learns what is likely, which is not always what is true.