Cross-entropy loss
The standard training loss for language models: how surprised the model was by the token that actually came next.
Also: log loss, negative log-likelihood
Cross-entropy takes the probability the model gave the correct next token and computes minus its logarithm. If the model gave the right token a probability of one, the loss is zero. If it gave it a tiny probability, the loss is large. Averaging over every position in a batch gives the training loss. Lower cross-entropy means better prediction, and raising e to the power of the average cross-entropy gives perplexity, a more intuitive measure of the same thing.