Gradient descent
The basic training rule: move every parameter a small step in the direction that lowers the loss.
Also: optimiser, Adam, AdamW
Gradient descent uses the gradients from backpropagation to update weights, a little at a time, over many batches of examples. Practical training uses optimisers that improve on the plain rule. Adam, published by Diederik Kingma and Jimmy Ba in 2014, keeps running averages of past gradients to give each parameter its own step size, and its variant AdamW is the most common choice for language models. Newer optimisers such as Muon, used for Kimi K2, aim to train faster with less memory.