Reward model
A model trained to score answers the way human raters would, used to guide reinforcement learning.
Also: preference model
People cannot rate millions of answers during training, so labs train a reward model to do it for them. It is usually a copy of a language model with a single score output, trained on human comparisons of answer pairs. During RLHF, the policy model writes answers and the reward model grades them. Reward models are imperfect, and a policy can learn to exploit their blind spots, for example by writing longer answers because raters tended to prefer length.