Direct preference optimisation
A simpler alternative to RLHF that trains directly on preference pairs, with no separate reward model.
Also: DPO
Direct preference optimisation was published by Rafael Rafailov and colleagues at Stanford in 2023. They showed that the RLHF objective could be rewritten so that the language model is trained directly to raise the likelihood of preferred answers and lower that of rejected ones, compared with a frozen reference copy. No reward model or reinforcement learning loop is needed, which makes it cheaper and more stable. DPO and its variants were quickly taken up across open models, including Zephyr and many Llama and Mistral fine-tunes.