Knowledge distillation
Training a smaller student model to imitate a larger teacher model's outputs.
Also: distillation, teacher-student training
Distillation, described by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in 2015, trains a student on the teacher's full probability distribution or on text the teacher writes, which carries more information than plain right-or-wrong labels. Many small models are distilled from large ones, and DeepSeek released several smaller models distilled from DeepSeek-R1 in 2025. Distilling from a competitor's outputs can break that provider's terms of service, and in early 2025 OpenAI said it was reviewing whether its models had been used this way by DeepSeek.