Synthetic data
Training examples written by other models rather than collected from people, now a large share of post-training data.
As high-quality human text runs short, labs increasingly use models to generate training data: worked maths solutions, code with tests, conversations, translations and textbook-style explanations. Microsoft's Phi models from 2023 were trained heavily on synthetic textbook text and performed well for their small size. Synthetic data is also used in distillation, where a large model's answers teach a smaller one. The risks are that errors and biases are copied, and that models trained repeatedly on their own outputs can degrade, an effect called model collapse.