Training data
The text, code and other material a model learns from, now measured in trillions of tokens.
Also: training corpus, pre-training data
Language models are trained mostly on public web pages, often from Common Crawl, along with books, encyclopedias, scientific papers, code from public repositories, and increasingly synthetic and licensed data. GPT-3 was trained on about 300 billion tokens in 2020, Llama 2 on 2 trillion, Llama 3 on over 15 trillion. Most labs no longer disclose their data in detail. The use of copyrighted books, news and art in training is the subject of lawsuits in several countries, with publishers and authors on one side and labs arguing fair use on the other.