News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Model words

158 terms. The words of language models, training and evaluation, in plain language.

A

Active parameters
The parameters a mixture-of-experts model actually uses for each token, usually a small share of its total.
Agent
A model that works towards a goal over many steps, choosing and using tools and reacting to the results.
AI safety
The work of preventing harm from AI systems, from everyday misuse to severe risks from very capable models.
Alignment
Making AI systems pursue the goals and values their designers and users intend, and avoid harmful behaviour.
Attention
The step in a transformer where each token weighs every other token and pulls in information from the ones that matter.
Autoregressive generation
Producing text one token at a time, feeding each new token back in as input for the next.

B

Base model
A model straight out of pre-training, which continues text but has not yet been tuned to follow instructions.
Benchmark
A fixed set of test questions with a scoring method, used to compare models on one kind of skill.
Benchmark saturation
When top models score so close to the maximum on a benchmark that it can no longer tell them apart.
BERT
Google's 2018 encoder model, which read text in both directions and set new records across language tasks.
Byte-pair encoding
A method for building a vocabulary by repeatedly merging the most frequent pair of symbols in a body of text.

C

Calibration
Whether a model's confidence matches how often it is right.
Chain of thought
Having a model write out intermediate reasoning steps before its answer, which improves accuracy on hard problems.
Chatbot
An application in which people talk with a model in turns, such as ChatGPT, Claude, Gemini or Meta AI.
Chatbot Arena
A public site where people compare two anonymous models' answers, used to rank chat models by preference.
Chinchilla scaling
DeepMind's 2022 finding that, for a fixed budget, models should be trained on about 20 tokens per parameter.
Code-switching
Mixing two or more languages in one conversation or sentence, as in Hinglish, a common challenge for models.
Coding agent
An agent that reads a codebase, edits files, runs commands and tests, and iterates until a change works.
Computer use
A model operating a computer as a person would, looking at the screen and moving the mouse and keyboard.
Constitutional AI
Anthropic's method of training a model to critique and improve its own answers against a written list of principles.
Context window
The maximum number of tokens a model can take in at once, counting the prompt, any documents and its own reply.
Continuous batching
Serving many users at once by adding and removing requests from a running batch at every generation step.
Cross-entropy loss
The standard training loss for language models: how surprised the model was by the token that actually came next.

D

Data contamination
When benchmark questions or answers end up in a model's training data, inflating its scores.
Data filtering
Selecting and cleaning training text, removing spam, toxic content, personal data and low-quality pages.
Decoder
A model, or part of one, that writes text one token at a time, each token seeing only those before it.
Diffusion language model
A model that writes by starting from noise or masked text and refining the whole passage over several steps.
Direct preference optimisation
A simpler alternative to RLHF that trains directly on preference pairs, with no separate reward model.

E

ELMo
A 2018 model from the Allen Institute for AI that produced contextual word embeddings from a two-way recurrent network.
Embedding
A list of numbers that stands for a token, word or passage, placed so that similar meanings sit close together.
Embedding model
A model that turns a passage of text into a single vector, used for search, clustering and retrieval.
Emergent abilities
Skills that seem to appear suddenly once a model passes a certain size, and the debate over whether they are real.
Encoder
A model, or part of one, that reads a whole text at once in both directions to build a representation of its meaning.
Encoder-decoder
A design with two stacks: one reads the input, the other writes the output while looking back at it.
Evaluation
Testing what a model can do and how it behaves, with benchmarks, human ratings and targeted test sets.

F

Feed-forward network
The part of each transformer layer that processes every token on its own, holding much of the model's stored knowledge.
Few-shot prompting
Putting a handful of worked examples in the prompt so the model copies the pattern.
Fine-tuning
Further training of an already trained model on a smaller, focused dataset to adapt it to a task or style.
FlashAttention
A way of computing exact attention that avoids slow trips to GPU memory, making long contexts faster and cheaper.
Foundation model
A large model trained on broad data that can be adapted to many tasks. A wider term than language model.
Frontier model
One of the most capable models at a given time, usually from a handful of well-funded labs.
Function calling
An API feature in which a model returns a structured request to run a named function with given arguments.

G

Generative pre-trained transformer
OpenAI's name for its series of decoder models, which has become a general label for models of this kind.
GGUF
A single-file format for quantised models, used by llama.cpp and other tools to run models on ordinary computers.
GPQA
A 2023 benchmark of very hard science questions written by PhD experts so that web searching does not help.
Gradient descent
The basic training rule: move every parameter a small step in the direction that lowers the loss.
Graphics processing unit
The chip that trains and runs language models, built for doing many simple calculations at once.
Greedy decoding
Always picking the single most likely next token. Fast and repeatable, but prone to dull or looping text.
Grounding
Tying a model's answer to specific sources, such as documents or search results, rather than its memory alone.
Group relative policy optimisation
A reinforcement learning method from DeepSeek that scores each answer against a group of answers to the same question.
Grouped-query attention
A form of attention in which several query heads share one set of keys and values, shrinking the KV cache.
Guardrails
Checks around a model that screen inputs and outputs for harmful, private or off-topic content.

H

Hallucination
When a model states something false or made up, such as a fake citation, with apparent confidence.
Hidden state
The vector a model holds for each token between layers, carrying everything it has worked out so far.
Hugging Face Hub
The largest public site for sharing model weights, datasets and demos, run by the company Hugging Face.
HumanEval
A 2021 coding benchmark of 164 Python problems, each checked by hidden unit tests.
Hybrid model
A model that mixes attention layers with state-space or recurrent layers to get the strengths of both.

I

In-context learning
A model picking up a new task from examples or instructions in the prompt, without any change to its weights.
Indic language model
A model built for the languages of India, such as Hindi, Bengali, Tamil, Telugu and Marathi.
Inference
Running a trained model to get answers, as opposed to training it. Most of the world's AI computing is now inference.
Instruction tuning
Fine-tuning a model on many tasks phrased as instructions, so it learns to do what it is asked.
Interpretability
Research into what happens inside a model: which parts represent which ideas, and how they produce an answer.

J

Jailbreak
A prompt crafted to get around a model's safety training and make it produce what it would normally refuse.

K

Knowledge cutoff
The date after which a model has seen no training data, so it knows nothing of later events unless told.
Knowledge distillation
Training a smaller student model to imitate a larger teacher model's outputs.
KV cache
Stored keys and values from earlier tokens, kept so the model does not recompute them for every new token.

L

Language model
Any system that assigns probabilities to sequences of words, and so can predict or generate text.
Large language model
A neural network trained on very large amounts of text to predict the next token, and so to read and write language.
Latency
How long a model takes to respond: the wait for the first token and the speed of the tokens after it.
Layer
One repeated block of a model. Text passes through dozens of layers, each refining the representation a little.
Layer normalisation
A step that rescales each token's vector to a steady size inside every layer, keeping training stable.
Licence
The legal terms under which a model's weights may be used, changed and shared.
Logits
The raw scores a model gives every token in its vocabulary before they are turned into probabilities.
Long context
The ability to read and use very long inputs, from hundreds of thousands to millions of tokens.
Lost in the middle
The tendency of models to use information at the start and end of a long input better than information in the middle.
Low-rank adaptation
A cheap fine-tuning method that freezes the model and trains small add-on matrices instead.
Low-resource language
A language with little digital text available for training, so models handle it less well.

M

Masked language model
A model trained to fill in hidden words in a sentence using the words on both sides, as BERT was.
Mixed precision
Training with lower-precision numbers for most calculations to save memory and time, while keeping key values precise.
Mixture of experts
A design in which each layer holds many expert sub-networks and a router sends each token to just a few of them.
MMLU
A widely used 2020 benchmark of about 16,000 multiple-choice questions across 57 subjects.
Model API
A web service that lets programs send prompts to a hosted model and get answers back, usually paid per token.
Model card
A short document published with a model describing what it is for, how it was trained and evaluated, and its limits.
Model collapse
The loss of quality and variety when models are trained, generation after generation, on text written by earlier models.
Model Context Protocol
An open standard from Anthropic for connecting models and agents to tools and data sources.
Model specification
A published document setting out how a lab intends its models to behave, and how to weigh conflicting goals.
Multi-head attention
Running several attention operations side by side, so different heads can track different kinds of relationship.
Multi-head latent attention
DeepSeek's attention design that compresses keys and values into a small shared vector to save memory.
Multilingual model
A model trained on many languages so it can understand and write in each, and often translate between them.
Multimodal model
A model that takes in, and sometimes produces, more than one kind of data: text, images, audio or video.

N

Needle in a haystack test
A long-context test that hides one fact in a long text and asks the model to find it.
Neural network
A computing system made of layers of simple units whose connection strengths are learned from data.
Next-token prediction
The training task of guessing the next token of a text from all the tokens before it.

O

On-device AI
Running a model on the user's own phone, laptop or car rather than in a data centre.
Open source AI
A model released with its weights, code and enough detail on its data that others can study and rebuild it.
Open weights
A model whose trained weights are published for anyone to download and run, whatever the licence terms.

P

Parameter
One adjustable number inside a model, learned in training. Model size is usually given as a parameter count.
Perplexity
A measure of how well a model predicts a text. Lower means the model was less surprised.
Positional encoding
Information added to token embeddings so the model knows the order of words, which attention alone ignores.
Post-training
Everything done to a model after pre-training: instruction tuning, preference tuning, safety training and reasoning training.
Pre-training
The first and largest stage of training, in which a model learns from trillions of tokens of general text.
Process reward model
A model that scores each step of a solution, not just the final answer, to guide reasoning.
Prompt
The text, and any images or files, given to a model to respond to.
Prompt caching
Saving the processed form of a long, repeated prompt so later requests that start the same way are cheaper and faster.
Prompt engineering
Writing and testing prompts to get better, more reliable answers from a model.
Prompt injection
Hiding instructions in content a model reads, such as a web page or email, so that it follows them instead of its user.

Q

QLoRA
LoRA fine-tuning on top of a 4-bit quantised model, which let large models be tuned on a single GPU.
Quantisation
Storing a model's weights with fewer bits, such as 8 or 4 instead of 16, to cut memory and speed up inference.

R

Reasoning model
A model trained to think at length, in a hidden or visible chain of thought, before giving its final answer.
Recurrent neural network
A network that reads a sequence one step at a time, carrying a memory forward. The main language approach before transformers.
Red-teaming
Deliberately attacking a model to find ways it can be made to misbehave, before others find them.
Refusal
When a model declines a request, ideally only for genuinely harmful ones.
Reinforcement learning
Learning by trial and reward rather than from fixed examples. Used to tune chat models and to train reasoning models.
Reinforcement learning from human feedback
Training a model with rewards learned from people's comparisons of its answers, the method that made ChatGPT helpful.
Residual connection
A shortcut that adds a layer's input to its output, so signals and gradients can pass through very deep networks.
Retrieval-augmented generation
Searching a collection of documents for relevant passages and giving them to the model along with the question.
Reward hacking
A model finding a way to score well on its training reward without doing what the reward was meant to encourage.
Reward model
A model trained to score answers the way human raters would, used to guide reinforcement learning.
Rotary position embedding
A way of encoding word order by rotating query and key vectors by an angle that depends on position.

S

Sampling
Choosing each next token from the model's probabilities, with some randomness, rather than always taking the top one.
Scaling laws
Regular patterns showing that loss falls predictably as models, data and compute grow.
Self-attention
Attention in which the tokens of one text attend to each other, rather than to a separate input.
Self-consistency
Asking a model the same question several times and taking the most common answer.
SentencePiece
A tokenizer library from Google that treats text as a raw stream, spaces included, so it works for languages written without spaces.
Sliding-window attention
Attention limited to a fixed number of recent tokens, so cost stops growing with the full length of the text.
Small language model
A compact model, usually under about 10 billion parameters, that can run on a laptop, a phone or modest servers.
Softmax
A function that turns a list of raw scores into probabilities that are all positive and add up to one.
Sovereign AI
A country building its own models and computing capacity, so it does not depend entirely on foreign providers.
Special token
A reserved token that marks structure rather than text, such as the end of a document or the start of a user's turn.
Speculative decoding
Speeding up generation by letting a small fast model draft several tokens and the big model check them in one pass.
Speech model
A model that turns speech into text, text into speech, or holds a spoken conversation directly.
State-space model
A family of sequence models that carry a fixed-size running state instead of attending to every past token.
Structured output
Making a model reply in an exact machine-readable format, such as JSON matching a given schema.
Supervised fine-tuning
Fine-tuning on example prompts paired with good answers, usually the first step of turning a base model into an assistant.
SWE-bench
A benchmark of real bug reports from open-source Python projects, where a model must produce a fix that passes the tests.
Sycophancy
A model's tendency to tell users what they want to hear, agreeing with them even when they are wrong.
Synthetic data
Training examples written by other models rather than collected from people, now a large share of post-training data.
System card
A detailed report a lab publishes with a major model, describing its safety testing, risks and safeguards.
System prompt
Hidden instructions placed before the conversation that set a model's role, rules and tone.

T

T5
Google's 2019 encoder-decoder model that framed every language task as turning one text into another.
Temperature
A setting that controls how adventurous sampling is. Low temperature gives safe, predictable text; high gives varied text.
Test-time compute
Spending more computation while answering, for example by thinking longer, to get better results.
Token
The unit of text a model reads and writes: a whole word, a piece of a word, a punctuation mark or a byte.
Tokenizer
The program that splits text into tokens and turns them into numbers before a model sees them, and back again afterwards.
Tokenizer fertility
How many tokens a tokenizer needs, on average, for one word of a given language. Lower is cheaper and faster.
Tool use
A model asking an outside program, such as a calculator, search engine or database, to do part of a task.
Top-p sampling
Sampling only from the smallest group of tokens whose probabilities add up to a set share, such as 90 per cent.
Training compute
The total amount of computation used to train a model, usually counted in floating-point operations.
Training data
The text, code and other material a model learns from, now measured in trillions of tokens.
Transformer
The neural network design, introduced in 2017, that almost every modern language model is built on.

V

Vector database
A database built to store embeddings and quickly find those nearest to a query vector.
Vision encoder
The part of a multimodal model that turns an image into vectors, usually by cutting it into small patches.
Vision-language model
A model that reads images together with text, answering questions about photos, documents and charts.
Vocabulary
The fixed list of tokens a model knows, each with its own number and its own embedding.

W

Watermarking
Hiding a statistical signal in generated text so that it can later be identified as machine-written.
Weights
The full set of learned parameters of a model, saved as files. Having the weights means being able to run the model.
Word2vec
A 2013 Google method for learning word embeddings from which much of modern language modelling grew.

Z

Zero-shot
Asking a model to do a task with no examples, just an instruction.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.