News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM

Architectures

How the network is built inside.

Dense decoder transformers

The dense decoder transformer is the design behind GPT and most chatbots. It is a single stack of identical transformer layers that reads the tokens so far and predicts the next one, with a mask that stops each position from looking ahead

Mixture of experts

A mixture of experts (MoE) model replaces each dense feed-forward block with many smaller expert networks and a router that sends each token to only a few of them. The model can hold a very large number of parameters while using only a fraction for any one token, so it learns more without costing proportionally more to run

Encoder-only

Encoder-only models read a whole text in both directions at once and output a vector for each token, rather than generating text. Trained to fill in masked words, they learn rich representations of meaning and are then fine-tuned for classification, picking out names and places, question answering and search

Encoder-decoder

Encoder-decoder models use two stacks: an encoder reads the input in full, and a decoder writes the output one token at a time while attending to the encoder's results. This was the shape of the original 2017 transformer, built for translation, and it suits tasks that map one text to another, such as translation, summarisation and speech-to-text

State-space and recurrent

State-space and recurrent models process text as a running summary rather than attending back over every earlier token. They carry a fixed-size state from one token to the next, so memory and time grow only in step with length, and generating each new token costs the same however long the text

Hybrid designs

Hybrid models combine a few attention layers with many cheaper layers of another kind, usually state-space (Mamba) layers or forms of linear attention. Attention gives exact recall over the context, while the cheaper layers carry most of the load, cutting the memory needed for long inputs and speeding up generation

Diffusion language models

Diffusion language models write differently from the usual next-token models. They start from a passage that is fully masked or noisy and refine all of it over a series of steps, filling in many tokens at once, in the way image generators turn noise into a picture

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.