News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
BERT

BERT

An encoder-only transformer that reads text in both directions at once.

BERT is a language model released by Google in October 2018 that turns text into contextual vectors using self-supervised learning. It uses an encoder-only transformer and set new results on many language understanding tasks, and remains a standard building block in natural language processing research as of 2026.

Share on WhatsApp

  • Released October 2018 by Google Research under Apache 2.0
  • BERTBASE has 110 million parameters; BERTLARGE has 340 million
  • Trained on BookCorpus (800M words) and English Wikipedia (2,500M words)
  • 24 smaller models added March 11, 2020, down to BERTTINY at 4 million parameters
  • Google Search used BERT for over 70 languages by December 2019

What it is

BERT, short for bidirectional encoder representations from transformers, is a language model that Google introduced in October 2018. It turns text into a run of vectors through self-supervised learning and relies on an encoder-only transformer design. Two English sizes shipped first: BERTBASE with 110 million parameters and BERTLARGE with 340 million. Both were trained on the Toronto BookCorpus, about 800 million words, plus roughly 2,500 million words of English Wikipedia. The model pushed large language models to much better results and, as of 2026, is a routine part of natural language processing research.

How it came about

Google researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova put BERT forward in 2018. Its design grew out of earlier work on contextual word representations, among them semi-supervised sequence learning, generative pre-training, ELMo and ULMFit. Unlike those predecessors, BERT is deeply bidirectional and unsupervised, trained on plain text alone. Google began applying BERT to English search queries in the US on October 25, 2019, extended it to more than 70 languages by December 9, 2019, and by October 2020 nearly every English query was handled by a BERT model.

Inside the model

BERT is an encoder-only transformer made of four parts. A WordPiece tokenizer turns English text into integers over a 30,000-word vocabulary, marking anything outside it as [UNK]. An embedding stage adds token type, absolute position and segment type vectors, then normalises each token to 768 numbers. A stack of transformer blocks applies self-attention with no causal masking, so each token sees both sides at once. A task head maps the final vectors back to the vocabulary. Pre-training uses masked language modelling, where 15% of tokens are chosen and then masked 80% of the time, swapped for a random word 10% of the time, or left alone 10% of the time, plus next sentence prediction with [CLS] and [SEP] markers.

What it can do

BERT learns contextual vectors, so the same word gets different representations depending on its surroundings, unlike word2vec or GloVe which give one fixed vector per word. That helps with coreference and polysemy, and with tasks such as natural language inference, text classification and question answering. The original paper reported state-of-the-art results after fine-tuning on GLUE, SQuAD v1.1 and v2.0, and SWAG. Limits come from the encoder-only design: without a decoder, BERT cannot be prompted easily and does not generate fluent text, and masking many tokens at once degrades its output.

How to use it

Weights were published on GitHub under the Apache 2.0 licence. The first release covered BERTBASE and BERTLARGE in English. On March 11, 2020, 24 smaller models appeared, the tiniest being BERTTINY with 4 million parameters. For a new task, the task head is dropped and a fresh module is trained on top, which needs little data.

Reception and debate

BERT brought large gains on language understanding benchmarks and quickly became a standard reference point in the field. Its success also prompted work on interpretation, sometimes called BERTology, which probes what the model stores in its internal vectors and attention weights. That interpretive work notes that the reasons behind BERT's strong scores are still not fully understood.

Why it matters

BERT reshaped how the field approaches language representation and led directly to a wave of variants. RoBERTa kept the architecture but reworked training and dropped next sentence prediction; XLM-RoBERTa took the idea multilingual; DistilBERT and TinyBERT compressed the model while keeping most benchmark scores; ALBERT followed in the same year. Google's own search system adopted BERT across many languages.

Questions people ask about BERT

What does BERT actually do?

It converts text into vectors that carry the meaning of each word in its surrounding context. Because it reads in both directions at once, the same word gets a different vector depending on the sentence it appears in.

How is BERT trained?

Two objectives run together. In masked language modelling, some tokens are hidden and the model must recover them. In next sentence prediction, it decides whether one sentence really follows another. After that, the model is fine-tuned on a specific task.

Can BERT write text?

Not well. It has no decoder, so it cannot be prompted the way generative models can. Asking it to fill in many masked tokens at once shifts the input away from what it saw in training and the quality drops.

How do I use it for my own task?

Take the released weights, remove the pre-training task head and attach a new module for your task, then fine-tune. The original work found one hour on a single Cloud TPU was enough for BERTLARGE to reach strong results.

What came after BERT?

Several variants followed in 2019, including RoBERTa, XLM-RoBERTa, DistilBERT, TinyBERT and ALBERT. They adjust training, scale, multilingual coverage or compression while keeping the underlying encoder design.

Good reading and sources

Where this page comes from

The facts on this page come from the Wikipedia article "BERT (language model)", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.

It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.