BERT
An encoder-only transformer that reads text in both directions at once.
BERT is a language model released by Google in October 2018 that turns text into contextual vectors using self-supervised learning. It uses an encoder-only transformer and set new results on many language understanding tasks, and remains a standard building block in natural language processing research as of 2026.
- Released October 2018 by Google Research under Apache 2.0
- BERTBASE has 110 million parameters; BERTLARGE has 340 million
- Trained on BookCorpus (800M words) and English Wikipedia (2,500M words)
- 24 smaller models added March 11, 2020, down to BERTTINY at 4 million parameters
- Google Search used BERT for over 70 languages by December 2019
What it is
BERT, short for bidirectional encoder representations from transformers, is a language model that Google introduced in October 2018. It turns text into a run of vectors through self-supervised learning and relies on an encoder-only transformer design. Two English sizes shipped first: BERTBASE with 110 million parameters and BERTLARGE with 340 million. Both were trained on the Toronto BookCorpus, about 800 million words, plus roughly 2,500 million words of English Wikipedia. The model pushed large language models to much better results and, as of 2026, is a routine part of natural language processing research.
How it came about
Google researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova put BERT forward in 2018. Its design grew out of earlier work on contextual word representations, among them semi-supervised sequence learning, generative pre-training, ELMo and ULMFit. Unlike those predecessors, BERT is deeply bidirectional and unsupervised, trained on plain text alone. Google began applying BERT to English search queries in the US on October 25, 2019, extended it to more than 70 languages by December 9, 2019, and by October 2020 nearly every English query was handled by a BERT model.
Inside the model
BERT is an encoder-only transformer made of four parts. A WordPiece tokenizer turns English text into integers over a 30,000-word vocabulary, marking anything outside it as [UNK]. An embedding stage adds token type, absolute position and segment type vectors, then normalises each token to 768 numbers. A stack of transformer blocks applies self-attention with no causal masking, so each token sees both sides at once. A task head maps the final vectors back to the vocabulary. Pre-training uses masked language modelling, where 15% of tokens are chosen and then masked 80% of the time, swapped for a random word 10% of the time, or left alone 10% of the time, plus next sentence prediction with [CLS] and [SEP] markers.
What it can do
BERT learns contextual vectors, so the same word gets different representations depending on its surroundings, unlike word2vec or GloVe which give one fixed vector per word. That helps with coreference and polysemy, and with tasks such as natural language inference, text classification and question answering. The original paper reported state-of-the-art results after fine-tuning on GLUE, SQuAD v1.1 and v2.0, and SWAG. Limits come from the encoder-only design: without a decoder, BERT cannot be prompted easily and does not generate fluent text, and masking many tokens at once degrades its output.
How to use it
Weights were published on GitHub under the Apache 2.0 licence. The first release covered BERTBASE and BERTLARGE in English. On March 11, 2020, 24 smaller models appeared, the tiniest being BERTTINY with 4 million parameters. For a new task, the task head is dropped and a fresh module is trained on top, which needs little data.
Reception and debate
BERT brought large gains on language understanding benchmarks and quickly became a standard reference point in the field. Its success also prompted work on interpretation, sometimes called BERTology, which probes what the model stores in its internal vectors and attention weights. That interpretive work notes that the reasons behind BERT's strong scores are still not fully understood.
Why it matters
BERT reshaped how the field approaches language representation and led directly to a wave of variants. RoBERTa kept the architecture but reworked training and dropped next sentence prediction; XLM-RoBERTa took the idea multilingual; DistilBERT and TinyBERT compressed the model while keeping most benchmark scores; ALBERT followed in the same year. Google's own search system adopted BERT across many languages.
Questions people ask about BERT
What does BERT actually do?
It converts text into vectors that carry the meaning of each word in its surrounding context. Because it reads in both directions at once, the same word gets a different vector depending on the sentence it appears in.
How is BERT trained?
Two objectives run together. In masked language modelling, some tokens are hidden and the model must recover them. In next sentence prediction, it decides whether one sentence really follows another. After that, the model is fine-tuned on a specific task.
Can BERT write text?
Not well. It has no decoder, so it cannot be prompted the way generative models can. Asking it to fill in many masked tokens at once shifts the input away from what it saw in training and the quality drops.
How do I use it for my own task?
Take the released weights, remove the pre-training task head and attach a new module for your task, then fine-tune. The original work found one hour on a single Cloud TPU was enough for BERTLARGE to reach strong results.
What came after BERT?
Several variants followed in 2019, including RoBERTa, XLM-RoBERTa, DistilBERT, TinyBERT and ALBERT. They adjust training, scale, multilingual coverage or compression while keeping the underlying encoder design.
Good reading and sources
- Official websiteOfficial site
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingarXiv, Article
- Summary of the models — transformers 3.4.0 documentationHugging Face, Article
- BERT (language model) on WikipediaEncyclopedia
- More photos on Wikimedia CommonsPhotos
- Wikidata recordOpen data
The facts on this page come from the Wikipedia article "BERT (language model)", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.