News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
DistilBERT

DistilBERT

A smaller, faster version of the BERT language model.

DistilBERT is a compressed language model created by Hugging Face researchers. Released in 2019, it uses knowledge distillation to reduce the size of BERT by forty per cent while keeping most of its performance. It is designed for efficient text processing tasks.

Share on WhatsApp

  • Retains 97% of BERT's language understanding
  • 60% faster than the original BERT base model
  • Trained on 8 V100 GPUs for 90 hours
  • Uses a triple loss function for distillation
  • Supports context lengths up to 512 tokens

What it is

DistilBERT is a compact language representation model developed by researchers at Hugging Face. It was introduced in October 2019 as a distilled version of the BERT base model. The goal was to create a system that is smaller, faster, and cheaper to run while retaining strong language understanding capabilities. It is primarily used for tasks such as sequence classification, token classification, and question answering. The model processes text inputs up to 512 tokens in length.

How it came about

The model was proposed in a paper submitted to arXiv in October 2019 by Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. The work was accepted at the NeurIPS 2019 Workshop on Energy Efficient Machine Learning. The authors aimed to address the challenge of running large pre-trained models under constrained computational budgets. They demonstrated that knowledge distillation could be applied during the pre-training phase rather than just for specific tasks.

Inside the model

DistilBERT uses an encoder-only transformer architecture. It was trained using a triple loss function combining masked language modeling, distillation loss, and cosine embedding loss. The distillation loss ensures the student model matches the probability outputs of the teacher BERT model. Cosine loss aligns the hidden states of both models. Training data included BookCorpus and English Wikipedia, processed with WordPiece tokenization. The training ran on eight 16 GB V100 GPUs for ninety hours.

What it can do

The model retains ninety-seven per cent of the language understanding capabilities of BERT base while being sixty per cent faster. When fine-tuned, it achieves competitive scores on benchmarks such as MNLI (82.2), QQP (88.5), and SST-2 (91.3). It is suitable for tasks requiring whole-sentence analysis but is not designed for text generation. The uncased version does not distinguish between upper and lower case letters.

How to use it

The model weights are openly available through the Hugging Face model hub. It can be used directly with the Transformers library for tasks like masked language modeling or feature extraction. Code for the distillation process is provided in the research projects section of the library. Users can fine-tune the base model for specific downstream applications.

Safety and testing

The model inherits biases from its teacher, BERT. Examples show biased predictions related to gender and race when completing sentences. For instance, completions for prompts involving different demographic groups reflect societal stereotypes. These biases affect all fine-tuned versions of the model. The training data is described as fairly neutral, but the model can still produce biased outputs.

Why it matters

DistilBERT demonstrated that large models could be significantly compressed without major performance losses. It enabled more efficient on-device computations and reduced costs for inference. The approach influenced subsequent work in model distillation and efficient natural language processing. It showed that pre-training distillation was viable for general-purpose models.

Questions people ask about DistilBERT

How much smaller is DistilBERT than BERT?

DistilBERT is forty per cent smaller than the BERT base model. Despite this reduction in size, it retains ninety-seven per cent of the original model's language understanding capabilities.

What data was used to train DistilBERT?

It was pretrained on the same corpus as BERT. This includes the BookCorpus dataset, which contains over eleven thousand unpublished books, and the English Wikipedia, excluding lists, tables, and headers.

Can DistilBERT generate text?

No, it is not designed for text generation. It is primarily intended for tasks that use the whole sentence to make decisions, such as classification or question answering. For generation, models like GPT-2 are recommended.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on DistilBERT yet, so the facts on this page come from the sources listed above: huggingface.co/distilbert/distilbert-base-uncased, arxiv.org/abs/1910.01108. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.