DistilBERT
A smaller, faster version of the BERT language model.
DistilBERT is a compressed language model created by Hugging Face researchers. Released in 2019, it uses knowledge distillation to reduce the size of BERT by forty per cent while keeping most of its performance. It is designed for efficient text processing tasks.
- Retains 97% of BERT's language understanding
- 60% faster than the original BERT base model
- Trained on 8 V100 GPUs for 90 hours
- Uses a triple loss function for distillation
- Supports context lengths up to 512 tokens
What it is
DistilBERT is a compact language representation model developed by researchers at Hugging Face. It was introduced in October 2019 as a distilled version of the BERT base model. The goal was to create a system that is smaller, faster, and cheaper to run while retaining strong language understanding capabilities. It is primarily used for tasks such as sequence classification, token classification, and question answering. The model processes text inputs up to 512 tokens in length.
How it came about
The model was proposed in a paper submitted to arXiv in October 2019 by Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. The work was accepted at the NeurIPS 2019 Workshop on Energy Efficient Machine Learning. The authors aimed to address the challenge of running large pre-trained models under constrained computational budgets. They demonstrated that knowledge distillation could be applied during the pre-training phase rather than just for specific tasks.
Inside the model
DistilBERT uses an encoder-only transformer architecture. It was trained using a triple loss function combining masked language modeling, distillation loss, and cosine embedding loss. The distillation loss ensures the student model matches the probability outputs of the teacher BERT model. Cosine loss aligns the hidden states of both models. Training data included BookCorpus and English Wikipedia, processed with WordPiece tokenization. The training ran on eight 16 GB V100 GPUs for ninety hours.
What it can do
The model retains ninety-seven per cent of the language understanding capabilities of BERT base while being sixty per cent faster. When fine-tuned, it achieves competitive scores on benchmarks such as MNLI (82.2), QQP (88.5), and SST-2 (91.3). It is suitable for tasks requiring whole-sentence analysis but is not designed for text generation. The uncased version does not distinguish between upper and lower case letters.
How to use it
The model weights are openly available through the Hugging Face model hub. It can be used directly with the Transformers library for tasks like masked language modeling or feature extraction. Code for the distillation process is provided in the research projects section of the library. Users can fine-tune the base model for specific downstream applications.
Safety and testing
The model inherits biases from its teacher, BERT. Examples show biased predictions related to gender and race when completing sentences. For instance, completions for prompts involving different demographic groups reflect societal stereotypes. These biases affect all fine-tuned versions of the model. The training data is described as fairly neutral, but the model can still produce biased outputs.
Why it matters
DistilBERT demonstrated that large models could be significantly compressed without major performance losses. It enabled more efficient on-device computations and reduced costs for inference. The approach influenced subsequent work in model distillation and efficient natural language processing. It showed that pre-training distillation was viable for general-purpose models.
Questions people ask about DistilBERT
How much smaller is DistilBERT than BERT?
DistilBERT is forty per cent smaller than the BERT base model. Despite this reduction in size, it retains ninety-seven per cent of the original model's language understanding capabilities.
What data was used to train DistilBERT?
It was pretrained on the same corpus as BERT. This includes the BookCorpus dataset, which contains over eleven thousand unpublished books, and the English Wikipedia, excluding lists, tables, and headers.
Can DistilBERT generate text?
No, it is not designed for text generation. It is primarily intended for tasks that use the whole sentence to make decisions, such as classification or question answering. For generation, models like GPT-2 are recommended.
Good reading and sources
There is no English Wikipedia article on DistilBERT yet, so the facts on this page come from the sources listed above: huggingface.co/distilbert/distilbert-base-uncased, arxiv.org/abs/1910.01108. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.