DeBERTa
A BERT-style encoder with disentangled attention and an enhanced mask decoder.
DeBERTa is a pre-trained language model from Microsoft that improves on BERT and RoBERTa by separating word content from position in its attention mechanism. A 1.5 billion parameter version was the first single model to beat the human baseline on SuperGLUE, and a later V3 release raised efficiency further.
- First single model to beat the human baseline on SuperGLUE (89.9 vs 89.8)
- 1.5B parameter version with 48 transformer layers
- Disentangled attention separates word content from relative position
- DeBERTa-v3-large: 24 layers, hidden size 1024, 304M backbone parameters
- Trained on half the data of RoBERTa-Large yet scored higher on MNLI, SQuAD 2.0 and RACE
What it is
DeBERTa, short for Decoding-enhanced BERT with disentangled attention, is a pre-trained model for natural language understanding and generation tasks. It was proposed by researchers at Microsoft in June 2020 as an improvement on BERT and RoBERTa. The design represents each word with separate vectors for its content and its position, and adds absolute position information in the decoding layer. The largest version has 48 transformer layers and 1.5 billion parameters. DeBERTa-v3-large has 24 layers, a hidden size of 1024 and 304M backbone parameters.
How it came about
The DeBERTa paper was submitted in June 2020 by Pengcheng He, Xiaodong Liu, Jianfeng Gao and Weizhu Chen. A second version scaled the model to 1.5 billion parameters and, as of December 29, 2020, surpassed human performance on the SuperGLUE leaderboard for the first time. A third version replaced masked language modelling with a replaced token detection objective. The paper was revised several times, with the last revision in October 2021. DeBERTaV3 later added ELECTRA-style pre-training with gradient-disentangled embedding sharing.
Inside the model
DeBERTa uses disentangled attention: every word gets two vectors, one for content and one for relative position, and attention scores are computed with separate matrices for each. An enhanced mask decoder feeds absolute positions into the decoding layer when predicting masked tokens during pre-training. Fine-tuning uses a virtual adversarial training method to improve generalisation. DeBERTaV3 adopts ELECTRA-style pre-training, where a replaced token detection objective replaces masked language modelling, and shares embeddings in a gradient-disentangled way. The v3-large model has a 128K token vocabulary, adding 131M embedding parameters.
What it can do
A DeBERTa model trained on half the data used for RoBERTa-Large scored better across many tasks: 91.1% on MNLI against 90.2%, 90.7 on SQuAD v2.0 against 88.4, and 86.8 on RACE against 83.2. The 1.5B model reached a SuperGLUE macro-average of 89.9 versus the human 89.8, and an ensemble reached 90.3 as of January 6, 2021. DeBERTa-v3-large reports 91.5/89.0 F1/EM on SQuAD 2.0 and 91.8/91.9 accuracy on MNLI-m/mm.
How to use it
DeBERTa and DeBERTaV3 weights are published on Hugging Face under the microsoft organisation, with an official GitHub repository for implementation details. The models can be fine-tuned with the Hugging Face transformers library using standard text-classification scripts.
Reception and debate
The single DeBERTa model was the first to pass the human baseline on the SuperGLUE benchmark by macro-average score, and an ensemble version led the SuperGLUE leaderboard as of January 6, 2021, according to the authors.
Why it matters
DeBERTa showed that separating content and position in attention, plus an enhanced mask decoder, improves both understanding and generation tasks over BERT and RoBERTa. Its V3 follow-up applied ELECTRA-style pre-training to the design, and the models are widely used as fine-tuning baselines for natural language understanding.
Questions people ask about DeBERTa
What does DeBERTa stand for?
Decoding-enhanced BERT with disentangled attention. It is a Microsoft model that builds on BERT and RoBERTa by representing each word with separate content and position vectors and by adding absolute positions in the decoding layer.
How is DeBERTa different from BERT?
BERT mixes content and position in one representation. DeBERTa keeps them apart, computing attention with separate matrices for content and relative position, and adds an enhanced mask decoder that uses absolute positions when predicting masked tokens.
How large is DeBERTa?
The largest version has 48 transformer layers and 1.5 billion parameters. DeBERTa-v3-large has 24 layers, a hidden size of 1024, 304M backbone parameters and 131M embedding parameters from a 128K token vocabulary.
What is DeBERTaV3?
DeBERTaV3 is the follow-up release that improves training efficiency using ELECTRA-style pre-training with gradient-disentangled embedding sharing. It replaces masked language modelling with a replaced token detection objective and reports better downstream results than the original.
Good reading and sources
There is no English Wikipedia article on DeBERTa yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2006.03654, huggingface.co/microsoft/deberta-v3-large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.