News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
DeBERTa

DeBERTa

A BERT-style encoder with disentangled attention and an enhanced mask decoder.

DeBERTa is a pre-trained language model from Microsoft that improves on BERT and RoBERTa by separating word content from position in its attention mechanism. A 1.5 billion parameter version was the first single model to beat the human baseline on SuperGLUE, and a later V3 release raised efficiency further.

Share on WhatsApp

  • First single model to beat the human baseline on SuperGLUE (89.9 vs 89.8)
  • 1.5B parameter version with 48 transformer layers
  • Disentangled attention separates word content from relative position
  • DeBERTa-v3-large: 24 layers, hidden size 1024, 304M backbone parameters
  • Trained on half the data of RoBERTa-Large yet scored higher on MNLI, SQuAD 2.0 and RACE

What it is

DeBERTa, short for Decoding-enhanced BERT with disentangled attention, is a pre-trained model for natural language understanding and generation tasks. It was proposed by researchers at Microsoft in June 2020 as an improvement on BERT and RoBERTa. The design represents each word with separate vectors for its content and its position, and adds absolute position information in the decoding layer. The largest version has 48 transformer layers and 1.5 billion parameters. DeBERTa-v3-large has 24 layers, a hidden size of 1024 and 304M backbone parameters.

How it came about

The DeBERTa paper was submitted in June 2020 by Pengcheng He, Xiaodong Liu, Jianfeng Gao and Weizhu Chen. A second version scaled the model to 1.5 billion parameters and, as of December 29, 2020, surpassed human performance on the SuperGLUE leaderboard for the first time. A third version replaced masked language modelling with a replaced token detection objective. The paper was revised several times, with the last revision in October 2021. DeBERTaV3 later added ELECTRA-style pre-training with gradient-disentangled embedding sharing.

Inside the model

DeBERTa uses disentangled attention: every word gets two vectors, one for content and one for relative position, and attention scores are computed with separate matrices for each. An enhanced mask decoder feeds absolute positions into the decoding layer when predicting masked tokens during pre-training. Fine-tuning uses a virtual adversarial training method to improve generalisation. DeBERTaV3 adopts ELECTRA-style pre-training, where a replaced token detection objective replaces masked language modelling, and shares embeddings in a gradient-disentangled way. The v3-large model has a 128K token vocabulary, adding 131M embedding parameters.

What it can do

A DeBERTa model trained on half the data used for RoBERTa-Large scored better across many tasks: 91.1% on MNLI against 90.2%, 90.7 on SQuAD v2.0 against 88.4, and 86.8 on RACE against 83.2. The 1.5B model reached a SuperGLUE macro-average of 89.9 versus the human 89.8, and an ensemble reached 90.3 as of January 6, 2021. DeBERTa-v3-large reports 91.5/89.0 F1/EM on SQuAD 2.0 and 91.8/91.9 accuracy on MNLI-m/mm.

How to use it

DeBERTa and DeBERTaV3 weights are published on Hugging Face under the microsoft organisation, with an official GitHub repository for implementation details. The models can be fine-tuned with the Hugging Face transformers library using standard text-classification scripts.

Reception and debate

The single DeBERTa model was the first to pass the human baseline on the SuperGLUE benchmark by macro-average score, and an ensemble version led the SuperGLUE leaderboard as of January 6, 2021, according to the authors.

Why it matters

DeBERTa showed that separating content and position in attention, plus an enhanced mask decoder, improves both understanding and generation tasks over BERT and RoBERTa. Its V3 follow-up applied ELECTRA-style pre-training to the design, and the models are widely used as fine-tuning baselines for natural language understanding.

Questions people ask about DeBERTa

What does DeBERTa stand for?

Decoding-enhanced BERT with disentangled attention. It is a Microsoft model that builds on BERT and RoBERTa by representing each word with separate content and position vectors and by adding absolute positions in the decoding layer.

How is DeBERTa different from BERT?

BERT mixes content and position in one representation. DeBERTa keeps them apart, computing attention with separate matrices for content and relative position, and adds an enhanced mask decoder that uses absolute positions when predicting masked tokens.

How large is DeBERTa?

The largest version has 48 transformer layers and 1.5 billion parameters. DeBERTa-v3-large has 24 layers, a hidden size of 1024, 304M backbone parameters and 131M embedding parameters from a 128K token vocabulary.

What is DeBERTaV3?

DeBERTaV3 is the follow-up release that improves training efficiency using ELECTRA-style pre-training with gradient-disentangled embedding sharing. It replaces masked language modelling with a replaced token detection objective and reports better downstream results than the original.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on DeBERTa yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2006.03654, huggingface.co/microsoft/deberta-v3-large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.