News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
ELECTRA

ELECTRA

A pre-training method that trains text encoders to spot replaced tokens.

ELECTRA is a pre-training approach for text encoders, published in March 2020 by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning. Instead of masking words like BERT, it corrupts text with a small generator and trains a discriminator to detect the replacements, learning from every token.

Share on WhatsApp

  • Pre-trains encoders by detecting generator-replaced tokens, not by guessing masked ones
  • Task covers every input token, so training is more sample-efficient
  • A 4-day single-GPU model beat GPT on GLUE, with GPT using about 30x more compute
  • Matched RoBERTa and XLNet using under a quarter of their compute
  • Paper posted to arXiv on 23 March 2020

What it is

ELECTRA is a way to pre-train text encoders, the models that turn sentences into numerical representations for later tasks. It was set out in a March 2020 paper by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning. Rather than hiding some words and asking the model to guess them, the method feeds in text where certain tokens have been swapped for plausible substitutes produced by a small generator network. A second model then judges, token by token, which words were swapped. The authors report that this learns more from the same data and compute than masked language modelling.

How it came about

The work appeared on arXiv on 23 March 2020 as paper 2003.10555, under the title 'ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators'. The authors compare their approach against BERT, GPT, RoBERTa and XLNet. They report that a version trained on a single GPU for four days beat GPT on the GLUE benchmark despite GPT using roughly 30 times more compute.

Inside the model

The set-up has two parts. A small generator network fills gaps in the input with plausible replacement tokens. The main model, a discriminator, then labels every token as original or replaced. Because the task covers all tokens rather than only a masked subset, training is more sample-efficient. The authors say the learned representations beat BERT of the same size, on the same data and compute, with the largest gains in small models. At larger scale, the method matched RoBERTa and XLNet while using under a quarter of their compute, and passed them when given equal compute.

What it can do

Reported results centre on natural language understanding. On the GLUE benchmark, a small ELECTRA model trained for four days on one GPU outperformed GPT, which had used about 30 times more compute. Larger versions performed comparably to RoBERTa and XLNet with less than a quarter of their training compute, and better when compute was equalled. The paper stresses efficiency at small sizes as the main strength.

How to use it

The paper is public on arXiv with a DOI under 10.48550/arXiv.2003.10555. The arXiv listing links to code and media through third-party services including Hugging Face and CatalyzeX.

Why it matters

ELECTRA introduced replaced token detection as an alternative to masked language modelling, a pre-training task later reused in other encoder work. Its reported efficiency gains at small scale made it a common reference point for training compact encoders.

Questions people ask about ELECTRA

How is ELECTRA different from BERT?

BERT hides some tokens with a [MASK] symbol and trains the model to recover them. ELECTRA instead swaps tokens for plausible fakes from a small generator, then trains a discriminator to flag which positions were changed. The authors say this learns from every token rather than only the masked ones.

Who created it?

Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning, in a paper posted to arXiv on 23 March 2020.

What is it mainly good at?

The reported gains are in natural language understanding, especially at small model sizes. The authors highlight a model trained on one GPU for four days that beat GPT on GLUE despite GPT using roughly 30 times more compute.

What does the name stand for?

The sources give ELECTRA as the title of the paper and method. They do not expand it as an acronym.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on ELECTRA yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2003.10555. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.