ELECTRA
A pre-training method that trains text encoders to spot replaced tokens.
ELECTRA is a pre-training approach for text encoders, published in March 2020 by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning. Instead of masking words like BERT, it corrupts text with a small generator and trains a discriminator to detect the replacements, learning from every token.
- Pre-trains encoders by detecting generator-replaced tokens, not by guessing masked ones
- Task covers every input token, so training is more sample-efficient
- A 4-day single-GPU model beat GPT on GLUE, with GPT using about 30x more compute
- Matched RoBERTa and XLNet using under a quarter of their compute
- Paper posted to arXiv on 23 March 2020
What it is
ELECTRA is a way to pre-train text encoders, the models that turn sentences into numerical representations for later tasks. It was set out in a March 2020 paper by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning. Rather than hiding some words and asking the model to guess them, the method feeds in text where certain tokens have been swapped for plausible substitutes produced by a small generator network. A second model then judges, token by token, which words were swapped. The authors report that this learns more from the same data and compute than masked language modelling.
How it came about
The work appeared on arXiv on 23 March 2020 as paper 2003.10555, under the title 'ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators'. The authors compare their approach against BERT, GPT, RoBERTa and XLNet. They report that a version trained on a single GPU for four days beat GPT on the GLUE benchmark despite GPT using roughly 30 times more compute.
Inside the model
The set-up has two parts. A small generator network fills gaps in the input with plausible replacement tokens. The main model, a discriminator, then labels every token as original or replaced. Because the task covers all tokens rather than only a masked subset, training is more sample-efficient. The authors say the learned representations beat BERT of the same size, on the same data and compute, with the largest gains in small models. At larger scale, the method matched RoBERTa and XLNet while using under a quarter of their compute, and passed them when given equal compute.
What it can do
Reported results centre on natural language understanding. On the GLUE benchmark, a small ELECTRA model trained for four days on one GPU outperformed GPT, which had used about 30 times more compute. Larger versions performed comparably to RoBERTa and XLNet with less than a quarter of their training compute, and better when compute was equalled. The paper stresses efficiency at small sizes as the main strength.
How to use it
The paper is public on arXiv with a DOI under 10.48550/arXiv.2003.10555. The arXiv listing links to code and media through third-party services including Hugging Face and CatalyzeX.
Why it matters
ELECTRA introduced replaced token detection as an alternative to masked language modelling, a pre-training task later reused in other encoder work. Its reported efficiency gains at small scale made it a common reference point for training compact encoders.
Questions people ask about ELECTRA
How is ELECTRA different from BERT?
BERT hides some tokens with a [MASK] symbol and trains the model to recover them. ELECTRA instead swaps tokens for plausible fakes from a small generator, then trains a discriminator to flag which positions were changed. The authors say this learns from every token rather than only the masked ones.
Who created it?
Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning, in a paper posted to arXiv on 23 March 2020.
What is it mainly good at?
The reported gains are in natural language understanding, especially at small model sizes. The authors highlight a model trained on one GPU for four days that beat GPT on GLUE despite GPT using roughly 30 times more compute.
What does the name stand for?
The sources give ELECTRA as the title of the paper and method. They do not expand it as an acronym.
Good reading and sources
There is no English Wikipedia article on ELECTRA yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2003.10555. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.