BART
A denoising encoder-decoder transformer for generation and comprehension tasks
BART is a sequence-to-sequence transformer pre-trained by corrupting text and learning to rebuild it. It was introduced in 2019 by researchers at Facebook AI, and it matters because one design handles both text generation and understanding tasks well.
- Pre-training corrupts text, then trains the model to reconstruct it
- Best results came from sentence shuffling plus single-mask in-filling
- Reported ROUGE gains of up to 6 on summarisation and dialogue tasks
- Target-language pre-training added 1.1 BLEU over back-translation
- Weights released as facebook/bart-large via fairseq and Hugging Face
What it is
BART is a pre-trained transformer that reads text and writes text back out. It belongs to the sequence-to-sequence family, meaning an encoder handles the input while a decoder produces the output. The work came from researchers at Facebook AI and appeared as an arXiv paper in October 2019. The released large version is trained on English. It is aimed mainly at generation work such as summarisation and translation, though it also handles classification and question answering. The authors describe it as a generalisation of earlier pre-training schemes, combining a bidirectional encoder like BERT with a decoder that predicts left to right like GPT.
How it came about
The model was announced in a paper posted to arXiv on 29 October 2019, written by Mike Lewis and colleagues at Facebook AI. Code and weights first appeared in the fairseq repository on GitHub. The Hugging Face model card notes that the original team did not write one, so the card there was produced by Hugging Face staff instead.
Inside the model
Training has two steps. Text is first damaged by a noising function, then the model learns to restore the original wording. The team tested several ways of damaging text and settled on two together: scrambling the order of sentences and replacing stretches of text with one mask token, a method they call in-filling. The architecture is a standard transformer used for machine translation, with a bidirectional encoder and an autoregressive decoder. Ablation studies rebuilt other pre-training recipes inside this same framework to isolate which choices matter most.
What it can do
On GLUE and SQuAD, the authors report results matching RoBERTa given similar training resources. They report the strongest results at the time on abstractive dialogue, question answering and summarisation, with ROUGE gains of up to 6 points. For translation, adding target-language pre-training gave a 1.1 BLEU improvement over a back-translation baseline. Fine-tuning for generation is where the model is described as most effective.
How to use it
Weights for the large English version are published as facebook/bart-large and can be loaded through the Transformers library. The raw checkpoint supports text infilling, but the intended use is fine-tuning on labelled task data. Many fine-tuned variants are listed on the Hugging Face model hub.
Why it matters
The authors position BART as a framework that generalises BERT, GPT and other pre-training recipes under one denoising objective. Their ablations use it to compare those recipes on equal footing. Matching RoBERTa on understanding benchmarks while leading on generation tasks showed that a single encoder-decoder design could cover both kinds of work.
Questions people ask about BART
What kind of model is BART?
It is a transformer encoder-decoder, also called sequence-to-sequence. The encoder reads the input in both directions, similar to BERT, while the decoder writes output one token at a time, similar to GPT. That combination lets one model serve generation and understanding tasks.
How is it pre-trained?
Text is deliberately damaged by a noising function and the model must rebuild the original. The authors found the best recipe was to shuffle sentence order and replace spans of text with a single mask token. They also rebuilt other pre-training methods inside this framework to compare them.
What is it good at?
The authors report it is strongest after fine-tuning for generation, including summarisation, dialogue and translation. On GLUE and SQuAD it matched RoBERTa with comparable training resources. Reported gains reach up to 6 ROUGE on abstractive tasks.
How can I use it?
The large English checkpoint is published as facebook/bart-large and loads through the Transformers library in PyTorch. Out of the box it can do text infilling, but it is mainly meant to be fine-tuned on supervised data for a specific task.
Good reading and sources
There is no English Wikipedia article on BART yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1910.13461, huggingface.co/facebook/bart-large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.