XLNet
Permutation-based autoregressive transformer model created by Google AI
XLNet is an autoregressive transformer built by Google AI in June 2019 to improve upon BERT. By predicting tokens across random permutations of word order, it learns bidirectional context while preserving an autoregressive structure.
- Trained across 512 TPU v3 chips for 5.5 days on 32.89 billion SentencePiece tokens
- Employs a two-stream self-attention design to predict words in random permutations
- Released under the Apache 2.0 open licence with 110M and 340M parameter options
What it is
XLNet is an autoregressive language model created by Google AI and published on 19 June 2019. Built as an evolution beyond BERT, it was designed to blend the strengths of autoregressive pre-training with bidirectional context learning. The system came in two sizes: a base version with 110 million parameters and a larger edition containing 340 million parameters. Rather than reading text exclusively from left to right or relying on fixed mask tokens, it predicts words using shuffled sequence orders to capture relationships across entire passages.
How it came about
Google AI published the project in June 2019, providing open source code on GitHub under the permissive Apache 2.0 licence. The release followed the rapid adoption of early transformer architectures such as BERT and generative pre-trained transformers, introducing an alternative pre-training objective that could learn contextual patterns without standard denoising masks.
Inside the model
Instead of predicting tokens in their normal sequence, XLNet factorises text using random permutations. This lets it inspect context from both sides while retaining an autoregressive setup. To prevent the model from seeing the target token during prediction, the design uses a two-stream self-attention system. The content stream encodes text representations through causal masks, while the query stream relies on masked cross-attention to probe target positions without revealing token identities. The training dataset totalled 32.89 billion SentencePiece tokens drawn from English Wikipedia, BooksCorpus, ClueWeb 2012-B, Giga5, and Common Crawl. Computation ran across 512 TPU v3 accelerators for 5.5 days, covering 500,000 steps with an Adam optimiser and a batch size of 8,192.
What it can do
By training across all possible ordering permutations, the system captures broader dependencies across sequences. When evaluated on benchmark tests, it set top marks across multiple natural language processing challenges, showing strong execution in natural language inference, question answering, and general language modelling tasks. At the conclusion of training, the model remained under-fitted, suggesting that prolonged training could yield even lower loss values.
How to use it
Code and pre-trained weights for the cased Base and Large models are publicly available on GitHub. Google AI distributed the project under the terms of the Apache 2.0 open-source licence.
Questions people ask about XLNet
What makes XLNet different from standard autoregressive models?
Standard autoregressive systems generate text strictly in sequence order. XLNet samples random permutations of the token order, allowing it to incorporate bidirectional context without relying on artificial mask tokens during evaluation.
What model sizes are available?
Two primary configurations were released: XLNet-Base, which has 110 million parameters across 12 layers, and XLNet-Large, featuring 340 million parameters distributed over 24 layers.
What data was used to train the system?
It was pre-trained on roughly 33 billion words, equalling 32.89 billion tokens. Sources included English Wikipedia, BooksCorpus, Common Crawl, ClueWeb 2012-B, and Giga5.
Good reading and sources
- Pretrained models — transformers 2.0.0 documentationHugging Face, Article
- XLNet: Generalized Autoregressive Pretraining for Language UnderstandingarXiv, Article
The facts on this page come from the Wikipedia article "XLNet", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-16.