News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
XLNet

XLNet

Permutation-based autoregressive transformer model created by Google AI

XLNet is an autoregressive transformer built by Google AI in June 2019 to improve upon BERT. By predicting tokens across random permutations of word order, it learns bidirectional context while preserving an autoregressive structure.

Share on WhatsApp

  • Trained across 512 TPU v3 chips for 5.5 days on 32.89 billion SentencePiece tokens
  • Employs a two-stream self-attention design to predict words in random permutations
  • Released under the Apache 2.0 open licence with 110M and 340M parameter options

What it is

XLNet is an autoregressive language model created by Google AI and published on 19 June 2019. Built as an evolution beyond BERT, it was designed to blend the strengths of autoregressive pre-training with bidirectional context learning. The system came in two sizes: a base version with 110 million parameters and a larger edition containing 340 million parameters. Rather than reading text exclusively from left to right or relying on fixed mask tokens, it predicts words using shuffled sequence orders to capture relationships across entire passages.

How it came about

Google AI published the project in June 2019, providing open source code on GitHub under the permissive Apache 2.0 licence. The release followed the rapid adoption of early transformer architectures such as BERT and generative pre-trained transformers, introducing an alternative pre-training objective that could learn contextual patterns without standard denoising masks.

Inside the model

Instead of predicting tokens in their normal sequence, XLNet factorises text using random permutations. This lets it inspect context from both sides while retaining an autoregressive setup. To prevent the model from seeing the target token during prediction, the design uses a two-stream self-attention system. The content stream encodes text representations through causal masks, while the query stream relies on masked cross-attention to probe target positions without revealing token identities. The training dataset totalled 32.89 billion SentencePiece tokens drawn from English Wikipedia, BooksCorpus, ClueWeb 2012-B, Giga5, and Common Crawl. Computation ran across 512 TPU v3 accelerators for 5.5 days, covering 500,000 steps with an Adam optimiser and a batch size of 8,192.

What it can do

By training across all possible ordering permutations, the system captures broader dependencies across sequences. When evaluated on benchmark tests, it set top marks across multiple natural language processing challenges, showing strong execution in natural language inference, question answering, and general language modelling tasks. At the conclusion of training, the model remained under-fitted, suggesting that prolonged training could yield even lower loss values.

How to use it

Code and pre-trained weights for the cased Base and Large models are publicly available on GitHub. Google AI distributed the project under the terms of the Apache 2.0 open-source licence.

Questions people ask about XLNet

What makes XLNet different from standard autoregressive models?

Standard autoregressive systems generate text strictly in sequence order. XLNet samples random permutations of the token order, allowing it to incorporate bidirectional context without relying on artificial mask tokens during evaluation.

What model sizes are available?

Two primary configurations were released: XLNet-Base, which has 110 million parameters across 12 layers, and XLNet-Large, featuring 340 million parameters distributed over 24 layers.

What data was used to train the system?

It was pre-trained on roughly 33 billion words, equalling 32.89 billion tokens. Sources included English Wikipedia, BooksCorpus, Common Crawl, ClueWeb 2012-B, and Giga5.

Good reading and sources

Where this page comes from

The facts on this page come from the Wikipedia article "XLNet", written by its contributors and used under the CC BY-SA 4.0 licence, and from its Wikidata record (CC0). EveryLLM rewrote the text in its own words, with AI assistance, and compares every page with the article to make sure no passage is copied. It may be reused under the same CC BY-SA 4.0 licence with a link back.

It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-16.

The weekly digest

Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.