ALBERT
A slimmed-down BERT encoder that shares weights across its layers
ALBERT is a family of English encoder models for self-supervised language representation, published by six Google researchers in September 2019. It cuts BERT's memory needs through parameter sharing and factorised embeddings, and the authors report leading GLUE, RACE and SQuAD results with fewer parameters than BERT-large.
- Submitted to arXiv on 26 September 2019 by six Google researchers
- All transformer layers share one set of weights to cut memory use
- Adds a sentence-ordering objective alongside masked language modelling
- Trained on BookCorpus (11,038 books) and English Wikipedia
- Uses a 30,000-word SentencePiece vocabulary over lowercased text
What it is
ALBERT, short for A Lite BERT, is a group of pretrained language encoders released in 2019 by Zhenzhong Lan and colleagues at Google. The models come in four sizes, from base up to xxlarge, and are trained on raw English text without human labels. Their purpose is to produce sentence and token representations that can be fed into a classifier or a question-answering head after fine-tuning. The design goal was to keep BERT's accuracy while using far fewer parameters, so that larger versions fit in accelerator memory and train faster.
How it came about
The work appeared on arXiv on 26 September 2019, with a final revision in February 2020. Six authors are listed: Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma and Radu Soricut. Code and pretrained checkpoints were published through the google-research/albert repository. A second xxlarge release followed, differing from the first in dropout settings, extra training data and a longer training run, which the model card says improved nearly every downstream task.
Inside the model
Two tricks shrink the parameter count. Embedding tables are factorised so the word vector size and the hidden size are decoupled, and every transformer layer in the stack reuses the same weights instead of having its own. Repeating one layer many times keeps memory small, though the arithmetic cost stays close to a BERT of the same depth. Pretraining uses masked language modelling, hiding 15% of tokens, plus a sentence-ordering objective that asks the model to judge which of two consecutive text segments came first. Text is lowercased and split with SentencePiece into a 30,000-word vocabulary.
What it can do
The authors report that their strongest model set new top scores on GLUE, RACE and SQuAD while using fewer parameters than BERT-large, and that the sentence-ordering loss helps tasks with multi-sentence inputs. Fine-tuned results are listed for four sizes: xxlarge reaches an average of 90.9 across the reported tasks, with 94.6/89.1 on SQuAD1.1 and 86.8 on RACE. The model card notes the encoders are meant for whole-sentence decisions such as classification and question answering, and points users to GPT-2 style models for text generation.
How to use it
Checkpoints and code were released openly in the google-research/albert repository, and the models are hosted on Hugging Face, where the xxlarge v2 card was written by the community rather than the original team. They can be loaded through the Transformers library for masked-word prediction or for feature extraction in PyTorch and TensorFlow. The raw model is mainly intended to be fine-tuned on a labelled task.
Safety and testing
The model card warns that the system can produce biased completions even though its training corpus is fairly neutral. Its demonstration shows gendered differences in suggested occupations for otherwise identical prompts, and states that such bias carries over into any model fine-tuned from these weights.
Reception and debate
The paper's abstract presents the approach as a way to scale representation models past accelerator memory limits, and reports benchmark results in support. No external reviews, disputes or bans are described here.
Why it matters
ALBERT is presented as a demonstration that parameter reduction and cross-layer weight sharing can let a BERT-style encoder grow larger without a matching rise in memory use. Its checkpoints remain available for downstream fine-tuning.
Questions people ask about ALBERT
What does ALBERT stand for?
The name expands to A Lite BERT. It is a reworked version of Google's BERT encoder, designed to hold far fewer parameters while keeping comparable accuracy on language understanding tasks.
How does it reduce the number of parameters?
Two methods are used. The word embedding table is factorised so its dimensions can differ from the hidden size, and every transformer layer in the network shares one set of weights rather than each having its own copy.
What was it trained on?
Pretraining used BookCorpus, a collection of 11,038 unpublished books, together with English Wikipedia, with lists and tables removed. No human labelling was involved; inputs and targets were generated automatically from the raw text.
Can it write text?
No. The model card says it is built for tasks that read a whole sentence to make a judgement, such as classification, token labelling and question answering, and recommends a GPT-2 style model when you need generation.
Are the weights free to download?
Yes. The authors published code and pretrained models in the google-research/albert repository, and the checkpoints are also hosted on Hugging Face for use with the Transformers library.
Good reading and sources
There is no English Wikipedia article on ALBERT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1909.11942, huggingface.co/albert/albert-xxlarge-v2. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.