LLaDA
An 8-billion parameter diffusion language model
LLaDA is an 8-billion parameter language model built on diffusion architecture rather than autoregression. Released in 2025 by researchers including Shen Nie and Chongxuan Li, it challenges the assumption that large language capabilities require next-token prediction. The model matches LLaMA3 8B in general tasks and instruction following.
- Uses diffusion instead of autoregressive next-token prediction
- 8 billion parameters trained entirely from scratch
- Matches LLaMA3 8B in general task benchmarks
- Surpasses GPT-4o in reversal poem completion
- Open weights available on Hugging Face
What it is
LLaDA is a large language model with eight billion parameters that uses a diffusion process instead of the standard autoregressive method. It was introduced in early 2025 by a team of researchers associated with GSAI-ML. The model aims to prove that high-level language understanding and generation do not strictly depend on predicting the next word in a sequence. By employing a masking and unmasking strategy, it generates text through iterative refinement. The 8B instruct version is designed to follow user commands and engage in dialogue, showing performance comparable to established models like LLaMA3 8B.
How it came about
The project was first described in a paper submitted to arXiv in February 2025, titled Large Language Diffusion Models. The work was led by Shen Nie, Chongxuan Li, and colleagues including Fengqi Zhu and Ji-Rong Wen. They developed the model from scratch to test the scalability of diffusion approaches in natural language processing. An updated version of the code was released in October 2025 to support attention masks, improving its technical flexibility. The release included both the base pre-trained model and a supervised fine-tuned variant for instruction following.
Inside the model
Unlike traditional models that predict tokens one by one, LLaDA uses a forward masking process where parts of the data are hidden. A Transformer network then predicts these masked tokens during a reverse generation process. This approach optimizes a likelihood lower bound for probabilistic inference. The model was trained using pre-training and supervised fine-tuning paradigms. It does not rely on autoregressive baselines but constructs its own for comparison. The architecture allows it to address specific limitations found in sequential models, such as the reversal curse, by considering context more holistically during generation.
What it can do
The model demonstrates strong scalability across general tasks, mathematics, and coding benchmarks. In evaluations, the 8B parameter version performs comparably to LLaMA3 8B in in-context learning scenarios. After supervised fine-tuning, it shows robust instruction-following capabilities in multi-turn dialogues. Notably, the researchers claim it surpasses GPT-4o in reversal poem completion tasks, indicating a better grasp of bidirectional relationships in text. These results suggest that diffusion models can match the core competencies of leading autoregressive systems.
How to use it
The model weights and code are available openly through Hugging Face and GitHub. The developers provide an 8B instruct variant for direct use in chat and instruction-based applications. A project page offers demonstrations of its capabilities. The codebase was updated in late 2025 to include support for attention masks, allowing for more flexible integration into existing pipelines. Users can access the implementation to run the model locally or adapt it for further research.
Why it matters
LLaDA challenges the dominant view that autoregressive models are the only viable path for large-scale language intelligence. By showing competitive performance with a diffusion-based approach, it opens new avenues for architectural diversity in natural language processing. The work suggests that probabilistic inference via masking can be scaled effectively, potentially offering alternatives to next-token prediction for future model development.
Questions people ask about LLaDA
How does LLaDA generate text differently from other models?
Instead of predicting one word at a time from left to right, LLaDA uses a diffusion process. It starts with masked data and iteratively predicts the missing tokens using a Transformer, allowing it to consider the whole context simultaneously rather than sequentially.
Is LLaDA as powerful as LLaMA3?
The 8B parameter version of LLaDA is reported to be competitive with LLaMA3 8B in in-context learning and general benchmarks. After fine-tuning, it also shows strong instruction-following abilities comparable to established autoregressive models.
Can I download and run LLaDA myself?
Yes, the model weights and code are openly available. You can find the 8B-Instruct variant on Hugging Face and the implementation code on GitHub, allowing for local deployment and further experimentation.
Good reading and sources
There is no English Wikipedia article on LLaDA yet, so the facts on this page come from the sources listed above: huggingface.co/GSAI-ML/LLaDA-8B-Instruct, arxiv.org/abs/2502.09992. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.