Qwen-72B
A 72-billion parameter open-weight language model from Alibaba Cloud.
Alibaba Cloud released Qwen-72B in September 2023 as part of its Tongyi Qianwen series. This decoder-only transformer model features 72 billion parameters and supports a 32k token context window. It was trained on over three trillion tokens of multilingual data, offering strong performance in coding, mathematics, and general reasoning tasks while remaining accessible to researchers and developers.
- Contains 72 billion parameters in a decoder-only design.
- Trained on a corpus exceeding three trillion tokens.
- Supports a context window of 32,000 tokens.
- Uses a vocabulary of over 150,000 tokens for multilingual support.
What it is
Qwen-72B is a large language model developed by Alibaba Cloud. It contains 72 billion parameters and serves as the foundation for the broader Qwen series. The model handles text generation, logical reasoning, and code completion. Released in late 2023, it aims to provide a high-performance alternative to proprietary systems while maintaining open accessibility for the research community.
How it came about
The technical report for the Qwen series appeared on arXiv in September 2023. This publication introduced the base pretrained models and the chat-aligned variants. The team highlighted Qwen-72B as the flagship open-weight release within this initial lineup. It followed extensive internal development focused on optimizing training data distribution and model architecture for both English and Chinese tasks.
Inside the model
This model uses a standard decoder-only transformer architecture. It employs RoPE for position encoding, SwiGLU activation functions, and RMSNorm for stability. The tokenizer relies on tiktoken and includes a vocabulary of over 150,000 tokens. This large vocabulary improves compression rates for many languages beyond just English and Chinese. Training involved more than three trillion tokens of filtered web text, books, and code.
What it can do
Evaluations show strong results on benchmarks such as MMLU, C-Eval, and GSM8K. The model performs well in commonsense reasoning, mathematics, and programming tasks like HumanEval. It supports a context length of 32,000 tokens, allowing it to process lengthy documents. The developers claim it surpasses other open-source models of similar size in both Chinese and English evaluations.
How to use it
Alibaba Cloud provides the model weights openly via Hugging Face and ModelScope. Users can run the full precision version on hardware with at least 144GB of GPU memory. A quantized int4 version requires approximately 48GB. The codebase supports integration with standard libraries like Transformers and accelerates inference using flash attention techniques.
Questions people ask about Qwen-72B
What hardware is needed to run Qwen-72B?
Running the model in bf16 or fp16 precision requires at least 144GB of GPU memory, such as two A100-80G cards. For int4 quantization, you need a minimum of 48GB of VRAM.
How many tokens was Qwen-72B trained on?
The model underwent pretraining on a dataset comprising over three trillion tokens. This corpus includes web texts, professional books, code, and mathematical content after deduplication.
Does Qwen-72B support languages other than English and Chinese?
Yes, it uses a vocabulary of over 150,000 tokens optimized for multilingual efficiency. It shows high compression rates for languages including Japanese, Korean, Arabic, French, and Spanish.
Good reading and sources
There is no English Wikipedia article on Qwen-72B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2309.16609, huggingface.co/Qwen/Qwen-72B. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.