News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Qwen-72B

Qwen-72B

A 72-billion parameter open-weight language model from Alibaba Cloud.

Alibaba Cloud released Qwen-72B in September 2023 as part of its Tongyi Qianwen series. This decoder-only transformer model features 72 billion parameters and supports a 32k token context window. It was trained on over three trillion tokens of multilingual data, offering strong performance in coding, mathematics, and general reasoning tasks while remaining accessible to researchers and developers.

Share on WhatsApp

  • Contains 72 billion parameters in a decoder-only design.
  • Trained on a corpus exceeding three trillion tokens.
  • Supports a context window of 32,000 tokens.
  • Uses a vocabulary of over 150,000 tokens for multilingual support.

What it is

Qwen-72B is a large language model developed by Alibaba Cloud. It contains 72 billion parameters and serves as the foundation for the broader Qwen series. The model handles text generation, logical reasoning, and code completion. Released in late 2023, it aims to provide a high-performance alternative to proprietary systems while maintaining open accessibility for the research community.

How it came about

The technical report for the Qwen series appeared on arXiv in September 2023. This publication introduced the base pretrained models and the chat-aligned variants. The team highlighted Qwen-72B as the flagship open-weight release within this initial lineup. It followed extensive internal development focused on optimizing training data distribution and model architecture for both English and Chinese tasks.

Inside the model

This model uses a standard decoder-only transformer architecture. It employs RoPE for position encoding, SwiGLU activation functions, and RMSNorm for stability. The tokenizer relies on tiktoken and includes a vocabulary of over 150,000 tokens. This large vocabulary improves compression rates for many languages beyond just English and Chinese. Training involved more than three trillion tokens of filtered web text, books, and code.

What it can do

Evaluations show strong results on benchmarks such as MMLU, C-Eval, and GSM8K. The model performs well in commonsense reasoning, mathematics, and programming tasks like HumanEval. It supports a context length of 32,000 tokens, allowing it to process lengthy documents. The developers claim it surpasses other open-source models of similar size in both Chinese and English evaluations.

How to use it

Alibaba Cloud provides the model weights openly via Hugging Face and ModelScope. Users can run the full precision version on hardware with at least 144GB of GPU memory. A quantized int4 version requires approximately 48GB. The codebase supports integration with standard libraries like Transformers and accelerates inference using flash attention techniques.

Questions people ask about Qwen-72B

What hardware is needed to run Qwen-72B?

Running the model in bf16 or fp16 precision requires at least 144GB of GPU memory, such as two A100-80G cards. For int4 quantization, you need a minimum of 48GB of VRAM.

How many tokens was Qwen-72B trained on?

The model underwent pretraining on a dataset comprising over three trillion tokens. This corpus includes web texts, professional books, code, and mathematical content after deduplication.

Does Qwen-72B support languages other than English and Chinese?

Yes, it uses a vocabulary of over 150,000 tokens optimized for multilingual efficiency. It shows high compression rates for languages including Japanese, Korean, Arabic, French, and Spanish.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Qwen-72B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2309.16609, huggingface.co/Qwen/Qwen-72B. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.