Zamba2
Hybrid Mamba2-transformer language model suite by Zyphra
Zyphra released the Zamba2 suite in late 2024, featuring hybrid models with 1.2 billion to 7.4 billion parameters. These models combine state-space and transformer architectures to deliver high performance with low latency and efficient memory use.
- Hybrid Mamba2 and transformer architecture
- Trained on up to three trillion tokens
- Context window extendable to 16k tokens
- Open-source weights and training data
- Low inference latency and memory use
What it is
Zamba2 is a family of language models developed by Zyphra, available in sizes of 1.2 billion, 2.7 billion, and 7.4 billion parameters. Released in November 2024, these models are designed for general-purpose tasks while maintaining a small footprint. The largest variant, Zamba2-7B, serves as the flagship model, offering strong instruction-following capabilities suitable for various applications where efficiency is critical.
How it came about
The Zamba2 series builds upon the earlier Zamba1-7B model. Zyphra optimized the architecture, training procedures, and annealing datasets for this new suite. The team trained the models on up to three trillion tokens using a dataset called Zyda-2, which they also made publicly available. The technical report detailing these developments was published in November 2024.
Inside the model
These models use a hybrid design that mixes Mamba2 state-space layers with transformer blocks. To keep parameter costs low, the architecture uses shared weights for attention layers. It also applies LoRA projection matrices to shared multi-layer perceptrons, allowing each block to specialize slightly without adding significant overhead. The instruct version extends the context window from 4,000 to 16,000 tokens by adjusting rotary position embedding frequencies.
What it can do
Zyphra states that Zamba2 models achieve state-of-the-art performance compared to other open-weight models of similar size. They demonstrate strong instruction-following and reasoning skills. Tests show high success rates in retrieving specific information from long texts up to 16,000 tokens, though performance drops sharply beyond 18,000 tokens. The models are noted for rapid generation and low time to first token.
How to use it
Zyphra provides open-source weights for all models in the suite, including instruction-tuned variants. The pretraining dataset, Zyda-2, is also openly available. Users can run the models via Hugging Face transformers, though installing specific Mamba2 kernels from source is recommended for optimal latency and memory usage. A standalone PyTorch implementation is also provided.
Questions people ask about Zamba2
What makes Zamba2 different from standard transformers?
Zamba2 uses a hybrid architecture that combines Mamba2 state-space layers with transformer blocks. This design allows it to achieve lower inference latency and a smaller memory footprint than comparable pure transformer models while maintaining strong performance.
How large is the context window for Zamba2?
The base context length is 4,000 tokens. However, an experimental long-context mode extends this to 16,000 tokens by adjusting the rotation frequency of rotary position embeddings. Performance remains high up to this limit.
Is the training data for Zamba2 available?
Yes, Zyphra has open-sourced the pretraining dataset used for the Zamba2 series, which they call Zyda-2. This allows researchers and developers to inspect the data or use it for their own training purposes.
Good reading and sources
There is no English Wikipedia article on Zamba2 yet, so the facts on this page come from the sources listed above: huggingface.co/Zyphra/Zamba2-7B-Instruct, arxiv.org/abs/2411.15242. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.