News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Zamba2

Zamba2

Hybrid Mamba2-transformer language model suite by Zyphra

Zyphra released the Zamba2 suite in late 2024, featuring hybrid models with 1.2 billion to 7.4 billion parameters. These models combine state-space and transformer architectures to deliver high performance with low latency and efficient memory use.

Share on WhatsApp

  • Hybrid Mamba2 and transformer architecture
  • Trained on up to three trillion tokens
  • Context window extendable to 16k tokens
  • Open-source weights and training data
  • Low inference latency and memory use

What it is

Zamba2 is a family of language models developed by Zyphra, available in sizes of 1.2 billion, 2.7 billion, and 7.4 billion parameters. Released in November 2024, these models are designed for general-purpose tasks while maintaining a small footprint. The largest variant, Zamba2-7B, serves as the flagship model, offering strong instruction-following capabilities suitable for various applications where efficiency is critical.

How it came about

The Zamba2 series builds upon the earlier Zamba1-7B model. Zyphra optimized the architecture, training procedures, and annealing datasets for this new suite. The team trained the models on up to three trillion tokens using a dataset called Zyda-2, which they also made publicly available. The technical report detailing these developments was published in November 2024.

Inside the model

These models use a hybrid design that mixes Mamba2 state-space layers with transformer blocks. To keep parameter costs low, the architecture uses shared weights for attention layers. It also applies LoRA projection matrices to shared multi-layer perceptrons, allowing each block to specialize slightly without adding significant overhead. The instruct version extends the context window from 4,000 to 16,000 tokens by adjusting rotary position embedding frequencies.

What it can do

Zyphra states that Zamba2 models achieve state-of-the-art performance compared to other open-weight models of similar size. They demonstrate strong instruction-following and reasoning skills. Tests show high success rates in retrieving specific information from long texts up to 16,000 tokens, though performance drops sharply beyond 18,000 tokens. The models are noted for rapid generation and low time to first token.

How to use it

Zyphra provides open-source weights for all models in the suite, including instruction-tuned variants. The pretraining dataset, Zyda-2, is also openly available. Users can run the models via Hugging Face transformers, though installing specific Mamba2 kernels from source is recommended for optimal latency and memory usage. A standalone PyTorch implementation is also provided.

Questions people ask about Zamba2

What makes Zamba2 different from standard transformers?

Zamba2 uses a hybrid architecture that combines Mamba2 state-space layers with transformer blocks. This design allows it to achieve lower inference latency and a smaller memory footprint than comparable pure transformer models while maintaining strong performance.

How large is the context window for Zamba2?

The base context length is 4,000 tokens. However, an experimental long-context mode extends this to 16,000 tokens by adjusting the rotation frequency of rotary position embeddings. Performance remains high up to this limit.

Is the training data for Zamba2 available?

Yes, Zyphra has open-sourced the pretraining dataset used for the Zamba2 series, which they call Zyda-2. This allows researchers and developers to inspect the data or use it for their own training purposes.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Zamba2 yet, so the facts on this page come from the sources listed above: huggingface.co/Zyphra/Zamba2-7B-Instruct, arxiv.org/abs/2411.15242. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.