Kimi Linear
Hybrid linear attention model with 48 billion parameters
Kimi Linear is a hybrid language model released by the Kimi Team in October 2025. It uses 48 billion total parameters with only 3 billion active per token to improve speed and reduce memory use for long contexts.
- 48 billion total parameters with 3 billion active
- Supports context lengths up to 1 million tokens
- Reduces KV cache usage by up to 75 percent
- Trained on 5.7 trillion tokens of data
- Open weights available on Hugging Face
What it is
Kimi Linear is a language model architecture designed for high efficiency and performance across short and long contexts. Developed by the Kimi Team at Moonshot AI, it was released in October 2025. The model contains 48 billion total parameters but activates only 3 billion during inference. This design allows it to handle context windows up to one million tokens while significantly reducing memory usage compared to traditional full attention models.
How it came about
The Kimi Team introduced this architecture in a paper submitted to arXiv in late October 2025. The work builds on previous research into gated delta networks and linear attention mechanisms. The team released both base and instruction-tuned checkpoints trained on 5.7 trillion tokens. They also open-sourced the core kernel implementations to support further research and deployment in frameworks like vLLM.
Inside the model
The model uses a hybrid structure that combines Kimi Delta Attention (KDA) with Multi-Head Latent Attention (MLA) in a three-to-one ratio. KDA refines the gated delta rule with finer gating to better use finite-state recurrent memory. This approach reduces the need for large key-value caches by up to 75 percent. The architecture employs a chunkwise algorithm with specialized transition matrices to boost hardware efficiency and decoding throughput.
What it can do
Tests show the model achieves Pareto-optimal performance on long-context benchmarks like RULER. It records a score of 84.3 on RULER with a 128k context length. On MMLU-Pro with a 4k context, it scores 51.0. The system offers up to 6.3 times faster time per output token compared to standard MLA designs. It maintains competitive speed with full attention methods while handling sequences as long as one million tokens.
How to use it
Moonshot AI has made the model weights available under an open license. Developers can download the base and instruct versions from Hugging Face. The team provides code for integrating the model with Hugging Face Transformers and vLLM. This allows users to run local inference or create API endpoints compatible with OpenAI standards.
Questions people ask about Kimi Linear
How many parameters does Kimi Linear have?
The model has 48 billion total parameters. However, it is a mixture of experts style design where only 3 billion parameters are activated for each token during processing.
What is the maximum context length?
Kimi Linear supports a context window of up to one million tokens. It is designed to maintain efficiency and performance even at these extreme sequence lengths.
Is the model open source?
Yes, the team has released the model weights openly. You can download the base and instruction-tuned versions from Hugging Face and use them with standard libraries.
Good reading and sources
There is no English Wikipedia article on Kimi Linear yet, so the facts on this page come from the sources listed above: huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct, arxiv.org/abs/2510.26692. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.