StarCoder2
Open code generation model by BigCode
StarCoder2 is a family of open language models for code, released in 2024 by the BigCode project. It includes versions with 3 billion, 7 billion and 15 billion parameters. Trained on over four trillion tokens from more than 600 programming languages, it aims to support responsible development through transparent data sourcing.
- Trained on 4+ trillion tokens from 600+ languages
- 15B model outperforms CodeLlama-34B on many tasks
- Uses grouped-query and sliding window attention
- Weights released under OpenRAIL-M v1 license
- Dataset includes SWHIDs for full transparency
What it is
StarCoder2 is a set of language models designed for software engineering tasks. The BigCode project, an open scientific collaboration, developed these models to assist with code completion and reasoning. Released in early 2024, the family comprises three sizes: 3 billion, 7 billion and 15 billion parameters. These models process text inputs to generate source code snippets across hundreds of programming languages. They are not instruction-tuned chatbots but base models intended for further fine-tuning or direct code generation.
How it came about
The BigCode project built StarCoder2 as a successor to earlier models like StarCoderBase. In partnership with Software Heritage, the team constructed The Stack v2 dataset, which is four times larger than its predecessor. This new dataset draws from a digital commons of source code archives, covering 619 programming languages. The models were announced in February 2024 via a research paper detailing their training on trillions of tokens. The project emphasizes transparency by releasing persistent identifiers for the training data.
Inside the model
The architecture uses a transformer decoder design with specific attention mechanisms. It employs grouped-query attention and a sliding window attention span of 4,096 tokens within a larger context window of 16,384 tokens. Training utilized the Fill-in-the-Middle objective, allowing the model to predict missing code segments. The team trained the models on between 3.3 and 4.3 trillion tokens using the NVIDIA NeMo Framework. Computation was performed on the NVIDIA Eos Supercomputer, which relies on DGX H100 systems. The dataset includes GitHub pull requests, Kaggle notebooks and documentation.
What it can do
StarCoder2-3B outperforms other code models of similar size on most benchmarks. The 15-billion parameter version significantly exceeds comparable models and matches or beats CodeLlama-34B, despite having less than half the parameters. While DeepSeekCoder-33B leads in code completion for high-resource languages, StarCoder2-15B performs better on math and code reasoning tasks. It also shows strength in low-resource languages. The model can generate code snippets given context, though outputs may contain bugs or inefficiencies.
How to use it
The model weights are available under the BigCode OpenRAIL-M v1 license. This open access allows researchers and developers to download and use the models. The project provides tools for fine-tuning via GitHub repositories. Users can run the models on CPUs or GPUs, with support for quantization to reduce memory usage. The license requires adherence to responsible use guidelines, and the team offers a search index to trace generated code back to its training sources for attribution.
Safety and testing
The pretraining dataset was filtered to include only permissive licenses and code with no license. Opt-out requests were excluded from the training data. Despite these filters, the model might reproduce verbatim code from its dataset, which could require attribution. The developers provide a search tool to help users identify the origin of generated code and comply with license requirements. The model is not guaranteed to produce secure or efficient code, and users are warned about potential exploits.
Why it matters
StarCoder2 contributes to the open science movement in artificial intelligence by providing transparent data sources and open weights. Its performance challenges larger proprietary models, demonstrating that smaller, well-trained models can achieve competitive results. The release of The Stack v2 dataset supports broader research into code generation by offering a large, curated corpus of multilingual source code. This transparency helps address legal and ethical concerns regarding copyright in AI training.
Questions people ask about StarCoder2
Is StarCoder2 an instruction-following chatbot?
No, it is not an instruction model. Commands like writing a specific function do not work well without fine-tuning. It is a base model designed for code completion and generation tasks rather than conversational interaction.
What license covers the StarCoder2 models?
The models are licensed under the BigCode OpenRAIL-M v1 agreement. This open license allows for use and modification but includes conditions for responsible deployment and attribution requirements for generated code.
How large is the context window for StarCoder2?
The model supports a context window of 16,384 tokens. It uses a sliding window attention mechanism of 4,096 tokens to manage this context efficiently during processing and generation.
Good reading and sources
There is no English Wikipedia article on StarCoder2 yet, so the facts on this page come from the sources listed above: huggingface.co/bigcode/starcoder2-15b, arxiv.org/abs/2402.19173. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.