StarCoder
Open code generation model with 15.5 billion parameters
StarCoder is a 15.5 billion parameter language model for code, released in May 2023 by the BigCode community. It supports multiple programming languages and offers infilling capabilities, trained on one trillion tokens from permissively licensed GitHub repositories.
- Contains 15.5 billion parameters
- Trained on 1 trillion tokens from The Stack
- Supports 8K context length
- Achieves 40% pass@1 on HumanEval
- Uses multi-query attention for speed
What it is
StarCoder is a large language model designed specifically for programming tasks. Developed by the BigCode community, it contains 15.5 billion parameters and supports an context window of 8,000 tokens. The model handles multiple programming languages and includes features for code infilling. It was built to provide a powerful, open-access alternative for developers seeking automated coding assistance.
How it came about
The BigCode community, an open scientific collaboration, introduced StarCoder in May 2023. The project aimed to foster responsible development of code-specific language models. The team released two versions: StarCoderBase, the foundational model, and StarCoder, which underwent additional fine-tuning. This release represented a significant step in making high-performance code models available to the public under open licences.
Inside the model
The architecture uses a decoder-only transformer design with multi-query attention, which enables faster inference during large-batch processing. StarCoderBase was trained on one trillion tokens sourced from The Stack, a collection of permissively licensed GitHub repositories. The final StarCoder model resulted from fine-tuning this base on 35 billion Python tokens. The training process included an improved pipeline for redacting personally identifiable information.
What it can do
Evaluations indicate that StarCoderBase outperforms other open code models supporting multiple languages. It matches or exceeds the performance of OpenAI’s code-cushman-001 model. The fine-tuned StarCoder variant excels in Python tasks, achieving a 40 per cent pass rate on the HumanEval benchmark while maintaining competence in other programming languages. It supports infilling, allowing it to generate code within existing structures.
How to use it
The models are publicly available under a version of the Open Responsible AI Model license, which the developers describe as commercially viable. Users can access the weights directly. The release includes tools for inspection and an opt-out process for data contributors. This approach aims to balance open access with responsible usage and legal compliance for commercial applications.
Safety and testing
The development team implemented several safety measures before release. These include an enhanced pipeline for redacting personally identifiable information from the training data. They also introduced a novel attribution tracing tool to help track the origins of generated code. These steps were taken to ensure a safer open-access model release and to address concerns about data privacy and copyright.
Why it matters
StarCoder provided a comprehensive baseline for evaluating code language models at the time of its release. By offering a high-performing open-weight model, it allowed researchers and developers to build upon a robust foundation without relying solely on proprietary APIs. Its use of The Stack dataset set a precedent for using permissively licensed code in large-scale training.
Questions people ask about StarCoder
What is StarCoder used for?
StarCoder is designed for code generation and completion across multiple programming languages. It supports infilling tasks and can assist developers by predicting code segments based on surrounding context.
Is StarCoder open source?
Yes, the model weights are publicly available under the Open Responsible AI Model license. This licence is described as more commercially viable than some previous open licences, allowing broader use.
How was StarCoder trained?
The base model was trained on one trillion tokens from The Stack, a dataset of permissively licensed GitHub repositories. It was then fine-tuned on 35 billion Python tokens to create the final StarCoder model.
Good reading and sources
There is no English Wikipedia article on StarCoder yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2305.06161. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.