News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
CodeGen

CodeGen

Open autoregressive language model for code generation

Salesforce researchers released CodeGen in 2022 as a family of open large language models designed for program synthesis. With up to 16.1 billion parameters, it generates code from natural language prompts and supports multi-turn problem solving.

Share on WhatsApp

  • Family of models up to 16.1 billion parameters
  • Trained on Google TPU-v4-512 pods
  • Mono variant uses 71.7 billion Python tokens
  • Includes open training library JAXFORMER
  • Supports multi-turn program synthesis

What it is

CodeGen is a family of autoregressive language models built for program synthesis. Salesforce researchers developed these models to generate computer programs from natural language descriptions or input-output examples. The largest version contains 16.1 billion parameters. It aims to make advanced code generation tools accessible by releasing both the model checkpoints and the training library, known as JAXFORMER, under open licences.

How it came about

The team announced the project in March 2022 via an academic paper. They released four model sizes ranging from 350 million to 16 billion parameters. Three pre-training data variants were provided: NL, Multi, and Mono. The Mono variant was initialised from the Multi model and further trained on Python-specific data. The researchers updated their paper through early 2023 to include further details on the training process and benchmarks.

Inside the model

The models use an autoregressive transformer architecture trained with cross-entropy loss. Training leveraged data and model parallelism across multiple Google TPU-v4-512 pods. The Mono variant underwent additional pre-training on the BigPython dataset, which comprises 71.7 billion tokens of Python code. The team also open-sourced JAXFORMER, the library used to train the models, to support reproducibility and further research in the community.

What it can do

CodeGen performs zero-shot Python code generation, showing competitive results on the HumanEval benchmark. It excels at program synthesis where users provide English prompts as comment strings. The model can also complete partially written code. Researchers introduced the Multi-Turn Programming Benchmark (MTPB) with 115 problem sets. Tests showed that breaking problems into multiple prompts significantly improved synthesis performance compared to single-turn inputs.

How to use it

Salesforce released the model weights and the JAXFORMER training library as open source contributions. Checkpoints are available for download via Hugging Face and GitHub. The release includes variants for different data mixes and sizes. Users can load the models using standard transformer libraries. The providers state the release is primarily for research purposes and advise careful evaluation before deployment in production systems.

Safety and testing

The developers note that the models are not evaluated for all downstream purposes. They recommend users assess accuracy, safety, and fairness before deployment. The guidance highlights common limitations of AI systems. Users are urged to comply with applicable laws and avoid high-risk scenarios where errors could impact rights or safety. The release includes references to acceptable use policies for further guidance.

Why it matters

By opening the weights and training code, CodeGen aimed to democratise access to large language models for code. It provided a baseline for open models in program synthesis. The introduction of the MTPB benchmark offered a new way to evaluate multi-step reasoning in code generation. The availability of JAXFORMER allowed other researchers to replicate the training setup and experiment with similar architectures.

Questions people ask about CodeGen

What is CodeGen used for?

It is designed for program synthesis, generating executable code from natural language prompts or completing partial code. It works best when prompts are formatted as comment strings in English.

How large is the biggest CodeGen model?

The largest model in the family has 16.1 billion trainable parameters. Smaller variants include 350 million, 2 billion, and 6 billion parameters.

Is CodeGen open source?

Yes, Salesforce released the model checkpoints and the JAXFORMER training library as open source. Weights are available for download on platforms like Hugging Face.

What hardware was used to train CodeGen?

The models were trained using multiple TPU-v4-512 pods provided by Google. The training process leveraged both data and model parallelism to handle the large scale.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on CodeGen yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2203.13474, huggingface.co/Salesforce/codegen-16B-mono. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.