CodeBERT
Bimodal pre-trained model for code and text
CodeBERT is a bimodal pre-trained model designed to process both programming languages and natural language. Developed by researchers including Zhangyin Feng and Ming Zhou, it uses a Transformer architecture to support tasks like code search and documentation generation.
- Supports natural language code search
- Generates code documentation automatically
- Uses replaced token detection for training
- Outperforms prior models in zero-shot probing
- Processes both code and natural language
What it is
CodeBERT is a pre-trained model that handles both programming code and natural language text. It was introduced in early 2020 by a team of researchers including Zhangyin Feng and Ming Zhou. The model aims to learn general-purpose representations that help computers understand the relationship between code and human descriptions. It supports applications such as searching for code using natural language queries and automatically generating documentation for software. By processing these two types of data together, it bridges the gap between how humans describe tasks and how machines execute them.
How it came about
The model was first submitted to arXiv in February 2020 and revised later that year. It was accepted to the Findings of EMNLP 2020 conference. The development involved a team of eleven authors who focused on creating a system capable of understanding bimodal data. This work built upon existing Transformer architectures but adapted them specifically for the unique challenges of pairing code with natural language instructions or comments.
Inside the model
CodeBERT uses a Transformer-based neural architecture. It is trained with a hybrid objective function that includes replaced token detection. This task involves detecting plausible alternatives for tokens that have been swapped out, sampled from generators. The training process utilizes both bimodal data, which consists of natural language and programming language pairs, and unimodal data. The unimodal data helps the model learn better generators, while the paired data provides direct input for learning the relationships between code and text.
What it can do
The model excels at natural language code search and code documentation generation. Evaluations show it achieves state-of-the-art performance on these specific downstream applications when its parameters are fine-tuned. Researchers also constructed a dataset for NL-PL probing to investigate the knowledge stored within the model. In zero-shot settings, where the model parameters remain fixed, CodeBERT performed better than previous pre-trained models, indicating it learns robust representations of code and language relationships.
How to use it
The model weights and associated code are available through platforms like Hugging Face. It was released as part of academic research, allowing the community to use and build upon the architecture. The paper provides detailed information on the training objectives and evaluation methods, enabling other developers to replicate the results or adapt the model for similar bimodal tasks involving programming and natural language.
Reception and debate
The work was accepted to Findings of EMNLP 2020, indicating peer recognition in the computational linguistics community. It is cited in various bibliographic tools and has been integrated into platforms like Hugging Face and CatalyzeX, suggesting adoption by researchers and developers interested in code intelligence and natural language processing applications.
Why it matters
CodeBERT established a baseline for bimodal pre-training in the domain of programming languages. Its approach to combining replaced token detection with bimodal data influenced subsequent models aimed at code understanding. By demonstrating superior performance in zero-shot probing, it highlighted the value of learning general-purpose representations that capture the semantic links between human language and machine code.
Questions people ask about CodeBERT
What is CodeBERT used for?
It is used for tasks that involve both programming code and natural language, such as searching for code snippets using plain English queries and automatically generating documentation for existing code bases.
How does CodeBERT learn?
It uses a Transformer architecture trained with a hybrid objective. This includes replaced token detection, where the model identifies plausible alternatives for masked tokens, using both paired code-text data and unimodal data.
Is CodeBERT better than previous models?
Yes, evaluations show it achieves state-of-the-art performance on natural language code search and documentation generation. It also outperforms earlier pre-trained models in zero-shot probing tasks.
Good reading and sources
There is no English Wikipedia article on CodeBERT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2002.08155. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.