News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
BioBERT

BioBERT

Biomedical language model based on BERT architecture

BioBERT is a domain-specific language representation model designed for biomedical text mining. Developed by researchers including Jinhyuk Lee and Jaewoo Kang, it adapts the BERT architecture by pre-training on large-scale biomedical corpora to improve information extraction from scientific literature.

Share on WhatsApp

  • Pre-trained on large-scale biomedical corpora
  • Outperforms BERT in named entity recognition
  • Improves question answering by 12.24% MRR
  • Free pre-trained weights available online
  • Based on BERT encoder architecture

What it is

BioBERT is a specialised language model built for processing biomedical texts. It addresses the issue where general natural language processing tools struggle with the specific vocabulary and structure of medical literature. By adapting the BERT framework, this model learns from vast collections of biomedical documents. It serves as a foundation for tasks such as identifying named entities, extracting relationships between concepts, and answering complex questions within the life sciences domain.

How it came about

The model was introduced in January 2019 through a research paper submitted to arXiv. The work was led by Jinhyuk Lee and colleagues, who sought to apply recent advances in deep learning to the growing field of biomedical text mining. The team revised the initial submission later that year, with the final version appearing in October 2019. Their goal was to bridge the gap between general language models and the specialised needs of medical researchers.

Inside the model

BioBERT uses the same underlying architecture as BERT, which stands for Bidirectional Encoder Representations from Transformers. The key difference lies in its training data. Instead of relying on general web text, the developers pre-trained the model on large-scale biomedical corpora. This approach helps the system understand the unique word distribution found in scientific papers.

What it can do

The model demonstrates significant improvements over standard BERT in three key areas. For biomedical named entity recognition, it achieved a 0.62 per cent increase in F1 score. In relation extraction tasks, the improvement was 2.80 per cent. The most notable gain was in question answering, where BioBERT showed a 12.24 per cent improvement in Mean Reciprocal Rank. These results indicate that domain-specific pre-training allows the model to grasp complex biomedical texts more effectively than general-purpose alternatives.

How to use it

The creators have made the pre-trained weights for BioBERT freely available to the public. This open access approach allows other researchers and developers to build upon their work without needing to train a massive model from scratch. The resources are hosted online for easy download and integration into existing natural language processing pipelines.

Why it matters

BioBERT established that adapting general language models to specific domains yields better performance in specialised fields. It highlighted the importance of using relevant training data rather than relying solely on architectural changes. The model has become a reference point for subsequent developments in biomedical natural language processing, encouraging the creation of other domain-specific variants for different scientific disciplines.

Questions people ask about BioBERT

What is BioBERT used for?

It is designed for biomedical text mining tasks. These include identifying named entities in medical texts, extracting relationships between biological concepts, and answering questions based on scientific literature.

How does it differ from standard BERT?

While it uses the same architecture as BERT, BioBERT is pre-trained on biomedical corpora instead of general web text. This allows it to better understand the specific vocabulary and context of medical documents.

Is the model free to use?

Yes, the pre-trained weights are freely available. The developers also provide the source code needed for fine-tuning the model, allowing researchers to adapt it for their specific projects without cost.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on BioBERT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1901.08746. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.