News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
SciBERT

SciBERT

A BERT model trained on scientific text.

SciBERT is a pretrained language model based on BERT, designed for scientific text. Developed by researchers at the Allen Institute for AI, it was released in 2019 to address the lack of large-scale labelled data in scientific domains. It uses a custom vocabulary and training corpus from Semantic Scholar.

Share on WhatsApp

  • Trained on 1.14 million scientific papers
  • Uses a custom scivocab vocabulary
  • Released by Allen Institute for AI in 2019
  • Based on the BERT encoder architecture
  • Improves over BERT on scientific NLP tasks

What it is

SciBERT is a specialised language model built on the BERT architecture. It focuses on processing scientific literature rather than general web text. The Allen Institute for AI created it to help machines understand technical papers. Released in 2019, it aims to solve the problem of scarce annotated data in science. The model comes in cased and uncased versions. It uses a unique vocabulary called scivocab, tailored to match scientific terminology found in its training set.

How it came about

Researchers Iz Beltagy, Kyle Lo, and Arman Cohan developed the model. They submitted their work to arXiv in March 2019. The paper describes how they adapted BERT for scientific use. The team released the code and pretrained weights to the public. This allowed other scientists to use the model for their own natural language processing tasks. The project is hosted on GitHub and Hugging Face.

Inside the model

The model uses an encoder-only transformer design, similar to the original BERT. Its key difference is the training data and vocabulary. Instead of general internet text, it learned from 1.14 million full-text papers sourced from Semantic Scholar. This corpus contains 3.1 billion tokens. The creators built a custom wordpiece vocabulary, named scivocab, to better capture scientific terms. This approach helps the model handle domain-specific language more effectively than generic models.

What it can do

SciBERT performs well on tasks like sequence tagging, sentence classification, and dependency parsing. Tests show it achieves statistically significant improvements over standard BERT on scientific datasets. It reached new state-of-the-art results on several benchmarks involving various scientific domains. The model is particularly useful when high-quality labelled data is expensive or hard to find.

How to use it

The weights are open and available for download. Users can find the model on Hugging Face under the name allenai/scibert_scivocab_uncased. The original code repository is hosted on GitHub by the Allen Institute. The developers ask users to cite their 2019 paper if they use the model in their work. Both cased and uncased versions are provided.

Why it matters

SciBERT demonstrated that domain-specific pretraining improves performance on specialised tasks. It provided a strong baseline for natural language processing in science. By releasing the model openly, it enabled wider research into scientific text mining. Many subsequent projects have used its architecture or vocabulary as a starting point for other scientific language models.

Questions people ask about SciBERT

What is SciBERT used for?

It is used for natural language processing tasks on scientific text, such as classifying sentences or tagging sequences in academic papers.

How is it different from regular BERT?

SciBERT is trained on scientific publications instead of general web text. It also uses a custom vocabulary called scivocab to better understand technical terms.

Is SciBERT free to use?

Yes, the pretrained models and code are open. You can download them from Hugging Face or GitHub, but you should cite the original paper.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on SciBERT yet, so the facts on this page come from the sources listed above: huggingface.co/allenai/scibert_scivocab_uncased, arxiv.org/abs/1903.10676. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.