MuRIL
A BERT-style encoder pre-trained on 17 Indian languages and their transliterations
MuRIL is a multilingual language model built for Indian languages, pre-trained from scratch on a BERT base architecture using Indian text plus translated and transliterated pairs. It was introduced in March 2021 by a team of Google researchers led by Simran Khanuja and Partha Talukdar, and released with weights on Hugging Face and an encoder on TFHub.
- Pre-trained on 17 Indian languages plus their transliterated counterparts
- BERT base architecture, 1000K steps, batch size 4096, sequence length 512
- Trained on Wikipedia, Common Crawl, PMINDIA and Dakshina data
- Average PANX F1 of 77.60 versus 58.01 for multilingual BERT
- Weights released on Hugging Face, encoder on TFHub
What it is
MuRIL, short for Multilingual Representations for Indian Languages, is a language model designed for text in the languages of India. Google researchers built it on the BERT base architecture and trained it from scratch on corpora covering 17 languages, including Hindi, Bengali, Tamil, Telugu, Urdu and English. Unlike general multilingual models that spread capacity across a hundred or more languages, it concentrates on Indian text and also sees translated and transliterated versions of that text during training. The team released the pre-trained model with its masked language modelling head intact, so it can predict hidden words, and published an encoder version on TFHub with a pre-processing module.
How it came about
The model was presented in a paper by Simran Khanuja, Diksha Bansal, Sarvesh Mehtani and eleven co-authors, submitted to arXiv on 19 March 2021 and revised on 2 April 2021. The work set out to fix the weak showing of general multilingual systems on Indian languages, which the authors attributed to those models spreading their vocabulary and training data across 100 or more languages. The team followed a training approach close to multilingual BERT, changing how low-resource languages were upsampled and adding parallel signals. Weights were published on Hugging Face and an encoder on TFHub.
Inside the model
MuRIL uses the BERT base design, trained from scratch with a self-supervised masked language modelling objective using whole word masking and up to 80 predictions. Training ran for 1000K steps with a batch size of 4096 and a maximum sequence length of 512. Its data combined monolingual text from Wikipedia and Common Crawl with two kinds of parallel pairs: translations produced by the Google NMT pipeline plus the PMINDIA corpus, and transliterations made with the IndicTrans library plus the Dakshina dataset, with transliterated pairs limited to Wikipedia. Low-resource languages were upsampled using an exponent of 0.3 rather than 0.7. All parameters are trainable and the team recommends fine-tuning all of them.
What it can do
The model is meant for downstream NLP tasks in Indian languages and is trained to cope with text written in Latin script or mixed with English, a common pattern in informal Indian writing. On XTREME tasks with Indian test sets, evaluated zero-shot with English as the training language, it beat multilingual BERT on average in every reported task, for example 77.60 against 58.01 F1 on PANX and 74.07 against 66.81 accuracy on XNLI. Gains were larger on transliterated test sets, such as 57.70 against 14.24 F1 on transliterated PANX. The team notes it is not expected to do well on languages outside its 17 pre-training languages.
How to use it
The pre-trained model, with the masked language modelling layer left in place, is downloadable from Hugging Face as muril-base-cased. Google also published an encoder on TFHub with a pre-processing module that turns raw text into the expected input format. Queries and feedback go to a dedicated contact address.
Reception and debate
The paper reports that MuRIL significantly outperforms multilingual BERT on all tasks in the cross-lingual XTREME benchmark, and shows results on transliterated versions of the same test sets to demonstrate handling of Latin-script Indian text. The published tables compare the two models directly across named entity recognition, part-of-speech tagging, natural language inference, sentence retrieval and question answering.
Why it matters
MuRIL is presented as a model built specifically to close the gap for Indian languages in multilingual NLP, where general models trained on 100 or more languages give Indian text little representation. Its approach of adding translated and transliterated document pairs as supervised cross-lingual signals, and of upsampling low-resource languages more aggressively, offers a template for language-focused pre-training.
Questions people ask about MuRIL
What is MuRIL for?
It is a text encoder for Indian languages, meant to be fine-tuned on downstream NLP tasks such as named entity recognition, part-of-speech tagging, sentence retrieval and question answering. It is also built to handle Indian text written in Latin script or mixed with English.
Which languages does it cover?
Seventeen languages seen in pre-training: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Kashmiri, Malayalam, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Sindhi, Tamil, Telugu and Urdu. The team says it is not expected to perform well on languages outside this set.
How is it different from multilingual BERT?
It is trained only on Indian text rather than 100 or more languages, and it adds translated and transliterated document pairs as supervised cross-lingual signals. It also upsamples low-resource languages with an exponent of 0.3 instead of 0.7.
How do I get it?
The pre-trained model with its masked language modelling layer is on Hugging Face as muril-base-cased. An encoder with a pre-processing module is on TFHub. All parameters are trainable, and the team recommends fine-tuning all of them.
How large is it?
The published description gives the BERT base architecture and training settings, including 1000K steps and a maximum sequence length of 512, but it does not state a total parameter count.
Good reading and sources
There is no English Wikipedia article on MuRIL yet, so the facts on this page come from the sources listed above: huggingface.co/google/muril-base-cased, arxiv.org/abs/2103.10730. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.