AfriBERTa
Multilingual encoder model for eleven African languages
AfriBERTa is a pretrained language model with 126 million parameters, designed for text processing in eleven African languages. Released by researchers associated with Castorini, it uses an encoder architecture to support tasks like classification and entity recognition on low-resource data.
- Trained on less than 1 GB of text data
- Supports eleven African languages including Swahili and Hausa
- Contains 126 million parameters in an encoder architecture
- Effective for named entity recognition and text classification
What it is
AfriBERTa is a multilingual language model built for processing text in eleven African languages. It contains approximately 126 million parameters and follows an encoder-only design. The model supports languages such as Swahili, Hausa, Yorùbá, and Amharic. It aims to provide competitive performance on downstream tasks like text classification and named entity recognition, even for languages not included in its initial pretraining set.
How it came about
The model was introduced in 2021 alongside a paper titled Small Data? No Problem! This work explored the viability of pretrained models for low-resourced languages. The research was presented at the First Workshop on Multilingual Representation Learning in Punta Cana. The developers focused on creating a viable tool despite using a relatively small training corpus compared to major global models.
Inside the model
This encoder-based model features ten layers, six attention heads, and 768 hidden units. Its feed-forward size is set to 3072. Training relied on less than one gigabyte of text data aggregated from BBC News articles and Common Crawl. The dataset covers Afaan Oromoo, Amharic, Gahuza, Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yorùbá. Users must manually set the maximum sequence length to 512 tokens when using it with standard libraries due to SentencePiece compatibility issues.
What it can do
The model demonstrates competitive results in text classification and named entity recognition. It performs well on several African languages, including some it did not see during pretraining. However, its capabilities are constrained by the limited volume of training data. The reliance on news articles from a specific time period may affect its ability to generalize to other domains or linguistic styles.
How to use it
AfriBERTa is available with open weights through the Hugging Face platform under the identifier castorini/afriberta_large. Developers can integrate it into their workflows using the Transformers library for various downstream applications. The model is free to download and use for tasks such as token classification, provided users adjust the tokenizer settings correctly.
Questions people ask about AfriBERTa
Which languages does AfriBERTa support?
The model covers eleven African languages: Afaan Oromoo, Amharic, Gahuza, Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yorùbá. It may also perform reasonably well on related languages not explicitly included in the training set.
How large is the AfriBERTa model?
AfriBERTa large has approximately 126 million parameters. It uses an encoder-only architecture with ten layers, six attention heads, and 768 hidden units, making it relatively compact compared to larger global language models.
What are the main limitations of this model?
The model was trained on less than 1 GB of data, mostly from news articles. This limited scope means it may not capture complex linguistic relations or generalize well to domains outside of news media. Users should be aware of potential biases from the source data.
Good reading and sources
There is no English Wikipedia article on AfriBERTa yet, so the facts on this page come from the sources listed above: huggingface.co/castorini/afriberta_large. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.