MaLLaM
Large language models trained on Malaysian context and Malay language data.
MaLLaM is a family of large language models developed to address the lack of pretrained models with Malaysian context. Released in January 2024, it includes versions with 1.1 billion, 3 billion, and 5 billion parameters, trained on 90 billion tokens to improve natural language understanding for Malay.
- Trained on 349GB dataset equal to 90 billion tokens
- Available in 1.1B, 3B, and 5B parameter sizes
- Focuses on Malaysian context and Malay language
- Released with open weights in January 2024
- Competitive with ChatGPT-3.5 in instruction tasks
What it is
MaLLaM stands for Malaysia Large Language Model. It is a set of artificial intelligence models designed to process and generate text with a specific focus on the Malaysian context. The project addresses a gap in existing technology, as few large models were pretrained from scratch using local data. The team released three sizes: 1.1 billion, 3 billion, and 5 billion parameters. These models aim to enhance tasks involving natural language understanding and generation, particularly for the Malay language.
How it came about
The authors submitted their paper describing MaLLaM to arXiv in January 2024. The initial version appeared on 26 January, with a revision following on 29 January. The work was led by Husein Zolkepli, along with Aisyah Razak, Kamarul Adha, and Ariff Nazhan. They sought to create a model that captures linguistic nuances present in Malaysia, moving beyond generic global datasets to provide comprehensive language representations grounded in local context.
Inside the model
The models were trained on a dataset totaling 349 gigabytes. This volume equates to approximately 90 billion tokens when processed through their custom pretrained Byte Pair Encoding tokenizer. The training involved a single epoch over this data. The team employed instruction tuning to refine the base models. This approach allows the system to follow user commands more effectively. The architecture supports standard decoder-based operations common in modern language models, focusing on textual data rather than multimodal inputs.
What it can do
Despite being trained on a smaller dataset compared to some global counterparts, the instruction-tuned MaLLaM models perform competitively. The authors claim notable proficiency when compared to ChatGPT-3.5 and Malaysian Mistral. The models demonstrate effectiveness in capturing and understanding the specific nuances of the Malaysian language. They are suited for enhanced natural language understanding and generation tasks that require local cultural and linguistic awareness.
How to use it
All MaLLaM models are released openly. The weights are available for download, allowing researchers and developers to use them locally. The project emphasizes community access and transparency. Users can find the models on platforms such as Hugging Face. This open approach aims to pave the way for further advancements in large language models within the Malaysian context, enabling others to build upon the foundational work.
Why it matters
MaLLaM contributes to the field by providing comprehensive language representations grounded in Malaysian context. It highlights the effectiveness of training on localized data even at smaller scales. The project aims to advance capabilities for the Malay language, offering an alternative to global models that may not fully capture regional linguistic subtleties. It serves as a resource for those seeking to develop applications specific to Malaysia.
Questions people ask about MaLLaM
What languages does MaLLaM support?
MaLLaM is designed primarily for the Malay language and the broader Malaysian context. It aims to capture linguistic nuances specific to Malaysia, improving natural language understanding and generation for users in that region.
How large are the MaLLaM models?
The project released three model sizes. These include versions with 1.1 billion, 3 billion, and 5 billion parameters. All were trained on the same dataset of 90 billion tokens using a Byte Pair Encoding tokenizer.
Is MaLLaM free to use?
Yes, all MaLLaM models are released with open weights. This means developers and researchers can download and use them freely. The models are hosted on platforms like Hugging Face for easy access.
Good reading and sources
There is no English Wikipedia article on MaLLaM yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2401.14680. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.