News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
BLOOMZ

BLOOMZ

Multilingual instruction-following model based on BLOOM

BLOOMZ is a family of large language models developed by the BigScience Workshop. It was created by fine-tuning the pretrained BLOOM and mT5 models to follow human instructions in dozens of languages. The project demonstrates that multitask finetuning enables zero-shot generalization to unseen tasks and languages.

Share on WhatsApp

  • Finetuned on 2.09 billion tokens from 46 languages
  • Uses 288 A100 GPUs for training
  • Generalizes to unseen tasks and languages
  • Based on the BLOOM and mT5 architectures
  • Openly available weights and code

What it is

BLOOMZ is a set of language models designed to understand and execute human instructions across many languages. It was built by the BigScience Workshop as a finetuned version of the BLOOM and mT5 pretrained models. The system aims to perform tasks expressed in natural language without needing specific examples for each new job. It supports dozens of languages and can generalize to tasks it has not seen during training. The models are available for researchers and developers to use in various applications requiring crosslingual capabilities.

How it came about

The model was introduced in a paper submitted in November 2022 titled Crosslingual Generalization through Multitask Finetuning. The work was led by Niklas Muennighoff and a large team of collaborators from the BigScience community. They applied multitask prompted finetuning to existing multilingual models. This approach extended previous English-focused studies to a broader linguistic context. The project released its code, datasets, and model weights to the public to support further research in open science.

Inside the model

BLOOMZ uses the same architecture as the original BLOOM model, which is a decoder-only transformer. The team finetuned the pretrained checkpoints on a dataset called xP3, which contains supervised tasks in 46 languages. This process involved 2.09 billion tokens. Training utilized 288 Nvidia A100 GPUs with 80GB of memory each, connected via NVLink. The software stack included Megatron-DeepSpeed for orchestration and PyTorch for neural network operations. The team found that using machine-translated prompts improved performance on human-written prompts in non-English languages.

What it can do

The model demonstrates strong zero-shot generalization, meaning it can handle new tasks without prior specific training. It performs well on both English and non-English tasks when finetuned on multilingual data. Tests show it can translate text, classify sentiment, and generate creative writing in languages like Spanish and Telugu. The system can even generalize to languages that were only present in the pretraining corpus but not in the finetuning data. Performance varies depending on how clearly the prompt defines the end of the input.

How to use it

The model weights are openly available on Hugging Face under an open licence. Developers can run the model on CPUs or GPUs using standard libraries like Transformers and Accelerate. The team provides code examples for loading the model in full precision or quantized to 8-bit for lower memory usage. It is recommended for prompting in English for the BLOOMZ variant and non-English for the mT0 variant. A version finetuned only on P3 was released strictly for research purposes.

Why it matters

BLOOMZ showed that large multilingual models can learn task-agnostic and language-agnostic capabilities through multitask finetuning. This finding suggests that models can acquire higher-level reasoning skills that transfer across linguistic boundaries. The release of the xP3 dataset and the associated code has supported further work in crosslingual natural language processing. It established a baseline for zero-shot performance in dozens of languages within the open-source community.

Questions people ask about BLOOMZ

What is BLOOMZ used for?

It is used for performing tasks expressed in natural language across many languages. Examples include translation, sentiment analysis, and creative writing. It works best when prompts clearly indicate where the input ends.

How many languages does it support?

The model was finetuned on data from 46 languages. It can also generalize to other languages that were part of the original pretraining corpus, even if they were not in the finetuning set.

Is BLOOMZ free to use?

Yes, the model weights, code, and datasets are freely available. Researchers and developers can download them from Hugging Face and run them locally using standard deep learning libraries.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on BLOOMZ yet, so the facts on this page come from the sources listed above: huggingface.co/bigscience/bloomz, arxiv.org/abs/2211.01786. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.