Swallow
Japanese-enhanced Llama models by Tokyo Tech
Swallow is a series of large language models built by the Swallow team at Tokyo Institute of Technology. It adapts Meta's Llama architecture through continual pre-training on Japanese data to improve local language performance while retaining English capabilities.
- Built by Tokyo Institute of Technology researchers
- Uses 200 billion tokens for Japanese adaptation
- Based on Meta Llama 3.1 architecture
- Supports both Japanese and English languages
- Open weights available on Hugging Face
What it is
Swallow is a family of large language models developed by researchers at the Tokyo Institute of Technology. The project adapts Meta’s Llama models to better handle Japanese text. It uses continual pre-training, a method that further trains an existing model on new data without starting from scratch. The series includes versions with eight billion and seventy billion parameters. These models aim to support both Japanese and English tasks effectively.
How it came about
The initial research focused on adapting Llama 2 by expanding its vocabulary for Japanese characters. A paper describing this work appeared in April 2024. The team later updated the foundation to Llama 3.1. Releases in late 2024 included version 0.1 for both model sizes. Version 0.3 arrived in December 2024, offering improved instruction following and conversation skills for the larger model.
Inside the model
The models use a decoder-only transformer architecture inherited from Llama. Training involves continual pre-training on approximately two hundred billion tokens. This data comes from the Swallow Corpus Version 2, which includes Japanese web pages, Wikipedia articles, and coding examples. Instruction-tuned variants undergo supervised fine-tuning using synthetic data created specifically for Japanese interactions. The tokenizer and core structure remain consistent with the base Llama designs.
What it can do
Benchmarks show strong performance on Japanese tasks such as question answering and translation. The seventy billion parameter instruct model scored 71.15 on the Japanese MT-Bench in version 0.3. This represents an improvement over earlier versions. English capabilities remain competitive, with scores on standard tests like MMLU and GSM8K staying close to the base Llama 3.1 performance. The model handles multi-turn dialogue and reasoning tasks in both languages.
How to use it
The weights are available openly through Hugging Face. Users can download the base and instruction-tuned variants for local use. The project provides separate links for the eight billion and seventy billion parameter sizes. Documentation refers to the original Llama model cards for architectural details. Contact information is provided via an institutional email address for the research team.
Why it matters
The project demonstrates how continual pre-training can efficiently adapt large English-centric models to other languages. It offers a resource for developers needing robust Japanese language support without training a model from scratch. The approach preserves the general reasoning abilities of the base model while enhancing specific linguistic nuances.
Questions people ask about Swallow
What is the Swallow model?
Swallow is a series of large language models adapted from Meta's Llama. It is designed to improve Japanese language processing while keeping strong English skills. The Tokyo Institute of Technology developed it using continual pre-training techniques.
How was Swallow trained?
The team used continual pre-training on about 200 billion tokens. The data included Japanese web corpora, Wikipedia, and coding content. This method extends the existing Llama models rather than training a new one from zero.
Is Swallow open source?
Yes, the model weights are openly available. You can download the eight billion and seventy billion parameter versions from Hugging Face. The license terms follow those of the underlying Llama models.
Good reading and sources
There is no English Wikipedia article on Swallow yet, so the facts on this page come from the sources listed above: huggingface.co/tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3, arxiv.org/abs/2404.17790. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.