News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
Swallow

Swallow

Japanese-enhanced Llama models by Tokyo Tech

Swallow is a series of large language models built by the Swallow team at Tokyo Institute of Technology. It adapts Meta's Llama architecture through continual pre-training on Japanese data to improve local language performance while retaining English capabilities.

Share on WhatsApp

  • Built by Tokyo Institute of Technology researchers
  • Uses 200 billion tokens for Japanese adaptation
  • Based on Meta Llama 3.1 architecture
  • Supports both Japanese and English languages
  • Open weights available on Hugging Face

What it is

Swallow is a family of large language models developed by researchers at the Tokyo Institute of Technology. The project adapts Meta’s Llama models to better handle Japanese text. It uses continual pre-training, a method that further trains an existing model on new data without starting from scratch. The series includes versions with eight billion and seventy billion parameters. These models aim to support both Japanese and English tasks effectively.

How it came about

The initial research focused on adapting Llama 2 by expanding its vocabulary for Japanese characters. A paper describing this work appeared in April 2024. The team later updated the foundation to Llama 3.1. Releases in late 2024 included version 0.1 for both model sizes. Version 0.3 arrived in December 2024, offering improved instruction following and conversation skills for the larger model.

Inside the model

The models use a decoder-only transformer architecture inherited from Llama. Training involves continual pre-training on approximately two hundred billion tokens. This data comes from the Swallow Corpus Version 2, which includes Japanese web pages, Wikipedia articles, and coding examples. Instruction-tuned variants undergo supervised fine-tuning using synthetic data created specifically for Japanese interactions. The tokenizer and core structure remain consistent with the base Llama designs.

What it can do

Benchmarks show strong performance on Japanese tasks such as question answering and translation. The seventy billion parameter instruct model scored 71.15 on the Japanese MT-Bench in version 0.3. This represents an improvement over earlier versions. English capabilities remain competitive, with scores on standard tests like MMLU and GSM8K staying close to the base Llama 3.1 performance. The model handles multi-turn dialogue and reasoning tasks in both languages.

How to use it

The weights are available openly through Hugging Face. Users can download the base and instruction-tuned variants for local use. The project provides separate links for the eight billion and seventy billion parameter sizes. Documentation refers to the original Llama model cards for architectural details. Contact information is provided via an institutional email address for the research team.

Why it matters

The project demonstrates how continual pre-training can efficiently adapt large English-centric models to other languages. It offers a resource for developers needing robust Japanese language support without training a model from scratch. The approach preserves the general reasoning abilities of the base model while enhancing specific linguistic nuances.

Questions people ask about Swallow

What is the Swallow model?

Swallow is a series of large language models adapted from Meta's Llama. It is designed to improve Japanese language processing while keeping strong English skills. The Tokyo Institute of Technology developed it using continual pre-training techniques.

How was Swallow trained?

The team used continual pre-training on about 200 billion tokens. The data included Japanese web corpora, Wikipedia, and coding content. This method extends the existing Llama models rather than training a new one from zero.

Is Swallow open source?

Yes, the model weights are openly available. You can download the eight billion and seventy billion parameter versions from Hugging Face. The license terms follow those of the underlying Llama models.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on Swallow yet, so the facts on this page come from the sources listed above: huggingface.co/tokyotech-llm/Llama-3.1-Swallow-70B-Instruct-v0.3, arxiv.org/abs/2404.17790. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.