News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
TinyLlama

TinyLlama

A compact 1.1 billion parameter open language model.

TinyLlama is a small, open-source language model with 1.1 billion parameters. Developed by researchers including Peiyuan Zhang, it was pretrained on trillions of tokens using the Llama 2 architecture. The project aims to provide a efficient base for various applications requiring low computational resources.

Share on WhatsApp

  • Contains 1.1 billion parameters
  • Pretrained on 3 trillion tokens
  • Uses Llama 2 architecture and tokenizer
  • Trained on 16 A100 GPUs in 90 days
  • Open-source weights and code available

What it is

TinyLlama is a compact language model containing 1.1 billion parameters. It was designed to be pretrained on a massive dataset of three trillion tokens. The project utilizes the same architecture and tokenizer as Llama 2, allowing it to integrate easily with existing open-source tools built for that family. Its small size makes it suitable for applications with limited memory or computing power. The model serves as a base for further fine-tuning, including chat variants.

How it came about

The training process began in September 2023. The developers aimed to complete the pretraining phase within ninety days using sixteen A100 graphics processing units. The team included Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. They published their findings in a paper titled TinyLlama: An Open-Source Small Language Model in January 2024. The project leverages community contributions such as FlashAttention and Lit-GPT to improve efficiency.

Inside the model

The model follows the dense decoder transformer design of Llama 2 exactly. This includes using the same tokenizer, which helps with compatibility. Training involved processing approximately three trillion tokens over three epochs. The developers employed optimization techniques from the open-source community to achieve high computational efficiency. For the chat version, the base model was fine-tuned on synthetic dialogues from the UltraChat dataset. Further alignment used direct preference optimization on the UltraFeedback dataset.

What it can do

Despite its small scale, the model shows strong performance in downstream tasks. It outperforms other open-source models of similar size in various benchmarks. The chat variant can engage in dialogue and follow instructions. It supports multiple languages due to its Llama 2 heritage. The compact nature allows it to run on hardware that cannot support larger models, making it useful for edge devices or restricted environments.

How to use it

The model weights and code are publicly available on GitHub and Hugging Face. It is released under an open-source licence, permitting free use and modification. Users can access the base pretrained model as well as the chat-finetuned version. Integration requires the transformers library version 4.34 or higher. The open availability encourages community experimentation and deployment in diverse projects.

Why it matters

TinyLlama demonstrates that small models can be trained efficiently on large datasets. It provides a viable alternative for developers who need lightweight solutions. By matching the Llama 2 architecture, it lowers the barrier for adopting smaller models in existing pipelines. The project highlights the value of community-driven optimizations like FlashAttention in reducing training costs and time.

Questions people ask about TinyLlama

What is the size of TinyLlama?

TinyLlama has 1.1 billion parameters. This small size allows it to run on devices with limited memory and computational resources compared to larger language models.

Which architecture does TinyLlama use?

It uses the exact same architecture and tokenizer as Llama 2. This design choice ensures compatibility with many open-source projects and tools already built for the Llama ecosystem.

How much data was used for training?

The model was pretrained on approximately three trillion tokens. This large dataset helps the small model achieve competitive performance on various downstream natural language processing tasks.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on TinyLlama yet, so the facts on this page come from the sources listed above: huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0, arxiv.org/abs/2401.02385. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.