News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
ULMFiT

ULMFiT

Transfer learning method for NLP text classification tasks

ULMFiT is a transfer learning technique for natural language processing introduced in 2018 by Jeremy Howard and Sebastian Ruder. It allows pre-trained language models to be fine-tuned for specific text classification tasks, significantly reducing data needs and error rates compared to training from scratch.

Share on WhatsApp

  • Reduces error by 18-24% on most text classification datasets
  • Matches scratch-training performance with 100x less labeled data
  • Open-sourced pretrained models and code for public use
  • Introduced key techniques for effective language model fine-tuning

What it is

ULMFiT stands for Universal Language Model Fine-tuning. It is a method designed to apply transfer learning to natural language processing tasks. Developed by Jeremy Howard and Sebastian Ruder, it was first described in a paper submitted in January 2018. The approach allows a general language model to be adapted for specific text classification jobs. This reduces the need for large labeled datasets and improves performance over training models from scratch for each new task.

How it came about

The concept emerged from the observation that inductive transfer learning had already transformed computer vision. However, natural language processing still largely relied on task-specific modifications and training from scratch. Howard and Ruder proposed ULMFiT to bridge this gap. Their paper was submitted to arXiv in January 2018 and revised through May 2018. It was accepted for presentation at ACL 2018. The authors made their pretrained models and code available to the public.

Inside the model

The core design involves fine-tuning a pre-trained language model rather than starting anew. The authors introduced specific techniques crucial for this process. These methods allow the model to retain general linguistic knowledge while adapting to target tasks. The architecture relies on standard transformer or recurrent structures underlying the pre-trained model. The key innovation lies in the fine-tuning strategy, which includes gradual unfreezing of layers and discriminative learning rates. This ensures stable adaptation without catastrophic forgetting of prior knowledge.

What it can do

ULMFiT demonstrated significant improvements on six text classification benchmarks. It reduced error rates by 18 to 24 percent on most datasets compared to previous state-of-the-art methods. A notable capability is its data efficiency. With only 100 labeled examples, the method matched the performance of models trained from scratch on 100 times more data. This makes it particularly useful for domains where labeled data is scarce or expensive to obtain.

How to use it

The authors open-sourced both the pretrained models and the accompanying code. This allows researchers and developers to implement the method freely. The release supports reproducibility and further experimentation within the community. Users can download the weights and integrate the fine-tuning procedures into their own pipelines without restriction.

Reception and debate

The work was recognized by its acceptance at ACL 2018, a major conference in computational linguistics. It addressed a critical gap in NLP methodology by bringing transfer learning successes from vision to text. The reduction in error rates and data requirements drew attention to the practical benefits of universal fine-tuning. The open release of code facilitated rapid adoption and testing by other researchers.

Why it matters

ULMFiT influenced subsequent developments in NLP by proving the effectiveness of fine-tuning general language models. It paved the way for later architectures that rely heavily on transfer learning. The demonstration that small labeled datasets could yield high performance changed how practitioners approached low-resource tasks. It established fine-tuning as a standard practice in the field.

Questions people ask about ULMFiT

What is ULMFiT used for?

It is used for text classification tasks in natural language processing. It allows developers to take a general language model and adapt it to specific categories like sentiment analysis or topic detection with minimal data.

How much data does ULMFiT need?

It is highly data-efficient. The authors showed it can match the performance of models trained from scratch on large datasets using only 100 labeled examples. This makes it ideal for tasks with limited annotated data.

Is ULMFiT available for free?

Yes, the authors open-sourced the pretrained models and the code. Researchers and developers can download and use them without cost, facilitating widespread adoption and experimentation in the NLP community.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on ULMFiT yet, so the facts on this page come from the sources listed above: arxiv.org/abs/1801.06146. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.