News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
SOLAR 10.7B

SOLAR 10.7B

An 11-billion parameter language model using depth up-scaling.

Upstage released SOLAR 10.7B in late 2023 as a compact large language model with 10.7 billion parameters. It uses a technique called depth up-scaling to improve performance beyond its size, rivaling much larger systems while remaining efficient for fine-tuning and deployment.

Share on WhatsApp

  • Uses depth up-scaling instead of mixture of experts.
  • Base weights released under Apache 2.0 licence.
  • Outperforms Mixtral 8x7B-Instruct in benchmarks.
  • Built by scaling up Mistral 7B architecture.

What it is

SOLAR 10.7B is a large language model developed by Upstage. It contains 10.7 billion parameters, placing it in the smaller tier of modern foundation models. The system was designed to deliver high performance in natural language processing tasks without the computational cost of massive networks. It serves as a base for further customisation through fine-tuning.

How it came about

The model was introduced in December 2023 via a technical paper submitted to arXiv. The work was later accepted to the NAACL 2024 Industry Track. Upstage built the model by adapting weights from Mistral 7B, aiming to create a more capable system through architectural changes rather than starting from scratch.

Inside the model

The core innovation is depth up-scaling (DUS). This method involves scaling the depth of the network and continuing pre-training on the expanded structure. Unlike mixture of experts approaches, DUS keeps the architecture dense and simpler to train. The base model integrates modified layers from Mistral 7B before undergoing further training to stabilise performance.

What it can do

Upstage claims the model outperforms other systems with up to 30 billion parameters. In benchmark tests, the instruct variant scored 74.20, surpassing Mixtral 8x7B-Instruct and Llama-2-70b-chat. The base model also showed strong results, scoring higher than several 7B and 14B competitors in general language tasks.

How to use it

The base model weights are available under the Apache 2.0 licence, allowing broad commercial and research use. The instruction-tuned variant uses a Creative Commons Non-Commercial licence because it was trained on datasets like Alpaca. Users can load the model via standard transformer libraries for text generation.

Questions people ask about SOLAR 10.7B

What is depth up-scaling?

Depth up-scaling is a method used by Upstage to expand a smaller model. It involves modifying the network depth and continuing pre-training, avoiding the complexity of mixture of experts systems.

Can I use SOLAR 10.7B commercially?

Yes, the base model is licensed under Apache 2.0. However, the instruction-tuned version uses a non-commercial licence due to the training data sources used during its fine-tuning phase.

How does it compare to larger models?

Upstage states it outperforms models with up to 30 billion parameters. Benchmarks show it scoring higher than Mixtral 8x7B and Llama-2-70b in specific instruction-following tasks.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on SOLAR 10.7B yet, so the facts on this page come from the sources listed above: huggingface.co/upstage/SOLAR-10.7B-v1.0, arxiv.org/abs/2312.15166. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.