SOLAR 10.7B
An 11-billion parameter language model using depth up-scaling.
Upstage released SOLAR 10.7B in late 2023 as a compact large language model with 10.7 billion parameters. It uses a technique called depth up-scaling to improve performance beyond its size, rivaling much larger systems while remaining efficient for fine-tuning and deployment.
- Uses depth up-scaling instead of mixture of experts.
- Base weights released under Apache 2.0 licence.
- Outperforms Mixtral 8x7B-Instruct in benchmarks.
- Built by scaling up Mistral 7B architecture.
What it is
SOLAR 10.7B is a large language model developed by Upstage. It contains 10.7 billion parameters, placing it in the smaller tier of modern foundation models. The system was designed to deliver high performance in natural language processing tasks without the computational cost of massive networks. It serves as a base for further customisation through fine-tuning.
How it came about
The model was introduced in December 2023 via a technical paper submitted to arXiv. The work was later accepted to the NAACL 2024 Industry Track. Upstage built the model by adapting weights from Mistral 7B, aiming to create a more capable system through architectural changes rather than starting from scratch.
Inside the model
The core innovation is depth up-scaling (DUS). This method involves scaling the depth of the network and continuing pre-training on the expanded structure. Unlike mixture of experts approaches, DUS keeps the architecture dense and simpler to train. The base model integrates modified layers from Mistral 7B before undergoing further training to stabilise performance.
What it can do
Upstage claims the model outperforms other systems with up to 30 billion parameters. In benchmark tests, the instruct variant scored 74.20, surpassing Mixtral 8x7B-Instruct and Llama-2-70b-chat. The base model also showed strong results, scoring higher than several 7B and 14B competitors in general language tasks.
How to use it
The base model weights are available under the Apache 2.0 licence, allowing broad commercial and research use. The instruction-tuned variant uses a Creative Commons Non-Commercial licence because it was trained on datasets like Alpaca. Users can load the model via standard transformer libraries for text generation.
Questions people ask about SOLAR 10.7B
What is depth up-scaling?
Depth up-scaling is a method used by Upstage to expand a smaller model. It involves modifying the network depth and continuing pre-training, avoiding the complexity of mixture of experts systems.
Can I use SOLAR 10.7B commercially?
Yes, the base model is licensed under Apache 2.0. However, the instruction-tuned version uses a non-commercial licence due to the training data sources used during its fine-tuning phase.
How does it compare to larger models?
Upstage states it outperforms models with up to 30 billion parameters. Benchmarks show it scoring higher than Mixtral 8x7B and Llama-2-70b in specific instruction-following tasks.
Good reading and sources
There is no English Wikipedia article on SOLAR 10.7B yet, so the facts on this page come from the sources listed above: huggingface.co/upstage/SOLAR-10.7B-v1.0, arxiv.org/abs/2312.15166. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.