DeepSeek LLM 67B
A 67-billion-parameter open-source model trained on English and Chinese text.
DeepSeek LLM 67B is a large language model with 67 billion parameters, built by DeepSeek-AI and described in a paper posted in January 2024. It was trained from scratch on 2 trillion tokens of English and Chinese data, and released in both base and chat forms to support open research.
- 67 billion parameters, trained from scratch on 2 trillion tokens
- Training data covers both English and Chinese
- Base and chat versions of 7B and 67B released openly
- Chat version tuned with supervised fine-tuning and DPO
- Developers report it surpasses LLaMA-2 70B on code, maths and reasoning
What it is
DeepSeek LLM 67B is a general-purpose language model holding 67 billion parameters. DeepSeek-AI trained it from the ground up on a corpus of 2 trillion tokens covering English and Chinese. The work appears in a research paper posted on 5 January 2024, which frames the model as part of a project to push open-source language models forward over the long term. Two sizes were produced, 7B and 67B, each in a base version and a chat version. The chat model starts from the base weights and is then tuned on instruction data.
How it came about
The model comes from DeepSeek-AI, part of DeepSeek Inc., with a paper listing 86 authors led by Xiao Bi. The team set out to resolve disagreements in published scaling laws, the rules that predict how model quality improves with size and data. Their findings guided training in the 7B and 67B configurations. The pre-training dataset stood at 2 trillion tokens at the time of writing and was described as still growing. Base and chat weights for both sizes were opened to researchers.
Inside the model
The 67B chat model is initialised from the 67B base model and then fine-tuned on additional instruction data. The base model itself was trained from scratch on 2 trillion tokens of English and Chinese text. After supervised fine-tuning, the team applied Direct Preference Optimization, a method that aligns a model to human preferences without a separate reward model, to produce the chat variants. The tokenizer adds a beginning-of-sequence token to inputs by default. The team advises against system prompts with this release, as they are not supported.
What it can do
DeepSeek-AI reports that the 67B base model beats LLaMA-2 70B across several benchmarks, with the largest margins in code, mathematics and reasoning. In open-ended evaluation, the 67B chat model is reported to outperform GPT-3.5. These are the developers' own results from their paper.
How to use it
Weights for the 7B and 67B base and chat models were released openly for research. The code repository carries the MIT licence, while model use falls under a separate Model License that permits commercial use. The chat model can be run through the Transformers library, and DeepSeek also offers a chat interface online.
Why it matters
The work is presented as a contribution to open-source language models, built on a fresh look at scaling laws that the authors say had produced conflicting conclusions. Its scaling analysis was used to guide the 7B and 67B training runs, and the released weights were offered to the research community.
Questions people ask about DeepSeek LLM 67B
What is DeepSeek LLM 67B?
It is a 67-billion-parameter language model from DeepSeek-AI, trained from scratch on 2 trillion tokens of English and Chinese text. It comes in a base form and a chat form, the latter fine-tuned on instruction data and aligned with Direct Preference Optimization.
How does it compare with other models?
According to DeepSeek-AI's own paper, the 67B base model outperforms LLaMA-2 70B on a range of benchmarks, especially code, mathematics and reasoning. The chat version is reported to beat GPT-3.5 in open-ended evaluation.
Can I use it commercially?
Yes. The developers state that DeepSeek LLM supports commercial use under its Model License, while the code repository is under the MIT licence. The full model licence terms are in the LICENSE-MODEL file of the repository.
How do I run the chat model?
Load it with the Transformers library as a causal language model in bfloat16, then format messages with the provided chat template. The developers note that system prompts are not supported in this version and should be left out.
Good reading and sources
There is no English Wikipedia article on DeepSeek LLM 67B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2401.02954, huggingface.co/deepseek-ai/deepseek-llm-67b-chat. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.