News
ReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior ReleaseReleasesOVHcloud unveils Olala mini, its first in-house LLMChinaA 9B model beats a 1T model on FinQA, trained with Crusoe Managed RLReleasesThe internal metaphysics of an LLMReleasesMapping Language Model Internal Representations to Physical Laws and Scientific ReasoningResearchCan China keep rogue AI in check? Meet the start-up testing the top modelsFrontierMicrosoft aims to challenge OpenAI, Anthropic with cheaper, specialized AI modelsOpen modelsMicrosoft rolls out new first-party models and updates to agentic AI continuous improvementFrontierMicrosoft unveils AI models for coding, cybersecurity as it takes on Anthropic, OpenAIFrontierGoogle is making it easier to opt out of Gemini AI model training – here’s howFrontierOpenAI Unveils New Astra AI Model After Scrapping Prior Release
EveryLLM
DeepSeek LLM 67B

DeepSeek LLM 67B

A 67-billion-parameter open-source model trained on English and Chinese text.

DeepSeek LLM 67B is a large language model with 67 billion parameters, built by DeepSeek-AI and described in a paper posted in January 2024. It was trained from scratch on 2 trillion tokens of English and Chinese data, and released in both base and chat forms to support open research.

Share on WhatsApp

  • 67 billion parameters, trained from scratch on 2 trillion tokens
  • Training data covers both English and Chinese
  • Base and chat versions of 7B and 67B released openly
  • Chat version tuned with supervised fine-tuning and DPO
  • Developers report it surpasses LLaMA-2 70B on code, maths and reasoning

What it is

DeepSeek LLM 67B is a general-purpose language model holding 67 billion parameters. DeepSeek-AI trained it from the ground up on a corpus of 2 trillion tokens covering English and Chinese. The work appears in a research paper posted on 5 January 2024, which frames the model as part of a project to push open-source language models forward over the long term. Two sizes were produced, 7B and 67B, each in a base version and a chat version. The chat model starts from the base weights and is then tuned on instruction data.

How it came about

The model comes from DeepSeek-AI, part of DeepSeek Inc., with a paper listing 86 authors led by Xiao Bi. The team set out to resolve disagreements in published scaling laws, the rules that predict how model quality improves with size and data. Their findings guided training in the 7B and 67B configurations. The pre-training dataset stood at 2 trillion tokens at the time of writing and was described as still growing. Base and chat weights for both sizes were opened to researchers.

Inside the model

The 67B chat model is initialised from the 67B base model and then fine-tuned on additional instruction data. The base model itself was trained from scratch on 2 trillion tokens of English and Chinese text. After supervised fine-tuning, the team applied Direct Preference Optimization, a method that aligns a model to human preferences without a separate reward model, to produce the chat variants. The tokenizer adds a beginning-of-sequence token to inputs by default. The team advises against system prompts with this release, as they are not supported.

What it can do

DeepSeek-AI reports that the 67B base model beats LLaMA-2 70B across several benchmarks, with the largest margins in code, mathematics and reasoning. In open-ended evaluation, the 67B chat model is reported to outperform GPT-3.5. These are the developers' own results from their paper.

How to use it

Weights for the 7B and 67B base and chat models were released openly for research. The code repository carries the MIT licence, while model use falls under a separate Model License that permits commercial use. The chat model can be run through the Transformers library, and DeepSeek also offers a chat interface online.

Why it matters

The work is presented as a contribution to open-source language models, built on a fresh look at scaling laws that the authors say had produced conflicting conclusions. Its scaling analysis was used to guide the 7B and 67B training runs, and the released weights were offered to the research community.

Questions people ask about DeepSeek LLM 67B

What is DeepSeek LLM 67B?

It is a 67-billion-parameter language model from DeepSeek-AI, trained from scratch on 2 trillion tokens of English and Chinese text. It comes in a base form and a chat form, the latter fine-tuned on instruction data and aligned with Direct Preference Optimization.

How does it compare with other models?

According to DeepSeek-AI's own paper, the 67B base model outperforms LLaMA-2 70B on a range of benchmarks, especially code, mathematics and reasoning. The chat version is reported to beat GPT-3.5 in open-ended evaluation.

Can I use it commercially?

Yes. The developers state that DeepSeek LLM supports commercial use under its Model License, while the code repository is under the MIT licence. The full model licence terms are in the LICENSE-MODEL file of the repository.

How do I run the chat model?

Load it with the Transformers library as a causal language model in bfloat16, then format messages with the provided chat template. The developers note that system prompts are not supported in this version and should be left out.

Good reading and sources

    Where this page comes from

    There is no English Wikipedia article on DeepSeek LLM 67B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2401.02954, huggingface.co/deepseek-ai/deepseek-llm-67b-chat. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.

    It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.

    The weekly digest

    Every Monday: the model of the day, new pages on EveryLLM, and the conferences and anniversaries of the week ahead.