GLM-130B
Open bilingual pre-trained language model with 130 billion parameters
GLM-130B is a bilingual pre-trained language model with 130 billion parameters, developed by researchers including Aohan Zeng and Jie Tang. Released in October 2022, it aims to provide an open-source alternative to large proprietary models like GPT-3, supporting English and Chinese with efficient inference capabilities.
- Contains 130 billion parameters for bilingual processing
- Supports INT4 quantization without post-training loss
- Outperforms GPT-3 davinci on various English benchmarks
- Open-sourced weights, code, and training logs
- Runs effectively on affordable consumer GPUs
What it is
GLM-130B is a pre-trained language model featuring 130 billion parameters. It supports both English and Chinese languages. The project was initiated to create an open-source model comparable in quality to GPT-3 davinci. By making the weights publicly accessible, the developers aimed to demonstrate how models of this scale can be successfully trained. It stands out for its ability to run on affordable hardware through INT4 quantization.
How it came about
The model was introduced in October 2022 via a paper submitted to arXiv. The development team, led by authors such as Aohan Zeng and Jie Tang, faced significant engineering challenges during training. These included unexpected loss spikes and divergence issues. The team documented their training strategies and design choices to help the community understand the process of pre-training 100B-scale models. The paper was revised in October 2023.
Inside the model
GLM-130B utilizes a architecture designed for stability and efficiency at scale. The training process involved specific strategies to handle technical difficulties like divergence. A key design feature is its scaling property that allows for INT4 quantization without post-training adjustments. This approach minimizes performance loss while significantly reducing memory requirements. The model weights, code, and training logs are open-sourced to support reproducibility and further research.
What it can do
The model demonstrates strong performance on popular English benchmarks, outperforming GPT-3 175B davinci in several areas. It also shows consistent advantages over ERNIE TITAN 3.0 260B on Chinese-related benchmarks. Unlike some contemporaries such as OPT-175B and BLOOM-176B, GLM-130B maintains a performance edge in these tests. Its unique quantization capability allows effective inference on consumer-grade GPUs like the RTX 3090 or RTX 2080 Ti.
How to use it
The model weights are publicly accessible for download. The developers have open-sourced the associated code, training logs, and toolkits. This open approach allows researchers and engineers to study the training process and deploy the model locally. The INT4 quantization feature makes it feasible to run the model on hardware with limited memory, such as systems with four RTX 3090 cards.
Why it matters
GLM-130B contributes to the field by providing a detailed account of training large-scale models. It offers a viable open-source alternative to proprietary systems for bilingual tasks. The release of training logs and lessons learned helps other teams navigate similar engineering challenges. Its success in running on affordable hardware lowers the barrier for experimenting with 100B-scale models.
Questions people ask about GLM-130B
What languages does GLM-130B support?
GLM-130B is a bilingual model designed to process both English and Chinese. It performs strongly on benchmarks for both languages, often outperforming other large models in Chinese-specific tasks.
Can I run GLM-130B on my own computer?
Yes, thanks to INT4 quantization, it can run on affordable hardware. The developers state it works effectively on setups with four RTX 3090 GPUs or eight RTX 2080 Ti GPUs.
Is the model open source?
Yes, the model weights are publicly accessible. The team also open-sourced the code, training logs, and toolkits to help others understand how to train models of this scale.
Good reading and sources
There is no English Wikipedia article on GLM-130B yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2210.02414. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.