Search: Pre-training
Concepts
WorldGPT-1
OpenAI's first GPT model, a 117M-parameter decoder-only transformer from 2018.Research · Open weights · 2018
🇺🇸 United StatesT5
Encoder-decoder language model family from Google AI using a text-to-text framework.General · Fully open · 2019
🇯🇵 JapanSwallow
Japanese-enhanced Llama models by Tokyo TechGeneral · Open weights · 2024
WorldBioBERT
Biomedical language model based on BERT architectureDomain · Open weights · 2019
🇺🇸 United StatesDeBERTa
A BERT-style encoder with disentangled attention and an enhanced mask decoder.Encoder · Open weights · 2020
WorldBLIP-2
Efficient vision-language model using frozen encodersMultimodal · Research only · 2023
🇨🇳 ChinaDeepSeekMath
Open language model for mathematical reasoningReasoning · Open weights · 2024
WorldMM1
A family of multimodal language models up to 30B parameters, dense and MoE.Multimodal · Research only · 2024
🇯🇵 JapanLlama-3-ELYZA-JP-8B
Japanese-tuned Llama 3 model by ELYZAGeneral · Conditional licence
🇨🇳 ChinaERNIE 3.0
Knowledge-enhanced pre-training model for language tasksGeneral · Research only · 2021
🇨🇳 ChinaERNIE 3.0 Titan
Large-scale Chinese knowledge-enhanced pre-trained language modelGeneral · Research only · 2021
🇨🇳 ChinaInternVL3
Open-source multimodal large language model seriesMultimodal · Open weights · 2025
WorldELECTRA
A pre-training method that trains text encoders to spot replaced tokens.Research · Research only · 2020
🇺🇸 United StatesGrok-1
xAI's 314 billion parameter open-weights mixture-of-experts base modelGeneral · Open weights · 2024
🇸🇬 SingaporeSeaLLMs
Open language models for Southeast Asian languagesGeneral · Open weights · 2023
🇺🇸 United StatesBART
A denoising encoder-decoder transformer for generation and comprehension tasksGeneral · Conditional licence · 2019