Lab city, United States
Mountain View and Palo Alto
Google's headquarters and Stanford University: where the transformer, BERT and Gemini were born.
The stretch of Silicon Valley between Google's headquarters in Mountain View and Stanford University beside Palo Alto produced much of the science behind today's models. Google researchers published the transformer in 2017 and BERT in 2018, and Google DeepMind's Gemini is built partly here. Stanford trains many of the field's researchers and runs some of its best-known measurement projects.
Google Brain, started in 2011, and Google Research produced Word2Vec (2013), the transformer paper Attention Is All You Need (June 2017), BERT (2018), T5 (2019), LaMDA (2021) and PaLM (2022), a 540-billion-parameter model trained on 6,144 TPU v4 chips, Google's own AI accelerators. In April 2023 Google Brain merged with London's DeepMind to form Google DeepMind, which builds the Gemini models and the open Gemma family.
Stanford
Stanford's Institute for Human-Centered Artificial Intelligence (HAI), founded in 2019, publishes the annual AI Index, a widely cited record of models, costs and policy. Its Center for Research on Foundation Models coined the term foundation model in 2021 and runs the HELM evaluations. In March 2023 Stanford's Alpaca showed that a small Llama model tuned on 52,000 examples could mimic a commercial chatbot, starting a wave of cheap open fine-tunes.
Models of Mountain View and Palo Alto
Still to be written: Word2Vec.