PaLM-E
An embodied multimodal language model that grounds words in robot sensor input
PaLM-E is an embodied multimodal language model that folds continuous sensor readings, images and text into a single large language model so words connect to real-world percepts. Its largest version, PaLM-E-562B with 562 billion parameters, was presented in March 2023 and handles robot planning, visual question answering and captioning in one model.
- Largest version, PaLM-E-562B, has 562 billion parameters
- Paper posted to arXiv on 6 March 2023
- Trains sensor, visual and text encodings end to end with a pre-trained language model
- Handles robot manipulation planning, visual question answering and captioning
- Authors report state-of-the-art results on OK-VQA
What it is
PaLM-E is an embodied multimodal language model. It takes in what the authors call multimodal sentences: visual input, continuous state estimates from sensors, and text, all interleaved in one sequence. These encodings are trained end to end together with a pre-trained large language model. The point is grounding, that is, tying words to things a machine actually perceives in the world. The largest variant, PaLM-E-562B, carries 562 billion parameters. It is used for sequential robotic manipulation planning, visual question answering and image captioning, and it also works as a general visual-language model.
How it came about
The model was described in a paper posted to arXiv on 6 March 2023 by Danny Driess and 21 co-authors, working across machine learning, artificial intelligence and robotics. The paper introduces embodied language models as a way to fold real-world continuous sensor modalities straight into a language model. It reports that a single model can handle several embodied reasoning tasks across different observation modalities and different robot bodies.
Inside the model
The system feeds multimodal sentences into a pre-trained large language model. These sentences mix visual encodings, continuous state estimation and text. The encoders are trained end to end alongside the language model rather than bolted on afterwards. Training spans robotics tasks as well as internet-scale language and vision-language data. The authors report positive transfer: joint training across these diverse domains helps the model, rather than diluting it. General language ability is retained and improves as the model gets larger.
What it can do
PaLM-E addresses a range of embodied reasoning tasks, drawing on several observation modalities and working across multiple embodiments. Beyond robotics, the 562B version acts as a visual-language generalist and the authors report state-of-the-art performance on OK-VQA, a visual question answering benchmark. They also report that general language capability holds up and improves with scale.
Questions people ask about PaLM-E
What does 'embodied' mean here?
It means the model takes in real-world continuous sensor input, not just text. Visual readings and state estimates are fed in alongside words, so the language model is linked to things a robot actually perceives rather than to text alone.
How big is PaLM-E?
The largest version, PaLM-E-562B, has 562 billion parameters. The paper describes it as a single large embodied multimodal model that also works as a visual-language generalist.
What tasks is it used for?
The paper trains it on sequential robotic manipulation planning, visual question answering and captioning. It works across several observation modalities and multiple robot embodiments.
Does training on robot data hurt its language skills?
No. The authors report that general language capability is retained and improves with scale, and that joint training across language, vision and visual-language data brings positive transfer.
Good reading and sources
There is no English Wikipedia article on PaLM-E yet, so the facts on this page come from the sources listed above: arxiv.org/abs/2303.03378. Some are the subject's own pages, so its claims about itself are reported as its claims. EveryLLM wrote the text in its own words, with AI assistance, and compares it with the sources to make sure no passage is copied.
It may contain mistakes, and figures change over time. Nothing here is investment or buying advice. If something is wrong, tell us. Last refreshed 2026-11-14.