AI in robotics means using learned models instead of hand-written rules for the parts of a robot's job that the real world makes messy: seeing the scene, deciding how to move and predicting what happens next. Vision models turn camera images into objects and distances, and a learned policy turns that view, plus an instruction, into motor commands.
In October 2024, MIT CSAIL described Diffusion Forcing, a training method that combines the next-token prediction behind chatbots with the video diffusion behind video generators, and showed it helping a robot arm finish a task despite visual distractions. This guide explains the stack that research belongs to: how robots see, how they learn skills, what vision-language-action models are, how world models let a robot plan, where Diffusion Forcing fits, and what still limits AI-powered robots outside factories and warehouses.
What AI does inside a robot
A robot runs a loop: sense, decide, act, then sense again. In a classic industrial cell, engineers write every part of that loop by hand. Parts arrive in a fixture at a known position, the program moves the arm through recorded waypoints, and the same weld lands in the same spot all shift. Physical Intelligence calls robots built this way narrow specialists: they repeat the same motions in carefully arranged settings, and that works precisely because nothing in the cell changes.
Hand-written rules break when the world varies: a mug turned the other way, a shirt crumpled differently, a box half hidden behind another. AI replaces the parts of the loop that have to cope with that variation.
| Part of the loop | Classic approach | AI approach |
|---|---|---|
| Seeing | Fixed cameras, printed markers, tuned thresholds | Learned detection, segmentation and depth estimation |
| Deciding how to move | Programmed waypoints and motion planners | A learned policy, trained on demonstrations or practice |
| Understanding the task | One program per task | A vision-language model that reads a plain instruction |
| Looking ahead | Physics equations written by engineers | World models that learn from video what happens next |
The classic tools have not gone away. Motion planners, joint controllers and safety systems still run underneath, and the learned parts hand them targets or short sequences of actions. What changed is where the top-level decisions come from (which object, which grasp, which next move): a model trained on data rather than a rule someone wrote.
How robots see: computer vision, depth and segmentation
Robot vision starts with sensors. A manipulation robot typically carries colour cameras, often one on the wrist and one watching the whole workspace, and a depth sensor: a stereo camera pair, a structured-light projector or a time-of-flight camera that measures how far away each pixel is. Mobile robots often add LiDAR. Just as important is proprioception, the robot's sense of its own body: joint angles, gripper opening and, on some arms, force and torque. NVIDIA's GR00T models, for example, take three inputs: video from the onboard cameras, a language command and the robot's current joint positions.
From those signals, computer vision models answer the questions a grasp depends on:
- What is there? Object detection finds each object and names it.
- Which pixels belong to it? Segmentation outlines each object exactly. Meta's Segment Anything Model (SAM, 2023) is promptable: give it a point or a rough box and it returns the object's mask. It was trained on over 1 billion masks from 11 million images. SAM 2 (2024) extended it to video with a streaming memory, so an object can be tracked as it moves.
- How far away is it? Depth sensors measure distance directly. Monocular depth estimation infers it from a single colour image: Depth Anything V2 (NeurIPS 2024) offers models from 25 million to 1.3 billion parameters, trained on synthetic images and pseudo-labelled real ones.
- Which way is it facing? Pose estimation gives the object's position and orientation in 3D, which is what the gripper actually needs.
A classic pipeline chains these steps: segment the mug, combine the mask with depth into a 3D point cloud, compute a grasp, and send it to a motion planner.
Newer systems fold perception into a larger model. Google DeepMind's Gemini Robotics-ER, introduced in March 2025, points at object parts, matches points between views and detects objects in 3D, and roboticists can connect it to their existing low-level controllers. Vision-language-action models go further and never output a mask at all: the image goes in, motor commands come out, and the seeing happens inside the network.
Vision is also where robots fail first: glare on a shiny part, a transparent glass or an object half hidden behind another all change what the model sees. That is why the MIT research below cares so much about ignoring misleading observations.
How robots learn skills instead of following rules
A policy is the part of the robot that chooses the next action from what it currently observes. Writing a policy by hand for folding a shirt is hopeless, so modern robots learn policies from data, in three main ways.
Imitation learning from demonstrations
In imitation learning, also called behaviour cloning, a person performs the task through the robot, usually by teleoperation: with a VR headset and controllers, or with a small pair of leader arms that the robot's arms mirror. Each demonstration records what the robot's cameras saw and what its joints did, and a neural network learns to map one to the other. The ALOHA project (Zhao and colleagues, 2023) showed how far this goes on low-cost hardware: its Action Chunking with Transformers (ACT) method learned six fine tasks, such as slotting a battery and opening a translucent condiment cup, with 80 to 90% success from only 10 minutes of demonstrations.
Imitation has a known weakness: compounding errors. A learned policy makes a small mistake, the mistake puts the robot in a situation slightly unlike its training data, and there the next mistake is likely to be bigger, as Physical Intelligence explains. ACT's answer was to predict a chunk of future actions at once rather than one action at a time. NVIDIA's GR00T models, for one, also output action chunks: short predicted sequences of joint motions.
Diffusion policies
Demonstrations are messy in a particular way: people solve the same task differently. If half the demonstrations go around an obstacle on the left and half on the right, a model that averages them drives straight into it. Diffusion Policy (Chi and colleagues, RSS 2023) generates actions the way image generators make pictures: it starts from random noise and denoises it, step by step, into a sequence of actions conditioned on the camera images. It handles several valid ways of doing a task, and across 12 tasks from four benchmarks it beat the previous best methods by 46.9% on average.
Several of the models below also generate actions by denoising. π0 outputs actions with flow matching, a variant of diffusion, and GR00T N1 uses a diffusion transformer for its fast action module.
Reinforcement learning and sim-to-real
Reinforcement learning (RL) learns from outcomes instead of examples: the robot tries, a reward says how well it did, and the policy shifts toward what worked. On real hardware that is slow and hard on the machine, so RL usually starts in physics simulation. Rudin and colleagues (CoRL 2021) trained the ANYmal quadruped with thousands of simulated robots in parallel on a single workstation GPU: under four minutes for walking on flat ground and twenty for uneven terrain, then transferred the policies to the real robot.
The catch is the sim-to-real gap. Simulated friction, lighting and textures never match reality exactly, and a policy that exploits the simulator's quirks fails on hardware. Domain randomization is the classic fix: vary how the simulator renders the scene so widely that the real world looks like one more variation. In 2017, Tobin and colleagues trained an object detector only on simulated images with random, non-realistic textures and got real-world localization accurate to 1.5 cm, good enough to grasp objects in clutter. How synthetic data is generated to train robots is covered in our guide to AI-generated data for robotics.
RL also works on real robots once a policy is good enough to practise safely. Physical Intelligence's π*0.6 (November 2025) combined demonstrations, corrections from a teleoperator who takes over when the robot errs, and RL from the robot's own attempts. On some of the hardest tasks, that more than doubled throughput and cut failure rates by half or more; the resulting model made espresso drinks for an entire day and assembled boxes used for real packaging in a factory.
Vision-language-action models: from pixels and words to motion
A vision-language-action (VLA) model is a robot policy built on a vision-language model, the kind of model that answers questions about images. It takes camera images and an instruction in plain language, and it outputs robot actions. The point is to inherit what the vision-language model learned from the web (what a banana is, what "the bag about to fall off the table" means) and add what only robot data can teach: how to pick it up.
Google DeepMind's RT-2, announced on July 28, 2023, presented this as a new kind of model. It took vision-language models pretrained on web data and co-fine-tuned them on robot demonstrations collected with 13 robots over 17 months, writing the robot's actions as strings of numbers, like extra words in the model's vocabulary. Across more than 6,000 robot trials, success in situations not covered by the robot data rose from 32% for its predecessor RT-1 to 62%.
The models that followed, with dates as of September 2026:
| Model | Developer | Announced | What it added |
|---|---|---|---|
| RT-2 | Google DeepMind | July 2023 | Actions as tokens of a web-trained vision-language model |
| OpenVLA | Kim, Pertsch and team | June 2024 | Open weights: 7 billion parameters, 970,000 real robot demonstrations |
| π0 | Physical Intelligence | October 2024 | A flow-matching action expert for fast, smooth control; open weights in 2025 |
| GR00T N1 | NVIDIA | March 2025 | An open humanoid model with a slow reasoning module and a fast action module |
| Gemini Robotics | Google DeepMind | March 2025 | A VLA built on Gemini 2.0, with the Gemini Robotics-ER reasoning model |
| π0.5, π*0.6, π0.7 | Physical Intelligence | April 2025 to April 2026 | New homes, learning from experience, recombining skills for new tasks |
| Gemini Robotics 2 | Google DeepMind | July 2026 | Whole-body control of humanoids and multi-robot collaboration |
Three design choices matter more than the names.
- Two speeds. Reasoning about a scene is slow; motor control has to be fast. GR00T N1 pairs a vision-language module that interprets the scene with a diffusion transformer that generates fluid motor actions in real time. π0 adds an action expert to a 3-billion-parameter vision-language model so that it can output motor commands up to 50 times a second. In Gemini Robotics, the embodied-reasoning model plans the steps and coordinates with the VLA that carries them out.
- Open weights. OpenVLA (June 2024) is a 7-billion-parameter open model trained on 970,000 real-world robot demonstrations. It beat the closed 55-billion-parameter RT-2-X by 16.5 percentage points of absolute task success across 29 tasks, and it can be fine-tuned on consumer GPUs with low-rank adaptation. Physical Intelligence released π0's code and weights in February 2025, and GR00T N1 was published as an open, customizable model.
- Cross-embodiment. Robots differ in arms, hands and cameras, and data from one used to be of little use to another. π0 trained on data from 8 distinct robots plus the open-source Open X-Embodiment dataset. Gemini Robotics 2 controls Apptronik's Apollo 2 humanoid and a two-arm Franka platform with the same model checkpoint, and its on-device version adapts to a new two-arm robot in a few hours, typically with fewer than 200 examples.
Gemini Robotics sits on the same model family we explain in our guide to Google Gemini: it started as Gemini 2.0 with physical actions added as a new output.
How robots plan by predicting the future
A policy reacts: it maps what the robot sees now to what it does next. Planning adds a step: imagine where each option leads, then pick the best. For that the robot needs a world model, an AI system that simulates how an environment will change and how the agent's own actions will change it, in Google DeepMind's definition. Engineers have long written such models by hand as physics equations. The new ones learn from video, of which there is vastly more than robot data.
Three approaches show the range:
- Plan in pixels. UniPi (NeurIPS 2023) treated decision-making as text-conditioned video generation: given a goal in words, it generates frames showing the planned actions, then extracts the control actions from that generated video.
- Plan in a learned representation. Meta's V-JEPA 2 (June 2025) predicts compact representations of the future instead of pixels. Pretrained on over 1 million hours of internet video, then post-trained on less than 62 hours of unlabelled robot video from the DROID dataset, its action-conditioned version picked and placed objects with Franka arms in two labs, planning toward a goal image, without any data from those labs and without task-specific training or reward.
- Imagine a subgoal, then act. Physical Intelligence's π0.7 (April 2026) can take subgoal images generated at test time by a lightweight world model, each showing what the end of the current sub-step should look like, and the VLA works toward that picture.
General-purpose world models are growing too. Google DeepMind's Genie 3 (August 2025) generates interactive environments from a text prompt at 24 frames per second in 720p and keeps them consistent for a few minutes, and DeepMind presents it as a place to train agents. Its listed limits include a constrained range of actions an agent can take directly and no geographically accurate simulation of real places.
Pixel prediction has a structural trade-off. A model that predicts one frame at a time from its own previous output can run for any length but cannot aim at a goal far ahead, and its video can drift apart as errors build up. A model that generates a whole clip at once can be steered toward a goal but produces a fixed length. That trade-off is what Diffusion Forcing addresses.
Where MIT's Diffusion Forcing fits
Diffusion Forcing is a training method for sequence models by Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake and Vincent Sitzmann, published at NeurIPS 2024. It combines the two families of sequence models in one:
| Next-token prediction | Full-sequence diffusion | Diffusion Forcing | |
|---|---|---|---|
| How it generates | One token at a time | The whole sequence, by denoising | Tokens denoised in order, each at its own noise |
| Sequence length | Any | Fixed | Any, including past the training length |
| Steers toward a distant goal | Not without extra machinery | Yes | Yes |
| Familiar example | ChatGPT | Sora | Video prediction and robot planning |
The trick is in the noise. In a diffusion model, adding noise to data works like partial masking: a lightly noised frame is mostly known, a heavily noised one mostly unknown. Diffusion Forcing trains the network to clean up a sequence in which every token, a video frame or an action, has its own independent noise level, while predicting the next few tokens. At test time the model can keep the near future at low noise and the distant future at high noise, which builds in the intuition that what happens next is more certain than what happens later. It can also mark an observation it does not trust as noisy.
The experiments show what that buys:
- Memory for a robot arm. A Franka arm had to swap an apple and an orange using a third slot, starting from random positions. Halfway through, the current camera view does not say which fruit to move next; the robot has to remember how things started. Trained on 150 demonstrations recorded by teleoperation in VR, Diffusion Forcing succeeded 80% of the time, while Diffusion Policy, which has no memory, failed the task.
- Robustness to bad input. With visual distractions, such as a shopping bag dropped on the table, or with the camera fully blocked, Diffusion Forcing treated those observations as noisy and leaned on its own prediction: success fell only to 76%. A next-frame diffusion baseline that had to trust every frame reached 48%.
- Stable video and planning. Trained on Minecraft and DeepMind Lab footage, it kept generating stable video past the length it was trained on, where baselines diverged. As a planner, it reached the goal of a 2D maze with faster plans than six baselines, according to MIT.
The paper is candid about its limits: the implementation is a small recurrent network rather than a large transformer, and the authors did not test how it scales to internet-sized datasets. What makes it matter for robotics is that it joins video models that imagine and robot policies that act. The same model generated a video of the robot doing the task from a single frame while producing its actions, which points toward pretraining robot models on unlabelled video. In the MIT News article, Chen describes it as a possible backbone for a world model trained on internet video, one that could let a robot work out a task it was never shown, such as opening a door.
What still limits AI-powered robots
Data
There is no internet of robot data. Language models learn from a large share of the web; robot policies need recordings of a robot doing things, collected one episode at a time. RT-1's dataset took 13 robots 17 months in one office kitchen. The Open X-Embodiment collaboration pooled data from 22 different robots across 21 institutions to cover 527 skills. Video pretraining, simulation and world models all exist to get more experience without more teleoperation hours.
Reliability
Getting a learned policy to work some of the time is much easier than getting it to work every time, at a useful speed. Physical Intelligence puts it plainly: with demonstrations alone it is not hard to reach half success and very hard to reach consistent success. A 90% success rate sounds high until a robot attempts the task hundreds of times a day.
New places, new bodies and fine dexterity
Most commercial robots still work in tightly controlled settings such as factories and warehouses (the market figures are in the future of robotics and automation), and many impressive demos use data collected in the test scene or one very like it, as the π0.5 announcement notes. π0.5 (April 2025) was built to clean kitchens and bedrooms in homes it had never seen, and its developers say it does not always succeed on the first try. Google DeepMind reports that Gemini Robotics 2 reaches medium to high success on whole-body and gripper tasks, that multi-finger dexterous manipulation remains challenging, and that its robots still need to move faster.
Safety
A robot's mistakes are physical. The usual design is layered: collision avoidance, speed limits and contact-force limits run in dedicated controllers, and the AI model sits above them. Google DeepMind describes this layered approach for Gemini Robotics, from low-level motor control up to semantic judgments about whether an action is safe in context, and released the ASIMOV dataset to measure the latter. With Gemini Robotics 2 it added ASIMOV-Agentic, which tests whether the reasoning model refuses unsafe tool calls from the VLA and asks a person for help when uncertain, and it reports that Gemini Robotics ER 2 can bring the robot to a safe stop when someone comes too close.
For industrial robots, ISO 10218-1:2025 sets the safety requirements for the robot itself and ISO 10218-2:2025 covers its integration into a complete system; the 2025 edition replaced the 2011 one. Its scope excludes service robots in places the public can access, household consumer products and medical robots, so a general-purpose home robot falls outside it.
Warning
Treat a learned policy as a component that can be wrong, never as the safety function. Speed, force and keep-out limits belong in safety-rated controllers the model cannot override, and a person must always be able to stop the robot.
How to read an AI robotics demo
Many of the models above are research releases or previews. As of September 2026, the Gemini Robotics 2 VLA and its on-device version are available only to early-access partners, with the reasoning model, Gemini Robotics ER 2, on Google AI Studio, and NVIDIA's GR00T 1.7 early access explicitly excludes production deployment. A video shows one run, not the failure rate. These questions separate a capability from a clip:
- Was it autonomous? A teleoperated robot, or one where a person takes over, shows the hardware, not the AI. Physical Intelligence labels its laundry video uncut and fully autonomous; ask for the same statement.
- Is it real time? Sped-up footage hides slow policies.
- How often does it work? Ask for the success rate over many trials, and what counted as success.
- Was the scene new? Objects, rooms and lighting from the training data are the easy case. Ask what the robot had never seen.
- How much task-specific data did it need? Physical Intelligence found 1 to 20 hours of data enough to fine-tune π0 to a variety of tasks; ask any vendor for its number.
- What happens when it fails? Look for recovery: does the robot notice and retry, stop safely, or carry on?
- What sits outside the model? Ask which safety functions are independent of the AI, and who can stop the robot.
The last question applies to any AI system that acts. In business software the same principle holds: the model proposes, and a rule you signed off or a named person approves anything that changes data. That is how our AI and automation service builds workflows and agents, with an evaluation set of real cases run before every change. For how AI features fit inside ordinary applications, see our guide to intelligent apps, and for more on the topic, the AI category.
