# AI in robotics: how AI-powered robots see, learn and plan

> AI in robotics uses learned models to see a scene, turn an instruction into motor commands and predict what happens next. How each part works, and its limits.

- URL: https://computese.com/revolutionizing-robotics-with-ai-powered/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-10-15
- Updated: 2026-09-25
- Topics: AI & automation

## In short
- AI in robotics replaces hand-written rules with learned models: vision models find objects and how far away they are, a learned policy turns camera images and an instruction into motor commands, and some systems predict the future before they move.
- Robots learn skills from teleoperated demonstrations (imitation learning), from rewarded practice in simulation or on real hardware (reinforcement learning), and from the web images, text and video their base models were pretrained on.
- Vision-language-action models map pixels and words straight to motion: RT-2 (2023), OpenVLA and π0 (2024), GR00T N1 and Gemini Robotics (2025), and Gemini Robotics 2 (July 2026).
- MIT's Diffusion Forcing (NeurIPS 2024) gives every future frame its own noise level, so one model can predict video, plan toward a goal and discount bad camera input. Its robot arm kept 76% success with the camera distracted or blocked.
- The limits are data, reliability, new places and safety. Robot data is scarce, many demos run close to their training scenes, and a learned model is not a safety function: speed, force and keep-out limits stay outside it.

AI in robotics means using learned models instead of hand-written rules for the parts of a robot's job that the real world makes messy: seeing the scene, deciding how to move and predicting what happens next. Vision models turn camera images into objects and distances, and a learned policy turns that view, plus an instruction, into motor commands.

In October 2024, MIT CSAIL described [Diffusion Forcing](https://news.mit.edu/2024/combining-next-token-prediction-video-diffusion-computer-vision-robotics-1016), a training method that combines the next-token prediction behind chatbots with the video diffusion behind video generators, and showed it helping a robot arm finish a task despite visual distractions. This guide explains the stack that research belongs to: how robots see, how they learn skills, what vision-language-action models are, how world models let a robot plan, where Diffusion Forcing fits, and what still limits AI-powered robots outside factories and warehouses.

## What AI does inside a robot

A robot runs a loop: sense, decide, act, then sense again. In a classic industrial cell, engineers write every part of that loop by hand. Parts arrive in a fixture at a known position, the program moves the arm through recorded waypoints, and the same weld lands in the same spot all shift. Physical Intelligence calls robots built this way [narrow specialists](https://www.pi.website/blog/pi0): they repeat the same motions in carefully arranged settings, and that works precisely because nothing in the cell changes.

Hand-written rules break when the world varies: a mug turned the other way, a shirt crumpled differently, a box half hidden behind another. AI replaces the parts of the loop that have to cope with that variation.

| Part of the loop       | Classic approach                                 | AI approach                                             |
| ---------------------- | ------------------------------------------------ | ------------------------------------------------------- |
| Seeing                 | Fixed cameras, printed markers, tuned thresholds | Learned detection, segmentation and depth estimation    |
| Deciding how to move   | Programmed waypoints and motion planners         | A learned policy, trained on demonstrations or practice |
| Understanding the task | One program per task                             | A vision-language model that reads a plain instruction  |
| Looking ahead          | Physics equations written by engineers           | World models that learn from video what happens next    |

The classic tools have not gone away. Motion planners, joint controllers and safety systems still run underneath, and the learned parts hand them targets or short sequences of actions. What changed is where the top-level decisions come from (which object, which grasp, which next move): a model trained on data rather than a rule someone wrote.

## How robots see: computer vision, depth and segmentation

Robot vision starts with sensors. A manipulation robot typically carries colour cameras, often one on the wrist and one watching the whole workspace, and a depth sensor: a stereo camera pair, a structured-light projector or a time-of-flight camera that measures how far away each pixel is. Mobile robots often add LiDAR. Just as important is proprioception, the robot's sense of its own body: joint angles, gripper opening and, on some arms, force and torque. NVIDIA's GR00T models, for example, take [three inputs](https://developer.nvidia.com/isaac/gr00t): video from the onboard cameras, a language command and the robot's current joint positions.

From those signals, computer vision models answer the questions a grasp depends on:

- **What is there?** Object detection finds each object and names it.
- **Which pixels belong to it?** Segmentation outlines each object exactly. Meta's [Segment Anything Model](https://arxiv.org/abs/2304.02643) (SAM, 2023) is promptable: give it a point or a rough box and it returns the object's mask. It was trained on over 1 billion masks from 11 million images. [SAM 2](https://arxiv.org/abs/2408.00714) (2024) extended it to video with a streaming memory, so an object can be tracked as it moves.
- **How far away is it?** Depth sensors measure distance directly. Monocular depth estimation infers it from a single colour image: [Depth Anything V2](https://arxiv.org/abs/2406.09414) (NeurIPS 2024) offers models from 25 million to 1.3 billion parameters, trained on synthetic images and pseudo-labelled real ones.
- **Which way is it facing?** Pose estimation gives the object's position and orientation in 3D, which is what the gripper actually needs.

A classic pipeline chains these steps: segment the mug, combine the mask with depth into a 3D point cloud, compute a grasp, and send it to a motion planner.

![A camera above a table measures the distance to a mug, a box and a bowl. The mug is filled in orange as the segmented object, and a robot arm's open gripper reaches toward it.](https://computese.com/images/blog/revolutionizing-robotics-with-ai-powered/perception.2cb9bf6e37-1536.webp)

*A grasp needs two answers from vision: which pixels belong to the object, and how far away they are.*

Newer systems fold perception into a larger model. Google DeepMind's [Gemini Robotics-ER](https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/), introduced in March 2025, points at object parts, matches points between views and detects objects in 3D, and roboticists can connect it to their existing low-level controllers. Vision-language-action models go further and never output a mask at all: the image goes in, motor commands come out, and the seeing happens inside the network.

Vision is also where robots fail first: glare on a shiny part, a transparent glass or an object half hidden behind another all change what the model sees. That is why the MIT research below cares so much about ignoring misleading observations.

## How robots learn skills instead of following rules

A policy is the part of the robot that chooses the next action from what it currently observes. Writing a policy by hand for folding a shirt is hopeless, so modern robots learn policies from data, in three main ways.

### Imitation learning from demonstrations

In imitation learning, also called behaviour cloning, a person performs the task through the robot, usually by teleoperation: with a VR headset and controllers, or with a small pair of leader arms that the robot's arms mirror. Each demonstration records what the robot's cameras saw and what its joints did, and a neural network learns to map one to the other. The ALOHA project (Zhao and colleagues, 2023) showed how far this goes on low-cost hardware: its [Action Chunking with Transformers](https://arxiv.org/abs/2304.13705) (ACT) method learned six fine tasks, such as slotting a battery and opening a translucent condiment cup, with 80 to 90% success from only 10 minutes of demonstrations.

![Small leader arms on a desk drive a pair of larger robot arms that copy their pose over a table, and each demonstration is saved as a stack of camera frames and joint-angle traces.](https://computese.com/images/blog/revolutionizing-robotics-with-ai-powered/teleoperation.30a3ae5a70-1536.webp)

*Teleoperation turns a person's skill into training data: what the robot's own cameras saw, paired with what its own joints did.*

Imitation has a known weakness: compounding errors. A learned policy makes a small mistake, the mistake puts the robot in a situation slightly unlike its training data, and there the next mistake is likely to be bigger, as Physical Intelligence [explains](https://www.pi.website/blog/pistar06). ACT's answer was to predict a chunk of future actions at once rather than one action at a time. NVIDIA's GR00T models, for one, also output action chunks: short predicted sequences of joint motions.

### Diffusion policies

Demonstrations are messy in a particular way: people solve the same task differently. If half the demonstrations go around an obstacle on the left and half on the right, a model that averages them drives straight into it. [Diffusion Policy](https://arxiv.org/abs/2303.04137) (Chi and colleagues, RSS 2023) generates actions the way image generators make pictures: it starts from random noise and denoises it, step by step, into a sequence of actions conditioned on the camera images. It handles several valid ways of doing a task, and across 12 tasks from four benchmarks it beat the previous best methods by 46.9% on average.

Several of the models below also generate actions by denoising. π0 outputs actions with flow matching, a variant of diffusion, and [GR00T N1](https://arxiv.org/abs/2503.14734) uses a diffusion transformer for its fast action module.

### Reinforcement learning and sim-to-real

Reinforcement learning (RL) learns from outcomes instead of examples: the robot tries, a reward says how well it did, and the policy shifts toward what worked. On real hardware that is slow and hard on the machine, so RL usually starts in physics simulation. Rudin and colleagues (CoRL 2021) trained the ANYmal quadruped with [thousands of simulated robots in parallel](https://arxiv.org/abs/2109.11978) on a single workstation GPU: under four minutes for walking on flat ground and twenty for uneven terrain, then transferred the policies to the real robot.

The catch is the sim-to-real gap. Simulated friction, lighting and textures never match reality exactly, and a policy that exploits the simulator's quirks fails on hardware. Domain randomization is the classic fix: vary how the simulator renders the scene so widely that the real world looks like one more variation. In 2017, Tobin and colleagues trained an object detector [only on simulated images](https://arxiv.org/abs/1703.06907) with random, non-realistic textures and got real-world localization accurate to 1.5 cm, good enough to grasp objects in clutter. How synthetic data is generated to train robots is covered in our guide to [AI-generated data for robotics](https://computese.com/ai-generated-data-for/).

RL also works on real robots once a policy is good enough to practise safely. Physical Intelligence's π\*0.6 (November 2025) combined demonstrations, corrections from a teleoperator who takes over when the robot errs, and RL from the robot's own attempts. On some of the hardest tasks, that more than doubled throughput and cut failure rates by half or more; the resulting model made espresso drinks for an entire day and assembled boxes used for real packaging in a factory.

## Vision-language-action models: from pixels and words to motion

A vision-language-action (VLA) model is a robot policy built on a vision-language model, the kind of model that answers questions about images. It takes camera images and an instruction in plain language, and it outputs robot actions. The point is to inherit what the vision-language model learned from the web (what a banana is, what "the bag about to fall off the table" means) and add what only robot data can teach: how to pick it up.

Google DeepMind's [RT-2](https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/), announced on July 28, 2023, presented this as a new kind of model. It took vision-language models pretrained on web data and co-fine-tuned them on robot demonstrations collected with 13 robots over 17 months, writing the robot's actions as strings of numbers, like extra words in the model's vocabulary. Across more than 6,000 robot trials, success in situations not covered by the robot data rose from 32% for its predecessor RT-1 to 62%.

The models that followed, with dates as of September 2026:

| Model              | Developer             | Announced                | What it added                                                                |
| ------------------ | --------------------- | ------------------------ | ---------------------------------------------------------------------------- |
| RT-2               | Google DeepMind       | July 2023                | Actions as tokens of a web-trained vision-language model                     |
| OpenVLA            | Kim, Pertsch and team | June 2024                | Open weights: 7 billion parameters, 970,000 real robot demonstrations        |
| π0                 | Physical Intelligence | October 2024             | A flow-matching action expert for fast, smooth control; open weights in 2025 |
| GR00T N1           | NVIDIA                | March 2025               | An open humanoid model with a slow reasoning module and a fast action module |
| Gemini Robotics    | Google DeepMind       | March 2025               | A VLA built on Gemini 2.0, with the Gemini Robotics-ER reasoning model       |
| π0.5, π\*0.6, π0.7 | Physical Intelligence | April 2025 to April 2026 | New homes, learning from experience, recombining skills for new tasks        |
| Gemini Robotics 2  | Google DeepMind       | July 2026                | Whole-body control of humanoids and multi-robot collaboration                |

Three design choices matter more than the names.

- **Two speeds.** Reasoning about a scene is slow; motor control has to be fast. GR00T N1 pairs a vision-language module that interprets the scene with a diffusion transformer that generates fluid motor actions in real time. π0 adds an action expert to a 3-billion-parameter vision-language model so that it can output motor commands [up to 50 times a second](https://www.pi.website/blog/pi0). In Gemini Robotics, the embodied-reasoning model plans the steps and coordinates with the VLA that carries them out.
- **Open weights.** [OpenVLA](https://arxiv.org/abs/2406.09246) (June 2024) is a 7-billion-parameter open model trained on 970,000 real-world robot demonstrations. It beat the closed 55-billion-parameter RT-2-X by 16.5 percentage points of absolute task success across 29 tasks, and it can be fine-tuned on consumer GPUs with low-rank adaptation. Physical Intelligence [released π0's code and weights](https://www.pi.website/blog/openpi) in February 2025, and GR00T N1 was published as an open, customizable model.
- **Cross-embodiment.** Robots differ in arms, hands and cameras, and data from one used to be of little use to another. π0 trained on data from 8 distinct robots plus the open-source Open X-Embodiment dataset. [Gemini Robotics 2](https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/) controls Apptronik's Apollo 2 humanoid and a two-arm Franka platform with the same model checkpoint, and its on-device version adapts to a new two-arm robot in a few hours, typically with fewer than 200 examples.

Gemini Robotics sits on the same model family we explain in our [guide to Google Gemini](https://computese.com/google-unveils-gemini-the-most-advanced-and-versatile-ai-model-yet/): it started as Gemini 2.0 with physical actions added as a new output.

## How robots plan by predicting the future

A policy reacts: it maps what the robot sees now to what it does next. Planning adds a step: imagine where each option leads, then pick the best. For that the robot needs a world model, an AI system that simulates how an environment will change and how the agent's own actions will change it, in Google DeepMind's [definition](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/). Engineers have long written such models by hand as physics equations. The new ones learn from video, of which there is vastly more than robot data.

Three approaches show the range:

1. **Plan in pixels.** [UniPi](https://arxiv.org/abs/2302.00111) (NeurIPS 2023) treated decision-making as text-conditioned video generation: given a goal in words, it generates frames showing the planned actions, then extracts the control actions from that generated video.
2. **Plan in a learned representation.** Meta's [V-JEPA 2](https://arxiv.org/abs/2506.09985) (June 2025) predicts compact representations of the future instead of pixels. Pretrained on over 1 million hours of internet video, then post-trained on less than 62 hours of unlabelled robot video from the DROID dataset, its action-conditioned version picked and placed objects with Franka arms in two labs, planning toward a goal image, without any data from those labs and without task-specific training or reward.
3. **Imagine a subgoal, then act.** Physical Intelligence's [π0.7](https://www.pi.website/blog/pi07) (April 2026) can take subgoal images generated at test time by a lightweight world model, each showing what the end of the current sub-step should look like, and the VLA works toward that picture.

General-purpose world models are growing too. Google DeepMind's Genie 3 (August 2025) generates interactive environments from a text prompt at 24 frames per second in 720p and keeps them consistent for a few minutes, and DeepMind presents it as a place to train agents. Its listed limits include a constrained range of actions an agent can take directly and no geographically accurate simulation of real places.

Pixel prediction has a structural trade-off. A model that predicts one frame at a time from its own previous output can run for any length but cannot aim at a goal far ahead, and its video can drift apart as errors build up. A model that generates a whole clip at once can be steered toward a goal but produces a fixed length. That trade-off is what Diffusion Forcing addresses.

## Where MIT's Diffusion Forcing fits

[Diffusion Forcing](https://proceedings.neurips.cc/paper_files/paper/2024/hash/2aee1c4159e48407d68fe16ae8e6e49e-Abstract-Conference.html) is a training method for sequence models by Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake and Vincent Sitzmann, published at NeurIPS 2024. It combines the two families of sequence models in one:

|                              | Next-token prediction       | Full-sequence diffusion          | Diffusion Forcing                               |
| ---------------------------- | --------------------------- | -------------------------------- | ----------------------------------------------- |
| How it generates             | One token at a time         | The whole sequence, by denoising | Tokens denoised in order, each at its own noise |
| Sequence length              | Any                         | Fixed                            | Any, including past the training length         |
| Steers toward a distant goal | Not without extra machinery | Yes                              | Yes                                             |
| Familiar example             | ChatGPT                     | Sora                             | Video prediction and robot planning             |

The trick is in the noise. In a diffusion model, adding noise to data works like partial masking: a lightly noised frame is mostly known, a heavily noised one mostly unknown. Diffusion Forcing trains the network to clean up a sequence in which every token, a video frame or an action, has its own independent noise level, while predicting the next few tokens. At test time the model can keep the near future at low noise and the distant future at high noise, which builds in the intuition that what happens next is more certain than what happens later. It can also mark an observation it does not trust as noisy.

![A strip of six frames of a robot gripper over two fruits: two sharp frames of the past, then the next frame, outlined in orange, which a gear cleans up first, then three with increasing noise.](https://computese.com/images/blog/revolutionizing-robotics-with-ai-powered/diffusion-forcing.e660677bcc-1536.webp)

*Each future frame carries its own noise level, so the model commits to the next step and keeps the distant ones open.*

The experiments show what that buys:

- **Memory for a robot arm.** A Franka arm had to swap an apple and an orange using a third slot, starting from random positions. Halfway through, the current camera view does not say which fruit to move next; the robot has to remember how things started. Trained on 150 demonstrations recorded by teleoperation in VR, Diffusion Forcing succeeded 80% of the time, while Diffusion Policy, which has no memory, failed the task.
- **Robustness to bad input.** With visual distractions, such as a shopping bag dropped on the table, or with the camera fully blocked, Diffusion Forcing treated those observations as noisy and leaned on its own prediction: success fell only to 76%. A next-frame diffusion baseline that had to trust every frame reached 48%.
- **Stable video and planning.** Trained on Minecraft and DeepMind Lab footage, it kept generating stable video past the length it was trained on, where baselines diverged. As a planner, it reached the goal of a 2D maze with faster plans than six baselines, [according to MIT](https://news.mit.edu/2024/combining-next-token-prediction-video-diffusion-computer-vision-robotics-1016).

The paper is candid about its limits: the implementation is a small recurrent network rather than a large transformer, and the authors did not test how it scales to internet-sized datasets. What makes it matter for robotics is that it joins video models that imagine and robot policies that act. The same model generated a video of the robot doing the task from a single frame while producing its actions, which points toward pretraining robot models on unlabelled video. In the MIT News article, Chen describes it as a possible backbone for a world model trained on internet video, one that could let a robot work out a task it was never shown, such as opening a door.

## What still limits AI-powered robots

### Data

There is no internet of robot data. Language models learn from a large share of the web; robot policies need recordings of a robot doing things, collected one episode at a time. RT-1's dataset took [13 robots 17 months](https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/) in one office kitchen. The [Open X-Embodiment](https://arxiv.org/abs/2310.08864) collaboration pooled data from 22 different robots across 21 institutions to cover 527 skills. Video pretraining, simulation and world models all exist to get more experience without more teleoperation hours.

### Reliability

Getting a learned policy to work some of the time is much easier than getting it to work every time, at a useful speed. Physical Intelligence [puts it plainly](https://www.pi.website/blog/pistar06): with demonstrations alone it is not hard to reach half success and very hard to reach consistent success. A 90% success rate sounds high until a robot attempts the task hundreds of times a day.

### New places, new bodies and fine dexterity

Most commercial robots still work in tightly controlled settings such as factories and warehouses (the market figures are in [the future of robotics and automation](https://computese.com/the-future-of-robotics-and-automation/)), and many impressive demos use data collected in the test scene or one very like it, as the [π0.5 announcement](https://www.pi.website/blog/pi05) notes. π0.5 (April 2025) was built to clean kitchens and bedrooms in homes it had never seen, and its developers say it does not always succeed on the first try. Google DeepMind [reports](https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/) that Gemini Robotics 2 reaches medium to high success on whole-body and gripper tasks, that multi-finger dexterous manipulation remains challenging, and that its robots still need to move faster.

### Safety

A robot's mistakes are physical. The usual design is layered: collision avoidance, speed limits and contact-force limits run in dedicated controllers, and the AI model sits above them. Google DeepMind describes this [layered approach](https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/) for Gemini Robotics, from low-level motor control up to semantic judgments about whether an action is safe in context, and released the ASIMOV dataset to measure the latter. With Gemini Robotics 2 it added [ASIMOV-Agentic](https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/), which tests whether the reasoning model refuses unsafe tool calls from the VLA and asks a person for help when uncertain, and it reports that Gemini Robotics ER 2 can bring the robot to a safe stop when someone comes too close.

For industrial robots, [ISO 10218-1:2025](https://www.iso.org/standard/73933.html) sets the safety requirements for the robot itself and ISO 10218-2:2025 covers its integration into a complete system; the 2025 edition replaced the 2011 one. Its scope excludes service robots in places the public can access, household consumer products and medical robots, so a general-purpose home robot falls outside it.

> [!WARNING]
> Treat a learned policy as a component that can be wrong, never as the safety function. Speed, force and keep-out limits belong in safety-rated controllers the model cannot override, and a person must always be able to stop the robot.

## How to read an AI robotics demo

Many of the models above are research releases or previews. As of September 2026, the Gemini Robotics 2 VLA and its on-device version are [available only to early-access partners](https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/), with the reasoning model, Gemini Robotics ER 2, on Google AI Studio, and NVIDIA's [GR00T 1.7 early access](https://developer.nvidia.com/isaac/gr00t) explicitly excludes production deployment. A video shows one run, not the failure rate. These questions separate a capability from a clip:

1. **Was it autonomous?** A teleoperated robot, or one where a person takes over, shows the hardware, not the AI. Physical Intelligence [labels its laundry video](https://www.pi.website/blog/pi0) uncut and fully autonomous; ask for the same statement.
2. **Is it real time?** Sped-up footage hides slow policies.
3. **How often does it work?** Ask for the success rate over many trials, and what counted as success.
4. **Was the scene new?** Objects, rooms and lighting from the training data are the easy case. Ask what the robot had never seen.
5. **How much task-specific data did it need?** Physical Intelligence [found 1 to 20 hours of data](https://www.pi.website/blog/openpi) enough to fine-tune π0 to a variety of tasks; ask any vendor for its number.
6. **What happens when it fails?** Look for recovery: does the robot notice and retry, stop safely, or carry on?
7. **What sits outside the model?** Ask which safety functions are independent of the AI, and who can stop the robot.

The last question applies to any AI system that acts. In business software the same principle holds: the model proposes, and a rule you signed off or a named person approves anything that changes data. That is how our [AI and automation service](https://computese.com/services/ai-automation/) builds workflows and agents, with an evaluation set of real cases run before every change. For how AI features fit inside ordinary applications, see our guide to [intelligent apps](https://computese.com/artificial-intelligence-and-intelligent-apps/), and for more on the topic, the [AI category](https://computese.com/category/ai/).

## Key terms
- **Computer vision**: The part of AI that turns camera images into usable facts: which objects are present, where they are, which pixels belong to each one and how far away they are.
- **Segmentation**: Labelling every pixel of an image with the object it belongs to, so the robot gets an exact outline rather than a rough box. Models like SAM return a mask for an object you point at.
- **Proprioception**: A robot's sense of its own body: joint angles, gripper opening and, on some arms, force and torque. Policies read it alongside the camera images.
- **Policy**: The part of a robot's software that chooses the next action from what it currently observes. In AI robotics it is a neural network trained on data rather than a hand-written program.
- **Imitation learning**: Training a policy to copy demonstrations, usually recorded by a person teleoperating the robot. Also called behaviour cloning.
- **Diffusion policy**: A policy that generates a short sequence of actions by starting from random noise and denoising it step by step, conditioned on what the cameras see. It copes well with tasks that can be done in several valid ways.
- **Sim-to-real gap**: The difference between a physics simulator and the real world (friction, lighting, textures) that makes a policy trained in simulation fail on real hardware unless training accounts for it.
- **Vision-language-action (VLA) model**: A robot policy built on a vision-language model: it takes camera images and a plain-language instruction and outputs robot actions, inheriting knowledge the underlying model learned from the web.
- **World model**: A learned model that predicts how an environment will change, including how the robot's own actions will change it, so the robot can compare options before acting.
- **Diffusion Forcing**: A training method from MIT CSAIL (NeurIPS 2024) in which a sequence model denoises tokens that each carry their own noise level, combining next-token prediction with full-sequence diffusion.

## Common questions

### How is AI used in robotics?

AI handles the parts of a robot's job that vary: recognizing and locating objects in camera images, choosing movements with a policy learned from demonstrations or practice, understanding instructions in plain language and, in newer systems, predicting what will happen before acting. Classic controllers and safety systems still run underneath.

### How do robots see with AI?

Robots combine colour cameras, depth sensors and their own joint readings. Computer vision models detect objects, segment them into exact outlines and estimate distance and orientation, which tells the gripper where to go. In vision-language-action models this happens inside one network that goes from image to motor command.

### What is a vision-language-action (VLA) model?

It is a vision-language model, the kind that answers questions about images, trained further on robot data so that it outputs actions instead of only text. Google DeepMind's RT-2 introduced the approach in July 2023; OpenVLA, π0, GR00T N1 and Gemini Robotics followed.

### What is a world model in robotics?

A world model predicts how a scene will change, including the effect of the robot's own actions. A robot can use it to imagine several plans and pick the one whose predicted outcome matches the goal. Recent world models learn from video, predicting either future frames or compact representations of them.

### What did MIT's Diffusion Forcing research show?

It showed that one sequence model can do both next-token prediction and full-sequence diffusion by giving each token its own noise level. In the paper's robot test, an arm that had to remember the starting layout of two fruits succeeded 80% of the time, and still 76% with the camera distracted or blocked.

### Are AI-powered robots safe to work around people?

Only when safety does not depend on the AI model. For industrial robots, ISO 10218-1 and ISO 10218-2 (2025 editions) set the safety requirements, and limits such as speed, force and keep-out zones belong in safety systems the learned model cannot override. Household and public-space service robots fall outside that standard's scope.

## Sources
1. [Combining next-token prediction and video diffusion in computer vision and robotics](https://news.mit.edu/2024/combining-next-token-prediction-video-diffusion-computer-vision-robotics-1016), MIT News
2. [π0: Our First Generalist Policy](https://www.pi.website/blog/pi0), Physical Intelligence
3. [Isaac GR00T: Generalist Robot 00 Technology](https://developer.nvidia.com/isaac/gr00t), NVIDIA Developer
4. [Segment Anything](https://arxiv.org/abs/2304.02643), Meta AI (arXiv)
5. [SAM 2: Segment Anything in Images and Videos](https://arxiv.org/abs/2408.00714), Meta AI (arXiv)
6. [Depth Anything V2](https://arxiv.org/abs/2406.09414), NeurIPS 2024 (arXiv)
7. [Gemini Robotics brings AI into the physical world](https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/), Google DeepMind
8. [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://arxiv.org/abs/2304.13705), arXiv
9. [π*0.6: a VLA that Learns from Experience](https://www.pi.website/blog/pistar06), Physical Intelligence
10. [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://arxiv.org/abs/2303.04137), RSS 2023 (arXiv)
11. [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://arxiv.org/abs/2503.14734), NVIDIA (arXiv)
12. [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://arxiv.org/abs/2109.11978), CoRL 2021 (arXiv)
13. [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://arxiv.org/abs/1703.06907), arXiv
14. [RT-2: New model translates vision and language into action](https://deepmind.google/blog/rt-2-new-model-translates-vision-and-language-into-action/), Google DeepMind
15. [OpenVLA: An Open-Source Vision-Language-Action Model](https://arxiv.org/abs/2406.09246), arXiv
16. [Open Sourcing π0](https://www.pi.website/blog/openpi), Physical Intelligence
17. [Gemini Robotics 2 brings whole body intelligence to robots](https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/), Google DeepMind
18. [Genie 3: A new frontier for world models](https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/), Google DeepMind
19. [Learning Universal Policies via Text-Guided Video Generation](https://arxiv.org/abs/2302.00111), NeurIPS 2023 (arXiv)
20. [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://arxiv.org/abs/2506.09985), Meta AI (arXiv)
21. [π0.7: a Steerable Model with Emergent Capabilities](https://www.pi.website/blog/pi07), Physical Intelligence
22. [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://proceedings.neurips.cc/paper_files/paper/2024/hash/2aee1c4159e48407d68fe16ae8e6e49e-Abstract-Conference.html), NeurIPS 2024
23. [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://arxiv.org/abs/2310.08864), arXiv
24. [π0.5: a VLA with Open-World Generalization](https://www.pi.website/blog/pi05), Physical Intelligence
25. [ISO 10218-1:2025 Robotics: Safety requirements, Part 1: Industrial robots](https://www.iso.org/standard/73933.html), ISO