AI-generated data for robotics is training data that software produces instead of a robot recording it: physics simulations, images and videos from generative models, and new demonstrations derived from a few real ones. Teams use it because real robot data is slow and expensive to collect, and it is only as useful as its measured match with reality.
This guide explains why robot data is so scarce, how simulation and domain randomization work, how generative models such as MIT CSAIL's LucidSim and NVIDIA's Cosmos add realism, how teams measure the sim-to-real gap, and what goes wrong when a model learns from data nobody recorded. The models that consume this data, including vision-language-action models, are covered in our guide to AI in robotics.
What counts as AI-generated data for robotics
Synthetic data is anything a robot learns from that no sensor captured in the real world. In robotics it comes in three families, and most working pipelines combine at least two:
| Family | How the data is made | What it gets right | What it gets wrong | Examples |
|---|---|---|---|---|
| Physics simulation | A simulator renders scenes and computes motion and contact from rules | Geometry, controllable physics, exact labels for free | Simplified appearance, friction and contact | Domain randomization, Isaac Sim |
| Generative models | Image, video or scene generators trained on real media | Realistic appearance and wide variety | Physics is learned from pixels, not guaranteed | LucidSim, Cosmos, DreamGen |
| Demonstration multiplication | A few real demonstrations adapted to new objects, poses and robots | Motions that start from real human skill | Only variations of what someone demonstrated | MimicGen |
A training example for a robot is more than a picture. For perception it pairs camera frames with labels: object boxes, segmentation masks that outline each object pixel by pixel, depth. For learning a policy, the part of the robot's software that chooses the next move, it pairs what the cameras saw with what the robot did: joint positions and gripper commands. Software can produce both halves at once, because a simulator knows exactly where every object is and what every joint did. That is the core appeal.
Why real robot data is scarce and expensive
Language models learn from text that already exists. Meta pretrained its largest Llama 3 model on 15.6 trillion text tokens. Robots have no equivalent: every real training example has to be produced by a robot in a physical place, usually with a person teleoperating it.
DROID shows what that costs. It is a real-world manipulation dataset released in March 2024 and built for diversity: 76,000 demonstration trajectories, or 350 hours of interaction, collected in 564 scenes and 84 tasks by 50 data collectors in North America, Asia and Europe over 12 months. Its authors name the obstacles plainly: collecting robot data in diverse places poses logistical and safety challenges and needs substantial investment in hardware and human labour. As a result, they note, even the most general manipulation policies of the time were trained mostly on data from a small number of environments.
Three costs keep real data from scaling:
- Every scene is physical. Someone sets up the table, the objects and the lighting, then resets them after every attempt. MIT CSAIL's Ge Yang made the point about LucidSim: demonstrations are easy to collect, but scaling teleoperation to thousands of skills is hard because a person has to physically set up every scene.
- Failures are expensive. The most useful lessons (a slip, a collision, a fall) are the ones you least want to stage on real hardware. Simulation lets a robot explore failure scenarios that are difficult to produce in the real world, and learn from as much of its own experience as the compute allows.
- Labels cost time. A real image has no object boxes or masks until someone draws them. A simulator writes them out with every frame.
How simulation and domain randomization generate data
The oldest source of synthetic robot data is a physics simulator. It computes how objects move and collide, renders what the robot's cameras would see and records every action. It can run faster than real time and in parallel, and nothing breaks. NVIDIA's Isaac Sim, for example, includes Replicator, a set of synthetic data generation tools with randomizers and "annotators" that output ground truth such as bounding boxes and semantic segmentation for every frame.
The catch is that simulated images and physics are simplified, and a model trained on them learns the simulator's quirks. Domain randomization is the classic answer. In a 2017 paper from OpenAI and UC Berkeley, Tobin and colleagues trained an object detector on hundreds of thousands of low-fidelity rendered images with random camera positions, lighting, object positions and deliberately non-realistic textures. The bet was that a model trained on enough variation would treat the real world as one more variation. Trained only on those images, the detector located real objects to within 1.5 cm, coped with distractors and partial occlusion, and was accurate enough to guide grasping in clutter. The authors described it as the first deep network trained only on simulated RGB images, with no pretraining on real images, to transfer to the real world for robotic control.
The same idea extends from appearance to physics:
- Dynamics randomization. Peng and colleagues randomized the simulator's dynamics during training. Their robot arm's pushing policy learned to adapt to physics it had never seen and kept a similar level of performance on the real robot, without any training on it.
- Automatic domain randomization. OpenAI's Rubik's cube work (October 2019) generated randomized environments of ever-increasing difficulty, and policies trained only in simulation solved the cube with a humanoid robot hand.
Randomization buys robustness by making the model cope with everything, but the robot never sees what the real world actually looks like. The LucidSim team sums up the trade-off: domain randomization is still the go-to method, and its data is diverse but not realistic. That gap is where generative models come in.
Generative models that repaint the simulator
Generative models trained on real images and video can make a simulated scene look real. The best-known systems share a division of labour: the simulator stays in charge of geometry and physics, and the generative model only decides how things look. Because the geometry does not change, what the simulator knows about the scene (where a step edge is, how tall a box is) still holds for the generated picture.
LucidSim (MIT CSAIL, 2024)
LucidSim trained a four-legged robot dog to do visual parkour with a single low-cost colour camera and no depth sensor, without any real-world training data. The research was presented at the Conference on Robot Learning (CoRL) in November 2024, in the paper Learning Visual Parkour from Generated Images. The pipeline works in three steps:
- Describe. A large language model writes varied descriptions of environments. The team sourced diverse prompts from ChatGPT because a fixed prompt kept producing near-identical images.
- Condition. The physics simulator supplies a depth map and a semantic mask of each scene, and an image generator paints a realistic picture that respects them.
- Animate. A technique the team calls Dreams In Motion computes how each pixel moves between frames from the scene's 3D geometry and the robot's change of viewpoint, and warps one generated image into a short, consistent video.
MIT reported a sharp difference in its comparison: robots that learned from an expert teacher succeeded 15% of the time, and quadrupling the expert's data barely helped, while robots that gathered their own experience in LucidSim reached 88% once their dataset was doubled. The team also reported outperforming domain randomization.
NVIDIA Cosmos (2025 onward)
NVIDIA announced Cosmos at CES on January 6, 2025: a platform of world foundation models, tokenizers, guardrails and a video processing pipeline for robots and autonomous vehicles. NVIDIA presents the models as a way to generate large amounts of photoreal, physics-based synthetic data to train and evaluate existing models, from text, image and video inputs as well as robot sensor or motion data, under an open model license. The accompanying paper frames the idea: a physical AI system needs a digital twin of itself, the policy, and a digital twin of the world, the world model.
Cosmos-Transfer1, published in March 2025, is the part that repaints simulation. It generates video conditioned on spatial inputs such as segmentation, depth and edge maps, which can be weighted differently in different parts of the frame, and NVIDIA demonstrates it for robotics sim-to-real transfer and for enriching autonomous-driving data. It applies LucidSim's division of labour in an open model: render the scene in a simulator, keep its structure, change its appearance.
Generative models that invent scenes, videos and actions
The second use of generative AI goes further. Instead of repainting a scene someone built, it creates the scene, or the robot's whole experience:
- Scenes. MIT CSAIL and the Toyota Research Institute's steerable scene generation, presented at CoRL in September 2025, trains a diffusion model on over 44 million 3D rooms and then steers it, with Monte Carlo tree search or reinforcement learning, toward scenes that are physically feasible (a fork does not pass through a bowl) and useful for a task. It followed text prompts 98% of the time for pantry shelves and 86% for messy breakfast tables. The output is a simulation scene made of 3D object models, so a virtual robot can interact with it.
- Robot videos with inferred actions. NVIDIA's DreamGen (May 2025) fine-tunes image-to-video models on a target robot, prompts them to show familiar or new tasks in new places, then recovers the missing actions with a latent action model or an inverse dynamics model. The authors call the results neural trajectories. With teleoperation data from only a single pick-and-place task in one environment, a humanoid robot performed 22 new behaviours in both seen and unseen environments.
- Actions directly. Cosmos 3, announced on May 31, 2026, combines vision reasoning with generation across text, video, images, sound and action in one model. NVIDIA says it can output numerical action data such as joint angles, gripper positions and trajectory points, and that developers can fine-tune it for a particular robot, camera layout, workspace or task.
Not every data generator is a generative model. MimicGen (CoRL 2023) adapts a few human demonstrations to new scene configurations, object instances and robot arms: about 200 human demonstrations became over 50,000 across 18 tasks, and the authors found the generated data compared favourably with collecting more human demonstrations.
The sim-to-real gap, and how teams measure it
The sim-to-real gap, also called the reality gap, is the difference between what a policy does in simulation and what it does on the real robot. The SIMPLER authors single out two kinds of disparity: control, where the simulated robot and objects respond differently to the same commands (friction, contact, masses, motor response), and visual, where the scene looks different (textures, lighting, reflections, camera noise).
An unmeasured gap is dangerous for a specific reason: models exploit simulators. Kadian and colleagues scanned a real lab into a virtual replica and ran nine navigation models in both. Agents trained for a simulator-based challenge had learned to abuse the simulator's collision handling, sliding along walls to take shortcuts through space that is not passable in reality. Tricks like that score well in simulation and fail on hardware.
Teams measure the gap in four ways:
- Real success rate over many trials. This is the number that matters. Define success before testing, run enough trials for the rate to mean something, and include real scenes the training data never showed.
- Paired simulation and real evaluation. Run the same set of policies (checkpoints, ablations, competing methods) in simulation and on the robot, and check whether simulation ranks them the way reality does. Kadian's team introduced the Sim-vs-Real Correlation Coefficient (SRCC): for the Habitat simulator as used in the CVPR 2019 challenge, SRCC for success was 0.18, so rankings in simulation barely predicted real ones. Tuning the simulator's parameters raised it to 0.844. SIMPLER (2024) uses Pearson correlation and Mean Maximum Rank Violation (MMRV), which weighs each case where simulation ranks two policies the wrong way round by how far apart they really are.
- Ablation of the synthetic data. Train with and without it, and with different amounts, then compare on the same evaluation. DreamGen varied its neural trajectories from 0 to 240,000 in the RoboCasa simulation benchmark and found a log-linear relation between their number and policy performance.
- Scoring the generator itself. DreamGen Bench rates video models on instruction following and physics following, and its authors found the scores correlate with downstream policy success.
Closing the gap uses the same tools in reverse. System identification measures the real robot and sets the simulator's parameters to match; SIMPLER uses it offline, and adds visual matching: "green screening" real backgrounds behind the simulated objects and projecting real object textures onto them. Kadian's jump from 0.18 to 0.844 shows how much predictivity depends on simulator settings. Even good system identification has limits: Tobin and colleagues point to effects that physics simulators do not capture, such as parts that flex, gear backlash, wear and fluid dynamics.
The risks of training on synthetic data
Synthetic data fails in ways that are easy to miss, because the model looks fine on the data you generated.
| Risk | What happens | What limits it |
|---|---|---|
| Physics that looks right | Generated video is convincing and physically wrong, so a policy can learn motion no object follows | A simulator in charge of geometry and contact; real trials |
| Simulator exploits | The policy finds shortcuts that only exist in the simulator | Paired simulation and real evaluation |
| Missing edge cases | Generators reproduce what is common; rare events are thin or absent | Generate rare cases on purpose; keep real failures in the dataset |
| Bias | Scenes inherit the defaults of prompts and generators: typical rooms, objects, lighting | Compare coverage with the places the robot will actually work |
| Inferred labels | Actions recovered from generated video are estimates, not measurements | Validate against real, measured trajectories |
| Model collapse | Generators trained on their own output lose the tails of the real distribution | Accumulate real and synthetic data; never replace the real set |
Realistic is not the same as correct. The Physics-IQ benchmark (January 2025) tested video models including Sora, Runway, Pika, Lumiere, Stable Video Diffusion and VideoPoet on fluid dynamics, optics, solid mechanics, magnetism and thermodynamics. Physical understanding was severely limited, and it was unrelated to how realistic the videos looked. A robot trained on such video can learn motion that no real object follows, which is why the pipelines above keep a simulator responsible for contact. Our article on AI limitations in understanding covers the same pattern in language models.
Model collapse removes the rare cases first. In a paper published in Nature on July 24, 2024, Shumailov and colleagues showed that training indiscriminately on model-generated content does irreversible damage: the tails of the original distribution vanish, in language models as well as variational autoencoders and Gaussian mixture models. In their language model experiment, keeping 10% of the original data in each generation led to only minor degradation. Gerstgrasser and colleagues (2024) confirmed that replacing real data with each generation's synthetic data tends toward collapse, and found that accumulating synthetic data alongside the original real data avoided it across the model sizes and architectures they tested. For robots, the tails are exactly what matters: the slip, the odd object, the person stepping into the workspace.
Bias travels with the generator. Whatever a generator treats as a typical kitchen, warehouse or street becomes the robot's idea of one. The Nature authors make the fairness point directly: low-probability events are often the ones relevant to marginalized groups. Our guide to fairness in AI decision-making explains how to test for that kind of skew.
Warning
Never let synthetic data grade itself. A policy that scores well on generated scenes has shown only that it fits the generator. Keep a real test set that no training run ever sees, and decide on real trials.
How to use synthetic data without fooling yourself
Whether you train robot policies or ordinary vision and machine learning models, the discipline is the same:
- Start from a real test set collected where the model will work, and never train on it.
- Name the gap you are closing. Appearance calls for visual randomization, generative repainting or visual matching; physics calls for system identification and dynamics randomization; coverage calls for scene generation or demonstration multiplication.
- Keep physics with the simulator where you can, and let generative models change appearance rather than invent contact.
- Mix, do not replace. Keep real data in every training run, and record which examples are synthetic, from which generator, version and prompt.
- Check predictivity before trusting simulation scores. Run a handful of policies in both worlds and compute a rank correlation.
- Generate the rare cases on purpose, then check them by hand, because the generator will not volunteer them.
- Judge the final model only on real trials, including scenes the training data never showed.
Most of that list is data engineering: datasets that are versioned and traceable, quality checks that run before anything reaches training, and a clear line between real and generated records. Our data platform service builds that kind of foundation for businesses, with tested pipelines, a catalog with lineage, and features and datasets served to ML and AI. When a model goes into a business workflow, our AI and automation service applies the rule from step 7: an evaluation set built from real cases, passed before any model, prompt or source change goes live. For more on the topic, see the AI category.
