Interactive World Simulator for Robot Policy Training and Evaluation

Imagine that you want to teach a robot to push something on a table. The standard recipe in robot learning is to collect hundreds of expert performances on a real robot, train an imitation learning policy on that data, and then evaluate the policy by running it multiple times on the same real robot. Both phases (data collection and evaluation) are slow, expensive, and difficult to reproduce: equipment breaks, lighting changes, objects drift out of place, and each new task means more hours in the lab.

A natural question is whether we can replace some of the actions of a real robot with a simulator. Classic physics-based simulators are powerful, but building one for a new task means manually modeling the geometry, contacts, friction, and deformation, and the resulting simulator still often does not match reality closely enough to transfer the policies trained within it.

In our work, we take a different path. We are building Interactive world simulation: A learned, motion-conditional video prediction model, which, given the current image and sequence of robot actions, predicts the next frames in pixel space only, with no physics engine inside. You can connect a remote control and control a robot through this universal learning model for over 10 minutes at 15 fps on a single RTX 4090, and the expected video remains stable and physically reasonable.

The basic idea is that if the simulator is faithful enough, we can solve two long-standing bottlenecks in robot learning:

– Generating data for training purposes has become cheaper, because we can collect demonstrations within the simulator.
– Policy evaluation becomes scalable and repeatable, because we can put many policies through the simulator under identical conditions.

What a world simulator can do
We trained our scientist simulator on four manipulation tasks involving very different physical systems: T-pushing (contact with the rigid object), rope guidance (deformable solid interaction with the clamp), cup grasping (gripper microdynamics), and pile sweeping (manipulating piles of objects). All four behaviors are learned from interaction data alone, without any prior physical principles having to be encoded.

Some examples of what the model captures:

Rope routing: Correctly distinguish between a rope that is actually inserted into the clamp and a rope that is swung in front of it without contacting it. Most importantly, it does not take sides in either outcome, but rather follows what the actions mean.

Grip the Cup: It captures subtle impacts such as the cup sliding from the handle, or the handle being pushed and rotated.

Stacks Sweeping: It can create video for multiple viewpoints continuously

You can try it directly on our website Project page: Just open your browser and play with your keyboard!

How we built the interactive world simulator

At a high level, training of the interactive world simulator takes place in two stages. First, we trained an autoencoder that compresses RGB images into compact 2D latent representations and reconstructs them back into images. This allows the model to think about a lower dimensional space while still producing high-resolution output at the pixel level.

Second, we immobilized this autoencoder and trained a dynamic model conditional on movement in the latent space. Given previous visual latencies and robot actions, the model predicts the next latent state, which is then decoded back into an image. At inference time, this process is repeated autoregressively: predicted frames become part of the context of predicting future frames. Because the prediction occurs in latent space and uses coherence models, the simulator can be run interactively while remaining stable over long horizons.

How is it different from previous works?

This differs from many existing approaches to robotics simulation and global modeling. Classical physics simulators can be powerful, but they often require manual definition of the object’s geometry, contact dynamics, friction, and deformation, and the resulting simulation may still have a large gap between simulation and reality. Modern video generation methods and global models offer a more data-driven alternative, but many of them are either not explicitly conditioned by robot actions, too slow for real-time interaction, closed source, or unstable under long-term deployments.

In contrast, our interactive world simulator is motion-adapted, produces virtually accurate predictions at the pixel level, and supports stable interactions for over 10 minutes at 15 fps on a single consumer RTX 4090 GPU. This makes it practical not only as a visual prediction model, but also as an interactive engine to collect policy training data and repeatedly evaluate robotics policies. It has multiple applications.

Application 1: Scalable data generation

If our simulation is good enough, can combined demonstrations using this global model actually replace real demonstrations of learning and imitation policies training?

To answer this question, we collected the entire demos inside our world simulator and then trained imitation learning policies on 0% of the real-world data and 100% of the generated data. We deployed the trained policies directly on the real robot. Published policies don’t just complete tasks. They remain robust in the face of ongoing human disturbances. This shows that the quality of the generated data is comparable to real data.

Application 2: Evaluating robust policies

Evaluating a bot policy in the real world is difficult: you need to reset the scene, rerun the policy several times, and compare across checkpoints under identical conditions. In practice, this is not scalable and repeatable.

In turn, we can introduce policies into our global model for replicable evaluation. We evaluated the same four policies (DP, ACT, π0, π0.5) inside the simulator and on the real robot, with the same initial configurations. Each point in our assessment is a political checkpoint, captured in both the real world and the global model. We observed a strong relationship between the two outcomes, across tasks and across policies. Qualitatively, good policies that work in the simulator also work on the real robot, and bad policies that fail in the simulator also work on the real robot.

Looking forward

Conditional video prediction has long shown great potential for robotics, but existing methods have been either too slow for interactive use or too fragile under long-horizon inference. By addressing these two things – stability and computational efficiency – the Interactive World Simulator becomes a practical engine for two things at once: training imitation policies on simulator-generated data that perform similarly to those trained on real-world demonstrations, and evaluating policies in a way that closely tracks real-world performance.

Going forward, we will expand the framework to include more diverse environments and increasingly complex processing tasks, in order to unleash the potential of large-scale automated data. An important direction for future work is to examine how to measure the performance of global models with increasing amounts of interaction data and computational resources. Understanding these scaling behaviors can guide the design of larger, more capable global models of robots.