WALL-SS bridges the gap between simulated and real-world robotics with precision

Demonstrations of robots executing martial arts or dynamic movements often overlook the subtle disruptions of everyday settings: a shifted cup or stray cloth can disrupt pre-trained motions, leading to collisions or spills. This highlights the persistent gap between virtual training and physical execution, where even minor scene changes render motion plans ineffective.

World models aim to replicate real-world physics within simulations, offering a promising path for embodied AI. Yet most current approaches fail practical tests, misrepresenting visual inputs while ignoring physical constraints. Objects may appear graspable in simulation but behave unpredictably in reality—a phenomenon X-Square Robot calls “magnet-like grasping.” This inconsistency undermines real-world applicability.

To address these flaws, WALL-SS introduces an autoregressive world model designed for long-horizon robotic tasks. Its core innovation, Next-Scale Autoregression, avoids overestimating action outcomes by generating predictions in stages. First, it produces a low-resolution preview of broad movements—such as the arm’s trajectory or object displacement—before refining it into high-definition details like contact points. If an input command to move left produces a visual output showing rightward motion, the model penalizes this misalignment. Testing confirmed its superior action sensitivity score of 0.290, surpassing Cosmos3 (0.044) and models scoring zero, ensuring the simulation adheres to the robot’s physical feedback.

Long-horizon tasks like pouring water—spanning approach, grasp, tilt, and replacement—require precise memory management over 60 seconds, exceeding a robot’s immediate recall capacity. WALL-SS implements Scale-Compressed Long-term Memory, storing recent interactions in high fidelity while summarizing older data. For instance, it retains the exact gripper contact with a cup two seconds ago but compresses earlier observations—such as the cup’s presence on the table or an open drawer—into contextual notes without pixel-level precision. To prevent cumulative errors, Per-Scale Dream Forcing injects controlled noise during training, forcing the model to correct trajectories despite disturbances. This resilience helps robots recover from minor deviations rather than compounding mistakes.

Beyond simulation fidelity, WALL-SS enables direct policy evaluation by comparing virtual and real-world performance. Through Online Policy Alignment of Visual Dynamics, the system employs two internal evaluators: Judge 1 verifies if the simulated arm follows the intended trajectory (Action Consistency), while Judge 2 ensures objects remain coherent across camera views (Long-term Consistency). Policies that succeed in this virtual “dream” environment reliably transfer to chaotic human environments, like cluttered kitchens.

The project’s foundations are available at http://x2robot.com/pages/ss for further exploration. The technical paper and implementation details are published on GitHub.