Runway GWM Worlds 2 uses WorldPrompt to make real-time interactive simulations

Runway just released a research preview of GWM Worlds 2, which shifts high-fidelity video and audio generation into real-time interactive simulations. They are using an "autoregressive diffusion" model to handle how the environment generates over time. The most interesting part here is WorldPrompt, a specific input format that lets you define a generated world and the actions happening inside it.

How WorldPrompt handles environment control

WorldPrompt acts as a control layer for cameras, characters, and the environment. It isn't a programming language or a scripting tool like you'd find in Roblox or Minecraft; it's a prompting mechanism. You can lock in specific aspects of the simulation, including the first frame, and then trigger a series of events tied to specific timestamps. These actions can even be prompted while the simulation is running.

According to Runway's Principal Research Scientist, Robin Kahlow, this allows for detailed control over subjects in the scene. For example, you could prompt an NPC to walk up to the user and speak. While it isn't perfect since it's still a research preview, movement is reported to be quite reliable, though complex physics or specific character abilities can still be hit-or-miss.

The landscape of real-time world models

Runway isn't the only one pushing for real-time interactive video and audio. There are a few other heavy hitters in this space, though all of them are fighting massive latency and stability issues.

• Runway GWM Worlds 2: Focuses on the WorldPrompt control layer and autoregressive diffusion.
• Google DeepMind Genie 3: Generates at 720p and 24 fps, though Google admits it only supports a few minutes of interaction rather than hours of continuous play.
• Odyssey-2 Pro: Another competitor in the real-time generation space.
• World Labs RTFM: Known as the Real-Time Frame Model.

Transitioning from video models to runtimes

The goal here is to move away from pre-designed, scripted worlds and toward fully self-generated experiences. The engineering hurdle is immense because the model has to generate frames fast enough to feel like a runtime rather than a rendered video.

Runway is reportedly valued at $5.3 billion after a $315 million raise in February, and they've been iterating since the first GWM Worlds launch last December. CTO Kamil Sindi notes that scaling data and models is currently the primary way they are improving how well the world model follows specific prompts. The promise is that we'll eventually have on-demand promptable worlds with synced audio and video across any environment, which is moving out of the "hypothetical" stage and into actual research previews.