Runway AI Updates & Release Notes

Research Preview

Introducing GWM Worlds 2

Interactive worlds generated in real time: continuous 720p video at 24 fps and audio at 48000 Hz, responding to your inputs as you explore.

Built on top of our foundational audio-visual generation model, GWM Worlds 2 turns high-fidelity video and audio generation into real-time interactive simulation. You define the environment, subjects, visual style, physical rules and ambience. Once inside, you steer the world with text actions addressed to any subject or to the scene itself, alongside continuous camera motion. A text action can carry a movement, a gesture, a line of dialogue or a change in the weather.

The world continues from each new input instead of following a fixed clip or script so sessions have no preset length. This creates a foundation for interactive entertainment, virtual characters, robotics and embodied-agent simulation, and generative design and interfaces. Extending GWM Worlds, showcased in December last year, GWM Worlds 2 supports generated audio and rich subject and scene control.

Capabilities

The clips below were generated with GWM Worlds 2. In each one, a user is steering the world with text actions addressed to subjects and the scene, plus continuous camera motion. Recurring actions can be bound to keys for fast play.

Navigation

The model supports both first-person and third-person navigation like walking, driving, riding, looking around and flying through a world. The camera and the subject can be controlled independently.

WorldPrompt

Rich control over a world requires an input representation that generalizes across use cases. We base ours on the insight that a world can be split into two kinds of state: what persists and what changes over time. We formalize this as WorldPrompt, a format with two complementary layers.

• Persistent world context
  • A genesis prompt that includes a scene description covering the environment, its layout, materials, lighting and ambient sound; the subjects that can participate in events, with their attributes; and the laws, reusable conventions that govern behavior, from gravity and collision to character abilities and the camera perspective.
  • A first frame to ground the generation visually.
• A timestamped event stream
  • Actions that describe movement, gestures, object interactions, speech and sound. Each action is a free-form text prompt with start and end timestamps, addressed to a subject or to the scene itself. Multiple actions can overlap.
  • Camera input to represent a per-frame stream of viewpoint translation and rotation.

Example

Here is a concrete example: a street-corner chat between two characters, written as WorldPrompt.

Model Overview

GWM Worlds 2 is an autoregressive diffusion video and audio model generating 720p video at 24 fps and audio at 48000 Hz. GWM Worlds 2 conditions each AR step on three kinds of context: a global context (the genesis prompt and first frame), the current frame's inputs (camera input and the text actions that span the current frame) and the past generated frames, cached in a sliding window. The video and audio decoders are causal and run with a cache for faster decoding.

Attention pattern.

Every token attends to the global tokens (first frame and genesis prompt), and each frame's video, text and audio tokens causally attend to themselves and past frames in a sliding KV-cache window; older frames are evicted. Global tokens attend only themselves. Per frame: v = video, a = audio, t = text, c = camera.

Using the Model

The model itself takes the WorldPrompt, with its persistent and event-based context, and outputs video and audio. There are various ways we could imagine using such a model:

• Ahead of time.
  The user (possibly with the assistance of an LLM) authors the timestamped event stream once at the start, and the model generates the whole video and audio with it.
  Uses: filmmaking, directing.

• Turn-based.
  The generation runs until the user has to make a decision. The user then chooses an action, which an LLM could put into the timestamped event stream format. The model proceeds to generate until the next decision point.
  Uses: visual novels.

• Real-time.
  The model continuously generates video and audio, and the user can take actions that immediately affect the generation stream. This is the most challenging scenario, as it requires text to be available with low latency, reacting immediately to the world. Making users type actions would be too slow, and even VLMs cannot react within tens of milliseconds.
  Uses: games, interactive experiences.

Demo

For some videos in the capabilities section we used the ahead-of-time method to author rich actions beforehand without having to come up with them on the fly. For most of them we used our demo, which implements the real-time method.

Instead of coming up with text prompts on the fly, we bind key and mouse inputs to premade prompts. For example, the W key might be "The character moves forward", and left-click might be "The character throws a ball". While this works, we found the ahead-of-time method produces better quality because the text prompts can describe much more of the scene. For example, if we turn around, we can describe the entire scene that becomes visible, whereas in the real-time demo the model has to come up with every detail on its own, with no anchor besides the persistent context and previous frames.

Multiplayer

Because the model knows the concept of subjects, we can also let different users control different subjects. This can be used to implement multiplayer experiences by letting each user control their own character or by having one user control the world.

In the demo, we add this functionality as roles, where each role (e.g. player 1, player 2, director) can have its own set of actions addressed to its own subjects. The demo uses LiveKit to broadcast the video and audio to everyone accessing it, and each client can select a different role to participate in the experience.

World Authoring

In our demo, sessions are started from presets that contain the first frame, the genesis prompt and the possible actions for the different subjects, bound to different keys. Instead of manually typing out these prompts and coming up with actions, we allow users to create their own presets quickly by generating everything with the assistance of an LLM. A user can type a simple prompt like "third person perspective dirt bike in a snowy landscape" and the assistant takes care of generating everything. This way, we can go from an idea to a world within seconds.

From prompt to preset.

The authoring flow, step by step: describe the world, get a generated first frame, a drafted genesis prompt and a set of subject actions bound to keys — all editable before you hit play.

Limitations

GWM Worlds 2 is a research preview, and real-time generation still trades fidelity for speed. Over time, details, textures and geometry can drift, especially with quick camera movements. Long-term consistency is also imperfect, and the model doesn't support image references beyond the first frame.

While free-form text makes the control surface general, fully leveraging it can require an external real-time harness that tracks the state of the world and generates actions on the fly, for example the NPC dialogue in an NPC–player interaction.

Real-time video generation is still in its earliest stages, and the constraints outlined in this post will be solved with continued research. We expect the same curve that took offline video generation from short, rough clips to high-fidelity production footage to play out again here. GWM Worlds 2 is a very early point on that curve.

Digital interfaces are traditionally implemented through intermediate representations such as code, requiring their appearance and behavior to be specified in advance. We introduce Solaris, an interface world model that instead generates an interactive UI directly, frame by frame, in response to user actions. Solaris treats mouse interactions as conditioning signals and autoregressively synthesizes the resulting visual state at interactive speeds. To enable real-time generation while maintaining visual coherence over extended interactions, we combine autoregressive frame generation with few-step distillation and training on the model's own outputs. A language model complements the visual world model by interpreting user intent and specifying how interactions should affect the generated environment, separating high-level reasoning from visual rendering. By generating both the appearance and behavior of an interface dynamically, Solaris enables open-ended interactions that need not be explicitly programmed in advance. We view interface world models as a step toward a new paradigm for software, where interfaces are generated and adapted continuously around user intent rather than implemented as fixed collections of predefined states and