Commands

Commands come from the model

A model declares typed input fields; each public field becomes a set_<field>command with a validated payload. We do not define the command set, so a new model with different controls needs no API change. Read a session's commands from its schema:

GET /v1/realtime/sessions/{id}/schema

The response is an OpenAPI 3.1 document: one /events/set_<field> path per command, with the payload schema, default, bounds and description, and an x-reactor.tracks list of the media tracks. The schema is available once the session is ready. For the models we serve, see the model catalog.

Setting state before the first frame

Pass starting_input when creating the session to apply state before any viewer connects, so the first frames already reflect it:

{
  "state":    { "prompt": "a red fox in fresh snow", "seed": 7 },
  "commands": [ { "command": "set_paused", "data": { "paused": false } } ]
}

Both keys are optional. state sets fields directly; commands applies commands in order.

Sending commands while streaming

A command is a protobuf DataClientMessage carrying a Command with the command name and a JSON-like payload. In the reference Python client:

viewer.command("set_prompt", prompt="neon city at night")
viewer.command("set_paused", paused=True)

When a command takes effect

Generation is block-by-block, so a command can only apply at the next block boundary. How long that is depends on the model: for longlive-1.3ba block is 12 frames, which is 750 ms of video at 16 fps, plus the time to generate it. A prompt change on that model also re-builds the model's context, which makes the block that applies it slower than normal. Measured end-to-end command latency is not published yet.

Validation