# Policy Deployment (lerobot-rollout)

`lerobot-rollout` is the single CLI for deploying trained policies on real robots. It supports multiple execution strategies and inference backends, from quick evaluation to continuous recording and human-in-the-loop data collection.

## Quick Start

No extra dependencies are needed beyond your robot and policy extras.

```bash
lerobot-rollout \
    --strategy.type=base \
    --policy.path=lerobot/act_koch_real \
    --robot.type=koch_follower \
    --robot.port=/dev/ttyACM0 \
    --task="pick up cube" \
    --duration=30
```

This runs the policy for 30 seconds with no recording.

### Choosing a device

Pass `--device=cpu` or `--device=cuda` to override the checkpoint device on a host where that device is available.
The resolved device is applied to the policy configuration before loading the policy and constructing its processors.
For example, add `--device=cpu` to the command above to run a CUDA-trained checkpoint on a CPU-only host, if the policy supports CPU execution.
Check that CPU inference can meet your required control rate before deploying on a robot.

When `--device` is omitted, rollout uses the policy configuration device if set, otherwise it selects an available device automatically.
The same behavior applies to `RolloutConfig(device=...)` with `build_rollout_context`.

---

## Strategies

Select a strategy with `--strategy.type=<name>`. Each strategy defines a different control loop with its own recording and interaction semantics.

### Base (`--strategy.type=base`)

Autonomous policy execution with no data recording. Use this for quick evaluation, demos, or when you only need to observe the robot.

```bash
lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="Put lego brick into the box" \
    --duration=60
```

| Flag             | Description                                            | Default |
| ---------------- | ------------------------------------------------------ | ------- |
| `--duration`     | Run time in seconds (0 = infinite)                     | 0       |
| `--task`         | Task description passed to the policy                  | --      |
| `--display_data` | Stream observations/actions to Rerun for visualization | false   |

### Sentry (`--strategy.type=sentry`)

Continuous autonomous recording with periodic upload to the Hugging Face Hub. Episode boundaries are auto-computed from camera resolution and FPS so each saved episode produces a complete video file, keeping uploads efficient.

Policy state (hidden state, RTC queue) persists across episode boundaries: the robot does not reset between episodes.

```bash
lerobot-rollout \
    --strategy.type=sentry \
    --strategy.upload_every_n_episodes=5 \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --dataset.repo_id=${HF_USER}/rollout_eval_data \
    --dataset.single_task="Put lego brick into the box" \
    --duration=3600
```

| Flag                                   | Description                                           | Default |
| -------------------------------------- | ----------------------------------------------------- | ------- |
| `--strategy.upload_every_n_episodes`   | Push to Hub every N episodes                          | 5       |
| `--strategy.target_video_file_size_mb` | Target video file size for episode rotation           | auto    |
| `--dataset.repo_id`                    | **Required.** Hub repository for the recorded dataset | --      |
| `--dataset.push_to_hub`                | Whether to push to Hub on teardown                    | true    |

### Highlight (`--strategy.type=highlight`)

Autonomous rollout with on-demand recording via a memory-bounded ring buffer. The buffer captures the last N seconds of telemetry. Press the save key to flush the buffer and start live recording; press it again to save the episode. With an asynchronous backend, each toggle pauses inference, invalidates queued/interpolated motion and requests a supported local hold before the blocking drain or save. A healthy run resumes from a fresh capture afterward.

```bash
lerobot-rollout \
    --strategy.type=highlight \
    --strategy.ring_buffer_seconds=30 \
    --strategy.save_key=s \
    --strategy.push_key=h \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=koch_follower \
    --robot.port=/dev/ttyACM0 \
    --dataset.repo_id=${HF_USER}/rollout_highlight_data \
    --dataset.single_task="Pick up the red cube"
```

**Keyboard controls:**

| Key                | Action                                                   |
| ------------------ | -------------------------------------------------------- |
| `s` (configurable) | Start recording (flushes buffer) / stop and save episode |
| `h` (configurable) | Push dataset to Hub                                      |
| `ESC`              | Stop the session                                         |

| Flag                                   | Description                                    |
| -------------------------------------- | ---------------------------------------------- |
| `--strategy.ring_buffer_seconds`       | Duration of buffered telemetry (default: 10)   |
| `--strategy.ring_buffer_max_memory_mb` | Memory cap for the ring buffer (default: 1024) |
| `--strategy.save_key`                  | Key to toggle recording (default: `s`)         |
| `--strategy.push_key`                  | Key to push to Hub (default: `h`)              |

### DAgger (`--strategy.type=dagger`)

Human-in-the-loop data collection. Alternates between autonomous policy execution and human intervention via a teleoperator. Intervention frames are tagged with `intervention=True`. Requires a teleoperator (`--teleop.type`).

See the [Human-In-the-Loop Data Collection](./hil_data_collection) guide for a detailed walkthrough.

**Corrections-only mode** (default): Only human correction windows are recorded. Each correction becomes one episode.

```bash
lerobot-rollout \
    --strategy.type=dagger \
    --strategy.num_episodes=20 \
    --policy.path=outputs/pretrain/checkpoints/last/pretrained_model \
    --robot.type=bi_openarm_follower \
    --teleop.type=bi_openarm_mini \
    --dataset.repo_id=${HF_USER}/rollout_hil_data \
    --dataset.single_task="Fold the T-shirt"
```

**Continuous recording mode** (`--strategy.record_autonomous=true`): Both autonomous and correction frames are recorded with time-based episode rotation (same as Sentry).

```bash
lerobot-rollout \
    --strategy.type=dagger \
    --strategy.record_autonomous=true \
    --strategy.num_episodes=50 \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --teleop.type=so101_leader \
    --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=${HF_USER}/rollout_dagger_data \
    --dataset.single_task="Grasp the block"
```

**Keyboard controls** (default input device):

| Key     | Action                                      |
| ------- | ------------------------------------------- |
| `Space` | Pause / resume policy execution             |
| `Tab`   | Start / stop human correction               |
| `Enter` | Push dataset to Hub (corrections-only mode) |
| `ESC`   | Stop the session                            |

Foot pedal input is also supported via `--strategy.input_device=pedal`. Configure pedal codes with `--strategy.pedal.*` flags.

Where the robot supports the rollout position hold, a pause refreshes the last position target actually applied by the driver, including its clipping and the gripper target. Before the first command, a validated measured pose supplies the target. This preserves the commanded position error instead of changing the target to the measured pose; the robot can finish settling toward that target while policy trajectory advancement stops. Validate loaded-arm and gripper behavior on the actual robot; this software hold does not guarantee retained grip or physical immobility.

| Flag                                 | Description                                                                                                                                                    |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--strategy.num_episodes`            | Number of correction episodes to record (default: `--dataset.num_episodes`)                                                                                    |
| `--strategy.record_autonomous`       | Record autonomous frames too (default: false)                                                                                                                  |
| `--strategy.upload_every_n_episodes` | Push to Hub every N episodes (default: 5)                                                                                                                      |
| `--strategy.input_device`            | Input device: `keyboard` or `pedal` (default: keyboard)                                                                                                        |
| `--strategy.smooth_handover`         | Smoothly hand control over at pause / correction start (default: true). Disable for clutch-style teleops that re-reference at the current robot pose on engage |
| `--teleop.type`                      | **Required.** Teleoperator type                                                                                                                                |

### Episodic (`--strategy.type=episodic`)

Episode-oriented recording that mirrors the behavior of `lerobot-record`. The policy drives the robot for each episode; an optional teleoperator can drive the robot during the reset phase between episodes.

```bash
lerobot-rollout \
    --strategy.type=episodic \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --teleop.type=so100_leader \
    --teleop.port=/dev/ttyACM1 \
    --dataset.repo_id=${HF_USER}/rollout_eval_data \
    --dataset.num_episodes=20 \
    --dataset.episode_time_s=30 \
    --dataset.reset_time_s=10 \
    --dataset.single_task="Pick up the red cube"
```

Teleop is optional — if omitted the robot holds its position during the reset phase.

**Keyboard controls:**

| Key         | Action                           |
| ----------- | -------------------------------- |
| `→` (right) | End the current episode early    |
| `←` (left)  | Discard episode and re-record it |
| `ESC`       | Stop the recording session       |

| Flag                                            | Description                                                                                                                                                |
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dataset.num_episodes`                        | Number of episodes to record                                                                                                                               |
| `--dataset.episode_time_s`                      | Duration of each recording episode in seconds. Defaults to `--duration` when that is set and this one is not                                               |
| `--dataset.reset_time_s`                        | Duration of the reset phase between episodes in seconds                                                                                                    |
| `--teleop.type`                                 | Optional. Teleoperator to drive the robot during resets                                                                                                    |
| `--strategy.reset_to_initial_position`          | Whether to reset the robot to its initial position between episodes                                                                                        |
| `--strategy.smooth_leader_to_follower_handover` | Whether to turn on or off the leader -> follower smooth handover behavior.                                                                                 |
| `--strategy.smooth_handover`                    | Smoothly hand control to the teleop at reset start (default: true). Disable for clutch-style teleops that re-reference at the current robot pose on engage |

### Bring your own strategy

`--strategy.type` accepts any strategy registered in the environment, not only the built-ins above. A strategy is two classes in a pip-installable package: a config registered on `RolloutStrategyConfig`, and a `RolloutStrategy` subclass named after it without the `Config` suffix — the same `SomethingConfig`/`Something` convention as [custom hardware](./integrate_hardware#the-4-core-conventions), resolved from the config's module or package.

```python
from dataclasses import dataclass
from typing import ClassVar

from lerobot.rollout import RolloutContext, RolloutStrategy, RolloutStrategyConfig

@RolloutStrategyConfig.register_subclass("patrol")
@dataclass
class PatrolStrategyConfig(RolloutStrategyConfig):
    dataset_mode: ClassVar[str] = "required"  # the engine creates the dataset for you
    legs: int = 2  # becomes --strategy.legs

class PatrolStrategy(RolloutStrategy):
    def run(self, ctx: RolloutContext) -> None:
        ...  # the paced control loop: see the contract on RolloutStrategy.run

    def teardown(self, ctx: RolloutContext) -> None:
        ctx.data.dataset.finalize()
        self._teardown_hardware(ctx.hardware, ctx.runtime.cfg.return_to_initial_position)
```

The config declares what the engine arranges on the strategy's behalf, so nothing in LeRobot needs to know its type:

| Declaration                     | Effect                                                                                                                                     |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `dataset_mode`                  | `"none"` (default) rejects `--dataset.*` flags, `"optional"` records when they are given, `"required"` makes `--dataset.repo_id` mandatory |
| `requires_teleop`               | Makes `--teleop.type` mandatory                                                                                                            |
| `supports_interactive`          | Allows `--interactive=true`; `run()` must then honour the restartable contract documented on `RolloutStrategy`                             |
| `supports_sim_robot`            | Allows `--robot.type=sim`; `run()` must then start a new episode whenever one ends (see `lerobot.robots.sim_robot`)                        |
| `requires_streaming_encoding()` | Return `True` when frames are written from inside the control loop: `--dataset.streaming_encoding` is forced on                            |
| `extra_dataset_features()`      | Strategy-owned dataset columns (DAgger's `intervention`, for instance); every recorded frame must carry them                               |

Name the distribution `lerobot_strategy_<name>` and `lerobot-rollout` imports the package before parsing the CLI — only its `__init__.py` runs, so import the config there. A package with any other name is loaded, submodules included, with `--strategy.discover_packages_path=<package>`. The strategy's dataclass fields become `--strategy.*` flags:

```bash
lerobot-rollout \
    --strategy.type=patrol \
    --strategy.legs=3 \
    --policy.path=${HF_USER}/my_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --dataset.repo_id=${HF_USER}/rollout_patrol \
    --dataset.single_task="patrol the aisle"
```

`SentryStrategy` (recording, interactive-safe) and `BaseStrategy` (no recording) are the reference implementations to copy from; `send_next_action`, `safe_push_to_hub` and `estimate_max_episode_seconds` from `lerobot.rollout.strategies` are the helpers they build on.

For asynchronous backends, the dispatch contract is mandatory. Publish each captured observation with `engine.notify_observation(obs_processed)`, call `send_next_action(...)` on **every autonomous motor tick**, including interpolation and held ticks, then call `engine.pump_query(obs_processed)`. Calling only `engine.get_action()` bypasses the permission and hold handshake: a planned text query can wait indefinitely, and a terminal fault may never signal shutdown.

Third-party strategies can call `self._tick(timer, new_cycle=interpolator.needs_new_action())`
in place of `timer.tick(...)` to include periodic health reporting after `_init_engine(ctx)`.

`send_next_action` is built on `lerobot.inference.dispatch_tick` and its parts (`next_interpolated_action`, `commit_action`, `dispatch_permitted`), which encode that order once: `dispatch_allowed()` before and after pulling an action and again before sending, interpolation invalidation plus a supported local hold and `acknowledge_hold()` when denied, and `record_dispatch()` after sending. A hold that fails is never acknowledged. Use those helpers rather than re-implementing the sequence; `send_next_action` only adds the dataset frame, the robot-side action processor and cadence accounting. Remote contexts have `ctx.policy.policy`, `preprocessor` and `postprocessor` set to `None`; use `ctx.policy.inference` and the robot-side pipelines in `ctx.processors`.

Wrap known blocking mid-run recording work in `self._pause_for_recording(ctx)`, including ring-buffer drains and episode finalization. This invalidates asynchronous motion before the recording stall and resumes only if the run is still healthy and active. Strategies with operator-controlled phases must pass their current motion permission as `resume_allowed`. The next action must come from a fresh observation; saved work must not revive a paused, stopped or faulted run.

---

## Visualizing policy predictions

With `--display_data`, the viewer also shows what the policy predicts besides actions: a world
model's imagined camera frames, a predicted subtask, bounding boxes drawn over a camera. Each policy
decides what it predicts and turns it on through its own config, for example
`--policy.return_predicted_video=true` for LingBot-VA. A prediction is named after what it predicts:
Rerun shows a predicted `observation.images.top` as `prediction.images.top`, a predicted subtask as
`prediction.subtask`, and boxes under their camera (`<camera>/prediction`); Foxglove publishes them
on `/prediction/...` topics and `<camera topic>/prediction` image annotations. With RTC, the
prediction refreshes once per chunk. See [Bring your own policy](./bring_your_own_policies) to return
a prediction from a policy.

## Inference Backends

Select a backend with `--inference.type=<name>`. Local synchronous and RTC inference use the policy configured below. The [remote backend](./remote_inference) obtains its policy contract from a server and does not load weights on the robot computer.

### Sync (default)

One policy call per control tick. The main loop blocks until the action is computed.

Works with all policies. No extra flags needed.

### Real-Time Chunking (`--inference.type=rtc`)

A background thread produces action chunks asynchronously. The main control loop polls for the next ready action while the policy computes the next chunk in parallel.

Use RTC with compatible VLA models (Pi0, Pi0.5, SmolVLA) to overlap inference and motion. Continuous execution still requires enough usable action playback to cover inference latency and its variation.

```bash
lerobot-rollout \
    --strategy.type=base \
    --inference.type=rtc \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --policy.path=${HF_USER}/pi0_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="Pick up the cube" \
    --duration=60 \
    --device=cuda
```

| Flag                                        | Description                                                                                         |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `--inference.rtc.execution_horizon`         | Steps to blend with previous chunk (default: varies by policy)                                      |
| `--inference.rtc.mode`                      | `guided` (default) or trained-prefix `trained` for compatible Pi05 checkpoints                      |
| `--inference.rtc.max_guidance_weight`       | Consistency enforcement strength (default: varies by policy)                                        |
| `--inference.rtc.prefix_attention_schedule` | Blend schedule: `LINEAR`, `EXP`, `ONES`, `ZEROS`                                                    |
| `--inference.queue_threshold`               | Backpressure threshold; trained RTC requires at least its maximum delay                             |
| `--inference.action_starvation_grace_s`     | Wait for fresh motion after starvation on a supported robot (default `1.0`; `0` faults immediately) |

See the [Real-Time Chunking](./rtc) guide for details on tuning RTC parameters.

Local RTC runs in a background thread in the rollout process, without transport. A
server and client on the same computer use `--inference.type=remote` instead.
See the [remote guide](./remote_inference) for chunk alignment/blending and the
[waiting behavior](./remote_inference#what-the-robot-does-while-waiting-or-stopping)
shared by asynchronous backends.

**Waiting requires a supported robot.** Ordinary local RTC remains available without
that capability, but planned text/instruction-change pauses and recoverable starvation
need it. Position-controlled SO, OMX, Koch, HopeJr and Rebot followers support it;
OpenArm, Reachy2 and bimanual wrappers have configuration constraints listed in the
[robot support table](./integrate_hardware#waiting-between-predictions).
With support, an empty buffer can retain the last applied target for
`action_starvation_grace_s` (default 1 second), including the time needed to drain old
inference and predict from a fresh capture. After motion has started, zero grace or
an unsupported robot faults immediately on exhaustion; startup has its own deadline.
Unlike the former local RTC path, exhaustion is no longer an indefinitely skipped
tick. Grace never extends expired request or freshness limits.

Local RTC exposes these additional budgets through `--inference.`:

| Parameter               | Default | Meaning                                      |
| ----------------------- | ------- | -------------------------------------------- |
| `max_observation_age_s` | `5 s`   | Maximum source age through action execution  |
| `action_timeout_s`      | `10 s`  | Outstanding action request                   |
| `startup_timeout_s`     | `120 s` | Initial preparation and fresh-action startup |
| `language_timeout_s`    | `120 s` | Planned language operation                   |

These are starting values, not measured hardware guarantees. RTC must retain enough
**post-trim playback** for another inference and timing margin. Raising
`queue_threshold` cannot help if inference routinely consumes the usable prediction
horizon; reduce latency or validate a longer horizon/different action frequency.

A terminal fault never automatically resumes policy motion. With usable observation
acquisition and actuator control, shutdown honors `--return_to_initial_position`
(default true) and disconnects. Returning reads a full observation, so a persistent
camera failure can prevent it. Failed command application suppresses further shutdown
movement for all backends, including sync. This does not change synchronous prediction
or introduce camera-specific recovery.

Compilation and instruction changes

RTC compilation warms both initial and prefixed calls, including a representative
nonzero delay, before motion. Although `--compile_warmup_inferences` defaults to 2,
enabled RTC requires at least 3; startup logs report the effective count. Warmup must
fit `startup_timeout_s`. Later instructions that change tensor shapes can trigger
recompilation and exceed playback or an operation deadline.

A task change during inference discards the obsolete result. Supported robots pause
and request fresh actions; without that waiting capability, remaining actions can run
out. Prefer changing tasks between runs on those robots.

---

## Interactive Sessions

Add `--interactive=true` to drive the rollout from the terminal instead of starting immediately. Hardware connects and the policy loads as usual, but **the robot stays still until you type `/start`** — useful when you want to position the scene first, re-instruct the policy between attempts, or run several takes without paying the load time again.

```bash
lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USER}/my_smolvla_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="pick up the cube" \
    --interactive=true
```

| Command             | Action                                                                                                                                                                                                                                                                                                                  |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/start`            | Start (or restart) the policy control loop                                                                                                                                                                                                                                                                              |
| `/subtask <text>`   | Change the instruction within the current run; asynchronous inference may enter a planned hold. No argument prints the current task. Only affects policies that condition on language (SmolVLA, π0/π0.5, and similar). With an [external planner](#external-planner), `/subtask` also hands control back from autosteer |
| `/vqa <text>`       | Ask a question about what the robot currently sees. Only while a run is in progress, and only with a policy text head or an [external planner](#external-planner)                                                                                                                                                       |
| `/autosteer <goal>` | Hand over a high-level goal and let the policy's text head or the [external planner](#external-planner) choose the subtasks. `/autosteer off` hands control back; no argument reports the current goal                                                                                                                  |
| `/reset`            | Stop movement, return the robot to its startup position, and restore the `--task` instruction                                                                                                                                                                                                                           |
| `/stop`             | End the session and run the normal shutdown routines                                                                                                                                                                                                                                                                    |
| `/help`             | List the commands                                                                                                                                                                                                                                                                                                       |

```text
> /start
Rollout running — task 'pick up the cube'. /subtask <text> to change it, ...
> /subtask put the cube in the box
Task: 'pick up the cube' → 'put the cube in the box' (applies from the next policy inference)
> /vqa is the cube inside the box?
Asked: 'is the cube inside the box?' — answering from the next observation...
Q: is the cube inside the box?
A: No — the cube is on the table to the left of the box.
> /reset
Task restored to 'pick up the cube'
Resetting — returning the robot to its initial position...
Robot reset — holding at initial position. /start to run.
> /stop
```

Lines starting with `>` are what you type; the session renders no prompt of its own.

`Ctrl-C` still shuts down as usual, and closing stdin (`Ctrl-D`, or the end of a piped script) ends the session — so a piped script must keep stdin open for the intended duration:

```bash
(printf '/start\n'; sleep 60; printf '/stop\n') | lerobot-rollout ... --interactive=true
```

**`/subtask`** updates the task for subsequent inference. With `--inference.type=sync`, precomputed chunk actions are dropped. With local RTC or remote inference, a change during an action request discards the obsolete result. On a robot supporting local hold, the engine invalidates queued/interpolated motion, waits for the control loop to acknowledge the hold, then requests fresh motion under the new task. Expect a stop during this transition; it is not a guaranteed continuous blend. A change between requests can retain already accepted actions for continuity. Local RTC on a robot without hold support still discards the obsolete result, but can exhaust its remaining actions and terminate. Change instructions between runs if that robot cannot hold appropriately.

With local `--use_torch_compile=true`, an instruction that changes processed tensor shapes can trigger recompilation. Padding may keep some task changes at the same shape, but later compilation is not guaranteed to fit the remaining playback or the instruction-change deadline: it can cause a terminal fault. Leave compilation off when exploring instructions unless those transitions have been validated. Changes during initial compile warmup do not request a motion hold, because policy motion has not started yet.

`/vqa` and `/autosteer` require a policy that declares text generation support and implements `PreTrainedPolicy.generate_text()`, such as an appropriately trained EO1 checkpoint, or an [external planner](#external-planner). Capability is checked before accepting a query; checkpoint-specific behavior still needs validation.

**`/vqa`** answers from the next observation and prints when ready, so you can keep typing meanwhile; only one question is in flight at a time (autosteer's own queries share that slot). With an external planner the question is answered on a worker thread while the robot keeps acting, and nothing below applies. Asynchronous backends first enter an acknowledged local hold and invalidate queued/interpolated motion, then submit a fresh observation to the text path. A healthy active run resumes from a fresh action chunk after the answer; paused or stopped runs stay inactive. Synchronous text generation runs inline on the control thread and drops the policy's precomputed actions, so the next tick resumes from fresh inference. Query time counts against `--duration`. Questions are only accepted while a run is in progress — an idle engine has no current view. If the instruction or execution context invalidates an asynchronous answer, the question receives one cancellation notice instead of the obsolete answer. Segment end also reports unanswered questions, including generation still in progress; reporting cancellation does not interrupt a running model call or free its query slot early.

**`/autosteer`** hands the policy a goal and asks it for the next subtask every `--autosteer_interval_s` seconds (default 10), applying each one through the same path as `/subtask` and announcing it in the chat. Every query is a full text generation, so the interval is a cost knob: lower values re-plan sooner but spend more of the loop generating text instead of acting. Plan progress lives in the policy, not the rollout, so the plan dies with the segment; the sequencer also stops on `/reset`, `/subtask`, `/autosteer off`, and the first planning failure.

**Logs below ERROR are muted while the session runs** so routine output doesn't interleave with what you're typing; errors and fatal inference failures still show, and normal logging resumes when the session ends. The [cadence summaries](#cadence-reporting) are printed in the chat instead, and running without `--interactive` gets you the live log back.

Sessions work over SSH and on headless machines — no display server needed.

**Recording while interactive.** `--strategy.type=sentry` also accepts `--interactive=true` and records continuously while you steer: each `/start`…`/reset` segment saves complete episodes plus one final partial episode, and the dataset stays open until shutdown. Frames carry the task provenance of the action actually sent, so actions already queued or interpolated under the old task keep the old label.

Blocking mid-run episode saves pause asynchronous inference, invalidate queued/interpolated actions and request a supported local hold. They resume from a fresh capture only if the run remains active and healthy. The same protection applies to DAgger saves and Highlight ring-buffer drains/saves. Recording pauses and ordinary starvation share the physical waiting behavior, but their deadlines and reasons for resumption remain separate.

```bash
lerobot-rollout \
    --strategy.type=sentry \
    --policy.path=${HF_USER}/my_smolvla_policy \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --dataset.repo_id=${HF_USER}/rollout_cube_sessions \
    --task="pick up the cube" \
    --interactive=true
```

The other recording strategies (episodic, DAgger, highlight) are not supported: they bind their own keyboard controls and finalize the dataset when `run()` exits, so they cannot be restarted segment by segment.

### External planner

`--planner.model_id=<name>` routes `/vqa` and `/autosteer` through the annotation pipeline's VLM client (`annotations` extra) instead of the policy's text head. Calls run on a worker thread while the robot keeps acting, and no language hold is requested for them, with one deliberate exception: when `/autosteer` starts, the inference backend is paused until the planner's first reply, because the goal is context for the planner, not an instruction for the policy. Asynchronous backends (`rtc`, `remote`) drop actions already queued or being computed for the previous instruction; the synchronous backend holds the robot. Once the first subtask is applied the backend resumes from a fresh observation. `/subtask` still applies your instruction directly and cancels automatic planning; late planner replies cannot override it.

The planner works with every inference backend. With local inference it reuses the loaded processor's subtask recipe when there is one, and otherwise requests a short instruction; this does not establish that a checkpoint can follow it. `--planner.instructions='["pick up the cup","open the drawer"]'` restricts planner output: the VLM is asked to pick one of these strings, and a next-subtask reply outside the list is an error. Case, extra spaces and a trailing period are ignored, and the list's own wording is sent to the policy. Without this flag, the list is read from the task labels of the checkpoint's training dataset (via its `train_config.json`). Every label goes into the prompt, so pass the flag with a shorter list for datasets with many tasks. If the dataset cannot be read, a warning is logged and planner output stays free-form. Remote inference owns the policy, so neither the recipe nor the training vocabulary is available locally: pass `--planner.instructions` explicitly or answers stay free-form. The planner has been exercised on hardware with local inference only.

The external planner receives the last `--planner.history` observation/command pairs (default 4), followed by the current images. Each pair contains the observation used to request an instruction and the instruction subsequently accepted by the engine; acceptance does not imply successful execution. Repeated instructions remain in history so the VLM can assess lack of progress. The history clears on manual takeover, reset, stop, segment end, or a new autosteer goal. Set `--planner.history=0` to disable it. `/vqa` can inspect this history but does not add entries. For a next-subtask request the VLM must first describe the scene and state whether the previous command is completed, in progress or failed, and only then name the next instruction. A reply of `done` means the goal is complete: autosteer stops, the run ends, and the robot holds its pose until `/reset` or `/start`. A late `done` from before a `/subtask`, `/reset` or new goal is discarded. Each applied subtask is printed with the model's scene assessment and the request latency.

Start an OpenAI-compatible VLM server separately, then connect:

```bash
transformers serve Qwen/Qwen3-VL-8B-Instruct --port 8000
# Alternatively: vllm serve Qwen/Qwen3-VL-8B-Instruct --port 8000
lerobot-rollout ... --task="pick up the cup" --interactive=true \
    --planner.model_id=Qwen/Qwen3-VL-8B-Instruct \
    --planner.api_base=http://localhost:8000/v1
> /start
> /vqa What is the next short instruction to clear the table?
> /subtask put the cup in the sink
> /autosteer clear the table
```

A hosted model works the same way. This example uses Hugging Face Inference Providers, with a token that has the "Make calls to Inference Providers" permission:

```bash
export HF_TOKEN=hf_...
lerobot-rollout ... --task="put all colored cubes in the green bin" --interactive=true \
    --planner.model_id=Qwen/Qwen3.8-27B:novita \
    --planner.api_base=https://router.huggingface.co/v1 \
    --planner.api_key_env=HF_TOKEN
```

`/vqa` only reports an answer; `/autosteer` applies instructions. Configure the endpoint with `--planner.api_base`, and the key with `--planner.api_key_env=<variable>` (preferred, the key stays out of commands and logs) or `--planner.api_key`. `--planner.request_timeout_s` (default 600) and `--planner.request_max_retries` (default 2) bound each request; `--planner.camera_key` selects one camera; `--planner.log_path` appends every exchange as one JSON line. Model serving dependencies belong in the server environment. `--autosteer_interval_s` measures robot-motion time after a reply is applied, in addition to request latency. Multi-level steering and automatic scene inventory are outside this adapter's scope.

### Programmatic control

`RolloutController` exposes the same operations as a library API — thread-safe `start()` / `reset()` / `stop()` / `set_task()` plus a `RolloutEvent` callback, with no stdin, printing or log muting attached:

```python
from threading import Event, Thread

from lerobot.rollout import (
    LinkedEvent,
    RolloutController,
    RolloutEvent,
    build_rollout_context,
    create_strategy,
)

parent = Event()  # your application's shutdown signal
ctx = build_rollout_context(cfg, LinkedEvent(parent))  # loads policy, connects robot
strategy = create_strategy(cfg.strategy)
strategy.setup(ctx)

controller = RolloutController(strategy, ctx, on_event=print)  # or your own observer
serve_thread = Thread(target=controller.serve)  # serve() blocks; run it where you like
serve_thread.start()

controller.start()                        # robot starts executing the policy
controller.set_task("grab the red cube")  # re-instruct mid-run
controller.reset()                        # stop movement, return home, stay warm
controller.stop()                         # end serve()

serve_thread.join()
strategy.teardown(ctx)  # teardown stays with the caller
```

`RolloutController` requires the context's shutdown event to be a `LinkedEvent`, built as above: the controller ends run segments through its local flag, while your `parent` event still forces a full shutdown. Set `play_sounds=False` unless you want the vocal announcements.

---

## Common Flags

| Flag                              | Description                                                       | Default |
| --------------------------------- | ----------------------------------------------------------------- | ------- |
| `--policy.path`                   | **Required.** HF Hub model ID or local checkpoint path            | --      |
| `--robot.type`                    | **Required.** Robot type (e.g. `so100_follower`, `koch_follower`) | --      |
| `--robot.port`                    | Serial port for the robot                                         | --      |
| `--robot.cameras`                 | Camera configuration (JSON dict)                                  | --      |
| `--fps`                           | Policy inference and dataset recording frequency                  | 30      |
| `--status_interval_s`             | Seconds between health logs (0 disables; non-interactive only)    | 5       |
| `--duration`                      | Run time in seconds (0 = infinite; per segment if interactive)    | 0       |
| `--device`                        | Torch device (`cpu`, `cuda`, `mps`)                               | auto    |
| `--task`                          | Task description (used when no dataset is provided)               | --      |
| `--display_data`                  | Stream telemetry to Rerun visualization                           | false   |
| `--display_ip` / `--display_port` | Remote Rerun server address                                       | --      |
| `--interpolation_multiplier`      | Robot commands per policy action (control at `fps × N` Hz)        | 1       |
| `--interactive`                   | Chat session on stdin; robot idles until `/start` (base, sentry)  | false   |
| `--use_torch_compile`             | Enable `torch.compile` for inference                              | false   |
| `--resume`                        | Resume a previous recording session                               | false   |
| `--play_sounds`                   | Vocal synthesis for events                                        | true    |

---

## Cadence Reporting

Non-interactive rollout logs its status every five seconds. Change the interval with `--status_interval_s=10`, or disable it with `--status_interval_s=0`.

```text
Rollout: loop 29.7/30 Hz · over budget 2/149 · starved 0 · infer 18.7 ms (worst 40.1) · process 2.41 GiB · GPU 5.12/79.6 GiB
```

Latency is median/worst over the interval (`turnaround` for RTC/remote). Unavailable fields are omitted; GPU memory is device-wide, and remote memory is local. Reporting pauses when the loop blocks.

Every strategy reports what its control loop actually achieved, as do `lerobot-record`, `lerobot-teleoperate` and `lerobot-replay` — they all pace through the same timer. Each saved episode gets a one-line digest, and when the loop ends — including on `Ctrl-C` — a block covering the whole run:

```
Cadence (episode 1): 29.84 Hz policy vs 30 Hz target · 1200 ticks, 20.1 s measured · 6/598 cycles over the 33.3 ms budget (work mean 18.7 ms, worst 48.1 ms)
Cadence (episode 2): 29.84 Hz policy vs 30 Hz target · 1200 ticks, 20.1 s measured · 6/599 cycles over the 33.3 ms budget (work mean 18.7 ms, worst 48.1 ms)
Cadence summary — whole run, 2 episodes · target 30 Hz × 2 (16.7 ms tick slot, 33.3 ms cycle budget): 2400 ticks, 1197 cycles judged
  effective cadence: 29.84 Hz policy / 59.67 Hz commands over 40.2 s measured
  cycles over the 33.3 ms work budget: 12/1197 (1.0%) — work mean 18.7 ms, worst 48.1 ms
  ticks over their 16.7 ms slot: 40/2400 (costs interpolation smoothness only)
  loop-body steps (share of measured work):
    observe      mean   3.13 ms · worst   9.00 ms ·  31.7% of work · 2400 calls
    process_obs  mean   0.50 ms · worst   1.00 ms ·   5.1% of work · 2400 calls
    infer        mean   5.18 ms · worst  40.00 ms ·  52.4% of work · 2400 calls
    send         mean   0.40 ms · worst   0.40 ms ·   4.1% of work · 2400 calls
    record       mean   0.67 ms · worst 620.00 ms ·   6.8% of work · 2402 calls
  pacing headroom: 7.4 ms slept per tick on average (max 13.0 ms) — near zero means the loop is saturated
```

Read it top-down: **effective cadence** is what you got against `--fps`, **cycles over budget** is how often the loop could not hold it, the **step breakdown** says where the time went, and **pacing headroom** near zero means there is nothing left to absorb a slow tick. A high `infer` share points at the policy or the device; a high `observe` share points at the cameras. A step's `worst` far above its `mean` is a one-off — above, `record` is 0.67 ms per frame and the 620 ms outlier is the blocking `save_episode` that closed an episode.

A _cycle_ here is one policy action's worth of ticks (`--interpolation_multiplier` of them). Only whole cycles over the `1/fps` budget can cost you frames; a single tick overrunning its slot only makes the interpolated motion less smooth, which is why the two are counted separately. Without interpolation a cycle is a single tick, so the report drops the distinction and counts ticks throughout.

In [interactive sessions](#interactive-sessions) each `/start` segment owns its own timer, so the report lands in the chat once per segment (`/reset`, `/stop`, or `--duration`) rather than once per process.

---

## Programmatic Usage

For extensions, `lerobot.inference` is the public package for backend configurations, local engines and shared scheduling contracts. `lerobot.remote_inference` owns the remote engine, protocol, deployment and session implementation; `lerobot.rollout` owns the robot loop, strategies, interpolation and recording. Import public symbols from their package rather than implementation files. The optional remote dependencies are needed only when the remote backend or server is used.

For custom deployments (e.g. with kinematics processors), use the rollout module API directly:

```python
from lerobot.rollout import BaseStrategyConfig, RolloutConfig, build_rollout_context
from lerobot.inference import SyncInferenceConfig
from lerobot.rollout.strategies import BaseStrategy
from lerobot.utils.process import ProcessSignalHandler

cfg = RolloutConfig(
    robot=my_robot_config,
    policy=my_policy_config,
    strategy=BaseStrategyConfig(),
    inference=SyncInferenceConfig(),
    fps=30,
    duration=60,
    task="my task",
)

signal_handler = ProcessSignalHandler(use_threads=True)
ctx = build_rollout_context(
    cfg,
    signal_handler.shutdown_event,
    robot_action_processor=my_custom_action_processor,       # optional
    robot_observation_processor=my_custom_obs_processor,     # optional
)

strategy = BaseStrategy(cfg.strategy)
try:
    strategy.setup(ctx)
    strategy.run(ctx)
finally:
    strategy.teardown(ctx)
```

See `examples/so100_to_so100_EE/rollout.py` and `examples/phone_to_so100/rollout.py` for full examples with kinematics processors.