# Simulation: local or served

A simulated environment can be driven in two ways:

- as a **batched env** (`--env.*`): many environments for one task, stepped together by each `step()` call. `lerobot-eval` and evaluation during training use it.
- as a **sim robot** (`--robot.type=sim`): one environment behind the `Robot` interface, swappable with real hardware. `lerobot-rollout`, `lerobot-record`, `lerobot-teleoperate` and `lerobot-replay` use it.

Either runs **locally**, through the environment API, or **served** by `lerobot-env-server`, usually in its own Docker image, so that the simulator's dependencies never meet the policy's. Setting an `endpoint` selects served execution; both paths expose the same canonical observations (HWC uint8 images, flat float32 state), episode ends, and control conventions.

The code keeps three layers apart:

- an **env** (`lerobot.envs`, `--env.type`) is what a LeRobot command runs, configured by an `EnvConfig`. Not every env is a simulator: `gym_manipulator` drives a real robot.
- a **sim** (`lerobot.sim`) is the simulator behind a simulated env: a `SimConfig` naming its integration, stepped by `LocalVectorEnv`, without torch.
- the **env server** (`lerobot.env_server`) serves a sim; `RemoteVectorEnv` drives it with the same API as `LocalVectorEnv`.

## Evaluate a checkpoint

Locally, with the simulator installed next to LeRobot (see [LIBERO](./libero)):

```bash
lerobot-eval \
  --policy.path=lerobot/smolvla_libero \
  --policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
  --policy.device=cuda \
  --env.type=libero --env.task=libero_spatial --env.task_ids='[0]' \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
  --eval.n_episodes=1 --eval.batch_size=1 \
  --output_dir=outputs/eval_libero
```

Served: start the server with the same environment (see [Serve an environment](#serve-an-environment)), and add its endpoint:

```bash
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true
lerobot-eval ... --env.server.endpoint=tcp/127.0.0.1:7448
```

Every option for the server is under `--env.server.*`, and none applies to a local run. The client asks for the deployment named after `--env.type`, as the server configurations name it; pass `--env.server.deployment` for another. A wrong endpoint or deployment fails within `--env.server.handshake_timeout_s` (10 s) and names what the endpoint serves. Against a server secured with TLS, pass its Zenoh JSON5 file with `--env.server.zenoh_config_path`, as for remote inference. The server must run the environment the command describes: the same type, task, control conventions and simulated body (`robot_type`), or the evaluation refuses to start and lists the differences. Other options that differ (resolution, episode length) are logged as warnings.

Observation names come from the environment (`observation.images.<camera>`, `observation.state`). When a checkpoint was trained with other camera names, map them with `--rename_map`, as for any robot, and preserve each camera's identity even when several have the same shape. Placeholder cameras (`empty_cameras`) are a policy setting saved with the checkpoint; override it with `--policy.empty_cameras` only when needed.

Before the first step, every checkpoint input must exist in the environment after renaming, with matching state and action dimensions and RGB channel count. Image height and width may differ: policy preprocessing owns resizing. Controller conventions (delta vs absolute actions, rotation format) are not checked; use a checkpoint trained on the environment's convention.

Each `step()` advances every unfinished environment in the batch once. An environment that ends keeps its terminal observation until the next reset; stepping the others never restarts it. With `--eval.use_async_envs=true` (the default), each environment runs in its own worker process, locally or on the server, and the batch steps in parallel. A server admits up to `max_envs` environments per batch (32 by default). `--env.server.encoding=jpeg` (or `--robot.env.server.encoding=jpeg`, with `--env.server.jpeg_quality`, 90 by default) sends served camera frames as JPEG: smaller replies for a remote server and larger batches, at the cost of encoding time and lossy frames. On localhost, raw frames are faster. A reply carries the frames of every environment in the batch and must fit `max_payload_bytes` (32 MiB by default) at both ends, or the server refuses the batch when it opens. For large batches or high resolution cameras, raise it on the server (`max_payload_bytes` in the YAML's `zenoh` section, next to `listen_endpoints`) and on the client (`--env.server.max_payload_bytes`, or `--robot.env.server.max_payload_bytes`).

## Run a sim robot

The same environment as a batched env and as a sim robot:

```bash
lerobot-eval    --env.type=libero --env.task=libero_spatial --env.task_ids='[0]' ...
lerobot-rollout --robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 ...
```

For example:

```bash
lerobot-rollout \
  --robot.type=sim --robot.env.type=libero --robot.env.task=libero_spatial --robot.task_id=0 \
  --task="pick up the black bowl between the plate and the ramekin and place it on the plate" \
  --policy.path=lerobot/smolvla_libero \
  --policy.pretrained_revision=a24f1f380461b665158d52acd140152c79e8b36d \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
  --device=cuda --fps=20 --inference.type=sync \
  --display_data=true --display_mode=rerun \
  --return_to_initial_position=false --play_sounds=false
```

- `--robot.env.*` takes the same options as `--env.*` in evaluation. `--robot.task_id` picks the one task the robot runs, where evaluation runs every task of the suite or those in `--env.task_ids`. EnvHub environments are `--robot.env.type=envhub --robot.env.hub_path=<user/repo> --robot.env.trust_remote_code=true`.
- Some policies need an environment's processors (XVLA and Cosmos3 on LIBERO, Isaac Lab Arena). `lerobot-eval` applies them, and so does a sim robot with `--inference.type=sync`. Remote inference does not, so it refuses such a checkpoint at startup.
- `--robot.env.server.endpoint=tcp/...` drives the environment on an env server instead, as `--env.server.endpoint` does for a batched env. The robot checks that the server runs that environment.
- Each `send_action()` advances the environment exactly once. Observation reads, policy inference, and waiting do not advance simulation; the existing control loop owns wall-clock pacing.
- `--fps` must equal the environment's control rate, divided by `--interpolation_multiplier` when interpolating, since every command advances the environment once. When recording, `--dataset.fps` must equal `--fps`. Rollout refuses to start otherwise.

A sim robot is a robot: swap `--robot.type=sim --robot.env.*` for the real follower's `--robot.*`, and everything else stays. A teleoperator drives it as it drives the real follower (`--teleop.type=so101_leader` for an SO-101 twin), when the environment declares the follower's action names and units; a mismatch is reported at the first action. `lerobot-record` stores the body the environment declares as the dataset's `robot_type` (`panda` for LIBERO), or `sim_robot` when it declares none.

Recorded episodes follow the environment's. `lerobot-record` and the episodic rollout strategy restart the simulated environment at the start of each episode, and end it when the environment's episode ends (success, termination or its step limit) or when its duration is up, whichever comes first. Each recorded episode is therefore one simulated episode, from its initial state. Nothing needs resetting by hand, so skip the reset phase between episodes with `--dataset.reset_time_s=0`, and with the episodic rollout strategy also pass `--strategy.reset_to_initial_position=false`. When an episode ends, the robot stops: it ignores actions and keeps its final state, and `latest` keeps the final reward, termination and success. Teleoperation and the base rollout strategy then start a new episode (the strategy also resets the policy), and replay stops. The sentry, highlight and DAgger strategies don't support sim robots yet, since they record by time or on demand rather than per episode; use the episodic strategy to record. Real robots are unaffected: their episodes are timed, and their scene is reset by hand. For success rates, evaluate with `lerobot-eval` and a batched env.

Robot camera features use the same `observation.images.<camera>` names as real-hardware recording. A single unnamed camera is `observation.images.image`. Batched evaluation keeps the environment's own keys, such as `observation.image` for a single camera; when deploying a checkpoint with that name through `SimRobot`, use `--rename_map='{"observation.images.image": "observation.image"}'`.

For remote inference, keep the same `--robot.*` options and replace the local policy and inference options with:

```bash
--inference.type=remote \
--inference.endpoint=tcp/127.0.0.1:7447 \
--inference.deployment=libero \
--inference.semantics=libero-delta-v1
```

Start the policy server separately using [remote inference setup](./remote_inference). Its schema must match the robot's observations after `--rename_map`. Omit `--policy.path` from the rollout command because the policy server loads the checkpoint. For local RTC, use `--inference.type=rtc` with the policy's RTC settings.

When the policy is late, holding a sim robot does not advance physics. Simulator failures propagate to rollout shutdown.

## Serve an environment

From a LeRobot source checkout (the image is built from it), run from the repository root:

```bash
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true
```

`--docker=true` uses `docker/sim/compose.server.yaml`, selects the image from `sim.type`, mounts the effective YAML, and runs the container on the host network, so the server listens on `localhost:7448` unless `--zenoh.listen_endpoints` says otherwise. Swap the YAML to select another simulator. Each simulator's control rate, action convention, component names and robot type come from its integration's `conventions()`, so a YAML only names the task and the simulator's options.

As for `lerobot-policy-server`, every YAML field accepts a nested command-line override, and the file stays unchanged. A dictionary value such as `sim.kwargs` is merged into the YAML's:

```bash
lerobot-env-server --config_path=docker/sim/configs/libero.yaml --docker=true \
  --sim.task=libero_goal \
  --sim.kwargs='{gym_kwargs: {observation_height: 256, observation_width: 256}}'
```

The effective configuration is kept under `$HF_LEROBOT_HOME/sim`, so a restarted container finds it. Switching configuration recreates the server and ends active sessions. The image is built when missing; add `--build=true` to rebuild it after changing the simulator or LeRobot source, or `--stop=true` to shut down the server. The container requires Docker GPU access for EGL rendering. [EnvHub](./envhub) repositories are served with `--envhub=<user/repo>`; see [EnvHub and the simulator server](./envhub#envhub-and-the-simulator-server).

> [!TIP]
> If a simulator that renders with Vulkan (Isaac Sim, or ManiSkill/SAPIEN as in RoboMME, whose image ships its own Vulkan file) fails in the container with `vkCreateInstance failed` or `ERROR_INCOMPATIBLE_DRIVER` while it works outside Docker, the container has the NVIDIA driver's libraries but not the loader files that point at them (`/usr/share/vulkan/icd.d/nvidia_icd.json`, `/usr/share/glvnd/egl_vendor.d/10_nvidia.json`). The NVIDIA Container Toolkit's default mode, used by `gpus: all`, may not mount them; its [CDI mode](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html) does. Request the GPU through CDI (`--device nvidia.com/gpu=all` for `docker run`, `devices: [nvidia.com/gpu=all]` in place of `gpus: all` in `docker/sim/compose.server.yaml`), or add the two files to the image.

Without `--docker=true`, the same command runs the server in the current environment: install the simulator's native dependencies in an isolated environment, install LeRobot with `--no-deps`, and install `docker/sim/requirements.txt`. Either way it listens on localhost; to accept other machines, pass the address of the network interface they reach, as in `--zenoh.listen_endpoints='[tcp/192.168.1.20:7448]'`, and add TLS and access control with a `zenoh.config_file` in the YAML.

LIBERO runtime images use converted NumPy initial-state files. The image build converts trusted upstream assets with `scripts/ci/convert_libero_init_states.py`; the server rejects missing converted files. RoboTwin and RoboMME require their own torch for native planning and ManiSkill, respectively; their torch installations are isolated from the policy environment.

Server requests are never retried. After an ambiguous timeout, the client fails; close it and open a new session.

## Add a benchmark

See [Adding a benchmark](./adding_benchmarks) for simulator modules, conventions, images, and CI integration.