LIBERO-plus

LIBERO-plus is a robustness benchmark for Vision-Language-Action (VLA) models built on top of LIBERO. It systematically stress-tests policies by applying seven independent perturbation dimensions to the original LIBERO task set, exposing failure modes that standard benchmarks miss.

• Paper: In-depth Robustness Analysis of Vision-Language-Action Models
• GitHub: sylvestf/LIBERO-plus
• Dataset: lerobot/libero_plus

Perturbation dimensions

LIBERO-plus creates ~10 000 task variants by perturbing each original LIBERO task along these axes:

Available task suites

LIBERO-plus perturbs the 40 original tasks in the Spatial, Object, Goal, and Long suites. Each registered task in these four suites is a perturbed variant, rather than one of the original tasks. The fork also retains the 90 unperturbed tasks in LIBERO-90.

These counts come from the upstream task registry. The four perturbed suites contain 10,030 task variants in total; LIBERO-90 is not included in that total. Counts and task indices depend on the installed fork revision.

Installing LIBERO-plus **replaces** vanilla LIBERO — it uninstalls `hf-libero` so that `import libero` resolves to the LIBERO-plus fork. You cannot have both installed at the same time. To switch back to vanilla LIBERO, uninstall the fork and reinstall with `pip install -e ".[libero]"`.

Installation

System dependencies (Linux only)

sudo apt install libexpat1 libfontconfig1-dev libmagickwand-dev

Python package

pip install -e ".[libero]" "robosuite==1.4.1" bddl easydict mujoco wand scikit-image gym
git clone https://github.com/sylvestf/LIBERO-plus.git
cd LIBERO-plus && pip install --no-deps -e .
pip uninstall -y hf-libero  # so `import libero` resolves to the fork

LIBERO-plus is installed from its GitHub fork rather than a pyproject extra — the fork ships as a namespace package that pip can’t handle, so it must be cloned and added to PYTHONPATH. See docker/Dockerfile.benchmark.libero_plus for the canonical install. MuJoCo is required, so only Linux is supported.

Set the MuJoCo rendering backend before running evaluation:

Copied

export MUJOCO_GL=egl   # headless / HPC / cloud

Download LIBERO-plus assets

LIBERO-plus ships its extended asset pack separately. Download assets.zip from the Hugging Face dataset and extract it into the LIBERO-plus package directory:

# After installing the package, find where it was installed:
python -c "import libero; print(libero.__file__)"
# Then extract assets.zip into <package_root>/libero/assets/

Evaluation

Default evaluation (recommended)

Evaluate every registered variant across the four perturbed suites, using one episode per variant as recommended by the upstream evaluation guide:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
  --eval.batch_size=1 \
  --eval.n_episodes=1 \
  --env.max_parallel_tasks=1

This runs 10,030 episodes for the registry above. With --eval.n_episodes=10, the same command runs 100,300 episodes, not 400.

Smoke test

Before launching a full evaluation, check your setup with one registered variant and one episode:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial \
  --env.task_ids='[0]' \
  --eval.batch_size=1 \
  --eval.n_episodes=1

This is a setup check, not a representative robustness evaluation.

Single-suite evaluation

Evaluate on one LIBERO-plus suite:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial \
  --eval.batch_size=1 \
  --eval.n_episodes=1

• --env.task picks the suite (libero_spatial, libero_object, etc.).
• --env.task_ids restricts to registered task indices ([0], [1,2,3], etc.), which identify variants in the four perturbed suites. Omit to run all registered tasks. When evaluating multiple suites, the same index list is applied independently to each suite.
• --eval.batch_size controls how many environments run in parallel.
• --eval.n_episodes sets how many episodes to run per registered task. Increasing it repeats evaluation of the selected variants; it does not select additional distinct variants.

Multi-suite evaluation

Benchmark a policy across multiple suites at once by passing a comma-separated list:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object \
  --eval.batch_size=1 \
  --eval.n_episodes=1

Control mode

LIBERO-plus supports two control modes — relative (default) and absolute. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:

--env.control_mode=relative   # or "absolute"

Reset performance

By default, LeRobot preserves LIBERO’s hard-reset behavior. With fixed initial states enabled, you can opt into soft resets to skip rebuilding the simulator model and renderer on every episode:

--env.init_states=true --env.hard_reset=false

Soft resets are faster but are not bit-identical to hard resets after the environment’s settling steps, so camera observations and policy results may differ slightly. Use hard resets when reproducing benchmark results.

Policy inputs and outputs

Observations:

• observation.state — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
• observation.images.image — main camera view (agentview_image), HWC uint8
• observation.images.image2 — wrist camera view (robot0_eye_in_hand_image), HWC uint8

Actions:

• Continuous control in Box(-1, 1, shape=(7,)) — 6D end-effector delta + 1D gripper

Evaluation workload and reproducibility

The total number of evaluation episodes is the number of selected registered tasks across all suites × --eval.n_episodes. For example, all Spatial variants with one episode each give 2,402 episodes; selecting --env.task_ids='[0,1,2]' in Spatial and Object with one episode each gives six episodes.

The 400-episode calculation (40 original tasks × 10 episodes) applies to vanilla LIBERO, not a full LIBERO-plus evaluation. For LIBERO-plus, start with one episode per registered variant. Repeated episodes of a variant are not equivalent to sampling distinct variants of an original task.

For a custom subset, use the fork’s task classification to identify perturbation categories and difficulty levels. Do not assume that every original task has ten distinct variants in every perturbation category. Record the fork revision, selected suite names and task indices, episodes per variant, seed, and reset settings when reporting results. To compare with published results, follow the benchmark’s sampling and aggregation protocol rather than treating an arbitrary subset as equivalent.

Training

Dataset

A LeRobot-format training dataset for LIBERO-plus is available at:

• lerobot/libero_plus

Example training command

lerobot-train \
    --policy.type=smolvla \
    --policy.repo_id=${HF_USER}/smolvla_libero_plus \
    --policy.load_vlm_weights=true \
    --dataset.repo_id=lerobot/libero_plus \
    --env.type=libero_plus \
    --env.task=libero_spatial \
    --output_dir=./outputs/ \
    --steps=100000 \
    --batch_size=4 \
    --eval.batch_size=1 \
    --eval.n_episodes=1 \
    --env_eval_freq=1000

Relationship to LIBERO

LIBERO-plus is a drop-in extension of LIBERO:

• Same Python gym interface (LiberoEnv, LiberoProcessorStep)
• Same camera names and observation/action format
• Same task suite names
• Installs under the same libero Python package name (different GitHub repo)

To use the original LIBERO benchmark, see LIBERO and use --env.type=libero.