Unified interfaces for fast and scalable embodied-AI evaluation
Paper · Project page · Package guide · Changelog · Contributing
RoboScope Kit evaluates your model on standard robot simulation benchmarks. You wrap a model once in a Python
Policy class; the benchmark runtime drives the simulators, schedules the evaluation, and scores the results.
Each benchmark is wrapped once in an adapter around its own evaluator, so its rules and scoring stay unchanged.
- 🤖 2026-11-10: RLBench (the PerAct 18-task suite) is supported.
- 🤖 2026-11-06: robomimic and LIBERO-Pro are supported.
- 🎉 2026-11-05: RoboScope Kit is released! See the technical report.
- 🤖 2026-11-04: RoboDojo is supported.
- 🚀 Faster evaluation: on 500 episodes with a StarVLA checkpoint, evaluation is 6.16× faster than the official StarVLA evaluator on LIBERO-Long (one RTX 4090, CUDA MPS) and 25.04× faster on RoboTwin Easy (four RTX 4090s vs. a one-GPU baseline). See Results.
- 🧩 One interface: every supported benchmark runs through the same
PolicyAPI and sends its canonical observations unchanged. - ⚙️ Measured concurrency: simulator Workers and Policy replicas run as separate pools. Autotune sizes both
from real probe episodes, or you fix them with
--concurrencyand--replicas. - 🌐 Scale-out (Beta): an evaluation can be sharded deterministically across GPU machines and merged afterwards.
- 📦 Reproducible environments: each benchmark's simulator is built from pinned sources by one script.
- ♻️ Recoverable runs: Work Items are claimed from a durable queue. A failed Worker's item returns to the queue, and a finished item is scored exactly once.
The example metrics are for 500 LIBERO-Long episodes with 12 Workers and 3 replicas on one RTX 4090. See the interactive overview.
From the technical report, each run covering 500 episodes with a StarVLA checkpoint. Evaluation time excludes initialization, concurrency search or offline tuning, and cleanup. LIBERO-Long, on one 48 GiB RTX 4090, with success rates equivalent to the official evaluation's:
| Configuration | vs. official StarVLA | vs. RLinf | vs. vla-eval |
|---|---|---|---|
| 12 Workers, 3 replicas | 4.45× | 2.78× | 1.52× |
| 16 Workers, 4 replicas, CUDA MPS | 6.16× | 3.85× | 2.10× |
RoboTwin Easy, on four GPUs with chunk-boundary observation, takes 14.4 minutes instead of 359.7 with one simulator and one model on one GPU (25.04×), with identical success outcomes and terminal steps. On the same four GPUs, one simulator and one model per GPU takes 98.9 minutes, so RoboScope Kit is 6.88× faster on equal hardware.
| Benchmark | Suites | Contract |
|---|---|---|
| LIBERO | libero_spatial, libero_object, libero_goal, libero_10, libero_90 |
LIBERO protocol |
| LIBERO-Plus | libero_spatial, libero_object, libero_goal, libero_10 |
LIBERO protocol |
| LIBERO-Pro | libero_{goal,spatial,10,object}_{object,swap,lan,task} (16 single-perturbation suites) |
LIBERO protocol, section 11 |
| RoboTwin | demo_clean, demo_randomized |
Seed protocol |
| RoboDojo | arx_x5, arx_x5_randomized |
RoboDojo guide |
| RoboCasa GR-1 Tabletop | tabletop |
Manifest |
| CALVIN | task_D_D, task_ABC_D, task_ABCD_D |
CALVIN protocol |
| MetaWorld | mt50 |
MetaWorld protocol |
| RLBench | peract_18 (the PerAct 18-task reproduction profile, not all of RLBench) |
RLBench protocol |
| robomimic | ph_low_dim |
robomimic protocol |
| SimplerEnv | google_robot_visual_matching, google_robot_variant_aggregation, widowx_visual_matching |
SimplerEnv protocol |
The simulators, datasets, and assets are not dependencies of the PyPI package; build-runtime.sh builds them
separately. Simulator builds, real episodes, and model loading need Linux x86_64, a GPU, and network access.
Start from an existing Policy if you use one of these model families:
| Model | Benchmarks | Example |
|---|---|---|
| StarVLA | LIBERO, LIBERO-Plus, LIBERO-Pro, RoboTwin, RoboCasa GR-1 Tabletop, CALVIN, MetaWorld, with a published checkpoint for each | examples/starvla/ |
| OpenVLA | LIBERO Spatial, Object, Goal, and Long with suite-specific checkpoints | examples/openvla/ |
| X-VLA | LIBERO, LIBERO-Plus, RoboTwin, CALVIN | examples/xvla/ |
| XPolicyLab | RoboDojo | examples/xpolicylab/ |
| PerAct | RLBench (the released 600k multi-task checkpoint) | examples/peract/ |
| robomimic model zoo | robomimic (the v0.1 PH low-dim BC-RNN checkpoints) | examples/robomimic/ |
| RT-1-X | SimplerEnv (Google Robot and WidowX, through SimplerEnv's own RT-1 inference code) | examples/rt1/ |
Python 3.11 or newer is required.
python -m pip install roboscope-kitload_model stands for your own model code:
import numpy as np
from roboscope_kit import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run
class UserPolicy(Policy):
def load(self, context):
self.model = load_model(context.checkpoint_path)
return PolicyCapabilities(
supported_benchmarks=(BenchmarkName.LIBERO,),
required_features=("images.primary", "state.proprioception"),
supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
supported_action_chunk_sizes=(1,),
)
def act(self, observation):
return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))
if __name__ == "__main__":
run(UserPolicy())Save this as roboscope-policy.json:
{
"schema_version": 2,
"command": ["/opt/my-policy/bin/python", "/opt/my-policy/user_policy.py"],
"cwd": "/opt/my-policy"
}From a checkout of this repository:
benchmarks/libero/build-runtime.sh /sim/libero
python -m roboscope_kit.runtime.benchmark \
--benchmark libero --suite libero_10 --checkpoint-source /models/my-checkpoint \
--benchmark-manifest /sim/libero/benchmark-manifest.json \
--policy-config /path/to/roboscope-policy.json \
--output-dir /runs/libero_10Add --quick-test for a short real evaluation with one Worker and one Server. Results, progress, and logs are
written under --output-dir.
Use a new or empty output directory for each run, including --self-test. Only a directory containing
errors.jsonl alone can be retried in place; existing run artifacts are never resumed or overwritten.
Keep shell and scheduler logs outside the output directory.
The package guide covers the rest: how evaluation works, automatic tuning, writing a Policy, CUDA MPS, and cross-instance sharding.
| Path | Contents |
|---|---|
packages/roboscope-kit/ |
The roboscope-kit package: public Policy API, protocol client and server, benchmark runtime, and the evaluation protocol documents |
benchmarks/<benchmark>/ |
For CALVIN, LIBERO, LIBERO-Plus, LIBERO-Pro, MetaWorld, RLBench, RoboCasa GR-1 Tabletop, RoboDojo, RoboTwin, robomimic, and SimplerEnv: the benchmark-manifest.json and build-runtime.sh, which builds the pinned simulator environment; RoboTwin also has its launcher and seed-cache tooling |
examples/starvla/ |
The StarVLA Policy and one demo per benchmark, including the configurations behind the RoboScope Kit technical report |
examples/xvla/ |
An X-VLA Policy example: a stateful Policy serving several benchmarks |
examples/xpolicylab/ |
An XPolicyLab Policy example: serves any XPolicyLab policy on RoboDojo, validated with Simate-beta |
examples/peract/ |
A PerAct Policy example for RLBench, with PerAct's native eval.py runner and an episode-by-episode comparison |
examples/robomimic/ |
A robomimic model-zoo Policy example: per-episode RNN and RNG state, with a native comparison script |
examples/rt1/ |
An RT-1-X Policy example for SimplerEnv, with a runner for SimplerEnv's official scripts and an episode-by-episode comparison |
ci/ |
Build, compatibility, and PyPI release scripts used by GitHub Actions |
Contributions are very welcome! 🎉 Bug reports, documentation fixes, new Policy examples, new benchmarks, and
performance work all help, and questions are contributions too. Open an
Issue to report a bug or propose a change, and read
CONTRIBUTING.md for how to set up a checkout, run the checks, and what a pull request
needs. Performance pull requests need a same-machine A/B measurement.
If you find RoboScope Kit helpful in your research, please consider citing our work:
@misc{wu2026roboscopekit,
title = {RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation},
author = {Wu, Deng-Ke and Jin, Yifan and Yang, Yida and Qian, Lei and Zhou, Zhaoye and Zhang, Hongru and Ye, Kai and Hu, Shuyuan and Mao, Yiran and Ye, Zewei and Zhao, Bo},
year = {2026},
eprint = {2611.03850},
archivePrefix = {arXiv},
primaryClass = {cs.RO}
}Copyright 2026 RoboScope Team. Licensed under the Apache License, Version 2.0. See LICENSE.
Ask questions, share your results, and discuss new benchmarks in GitHub Discussions. Report bugs as Issues.
WeChat users can also scan the QR code to join the RoboScope Kit group (the invitation is valid before November 19, 2026):


