Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

63 stars

Watchers

0 watching

Forks

Repository files navigation

RoboScope Kit

Unified interfaces for fast and scalable embodied-AI evaluation

MINT Lab@SJTU Project page arXiv PyPI

Paper · Project page · Package guide · Changelog · Contributing

Evaluation time in minutes, 500 episodes: on LIBERO-Long RoboScope Kit finishes in 8.1 minutes (6.16× faster than the official StarVLA evaluation); on RoboTwin Easy it goes from 359.7 to 14.4 minutes (25.04×)

RoboScope Kit evaluates your model on standard robot simulation benchmarks. You wrap a model once in a Python Policy class; the benchmark runtime drives the simulators, schedules the evaluation, and scores the results. Each benchmark is wrapped once in an adapter around its own evaluator, so its rules and scoring stay unchanged.

News

Highlights

  • 🚀 Faster evaluation: on 500 episodes with a StarVLA checkpoint, evaluation is 6.16× faster than the official StarVLA evaluator on LIBERO-Long (one RTX 4090, CUDA MPS) and 25.04× faster on RoboTwin Easy (four RTX 4090s vs. a one-GPU baseline). See Results.
  • 🧩 One interface: every supported benchmark runs through the same Policy API and sends its canonical observations unchanged.
  • ⚙️ Measured concurrency: simulator Workers and Policy replicas run as separate pools. Autotune sizes both from real probe episodes, or you fix them with --concurrency and --replicas.
  • 🌐 Scale-out (Beta): an evaluation can be sharded deterministically across GPU machines and merged afterwards.
  • 📦 Reproducible environments: each benchmark's simulator is built from pinned sources by one script.
  • ♻️ Recoverable runs: Work Items are claimed from a durable queue. A failed Worker's item returns to the queue, and a finished item is scored exactly once.

Architecture

Overview of the RoboScope Kit evaluation architecture

The example metrics are for 500 LIBERO-Long episodes with 12 Workers and 3 replicas on one RTX 4090. See the interactive overview.

Results

From the technical report, each run covering 500 episodes with a StarVLA checkpoint. Evaluation time excludes initialization, concurrency search or offline tuning, and cleanup. LIBERO-Long, on one 48 GiB RTX 4090, with success rates equivalent to the official evaluation's:

Configuration vs. official StarVLA vs. RLinf vs. vla-eval
12 Workers, 3 replicas 4.45× 2.78× 1.52×
16 Workers, 4 replicas, CUDA MPS 6.16× 3.85× 2.10×

RoboTwin Easy, on four GPUs with chunk-boundary observation, takes 14.4 minutes instead of 359.7 with one simulator and one model on one GPU (25.04×), with identical success outcomes and terminal steps. On the same four GPUs, one simulator and one model per GPU takes 98.9 minutes, so RoboScope Kit is 6.88× faster on equal hardware.

Supported benchmarks

Benchmark Suites Contract
LIBERO libero_spatial, libero_object, libero_goal, libero_10, libero_90 LIBERO protocol
LIBERO-Plus libero_spatial, libero_object, libero_goal, libero_10 LIBERO protocol
LIBERO-Pro libero_{goal,spatial,10,object}_{object,swap,lan,task} (16 single-perturbation suites) LIBERO protocol, section 11
RoboTwin demo_clean, demo_randomized Seed protocol
RoboDojo arx_x5, arx_x5_randomized RoboDojo guide
RoboCasa GR-1 Tabletop tabletop Manifest
CALVIN task_D_D, task_ABC_D, task_ABCD_D CALVIN protocol
MetaWorld mt50 MetaWorld protocol
RLBench peract_18 (the PerAct 18-task reproduction profile, not all of RLBench) RLBench protocol
robomimic ph_low_dim robomimic protocol
SimplerEnv google_robot_visual_matching, google_robot_variant_aggregation, widowx_visual_matching SimplerEnv protocol

The simulators, datasets, and assets are not dependencies of the PyPI package; build-runtime.sh builds them separately. Simulator builds, real episodes, and model loading need Linux x86_64, a GPU, and network access.

Ready-made policies

Start from an existing Policy if you use one of these model families:

Model Benchmarks Example
StarVLA LIBERO, LIBERO-Plus, LIBERO-Pro, RoboTwin, RoboCasa GR-1 Tabletop, CALVIN, MetaWorld, with a published checkpoint for each examples/starvla/
OpenVLA LIBERO Spatial, Object, Goal, and Long with suite-specific checkpoints examples/openvla/
X-VLA LIBERO, LIBERO-Plus, RoboTwin, CALVIN examples/xvla/
XPolicyLab RoboDojo examples/xpolicylab/
PerAct RLBench (the released 600k multi-task checkpoint) examples/peract/
robomimic model zoo robomimic (the v0.1 PH low-dim BC-RNN checkpoints) examples/robomimic/
RT-1-X SimplerEnv (Google Robot and WidowX, through SimplerEnv's own RT-1 inference code) examples/rt1/

Quick start

1. Install

Python 3.11 or newer is required.

python -m pip install roboscope-kit

2. Write a Policy

load_model stands for your own model code:

import numpy as np

from roboscope_kit import ActionChunk, BenchmarkName, Policy, PolicyCapabilities, run


class UserPolicy(Policy):
    def load(self, context):
        self.model = load_model(context.checkpoint_path)
        return PolicyCapabilities(
            supported_benchmarks=(BenchmarkName.LIBERO,),
            required_features=("images.primary", "state.proprioception"),
            supported_suites={BenchmarkName.LIBERO: ("libero_10",)},
            supported_action_chunk_sizes=(1,),
        )

    def act(self, observation):
        return ActionChunk(np.asarray(self.model.predict(observation.features), dtype=np.float32))


if __name__ == "__main__":
    run(UserPolicy())

3. Tell the runtime how to start it

Save this as roboscope-policy.json:

{
  "schema_version": 2,
  "command": ["/opt/my-policy/bin/python", "/opt/my-policy/user_policy.py"],
  "cwd": "/opt/my-policy"
}

4. Build the simulator and evaluate

From a checkout of this repository:

benchmarks/libero/build-runtime.sh /sim/libero

python -m roboscope_kit.runtime.benchmark \
    --benchmark libero --suite libero_10 --checkpoint-source /models/my-checkpoint \
    --benchmark-manifest /sim/libero/benchmark-manifest.json \
    --policy-config /path/to/roboscope-policy.json \
    --output-dir /runs/libero_10

Add --quick-test for a short real evaluation with one Worker and one Server. Results, progress, and logs are written under --output-dir. Use a new or empty output directory for each run, including --self-test. Only a directory containing errors.jsonl alone can be retried in place; existing run artifacts are never resumed or overwritten. Keep shell and scheduler logs outside the output directory.

The package guide covers the rest: how evaluation works, automatic tuning, writing a Policy, CUDA MPS, and cross-instance sharding.

Repository layout

Path Contents
packages/roboscope-kit/ The roboscope-kit package: public Policy API, protocol client and server, benchmark runtime, and the evaluation protocol documents
benchmarks/<benchmark>/ For CALVIN, LIBERO, LIBERO-Plus, LIBERO-Pro, MetaWorld, RLBench, RoboCasa GR-1 Tabletop, RoboDojo, RoboTwin, robomimic, and SimplerEnv: the benchmark-manifest.json and build-runtime.sh, which builds the pinned simulator environment; RoboTwin also has its launcher and seed-cache tooling
examples/starvla/ The StarVLA Policy and one demo per benchmark, including the configurations behind the RoboScope Kit technical report
examples/xvla/ An X-VLA Policy example: a stateful Policy serving several benchmarks
examples/xpolicylab/ An XPolicyLab Policy example: serves any XPolicyLab policy on RoboDojo, validated with Simate-beta
examples/peract/ A PerAct Policy example for RLBench, with PerAct's native eval.py runner and an episode-by-episode comparison
examples/robomimic/ A robomimic model-zoo Policy example: per-episode RNN and RNG state, with a native comparison script
examples/rt1/ An RT-1-X Policy example for SimplerEnv, with a runner for SimplerEnv's official scripts and an episode-by-episode comparison
ci/ Build, compatibility, and PyPI release scripts used by GitHub Actions

Contributing

Contributions are very welcome! 🎉 Bug reports, documentation fixes, new Policy examples, new benchmarks, and performance work all help, and questions are contributions too. Open an Issue to report a bug or propose a change, and read CONTRIBUTING.md for how to set up a checkout, run the checks, and what a pull request needs. Performance pull requests need a same-machine A/B measurement.

Citation

If you find RoboScope Kit helpful in your research, please consider citing our work:

@misc{wu2026roboscopekit,
  title         = {RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation},
  author        = {Wu, Deng-Ke and Jin, Yifan and Yang, Yida and Qian, Lei and Zhou, Zhaoye and Zhang, Hongru and Ye, Kai and Hu, Shuyuan and Mao, Yiran and Ye, Zewei and Zhao, Bo},
  year          = {2026},
  eprint        = {2611.03850},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

License

Copyright 2026 RoboScope Team. Licensed under the Apache License, Version 2.0. See LICENSE.

Community

Ask questions, share your results, and discuss new benchmarks in GitHub Discussions. Report bugs as Issues.

WeChat users can also scan the QR code to join the RoboScope Kit group (the invitation is valid before November 19, 2026):

WeChat QR code for the RoboScope Kit community

About

No description, website, or topics provided.

Resources

Contributing

Stars

63 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages