The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Shared: Panda Dynamic Scenes README row, taxonomy, label, checks and resource rules (#178)
Changhe Chena58de3ec244e
diff --git a/README.md b/README.mdindex 1984391ef5..bbd4bd70f2 100644--- a/README.md+++ b/README.md@@ -66,6 +66,7 @@ drives it from a controller program of its own. It is offered where a benchmark'| LIBERO-MAX (`libero-max-<suite>-<task>-<event>-iNN`) | robosuite 1.4 (MuJoCo 3.2.6, CPU) · Franka Panda; LIBERO tasks with one change in mid-task (a target, receptacle or obstacle moves, the camera, lighting or scene theme changes, sensor noise sets in, distractors appear) | 64 | full || ManiGuard (`maniguard-<family>-iNN`) | BEHAVIOR-1K 3.7.2 (OmniGibson on Isaac Sim 4.5) · Franka Panda with long fin-ray fingers; tabletop tasks with a safety specification checked at every control step | 30 (6 families × 5) | full on 14, none on 16 (human review) || MolmoSpaces (`molmospaces-<family>-i00`) | Ai2 MolmoSpaces on MuJoCo 3.5 (CPU physics) · Franka FR3 + Robotiq 2F-85 on a fixed base; Rainbow RB-Y1 mobile manipulator | 8 | none (human review) |+| Panda Dynamic Scenes (`pandadyn-<code>-<name>-i00`) | ManiSkill 3 / SAPIEN (CPU PhysX at 1000 Hz) · one or two Franka Panda arms with task tools (a plate, a cup, a paddle, a scoop, ...); multi-stage dynamic tasks: catch, strike, carry, hand over, release objects that move by themselves | 25 (12 one-arm + 13 two-arm) | full (the benchmark's recorded reference rollouts) |Each task directory carries a README in one layout: a key table (benchmark, mode, twin, simulator and robot, family andinstance, oracle, budget, image), then Task, Difficulty, Reference solution, Verification and Details.@@ -80,7 +81,7 @@ instance, oracle, budget, image), then Task, Difficulty, Reference solution, Ver- **DexToolBench generators:** `scripts/dtb_mk_task.py`, `dtb_mk_oracle.py`.- **MuJoCo standard-mode twins:** generated by `scripts/mj_mk_limited.py`.- **KinDER, HumanoidBench:** both modes (HumanoidBench: also standard-realtime) rendered by `scripts/kinder/gen_tasks.py` and `scripts/humanoidbench/gen_tasks.py` (`render`, `check`).-- **Other benchmarks:** each has its generator under `scripts/<bench>/` (RoboWits, RoboLab, BEHAVIOR-1K, RoboTangle, YAMLab, RoboDojo, ManiGuard, MolmoSpaces: `mk_tasks.py --check`; VLABench, RoboCasa, RoboCasa365, RoboCasa-GR1, LIBERO-MAX, DexJoCo, MuJoCo Playground's quadrupeds: `generate.py --check`; BiGym 2.0: `gen_tasks.py check`; MyoSuite: `scripts/myosuite/generate.py --check`; LabUtopia: `scripts/labutopia/generate.py --check`; AutoBio: `scripts/autobio/generate.py --check`; MetaWorld+: `generate.py`; RoboPaint: `scripts/robopaint_strict/mk_tasks.py --check`, which runs `scripts/robopaint_brush_mk_task.py --refresh` and adds the process rules). CI runs every check mode (`ci_checks/check-generated-tasks.sh`). Per-benchmark notes: [docs/benchmarks/](docs/benchmarks/).+- **Other benchmarks:** each has its generator under `scripts/<bench>/` (RoboWits, RoboLab, BEHAVIOR-1K, RoboTangle, YAMLab, RoboDojo, ManiGuard, MolmoSpaces, Panda Dynamic Scenes: `mk_tasks.py --check`; VLABench, RoboCasa, RoboCasa365, RoboCasa-GR1, LIBERO-MAX, DexJoCo, MuJoCo Playground's quadrupeds: `generate.py --check`; BiGym 2.0: `gen_tasks.py check`; MyoSuite: `scripts/myosuite/generate.py --check`; LabUtopia: `scripts/labutopia/generate.py --check`; AutoBio: `scripts/autobio/generate.py --check`; MetaWorld+: `generate.py`; RoboPaint: `scripts/robopaint_strict/mk_tasks.py --check`, which runs `scripts/robopaint_brush_mk_task.py --refresh` and adds the process rules). CI runs every check mode (`ci_checks/check-generated-tasks.sh`). Per-benchmark notes: [docs/benchmarks/](docs/benchmarks/).- **RoboFunOlympics:** `scripts/robofunolympics/gen_tasks.py` (`render`, `check`) from the families under`scripts/robofunolympics/shared/rfo/tasks/`; how to write a family: `scripts/robofunolympics/AUTHORING.md`; notes in[docs/benchmarks/robofunolympics.md](docs/benchmarks/robofunolympics.md).diff --git a/ci_checks/check-generated-tasks.sh b/ci_checks/check-generated-tasks.shindex 505ab5b32a..907c2f89ac 100755--- a/ci_checks/check-generated-tasks.sh+++ b/ci_checks/check-generated-tasks.sh@@ -39,6 +39,10 @@ if [ -f "$root/scripts/yamlab/mk_tasks.py" ]; then # YAMLab (its tasks pull reif out=$(python3 "$root/scripts/yamlab/mk_tasks.py" --check 2>&1); then echo "ok scripts/yamlab/mk_tasks.py: ${out##*$'\n'}"else echo "FAIL scripts/yamlab/mk_tasks.py --check:"; echo "$out" | tail -n 40; rc=1; fifi+if [ -f "$root/scripts/pandadyn/mk_tasks.py" ]; then # Panda Dynamic Scenes (its tasks pull request adds the generator)+ if out=$(python3 "$root/scripts/pandadyn/mk_tasks.py" --check 2>&1); then echo "ok scripts/pandadyn/mk_tasks.py: ${out##*$'\n'}"+ else echo "FAIL scripts/pandadyn/mk_tasks.py --check:"; echo "$out" | tail -n 40; rc=1; fi+fiif [ -f "$root/scripts/robodojo/mk_tasks.py" ]; then # RoboDojo (its tasks pull request adds the generator)if out=$(python3 "$root/scripts/robodojo/mk_tasks.py" --check 2>&1); then echo "ok scripts/robodojo/mk_tasks.py: ${out##*$'\n'}"else echo "FAIL scripts/robodojo/mk_tasks.py --check:"; echo "$out" | tail -n 40; rc=1; fidiff --git a/ci_checks/check-tooling-identity.sh b/ci_checks/check-tooling-identity.shindex 0c32156157..aa580d7095 100755--- a/ci_checks/check-tooling-identity.sh+++ b/ci_checks/check-tooling-identity.sh@@ -1,13 +1,13 @@#!/usr/bin/env bash# Shared-source rules (PROTOCOL §2.1, D9: copies of shared sources are byte-identical). Two parts:-# 1. BEHAVIOR-1K, RoboLab, RoboWits, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, RoboTangle, BiGym 2.0, YAMLab, RoboDojo, LabUtopia and AutoBio: within one benchmark, every mode, every copy of a shared file is byte-identical+# 1. BEHAVIOR-1K, RoboLab, RoboWits, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, RoboTangle, BiGym 2.0, YAMLab, RoboDojo, LabUtopia, AutoBio and Panda Dynamic Scenes: within one benchmark, every mode, every copy of a shared file is byte-identical# (the generators copy them from the benchmark's template task unchanged):# - the tooling module rcb_<sim>.py wherever it sits (privileged environment/ and tests/, standard environment/sim/);# - the standard agent side environment/rcb_client.py + rcb1.py and the sim service environment/sim/ (server.py,# finalize.sh, run.sh, rcb1.py);# - the graders, tests/grade.py (one version per mode).# A change to one of them is rolled out to every task of the benchmark at once. Same approach as check-robopaint-tooling.sh.-# 2. The eai-standard/2.2 layer (every task whose metadata.protocol is 2.2: BEHAVIOR, RoboLab, RoboWits, RoboPaint, MolmoSpaces, MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia, AutoBio): the+# 2. The eai-standard/2.2 layer (every task whose metadata.protocol is 2.2: BEHAVIOR, RoboLab, RoboWits, RoboPaint, MolmoSpaces, MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia, AutoBio, Panda Dynamic Scenes): the# agent's client environment/rcb_client.py + rcb1.py and the service's environment/sim/rcb_limited.py + rcb1.py +# run.sh are byte-identical to their source in scripts/eai_standard/ (docs/STANDARD_MODE.md), and the sidecar is# set up for crash recovery: `command: sh /srv/run.sh` (run.sh exits non-zero when the service crashes) and@@ -39,7 +39,8 @@ tasks, src = sys.argv[1], sys.argv[2]MODULE = {"behavior": "rcb_behavior.py", "robolab": "rcb_robolab.py", "robowits": "rcb_robowits.py", "maniguard": "rcb_maniguard.py","molmospaces": "rcb_molmospaces.py", "myosuite": "rcb_myosuite.py","dexjoco": "rcb_dexjoco.py", "robotangle": "rcb_topogym.py", "bigym2": "rcb_bigym2.py", "yamlab": "rcb_yamlab.py", "robodojo": "rcb_robodojo.py",- "labutopia": "rcb_labutopia.py", "autobio": "rcb_autobio.py", "robofunolympics": "rcb_rfo.py"}+ "labutopia": "rcb_labutopia.py", "autobio": "rcb_autobio.py", "robofunolympics": "rcb_rfo.py",+ "pandadyn": "rcb_pandadyn.py"}SHARED = {"environment/rcb_client.py", "environment/rcb1.py", "environment/sim/server.py", "environment/sim/finalize.sh","environment/sim/run.sh", "environment/sim/rcb1.py", "environment/sim/rcb_limited.py", "tests/grade.py"}LAYER = {"environment/rcb_client.py": "rcb_client.py", "environment/rcb1.py": "rcb1.py", "environment/sim/rcb_limited.py": "rcb_limited.py",diff --git a/ci_checks/check-unit-tests.sh b/ci_checks/check-unit-tests.shindex 0fd122f177..539a119fe5 100755--- a/ci_checks/check-unit-tests.sh+++ b/ci_checks/check-unit-tests.sh@@ -47,6 +47,8 @@ if python3 -c "import numpy, requests" 2>/dev/null; thenrobodojo/tests/test_privileged_grading.py; dorun 900 "$t"done+ # Panda Dynamic Scenes' graders, privileged and standard (its tasks pull request adds the test)+ if [ -f "$root/scripts/pandadyn/tests/test_grading.py" ]; then run 300 pandadyn/tests/test_grading.py; fi# LIBERO-MAX's rcb_service.py is its own variant (its verdict and video protocol differ): the classes of the shared# service test that apply to it, run against that copyRCB_SERVICE_DIR="$root/scripts/libero-max/shared/standard" run 900 humanoidbench/tests/test_standard_service.py \diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.mdindex e7363ed239..ae99510763 100644--- a/docs/PROTOCOL.md+++ b/docs/PROTOCOL.md@@ -491,8 +491,8 @@ under an unrelated name would be missed.| LIBERO-MAX | `libero`, `hf-libero`, `rlinf-libero`, `rpent-libero`, `praxis-libero`, `libero-infinity`, `rlinf-liberopro`, `rpent-liberopro`, `rlinf-liberoplus` | LIBERO's goal definitions and initial states; no meshes, no demonstrations, not LIBERO-MAX's events |Not on PyPI under any name found: RoboTwin's own package, DexToolBench / SimToolReal, HumanoidBench, RoboWits, RoboLab,-ManiGuard, DexJoCo, LabUtopia, AutoBio, VLABench, RoboDojo, RoboTangle, RoboOlympics, RoboFunOlympics, BiGym, YAMLab; Isaac Sim does not-install from pypi.org.+ManiGuard, DexJoCo, LabUtopia, AutoBio, VLABench, RoboDojo, RoboTangle, RoboOlympics, RoboFunOlympics, BiGym, YAMLab, Panda Dynamic Scenes+(not published at all: its environment package is this repository's `images/pandadyn/`); Isaac Sim does not install from pypi.org.**Per-benchmark exceptions.** A benchmark's agents have open egress, and its instructions say so, only by ownerdecision, recorded here, in `docs/benchmarks/<bench>.md` and in the three lists that implement it (the same directory@@ -657,7 +657,7 @@ Wire details (endpoints, RCB1 frames, `meta`, error codes, crash recovery) are i| Version | Date | Content ||---|---|---|| eai-standard/2.1 | 2026-09-30 | the base (formerly `rcb-limited/2.1`): no reset; `/submit` action chunks, `/observe`; binary RCB1 responses; 82-column proprioception, optional depth noise; crash recovery. Served by the standard tasks of RoboTwin, DexToolBench, MuJoCo Playground (manipulation), KinDER, HumanoidBench, MetaWorld+, VLABench, the three RoboCasa benchmarks, LIBERO-MAX, BiGym 2.0 and RoboFunOlympics |-| eai-standard/2.2 | 2026-10-01 | additive extension (§6, [`STANDARD_MODE.md`](STANDARD_MODE.md) Part 2) with crash recovery; served by BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, MolmoSpaces, MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia, AutoBio, ManiGuard and MuJoCo Playground's quadruped tasks |+| eai-standard/2.2 | 2026-10-01 | additive extension (§6, [`STANDARD_MODE.md`](STANDARD_MODE.md) Part 2) with crash recovery; served by BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, MolmoSpaces, MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia, AutoBio, ManiGuard, Panda Dynamic Scenes and MuJoCo Playground's quadruped tasks || eai-standard/2.3 | 2026-10-08 | 2.2 + the realtime capability (§4b; [`STANDARD_MODE.md`](STANDARD_MODE.md) Part 3, "Standard-realtime mode"); every 2.2 addition stays a MAY. Served by standard-realtime tasks only (HumanoidBench, RoboOlympics, BiGym 2.0); standard tasks stay on 2.1 / 2.2. Issue #86 || eai-standard/2.4 | 2026-10-08 | 2.3 + the operations capability ([`STANDARD_MODE.md`](STANDARD_MODE.md) Part 4): operation ids and request deadlines (408 / 409), `GET /operations/<id>`, a service-owned call watchdog, a physics-tick ledger, fsynced journals; every earlier addition, the realtime capability included, stays a MAY. Served by standard tasks (RoboTangle); reference implementation `scripts/eai_standard/v2_4/`. Issue #89 |diff --git a/docs/STANDARD_MODE.md b/docs/STANDARD_MODE.mdindex c2b81d8afa..d51619d767 100644--- a/docs/STANDARD_MODE.md+++ b/docs/STANDARD_MODE.md@@ -227,10 +227,11 @@ episode is graded. Results with `N > 0` are reported separately and never ranked> says otherwise, every 2.2 addition is a MAY capability, and a request written for 2.0 or 2.1 gets the same answer as> before (plus parts and keys an old client ignores). Which parts are required is listed in PROTOCOL §4.-**Who serves it.** The standard twins of twelve benchmarks: BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces,-MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia and AutoBio ([docs/benchmarks/](benchmarks/); MyoSuite and AutoBio: MuJoCo on the CPU, resume class `exact`; YAMLab: Isaac+**Who serves it.** The standard twins of thirteen benchmarks: BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces,+MyoSuite, DexJoCo, YAMLab, RoboDojo, LabUtopia, AutoBio and Panda Dynamic Scenes ([docs/benchmarks/](benchmarks/); MyoSuite and AutoBio: MuJoCo on the CPU, resume class `exact`; YAMLab: IsaacLab with CPU PhysX, resume class `exact`; RoboDojo: Isaac Lab with GPU PhysX, resume class `exact`, `approx` for its six-standard-only tasks graded by the live episode; LabUtopia: Isaac Sim, PhysX on the CPU, resume class `exact`), and MuJoCo Playground's quadruped tasks. The RoboTwin and the other MuJoCo standard tasks (DexToolBench, MuJoCo Playground's manipulation tasks) serve 2.1.+standard-only tasks graded by the live episode; LabUtopia: Isaac Sim, PhysX on the CPU, resume class `exact`; Panda Dynamic Scenes:+ManiSkill / SAPIEN with CPU PhysX, resume class `exact`), and MuJoCo Playground's quadruped tasks. The RoboTwin and the other MuJoCo standard tasks (DexToolBench, MuJoCo Playground's manipulation tasks) serve 2.1.**One implementation.** The source is [`scripts/eai_standard/`](../scripts/eai_standard/), the D9 shared source of these sixbenchmarks: the service layer `rcb_limited.py` (HTTP, RCB1 responses, the protocol helpers, the `/spec` self-check, crash@@ -443,7 +444,7 @@ The resume check depends on the simulator's determinism class, which the adapter| class | rule | benchmarks ||---|---|---|-| `exact` | the replayed state hash must equal the checkpoint's; otherwise the episode ends `resume_failed` (`infra_error = 1`) | RoboWits on Genesis' CPU backend; RoboLab (Isaac, CPU PhysX: the resumed service is a fresh process, whose first replay is bit-exact); RoboPaint (ManiSkill/SAPIEN, CPU PhysX, CPU brush model); MolmoSpaces (MuJoCo, CPU physics); YAMLab (Isaac, CPU PhysX, a fixed env seed: the resumed service is a fresh process, whose first run is bit-exact); LabUtopia (Isaac Sim, CPU PhysX, a fresh process as RoboLab; one control step is one physics step); AutoBio (MuJoCo 3.3, CPU physics; one control step is one physics step); RoboDojo (Isaac, GPU PhysX: the resumed service is a fresh process, whose first run is bit-exact, measured on 48 of its 54 tasks; `make_toast` with PhysX enhanced determinism) |+| `exact` | the replayed state hash must equal the checkpoint's; otherwise the episode ends `resume_failed` (`infra_error = 1`) | RoboWits on Genesis' CPU backend; RoboLab (Isaac, CPU PhysX: the resumed service is a fresh process, whose first replay is bit-exact); RoboPaint (ManiSkill/SAPIEN, CPU PhysX, CPU brush model); MolmoSpaces (MuJoCo, CPU physics); YAMLab (Isaac, CPU PhysX, a fixed env seed: the resumed service is a fresh process, whose first run is bit-exact); LabUtopia (Isaac Sim, CPU PhysX, a fresh process as RoboLab; one control step is one physics step); AutoBio (MuJoCo 3.3, CPU physics; one control step is one physics step); RoboDojo (Isaac, GPU PhysX: the resumed service is a fresh process, whose first run is bit-exact, measured on 48 of its 54 tasks; `make_toast` with PhysX enhanced determinism); Panda Dynamic Scenes (ManiSkill / SAPIEN, CPU PhysX with enhanced determinism, one thread; every reset rebuilds the scene; ten physics steps per control step) || `approx` | the hash is compared and the result recorded; a mismatch does not end the episode: `resume_state_mismatch = 1`, and the robot's deviation from its checkpointed proprio is recorded (max \|Δ\| over `joint_pos`; the largest TCP position error from `tcp_pose`). Only a gross divergence, a TCP more than `RCB_RESUME_MAX_DEV_M` (0.05 m) from where it was, ends the episode as `resume_failed` (`infra_error = 1`) | simulators declared nondeterministic: RoboWits on the GPU backend (the particle families separate_marbles_and_sand and stabilize_bottle, `RCB_ROBOWITS_BACKEND=gpu`); BEHAVIOR-1K (OmniGibson declares rollouts nondeterministic; measured so far: live = fresh replay on every 2.2 run, so it normally resumes exactly); ManiGuard (OmniGibson 3.7.2; not measured yet); RoboDojo's six standard-only tasks graded by the live episode (`make_toast_random`, the three liquid tasks and the two cloth tasks: GPU PhysX particles and cloth do not replay bit for bit; PROTOCOL §4.8) |This relaxes "the hash checked" of PROTOCOL §4.7 for declared-nondeterministic simulators only. It cannot raise a score:diff --git a/docs/TAXONOMY.md b/docs/TAXONOMY.mdindex ebeb86dfd3..1b954473b6 100644--- a/docs/TAXONOMY.md+++ b/docs/TAXONOMY.md@@ -7,9 +7,9 @@ Known values; a new value is added by pull request. See [PROTOCOL.md](PROTOCOL.m| `author_name` | the author's name (an organization may go in `author_org`) || `task_type` | `trajectory` · `policy` || `mode` | `privileged` · `standard` (formerly `unlimited` / `limited`) · `standard-realtime` (protocol v1.1; named so since v1.1.1, PROTOCOL §4b) |-| `protocol` | standard and standard-realtime mode only. Standard-realtime: `eai-standard/2.3`. Standard: `eai-standard/2.1` · `eai-standard/2.2` (BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, MuJoCo Playground's quadruped tasks, YAMLab, RoboDojo, LabUtopia, AutoBio) · `eai-standard/2.4` (the operations capability: RoboTangle). All are specified in [STANDARD_MODE.md](STANDARD_MODE.md) |-| `simulator` | `robosuite` · `robotwin` (RoboTwin 2.0 = SAPIEN 3 + CuRobo) · `mujoco` · `kinder` (KinDER: MuJoCo and PyBullet) · `humanoidbench` (HumanoidBench: MuJoCo + dm_control) · `robowits` (RoboWits on Genesis) · `robolab` (NVIDIA RoboLab on Isaac Lab) · `labutopia` (LabUtopia on Isaac Sim) · `behavior` (BEHAVIOR-1K: OmniGibson on Isaac Sim) · `maniskill` (ManiSkill 3 / SAPIEN: RoboPaint) · `vlabench` (VLABench: MuJoCo + dm_control) · `maniguard` (ManiGuard: BEHAVIOR-1K 3.7.2, OmniGibson on Isaac Sim 4.5) · `molmospaces` (Ai2 MolmoSpaces: MuJoCo) · `myosuite` (MyoSuite: musculoskeletal models on MuJoCo) · `dexjoco` (DexJoCo: MuJoCo 3.4 + its gymnasium envs) · `bigym2` (BiGym 2.0: MuJoCo 3.8.1 with NVIDIA's GR00T-WBC lower-body policy in every step) · `robotangle` (RoboTangle, formerly TopoGym: Genesis 1.2.1 rigid bodies and libuipc IPC / PBD deformables, Nyx RGB) · `yamlab` (YAMLab on Isaac Lab / Isaac Sim 5.1, PhysX on the CPU) · `robodojo` (RoboDojo's simulation benchmark: its Isaac Lab 2.3 fork on Isaac Sim 5.1, PhysX with GPU dynamics, cuRobo IK) · `autobio` (AutoBio: MuJoCo 3.3 with AutoBio's own thread, detent and liquid-surface mechanisms) — the simulator, not the image: benchmarks on one simulator may use different images and versions (PROTOCOL D5) |-| `robot` | `franka-panda` · `aloha-agilex` · `aloha-2` (ALOHA 2, MuJoCo Menagerie: MuJoCo Playground, RoboTangle) · `kuka-iiwa14-sharpa-ha4` · `tidybot-plus-plus` · `unitree-h1` · `unitree-h1hand` (H1 with two Shadow hands) · `sawyer` (Rethink Robotics Sawyer, as in Meta-World) · `marvin-bimanual` (RoboWits) · `franka-robotiq-2f85` (a Franka arm with a Robotiq 2F-85, the DROID setup: RoboLab, MolmoSpaces' FR3, RoboTangle's Panda) · `r1pro` (BEHAVIOR-1K) · `franka-panda-omron` (RoboCasa's PandaOmron: Franka Panda on an Omron mobile base with a lift torso) · `fourier-gr1` (Fourier GR1 humanoid: two arms, waist and two Fourier dexterous hands, fixed base; RoboCasa-GR1) · `franka-panda-longfinger` (ManiGuard: Franka Panda with long fin-ray fingers) · `rainbow-rby1` (Rainbow Robotics RB-Y1: holonomic base, 6-joint torso, two 7-joint arms with parallel grippers; MolmoSpaces) · `myohand` (MyoSuite's MyoHand: a musculoskeletal human forearm and hand, 39 muscles) · `myoarm` (MyoSuite's MyoArm: a musculoskeletal human shoulder, arm and hand, 34 or 63 muscles) · `myoarm-torso` (MyoSuite's table-tennis player: torso, right arm and hand on a sliding pelvis) · `myoarm-mpl` (MyoSuite's MyoArm next to the Modular Prosthetic Limb, a motorized arm) · `myolegs` (MyoSuite's MyoLegs: a free-standing musculoskeletal torso and both legs) · `myoleg-osl` (MyoSuite's MyoLeg with an Open-Source Leg knee-ankle prosthesis) · `unitree-go1` (Unitree Go1 quadruped; MuJoCo Playground) · `boston-dynamics-spot` (Boston Dynamics Spot quadruped; MuJoCo Playground) · `google-barkour-vb` (Google Barkour vB quadruped; MuJoCo Playground) · `franka-allegro` (Franka Panda with a 16-DoF Allegro hand; DexJoCo) · `franka-allegro-bimanual` (two of them side by side; DexJoCo) · `kuka-iiwa14` (KUKA LBR iiwa 14 with a tool, here an air hockey mallet; RoboOlympics) · `unitree-g1` (Unitree G1 29-DoF humanoid with its rubber hands, a switchable grip on each palm; RoboOlympics' robot library with Unitree's limits and motor curves; RoboFunOlympics) · `unitree-h1-2` (Unitree H1-2, the adult-size 27-DoF humanoid, with the same grip; RoboOlympics' robot library; RoboFunOlympics' tasks whose official venue needs an adult-size body) · `unitree-g1-dex1` (Unitree G1 29-DoF humanoid with Dex1-1 two-finger grippers; legs and waist driven by GR00T-WBC; BiGym 2.0) · `i2rt-yam-bimanual` (two I2RT YAM arms, 6 joints each, with parallel two-finger grippers on a workstation table; YAMLab) · `arx-x5-bimanual` (two ARX X5 arms, 6 joints each, with parallel two-finger grippers at a table; RoboDojo) · `aloha-2-arm` (one arm of ALOHA 2 with its parallel gripper, fixed to a bench; AutoBio, whose two-arm tasks use `aloha-2`) · `ur5e-robotiq-2f85` (a UR5e with a Robotiq 2F-85; AutoBio) · `ur5e-dexhand-2f85-bimanual` (two UR5e arms, one with a DexHand021 dexterous hand that holds a pipette, one with a Robotiq 2F-85; AutoBio) |+| `protocol` | standard and standard-realtime mode only. Standard-realtime: `eai-standard/2.3`. Standard: `eai-standard/2.1` · `eai-standard/2.2` (BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, MuJoCo Playground's quadruped tasks, YAMLab, RoboDojo, LabUtopia, AutoBio, Panda Dynamic Scenes) · `eai-standard/2.4` (the operations capability: RoboTangle). All are specified in [STANDARD_MODE.md](STANDARD_MODE.md) |+| `simulator` | `robosuite` · `robotwin` (RoboTwin 2.0 = SAPIEN 3 + CuRobo) · `mujoco` · `kinder` (KinDER: MuJoCo and PyBullet) · `humanoidbench` (HumanoidBench: MuJoCo + dm_control) · `robowits` (RoboWits on Genesis) · `robolab` (NVIDIA RoboLab on Isaac Lab) · `labutopia` (LabUtopia on Isaac Sim) · `behavior` (BEHAVIOR-1K: OmniGibson on Isaac Sim) · `maniskill` (ManiSkill 3 / SAPIEN: RoboPaint; Panda Dynamic Scenes, ManiSkill 3.0.1 / SAPIEN 3.0.3 with CPU PhysX at 1000 Hz) · `vlabench` (VLABench: MuJoCo + dm_control) · `maniguard` (ManiGuard: BEHAVIOR-1K 3.7.2, OmniGibson on Isaac Sim 4.5) · `molmospaces` (Ai2 MolmoSpaces: MuJoCo) · `myosuite` (MyoSuite: musculoskeletal models on MuJoCo) · `dexjoco` (DexJoCo: MuJoCo 3.4 + its gymnasium envs) · `bigym2` (BiGym 2.0: MuJoCo 3.8.1 with NVIDIA's GR00T-WBC lower-body policy in every step) · `robotangle` (RoboTangle, formerly TopoGym: Genesis 1.2.1 rigid bodies and libuipc IPC / PBD deformables, Nyx RGB) · `yamlab` (YAMLab on Isaac Lab / Isaac Sim 5.1, PhysX on the CPU) · `robodojo` (RoboDojo's simulation benchmark: its Isaac Lab 2.3 fork on Isaac Sim 5.1, PhysX with GPU dynamics, cuRobo IK) · `autobio` (AutoBio: MuJoCo 3.3 with AutoBio's own thread, detent and liquid-surface mechanisms) — the simulator, not the image: benchmarks on one simulator may use different images and versions (PROTOCOL D5) |+| `robot` | `franka-panda` · `aloha-agilex` · `aloha-2` (ALOHA 2, MuJoCo Menagerie: MuJoCo Playground, RoboTangle) · `kuka-iiwa14-sharpa-ha4` · `tidybot-plus-plus` · `unitree-h1` · `unitree-h1hand` (H1 with two Shadow hands) · `sawyer` (Rethink Robotics Sawyer, as in Meta-World) · `marvin-bimanual` (RoboWits) · `franka-robotiq-2f85` (a Franka arm with a Robotiq 2F-85, the DROID setup: RoboLab, MolmoSpaces' FR3, RoboTangle's Panda) · `r1pro` (BEHAVIOR-1K) · `franka-panda-omron` (RoboCasa's PandaOmron: Franka Panda on an Omron mobile base with a lift torso) · `fourier-gr1` (Fourier GR1 humanoid: two arms, waist and two Fourier dexterous hands, fixed base; RoboCasa-GR1) · `franka-panda-longfinger` (ManiGuard: Franka Panda with long fin-ray fingers) · `rainbow-rby1` (Rainbow Robotics RB-Y1: holonomic base, 6-joint torso, two 7-joint arms with parallel grippers; MolmoSpaces) · `myohand` (MyoSuite's MyoHand: a musculoskeletal human forearm and hand, 39 muscles) · `myoarm` (MyoSuite's MyoArm: a musculoskeletal human shoulder, arm and hand, 34 or 63 muscles) · `myoarm-torso` (MyoSuite's table-tennis player: torso, right arm and hand on a sliding pelvis) · `myoarm-mpl` (MyoSuite's MyoArm next to the Modular Prosthetic Limb, a motorized arm) · `myolegs` (MyoSuite's MyoLegs: a free-standing musculoskeletal torso and both legs) · `myoleg-osl` (MyoSuite's MyoLeg with an Open-Source Leg knee-ankle prosthesis) · `unitree-go1` (Unitree Go1 quadruped; MuJoCo Playground) · `boston-dynamics-spot` (Boston Dynamics Spot quadruped; MuJoCo Playground) · `google-barkour-vb` (Google Barkour vB quadruped; MuJoCo Playground) · `franka-allegro` (Franka Panda with a 16-DoF Allegro hand; DexJoCo) · `franka-allegro-bimanual` (two of them side by side; DexJoCo) · `franka-panda-bimanual` (two stock Franka Panda (FER) arms with their standard hands, facing each other or side by side at one table, each with a rigid task tool on its hand where the task has one; Panda Dynamic Scenes) · `kuka-iiwa14` (KUKA LBR iiwa 14 with a tool, here an air hockey mallet; RoboOlympics) · `unitree-g1` (Unitree G1 29-DoF humanoid with its rubber hands, a switchable grip on each palm; RoboOlympics' robot library with Unitree's limits and motor curves; RoboFunOlympics) · `unitree-h1-2` (Unitree H1-2, the adult-size 27-DoF humanoid, with the same grip; RoboOlympics' robot library; RoboFunOlympics' tasks whose official venue needs an adult-size body) · `unitree-g1-dex1` (Unitree G1 29-DoF humanoid with Dex1-1 two-finger grippers; legs and waist driven by GR00T-WBC; BiGym 2.0) · `i2rt-yam-bimanual` (two I2RT YAM arms, 6 joints each, with parallel two-finger grippers on a workstation table; YAMLab) · `arx-x5-bimanual` (two ARX X5 arms, 6 joints each, with parallel two-finger grippers at a table; RoboDojo) · `aloha-2-arm` (one arm of ALOHA 2 with its parallel gripper, fixed to a bench; AutoBio, whose two-arm tasks use `aloha-2`) · `ur5e-robotiq-2f85` (a UR5e with a Robotiq 2F-85; AutoBio) · `ur5e-dexhand-2f85-bimanual` (two UR5e arms, one with a DexHand021 dexterous hand that holds a pipette, one with a Robotiq 2F-85; AutoBio) || `family` | free-form slug naming the scene/goal, shared by its instances (e.g. `stack_blocks_two`, `dextoolbench/brush/blue_brush/sweep_forward`) || `oracle` | `full` (oracle scores 1) · `partial` (oracle reaches part of the goal) · `none` — see PROTOCOL §2.7 || `oracle_evidence` | required unless `oracle = "full"`: where the evidence for solvability is (PR, results site, upstream record) |diff --git a/scripts/task_conventions.py b/scripts/task_conventions.pyindex dbf65e2def..e0d42efff1 100644--- a/scripts/task_conventions.py+++ b/scripts/task_conventions.py@@ -73,6 +73,7 @@ BENCHMARKS = [ # (directory prefix, label)("yamlab-", "YAMLab"),("robodojo-", "RoboDojo"),("robofunolympics-", "RoboFunOlympics"),+ ("pandadyn-", "Panda Dynamic Scenes"),]# Documented exceptions: tasks without a privileged twin (the same lists: ci_checks/check-instance-copies.sh NO_TWIN and# STANDARD_ONLY). RoboOlympics (PROTOCOL §4b, owner 2026-10-08): standard-realtime only. RoboDojo's six tasks whose physicsdiff --git a/scripts/task_resources.toml b/scripts/task_resources.tomlindex cf6aa365f6..7434348282 100644--- a/scripts/task_resources.toml+++ b/scripts/task_resources.toml@@ -820,6 +820,34 @@ verifier_min_memory_gb = 3.0verifier_min_gpu_memory_gb = 0.0measured = "author estimate (ws-1, 2026-10-11): CPU-only; the sim sidecar steps and renders on one or two cores (under 1 GB), the agent's container holds the harness and the client"+# pandadyn, privileged: 25 task(s)+[[rule]]+match = "^pandadyn-.*-privileged$"+min_cpus = 2.0+min_memory_gb = 4.0+min_gpu_memory_gb = 1.0+gpu_required = true+gpu_class = "any"+recommended_gpu = "any 8 GB+ NVIDIA GPU with Vulkan (RTX 4090 / L4)"+verifier_min_cpus = 2.0+verifier_min_memory_gb = 3.0+verifier_min_gpu_memory_gb = 1.0+measured = "50 oracle trials, 25 per mode (rcb_monitor samples every 5 s, eai-pandadyn:0.1.0, RTX 4090, 2026-10-13): privileged trial peak 0.79 GiB (anon + shmem), GPU 0.05 GiB; standard sim service peak 0.94 GiB, GPU 0.35 GiB, agent container 0.03 GiB; CPU p90 about 1 core per container; one filmed scene (render_video, S12 and D07): RSS 1.2-1.3 GiB, GPU 0.25 GiB; CPU: one core per scene (CPU PhysX, single-threaded) + one for the agent"++# pandadyn, standard: 25 task(s)+[[rule]]+match = "^pandadyn-.*-standard$"+min_cpus = 2.0+min_memory_gb = 4.0+min_gpu_memory_gb = 1.0+gpu_required = true+gpu_class = "any"+recommended_gpu = "any 8 GB+ NVIDIA GPU with Vulkan (RTX 4090 / L4)"+verifier_min_cpus = 2.0+verifier_min_memory_gb = 3.0+verifier_min_gpu_memory_gb = 1.0+measured = "50 oracle trials, 25 per mode (rcb_monitor samples every 5 s, eai-pandadyn:0.1.0, RTX 4090, 2026-10-13): privileged trial peak 0.79 GiB (anon + shmem), GPU 0.05 GiB; standard sim service peak 0.94 GiB, GPU 0.35 GiB, agent container 0.03 GiB; CPU p90 about 1 core per container; one filmed scene (render_video, S12 and D07): RSS 1.2-1.3 GiB, GPU 0.25 GiB; the sim service is one scene, the collect hook replays in fresh processes one after the other; CPU: one core per scene + one for the agent"+[[limit]]match = "^robofunolympics-.*-privileged$"scope = "trial"@@ -1752,6 +1780,36 @@ trial_max_gpu_memory_gb = 16.0trial_max_memory_gb = 27.0measured = "GPU: 10.4 GiB on a 48 GB card, 6.5 on 24 GB (batch_eval_live vram_profile (per-trial GPU peaks by card, 10-03..10)) -> max(1.5 x peak, peak + 4), at least 16 for Isaac Sim; memory: agent container docker peak 1.5 GiB over 1 samples (mem_guard (docker memory incl. page cache, top-4 containers every 10 min, ws-1, 10-02..10)) -> max(1.5 x peak, peak + 4, 8), at most the 24 GiB cap; whole trial: + the sim service's 19 GiB (docker peak 12.4 GiB, 1 samples); processes 512 and disk 20 GiB: design defaults, not yet measured"+[[limit]]+match = "^pandadyn-.*-privileged$"+scope = "trial"+max_gpu_memory_gb = 5.0+max_memory_gb = 16.0+max_processes = 512+max_disk_gb = 10.0+disk_paths = ["/app", "/tmp", "/root"]+disk_exclude = []+sustain_sec = 60.0+window_sec = 120.0+trial_max_gpu_memory_gb = 5.0+trial_max_memory_gb = 16.0+measured = "GPU: 0.25 GiB per scene -> max(1.5 x peak, peak + 4) with room for a few scenes at once; memory: 1.3 GiB per scene, 16 GiB lets an agent run about ten scenes in parallel (50 oracle trials, 25 per mode (rcb_monitor samples every 5 s, eai-pandadyn:0.1.0, RTX 4090, 2026-10-13): privileged trial peak 0.79 GiB (anon + shmem), GPU 0.05 GiB; standard sim service peak 0.94 GiB, GPU 0.35 GiB, agent container 0.03 GiB; CPU p90 about 1 core per container; one filmed scene (render_video, S12 and D07): RSS 1.2-1.3 GiB, GPU 0.25 GiB); processes 512 and disk 10 GiB: design defaults, not yet measured"++[[limit]]+match = "^pandadyn-.*-standard$"+scope = "agent"+max_gpu_memory_gb = 0.0+max_memory_gb = 8.0+max_processes = 512+max_disk_gb = 10.0+disk_paths = ["/app", "/tmp", "/root"]+disk_exclude = []+sustain_sec = 60.0+window_sec = 120.0+trial_max_gpu_memory_gb = 4.0+trial_max_memory_gb = 14.0+measured = "memory: the agent container holds only Python and the client (0.03 GiB in the oracle trials): max(1.5 x peak, peak + 4, 8) -> 8 GiB; whole trial: + the sim service's 6 GiB (one scene, 1.3 GiB, with its frames and the replays' scenes); GPU 0.25 GiB in the sim service (50 oracle trials, 25 per mode (rcb_monitor samples every 5 s, eai-pandadyn:0.1.0, RTX 4090, 2026-10-13): privileged trial peak 0.79 GiB (anon + shmem), GPU 0.05 GiB; standard sim service peak 0.94 GiB, GPU 0.35 GiB, agent container 0.03 GiB; CPU p90 about 1 core per container; one filmed scene (render_video, S12 and D07): RSS 1.2-1.3 GiB, GPU 0.25 GiB); processes 512 and disk 10 GiB: design defaults, not yet measured"+[[cap]]match = "^autobio-.*-privileged$"main = { cpus = 4.0, memory_gb = 12.0, pids = 8192 }@@ -2027,3 +2085,12 @@ main = { cpus = 8.0, memory_gb = 16.0, pids = 8192 }match = "^robofunolympics-.*-standard$"main = { cpus = 4.0, memory_gb = 10.0, pids = 8192 }sim = { cpus = 4.0, memory_gb = 8.0, pids = 8192 }++[[cap]]+match = "^pandadyn-.*-privileged$"+main = { cpus = 8.0, memory_gb = 20.0, pids = 8192 }++[[cap]]+match = "^pandadyn-.*-standard$"+main = { cpus = 4.0, memory_gb = 10.0, pids = 8192 }+sim = { cpus = 4.0, memory_gb = 8.0, pids = 8192 }