The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Shared: simulator `bigym2`, robot `unitree-g1-dex1`, the BiGym 2.0 README row, label and checks (for #121) (#122)
Pingyue Zhang20cf70aa6502
diff --git a/README.md b/README.mdindex 4782fd1bdf..3cd78b9f87 100644--- a/README.md+++ b/README.md@@ -28,7 +28,7 @@ measures the same thing under real-robot constraints. The standard-mode wire proA third mode, **standard-realtime** (protocol v1.1, `eai-standard/2.3`, PROTOCOL §4b), is standard mode on a clock that does notstop: once an episode starts, the robot runs in real time and holds its last targets while the agent thinks, so the agent-drives it from a controller program of its own. It is offered where a benchmark's tasks call for it (HumanoidBench, RoboOlympics), as+drives it from a controller program of its own. It is offered where a benchmark's tasks call for it (HumanoidBench, RoboOlympics, BiGym 2.0), as`<base>-standard-realtime` next to `<base>-privileged` and `<base>-standard`; its results are reported separately.> **Names.** Protocol v1.0 renamed *unlimited/limited* to *privileged/standard* and `rcb-limited/2.1` to@@ -56,6 +56,7 @@ drives it from a controller program of its own. It is offered where a benchmark'| DexJoCo (`dexjoco-<family>-i00`) | MuJoCo 3.4 · Franka Panda + Allegro hand, one or two arms | 11 | full (upstream human demonstrations) || MyoSuite (`myosuite-<family>-i00`) | MyoSuite 2.12.2 (MuJoCo 3.6, CPU) · musculoskeletal hand, arm, torso and legs (MyoHand, MyoArm, MyoLegs, with prostheses in two tasks), muscle commands; MyoChallenge and MyoDex tasks | 22 | full on 3 (MyoSuite's NPG baselines), none on 19 (human review) || RoboTangle (`robotangle-<family>-<tier>-i00`; formerly TopoGym) | Genesis 1.2.1 (rigid bodies on CPU float64 or GPU fp32, libuipc IPC and PBD deformables), Nyx RGB · ALOHA 2 bimanual; Franka Panda + Robotiq 2F-85 | 52 (11 families × up to 5 tiers) | full |+| BiGym 2.0 (`bigym2-<family>-i00`) | MuJoCo 3.8.1 + NVIDIA GR00T-WBC (CPU; EGL rendering) · Unitree G1 humanoid with Dex1-1 grippers, legs and waist driven by the whole-body controller | 13 (+ 13 standard-realtime) | none (human review) || RoboOlympics (`roboolympics-<sport>-<format>-i00`) | MuJoCo 3.3.7 (CPU, OSMesa) · Franka Panda with a paddle (table tennis) or with a racket on an x-y base (badminton); KUKA iiwa 14 with a mallet (air hockey) | 3, standard-realtime only (real-time sports against a scripted opponent; S = 10 and no privileged twin, PROTOCOL §4b exceptions) | full (RoboOlympics' reference players, camera images only) |Each task directory carries a README in one layout: a key table (benchmark, mode, twin, simulator and robot, family anddiff --git a/ci_checks/check-generated-tasks.sh b/ci_checks/check-generated-tasks.shindex 67afb8f557..12f07a6292 100755--- a/ci_checks/check-generated-tasks.sh+++ b/ci_checks/check-generated-tasks.sh@@ -28,4 +28,8 @@ if [ -f "$root/scripts/robotangle/mk_tasks.py" ]; then # RoboTangle (its tasksif out=$(python3 "$root/scripts/robotangle/mk_tasks.py" --check 2>&1); then echo "ok scripts/robotangle/mk_tasks.py: ${out##*$'\n'}"else echo "FAIL scripts/robotangle/mk_tasks.py --check:"; echo "$out" | tail -n 40; rc=1; fifi+if [ -f "$root/scripts/bigym2/gen_tasks.py" ]; then # BiGym 2.0 (its tasks pull request adds the generator)+ if out=$(python3 "$root/scripts/bigym2/gen_tasks.py" check 2>&1); then echo "ok scripts/bigym2/gen_tasks.py: ${out##*$'\n'}"+ else echo "FAIL scripts/bigym2/gen_tasks.py check:"; echo "$out" | tail -n 40; rc=1; fi+fiexit $rcdiff --git a/ci_checks/check-tooling-identity.sh b/ci_checks/check-tooling-identity.shindex 16d0537604..f9558380e6 100755--- a/ci_checks/check-tooling-identity.sh+++ b/ci_checks/check-tooling-identity.sh@@ -1,6 +1,6 @@#!/usr/bin/env bash# Shared-source rules (PROTOCOL §2.1, D9: copies of shared sources are byte-identical). Two parts:-# 1. BEHAVIOR-1K, RoboLab, RoboWits, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo and RoboTangle: within one benchmark, both modes, every copy of a shared file is byte-identical+# 1. BEHAVIOR-1K, RoboLab, RoboWits, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, RoboTangle and BiGym 2.0: within one benchmark, every mode, every copy of a shared file is byte-identical# (the generators copy them from the benchmark's template task unchanged):# - the tooling module rcb_<sim>.py wherever it sits (privileged environment/ and tests/, standard environment/sim/);# - the standard agent side environment/rcb_client.py + rcb1.py and the sim service environment/sim/ (server.py,@@ -22,7 +22,7 @@ import collections, hashlib, os, re, sys, tomllibtasks, src = sys.argv[1], sys.argv[2]MODULE = {"behavior": "rcb_behavior.py", "robolab": "rcb_robolab.py", "robowits": "rcb_robowits.py", "maniguard": "rcb_maniguard.py","molmospaces": "rcb_molmospaces.py", "myosuite": "rcb_myosuite.py",- "dexjoco": "rcb_dexjoco.py", "robotangle": "rcb_topogym.py"}+ "dexjoco": "rcb_dexjoco.py", "robotangle": "rcb_topogym.py", "bigym2": "rcb_bigym2.py"}SHARED = {"environment/rcb_client.py", "environment/rcb1.py", "environment/sim/server.py", "environment/sim/finalize.sh","environment/sim/run.sh", "environment/sim/rcb1.py", "environment/sim/rcb_limited.py", "tests/grade.py"}LAYER = {"environment/rcb_client.py": "rcb_client.py", "environment/rcb1.py": "rcb1.py", "environment/sim/rcb_limited.py": "rcb_limited.py",@@ -95,6 +95,6 @@ if bad:if len(bad) > 60:print(f" ... {len(bad) - 60} more")sys.exit(1)-print("ok shared sources: " + ("; ".join(f"{b} {dirs[b]} task dirs, {len(seen[b])} shared files" for b in sorted(seen)) or "no BEHAVIOR / RoboLab / RoboWits / ManiGuard / MolmoSpaces / MyoSuite / DexJoCo / RoboTangle tasks")+print("ok shared sources: " + ("; ".join(f"{b} {dirs[b]} task dirs, {len(seen[b])} shared files" for b in sorted(seen)) or "no BEHAVIOR / RoboLab / RoboWits / ManiGuard / MolmoSpaces / MyoSuite / DexJoCo / RoboTangle / BiGym 2.0 tasks")+ f"; the eai-standard/2.2 / 2.4 layers in {n_layer} task dirs")PYdiff --git a/docs/TAXONOMY.md b/docs/TAXONOMY.mdindex 76c9b3a7c7..dd29d28072 100644--- a/docs/TAXONOMY.md+++ b/docs/TAXONOMY.md@@ -8,8 +8,8 @@ Known values; a new value is added by pull request (CI validation of these vocab| `task_type` | `trajectory` · `policy` || `mode` | `privileged` · `standard` (formerly `unlimited` / `limited`) · `standard-realtime` (protocol v1.1; named so since v1.1.1, PROTOCOL §4b) || `protocol` | standard and standard-realtime mode only. Standard-realtime: `eai-standard/2.3`. Standard: `eai-standard/2.1` · `eai-standard/2.2` (BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, MuJoCo Playground's quadruped tasks; [STANDARD_MODE_2_2.md](STANDARD_MODE_2_2.md)) · `eai-standard/2.4` ([STANDARD_MODE_2_4.md](STANDARD_MODE_2_4.md), the operations capability: RoboTangle) |-| `simulator` | `robosuite` · `robotwin` (RoboTwin 2.0 = SAPIEN 3 + CuRobo) · `mujoco` · `kinder` (KinDER: MuJoCo and PyBullet) · `humanoidbench` (HumanoidBench: MuJoCo + dm_control) · `robowits` (RoboWits on Genesis) · `robolab` (NVIDIA RoboLab on Isaac Lab) · `behavior` (BEHAVIOR-1K: OmniGibson on Isaac Sim) · `maniskill` (ManiSkill 3 / SAPIEN: RoboPaint) · `vlabench` (VLABench: MuJoCo + dm_control) · `maniguard` (ManiGuard: BEHAVIOR-1K 3.7.2, OmniGibson on Isaac Sim 4.5) · `molmospaces` (Ai2 MolmoSpaces: MuJoCo) · `myosuite` (MyoSuite: musculoskeletal models on MuJoCo) · `dexjoco` (DexJoCo: MuJoCo 3.4 + its gymnasium envs) · `robotangle` (RoboTangle, formerly TopoGym: Genesis 1.2.1 rigid bodies and libuipc IPC / PBD deformables, Nyx RGB) — the simulator, not the image: benchmarks on one simulator may use different images and versions (PROTOCOL D5) |-| `robot` | `franka-panda` · `aloha-agilex` · `aloha-2` (ALOHA 2, MuJoCo Menagerie: MuJoCo Playground, RoboTangle) · `kuka-iiwa14-sharpa-ha4` · `tidybot-plus-plus` · `unitree-h1` · `unitree-h1hand` (H1 with two Shadow hands) · `sawyer` (Rethink Robotics Sawyer, as in Meta-World) · `marvin-bimanual` (RoboWits) · `franka-robotiq-2f85` (a Franka arm with a Robotiq 2F-85, the DROID setup: RoboLab, MolmoSpaces' FR3, RoboTangle's Panda) · `r1pro` (BEHAVIOR-1K) · `franka-panda-omron` (RoboCasa's PandaOmron: Franka Panda on an Omron mobile base with a lift torso) · `fourier-gr1` (Fourier GR1 humanoid: two arms, waist and two Fourier dexterous hands, fixed base; RoboCasa-GR1) · `franka-panda-longfinger` (ManiGuard: Franka Panda with long fin-ray fingers) · `rainbow-rby1` (Rainbow Robotics RB-Y1: holonomic base, 6-joint torso, two 7-joint arms with parallel grippers; MolmoSpaces) · `myohand` (MyoSuite's MyoHand: a musculoskeletal human forearm and hand, 39 muscles) · `myoarm` (MyoSuite's MyoArm: a musculoskeletal human shoulder, arm and hand, 34 or 63 muscles) · `myoarm-torso` (MyoSuite's table-tennis player: torso, right arm and hand on a sliding pelvis) · `myoarm-mpl` (MyoSuite's MyoArm next to the Modular Prosthetic Limb, a motorized arm) · `myolegs` (MyoSuite's MyoLegs: a free-standing musculoskeletal torso and both legs) · `myoleg-osl` (MyoSuite's MyoLeg with an Open-Source Leg knee-ankle prosthesis) · `unitree-go1` (Unitree Go1 quadruped; MuJoCo Playground) · `boston-dynamics-spot` (Boston Dynamics Spot quadruped; MuJoCo Playground) · `google-barkour-vb` (Google Barkour vB quadruped; MuJoCo Playground) · `franka-allegro` (Franka Panda with a 16-DoF Allegro hand; DexJoCo) · `franka-allegro-bimanual` (two of them side by side; DexJoCo) · `kuka-iiwa14` (KUKA LBR iiwa 14 with a tool, here an air hockey mallet; RoboOlympics) |+| `simulator` | `robosuite` · `robotwin` (RoboTwin 2.0 = SAPIEN 3 + CuRobo) · `mujoco` · `kinder` (KinDER: MuJoCo and PyBullet) · `humanoidbench` (HumanoidBench: MuJoCo + dm_control) · `robowits` (RoboWits on Genesis) · `robolab` (NVIDIA RoboLab on Isaac Lab) · `behavior` (BEHAVIOR-1K: OmniGibson on Isaac Sim) · `maniskill` (ManiSkill 3 / SAPIEN: RoboPaint) · `vlabench` (VLABench: MuJoCo + dm_control) · `maniguard` (ManiGuard: BEHAVIOR-1K 3.7.2, OmniGibson on Isaac Sim 4.5) · `molmospaces` (Ai2 MolmoSpaces: MuJoCo) · `myosuite` (MyoSuite: musculoskeletal models on MuJoCo) · `dexjoco` (DexJoCo: MuJoCo 3.4 + its gymnasium envs) · `bigym2` (BiGym 2.0: MuJoCo 3.8.1 with NVIDIA's GR00T-WBC lower-body policy in every step) · `robotangle` (RoboTangle, formerly TopoGym: Genesis 1.2.1 rigid bodies and libuipc IPC / PBD deformables, Nyx RGB) — the simulator, not the image: benchmarks on one simulator may use different images and versions (PROTOCOL D5) |+| `robot` | `franka-panda` · `aloha-agilex` · `aloha-2` (ALOHA 2, MuJoCo Menagerie: MuJoCo Playground, RoboTangle) · `kuka-iiwa14-sharpa-ha4` · `tidybot-plus-plus` · `unitree-h1` · `unitree-h1hand` (H1 with two Shadow hands) · `sawyer` (Rethink Robotics Sawyer, as in Meta-World) · `marvin-bimanual` (RoboWits) · `franka-robotiq-2f85` (a Franka arm with a Robotiq 2F-85, the DROID setup: RoboLab, MolmoSpaces' FR3, RoboTangle's Panda) · `r1pro` (BEHAVIOR-1K) · `franka-panda-omron` (RoboCasa's PandaOmron: Franka Panda on an Omron mobile base with a lift torso) · `fourier-gr1` (Fourier GR1 humanoid: two arms, waist and two Fourier dexterous hands, fixed base; RoboCasa-GR1) · `franka-panda-longfinger` (ManiGuard: Franka Panda with long fin-ray fingers) · `rainbow-rby1` (Rainbow Robotics RB-Y1: holonomic base, 6-joint torso, two 7-joint arms with parallel grippers; MolmoSpaces) · `myohand` (MyoSuite's MyoHand: a musculoskeletal human forearm and hand, 39 muscles) · `myoarm` (MyoSuite's MyoArm: a musculoskeletal human shoulder, arm and hand, 34 or 63 muscles) · `myoarm-torso` (MyoSuite's table-tennis player: torso, right arm and hand on a sliding pelvis) · `myoarm-mpl` (MyoSuite's MyoArm next to the Modular Prosthetic Limb, a motorized arm) · `myolegs` (MyoSuite's MyoLegs: a free-standing musculoskeletal torso and both legs) · `myoleg-osl` (MyoSuite's MyoLeg with an Open-Source Leg knee-ankle prosthesis) · `unitree-go1` (Unitree Go1 quadruped; MuJoCo Playground) · `boston-dynamics-spot` (Boston Dynamics Spot quadruped; MuJoCo Playground) · `google-barkour-vb` (Google Barkour vB quadruped; MuJoCo Playground) · `franka-allegro` (Franka Panda with a 16-DoF Allegro hand; DexJoCo) · `franka-allegro-bimanual` (two of them side by side; DexJoCo) · `kuka-iiwa14` (KUKA LBR iiwa 14 with a tool, here an air hockey mallet; RoboOlympics) · `unitree-g1-dex1` (Unitree G1 29-DoF humanoid with Dex1-1 two-finger grippers; legs and waist driven by GR00T-WBC; BiGym 2.0) || `family` | free-form slug naming the scene/goal, shared by its instances (e.g. `stack_blocks_two`, `dextoolbench/brush/blue_brush/sweep_forward`) || `oracle` | `full` (oracle scores 1) · `partial` (oracle reaches part of the goal) · `none` — see PROTOCOL §2.7 || `oracle_evidence` | required unless `oracle = "full"`: where the evidence for solvability is (PR, results site, upstream record) |diff --git a/scripts/task_conventions.py b/scripts/task_conventions.pyindex d5373222e9..39c89ed836 100644--- a/scripts/task_conventions.py+++ b/scripts/task_conventions.py@@ -53,6 +53,7 @@ BENCHMARKS = [ # (directory prefix, label)("libero-max-", "LIBERO-MAX"),("robotangle-", "RoboTangle"),("roboolympics-", "RoboOlympics"),+ ("bigym2-", "BiGym 2.0"),]# Documented per-benchmark exception (PROTOCOL §4b, owner 2026-10-08): standard-realtime tasks without a privileged twin# (the same list: ci_checks/check-instance-copies.sh NO_TWIN)