The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
BiGym 2.0
Household loco-manipulation on a walking humanoid: a Unitree G1 with two parallel grippers, kept balanced and walking by NVIDIA's GR00T-WBC lower-body policy inside every step. The agent commands what a real G1 running that controller takes: walking velocity, pelvis height, torso lean, arm joints, grippers. No benchmark in the set has this: HumanoidBench controls an H1 joint by joint, RoboCasa-GR1's GR1 has a fixed base, and the kitchen tasks of RoboCasa and BEHAVIOR-1K run on wheeled bases.
Quick start
harbor run -d embodied-first-exam@all -p tasks/bigym2 -a codex -m openai/gpt-6-astra
Every image is public: Harbor pulls it.
| upstream | BiGym 2.0 (Imperial College London, S. James; arXiv 2610.07594), bigym 1.0.0 at 93ad4b3 (first published as 6a6632b; upstream rewrote its history, the package is unchanged and the image checks its git tree); a fork of BiGym (CoRL 2024) |
| licences | code Apache-2.0; G1 model BSD-3; props CC0 / CC BY 4.0; GR00T-WBC weights NVIDIA Open Model License (redistributable): the image can be published |
| simulator | MuJoCo 3.8.1 (pinned <3.9, Newton solver), CPU physics; GR00T-WBC's two ONNX policies on onnxruntime (CPU); EGL rendering; Python 3.12 |
| robot | Unitree G1 29-DoF with Dex1-1 two-finger grippers (no palm); legs and waist belong to the controller |
| action (50 Hz) | vx, vy (±1 m/s), pelvis height (0.4–1.0 m), wz (±1 rad/s), torso pitch (−0.2–0.8 rad; 11 of the 13 tasks), 14 arm joint targets (rad), 2 grippers (0 open – 1 closed): 20 or 21 floats |
| sensors | three RGB cameras on the robot (head, left_wrist, right_wrist, 60° fovy; upstream renders 84×84); arm and finger (+ waist) joints, gripper states, pelvis x, y, z, yaw from the base controller |
| score | upstream's predicate held for 1 s (50 steps) and no fall during the episode (pelvis tilt > ~53° or height < 0.30 m); fail-fast: an object on the floor (plates, blocks, dishwasher loads), a gripper on the sandwich |
| horizon | per task, 700–8750 control steps (14–175 s): twice the longest of 60 human demonstrations |
| tasks | 13 families × 3 modes: tasks/bigym2/bigym2-<family>-i00-privileged, -standard (eai-standard/2.1) and -realtime (eai-standard/2.3) |
| image | ghcr.io/mll-lab-nu/efe:bigym2-0.1.1 (images/bigym2/, 1.5 GB; dependencies at upstream's uv.lock, plus imageio-ffmpeg for the service's mp4 videos) |
| oracle | none: upstream has no expert or planner; human review |
Upstream ships its own coding-agent protocol (bigym-agent: one policy.py written from a task sentence and a
demonstration video, scored on 100 hidden seeds at 84×84). Here the EFE protocol applies: one frozen instance per task,
no demonstration.
Tasks: upstream's 20, ours ✅ (13)
| group | task | the robot must | horizon | ours | note |
|---|---|---|---|---|---|
| reaching | reach_target_single | touch a ball with the left gripper | 18 s | ⬜ | reach_target_dual's predicate, one hand |
| reach_target_multi_modal | touch a ball with either gripper | 14 s | ⬜ | the same | |
| reach_target_dual | touch two balls at once, one per gripper | 14 s | ✅ | ||
| table-top | move_plate | move a plate between two draining racks | 34 s | ⬜ | move_two_plates' predicate, one plate |
| move_two_plates | move two plates at once: in a slot, upright within 20°, released | 46 s | ✅ | ||
| flip_cup | turn an upside-down cup upright (within 5°) | 37 s | ✅ | ||
| flip_cutlery | take a spoon out of a mug, flip it, put it back | 39 s | ✅ | ||
| stack_blocks | carry three blocks to the far table and stack them | 175 s | ✅ | walks across | |
| dishwasher | dishwasher_close | push both trays in and close the door | 93 s | ✅ | three joints |
| dishwasher_load_cups | two cups into the upper tray | 40 s | ⬜ | load_plates' scene and predicate, no orientation clause | |
| dishwasher_load_cutlery | a knife and a fork from a mug into the basket | 53 s | ⬜ | the same | |
| dishwasher_load_plates | two plates from a drainer into the lower tray, upright | 63 s | ✅ | ||
| kitchen counter | drawer_top_open | open the top drawer | 27 s | ✅ | grasp and pull |
| drawer_top_close | close the top drawer | 17 s | ⬜ | the closing twin: a push does it | |
| wall_cupboard_open | open both doors of the wall cabinet | 29 s | ✅ | ||
| wall_cupboard_close | close both doors | 34 s | ⬜ | the closing twin | |
| pick_box | carry a 3 kg box from a low side table to the counter | 88 s | ✅ | squat; the box is squeezed between the grippers | |
| put_cups | two cups into the closed wall cabinet | 60 s | ✅ | opens the glass door first | |
| saucepan_to_hob | the saucepan from the closed cabinet onto the hob | 93 s | ✅ | opens the door, squats | |
| sandwich_remove | move the sandwich from the pan onto the board with a spatula | 63 s | ✅ | tool; a gripper on the sandwich fails |
Why these (no model's results enter the choice):
- Valid (20 of 20): holding the start pose for the whole horizon scores 0 and does not fall; replay is deterministic (below); every goal is an object state a sentence names. At t = 0 the head camera looks down at the hands: the balls, racks and blocks are in view, the cups, the box and the wall cabinet only after the robot leans, turns or points a wrist camera (upstream has no outside camera).
- One per success predicate (13): a task whose predicate is another's on fewer objects or one hand goes; of an open/close pair the opening twin stays (a grasp and a pull; the closing twin yields to a push); of the three dishwasher loads (one scene, one "in the rack, released" predicate) the plates stay, the only one with an orientation clause.
- Instance: seed 0. No published demonstration uses it (their seeds are 1–199), and it is outside upstream's reserved evaluation range 620000–629999, whose rollouts upstream publishes.
Modes
One module, rcb_bigym2.py, in every task's environment/ and tests/: the instance is upstream's official
make(task) reset with seed 0 (its deterministic reset and 200-step controller warm-up), one control step is one
upstream env step of a physical action row (upstream's EnvTools contract), success is upstream's is_success. A
load accepts the instance when the state matches bit for bit, or when the scene matches and every joint is within
5e-4 (upstream's golden-test tolerance for CPU kernel differences in the controller): replays on one host stay
bit-exact among themselves either way. "Held" in every text is upstream's is_gripper_holding_object: a finger
touching the object. On the G1, upstream's pinch_position is the wrist link's origin, 0.148 m behind the
finger pads; the point between the pads is the *_end_effector site (wrist_position, what tools.ik and the reach
check use), as the privileged instruction says.
| mode | the agent gets | graded |
|---|---|---|
| privileged | the bigym package (Session: step, reset, replay, MuJoCo model and data, onboard images) |
/app/output/trajectory.npz, replayed in two fresh processes |
| standard | the robot's three onboard cameras (84×84 as upstream renders them; RCB_IMAGE_HW for others, reported apart), upstream's proprioception, whole_body chunks; one episode, no reset |
the live episode, as it ran (rcb_service.py, as HumanoidBench and KinDER; nothing replayed) |
| realtime | the same on a clock that does not stop; between commands the robot stands still on the last one | as standard (rcb_realtime.py, as HumanoidBench) |
Every instruction states, besides upstream's task sentence and success rule: where the named objects are at the start
(one sentence measured on the frozen scene: distance, direction, height; upstream gives its own agents a demonstration
video instead, which a task here cannot ship), the action table, and upstream's own notes on the whole-body controller
(bigym/loco/agent/templates/api_strict.md). Standard spec() adds the robot's description: joint axes and positions,
the point between each gripper's pads, camera mounts. Reset budget: 0 in both service modes (the benchmark's choice, PROTOCOL §2): one episode, whose horizon the practice
and the graded attempt share. (The robot itself does not fall: 15 adversarial command patterns of 12 s each, no fall.)
RCB_RESET_BUDGET=N runs a configuration with N resets, reported separately.
Grading in both service modes: the live episode. The sim sidecar judges the episode it runs with upstream's
is_success at every control step (step() in each standard task's environment/sim/server.py) and the episode ends where
upstream ends it; rcb_service.py writes that verdict (graded_by = "live", replays = 0) once the agent's container
is stopped. The 1 s hold is upstream's, inside the episode; nothing is added after it. The frozen start is checked on
the live simulator when the scene is built (instance_check: the instance's t0 state bit for bit, or within
upstream's tolerance); a scene that does not rebuild it is instance_mismatch, and start_sha (that accepted start)
is checked when an episode begins; both give up as infrastructure (infra_error = 1) before the agent is served, and
the service gives up the same way at once if a reset does not return to start_sha (untested: BiGym 2.0 grants no
reset by default; upstream's reset with the seed and its warm-up is deterministic on one host, which
RCB_RESET_BUDGET=N runs rely on). The video is one render pass over trajectory.npz after the verdict
(render_video: rcb_bigym2.replay with the recorder; CPU physics and controller, EGL rendering; about a minute a
trial at the median); it never changes the verdict. final.json is finalized the moment the verdict is written,
before the video is rendered (video_pending while it renders); the collect hook waits for the verdict and then, as
long as its timeout allows, for the video, so a slow, hung or killed render costs the video and never the verdict.
reward.json carries graded_by_live, replays and sim_error.
infra_error = 1 only for no or an unfinalized final.json, a give-up after crashes not all inside the agent's calls,
a reset whose scene does not rebuild and instance_mismatch; simulator errors raised by the agent's calls, crashes
inside its calls, timeouts and failures are the agent's 0. In realtime: once the agent's first command has
run, a simulator error raised by a control step is the agent's 0 (sim_error = 1, no infra_error), and so is a
simulator crash inside a control step (a death whose fatal-signal record in crash.log -- SIGSEGV, SIGBUS, SIGFPE,
SIGILL, SIGABRT -- shows the crashing thread in the step; past RCB_MAX_RESUMES such crashes the trial ends
sim_crashed_on_call); a crash outside a step (building the scene, resuming, resetting, rendering, finalizing),
anything before the first command, a failed reset rebuild, a kill (SIGKILL, SIGTERM, the OOM killer) and a failed
resume are infrastructure. Standard mode keeps its rule: a crash is the agent's only inside its call (fatal_signal is
logged as evidence). A simulator that fails inside POST /reset ends the episode error (infra_error = 1: the
agent's reset was legitimate), not reset. A negative Content-Length is refused unread (400).
Realtime (the shared 2.3 layer): a gap before at is the plan's last row, walking velocity included; the
agent's requests cannot slow the clock -- bodies are read, decoded and checked off the simulator's thread, waiting
requests are served once the clock has caught up, the cameras are rendered once per control step
(docs/protocol/standard_mode.md, "What a request can cost the clock") -- and final.json / reward.json carry the clock's
record: clock_s, max_lag_s, stalls, stalled_s, stalled_share, held_steps, late_steps, requests_refused,
request_busy_s. They are evidence and decide no verdict.
The success check stays visible in privileged mode: it is sparse, and every instruction
states its thresholds. Upstream's reward shaping for the plate tasks (event_progress) is off by default and never
graded.
Where upstream holds answers
| where | what | handling |
|---|---|---|
bigym/envs/*.py (_success, _fail), bigym/loco/eval/ |
predicates, tolerances, fall check | privileged: kept; standard: sidecar only |
HF SWIRL-Lab/bigym-g1-native60 (public) |
60 successful demonstrations per task with full simulator state | seed 0 is none of them; the dataset is not in the image; the agent reaches the model APIs and the package registries only |
HF SWIRL-Lab/bigym2-agent-rollouts (public) + the project page |
two coding agents' programs (policy.py, on the page) and their 5,400 rollouts on seeds 620000–620099, for 9 tasks (5 of ours: reach_target_dual, move_two_plates, dishwasher_close, drawer_top_open, pick_box) |
a program that generalises over seeds solves seed 0 too: only the egress rule keeps it out |
tests/fixtures/demo_endpoints.npz |
the final state of one demonstration per task | not shipped: the wheel installs bigym/ only |
dishwasher_close, wall_cupboard_open |
no reset randomisation: every demonstration starts from this scene | covered by the dataset row |
There is no scripted expert or planner upstream; IK exists only as teleoperation and agent tooling (bigym/vr/ik/,
tools.ik in bigym/loco/agent/).
Upstream behaviour kept as it is
sandwich_remove's check does not ask that the spatula was used or that the sandwich left the pan: a sandwich lying flat on the board passes (a finger on the sandwich fails the episode).- The reach tasks light a sphere up when the success check finds a gripper inside it, in every camera.
put_cups's check asks only that each cup touch the wall cabinet's bottom shelf and not be held: contact from below the shelf, outside the closed cabinet, would count.flip_cutlery's check asks neither that the spoon end inside the mug nor that the mug stay upright. These notes stay in this page: no task file the agent sees mentions them.
Build and run
docker build -f images/bigym2/Dockerfile -t ghcr.io/mll-lab-nu/efe:bigym2-$(cat images/bigym2/VERSION) .
harbor run -p tasks/bigym2/bigym2-flip-cup-i00-standard -a codex -m openai/gpt-6-astra --env-file .env # agent settings: docs/protocol/running.md
The frozen instances live in the task dirs only (tasks/bigym2/bigym2-<family>-i00-privileged/environment/instance.json).
The task directories are the deliverable; the tooling that built them is kept outside this repository.
Measured (RTX 4090, host load ~25 on 32 cores)
- Every task builds in ~1 s; a reset (with the controller's 200-step warm-up) takes under 1 s; 5–12 ms per control step with three 84×84 cameras (1.7–4× real time). Install: a 0.7 GB environment; the GPU only renders.
- Holding the start pose for the whole horizon: 0 of 20 succeed, none falls (seed 620000; the ✅ tasks also at seed 0).
- 300 steps of a fixed arm-and-walk motion in two fresh processes: identical MuJoCo state (
qpos,qvel,act,ctrl) on 20 of 20 (the ✅ tasks also at seed 0); a reset in the same process and a fresh env in the same process land on the same state as well (the tooling's hash adds the controller's state and the props' poses). - The image rebuilds every instance's t = 0 hash offline; no layer of it holds a file of the upstream repository
outside the package.
nopscores 0 in all three modes. - Other CPUs (on the 7950X): every instance rebuilds bit for bit with OpenBLAS forced to its Prescott, Haswell or Sapphire Rapids kernels, with numpy's AVX-512 loops off, and on 2 cores. onnxruntime picks its kernels by CPU and has no switch: rounding-level noise (±2e-7 relative) on the controller's outputs moves the t = 0 joints by at most 2.2e-7, far inside 5e-4. Not yet checked on an Intel or AVX2-only host.