The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
ABC Sim
Twenty-nine tasks of ABC Sim (abc_sim at d0832d1, Apache-2.0; assets CC BY 4.0 /
MIT), the MuJoCo simulator of ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation: two I2RT YAM
arms (6 revolute joints and a parallel two-finger gripper each, the robot of YAMLab) side by side at a table, controlled
by joint-position targets. Every task the upstream catalogue scores is here, each as one frozen instance in two modes:
58 task directories tasks/abc-sim/abc-sim-<family>-i00-{privileged,standard}, and a realtime twin for the two
families whose scene moves by itself (abc-sim-conveyor-pick-i00-realtime, abc-sim-ball-tray-i00-realtime):
60 in all.
Quick start
harbor run -d embodied-first-exam@all -p tasks/abc-sim -a codex -m openai/gpt-6-astra
The tasks need an NVIDIA GPU (the cameras render over EGL; the physics runs on the CPU). The base image is
ghcr.io/mll-lab-nu/efe:abc-sim-0.2.0 (images/abc-sim/; the privileged tasks keep abc-sim-0.1.0, whose
simulator and scenes are the same), built from public sources.
Tasks
| family | upstream task | directive of the frozen instance | reference rows |
|---|---|---|---|
throw_bottles |
throw_plastic_bottles_in_bin |
throw plastic bottles in bin | 563 |
put_bottles |
put_plastic_bottles_in_bin |
put the plastic bottles in the bin | 476 |
conveyor_pick |
conveyor_pick |
pick objects from the conveyor belt and put them in the bin | 274 |
count |
count_into_opaque_box |
put all yellow triangles in the box | 277 |
count_one |
count_one_into_opaque_box |
put exactly 1 red object in the box | 264 |
count_two |
count_two_into_opaque_box |
put exactly 2 objects in the box | 291 |
count_three |
count_three_into_opaque_box |
put exactly 3 objects in the box | 538 |
count_four |
count_four_into_opaque_box |
put exactly 4 objects in the box | 671 |
put_relative |
put_relative |
put the soap dispenser to the right of the boxed food | 237 |
grab_clutter |
grab_specific_object_from_clutter |
grab the ketchup and place it in the box | 226 |
ball_tray |
ball_tray_balancing |
grab the tray handles and keep the ball balanced on the tray | 453 |
inhand_transfer |
inhand_transfer_item_to_other_side |
inhand transfer the item to other side (a cream cheese stick) | 374 |
drawer_search |
multi_drawer_search |
find the measuring cup, then the dish brush | 432 |
pouring |
pouring |
pour from one container into another | 307 |
hang_mug |
hang_mug_on_mug_rack |
hang mug on mug rack | 314 |
turn_mug |
turn_mug_right_side_up |
turn mug right side up (two mugs) | 561 |
sweep |
sweep_away_paper_scraps_from_table |
sweep away paper scraps from the table (two scraps) | 1288 |
nuts_bolts |
sort_nuts_and_bolts |
sort the nuts and bolts into their matching bins (9 nuts, 10 bolts) | 2872 |
lego |
sort_lego_blocks |
sort the red, yellow, and blue lego blocks into their matching bins (12 blocks) | 2205 |
dish_rack |
load_plates_into_dish_rack |
load plates into tabletop dish rack (one plate) | 375 |
chess |
set_up_chess_pieces_on_board |
set up chess pieces on the board (six black pieces on the table) | 1872 |
spell_cat, spell_dog, spell_fish, spell_bair, spell_xdof, spell_abc, spell_yam, spell_agi |
spell_<word> |
spell | 909, 580, 675, 1360, 719, 507, 581, 581 |
Left out: the four catalogue tasks upstream does not score (it has no success evaluator for them): the three
put_markers_in_*_drawer tasks and build_wood_block_tower. The count variants are one scene (upstream draws a new
counting directive at every reset whatever the variant asks); here each is a frozen instance with its own directive:
"exactly N objects" for N = 2, 3, 4, "exactly 1 red object" for one (upstream's randomizer never draws "exactly 1 object"
without a colour), and an "all of a colour and shape" directive for the generic one.
Task definition
- Instance. One frozen instance per family,
instance.npz(the privileged task'senvironment/andtests/, the standard twins'environment/sim/, byte-identical): the assembled scene XML (every object, its variant, size and pose already chosen), the full initialqposat rest, and the scene's randomization record (the directive and the per-task parameters the evaluator and the runtime read). The scene is the one of an upstream validation demonstration whose physics replay succeeded and was bit-identical in two fresh processes (spell_bair has no published demonstration: its scene is upstream's randomizer atreset(seed=243)). The record's seed is set to 0 except for the ball tray, whose runtime seeds its perturbations with it. Two families keep state on the model that the saved scene XML of the demonstrations lost (the randomizer writes it at run time): drawer search (the drawer stacks' exact poses; the collisions of the unused objects, parked below the table, are off) and chess (the tin's collisions are on only in the tin scenario); the instances write it into the scene XML, so the scene is complete in the file. Without it the demonstrations replay 0/20 (drawer search) and start numerically unstable (chess). The state hash (MuJoCo's integration state and the model's numeric arrays) is recorded and checked on every load. - Episode. It starts at the frozen instance. One control step = 17 physics steps of 2 ms = 0.034 s. Unused objects some scenes park below or far above the table (drawer search, chess, sweep) fall freely and touch nothing; they are upstream's.
- Success is upstream's evaluator for the task (
abc_sim/task_eval/, the catalogue'sevaluator_nameand options), read after every control step, with three changes:- The goal must hold 30 consecutive control steps (1.02 s); the episode then ends in success. Upstream counts any
single step (
ever_success), and several of its rules count an object while it passes through a region (thrown, carried over a bin): the hold stops a pass-through from counting. - Hang a mug: upstream's test (the mug's centre within 0.115 m of the tree's axis, between its base and 0.05 m above its top) also passes a mug standing on the table next to the tree; here the mug must also touch neither the table nor a gripper, i.e. hang from the tree, released.
- Ball-tray balancing: upstream's evaluator only checks that the ball is still on the tray, so doing nothing scores 1. Here the task's runtime decides: the episode succeeds when its countdown (started the moment both handle bars are touched by the fingers, one arm on each; 11 s in the frozen instance) completes with the ball on the tray and both handles held throughout (each bar touched within the last 0.1 s at every control step); it fails the moment the ball leaves the tray or a handle is let go during the countdown (the runtime's countdown never restarts). Every rule is stated in words, with its thresholds, in each task's instruction.
- The goal must hold 30 consecutive control steps (1.02 s); the episode then ends in success. Upstream counts any
single step (
- Texts. The privileged instruction names the scene's bodies and points to
s.state(); the standard instructions state the goal and the rule in words, with the robot's conventions. World-frame information: the standard texts give no world coordinates of workspace items (no object, bin or table positions; directions are relative to the arms); the robot's base poses, the camera calibration and the world frame's axes are given. - Time limit: 3600 s, the same for every task and mode (each task's
[agent] timeout_sec).
Modes
- Privileged:
rcb_abc.Session(/opt/abc-tools/rcb_abc.pyin the image) in the agent's container: reset, the scene's full state (every free body's pose, the evaluator's and the runtime's view, the randomization record), MuJoCo model and data, exact mid-episode snapshots, rendering (RGB, depth, calibration), the robot's kinematics (FK, IK), recording and self-replay. Deliverable/app/output/trajectory.npz,(T, 14)rows, at most 20,000. The grader (/opt/abc-tools/grade.py) checks the file (readable,(T, 14), finite, not too long: otherwisebad_trajectory, the agent's 0) before it replays it twice, side by side, in fresh processes; a replay that then gives no result isreplay_failedwithgrader_error = 1andinfra_error = 1. A third replay rendersreplay.mp4after the verdict. - Standard (
eai-standard/2.2): the shared layer ofembodied_first_exam/standard/(from the pinnedstandard-filesimage) and the adapter/opt/abc-tools/standard/server.py(in the base image;run.shstarts it throughRCB_SERVER):- action type
joint_position: the 14 numbers per control step, as in privileged mode;linearorholdbetween rowsdtapart; rows are executed as sent (abc_sim rounds each command to float32 and the servos clamp it to the actuator's range; the clamped entries are reported inmeta.clipped, not changed: clipping a row before abc_sim's float32 rounding changes the physics in the last bits); - six cameras, 640 x 480, depth on request:
top,overhead,left_side,right_side(static) and the wrist camerasleft,right(on the arms' sixth links); defaulttop,left,right; - proprioception:
joint_pos(the 14 action coordinates measured),joint_vel,gripper_opening(m),tcp_pose(each arm's tool centre point, world frame,wxyz),joint_effort; never an object's pose; /kinematics: FK of the tool frames (left_tcp,right_tcp,left_grasp,right_grasp), the arm links and the wrist-camera mounts; IK of the tool frames with each arm's own six joints (damped least squares, several starts);- resume class
exact(MuJoCo on the CPU repeats bit for bit in a fresh process);max_frames_per_call1000; - reset budget 0 (the mode's default): one episode. The adapter implements the optional reset capability, so a run
can grant resets with
RCB_RESET_BUDGET.
- action type
- Realtime (
eai-standard/2.3): the shared realtime layer (rcb_realtime.pyfromstandard-files,rcb_service.pybyte-identical to the other realtime benchmarks' copies, in the base image at/opt/abc-tools/realtime/) and the adapter/opt/abc-tools/realtime/server.pyon the same Session: the standard twin's instance, action, success rule, cameras and proprioception (not served:/kinematics, per-frame camera poses, depth modes). The scene-dynamic families get the twin: the conveyor belt carries the objects, and the tray's runtime pushes the ball. S = 1: the service's real-time factor (0.034 s of simulation per control step) is 11.2 for the conveyor and 17.1 for the ball tray without rendering, 5.3 and 10.2 rendering three cameras every step (RTX 4090, EGL), above PROTOCOL §2's 3. Reset budget 10 (this benchmark's choice; the mode's default is 0).
Grading (standard and realtime)
The verdict is the live episode's (PROTOCOL H.4): the episode the sim sidecar ran, under the task's rule as
Session.step applied it (the 1.02 s hold is part of the rule, run on the live simulator). Standard: the collect hook
(/opt/abc-tools/standard/finalize.sh) ends the service, writes final.json (graded_by = live, replays = 0) and then
renders replay.mp4 in one pass over the trajectory (EGL on the sidecar's GPU); the video decides nothing. Realtime: the
shared layer does the same from the adapter's step(). The verifier is the shared v2_1/grade.py (from the pinned
standard-files image), for both modes. At a fresh episode's start the service compares the built scene's state hash
with the instance's recorded one; a mismatch gives up before the agent is served (instance_mismatch, infra_error).
Reference solutions and oracle levels
oracle = "full" for all 60 directories. The reference of a family (solution/oracle.npy) is the commanded actions of
the validation demonstration whose scene is the frozen instance (after its first frame), replayed from the instance and
cut at the success step; spell_bair's is a hand-built trajectory (scripted inverse kinematics on the grasp frame, top-down
grasps with the clearest finger yaw, place checks and re-placing) on its seed-243 layout. The rows are float32 values
(abc_sim rounds every command to float32 anyway), so the privileged replay and the standard client execute the same rows.
Privileged: solution/oracle_replay.py replays and saves them. Standard: solution/oracle_stream.py streams them as
joint_position chunks. Realtime: solution/oracle_plan.py plans them into the episode at the control steps
where they ran (submit(..., at=step), sent ahead of the clock).
How the instances were chosen (the feasibility spike, 2026-10-20/21, issue #473): every validation demonstration of each family was replayed from its recorded scene twice in fresh processes under MuJoCo 3.6.0, the version the demonstrations were recorded with (under 3.8.1, which upstream pins now, fewer replays succeed: nuts and bolts 0/12 against 6/12); 510/510 final states were bit-identical. The first demonstration (in a fixed order) whose replay succeeded became the instance. Replay success per family (the rest is open-loop drift: the recordings themselves mostly succeed): pouring, grab from clutter, in-hand transfer, turn a mug and four spell words 20/20; spell yam 18/20, abc 17/21, xdof 16/20; put relative 18/20; put bottles 18/20; count 17/20 (exactly 1 / 2 / 3 / 4: 10/10, 9/10, 10/10, 9/10); hang a mug 16/20; drawer search 15/20; throw bottles 11/20; conveyor 10/20; dish rack 7/20 (its demonstrations fail 11/20 themselves); nuts and bolts 6/12; sweep 5/20; LEGO 5/20; chess 3/17; ball tray (the rule above) 29/100.
Verification (efe:abc-sim-0.1.0)
Oracle and nop with Harbor on an RTX 4090 workstation (AMD Ryzen 9 7950X), 2026-10-21, the image built from
images/abc-sim/: oracle 60/60, every one on its first attempt; nop 60/60 scored 0 (infra_error 0). In every
family both privileged replays, the standard service's live episode and the realtime service's live episode end in the
same final state (the hash below), which is also the reference's final state on the host where the instances were frozen
(Python 3.11, NumPy 2.4.6; the image runs Python 3.12): MuJoCo 3.6.0 on the CPU is bit-exact across these processes.
The realtime oracles (the reference planned with submit(..., at=step) at S = 1) ran with no late step and no stall (the
largest lag 11 ms).
| family | reference rows | oracle, privileged (both replays) | oracle, standard (live) | oracle, realtime (live) | final state | nop (priv / std / realtime) |
|---|---|---|---|---|---|---|
ball_tray |
453 | 1 | 1 | 1 | 0d05516cffd7e620 |
0 / 0 / 0 |
chess |
1872 | 1 | 1 | -- | 95d72acb81802ac4 |
0 / 0 |
conveyor_pick |
274 | 1 | 1 | 1 | 80563de74a724075 |
0 / 0 / 0 |
count |
277 | 1 | 1 | -- | b505c774f544eaf8 |
0 / 0 |
count_four |
671 | 1 | 1 | -- | 61409c4d0e466e6a |
0 / 0 |
count_one |
264 | 1 | 1 | -- | dabc9c60b668c109 |
0 / 0 |
count_three |
538 | 1 | 1 | -- | cfb8e8f612a52bd7 |
0 / 0 |
count_two |
291 | 1 | 1 | -- | 0514bf38465fb2ea |
0 / 0 |
dish_rack |
375 | 1 | 1 | -- | 12ee39cc6ca7725b |
0 / 0 |
drawer_search |
432 | 1 | 1 | -- | 5f4d8d12d9027048 |
0 / 0 |
grab_clutter |
226 | 1 | 1 | -- | 1de5d81724c1628a |
0 / 0 |
hang_mug |
314 | 1 | 1 | -- | c13c9ccdf019b28e |
0 / 0 |
inhand_transfer |
374 | 1 | 1 | -- | b15263ed28fedcff |
0 / 0 |
lego |
2205 | 1 | 1 | -- | 697c1c23c81322bf |
0 / 0 |
nuts_bolts |
2872 | 1 | 1 | -- | adc51944b0096256 |
0 / 0 |
pouring |
307 | 1 | 1 | -- | 14a0bf1cea242cde |
0 / 0 |
put_bottles |
476 | 1 | 1 | -- | 011999892aad55fe |
0 / 0 |
put_relative |
237 | 1 | 1 | -- | b3bacfe0c48af964 |
0 / 0 |
spell_abc |
507 | 1 | 1 | -- | 8cbaf7ef49ff90a9 |
0 / 0 |
spell_agi |
581 | 1 | 1 | -- | 6b54dcc0c6e409c3 |
0 / 0 |
spell_bair |
1360 | 1 | 1 | -- | f20fdf438e9ab63f |
0 / 0 |
spell_cat |
909 | 1 | 1 | -- | a75f73d72cf9b18c |
0 / 0 |
spell_dog |
580 | 1 | 1 | -- | b2090d429824e80e |
0 / 0 |
spell_fish |
675 | 1 | 1 | -- | 8a96253cc31a34ec |
0 / 0 |
spell_xdof |
719 | 1 | 1 | -- | 69efe25ba8cd6cbc |
0 / 0 |
spell_yam |
581 | 1 | 1 | -- | 0d6b71b98369775d |
0 / 0 |
sweep |
1288 | 1 | 1 | -- | 0c0685ed6ef4b04c |
0 / 0 |
throw_bottles |
563 | 1 | 1 | -- | d5a8babe7c4be42b |
0 / 0 |
turn_mug |
561 | 1 | 1 | -- | 75511085888777f6 |
0 / 0 |
The oracle batch ran on the task directories before their last edit, which changed only texts (the ball tray's
instruction and README) and the resource declarations of task.toml; nop ran on the final directories, and so did the
oracle again on twelve of them (the ball tray's three, the conveyor's three, chess and spell bair in both modes, nuts and
bolts standard, drawer search privileged): 12/12, the same final states.
bash ci_checks/run_all.sh tasks/abc-sim/abc-sim-*: every invariant holds; the one advisory finding is the root README's benchmark list (regenerated after merge).python3 scripts/task_conventions.py --check tasks/abc-sim/abc-sim-*: clean.ci_checks/check-standard-protocol.pyagainst a live spell_abc service: 222 PASS, 0 FAIL, 16 WARN (velocity limits: the simulator enforces none).- Crash recovery: the spell_abc service killed (
kill -9) after 300 of the reference's 507 rows; it restarted, resumed the same episode (resumes1, no state mismatch) and the reference ended in success at the same final state. - In one process: a reset after 150 steps returns to the instance's hash, a replay after it ends at the fresh-process
final state, and a mid-episode
snapshot()/restore()continues bit for bit (all 29 families). - Measured (the oracle trials): the privileged agent container 1.9 GB and its verifier 3.9 GB (chess); the standard sidecar 2.1 GB and 0.83 GiB of GPU memory (chess), the realtime sidecar 0.8 GB and 0.72 GiB; the client container 0.03 GB.
Integrity
- abc_sim ships no scripted expert or policy. The image takes the package from the upstream repository at its commit
and the asset packages by checksum in a fetch stage whose layers are not part of the image, and leaves out what
points at the demonstrations, checkpoints or the asset mirror: the package's README files, the asset manifest, the
Blender exporter of recorded dataset episodes and the comments of the scene XMLs (one described the training data).
Only the meshes and textures the 29 frozen scenes open are kept (
images/abc-sim/assets.txt), with every package's licence and attribution files (/opt/abc/ASSET_LICENSES.md). - Nothing an agent can read names its source: the instances carry no demonstration or episode id, and the reference
solutions live under
solution/only. - Agent egress is the model APIs and the package registries (PROTOCOL H.6). The demonstrations and checkpoints are on
Hugging Face and the upstream mirror, both unreachable. ABC Sim is not on PyPI (
abc-simandabc-minimaldo not exist there).
Files
Each task directory carries its frozen instance (instance.npz) and its reference solution (solution/); the standard
twins also the task's text for the service (environment/sim/task.json). Everything the tasks share is in the base image
(images/abc-sim/: the Dockerfile, the pinned packages, the asset list, the fetch stage's assembly script, and
tools/: rcb_abc.py, grade.py, video.py, standard/server.py, standard/finalize.sh, realtime/server.py,
realtime/rcb_service.py). The tooling that froze the instances and wrote the task directories is kept outside this
repository.
Standard-mode service details
Serves eai-standard/2.2 (MuJoCo on the CPU, EGL rendering, resume class exact); max_frames_per_call = 1000. The
optional reset capability is implemented with a budget of 0 by default; the 2.2 client has no reset(), so a run that
grants resets calls POST /reset through Robot._post. The scene builds in 1-5 s (RCB_STARTUP_TIMEOUT_S 300).