The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
HumanCLAW
Find, navigate and interact with a simulated human body: in a furnished HSSD house, find an object of a category, walk until the body touches it and, for a bed, couch or toilet, sit on it, using nothing but whole-body motion skills. No arm, no gripper: the robot is an SMPL-X human body, moved by HumanCLAW's MotionDiT skill models under Half-Physics.
| upstream | HumanCLAW (HumanClawBench, Human-CLAW/HumanCLAW), a source snapshot at commit fe3d376 (tree f60d71a0 of harnesses/humanclaw); profile paper_fullval_v1 (1,218 val episodes in 41 HSSD scenes) |
| simulator | Habitat-Sim acbe6f4 with HumanCLAW's Half-Physics patch (Bullet 2c204c4), MotionDiT flow-matching skills (30 NFE midpoint), GPU (EGL rendering, CUDA) |
| robot | smplx-human: HumanCLAW's SMPL-X neutral body, 8 motion skills (walk, turn, side step, step back, climb up / down a step, sit, stop) |
| score | HumanCLAW's own metric code, unchanged (evaluation/metrics/episode.py): NavSR@20cm (chair, potted plant, TV) or InteractSR (bed, couch, toilet) |
| tasks | 55 frozen episodes x 2 modes (110 task dirs; Selection below): tasks/humanclaw/humanclaw-{nav,sit}-<category>-i<NN>-privileged and -standard (eai-standard/2.1) |
| image | ghcr.io/mll-lab-nu/efe-private:humanclaw-<VERSION> (images/humanclaw/, BUILD_ONLY: without access to the organization's packages, build it once yourself, Before you run below) |
The task
One task = one frozen episode of HumanCLAW's full validation set: the scene, the body's start pose (HumanCLAW's own
spawn values), the target category and its goal objects, HumanCLAW's instruction word for word ("Find the chair and
move until your body is touching the target, with zero distance to the target object.", plus "Finally, sit on the
bed." for an interaction category). An action is a row [action_id, value], one motion skill of 0.5 s (15 frames at
30 Hz), mapped to HumanCLAW's skill calls by its own planner code (_chooser_action, so its clamps apply). An episode
allows 100 actions, the stop included, and has no reset; success needs an explicit stop (HumanCLAW's active stop).
| privileged | standard | |
|---|---|---|
| the agent has | the simulator (hc_tools.Env): the body's pose, the targets' boxes, the navmesh, free resets |
a client: the head camera (448 x 448 RGB, 90° fov), actions used / left, the pelvis' height change |
| deliverable | trajectory.npz (actions, T <= 100 rows) |
none: the sidecar records and grades the live episode |
| graded by | two fresh-process replays from the verifier's copy of the instance: both succeed, equal final state hashes | the live episode (HumanCLAW's finalize at the stop, in the sidecar); nothing replayed |
| video | the first replay's recorded poses, rendered after the verdict | the live episode's recorded poses, rendered after the verdict |
| time cutoff | 3600 s | 3600 s |
What differs from HumanCLAW's own evaluator:
- The motion model's noise is seeded at every reset (torch / numpy / random, seed 0). HumanCLAW never seeds it, so its rollouts of the same actions differ run to run; seeded, an episode is a function of the actions on a given GPU model, which a submitted trajectory and a crash-resume both need. Across GPU models the motion model's arithmetic differs in the last bits and the physics amplifies it, so the verifier replays on the machine the agent ran on.
- A reset rebuilds the simulator (HumanCLAW builds a fresh one per rollout too), so moved furniture comes back.
- The agent is not HumanCLAW's VLM pipeline (planner + verifier, history of 10): it is whatever the harness runs, and it calls the skills directly.
- Videos are rendered after the verdict from the poses the episode recorded frame by frame (HumanCLAW's own delayed renderer: it restores each recorded post-physics pose and renders; simulation never advances), so the frames are the graded episode's, not a re-run. The live video writer stays off as in HumanCLAW's paper runs (its third-person sensor can change Bullet's contact cache).
view_pointsin the frozen goals are cut to the 5 best per goal (HumanCLAW's loader needs one; nothing it scores reads them).
Network: HumanCLAW's public repository holds recorded rollouts of these very episodes (its docs/analysis has
action chains), so the agent reaches the model APIs and the package registries only (allowlist), in both modes.
Selection
The 55 episodes are those of HumanCLAW's 1,218 validation episodes that GPT-6 Luna (Codex 0.164, effort xhigh) failed in both modes, in one run over all 1,218 episodes x 2 modes (privileged 1,161 / 1,218 solved, standard 295 / 1,218): bed 22, couch 12, toilet 9, TV 11, chair 1, in 23 of the 41 scenes. Each episode keeps the number it has among all 1,218, so the numbers have gaps. None of the 55 is in the core set. Upstream episode ids and the per-episode outcomes are kept as trusted provenance outside the repository; no agent-readable file names them.
Before you run
The tasks' shared code (this repository's embodied_first_exam/benchmarks/humanclaw/: the privileged tools, the
sidecar service, the grader) comes from the public files-only image images/humanclaw-files/ (COPY --from, pinned by
digest); only the simulator image is private.
images/humanclaw/ builds ghcr.io/mll-lab-nu/efe-private:humanclaw-<VERSION> (21.8 GB): HumanCLAW's code, Habitat-Sim,
the motion weights and the 41 HSSD scenes. It holds data that may not be redistributed, so it is BUILD_ONLY
(images/humanclaw/BUILD_ONLY has the licences and the full recipe notes). A runner with access to the organization's
packages pulls it. Anyone else builds it once, from the repository root:
- On Hugging Face, request access to the gated dataset
HumanCLAW/HumanCLAW-HSSD(approved automatically) and accept the terms ofhssd/hssd-hab. - Make a read token and export it:
export HF_TOKEN=hf_... - Build under the tag the tasks'
cache_fromnames (the token goes in as a BuildKit secret, never into a layer):docker buildx build --builder default --secret id=hf_token,env=HF_TOKEN -t ghcr.io/mll-lab-nu/efe-private:humanclaw-$(cat images/humanclaw/VERSION) images/humanclawIt takes about 30 minutes on a 32-core host (Habitat-Sim's compile and the HSSD download), no GPU, and about 40 GB of free disk under Docker's data root. harbor run -p tasks/humanclaw/<task> ...then builds each task from the local build cache: no token, no download, seconds (the failed cache import from the private package is only logged). Keep that cache (docker builder prunedrops it).
Running needs an NVIDIA GPU (EGL rendering and the motion model on CUDA; compute capability 8.0 to 9.0). A privileged trajectory replays bit for bit only on the GPU model it was made on (The task, above).
Licences
| part | licence | in the image |
|---|---|---|
| HumanCLAW code (episode loader, Half-Physics env, skill runner, metrics) | Apache-2.0 | yes (LICENSE, NOTICE in /opt/humanclaw) |
| Habitat-Sim, its Half-Physics patch; Bullet | MIT; zlib | yes |
HSSD scenes (hssd/hssd-hab) and HumanCLAW's val41 supplement (HumanCLAW/HumanCLAW-HSSD) |
CC BY-NC 4.0, gated (terms accepted per user on Hugging Face) | yes: may be used, not redistributed |
| the body model (convex hulls and joint centres derived from SMPL-X / SMPL) | SMPL-X / SMPL model licences: non-commercial research, no redistribution | yes |
MotionDiT paper_fullval_v1 weights (HumanCLAW/HumanCLAW) |
released with the code; trained on AMASS / BABEL (non-commercial research) | yes |
So the image is private (BUILD_ONLY) and each user builds it under their own acceptance of these terms; the task directories hold only the frozen episodes (scene ids, start poses, goal objects) and this repository's code.
Reference solution
A navmesh oracle with the privileged tools (solution/oracle.py): Habitat's shortest path (on a navmesh for a wider
body) to the nearest goal viewpoint or to a reachable point beside a target box, turn-and-walk along it, close in with
short strides, then, for an interaction episode, HumanCLAW's own sitting recipe (turn the back to the target, step
back, sit at the category's height until the pelvis touches it), then stop; it tries a few variants with free resets.
It solves 4 of the 55 episodes. On 7 it found no solution: targets behind furniture or up a step, and seats the
body does not land on. The other 44 were not searched (oracle = "none"; their oracle_evidence says so).
- privileged:
solution/actions.jsonis the oracle's trajectory, replayed twice in fresh processes to success with equal final hashes on the GPU model named inoracle_evidence. The physics amplifies the last-bit differences of the motion model between GPU models (an RTX 4090 and an L40S agree on the start hash, never on the final one, and on about 1 trajectory in 10 not even on the verdict), sosolve.shreplays it first and, where it fails on the machine at hand, runs the oracle again closed loop there. - standard: the same actions streamed as
skillrows through the client (open loop, so they hold on the GPU model named in the evidence).