The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
KinDER
Physical reasoning for robot planning and control, kept apart from perception and language: tool use, non-prehensile multi-object manipulation, geometric and dynamic constraints.
Quick start
harbor run -d embodied-first-exam@all -p tasks/kinder -a codex -m openai/gpt-6-astra
Every image is public: Harbor pulls it.
| upstream | KinDER (RSS 2026, arXiv 2604.25788), MIT, pinned at 5b2dbac |
| simulator | MuJoCo 3.3 (Dynamic3D) in KinDER's MimicLabs lab, CPU physics |
| robot | TidyBot++: holonomic base, Kinova Gen3 7-DoF arm, Robotiq 2F-85 gripper; delta actions (MuJoCo: 10 Hz) |
| score | pass = KinDER's own goal check within the episode (1000 control steps) |
| tasks | 5 families × 2 modes: tasks/kinder/kinder-<family>-i00-privileged (scoop-pour, sweep-simple: -i01-) and -standard (eai-standard/2.1) |
| image | ghcr.io/mll-lab-nu/efe:kinder-0.1.6 (images/kinder/, 4.6 GB with the lab scenes) |
| oracle | dynamo (the fixture): full, both modes; the other four: none, human review (positive examples below; scoop-pour and sweep-simple: none yet, unvalidated) |
Tasks (one variant each, seed 0)
| family | challenge | the robot must | positive example |
|---|---|---|---|
| dynamo | nonprehensile | drive into a floor square that three chairs block (fixture) | oracle, both modes; Luna, both |
| sweep-into-drawer | nonprehensile, tool, dynamic | open a drawer, sweep five cubes into it | Luna, privileged; run codex-gpt6_1_sol-high, standard (no reset) |
| scoop-pour | nonprehensile, tool, dynamic | move 50 cubes between bins with a scoop | none yet (i01) |
| constrained-cupboard | geometric | six identical rods, each into the compartment it is owed | Luna, privileged only |
| sweep-simple | nonprehensile, tool | fifty cubes into a box marked on the floor, with a wiper | none yet (i01) |
Why these (no model's results enter the choice):
- Valid: KinDER's 3D families whose goal a camera can see or a sentence can name (the 2D ones move an abstract planar robot with no real counterpart; LimbRepositioning3D's IKFast module cannot be built in our image).
- Most task objects (upstream's
oN): scoop-pour 50, sweep-simple 50, constrained-cupboard 6, sweep-into-drawer 5; every other 3D family has at most 4. - Variant: both cube families use their o50 variant; sweep-simple's o50 has three goals, and the one used is upstream's README example (the right side of the kitchen island). Scoop-pour's o100 is not used: it simulates about 5x slower, and known 30-cube solutions already use 770 and 845 of the 1000 control steps. Cupboard (o1-o6) and drawer (o5 only) are at their largest.
Positive examples are runs published on the results site; each scores 1 under today's privileged verifier, and "both" means it also passes when sent through the standard client. "privileged only": the trajectory moves the base diagonally by up to 0.14 m a step, which standard mode clips to a 0.1 m disc (the largest move that stays in upstream's per-axis bound after the odometry rotation).
Open: tasks without a positive example in their own mode (PROTOCOL §4)
Scoop-pour and sweep-simple (i01, o50) have no positive example yet in either mode: they are unvalidated
([metadata] validated = false) until a human review records one, and are ranked with the validated tasks (PROTOCOL
§2). Every other family has one (human review, PROTOCOL §4: per family, at least one), so its predicate and instance are known
to be solvable. One standard twin has none of its own yet:
kinder-constrained-cupboard-i00-standard. Its privileged example moves the base by up to 0.14 m a step.
Modes
| mode | the agent gets | resets | graded |
|---|---|---|---|
| privileged | the simulator, its full state (get_state/set_state) and the goal check |
unlimited | its trajectory.npz, replayed twice in a fresh process |
| standard | the robot's own sensors, through rcb_client.py; no world coordinates, no goal check |
0 (the protocol's default) | the last episode, as it ran (live; nothing replayed) |
Grading in standard mode: the live episode. The sim sidecar judges the episode it runs with KinDER's own goal check
(the environment's terminated, at every control step; step() in each standard task's environment/sim/server.py) and
ends it there; rcb_service.py writes that verdict (graded_by = "live", replays = 0) once the agent's container is
stopped. The frozen start is checked on the live simulator: start_sha is the instance's recorded scene_sha (the
state hash right after reset), and a KinDER that no longer builds the instance (rcb_kinder's own check) is
instance_mismatch; both give up as infrastructure (infra_error = 1) before the agent is served. The video is one
render pass over trajectory.npz after the verdict (render_video, the task_view camera; MuJoCo EGL or PyBullet on
the CPU; median about 90 s a trial), which never changes the verdict. infra_error = 1 only for no or an unfinalized
final.json, a give-up after crashes not all inside the agent's calls, a reset whose scene does not rebuild (the
agent's reset was legitimate) and instance_mismatch; a simulator error raised by the agent's call, crashes inside its
calls, a timeout and a failure are the agent's 0.
World-frame information (PROTOCOL §5): privileged mode gives KinDER's world state; standard mode gives none -- odometry starts at the robot's start pose, and the goal is named by what the cameras see. One exception: where the goal is a translucent region marked on the floor (dynamo's green square, sweep-simple's grey box), faint in the robot's cameras, the standard instruction states its centre, size (and the box's height) in the odometry frame at t = 0, measured on the frozen instance.
| standard mode: given (a real TidyBot++ has it) | not given |
|---|---|
| the task sentence: goal and success condition; dynamo's and sweep-simple's marked goal region at t = 0 (odometry frame) | any object's position; the other goals' positions |
robot_base RGB, robot_wrist RGB-D (the Gen3's RealSense D410), 480×640 |
world coordinates; a room camera (RCB_OBS_ROOM_CAMERA=1, never overhead) is a knob, off |
| base odometry (pose, velocity), arm joint angles and velocities, finger position | the goal check before the episode ends (success / failed only) |
spec(): action layout and bounds, arm joint axes, positions, limits and torques, the grasp point, camera fields of view and poses, control rate, episode length, budget |
masses, friction, the simulator |
Run
harbor run -p tasks/kinder/kinder-scoop-pour-i01-standard -a codex -m openai/gpt-6-astra --env-file .env
harbor run -p tasks/kinder/kinder-dynamo-i00-privileged -a oracle # or -a nop: scores 0
Agent, provider and effort settings: docs/protocol/running.md. Agent time cutoff: 1800 s in both modes. Standard tasks grant no reset by default (the protocol's default): one episode, whose 1000 control steps the practice and the graded attempt share;
RCB_RESET_BUDGET=N harbor run ... runs a configuration with N resets, whose results are reported separately (the shipped instruction describes the default, one episode). Build the image: docker build -f images/kinder/Dockerfile -t ghcr.io/mll-lab-nu/efe:kinder-$(cat images/kinder/VERSION) . (from the repository root).
Results
Agent results are not kept in this repository: they are published on the results site, keyed by run id. Standard-mode runs with another reset budget are reported separately.
Standard-mode service details
KinDER and HumanoidBench service (eai-standard/2.1). Same endpoints, RCB1 frames and crash recovery, served by one
shared service (rcb_service.py in each standard task's environment/sim/, identical in both) with a per-benchmark
simulator adapter (server.py). Actions are delta or joint-position rows at the task's control rate (KinDER 10 Hz,
PyBullet tasks kinematic: a refused move comes back blocked and stops the chunk; HumanoidBench 50 Hz); the step
counters are rows and control_steps. No world coordinates are given (odometry and camera poses are in the robot's
starting frame). A request body over 64 MB gets 413 before it is decoded. The service implements the optional reset
capability (RCB_RESET_BUDGET).