The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
AGENTS.md — map for coding agents working in this repo
Read README.md, docs/protocol/README.md (v2.3: recommended conventions, each task declares its own settings, §2; the eight
invariants of Appendix H bind and no task value overrides them), docs/protocol/running.md (official runs) and CONTRIBUTING.md first.
- Modes: privileged, standard (wire protocol
eai-standard/2.x) and realtime (standard mode on a clock that does not stop,eai-standard/2.3), PROTOCOL §3. Task directories end in-privileged,-standardor-realtime; base images areghcr.io/mll-lab-nu/efe:<name>-<version>(private:efe-private:<name>-<version>), one image per benchmark (PROTOCOL §1). Noresults/in task dirs. - A pull request delivers tasks only (PROTOCOL §1, "What a contribution delivers"): no task-generation code in the
repository;
scripts/holds only what running the benchmark needs. Every task is in the normal form ofscripts/task_conventions.py(normalize): fields, protocol spelling, test.sh, graders, README layout, noresults/.python3 scripts/task_conventions.py --check tasks/<benchmark>/<task>reports a task it would change. - Tasks are self-contained Harbor task dirs, one folder per benchmark:
tasks/<benchmark>/<task>/(the folders:scripts/task_conventions.pyBENCHMARK_TABLE; find a task dir withtask_dir(name)/iter_tasks(), never by joiningtasks/and a name). Nothing at run time imports repository code outside the task directory. Reference tasks:tasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/and its standard-mode twintasks/robotwin-2/robotwin_2_0_stack_blocks_two-standard/. - Code several tasks share: recommended in the benchmark's base image (
images/<sim>/); elseembodied_first_exam/benchmarks/<benchmark>/(the same folder name astasks/<benchmark>/; tooling, graders, service adapter;embodied_first_exam/README.md). Its tasks get it from a files-only image (scripts/files_image.py,images/<name>/FILES.json): the tasks' DockerfilesCOPY --from=<image@digest>each file to the path a copy would have had (a sidecar's too: built from its ownenvironment/sim/Dockerfile, which puts the directory and those files at/srv). images/<sim>/Dockerfile: simulator base images, tagged fromimages/<sim>/VERSIONif present elseimages/VERSION;scripts/build_images.sh. MuJoCo tasks (dextoolbench_*,mujoco_playground_*) useimages/mujoco/(runprepare_assets.shfirst); notes intasks/<benchmark>/README.md.- Every resource number is a cap for one container; the format, where each is written and what enforces it: PROTOCOL
§6 (running.md "Resources" for runs). Edit the
[[cap]]rules ofscripts/task_resources.toml, never a task's caps or its## Resourcesparagraph:normalizewrites both. The monitor sidecar (images/monitor/rcb_monitor/, in the pinnedghcr.io/mll-lab-nu/efe:monitor-<version>image;normalizewires it into every task, which keeps only itslimits.json;ci_checks/check-monitor.pyruns its tests) changes with a new image version only. A CPU cap above the host's CPU count is the runner's to clamp (docker refuses it); the values inscripts/task_resources.tomlstay. The same step'ssanitizeline in everytests/test.shremoves links and special files from/app/outputbefore the grader reads the agent's files. - The agent's time cutoff is each task's own
[agent] timeout_sec, set by its author in the pull request (rules: PROTOCOL §2; whatnormalizerenders from it,## Time limitlast in everyinstruction.md, and how to change it: docs/protocol/running.md "The agent's time"). A benchmark's own text says "the time limit (## Time limit)", never a duration. Checks:ci_checks/check-cutoffs.py(advisory), the twins' equal numberci_checks/check-twins.py(H.7). - Seeds:
[metadata] instance_seedis the default seed,[metadata.seeds]only where a task allows others (PROTOCOL §2; how a runner draws one: docs/protocol/running.md "Seeds"). Committed non-default instances live under the privileged task'stests/seeds/<seed>/(never underenvironment/);ci_checks/check-seeds.pychecks them. - CI has two tiers.
ci_checks/run_invariants.sh [task dirs](the required jobinvariants) runs only the checks that enforce PROTOCOL Appendix H and fails on any failure, error or incomplete run;ci_checks/run_all.sh [task dirs]runs it and then every advisory check (--tier advisoryalone: exit 3 = warnings only). New per-task check =check-<thing>.sh <task_dir>, listed inci_checks/check-task.shunder INVARIANT_CHECKS (it guards Appendix H) or ADVISORY_CHECKS; a benchmark-specific check is that benchmark's hookci_checks/benchmarks/<bench>/invariant-*oradvisory-*(sharedci_checks/names no benchmark); an invariant rule also gets a mutation inci_checks/check-ci-selftest.py(a real task broken that way must fail the rule;--tier invariants). Run the checks from a checkout made withumask 022(run_all.shsets it for itself).ci_checks/check-unit-tests.sh --allruns the service-layer tests that need numpy + requests (CI jobunit-tests). Everything runs concurrently (ci_checks/parallel.py,RCB_CI_JOBScaps it). CI on a pull request runsrun_all.sh --changed: the scope fromci_checks/ci_scope.py(one benchmark's own paths -> its tasks; shared code -> the full suite); pushes to main and the nightly run are the full suite. A benchmark's own paths come fromscripts/task_conventions.pyBENCHMARK_TABLE; aci_scope.pyUNITSrow only names an image directory of another name (ci_scope.py --check). tasks/<benchmark>/<task>/DIGEST: the task's content digest (PROTOCOL Appendix H.8), written last bynormalize(after a hand edit:python3 scripts/task_conventions.py tasks/<benchmark>/<task>);ci_checks/check-dataset-manifest.shrecomputes it. The release manifesttasks/dataset.tomlis generated (scripts/dataset_manifest.py write) and not committed, so pull requests do not conflict on it. The task list (TASKS.md,tasks.json,dataset.toml) is generated after every merge to main and pushed to the branchtask-list(.github/workflows/task-list.yml;scripts/dataset_manifest.py list --out DIRlocally).- Local docker sandbox has no
[environment].gpus. RoboTwin and MuJoCo tasks pass a GPU through withenvironment/docker-compose.yaml(servicemain, nvidia device reservation;simin standard mode) — copy it fromtasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/. - Run agents with Harbor's native agents only (
-a codex,-a claude-code); no adapters, and no gateway that holds credentials or changes requests. The usage gateway (scripts/usage_gateway/,GATEWAY=1inrun_agent_batch.sh; official runs use it) forwards model-API calls only (api_paths.json; anything else at a provider is a 403), records token usage, and is then, with the package registries, the agent container's only egress (docs/protocol/running.md "Official runs"). No tool that runs outside the container in official runs: the harness's web search is off (run_agent_batch.shdefaultWEB_SEARCH=disabled), as are account connectors, apps and plugins. - A task's agent egress: Appendix H.6 and docs/protocol/running.md "Network" (
[agent] network_mode, "allowlist" or "public");normalizeends everyinstruction.mdwith## External material(PROTOCOL §5). Scene dynamics are each task's[metadata] dynamics(no central table). A task needs no download at run time: bake it into the image or the task directory. Nothing an agent can read (image, tooling, instance metadata) names or locates a solution. Unattended batches:scripts/run_agent_batch.sh(lanes with per-job claims; configured by environment variables, docs/protocol/running.md); its runner-side tools (log_mirror.py,mem_guard.py,scan_secrets.py,account_failure_check.py) are not task code. eai-standard/2.2(docs/protocol/standard_mode.md Part 2; which benchmarks serve it: docs/TAXONOMY.mdprotocol): its layer (rcb_limited.pywith in-trial crash recovery, an implementation detail; the protocol's rule is the runner's infrastructure rerun, Appendix H.5,rcb_client.py,rcb1.py,run.sh) has one source,embodied_first_exam/standard/, copied unchanged into every 2.2 task (check-tooling-identity.sh); each benchmark adds only its adapterenvironment/sim/server.py(embodied_first_exam/standard/toy_server.pyis the smallest one).eai-standard/2.4(docs/protocol/standard_mode.md Part 4, RoboARC): 2.3 + the operations capability. Its layer isembodied_first_exam/standard/v2_4/rcb_limited.py+rcb_client.pywith 2.2'srcb1.pyandrun.sh, copied unchanged into every 2.4 task (check-tooling-identity.sh).python3 embodied_first_exam/standard/selftest.pytests the crash recovery on the CPU in about two minutes.- Benchmark-level rules (templates, budgets, tooling file names, the world-frame and tooling declarations):
tasks/<benchmark>/README.md.