Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

9.0 KB
Raw Blame

AGENTS.md — map for coding agents working in this repo

Read README.md, docs/protocol/README.md (v2.3: recommended conventions, each task declares its own settings, §2; the eight invariants of Appendix H bind and no task value overrides them), docs/protocol/running.md (official runs) and CONTRIBUTING.md first.

  • Modes: privileged, standard (wire protocol eai-standard/2.x) and realtime (standard mode on a clock that does not stop, eai-standard/2.3), PROTOCOL §3. Task directories end in -privileged, -standard or -realtime; base images are ghcr.io/mll-lab-nu/efe:<name>-<version> (private: efe-private:<name>-<version>), one image per benchmark (PROTOCOL §1). No results/ in task dirs.
  • A pull request delivers tasks only (PROTOCOL §1, "What a contribution delivers"): no task-generation code in the repository; scripts/ holds only what running the benchmark needs. Every task is in the normal form of scripts/task_conventions.py (normalize): fields, protocol spelling, test.sh, graders, README layout, no results/. python3 scripts/task_conventions.py --check tasks/<benchmark>/<task> reports a task it would change.
  • Tasks are self-contained Harbor task dirs, one folder per benchmark: tasks/<benchmark>/<task>/ (the folders: scripts/task_conventions.py BENCHMARK_TABLE; find a task dir with task_dir(name) / iter_tasks(), never by joining tasks/ and a name). Nothing at run time imports repository code outside the task directory. Reference tasks: tasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/ and its standard-mode twin tasks/robotwin-2/robotwin_2_0_stack_blocks_two-standard/.
  • Code several tasks share: recommended in the benchmark's base image (images/<sim>/); else embodied_first_exam/benchmarks/<benchmark>/ (the same folder name as tasks/<benchmark>/; tooling, graders, service adapter; embodied_first_exam/README.md). Its tasks get it from a files-only image (scripts/files_image.py, images/<name>/FILES.json): the tasks' Dockerfiles COPY --from=<image@digest> each file to the path a copy would have had (a sidecar's too: built from its own environment/sim/Dockerfile, which puts the directory and those files at /srv).
  • images/<sim>/Dockerfile: simulator base images, tagged from images/<sim>/VERSION if present else images/VERSION; scripts/build_images.sh. MuJoCo tasks (dextoolbench_*, mujoco_playground_*) use images/mujoco/ (run prepare_assets.sh first); notes in tasks/<benchmark>/README.md.
  • Every resource number is a cap for one container; the format, where each is written and what enforces it: PROTOCOL §6 (running.md "Resources" for runs). Edit the [[cap]] rules of scripts/task_resources.toml, never a task's caps or its ## Resources paragraph: normalize writes both. The monitor sidecar (images/monitor/rcb_monitor/, in the pinned ghcr.io/mll-lab-nu/efe:monitor-<version> image; normalize wires it into every task, which keeps only its limits.json; ci_checks/check-monitor.py runs its tests) changes with a new image version only. A CPU cap above the host's CPU count is the runner's to clamp (docker refuses it); the values in scripts/task_resources.toml stay. The same step's sanitize line in every tests/test.sh removes links and special files from /app/output before the grader reads the agent's files.
  • The agent's time cutoff is each task's own [agent] timeout_sec, set by its author in the pull request (rules: PROTOCOL §2; what normalize renders from it, ## Time limit last in every instruction.md, and how to change it: docs/protocol/running.md "The agent's time"). A benchmark's own text says "the time limit (## Time limit)", never a duration. Checks: ci_checks/check-cutoffs.py (advisory), the twins' equal number ci_checks/check-twins.py (H.7).
  • Seeds: [metadata] instance_seed is the default seed, [metadata.seeds] only where a task allows others (PROTOCOL §2; how a runner draws one: docs/protocol/running.md "Seeds"). Committed non-default instances live under the privileged task's tests/seeds/<seed>/ (never under environment/); ci_checks/check-seeds.py checks them.
  • CI has two tiers. ci_checks/run_invariants.sh [task dirs] (the required job invariants) runs only the checks that enforce PROTOCOL Appendix H and fails on any failure, error or incomplete run; ci_checks/run_all.sh [task dirs] runs it and then every advisory check (--tier advisory alone: exit 3 = warnings only). New per-task check = check-<thing>.sh <task_dir>, listed in ci_checks/check-task.sh under INVARIANT_CHECKS (it guards Appendix H) or ADVISORY_CHECKS; a benchmark-specific check is that benchmark's hook ci_checks/benchmarks/<bench>/invariant-* or advisory-* (shared ci_checks/ names no benchmark); an invariant rule also gets a mutation in ci_checks/check-ci-selftest.py (a real task broken that way must fail the rule; --tier invariants). Run the checks from a checkout made with umask 022 (run_all.sh sets it for itself). ci_checks/check-unit-tests.sh --all runs the service-layer tests that need numpy + requests (CI job unit-tests). Everything runs concurrently (ci_checks/parallel.py, RCB_CI_JOBS caps it). CI on a pull request runs run_all.sh --changed: the scope from ci_checks/ci_scope.py (one benchmark's own paths -> its tasks; shared code -> the full suite); pushes to main and the nightly run are the full suite. A benchmark's own paths come from scripts/task_conventions.py BENCHMARK_TABLE; a ci_scope.py UNITS row only names an image directory of another name (ci_scope.py --check).
  • tasks/<benchmark>/<task>/DIGEST: the task's content digest (PROTOCOL Appendix H.8), written last by normalize (after a hand edit: python3 scripts/task_conventions.py tasks/<benchmark>/<task>); ci_checks/check-dataset-manifest.sh recomputes it. The release manifest tasks/dataset.toml is generated (scripts/dataset_manifest.py write) and not committed, so pull requests do not conflict on it. The task list (TASKS.md, tasks.json, dataset.toml) is generated after every merge to main and pushed to the branch task-list (.github/workflows/task-list.yml; scripts/dataset_manifest.py list --out DIR locally).
  • Local docker sandbox has no [environment].gpus. RoboTwin and MuJoCo tasks pass a GPU through with environment/docker-compose.yaml (service main, nvidia device reservation; sim in standard mode) — copy it from tasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/.
  • Run agents with Harbor's native agents only (-a codex, -a claude-code); no adapters, and no gateway that holds credentials or changes requests. The usage gateway (scripts/usage_gateway/, GATEWAY=1 in run_agent_batch.sh; official runs use it) forwards model-API calls only (api_paths.json; anything else at a provider is a 403), records token usage, and is then, with the package registries, the agent container's only egress (docs/protocol/running.md "Official runs"). No tool that runs outside the container in official runs: the harness's web search is off (run_agent_batch.sh default WEB_SEARCH=disabled), as are account connectors, apps and plugins.
  • A task's agent egress: Appendix H.6 and docs/protocol/running.md "Network" ([agent] network_mode, "allowlist" or "public"); normalize ends every instruction.md with ## External material (PROTOCOL §5). Scene dynamics are each task's [metadata] dynamics (no central table). A task needs no download at run time: bake it into the image or the task directory. Nothing an agent can read (image, tooling, instance metadata) names or locates a solution. Unattended batches: scripts/run_agent_batch.sh (lanes with per-job claims; configured by environment variables, docs/protocol/running.md); its runner-side tools (log_mirror.py, mem_guard.py, scan_secrets.py, account_failure_check.py) are not task code.
  • eai-standard/2.2 (docs/protocol/standard_mode.md Part 2; which benchmarks serve it: docs/TAXONOMY.md protocol): its layer (rcb_limited.py with in-trial crash recovery, an implementation detail; the protocol's rule is the runner's infrastructure rerun, Appendix H.5, rcb_client.py, rcb1.py, run.sh) has one source, embodied_first_exam/standard/, copied unchanged into every 2.2 task (check-tooling-identity.sh); each benchmark adds only its adapter environment/sim/server.py (embodied_first_exam/standard/toy_server.py is the smallest one).
  • eai-standard/2.4 (docs/protocol/standard_mode.md Part 4, RoboARC): 2.3 + the operations capability. Its layer is embodied_first_exam/standard/v2_4/rcb_limited.py + rcb_client.py with 2.2's rcb1.py and run.sh, copied unchanged into every 2.4 task (check-tooling-identity.sh). python3 embodied_first_exam/standard/selftest.py tests the crash recovery on the CPU in about two minutes.
  • Benchmark-level rules (templates, budgets, tooling file names, the world-frame and tooling declarations): tasks/<benchmark>/README.md.