Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

19.9 KB
Raw Blame

Contributing

Thank you for helping build Embodied First Exam (site, Discord). Tasks follow the contract in docs/protocol/README.md; this page explains how a contribution gets from an idea to main. Reviewers follow docs/REVIEWING.md. Everyone taking part follows the code of conduct; report security problems privately (SECURITY.md).

Pull requests can be opened by the repository's collaborators only. To contribute a benchmark or a task fix from outside, open an issue or propose it as the contribute page describes; a maintainer brings it in.

Kinds of pull requests

Kind What it may change Before you start
New benchmark tasks/<benchmark>/<its tasks>, images/<sim>/, tasks/<benchmark>/README.md (and, if needed, embodied_first_exam/benchmarks/<benchmark>/, ci_checks/benchmarks/<benchmark>/) an approved benchmark proposal
Tasks for an existing benchmark the same, without images/ —
Task fix the affected task directories describe the agent behaviour it closes; include before/after runs
Shared files / protocol docs/protocol/, docs/TAXONOMY.md, ci_checks/, .github/, shared scripts/, README.md a protocol-change issue for protocol changes

One benchmark per pull request. A pull request that adds tasks and changes shared files is split before review.

A pull request delivers tasks only: task directories under tasks/<benchmark>/ (PROTOCOL §1, "What a contribution delivers"). Scripts that build or generate tasks (generators, oracle searches, instance freezers, release builders) stay outside this repository. Code several tasks share may go in embodied_first_exam/benchmarks/<benchmark>/; the recommended place is the benchmark's image (images/<sim>/), so the tasks get it from the image.

Proposal (new benchmarks)

Open a benchmark-proposal issue first (or e-mail the proposal to [email protected], as the contribute page describes). A good proposal shows that the tasks are:

  • verifiable by a program, in a fresh process, without a learned judge;
  • solvable — a positive example the verifier (standard modes: the sidecar) scores 1: an oracle, an upstream expert or policy, a successful model trial, or a hand-built trajectory (tasks without one are human-reviewed and marked unvalidated, PROTOCOL §2, §4);
  • difficult for today's coding agents, for a reason that matters for robotics;
  • reproducible — a frozen scene instance; a privileged trajectory replays bit for bit (and the hardware it was checked on); a standard episode's recorded trajectory reproduces it for the video and audits (where the simulator replays bit for bit; else the video is recorded live);
  • isolatable — the result is computed by a party the agent cannot reach: the verifier in its own sandbox from the declared artifacts (privileged), or the sim sidecar on the live episode after the agent's container stops (standard modes, PROTOCOL H.4);
  • scrubbable — upstream reference solutions can be removed from everything the agent can reach.

It also states the oracle source (PROTOCOL §4), the caps of each container of a trial (CPU cores, memory, processes, GPU memory and GPU type: RTX-class, any or none; §6), image size, whether the images can be published, and the expected size of the pull request. A new benchmark's shared-files change adds its row to scripts/task_conventions.py BENCHMARK_TABLE (task-directory prefix, label, folder), and to ci_checks/ci_scope.py UNITS only for image directories of other names; every benchmark has its own prefix (task_conventions.py benchmarks --check, ci_scope.py --check).

Authoring a task

  1. Copy the closest existing pair (tasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/ and -standard/), or generate tasks with your own tooling (kept outside this repository). Name them <base>-privileged / <base>-standard (and <base>-realtime for the realtime mode, PROTOCOL §1, §3) and write the README in the standard layout (§1); there is no results/ directory.

  2. Fill in task.toml (PROTOCOL §1, §2: its settings, among them [metadata] dynamics and [agent] network_mode), instruction.md (§5, written by a human), the verifier under tests/ (Appendix H.1, separate sandbox) and solution/ (H.2). A privileged task's task.toml, with the fields you write (normalize adds the resource monitor's artifact and hook, [verifier] network_mode and main's caps):

    schema_version = "1.3"
    artifacts = ["/app/output/"]          # must include "/app/output/" (check-task-fields.sh)
    
    [task]
    name = "embodied-first-exam/mybench-pick-cube-i00-privileged"   # embodied-first-exam/<directory name>
    
    [metadata]
    task_type = "trajectory"              # or "policy"
    simulator = "mujoco"                  # simulator and robot: docs/TAXONOMY.md
    robot = "franka-panda"
    family = "pick_cube"
    dynamics = "static"                   # or "dynamic" (docs/TAXONOMY.md "Dynamics")
    oracle = "full"                       # "partial" / "none" also need oracle_evidence = "..."
    instance_seed = 0                     # the default seed; [metadata.seeds] only for other seeds (PROTOCOL §2)
    expert_time_estimate_hours = 0.5      # a standard task adds protocol = "eai-standard/2.x"
    
    [verifier]
    timeout_sec = 1800
    environment_mode = "separate"         # Appendix H.1
    
    [agent]
    timeout_sec = 3600                    # the agent's cutoff (PROTOCOL §2)
    network_mode = "allowlist"            # or "public": open egress, the author's choice, with no host list
    

    On the allowlist, python3 scripts/set_agent_egress.py allowlist <task dirs> writes [agent] allowed_hosts: check-verifier-isolation.sh requires exactly its list. The cutoff, [agent] timeout_sec, is yours to set in this pull request within PROTOCOL §2 and to state in its Contract; what normalize renders from it and how the clock runs: docs/protocol/running.md "The agent's time". Your own text never repeats the limit as a duration: say "the time limit (## Time limit)", and add what is particular to your benchmark (a simulator's start-up cost). Your service's time_s / time_left_s count from the agent's first request to it, never from the service's start, and a resumed service keeps that moment (docs/protocol/standard_mode.md, "The agent's clock": the shared layers do it for you; a test must show it).

  3. State the caps of every container: each number is an upper bound for one container, never a minimum. The rule format and where each cap ends up: PROTOCOL §6. Add a [[cap]] rule to scripts/task_resources.toml; for a standard task with a sim sidecar that holds the GPU:

    [[cap]]
    match = "^mybench-.*-standard$"          # a regular expression on the task directory's name; the last match wins
                                             # (`-standard$` does not match `-realtime` tasks: give them a rule too)
    main = { cpus = 4, memory_gb = 3, pids = 1024, disk_gb = 10 }      # the agent's container (cpus: a whole number)
    sim = { cpus = 4, memory_gb = 5, pids = 1024, gpu_memory_gb = 4 }   # gpu_memory_gb: only on the service that holds the GPU
    recommended_gpu = "any 8 GB+ NVIDIA GPU (RTX 4090 / L4)"
    measured = "oracle trials on an RTX 4090: sim 1.9 GiB, 1.2 GiB of GPU memory; agent's container 0.1 GiB"
    caps_measured = "memory: the sim service 1.9 GiB -> max(1.5 x peak, peak + 3), rounded up: 5 GiB; the agent's container 0.1 GiB -> peak + 2, rounded up: 3 GiB; GPU memory (the sim service): 1.2 GiB -> max(1.5 x peak, peak + 2), rounded up: 4 GiB; processes and disk: no thread count measured, the minimums (1024, 10 GiB)"
    

    and for its privileged twin, where the agent's own container runs the simulator and holds the GPU:

    [[cap]]
    match = "^mybench-.*-privileged$"
    main = { cpus = 8, memory_gb = 10, pids = 1024, disk_gb = 10, gpu_memory_gb = 4 }
    gpu_types = ["L40S", "L4", "RTX 4090"]   # only if the task needs ray-tracing cores (RTX-class models); else leave it out
    caps_measured = "the agent's container: memory 6.2 GiB -> max(1.5 x peak, peak + 2), rounded up: 10 GiB; GPU memory 1.3 GiB -> max(1.5 x peak, peak + 2), rounded up: 4 GiB; 300 threads -> 1024 (2 x peak, at least 1024); disk: the minimum (10 GiB)"
    

    One table per compose service (the names must be the compose file's), gpu_types only when the task needs an RTX-class GPU. The GPU itself is a device reservation you write in the compose file (deploy.resources.reservations. devices, driver: nvidia, capabilities: [gpu]); caps go in cpus / mem_limit / pids_limit, never in deploy.resources.limits. A separate verifier's caps are cpus / memory_mb you set in [verifier.environment] (PROTOCOL §6). Measure the oracle (and agent trials if you can: docker stats / cgroup memory.peak, nvidia-smi), set the caps as PROTOCOL §6 "How a rule's caps are chosen" says (the peaks with a margin, and the largest call the task allows) and make sure the oracle stays inside them, unless the benchmark's README states an exception: a process over any cap but CPU can be killed (PROTOCOL §6). Then run scripts/task_conventions.py on the tasks; check-resources.py checks the result.

  4. Base images: images/<sim>/Dockerfile, versioned in images/<sim>/VERSION, built and pushed with scripts/build_images.sh / scripts/push_images.sh. Images that cannot be published carry images/<sim>/BUILD_ONLY and build instructions. A task on a private image builds it itself when the runner cannot pull it: its Dockerfile starts with the recipe between # >>> images/<sim>/Dockerfile and # <<< images/<sim>/Dockerfile (the recipe's first FROM ... AS <sim>-base, the task's own stage FROM <sim>-base; normalize writes the current recipe there), and its compose service names the private image in build.cache_from (main: build: with cache_from only; a sidecar: build: with context / dockerfile inside environment/ and cache_from). The recipe needs no build context: every file it takes comes from a public files-only image (COPY --from=<image@digest>) or a public URL (ADD --checksum; a URL into this repository names a commit that a tag keeps reachable, since main is squashed and GitHub drops unreachable commits), system packages from a dated snapshot (e.g. snapshot.debian.org). The private image is that recipe built with BuildKit's inline cache (scripts/build_images.sh adds it to every BUILD_ONLY image): a runner that can pull it takes its layers, any other runner builds the same layers from public sources.

  5. Run agents with Harbor's native agents only (-a codex, -a claude-code, …). No adapters, and no gateway that holds credentials or changes requests; the optional usage gateway that only records token usage is allowed (docs/protocol/running.md "Official runs").

Before opening the pull request

bash ci_checks/run_invariants.sh tasks/<benchmark>/<your tasks>   # must pass (the required CI job `invariants`)
bash ci_checks/run_all.sh tasks/<benchmark>/<your tasks>      # both tiers: fix the advisory findings or say why the task departs
bash ci_checks/run_all.sh --changed               # what CI runs on your pull request (scope: ci_checks/ci_scope.py)
python3 scripts/task_conventions.py --check tasks/<benchmark>/<your tasks>     # README, task.toml, test.sh in the standard layout
python3 ci_checks/check-resources.py tasks/<benchmark>/<your tasks>             # the caps of every container + README / instruction (PROTOCOL §6)
python3 ci_checks/check-cutoffs.py tasks/<benchmark>/<your tasks>               # the time cutoff: task.toml = instruction = README = sim service, twins equal, cap (PROTOCOL §2)
python3 scripts/stamp_mtimes.py tasks             # before any image build from the checkout (below)
harbor run -p tasks/<benchmark>/<task> -a oracle               # on a sample: >= 2 tasks per mode
harbor run -p tasks/<benchmark>/<task> -a nop                  # must score 0
python3 scripts/task_conventions.py tasks/<benchmark>/<your tasks>    # after a hand edit: normalize, which also refreshes each DIGEST

Never build a task image (a harbor run, a docker build of environment/ or tests/) from a tree you have not stamped with scripts/stamp_mtimes.py: Docker's BuildKit reuses another build's same-name context file when its size and mtime match, and a checkout gives many files one mtime, so a task can silently run or be graded with another task's file (issue #258, docs/protocol/running.md "Building task images"). Tools that build images run from a stamped copy (stamp_mtimes.py stage); a new one does the same.

For a new simulator or family, also replay the oracle (or a positive example) twice in fresh processes and include the matching state hashes (the privileged verdict replays; the standard modes' video is rendered from the trajectory).

Tasks without an oracle

Allowed (oracle = "partial" or "none" with oracle_evidence), but they need human review: the pull request is labelled needs-human-review and is merged only after a maintainer other than the author approves it. Provide, per family, a positive example the verifier scores 1, the nop result and a short description of the success predicate. The checklist is in PROTOCOL §4.

Review and merge

  1. Static checks run on every pull request (.github/workflows/static-checks.yml), on what the pull request can affect (ci_checks/ci_scope.py, from the diff against main): when it changes only one benchmark's own paths (its tasks/<benchmark>/, images/<bench>/, tasks/<benchmark>/README.md, ci_checks/benchmarks/<benchmark>/; plus the benchmarks that share code with it), the per-task checks (incl. each task's DIGEST) and task_conventions.py --check on that benchmark's tasks (and check-cutoffs.py on its tasks); the cheap repo-level checks (resources, protocol, ...) always run on every task. Anything shared -- ci_checks/, .github/, the shared scripts/ modules and tables (task_conventions.py, task_resources.toml, ...), embodied_first_exam/standard/, images/monitor/, images/VERSION, the protocol documents (docs/protocol/), or a path the scope table does not know -- runs the full suite, as does every push to main, the nightly run and a manual run (workflow_dispatch). The CI mutation self-test (check-ci-selftest.py) runs with the full suite. The checks are in two tiers (PROTOCOL Appendix H): the job invariants (ci_checks/run_invariants.sh: the checks that enforce PROTOCOL Appendix H) is the only one that blocks a merge, and it fails on any failure, error or incomplete run; the job advisories (ci_checks/run_all.sh --tier advisory) reports every other finding as a warning (annotations, the job summary). Locally, bash ci_checks/run_all.sh runs both: exit 0 clean, 1 an invariant failed, 3 advisory findings only.
  2. A sticky overview comment (posted by the job advisories, ci_checks/pr_overview.py) lists the tasks the pull request adds or changes, the cutoffs it changes, the scope of its checks and the advisory findings ("Advisories (not blocking)": fix them or say why the task departs), and labels it (new-task, task-fix, shared, cutoff-change). A maintainer reviews the files and comments /validate to run the oracle (where oracle = "full") and nop on the changed tasks (.github/workflows/validate-task.yml, on the GPU runner; the results come back as a comment; not working yet: it needs a dedicated runner with the docker label, which is not attached -- until then the reviewer runs oracle and nop by hand), and, for new benchmarks, runs an adversarial ("cheat") trial and a model trial.
  3. Merge requires: the required check invariants passing (branch protection), a maintainer approval (CODEOWNERS for shared files), human review where required, and a branch up to date with main.
  4. After merge, the task list is regenerated by .github/workflows/task-list.yml and pushed to the branch task-list (TASKS.md, tasks.json, dataset.toml; nothing in main lists the tasks); maintainers publish base images and update the results site. A release (tag v<semver>) gets the same files as assets (.github/workflows/release.yml).

Rules that apply to every contribution

  • A published task may be changed in place, its scene included: it keeps its name and gets a new DIGEST (PROTOCOL H.8), and results are attributed by digest. Say what changed and why in the pull request; the README describes the task as it is.
  • To change a time cutoff (the task author's decision; the maintainers step in only where official results must stay comparable), set the new [agent] timeout_sec in the tasks concerned, in both twins, run python3 scripts/task_conventions.py on them and say old -> new and why in the pull request (docs/protocol/running.md "The agent's time"). The task keeps its name and gets a new digest; results under the earlier cutoff are ranked separately (PROTOCOL §2, H.8).
  • A sidecar's directory (environment/sim/, environment/scorer/, served at /srv) holds only what is the task's own: its frozen instance, per-task grader data, files no other task has. Files it shares with other tasks come from a pinned files-only image, the benchmark's images/<folder>-files/ (sources under embodied_first_exam/benchmarks/<folder>/; the same image serves its agent and verifier images; besides it, a benchmark may have files images that hold only data, such as goal pictures or build inputs, and one per further simulator image it runs on, as RoboARC's IKEA suite), or images/standard-files/ for the 2.2 / 2.4 / realtime layers: the sidecar is built from its directory (build: context: sim, its Dockerfile: COPY . /srv/, then COPY --from=<image@digest> of those files at their paths under /srv; .dockerignore keeps the two build files out). A private evaluator's own data (hidden plants, case banks) and files that may not be redistributed stay in the task dir.
  • Pin everything (FROM tags or digests, pip versions); no network access during verification.
  • The agent's egress (PROTOCOL Appendix H.6, docs/protocol/running.md "Network"): the allowlist by default; a task whose author chooses open egress declares [agent] network_mode = "public" and writes its own ## External material paragraph. Bake what the task itself needs at run time into the image or the task directory (an agent may install packages; a task must not depend on it). normalize writes the ## External material paragraph, an honour rule, into instruction.md. If a distribution on PyPI ships your benchmark's solutions or task sources, add it to the table in docs/protocol/leakage.md ("Packages on PyPI"). Keep provenance (source repository, dataset, demonstration file, expert module) out of every file under environment/ that the agent reads, and look through a new image for recorded episodes and links to them (ci_checks/check-agent-pointers.py covers the task's own files only).
  • The agent is root in its container: assume it can modify anything under /app and /opt. Grading therefore runs in a separate sandbox, and only the declared artifacts cross over (PROTOCOL Appendix H.1). A benchmark-specific check goes into the benchmark's own ci_checks/benchmarks/<bench>/ as an invariant-* hook (it guards Appendix H: blocking) or an advisory-* hook (a convention: a warning); shared ci_checks/ names no benchmark.
  • No credentials, personal absolute paths, large binaries or agent results in the repository (results go to the results site and the Hugging Face dataset).
  • New Python code should be formatted with ruff format (line length 120).