The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Contributing
Thank you for helping build Embodied First Exam (site, Discord). Tasks follow the contract in docs/protocol/README.md;
this page explains how a contribution gets from an idea to main. Reviewers follow docs/REVIEWING.md.
Everyone taking part follows the code of conduct; report security problems privately (SECURITY.md).
Pull requests can be opened by the repository's collaborators only. To contribute a benchmark or a task fix from outside, open an issue or propose it as the contribute page describes; a maintainer brings it in.
Kinds of pull requests
| Kind | What it may change | Before you start |
|---|---|---|
| New benchmark | tasks/<benchmark>/<its tasks>, images/<sim>/, tasks/<benchmark>/README.md (and, if needed, embodied_first_exam/benchmarks/<benchmark>/, ci_checks/benchmarks/<benchmark>/) |
an approved benchmark proposal |
| Tasks for an existing benchmark | the same, without images/ |
— |
| Task fix | the affected task directories | describe the agent behaviour it closes; include before/after runs |
| Shared files / protocol | docs/protocol/, docs/TAXONOMY.md, ci_checks/, .github/, shared scripts/, README.md |
a protocol-change issue for protocol changes |
One benchmark per pull request. A pull request that adds tasks and changes shared files is split before review.
A pull request delivers tasks only: task directories under tasks/<benchmark>/ (PROTOCOL §1, "What a contribution
delivers"). Scripts that build or generate tasks (generators, oracle searches, instance freezers, release builders)
stay outside this repository. Code several tasks share may go in embodied_first_exam/benchmarks/<benchmark>/; the
recommended place is the benchmark's image (images/<sim>/), so the tasks get it from the image.
Proposal (new benchmarks)
Open a benchmark-proposal issue first (or e-mail the proposal to [email protected], as the contribute page describes). A good proposal shows that the tasks are:
- verifiable by a program, in a fresh process, without a learned judge;
- solvable — a positive example the verifier (standard modes: the sidecar) scores 1: an oracle, an upstream expert or policy, a successful model trial, or a hand-built trajectory (tasks without one are human-reviewed and marked unvalidated, PROTOCOL §2, §4);
- difficult for today's coding agents, for a reason that matters for robotics;
- reproducible — a frozen scene instance; a privileged trajectory replays bit for bit (and the hardware it was checked on); a standard episode's recorded trajectory reproduces it for the video and audits (where the simulator replays bit for bit; else the video is recorded live);
- isolatable — the result is computed by a party the agent cannot reach: the verifier in its own sandbox from the declared artifacts (privileged), or the sim sidecar on the live episode after the agent's container stops (standard modes, PROTOCOL H.4);
- scrubbable — upstream reference solutions can be removed from everything the agent can reach.
It also states the oracle source (PROTOCOL §4), the caps of each container of a trial (CPU cores, memory, processes, GPU memory
and GPU type: RTX-class, any or none; §6), image size, whether the images can be published, and
the expected size of the pull request.
A new benchmark's shared-files change adds its row to scripts/task_conventions.py BENCHMARK_TABLE (task-directory
prefix, label, folder), and to ci_checks/ci_scope.py UNITS only for image directories of other names; every benchmark
has its own prefix (task_conventions.py benchmarks --check, ci_scope.py --check).
Authoring a task
Copy the closest existing pair (
tasks/robotwin-2/robotwin_2_0_stack_blocks_two-privileged/and-standard/), or generate tasks with your own tooling (kept outside this repository). Name them<base>-privileged/<base>-standard(and<base>-realtimefor the realtime mode, PROTOCOL §1, §3) and write the README in the standard layout (§1); there is noresults/directory.Fill in
task.toml(PROTOCOL §1, §2: its settings, among them[metadata] dynamicsand[agent] network_mode),instruction.md(§5, written by a human), the verifier undertests/(Appendix H.1, separate sandbox) andsolution/(H.2). A privileged task'stask.toml, with the fields you write (normalizeadds the resource monitor's artifact and hook,[verifier] network_modeand main's caps):schema_version = "1.3" artifacts = ["/app/output/"] # must include "/app/output/" (check-task-fields.sh) [task] name = "embodied-first-exam/mybench-pick-cube-i00-privileged" # embodied-first-exam/<directory name> [metadata] task_type = "trajectory" # or "policy" simulator = "mujoco" # simulator and robot: docs/TAXONOMY.md robot = "franka-panda" family = "pick_cube" dynamics = "static" # or "dynamic" (docs/TAXONOMY.md "Dynamics") oracle = "full" # "partial" / "none" also need oracle_evidence = "..." instance_seed = 0 # the default seed; [metadata.seeds] only for other seeds (PROTOCOL §2) expert_time_estimate_hours = 0.5 # a standard task adds protocol = "eai-standard/2.x" [verifier] timeout_sec = 1800 environment_mode = "separate" # Appendix H.1 [agent] timeout_sec = 3600 # the agent's cutoff (PROTOCOL §2) network_mode = "allowlist" # or "public": open egress, the author's choice, with no host listOn the allowlist,
python3 scripts/set_agent_egress.py allowlist <task dirs>writes[agent] allowed_hosts:check-verifier-isolation.shrequires exactly its list. The cutoff,[agent] timeout_sec, is yours to set in this pull request within PROTOCOL §2 and to state in its Contract; whatnormalizerenders from it and how the clock runs: docs/protocol/running.md "The agent's time". Your own text never repeats the limit as a duration: say "the time limit (## Time limit)", and add what is particular to your benchmark (a simulator's start-up cost). Your service'stime_s/time_left_scount from the agent's first request to it, never from the service's start, and a resumed service keeps that moment (docs/protocol/standard_mode.md, "The agent's clock": the shared layers do it for you; a test must show it).State the caps of every container: each number is an upper bound for one container, never a minimum. The rule format and where each cap ends up: PROTOCOL §6. Add a
[[cap]]rule toscripts/task_resources.toml; for a standard task with asimsidecar that holds the GPU:[[cap]] match = "^mybench-.*-standard$" # a regular expression on the task directory's name; the last match wins # (`-standard$` does not match `-realtime` tasks: give them a rule too) main = { cpus = 4, memory_gb = 3, pids = 1024, disk_gb = 10 } # the agent's container (cpus: a whole number) sim = { cpus = 4, memory_gb = 5, pids = 1024, gpu_memory_gb = 4 } # gpu_memory_gb: only on the service that holds the GPU recommended_gpu = "any 8 GB+ NVIDIA GPU (RTX 4090 / L4)" measured = "oracle trials on an RTX 4090: sim 1.9 GiB, 1.2 GiB of GPU memory; agent's container 0.1 GiB" caps_measured = "memory: the sim service 1.9 GiB -> max(1.5 x peak, peak + 3), rounded up: 5 GiB; the agent's container 0.1 GiB -> peak + 2, rounded up: 3 GiB; GPU memory (the sim service): 1.2 GiB -> max(1.5 x peak, peak + 2), rounded up: 4 GiB; processes and disk: no thread count measured, the minimums (1024, 10 GiB)"and for its privileged twin, where the agent's own container runs the simulator and holds the GPU:
[[cap]] match = "^mybench-.*-privileged$" main = { cpus = 8, memory_gb = 10, pids = 1024, disk_gb = 10, gpu_memory_gb = 4 } gpu_types = ["L40S", "L4", "RTX 4090"] # only if the task needs ray-tracing cores (RTX-class models); else leave it out caps_measured = "the agent's container: memory 6.2 GiB -> max(1.5 x peak, peak + 2), rounded up: 10 GiB; GPU memory 1.3 GiB -> max(1.5 x peak, peak + 2), rounded up: 4 GiB; 300 threads -> 1024 (2 x peak, at least 1024); disk: the minimum (10 GiB)"One table per compose service (the names must be the compose file's),
gpu_typesonly when the task needs an RTX-class GPU. The GPU itself is a device reservation you write in the compose file (deploy.resources.reservations. devices,driver: nvidia,capabilities: [gpu]); caps go incpus/mem_limit/pids_limit, never indeploy.resources.limits. A separate verifier's caps arecpus/memory_mbyou set in[verifier.environment](PROTOCOL §6). Measure the oracle (and agent trials if you can:docker stats/ cgroupmemory.peak,nvidia-smi), set the caps as PROTOCOL §6 "How a rule's caps are chosen" says (the peaks with a margin, and the largest call the task allows) and make sure the oracle stays inside them, unless the benchmark's README states an exception: a process over any cap but CPU can be killed (PROTOCOL §6). Then runscripts/task_conventions.pyon the tasks;check-resources.pychecks the result.Base images:
images/<sim>/Dockerfile, versioned inimages/<sim>/VERSION, built and pushed withscripts/build_images.sh/scripts/push_images.sh. Images that cannot be published carryimages/<sim>/BUILD_ONLYand build instructions. A task on a private image builds it itself when the runner cannot pull it: its Dockerfile starts with the recipe between# >>> images/<sim>/Dockerfileand# <<< images/<sim>/Dockerfile(the recipe's firstFROM ... AS <sim>-base, the task's own stageFROM <sim>-base;normalizewrites the current recipe there), and its compose service names the private image inbuild.cache_from(main:build:withcache_fromonly; a sidecar:build:withcontext/dockerfileinside environment/ andcache_from). The recipe needs no build context: every file it takes comes from a public files-only image (COPY --from=<image@digest>) or a public URL (ADD --checksum; a URL into this repository names a commit that a tag keeps reachable, since main is squashed and GitHub drops unreachable commits), system packages from a dated snapshot (e.g. snapshot.debian.org). The private image is that recipe built with BuildKit's inline cache (scripts/build_images.shadds it to every BUILD_ONLY image): a runner that can pull it takes its layers, any other runner builds the same layers from public sources.Run agents with Harbor's native agents only (
-a codex,-a claude-code, …). No adapters, and no gateway that holds credentials or changes requests; the optional usage gateway that only records token usage is allowed (docs/protocol/running.md "Official runs").
Before opening the pull request
bash ci_checks/run_invariants.sh tasks/<benchmark>/<your tasks> # must pass (the required CI job `invariants`)
bash ci_checks/run_all.sh tasks/<benchmark>/<your tasks> # both tiers: fix the advisory findings or say why the task departs
bash ci_checks/run_all.sh --changed # what CI runs on your pull request (scope: ci_checks/ci_scope.py)
python3 scripts/task_conventions.py --check tasks/<benchmark>/<your tasks> # README, task.toml, test.sh in the standard layout
python3 ci_checks/check-resources.py tasks/<benchmark>/<your tasks> # the caps of every container + README / instruction (PROTOCOL §6)
python3 ci_checks/check-cutoffs.py tasks/<benchmark>/<your tasks> # the time cutoff: task.toml = instruction = README = sim service, twins equal, cap (PROTOCOL §2)
python3 scripts/stamp_mtimes.py tasks # before any image build from the checkout (below)
harbor run -p tasks/<benchmark>/<task> -a oracle # on a sample: >= 2 tasks per mode
harbor run -p tasks/<benchmark>/<task> -a nop # must score 0
python3 scripts/task_conventions.py tasks/<benchmark>/<your tasks> # after a hand edit: normalize, which also refreshes each DIGEST
Never build a task image (a harbor run, a docker build of environment/ or tests/) from a tree you have not
stamped with scripts/stamp_mtimes.py: Docker's BuildKit reuses another build's same-name context file when its size and
mtime match, and a checkout gives many files one mtime, so a task can silently run or be graded with another task's file
(issue #258, docs/protocol/running.md "Building task images"). Tools that build images run from a stamped copy
(stamp_mtimes.py stage); a new one does the same.
For a new simulator or family, also replay the oracle (or a positive example) twice in fresh processes and include the matching state hashes (the privileged verdict replays; the standard modes' video is rendered from the trajectory).
Tasks without an oracle
Allowed (oracle = "partial" or "none" with oracle_evidence), but they need human review: the pull request is
labelled needs-human-review and is merged only after a maintainer other than the author approves it. Provide, per
family, a positive example the verifier scores 1, the nop result and a short description of the success predicate.
The checklist is in PROTOCOL §4.
Review and merge
- Static checks run on every pull request (
.github/workflows/static-checks.yml), on what the pull request can affect (ci_checks/ci_scope.py, from the diff againstmain): when it changes only one benchmark's own paths (itstasks/<benchmark>/,images/<bench>/,tasks/<benchmark>/README.md,ci_checks/benchmarks/<benchmark>/; plus the benchmarks that share code with it), the per-task checks (incl. each task'sDIGEST) andtask_conventions.py --checkon that benchmark's tasks (andcheck-cutoffs.pyon its tasks); the cheap repo-level checks (resources, protocol, ...) always run on every task. Anything shared --ci_checks/,.github/, the sharedscripts/modules and tables (task_conventions.py,task_resources.toml, ...),embodied_first_exam/standard/,images/monitor/,images/VERSION, the protocol documents (docs/protocol/), or a path the scope table does not know -- runs the full suite, as does every push tomain, the nightly run and a manual run (workflow_dispatch). The CI mutation self-test (check-ci-selftest.py) runs with the full suite. The checks are in two tiers (PROTOCOL Appendix H): the jobinvariants(ci_checks/run_invariants.sh: the checks that enforce PROTOCOL Appendix H) is the only one that blocks a merge, and it fails on any failure, error or incomplete run; the jobadvisories(ci_checks/run_all.sh --tier advisory) reports every other finding as a warning (annotations, the job summary). Locally,bash ci_checks/run_all.shruns both: exit 0 clean, 1 an invariant failed, 3 advisory findings only. - A sticky overview comment (posted by the job
advisories,ci_checks/pr_overview.py) lists the tasks the pull request adds or changes, the cutoffs it changes, the scope of its checks and the advisory findings ("Advisories (not blocking)": fix them or say why the task departs), and labels it (new-task,task-fix,shared,cutoff-change). A maintainer reviews the files and comments/validateto run the oracle (whereoracle = "full") andnopon the changed tasks (.github/workflows/validate-task.yml, on the GPU runner; the results come back as a comment; not working yet: it needs a dedicated runner with thedockerlabel, which is not attached -- until then the reviewer runs oracle andnopby hand), and, for new benchmarks, runs an adversarial ("cheat") trial and a model trial. - Merge requires: the required check
invariantspassing (branch protection), a maintainer approval (CODEOWNERS for shared files), human review where required, and a branch up to date withmain. - After merge, the task list is regenerated by
.github/workflows/task-list.ymland pushed to the branchtask-list(TASKS.md,tasks.json,dataset.toml; nothing inmainlists the tasks); maintainers publish base images and update the results site. A release (tagv<semver>) gets the same files as assets (.github/workflows/release.yml).
Rules that apply to every contribution
- A published task may be changed in place, its scene included: it keeps its name and gets a new
DIGEST(PROTOCOL H.8), and results are attributed by digest. Say what changed and why in the pull request; the README describes the task as it is. - To change a time cutoff (the task author's decision; the maintainers step in only where official results must stay
comparable), set the new
[agent] timeout_secin the tasks concerned, in both twins, runpython3 scripts/task_conventions.pyon them and say old -> new and why in the pull request (docs/protocol/running.md "The agent's time"). The task keeps its name and gets a new digest; results under the earlier cutoff are ranked separately (PROTOCOL §2, H.8). - A sidecar's directory (
environment/sim/,environment/scorer/, served at/srv) holds only what is the task's own: its frozen instance, per-task grader data, files no other task has. Files it shares with other tasks come from a pinned files-only image, the benchmark'simages/<folder>-files/(sources underembodied_first_exam/benchmarks/<folder>/; the same image serves its agent and verifier images; besides it, a benchmark may have files images that hold only data, such as goal pictures or build inputs, and one per further simulator image it runs on, as RoboARC's IKEA suite), orimages/standard-files/for the 2.2 / 2.4 / realtime layers: the sidecar is built from its directory (build: context: sim, itsDockerfile:COPY . /srv/, thenCOPY --from=<image@digest>of those files at their paths under/srv;.dockerignorekeeps the two build files out). A private evaluator's own data (hidden plants, case banks) and files that may not be redistributed stay in the task dir. - Pin everything (
FROMtags or digests, pip versions); no network access during verification. - The agent's egress (PROTOCOL Appendix H.6, docs/protocol/running.md "Network"): the allowlist by default; a task
whose author chooses open egress declares
[agent] network_mode = "public"and writes its own## External materialparagraph. Bake what the task itself needs at run time into the image or the task directory (an agent may install packages; a task must not depend on it).normalizewrites the## External materialparagraph, an honour rule, intoinstruction.md. If a distribution on PyPI ships your benchmark's solutions or task sources, add it to the table in docs/protocol/leakage.md ("Packages on PyPI"). Keep provenance (source repository, dataset, demonstration file, expert module) out of every file underenvironment/that the agent reads, and look through a new image for recorded episodes and links to them (ci_checks/check-agent-pointers.pycovers the task's own files only). - The agent is root in its container: assume it can modify anything under
/appand/opt. Grading therefore runs in a separate sandbox, and only the declared artifacts cross over (PROTOCOL Appendix H.1). A benchmark-specific check goes into the benchmark's ownci_checks/benchmarks/<bench>/as aninvariant-*hook (it guards Appendix H: blocking) or anadvisory-*hook (a convention: a warning); sharedci_checks/names no benchmark. - No credentials, personal absolute paths, large binaries or agent results in the repository (results go to the results site and the Hugging Face dataset).
- New Python code should be formatted with
ruff format(line length 120).