Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

Add 12 EmbodiedSWE tasks to the core set (#519)

Kangrui Wangd95805a999e5
diff --git a/README.md b/README.md
index b28952a1e2..8ea8c04fed 100644
--- a/README.md
+++ b/README.md
@@ -131,12 +131,12 @@ uv tool install embodied-first-exam --with-executables-from harbor
### 2. Run
```bash
-# the core set: 443 tasks, each in its default mode
+# the core set: 455 tasks, each in its default mode
harbor run -d embodied-first-exam -a codex -m openai/gpt-6-astra
```
```bash
-# the core tasks in another mode: --privileged (367 tasks), --standard, --standard-realtime
+# the core tasks in another mode: --privileged (379 tasks), --standard, --standard-realtime
harbor run -d embodied-first-exam --privileged -a codex -m openai/gpt-6-astra
```
diff --git a/docs/core_set.toml b/docs/core_set.toml
index d6f6721e53..275dd15d92 100644
--- a/docs/core_set.toml
+++ b/docs/core_set.toml
@@ -515,6 +515,54 @@ default_mode = "standard"
task = "dextoolbench_spatula_spoon_spatula_serve_plate"
default_mode = "standard"
+[[task]]
+task = "embodiedswe-assembly-bulb-franka-osc-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-assembly-ikea-table-bimanual-franka-osc-i00"
+default_mode = "privileged"
+
+[[task]]
+task = "embodiedswe-assembly-pc-gpu-franka-osc-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-assembly-pc-gpu-ram-franka-osc-i00"
+default_mode = "privileged"
+
+[[task]]
+task = "embodiedswe-assembly-pc-motherboard-franka-osc-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-cutting-dice-franka-osc-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-packing-egg-carton-g1-pink-ik-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-packing-tool-packing-franka-osc-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-puzzle-coffee-franka-osc-i00"
+default_mode = "privileged"
+
+[[task]]
+task = "embodiedswe-puzzle-spatula-franka-osc-i00"
+default_mode = "privileged"
+
+[[task]]
+task = "embodiedswe-puzzle-stack-blocks-g1-pink-ik-i00"
+default_mode = "standard"
+
+[[task]]
+task = "embodiedswe-puzzle-syringe-bimanual-franka-osc-i00"
+default_mode = "privileged"
+
[[task]]
task = "ese-a01-i01"
default_mode = "standard"
diff --git a/tasks/embodiedswe/README.md b/tasks/embodiedswe/README.md
index 86da092731..5ce675ce13 100644
--- a/tasks/embodiedswe/README.md
+++ b/tasks/embodiedswe/README.md
@@ -4,7 +4,10 @@ EmbodiedSWE's robot benchmark (`robobench`, commit `67f44e8a2ca5`) on Isaac Lab
GPU: the agent writes a program, `solution/solve.py` exposing `solve(env)`, that completes a manipulation or
loco-manipulation task by stepping the env. `simulator = "embodiedswe"`. 24 tasks (assembly, packing, puzzle, cutting
and loco-manipulation scenes, one frozen instance each): EmbodiedSWE's 28 published tasks without its 4 deformable ones
-(they need Newton physics and a networked VLM judge). For now all 24 are in the `all` set and none is in the core set.
+(they need Newton physics and a networked VLM judge). All 24 are in the `all` set; 12 are in the core set
+(`docs/core_set.toml`), seven in standard mode (assembly-bulb, assembly-pc-gpu, assembly-pc-motherboard, cutting-dice,
+packing-egg-carton, packing-tool-packing, puzzle-stack-blocks) and five in privileged mode (assembly-ikea-table-bimanual,
+assembly-pc-gpu-ram, puzzle-coffee, puzzle-spatula, puzzle-syringe-bimanual).
Each is in privileged mode (`task_type = "policy"`) and, as `<task>-standard`, in standard mode
(`task_type = "trajectory"`, "Standard mode" below);
all 24 scenes are static, so there is no standard-realtime mode.