The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Add 12 EmbodiedSWE tasks to the core set (#519)
Kangrui Wangd95805a999e5
diff --git a/README.md b/README.mdindex b28952a1e2..8ea8c04fed 100644--- a/README.md+++ b/README.md@@ -131,12 +131,12 @@ uv tool install embodied-first-exam --with-executables-from harbor### 2. Run```bash-# the core set: 443 tasks, each in its default mode+# the core set: 455 tasks, each in its default modeharbor run -d embodied-first-exam -a codex -m openai/gpt-6-astra``````bash-# the core tasks in another mode: --privileged (367 tasks), --standard, --standard-realtime+# the core tasks in another mode: --privileged (379 tasks), --standard, --standard-realtimeharbor run -d embodied-first-exam --privileged -a codex -m openai/gpt-6-astra```diff --git a/docs/core_set.toml b/docs/core_set.tomlindex d6f6721e53..275dd15d92 100644--- a/docs/core_set.toml+++ b/docs/core_set.toml@@ -515,6 +515,54 @@ default_mode = "standard"task = "dextoolbench_spatula_spoon_spatula_serve_plate"default_mode = "standard"+[[task]]+task = "embodiedswe-assembly-bulb-franka-osc-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-assembly-ikea-table-bimanual-franka-osc-i00"+default_mode = "privileged"++[[task]]+task = "embodiedswe-assembly-pc-gpu-franka-osc-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-assembly-pc-gpu-ram-franka-osc-i00"+default_mode = "privileged"++[[task]]+task = "embodiedswe-assembly-pc-motherboard-franka-osc-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-cutting-dice-franka-osc-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-packing-egg-carton-g1-pink-ik-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-packing-tool-packing-franka-osc-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-puzzle-coffee-franka-osc-i00"+default_mode = "privileged"++[[task]]+task = "embodiedswe-puzzle-spatula-franka-osc-i00"+default_mode = "privileged"++[[task]]+task = "embodiedswe-puzzle-stack-blocks-g1-pink-ik-i00"+default_mode = "standard"++[[task]]+task = "embodiedswe-puzzle-syringe-bimanual-franka-osc-i00"+default_mode = "privileged"+[[task]]task = "ese-a01-i01"default_mode = "standard"diff --git a/tasks/embodiedswe/README.md b/tasks/embodiedswe/README.mdindex 86da092731..5ce675ce13 100644--- a/tasks/embodiedswe/README.md+++ b/tasks/embodiedswe/README.md@@ -4,7 +4,10 @@ EmbodiedSWE's robot benchmark (`robobench`, commit `67f44e8a2ca5`) on Isaac LabGPU: the agent writes a program, `solution/solve.py` exposing `solve(env)`, that completes a manipulation orloco-manipulation task by stepping the env. `simulator = "embodiedswe"`. 24 tasks (assembly, packing, puzzle, cuttingand loco-manipulation scenes, one frozen instance each): EmbodiedSWE's 28 published tasks without its 4 deformable ones-(they need Newton physics and a networked VLM judge). For now all 24 are in the `all` set and none is in the core set.+(they need Newton physics and a networked VLM judge). All 24 are in the `all` set; 12 are in the core set+(`docs/core_set.toml`), seven in standard mode (assembly-bulb, assembly-pc-gpu, assembly-pc-motherboard, cutting-dice,+packing-egg-carton, packing-tool-packing, puzzle-stack-blocks) and five in privileged mode (assembly-ikea-table-bimanual,+assembly-pc-gpu-ram, puzzle-coffee, puzzle-spatula, puzzle-syringe-bimanual).Each is in privileged mode (`task_type = "policy"`) and, as `<task>-standard`, in standard mode(`task_type = "trajectory"`, "Standard mode" below);all 24 scenes are static, so there is no standard-realtime mode.