The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Embodied First Exam
Code agents have passed many exams in the digital world.
What is their first exam before entering the physical world?
Our nine original suites: RoboPaint · RoboARC · RoboDynamicExam · Embodied Olympics - Fun · Embodied Olympics - Match · DexHandExam · RoboEngineer · Embodied Safety Exam · iSpatialGym
▶ Watch the intro video
News
- 🔥 2026-10-19: Embodied First Exam is released!
Framework
In every task, a coding agent writes code, runs it on a robot in a frozen scene from one of the suites, and sees what happens.

Original suites
Nine of the suites were written for this exam:
| Suite | Description |
|---|---|
| RoboPaint | Tests fine tool control. A robot arm reproduces a given picture with a brush or pen, from Chinese calligraphy to oil painting. |
| RoboARC | Tests manipulation of tangled, deformable and multi-part objects. Robot arms take apart metal wire puzzles, tie and thread ropes, fold cloth and assemble IKEA furniture. |
| RoboDynamicExam | Tests handling of moving objects. One or two robot arms catch, strike, carry and hand over objects in motion: tossed balls, sliding pucks, swinging loads. |
| Embodied Olympics - Fun | Tests humanoid motor skills with fun sports-inspired tasks. The agent gets a humanoid robot to perform moves taken from many sports. |
| Embodied Olympics - Match | Tests competitive play against an opponent. Robots play full matches of table tennis, badminton, soccer, foosball and fencing against scripted or learned opponents. |
| DexHandExam | Tests dexterous hand manipulation. Five-finger robot hands turn book pages, open jars, stir coffee, plug in a USB stick, fold paper and build a hamburger. |
| RoboEngineer | Tests the sim-to-real engineering loop. The agent builds a controller in a public simulator, deploys it to one with hidden dynamics, and fixes it from what the deployment shows. |
| Embodied Safety Exam | Tests safety. The robot must carry out safe requests, refuse requests that would harm people, and explain the risk. |
| iSpatialGym | Tests spatial reasoning from images. The goal is given only as a picture: a mobile two-armed robot rearranges the room, or moves itself, until its camera view matches it. |
Quick start
Before you run
- Host: Linux x86_64 with git, uv (it fetches a Python 3.12-3.14 for Harbor if needed) and Docker Engine with Compose v2 and BuildKit (the
buildxplugin). Without Docker, e.g. on a Slurm cluster: rootless podman with-e podman(docs/protocol/running.md). - GPU: the NVIDIA driver and the NVIDIA Container Toolkit for every task that uses a GPU. An RTX-class GPU (with RT cores; not A100 or H100): AM-Bench, BEHAVIOR-1K, EmbodiedSWE, iSpatialGym, LabUtopia, ManiGuard, RoboARC, RoboDojo, RoboLab, YAMLab. Any NVIDIA GPU: ABC Sim, BiGym 2.0, DexToolBench, HumanCLAW, MolmoSpaces, Panda Dynamic Scenes, RoboPaint, RoboTwin 2.0, RoboWits. Some tasks: Embodied Olympics - Fun, Embodied Olympics - Match, HumanoidBench, KinDER, MuJoCo Playground, MyoSuite, roboEngineer. No GPU: AutoBio, DexHand, DexJoCo, Embodied Safety Exam, LIBERO-MAX, MetaWorld+, RoboCasa, RoboCasa-GR1, RoboCasa365, VLABench.
- Disk: a benchmark's public images take 1-30 GB, pulled on its first run; a benchmark that builds its image needs the free disk its row below states.
- API keys for the agent's model in
.env(step 1), or a subscription (step 3).
These benchmarks need a step by hand, or build their image on their first run because it may not be redistributed. That build accepts the licences of what it installs on your behalf: read them in the linked section. Organization members logged in to ghcr.io pull the prebuilt layers instead (on podman, after podman login ghcr.io, harbor run -e podman pulls the prebuilt image and uses it in place of the recipe; docs/protocol/running.md).
| Benchmark | By hand, once | First run |
|---|---|---|
| AM-Bench | - | Builds its image (accepts NVIDIA's Isaac Sim terms): about 25 min, 60 GB free disk. |
| AutoBio | - | Builds its image from AutoBio's repository, which states no licence (CPU only): a few minutes, 3 GB free disk. |
| BEHAVIOR-1K | Download the BEHAVIOR dataset (its downloader asks you to accept the BEHAVIOR Data Bundle licence; about 32 GB, 40 GB on disk) and export RCB_BEHAVIOR_DATA. |
Builds its image (accepts NVIDIA's Isaac Sim terms): about 20 min, 30 GB free disk. |
| HumanCLAW | Without access to the organization's packages: request access to the gated Hugging Face dataset HumanCLAW/HumanCLAW-HSSD, accept the hssd/hssd-hab terms, export HF_TOKEN and build its image once (HSSD is CC BY-NC 4.0; the body model and motion weights carry SMPL-X / AMASS research terms): about 30 min, 40 GB free disk. |
- |
| iSpatialGym | The BEHAVIOR-1K dataset, RCB_BEHAVIOR_DATA (the same directory). |
Builds its image (accepts NVIDIA's Isaac Sim terms): about 20 min, 31 GB free disk. |
| LabUtopia | - | Builds its image (accepts NVIDIA's Isaac Sim terms; LabUtopia's assets are CC BY-NC 4.0): about 25 min, 90 GB free disk. |
| LIBERO-MAX | A CPU with AVX-512 (Intel Ice Lake or newer Xeon, AMD Zen 4 or newer). | - |
| ManiGuard | Prepare the data root RCB_MANIGUARD_DATA (3.4 GB; its key asks you to accept the BEHAVIOR Data Bundle licence). NVIDIA driver 580.x: newer drivers crashed at start. |
Builds its image (accepts NVIDIA's Isaac Sim terms): about 20 min, 50 GB free disk. |
| RoboARC | NVIDIA driver 580 or newer, one GPU per container; knot_untying and rope_threading need a 24 GB GPU of their own. | - |
| RoboDojo | - | Builds its image with 41 GB of assets (accepts NVIDIA's Isaac Sim terms and cuRobo's): 30-90 min, 150 GB free disk. |
| roboEngineer | Build the evaluator images with its scripts; the eight Neural Fly tasks also need the force fit made locally (fetch_assets.sh --fit-fly). |
- |
| RoboLab | - | Builds its image (accepts NVIDIA's Isaac Sim terms): 20-60 min, 60 GB free disk. |
| RoboWits | 24 of its 30 tasks need BlenderKit models in RCB_ROBOWITS_BLENDERKIT: from the gated Hugging Face dataset (accept its terms, hf auth login) or with your own BlenderKit Full Plan key. |
- |
| VLABench | - | Builds its image (CPU only): 5-20 min, 40 GB free disk. |
| YAMLab | - | Builds its image (accepts NVIDIA's Isaac Sim terms): about 25 min, 60 GB free disk. |
1. Install
# clone with the submodules; install Harbor 0.23.0 with this repository's flags
git clone --recurse-submodules https://github.com/mll-lab-nu/embodied-first-exam.git
cd embodied-first-exam
uv tool install --force -e . --with-executables-from harbor
# your API keys go in .env
cp .env.example .env
# or without a clone: install from PyPI (or pip install embodied-first-exam);
# the tasks are cloned on first use; API keys: in the shell or a .env in the working directory
uv tool install embodied-first-exam --with-executables-from harbor
2. Run
# the core set: 456 tasks, each in its default mode
harbor run -d embodied-first-exam -a codex -m openai/gpt-6-astra
# the core tasks in another mode: --privileged (391 tasks), --standard, --realtime
harbor run -d embodied-first-exam --privileged -a codex -m openai/gpt-6-astra
# every task: 2398 task dirs (1211 tasks in their modes, 36 benchmarks)
harbor run -d embodied-first-exam@all -a codex -m openai/gpt-6-astra
# one benchmark, here RoboARC (212 task dirs)
harbor run -d embodied-first-exam@all -p tasks/roboarc -a codex -m openai/gpt-6-astra
# one task (-y skips Harbor's confirmation prompt, e.g. in a script)
harbor run -d embodied-first-exam@all -p tasks/<benchmark>/<task> -a oracle
3. Or use a subscription instead of an API key
# ChatGPT: log in once
codex login
CODEX_FORCE_AUTH_JSON=1 harbor run -d embodied-first-exam -a codex -m openai/gpt-6-astra
# Claude: put the token it prints in .env as CLAUDE_CODE_OAUTH_TOKEN
claude setup-token
CLAUDE_FORCE_OAUTH=1 harbor run -d embodied-first-exam \
-a claude-code -m anthropic/claude-opus-5-5
Contribute
See embodied-first-exam.ai/contribute and CONTRIBUTING.md.
Acknowledgements
Nine of the suites were written for this exam. The others are adapted from existing benchmarks, and their scenes, robots and success criteria are the ground the exam is built on. We are grateful to their authors:
- BEHAVIOR-1K: BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation (paper, code)
- RoboLab: RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies (paper, code)
- RoboCasa: RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (paper, code)
- RoboCasa365: RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots (paper, code)
- RoboCasa-GR1: GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (paper, code)
- MetaWorld+: Meta-World+: An Improved, Standardized, RL Benchmark (paper, code)
- VLABench: VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks (paper, code)
- HumanoidBench: HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation (paper, code)
- MolmoSpaces: MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation (paper, code)
- KinDER: KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning (paper, code)
- RoboTwin 2.0: RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (paper, code)
- MuJoCo Playground: MuJoCo Playground (paper, code)
- DexToolBench: SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation (paper, code)
- DexJoCo: DexJoCo: A Benchmark and Toolkit for Task-Oriented Dexterous Manipulation on MuJoCo (paper, code)
- MyoSuite: MyoSuite -- A contact-rich simulation suite for musculoskeletal motor control (paper, code)
- RoboDojo: RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies (paper, code)
- BiGym 2.0: BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation (paper, code)
- YAMLab (code)
- LIBERO-MAX: LIBERO-MAX: Do Robot Policies Adapt When the World Changes? (paper, code)
- AM-Bench: AM-Bench: A Modular Simulation Suite and Benchmark for Aerial Manipulation Policy Learning (paper, code)
- RoboWits: RoboWits: Unexpected Challenges for Robotic Creative Problem Solving (paper, code)
- ManiGuard: MANIGUARD: A Benchmark and Data Suite for Specification-Grounded Safety Evaluation and Improvement of Robotic Manipulation (paper, code)
- LabUtopia: LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agents (paper, code)
- AutoBio: AutoBio: A Simulation and Benchmark for Robotic Automation in Digital Biology Laboratory (paper, code)
License
MIT for the code in this repository. Third-party simulators, assets, reference solutions derived from upstream experts and the contents of the base images keep their own licenses.
Citation
@misc{embodiedfirstexam2026,
title = {Embodied First Exam},
author = {Kangrui Wang*§ and Qineng Wang*§ and Changhe Chen*§ and Pingyue Zhang*§ and Chi Gui*§ and Feishi Wang§ and Canyu Chen§ and Wenli Xiao§ and Xiaoyu Huang§ and Tonghe Zhang§ and Linjie Li§ and Zihan Wang and Minyeong Kim and Yiyou Sun and Mingkai Deng and Jinyu Hou and Eric P. Xing and Han Liu and Qi Zhu and Huaxiu Yao and Zhiwen Fan and Yuejiang Liu and Dawn Song and Pieter Abbeel and Li Fei-Fei and Jiajun Wu and Yuke Zhu and Guanya Shi† and Ruohan Zhang† and Manling Li†},
year = {2026},
url = {https://www.embodied-first-exam.ai}
}








