The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Commits on main
RoboARC: draw the lock rotor discs where the physics has them (optional render fix, #210) (#218)
5dc5806 Qineng Wang committed 1 month ago
RoboOlympics: the tasks RoboOlympics exports itself (supersedes #180): grader fix, PROTOCOL §4c, cutoffs, seeds (owner E7, board 0111) (+1 more) (#202, #219)
caa9cf8 Kangrui Wang committed 1 month ago
RoboSafeBench 0.5.1: 294 standard and 15 standard-realtime tasks on three robots (UR5e, Franka FR3, two YAM arms), low-level control (#194)
049af57 Canyu Chen committed 1 month ago
DexHand: ten dexterous-hand tasks (Sharpa Wave, Wuji Hand 2.2), privileged and standard (20 task dirs) (#185)
9a2d4fa Feishi Wang committed 1 month ago
MuJoCo Playground: standard-realtime twin for BarkourJoystick (S = 1, 10 resets; board 0112) (+2 more) (#208, #205, #211)
2a1a577 Kangrui Wang committed 1 month ago
RoboARC 0.1.0: record the image's own Harbor results (oracle and nop evidence, READMEs re-rendered) (#203)
2f23694 Qineng Wang committed 1 month ago
CI: PRs check only what they can affect, in parallel; full suite on main (board 0117) (+1 more) (#214, #215)
633860c Kangrui Wang committed 1 month ago
AM-Bench: a first, privileged-only set of three aerial-manipulation tasks (placeholder) (#183)
1e0a648 Changhe Chen committed 1 month ago
MyoSuite table_tennis standard-realtime: the instruction says the realtime client sends float64 rows (it said float32, the 2.2 client's) (#213)
087bbeb Kangrui Wang committed 1 month ago
Shared: DexHand README row, taxonomy, label, checks and resource rules (#184)
8512f8b Feishi Wang committed 1 month ago
Remove RoboTangle's leftover cap exception and stale pointers (#209)
3138bd7 Kangrui Wang committed 1 month ago
Shared: AM-Bench README row, taxonomy, label, checks and resource rules (#182)
3f5801c Changhe Chen committed 1 month ago
DexJoCo: instance.npz no longer names its source demonstration; pointer check reads .npz; infra_error for instance mismatch (DexJoCo, Playground quadrupeds) (board 0105) (+8 more) (#190, #188, #197, #204, #200, #199, ...)
feb31cd Kangrui Wang committed 1 month ago
RoboARC shared files: rename RoboTangle to RoboARC in docs and checks (base of the RoboARC tasks PR) (+1 more) (#186, #187)
9974f0d Qineng Wang committed 1 month ago
Panda Dynamic Scenes: 25 multi-stage dynamic tasks, privileged and standard (50 task dirs) (#179)
a774cc8 Changhe Chen committed 1 month ago
Framework fixes from review F (board 0105): a success is never lost to a sidecar crash; no infra failure published as the agent's 0; CPU caps clamped by run_agent_batch.sh (#189)
8aba8e4 Kangrui Wang committed 1 month ago
Shared: Panda Dynamic Scenes README row, taxonomy, label, checks and resource rules (#178)
a58de3e Changhe Chen committed 1 month ago
RoboFunOlympics: 50 single-humanoid sports in official settings (100 tasks) (+10 more) (#161, #163, #165, #164, #167, #172, ...)
223c216 Kangrui Wang committed 1 month ago
CI: static-checks on the self-hosted rcb-ci runners (GitHub-hosted minutes ran out, 2026-10-12) (+3 more) (#160, #157, #159, #162)
64ecf9a Kangrui Wang committed 1 month ago
BiGym2, KinDER: standard tasks grant 10 resets by default (owner 2026-10-11) (#149)
a1c7728 Pingyue Zhang committed 1 month ago
RoboDojo: agent time limit one hour in both modes (owner 2026-10-11); every instruction states the cutoff (#155)
50bee2f Kangrui Wang committed 1 month ago
Shared: protocol exception for physics that does not replay bit for bit (standard only, graded by the live episode) (+1 more) (#153, #154)
ed6792f Changhe Chen committed 1 month ago
RoboTwin service test: wait for the port after the restart (CI flake) (+1 more) (#152, #151)
98c84e5 Kangrui Wang committed 1 month ago
Runner: Codex's provider-side tools (apps, plugins, image generation) off with the web search (+1 more) (#147, #148)
b64c562 Pingyue Zhang committed 1 month ago
Agent egress on the allowlist in every task; external-material rule in the protocol and every instruction; usage gateway forwards model-API calls only (board 0072) (#150)
f2117aa Kangrui Wang committed 1 month ago
RoboTwin standard instructions state the judging facts; protocol exception for RoboTangle safety readings (owner 2026-10-10) (#145)
aeb7f57 Kangrui Wang committed 1 month ago
RoboDojo: all 54 simulation tasks (42 + 12 randomised variants) × 2 modes (108 tasks) (#142)
01c2ae8 Changhe Chen committed 1 month ago
Review round 3 (board 0067): verdict-safe graders, request bounds in every service, give-up answers, CI holes (#144)
9887245 Kangrui Wang committed 1 month ago
Shared: RoboDojo README row, taxonomy, label, checks and resource rules (#141)
d7ad66c Changhe Chen committed 1 month ago
Shared: every task declares its minimum compute (task protocol v1.1.4) (+12 more) (#128, #129, #130, #131, #133, #134, ...)
5bab38c Kangrui Wang committed 1 month ago