The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Commits on main
ABC Sim: benchmark label, CI scope row and resource rules (shared files, #473) (#479)
0ea668f Changhe Chen committed 29 days ago
Restore the AutoBio and LabUtopia benchmarks (all set only) (#478)
a93ad61 Kangrui Wang committed 29 days ago
Private base images build from public sources when they cannot be pulled; Before you run (+12 more) (#462, #463, #464, #465, #466, #467, ...)
e695c32 Kangrui Wang committed 1 month ago
Embodied Safety Exam: simulator image ese-0.5.6 from Embodied Safety Exam 0.5.6 (rename finished in the asset data; no source projection) (#438)
ce96d0f Canyu Chen committed 1 month ago
README: website and Discord badges, Contribute section (+42 more) (#390, #391, #392, #393, #395, #396, ...)
b2280d8 Kangrui Wang committed 1 month ago
Embodied Safety Exam: standard mode only, 76 tasks, core 50 (+1 more) (#388, #389)
f2aa80f Canyu Chen committed 1 month ago
RoboOlympics: two benchmarks, RoboOlympics-Fun and RoboOlympics-Match (board 0160) (+58 more) (#330, #331, #332, #333, #334, #335, ...)
55b312b Kangrui Wang committed 1 month ago
DexJoCo, DexHand: graded by the live episode, no grading replay (board 0151) (+18 more) (#324, #326, #314, #254, #316, #328, ...)
72140ff Kangrui Wang committed 1 month ago
RoboARC V2: 98 bases (knot-untying-06/09, blanket-flatten-07/08 new), 11 standard references within the joint-speed rule, buzz-wire-3d-03/04 references, image 0.2.0 (#301)
daa8939 Qineng Wang committed 1 month ago
Token budget 10B / 10B, cached input counted (owner Q12); a task may change in place, scene included (Q24) (+19 more) (#304, #303, #306, #305, #312, #310, ...)
0d799ad Kangrui Wang committed 1 month ago
RoboOlympics: withdraw the two skill-test tasks (tabletennis-hitskill-i00, badminton-actskill-i00) (#302)
b078f83 Chi Gui committed 1 month ago
Shared: IKEA MuJoCo README row, taxonomy, label, CI scope and hooks, resource rules, sub-image build (+1 more) (#298, #299)
bcad1dc Qineng Wang committed 1 month ago
RoboSafeBench 0.5.4: an 84-task core set, the 0.5.3 tasks in the easy set, rsb- task names (follows #252) (#296)
874f1ca Canyu Chen committed 1 month ago
RoboWits standard: one settle window in the grading replay (audit M5); step-cap end without a window (M6); move_cube / stand_pages texts (M10, M13) (+1 more) (#284, #283)
dc4b5d4 Qineng Wang committed 1 month ago
AM-Bench: 8 bases into the easy set (solved in every mode; names unchanged) (+10 more) (#286, #287, #288, #289, #285, #290, ...)
d0c69ce Changhe Chen committed 1 month ago
iSpatialGym: 12 spatial-intelligence tasks given only as a goal image, privileged and standard (eai-standard/2.2) (#280)
6f23617 Pingyue Zhang committed 1 month ago
DexJoCo: four tasks into the easy set (kept, off the main table; names unchanged) (+1 more) (#278, #277)
76bf140 Chi Gui committed 1 month ago
Shared: DexHand wave 2 README row, robot id and resource rules (updated 2026-10-15: wave-3 Marvin-arm rules) (+1 more) (#240, #241)
272970d Feishi Wang committed 1 month ago
Easy set: 84 task bases kept but outside the final statistics (central list docs/easy_set.toml + set marker) (#265)
0e04664 Qineng Wang committed 1 month ago
RoboOlympics: fencing bout exported by RoboOlympics' own exporter (the first turn-based exported matches, eai-standard-turn/1.0) (+1 more) (#257, #281)
96daa3d Changhe Chen committed 1 month ago
BiGym2 standard: reset budget 10 -> 0 (author decision 2026-10-16); the instruction says "one episode, no reset" (+3 more) (#274, #275, #276, #279)
d93e5d8 Pingyue Zhang committed 1 month ago
Rank validated and unvalidated tasks together (owner 10-16, board 0124) (+5 more) (#268, #269, #271, #270, #272, #273)
9923de3 Kangrui Wang committed 1 month ago
Review fixes after the 10-15 merge batch: AM-Bench standard hold padding and GPU-fault infra; RoboLab / RoboARC re-grade tools (board 0130) (+3 more) (#253, #255, #256, #266)
519ac7d Kangrui Wang committed 1 month ago
RoboSafeBench 0.5.3: the walking G1's room families, f_shutdown_request, the FR3 and two-arm widening, 14 realtime mover tasks, every task on eai-robosafe:0.5.3 (follows #223) (#252)
b11de29 Canyu Chen committed 1 month ago
Shared: caps row for the RoboWits blenderkit service (merge before the RoboWits grader fix) (+4 more) (#242, #243, #248, #247, #244)
f694250 Qineng Wang committed 1 month ago
Shared: AM-Bench's full set (README row, 2.2 in TAXONOMY, standard resource rules, conformance check for a service without joint_position) (+1 more) (#245, #246)
3fff248 Changhe Chen committed 1 month ago
Usage gateway health check and infra classification; task.toml presence check; cutoff pointers (board 0121) (+4 more) (#225, #226, #227, #238, #239)
26a9d51 Kangrui Wang committed 1 month ago
RoboSafeBench 0.5.2: the TidyBot's room families, new families and hidden tiers on the arms, every task on eai-robosafe:0.5.2 (follows #194) (#223)
4259fe6 Canyu Chen committed 1 month ago
Remove the AutoBio and LabUtopia benchmarks (owner 2026-10-14, board 0119) (+5 more) (#216, #212, #206, #220, #221, #224)
b6e76c1 Kangrui Wang committed 1 month ago