The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Commits on main
HumanCLAW: current Time limit text and standard-files 0.4.1 (#529)
ce9d0dd Kangrui Wang committed 13 days ago
README: teaser with the final task-set totals; intro video link on MLL-Lab/embodied-exam-data (#530)
9441d47 Kangrui Wang committed 13 days ago
Fix problems found on main: resets, leaks, grader bounds, stale texts (#526)
7acae4c Kangrui Wang committed 13 days ago
Add HumanCLAW: 55 episodes x privileged and standard (#528)
2bc0292 Kangrui Wang committed 13 days ago
Embodied Olympics - Match: every match's standard and privileged twins, G1 table tennis (#527)
ebd10b6 Kangrui Wang committed 13 days ago
Rename the mode standard-realtime to realtime (task protocol v2.3)
03fae2a jameskrw committed 14 days ago
protocol: default reset budget 0 in standard-realtime (same as standard)
9d846e6 jameskrw committed 15 days ago
Agent-ended episodes (optional, protocol v2.2); RoboDojo standard uses them with RoboDojo's step limit (#520)
9dfae86 Kangrui Wang committed 16 days ago
RoboLab standard: end the episode on the main thread (robolab-files 0.5.1) (#518)
149a383 Kangrui Wang committed 17 days ago
Usage gateway: skip blank lines in the container scan (#514)
8b172da Kangrui Wang committed 19 days ago
Per-call image budgets and single-copy response builder (#508)
65712d5 Kangrui Wang committed 21 days ago
Monitor 0.5.0 backstop, tightened resource caps, README tagline (#498)
9c66aaa Kangrui Wang committed 22 days ago
Resources as per-container caps, author fields removed, task fields de-duplicated (#497)
4c6d01b Kangrui Wang committed 22 days ago
Embodied Olympics - Fun release and task protocol 2.1 (#488)
b88c450 Kangrui Wang committed 25 days ago
pandadyn 0.2.0: stricter checks, tools joined to their adapters, all instances re-frozen (#482)
6847111 Changhe Chen committed 27 days ago
ABC Sim: 60 task directories and the abc-sim image (29 families; privileged, standard, 2 realtime twins; #473) (#480)
b9e50dc Changhe Chen committed 27 days ago