The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Commits on main
Shared: simulator `labutopia`, its field, the LabUtopia README label and checks (+3 more) (#95, #96, #93, #94)
55d30a5 Canyu Chen committed 1 month ago
CI: run MuJoCo Playground's quadruped generator check (#67)
cf6d488 Changhe Chen committed 1 month ago
BEHAVIOR-1K: a fall ends the episode as a failure (OmniGibson's Falling rule held for 10 s), both modes (#127)
39284dd Qineng Wang committed 1 month ago
MyoSuite standard: joint_effort from the physics, and a reset budget of 10 (task protocol v1.1.3) (+2 more) (#116, #125, #126)
9079325 Kangrui Wang committed 1 month ago
Shared: YAMLab README row, taxonomy, label and checks (+1 more) (#115, #117)
4c22bb9 Changhe Chen committed 1 month ago
Shared + HumanoidBench + RoboOlympics: the standard-realtime service serves requests while it is behind its clock (+1 more) (#120, #123)
c9afcde Pingyue Zhang committed 1 month ago
RoboPaint: prune the saturated targets, add 草书 caoshu and engraved round hand, finer oil paintings (-i01) (+1 more) (#118, #119)
eaa692d Qineng Wang committed 1 month ago
Shared: simulator `bigym2`, robot `unitree-g1-dex1`, the BiGym 2.0 README row, label and checks (for #121) (#122)
20cf70a Pingyue Zhang committed 1 month ago
dataset.toml: refresh the 64 LIBERO-MAX standard digests after #110 (#124)
84efa4d Changhe Chen committed 1 month ago
HumanoidBench: image 0.2.0 — MuJoCo 3.3.7 and three physics fixes (feet on boxes, room's reset, powerlift's mass), and hurdle's text corrected (#113)
d2793dc Pingyue Zhang committed 1 month ago
LIBERO-MAX: record the authors' written permission; cite the paper (#112)
7b61b28 Changhe Chen committed 1 month ago
LIBERO-MAX: grade a standard episode once its replay verdict is in, not once its video is rendered; a timeout is a failure, not infra (+1 more) (#110, #111)
32035cd Kangrui Wang committed 1 month ago
RoboTangle: image 0.2.22, frozen rope meshes for the IPC instances (knot_untying medium/hard on AVX2-only CPUs) (#109)
b175d2e Qineng Wang committed 1 month ago
LIBERO-MAX: image 0.1.1 pins OpenBLAS's Zen kernels so frozen instances rebuild on Intel Xeon 8488C (+1 more) (#107, #106)
a97b5c1 Kangrui Wang committed 1 month ago
RoboTangle: start frozen instances on CPUs without AVX-512 (fix for #99's InstanceMismatch on AWS L4/L40S) (#104)
ed8339a Qineng Wang committed 1 month ago
BEHAVIOR: scale the grader's per-replay time limit with the trajectory; raise verifier/collect timeouts (time limits only) (+2 more) (#101, #103, #105)
07d72fd Kangrui Wang committed 1 month ago
LIBERO-MAX: 64 cases with a mid-task event × 2 modes (128 tasks) (#50)
8ec7c1c Changhe Chen committed 1 month ago
RoboTangle: public image 0.2.21, one GPU per container, libuipc for sm_86, grader fix (follow-up to #81) (#99)
08aeb72 Qineng Wang committed 1 month ago
Shared + HumanoidBench: rename the third mode realtime -> standard-realtime (task protocol v1.1.1, #97) (#98)
17cccee Kangrui Wang committed 1 month ago
Shared: RoboTangle (formerly TopoGym) README row, taxonomy, label and checks (+1 more) (#80, #81)
1ad505f Qineng Wang committed 1 month ago
HumanoidBench: realtime mode -- 10 tasks <base>-realtime (eai-standard/2.3) where time does not stop (+2 more) (#83, #88, #90)
df02469 Kangrui Wang committed 1 month ago
BEHAVIOR-1K: the 23 remaining challenge tasks in both modes (100 families), human review (+1 more) (#75, #76)
549b6fe Qineng Wang committed 1 month ago
RoboCasa series: camera images without the collision geoms (#70)
0c75e9e Changhe Chen committed 1 month ago
BEHAVIOR: accept the Isaac EULA before the texture cap's import isaacsim (+6 more) (#69, #73, #77, #78, #84, #85, ...)
3a3a42a Kangrui Wang committed 1 month ago
Merge pull request #68 from JamesKrW/dev/qineng-robopaint (+2 more) (#71, #72)
63193f6 Qineng Wang committed 1 month ago
Merge pull request #66 from JamesKrW/perf/behavior-texture-budget
26482ad Kangrui Wang committed 1 month ago
Merge pull request #61 from JamesKrW/dev/chi-dexjoco-shared (+1 more)
20c0f65 Chi Gui committed 1 month ago
Merge pull request #58 from JamesKrW/dev/changhe-playground-quadruped-shared (+1 more)
f8f4009 Changhe Chen committed 1 month ago
Merge pull request #64 from JamesKrW/dev/chi-myosuite-shared (+1 more)
71a4cf0 Chi Gui committed 1 month ago
Merge pull request #49 from JamesKrW/dev/changhe-libero-max-label (+1 more)
c0f85b0 Changhe Chen committed 1 month ago