Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

NameLast commit date
..
abc-sim-ball-tray-i00-privileged
abc-sim-ball-tray-i00-realtime
abc-sim-ball-tray-i00-standard
abc-sim-chess-i00-privileged
abc-sim-chess-i00-standard
abc-sim-conveyor-pick-i00-privileged
abc-sim-conveyor-pick-i00-realtime
abc-sim-conveyor-pick-i00-standard
abc-sim-count-four-i00-privileged
abc-sim-count-four-i00-standard
abc-sim-count-i00-privileged
abc-sim-count-i00-standard
abc-sim-count-one-i00-privileged
abc-sim-count-one-i00-standard
abc-sim-count-three-i00-privileged
abc-sim-count-three-i00-standard
abc-sim-count-two-i00-privileged
abc-sim-count-two-i00-standard
abc-sim-dish-rack-i00-privileged
abc-sim-dish-rack-i00-standard
abc-sim-drawer-search-i00-privileged
abc-sim-drawer-search-i00-standard
abc-sim-grab-clutter-i00-privileged
abc-sim-grab-clutter-i00-standard
abc-sim-hang-mug-i00-privileged
abc-sim-hang-mug-i00-standard
abc-sim-inhand-transfer-i00-privileged
abc-sim-inhand-transfer-i00-standard
abc-sim-lego-i00-privileged
abc-sim-lego-i00-standard
abc-sim-nuts-bolts-i00-privileged
abc-sim-nuts-bolts-i00-standard
abc-sim-pouring-i00-privileged
abc-sim-pouring-i00-standard
abc-sim-put-bottles-i00-privileged
abc-sim-put-bottles-i00-standard
abc-sim-put-relative-i00-privileged
abc-sim-put-relative-i00-standard
abc-sim-spell-abc-i00-privileged
abc-sim-spell-abc-i00-standard
abc-sim-spell-agi-i00-privileged
abc-sim-spell-agi-i00-standard
abc-sim-spell-bair-i00-privileged
abc-sim-spell-bair-i00-standard
abc-sim-spell-cat-i00-privileged
abc-sim-spell-cat-i00-standard
abc-sim-spell-dog-i00-privileged
abc-sim-spell-dog-i00-standard
abc-sim-spell-fish-i00-privileged
abc-sim-spell-fish-i00-standard
abc-sim-spell-xdof-i00-privileged
abc-sim-spell-xdof-i00-standard
abc-sim-spell-yam-i00-privileged
abc-sim-spell-yam-i00-standard
abc-sim-sweep-i00-privileged
abc-sim-sweep-i00-standard
abc-sim-throw-bottles-i00-privileged
abc-sim-throw-bottles-i00-standard
abc-sim-turn-mug-i00-privileged
abc-sim-turn-mug-i00-standard
README.md