Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

NameLast commit date
..
robolab-apple-and-yogurt-in-bowl-i00-privileged
robolab-apple-and-yogurt-in-bowl-i00-standard
robolab-banana-in-bowl-i00-privileged
robolab-banana-in-bowl-i00-standard
robolab-banana-then-rubiks-cube-i00-privileged
robolab-banana-then-rubiks-cube-i00-standard
robolab-bananas-in-bin-one-more-i00-privileged
robolab-bananas-in-bin-one-more-i00-standard
robolab-bananas-in-bin-three-total-i00-privileged
robolab-bananas-in-bin-three-total-i00-standard
robolab-bananas-out-of-bin-i00-privileged
robolab-bananas-out-of-bin-i00-standard
robolab-bbq-sauce-in-bin-i00-privileged
robolab-bbq-sauce-in-bin-i00-standard
robolab-block-stacking-order-agnostic-i00-privileged
robolab-block-stacking-order-agnostic-i00-standard
robolab-block-stacking-specified-order-i00-privileged
robolab-block-stacking-specified-order-i00-standard
robolab-blocks-in-bin-i00-privileged
robolab-blocks-in-bin-i00-standard
robolab-butter-above-raisin-i00-privileged
robolab-butter-above-raisin-i00-standard
robolab-clutter-plastic-i00-privileged
robolab-clutter-plastic-i00-standard
robolab-clutter-pumpkin-i00-privileged
robolab-clutter-pumpkin-i00-standard
robolab-condiments-in-bin-i00-privileged
robolab-condiments-in-bin-i00-standard
robolab-cooking-clear-plate-i00-privileged
robolab-cooking-clear-plate-i00-standard
robolab-cubes-and-blocks-in-bin-i00-privileged
robolab-cubes-and-blocks-in-bin-i00-standard
robolab-food-packing2-cans-i00-privileged
robolab-food-packing2-cans-i00-standard
robolab-food-packing3-cans-i00-privileged
robolab-food-packing3-cans-i00-standard
robolab-fruits-green-limes-on-plate-i00-privileged
robolab-fruits-green-limes-on-plate-i00-standard
robolab-fruits-on-plate3-i00-privileged
robolab-fruits-on-plate3-i00-standard
robolab-keyboard-out-of-bin-i00-privileged
robolab-keyboard-out-of-bin-i00-standard
robolab-larger-object-raisin-box-in-bin-i00-privileged
robolab-larger-object-raisin-box-in-bin-i00-standard
robolab-marker-in-mug-i00-privileged
robolab-marker-in-mug-i00-standard
robolab-mustard-above-raisin-i00-privileged
robolab-mustard-above-raisin-i00-standard
robolab-mustard-in-left-bin-i00-privileged
robolab-mustard-in-left-bin-i00-standard
robolab-mustard-in-right-bin-i00-privileged
robolab-mustard-in-right-bin-i00-standard
robolab-pick-glasses-i00-privileged
robolab-pick-glasses-i00-standard
robolab-pick-orange-object-i00-privileged
robolab-pick-orange-object-i00-standard
robolab-pick-up-blue-pitcher-i00-privileged
robolab-pick-up-blue-pitcher-i00-standard
robolab-plastic-bottles-in-square-pail-i00-privileged
robolab-plastic-bottles-in-square-pail-i00-standard
robolab-put-two-mugs-on-shelf-i00-privileged
robolab-put-two-mugs-on-shelf-i00-standard
robolab-recycle-carton-i00-privileged
robolab-recycle-carton-i00-standard
robolab-recycle-cartons-vertical-crate-i00-privileged
robolab-recycle-cartons-vertical-crate-i00-standard
robolab-red-dishes-in-bin-i00-privileged
robolab-red-dishes-in-bin-i00-standard
robolab-reorient-red-mug-i00-privileged
robolab-reorient-red-mug-i00-standard
robolab-rubiks-cube-and-banana-i00-privileged
robolab-rubiks-cube-and-banana-i00-standard
robolab-rubiks-cube-behind-bowl-i00-privileged
robolab-rubiks-cube-behind-bowl-i00-standard
robolab-rubiks-cube-in-front-of-bowl-i00-privileged
robolab-rubiks-cube-in-front-of-bowl-i00-standard
robolab-rubiks-cube-left-of-bowl-i00-privileged
robolab-rubiks-cube-left-of-bowl-i00-standard
robolab-rubiks-cube-right-of-bowl-i00-privileged
robolab-rubiks-cube-right-of-bowl-i00-standard
robolab-rubiks-cube-then-banana-i00-privileged
robolab-rubiks-cube-then-banana-i00-standard
robolab-rubiks-cubes-in-bin-i00-privileged
robolab-rubiks-cubes-in-bin-i00-standard
robolab-spoon-in-mug-i00-privileged
robolab-spoon-in-mug-i00-standard
robolab-stack-yellow-on-red-i00-privileged
robolab-stack-yellow-on-red-i00-standard
robolab-stack3-rubiks-cube-i00-privileged
robolab-stack3-rubiks-cube-i00-standard
robolab-take-measuring-spoon-out-i00-privileged
robolab-take-measuring-spoon-out-i00-standard
robolab-take-mugs-off-of-shelf-i00-privileged
robolab-take-mugs-off-of-shelf-i00-standard
robolab-take-spatula-off-shelf-i00-privileged
robolab-take-spatula-off-shelf-i00-standard
robolab-throw-away-snacks-i00-privileged
robolab-throw-away-snacks-i00-standard
robolab-tools-picking-drill-i00-privileged
robolab-tools-picking-drill-i00-standard
robolab-unstack-rubiks-cube-i00-privileged
robolab-unstack-rubiks-cube-i00-standard
robolab-utensils-in-mug-i00-privileged
robolab-utensils-in-mug-i00-standard
robolab-white-mug-in-center-of-table-i00-privileged
robolab-white-mug-in-center-of-table-i00-standard
README.md