Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

NameLast commit date
..
behavior-assembling-gift-baskets-i00-privileged
behavior-assembling-gift-baskets-i00-standard
behavior-attach-a-camera-to-a-tripod-i00-privileged
behavior-attach-a-camera-to-a-tripod-i00-standard
behavior-boxing-books-up-for-storage-i00-privileged
behavior-boxing-books-up-for-storage-i00-standard
behavior-boxing-food-after-dinner-i00-privileged
behavior-boxing-food-after-dinner-i00-standard
behavior-bringing-in-wood-i00-privileged
behavior-bringing-in-wood-i00-standard
behavior-bringing-paper-to-recycling-i00-privileged
behavior-bringing-paper-to-recycling-i00-standard
behavior-bringing-water-i00-privileged
behavior-bringing-water-i00-standard
behavior-can-meat-i00-privileged
behavior-can-meat-i00-standard
behavior-canning-food-i00-privileged
behavior-canning-food-i00-standard
behavior-carrying-in-groceries-i00-privileged
behavior-carrying-in-groceries-i00-standard
behavior-carrying-out-garden-furniture-i00-privileged
behavior-carrying-out-garden-furniture-i00-standard
behavior-chop-an-onion-i00-privileged
behavior-chop-an-onion-i00-standard
behavior-chopping-wood-i00-privileged
behavior-chopping-wood-i00-standard
behavior-clean-a-keyboard-i00-privileged
behavior-clean-a-keyboard-i00-standard
behavior-clean-a-patio-i00-privileged
behavior-clean-a-patio-i00-standard
behavior-clean-a-trumpet-i00-privileged
behavior-clean-a-trumpet-i00-standard
behavior-clean-boxing-gloves-i00-privileged
behavior-clean-boxing-gloves-i00-standard
behavior-clean-up-broken-glass-i00-privileged
behavior-clean-up-broken-glass-i00-standard
behavior-clean-up-your-desk-i00-privileged
behavior-clean-up-your-desk-i00-standard
behavior-clean-your-rusty-garden-tools-i00-privileged
behavior-clean-your-rusty-garden-tools-i00-standard
behavior-cleaning-up-branches-and-twigs-i00-privileged
behavior-cleaning-up-branches-and-twigs-i00-standard
behavior-cleaning-up-plates-and-food-i00-privileged
behavior-cleaning-up-plates-and-food-i00-standard
behavior-clearing-food-from-table-into-fridge-i00-privileged
behavior-clearing-food-from-table-into-fridge-i00-standard
behavior-collecting-aluminum-cans-i00-privileged
behavior-collecting-aluminum-cans-i00-standard
behavior-collecting-childrens-toys-i00-privileged
behavior-collecting-childrens-toys-i00-standard
behavior-composting-waste-i00-privileged
behavior-composting-waste-i00-standard
behavior-cook-a-brisket-i00-privileged
behavior-cook-a-brisket-i00-standard
behavior-cook-a-frozen-pie-i00-privileged
behavior-cook-a-frozen-pie-i00-standard
behavior-cook-bacon-i00-privileged
behavior-cook-bacon-i00-standard
behavior-cook-broccolini-i00-privileged
behavior-cook-broccolini-i00-standard
behavior-cook-brussels-sprouts-i00-privileged
behavior-cook-brussels-sprouts-i00-standard
behavior-cook-cabbage-i00-privileged
behavior-cook-cabbage-i00-standard
behavior-cook-hot-dogs-i00-privileged
behavior-cook-hot-dogs-i00-standard
behavior-dispose-of-batteries-i00-privileged
behavior-dispose-of-batteries-i00-standard
behavior-dispose-of-glass-i00-privileged
behavior-dispose-of-glass-i00-standard
behavior-freeze-fruit-i00-privileged
behavior-freeze-fruit-i00-standard
behavior-freeze-pies-i00-privileged
behavior-freeze-pies-i00-standard
behavior-getting-organized-for-work-i00-privileged
behavior-getting-organized-for-work-i00-standard
behavior-halve-an-egg-i00-privileged
behavior-halve-an-egg-i00-standard
behavior-hanging-pictures-i00-privileged
behavior-hanging-pictures-i00-standard
behavior-hiding-Easter-eggs-i00-privileged
behavior-hiding-Easter-eggs-i00-standard
behavior-installing-a-fax-machine-i00-privileged
behavior-installing-a-fax-machine-i00-standard
behavior-installing-a-modem-i00-privileged
behavior-installing-a-modem-i00-standard
behavior-installing-a-scanner-i00-privileged
behavior-installing-a-scanner-i00-standard
behavior-installing-smoke-detectors-i00-privileged
behavior-installing-smoke-detectors-i00-standard
behavior-laying-tile-floors-i00-privileged
behavior-laying-tile-floors-i00-standard
behavior-loading-the-car-i00-privileged
behavior-loading-the-car-i00-standard
behavior-make-cabinet-doors-i00-privileged
behavior-make-cabinet-doors-i00-standard
behavior-make-gift-bags-for-baby-showers-i00-privileged
behavior-make-gift-bags-for-baby-showers-i00-standard
behavior-make-microwave-popcorn-i00-privileged
behavior-make-microwave-popcorn-i00-standard
behavior-make-pizza-i00-privileged
behavior-make-pizza-i00-standard
behavior-make-rose-centerpieces-i00-privileged
behavior-make-rose-centerpieces-i00-standard
behavior-moving-boxes-to-storage-i00-privileged
behavior-moving-boxes-to-storage-i00-standard
behavior-organizing-art-supplies-i00-privileged
behavior-organizing-art-supplies-i00-standard
behavior-organizing-school-stuff-i00-privileged
behavior-organizing-school-stuff-i00-standard
behavior-outfit-a-basic-toolbox-i00-privileged
behavior-outfit-a-basic-toolbox-i00-standard
behavior-packing-meal-for-delivery-i00-privileged
behavior-packing-meal-for-delivery-i00-standard
behavior-picking-up-toys-i00-privileged
behavior-picking-up-toys-i00-standard
behavior-picking-up-trash-i00-privileged
behavior-picking-up-trash-i00-standard
behavior-polishing-shoes-i00-privileged
behavior-polishing-shoes-i00-standard
behavior-preparing-lunch-box-i00-privileged
behavior-preparing-lunch-box-i00-standard
behavior-put-together-a-basic-pruning-kit-i00-privileged
behavior-put-together-a-basic-pruning-kit-i00-standard
behavior-putting-away-Halloween-decorations-i00-privileged
behavior-putting-away-Halloween-decorations-i00-standard
behavior-putting-away-toys-i00-privileged
behavior-putting-away-toys-i00-standard
behavior-putting-dirty-dishes-in-sink-i00-privileged
behavior-putting-dirty-dishes-in-sink-i00-standard
behavior-putting-dishes-away-after-cleaning-i00-privileged
behavior-putting-dishes-away-after-cleaning-i00-standard
behavior-putting-shoes-on-rack-i00-privileged
behavior-putting-shoes-on-rack-i00-standard
behavior-putting-up-Christmas-decorations-inside-i00-privileged
behavior-putting-up-Christmas-decorations-inside-i00-standard
behavior-re-shelving-library-books-i00-privileged
behavior-re-shelving-library-books-i00-standard
behavior-rearrange-your-room-i00-privileged
behavior-rearrange-your-room-i00-standard
behavior-rearranging-kitchen-furniture-i00-privileged
behavior-rearranging-kitchen-furniture-i00-standard
behavior-scrubbing-bathroom-floor-i00-privileged
behavior-scrubbing-bathroom-floor-i00-standard
behavior-set-up-a-coffee-station-in-your-kitchen-i00-privileged
behavior-set-up-a-coffee-station-in-your-kitchen-i00-standard
behavior-setting-mousetraps-i00-privileged
behavior-setting-mousetraps-i00-standard
behavior-setting-the-fire-i00-privileged
behavior-setting-the-fire-i00-standard
behavior-setting-the-table-i00-privileged
behavior-setting-the-table-i00-standard
behavior-setup-a-bar-for-a-cocktail-party-i00-privileged
behavior-setup-a-bar-for-a-cocktail-party-i00-standard
behavior-slicing-vegetables-i00-privileged
behavior-slicing-vegetables-i00-standard
behavior-sorting-books-on-shelf-i00-privileged
behavior-sorting-books-on-shelf-i00-standard
behavior-sorting-bottles-cans-and-paper-i00-privileged
behavior-sorting-bottles-cans-and-paper-i00-standard
behavior-sorting-household-items-i00-privileged
behavior-sorting-household-items-i00-standard
behavior-sorting-vegetables-i00-privileged
behavior-sorting-vegetables-i00-standard
behavior-spraying-for-bugs-i00-privileged
behavior-spraying-for-bugs-i00-standard
behavior-spraying-fruit-trees-i00-privileged
behavior-spraying-fruit-trees-i00-standard
behavior-stacking-wood-i00-privileged
behavior-stacking-wood-i00-standard
behavior-store-batteries-i00-privileged
behavior-store-batteries-i00-standard
behavior-store-honey-i00-privileged
behavior-store-honey-i00-standard
behavior-store-produce-i00-privileged
behavior-store-produce-i00-standard
behavior-storing-food-i00-privileged
behavior-storing-food-i00-standard
behavior-sweeping-garage-i00-privileged
behavior-sweeping-garage-i00-standard
behavior-thawing-frozen-food-i00-privileged
behavior-thawing-frozen-food-i00-standard
behavior-tidying-bathroom-i00-privileged
behavior-tidying-bathroom-i00-standard
behavior-tidying-bedroom-i00-privileged
behavior-tidying-bedroom-i00-standard
behavior-tidying-living-room-i00-privileged
behavior-tidying-living-room-i00-standard
behavior-turning-on-radio-i00-privileged
behavior-turning-on-radio-i00-standard
behavior-turning-out-all-lights-before-sleep-i00-privileged
behavior-turning-out-all-lights-before-sleep-i00-standard
behavior-unloading-the-car-i00-privileged
behavior-unloading-the-car-i00-standard
behavior-vacuuming-floors-i00-privileged
behavior-vacuuming-floors-i00-standard
behavior-wash-a-baseball-cap-i00-privileged
behavior-wash-a-baseball-cap-i00-standard
behavior-wash-dog-toys-i00-privileged
behavior-wash-dog-toys-i00-standard
README.md