Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

NameLast commit date
..
vlabench-add-condiment-i00-privileged
vlabench-add-condiment-i00-standard
vlabench-book-rearrange-i00-privileged
vlabench-book-rearrange-i00-standard
vlabench-find-unseen-object-i00-privileged
vlabench-find-unseen-object-i00-standard
vlabench-friction-qa-i00-privileged
vlabench-friction-qa-i00-standard
vlabench-hammer-nail-and-hang-picture-i00-privileged
vlabench-hammer-nail-and-hang-picture-i00-standard
vlabench-heat-food-i00-privileged
vlabench-heat-food-i00-standard
vlabench-insert-bloom-flower-i00-privileged
vlabench-insert-bloom-flower-i00-standard
vlabench-insert-flower-i00-privileged
vlabench-insert-flower-i00-standard
vlabench-magnetism-qa-i00-privileged
vlabench-magnetism-qa-i00-standard
vlabench-play-math-game-i00-privileged
vlabench-play-math-game-i00-standard
vlabench-put-box-on-painting-i00-privileged
vlabench-put-box-on-painting-i00-standard
vlabench-reflection-qa-i00-privileged
vlabench-reflection-qa-i00-standard
vlabench-select-billiards-common-sense-i00-privileged
vlabench-select-billiards-common-sense-i00-standard
vlabench-select-billiards-i00-privileged
vlabench-select-billiards-i00-standard
vlabench-select-billiards-spatial-i00-privileged
vlabench-select-billiards-spatial-i00-standard
vlabench-select-book-spatial-i00-privileged
vlabench-select-book-spatial-i00-standard
vlabench-select-chemistry-tube-common-sense-i00-privileged
vlabench-select-chemistry-tube-common-sense-i00-standard
vlabench-select-chemistry-tube-i00-privileged
vlabench-select-chemistry-tube-i00-standard
vlabench-select-chemistry-tube-semantic-i00-privileged
vlabench-select-chemistry-tube-semantic-i00-standard
vlabench-select-chemistry-tube-spatial-i00-privileged
vlabench-select-chemistry-tube-spatial-i00-standard
vlabench-select-drink-i00-privileged
vlabench-select-drink-i00-standard
vlabench-select-fruit-common-sense-i00-privileged
vlabench-select-fruit-common-sense-i00-standard
vlabench-select-fruit-i00-privileged
vlabench-select-fruit-i00-standard
vlabench-select-mahjong-i00-privileged
vlabench-select-mahjong-i00-standard
vlabench-select-mahjong-semantic-i00-privileged
vlabench-select-mahjong-semantic-i00-standard
vlabench-select-mahjong-spatial-i00-privileged
vlabench-select-mahjong-spatial-i00-standard
vlabench-select-nth-largest-poker-i00-privileged
vlabench-select-nth-largest-poker-i00-standard
vlabench-select-painting-i00-privileged
vlabench-select-painting-i00-standard
vlabench-select-poker-i00-privileged
vlabench-select-poker-i00-standard
vlabench-select-poker-semantic-i00-privileged
vlabench-select-poker-semantic-i00-standard
vlabench-select-poker-spatial-i00-privileged
vlabench-select-poker-spatial-i00-standard
vlabench-select-specific-type-book-i00-privileged
vlabench-select-specific-type-book-i00-standard
vlabench-select-toy-i00-privileged
vlabench-select-toy-i00-standard
vlabench-select-unique-type-mahjong-i00-privileged
vlabench-select-unique-type-mahjong-i00-standard
vlabench-take-out-cool-drink-i00-privileged
vlabench-take-out-cool-drink-i00-standard
vlabench-texas-holdem-i00-privileged
vlabench-texas-holdem-i00-standard
README.md