The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
behavior: BEHAVIOR-1K 3.9.2, fixed
efe-private:behavior-0.1.2 (VERSION) = the official BEHAVIOR-1K 3.9.2 image by digest + ffmpeg (for replay
videos), the stack the 2026 BEHAVIOR Challenge pins. The recipe (Dockerfile) needs no build context; every BEHAVIOR-1K
task carries it verbatim and builds it when it cannot pull the private image (BUILD_ONLY).
| Base | stanfordvl/behavior:3.9.2@sha256:2702900192741904e69a573aa7869ea51c592586f9791050796efc37faa893fb (OmniGibson 3.9.2, Isaac Sim 5.1.0, cuRobo; 16 layers) |
| Our layers | 17: ffmpeg 7:6.1.1-3ubuntu5 + 146 dependencies (the package state dpkg.lock, checked by sha256); 18: WORKDIR /app (empty) |
| Published image | built 2026-10-19 with BuildKit's inline cache; layer 17 diff ID sha256:e08e357324a963d9a1976433a672fa6e210ce2ec60df862f26c60ebfa086964e |
| Dataset | none; mounted read-only at /data from $RCB_BEHAVIOR_DATA (BEHAVIOR Data Bundle license: no redistribution) |
Why the digest, not the tag
Tags move. Upstream released 3.9.3 on 2026-09-23, and stanfordvl/behavior:latest moved to it. A registry can also re-push
3.9.2 itself. The digest names exactly the bytes we tested.
Why apt is pinned to a snapshot
apt-get install ffmpeg without a pin takes whatever the archive holds that day (a rebuild on 2026-09-26 got libapparmor1
24.04.8, published 2026-09-23, instead of 24.04.7). Pinning only ffmpeg= would not help: its 146 dependencies still
float, and superseded update versions leave the archive index. The recipe therefore:
- reads the Ubuntu archive as it was on 2026-09-20 (
--snapshot 20260920T090000Z, snapshot.ubuntu.com); - fails the build unless the resulting package state (the sorted
dpkg-querylist) has the sha256 ofdpkg.lock, the tested package list. The build reads no file of this repository:dpkg.lockis the readable copy of that list.
A rebuild then has the tested package versions. The layer bytes still differ (file times), so a local build has another layer 17 diff ID than the published image; the packages are the same.
Private: the tasks' build cache (BUILD_ONLY)
The image contains NVIDIA Isaac Sim 5.1.0 and Omniverse Kit (from the official image), whose licences forbid a public copy
(NOTICE). It is pushed to the organization's private package with BuildKit's inline cache, and every BEHAVIOR-1K task
names it in build.cache_from: with access, Docker pulls its layers instead of building them; without, the task builds the
same recipe from Docker Hub (the base, 14.7 GB) and snapshot.ubuntu.com. To build it by hand:
docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t ghcr.io/mll-lab-nu/efe-private:behavior-$(cat images/behavior/VERSION) - < images/behavior/Dockerfile
To check an image against the recipe:
img=ghcr.io/mll-lab-nu/efe-private:behavior-0.1.2
base=stanfordvl/behavior@sha256:2702900192741904e69a573aa7869ea51c592586f9791050796efc37faa893fb
# 1. layers 1-16 are the official base
diff <(docker image inspect -f '{{range .RootFS.Layers}}{{println .}}{{end}}' $base | head -16) \
<(docker image inspect -f '{{range .RootFS.Layers}}{{println .}}{{end}}' $img | head -16)
# 2. the packages are the tested ones
docker run --rm --entrypoint dpkg-query $img -W -f='${db:Status-Status} ${Package}:${Architecture} ${Version}\n' \
| LC_ALL=C sort | diff - images/behavior/dpkg.lock
Both print nothing for a good image.