Skip to content

Secure, Fast, and Extensible Sandbox runtime for AI agents.

refactor(test): split the fast-sandbox integration env into entry + env + verify scripts

高然6aa98fc3e7be
diff --git a/.github/workflows/fast-sandbox-test.yml b/.github/workflows/fast-sandbox-test.yml
index ba0c860a1..5f2c46efc 100644
--- a/.github/workflows/fast-sandbox-test.yml
+++ b/.github/workflows/fast-sandbox-test.yml
@@ -53,15 +53,15 @@ jobs:
run: uv run --no-project --python 3.12 python -m unittest discover -s scripts/tests -p test_fast_sandbox_integration_env.py -v
- name: Run integration tests
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh up --auto-clean
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh up --auto-clean
- name: Run base SDK e2e
timeout-minutes: 30
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh sdk-e2e
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh sdk-e2e
- name: Run idle-pause SDK e2e
timeout-minutes: 45
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh idle-e2e
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh idle-e2e
# Collect the env's own failure dump ($WORK/logs/failure-*.txt: kind
# nodes/pods, controller / firecracker-runtime / fastlet / builder /
@@ -74,7 +74,7 @@ jobs:
mkdir -p diagnostics
sudo cp -a "$work/logs" "$work/run.log" diagnostics/ 2>/dev/null || true
sudo chown -R "$(id -u):$(id -g)" diagnostics || true
- sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh status \
+ sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh status \
> diagnostics/live-status.txt 2>&1 || true
- name: Upload failure diagnostics
@@ -87,4 +87,4 @@ jobs:
- name: Teardown
if: always()
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh down
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh down
diff --git a/.github/workflows/kubernetes-nightly-build.yml b/.github/workflows/kubernetes-nightly-build.yml
index 6377c50ee..f180e09a2 100644
--- a/.github/workflows/kubernetes-nightly-build.yml
+++ b/.github/workflows/kubernetes-nightly-build.yml
@@ -390,15 +390,15 @@ jobs:
uses: astral-sh/setup-uv@v7
- name: Run integration tests (idle-pause profile)
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh up --auto-clean
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh up --auto-clean
- name: Run base SDK e2e
timeout-minutes: 30
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh sdk-e2e
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh sdk-e2e
- name: Run idle-pause SDK e2e
timeout-minutes: 45
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh idle-e2e
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh idle-e2e
# Collect the env's own failure dump ($WORK/logs/failure-*.txt) plus
# live status, BEFORE teardown removes the cluster.
@@ -409,7 +409,7 @@ jobs:
mkdir -p diagnostics
sudo cp -a "$work/logs" "$work/run.log" diagnostics/ 2>/dev/null || true
sudo chown -R "$(id -u):$(id -g)" diagnostics || true
- sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh status \
+ sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh status \
> diagnostics/live-status.txt 2>&1 || true
- name: Upload failure diagnostics
@@ -422,7 +422,7 @@ jobs:
- name: Teardown
if: always()
- run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh down
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration.sh down
publish-nightly-latest:
name: Publish latest (${{ matrix.component }} nightly)
diff --git a/scripts/fast-sandbox-env/README.md b/scripts/fast-sandbox-env/README.md
index 17617db06..e5aabc68f 100644
--- a/scripts/fast-sandbox-env/README.md
+++ b/scripts/fast-sandbox-env/README.md
@@ -11,24 +11,24 @@ Requires a bare-metal Linux host with KVM (`/dev/kvm`), Docker, Go ≥ 1.25, cgr
## Usage
```bash
-./scripts/fast-sandbox-env/integration-env.sh up # full stack + end-to-end + pause/resume verify
-./scripts/fast-sandbox-env/integration-env.sh status # component / pool / DART / OpenSandbox health
-./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
-./scripts/fast-sandbox-env/integration-env.sh sdk-e2e # Python SDK e2e suite against the live stack
-./scripts/fast-sandbox-env/integration-env.sh idle-e2e # OSEP-0024 idle-pause/wake suite (needs IDLE_PAUSE=1 up)
-./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
+./scripts/fast-sandbox-env/integration.sh up # full stack + end-to-end + pause/resume verify
+./scripts/fast-sandbox-env/integration.sh status # component / pool / DART / OpenSandbox health
+./scripts/fast-sandbox-env/integration.sh pool # re-apply the pool only
+./scripts/fast-sandbox-env/integration.sh sdk-e2e # Python SDK e2e suite against the live stack
+./scripts/fast-sandbox-env/integration.sh idle-e2e # OSEP-0024 idle-pause/wake suite (needs IDLE_PAUSE=1 up)
+./scripts/fast-sandbox-env/integration.sh down # teardown, host left clean
```
### Idle-pause / wake-on-access profile (OSEP-0024)
-`IDLE_PAUSE=1 ./scripts/fast-sandbox-env/integration-env.sh up` additionally deploys an ephemeral Redis (`manifests/idle-pause/redis.yaml`, before the server — the sweeper connects at startup), enables the server `[automation.idle_pause]` sweeper (5s sweep) and the gateway's `--activity-enabled` + `--wake-enabled`, and runs an `idle-pause / wake-on-access` verify stage: an opted-in sandbox (`lifecycle.idlePolicy`, X=30s) is auto-paused with zero traffic, and the next gateway request wakes it (parked while the restore completes). The `idle-e2e` action then runs the SDK suite (`tests/python/tests/test_fsb_idle_pause_e2e.py`) covering auto-pause, wake-on-access through the SDK's wake retry contract, and the no-idlePolicy compatibility invariant. Sandboxes without `lifecycle.idlePolicy` are never auto-paused or auto-woken regardless of the profile.
+`IDLE_PAUSE=1 ./scripts/fast-sandbox-env/integration.sh up` additionally deploys an ephemeral Redis (`manifests/idle-pause/redis.yaml`, before the server — the sweeper connects at startup), enables the server `[automation.idle_pause]` sweeper (5s sweep) and the gateway's `--activity-enabled` + `--wake-enabled`, and runs an `idle-pause / wake-on-access` verify stage: an opted-in sandbox (`lifecycle.idlePolicy`, X=30s) is auto-paused with zero traffic, and the next gateway request wakes it (parked while the restore completes). The `idle-e2e` action then runs the SDK suite (`tests/python/tests/test_fsb_idle_pause_e2e.py`) covering auto-pause, wake-on-access through the SDK's wake retry contract, and the no-idlePolicy compatibility invariant. Sandboxes without `lifecycle.idlePolicy` are never auto-paused or auto-woken regardless of the profile.
After `up`, point any OpenSandbox SDK at `http://127.0.0.1:18080` with the `OPEN-SANDBOX-API-KEY: fast-sandbox-env` header; sandbox endpoints are signed `f1.*` header routes served by the gateway at `http://127.0.0.1:18081`. The installed CLI works against the same stack, e.g. `OPEN_SANDBOX_DOMAIN=127.0.0.1:18080 OPEN_SANDBOX_API_KEY=fast-sandbox-env osb sandbox list`.
The `up` verify stages are also available as Python SDK e2e tests (template create → gateway ping → networkpolicy convergence, lifecycle ops, pause/resume, snapshot round trip):
```bash
-./scripts/fast-sandbox-env/integration-env.sh sdk-e2e # needs uv on PATH
+./scripts/fast-sandbox-env/integration.sh sdk-e2e # needs uv on PATH
```
or, manually: `cd tests/python && OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$(cat "$WORK/template-id")" make test-fsb` (requires `OPENSANDBOX_TEST_FSB_TEMPLATE_ID`; the script stores the id in `$WORK/template-id`).
@@ -58,7 +58,21 @@ SDK ──header──> ingress gateway (source-built, --provider-type=fast-sand
## Layout
```
-integration-env.sh entrypoint: up / down / status / pool
+integration.sh entrypoint: up / down / status / pool / sdk-e2e / idle-e2e;
+ sources and chains the parts below (single process, so the
+ ERR/failure-dump/--auto-clean contract and the stage summary
+ stay unified)
+lib.sh shared config, logging/stage machinery, failure dump,
+ tooling, server/CLI helpers (sourced by every part)
+env.sh environment lifecycle, also runnable standalone:
+ up (provisioning stages, no verifies) / down / status / pool
+verify/ one standalone-runnable script per verify stage
+ sandbox.sh end-to-end create -> gateway route -> execd /ping,
+ egress policy convergence, lifecycle ops
+ pause-resume.sh pause/resume round trip through the checkpoint
+ vault.sh credential-vault relay lifecycle (OSEP-0022)
+ idle-pause.sh idle-pause / wake-on-access (OSEP-0024, IDLE_PAUSE=1)
+ snapshot.sh snapshot -> restore round trip
architecture.svg how the environment works (topology, pipeline,
P2P delivery, egress actions channel)
manifests/
diff --git a/scripts/fast-sandbox-env/create-bench.py b/scripts/fast-sandbox-env/create-bench.py
index 4c665c6db..6ba846ec7 100644
--- a/scripts/fast-sandbox-env/create-bench.py
+++ b/scripts/fast-sandbox-env/create-bench.py
@@ -15,7 +15,7 @@
"""create-bench.py — fsb sandbox create-latency bench for the fast-sandbox-env
kind deployment, written against the OpenSandbox Python SDK.
-Run AFTER `./integration-env.sh up`:
+Run AFTER `./integration.sh up`:
cd OpenSandbox/tests/python
uv run python ../../scripts/fast-sandbox-env/create-bench.py
diff --git a/scripts/fast-sandbox-env/env.sh b/scripts/fast-sandbox-env/env.sh
new file mode 100755
index 000000000..574b83073
--- /dev/null
+++ b/scripts/fast-sandbox-env/env.sh
@@ -0,0 +1,1178 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# env.sh — fast-sandbox integration environment lifecycle: bring-up
+# (preflight -> kind -> RustFS -> control plane -> firecracker runtime ->
+# server + gateway -> golden-image template -> SandboxPool), teardown
+# (down), health (status) and pool re-apply (pool). No verify stages: the
+# entry script (integration.sh) chains env_up_stages with the
+# verify/*.sh suites; this file can also run them standalone.
+#
+# Usage:
+# scripts/fast-sandbox-env/env.sh up # provision the full environment
+# scripts/fast-sandbox-env/env.sh down # teardown, host left clean
+# scripts/fast-sandbox-env/env.sh status # component/pool/DART/OpenSandbox health
+# scripts/fast-sandbox-env/env.sh pool # re-apply the SandboxPool only
+# Environment overrides: see the header of integration.sh.
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib.sh"
+
+preflight() {
+ [[ "$(uname -s)" == "Linux" ]] \
+ || die "this environment requires a Linux host with KVM (run it on the remote development VM)"
+ command -v docker >/dev/null || die "docker is required"
+ command -v go >/dev/null || die "go is required (>=1.25, used by make images and gen-registry)"
+ ensure_tool kind
+ ensure_tool kubectl
+ ensure_tool helm
+ ensure_tool jq
+ docker info >/dev/null 2>&1 || die "docker daemon is not reachable"
+ local cgver
+ cgver="$(docker info --format '{{.CgroupVersion}}' 2>/dev/null || true)"
+ log "docker cgroup version=$cgver"
+ # kind requires cgroup v2; v1 hosts fail kubepods cgroup creation.
+ if [[ "$cgver" == "1" ]]; then
+ die "docker cgroup Version is 1; kind requires cgroup v2. Enable it with the kernel cmdline 'systemd.unified_cgroup_hierarchy=1' and reboot"
+ fi
+ [[ -e /dev/kvm ]] || die "/dev/kvm is missing on this host (KVM required)"
+ # Fail fast per heavy-data target instead of ENOSPC mid-run.
+ local min_free_kb=$((20 * 1024 * 1024)) avail_kb target xfs_dir
+ xfs_dir="${XFS_LOOP_FILE%/*}"
+ mkdir -p "$WORK" "$RUSTFS_DATA" "$xfs_dir" 2>/dev/null || true
+ local -a targets=("$WORK" "$RUSTFS_DATA" "$xfs_dir")
+ for target in "${targets[@]}"; do
+ avail_kb="$(df -Pk "$target" 2>/dev/null | awk 'NR==2 {print $4}')"
+ [[ "$avail_kb" =~ ^[0-9]+$ ]] || die "cannot determine free disk space on $target"
+ if (( avail_kb < min_free_kb )); then
+ die "$target has $((avail_kb / 1024 / 1024))G free; at least 20G is required (built images, XFS StateRoot, RustFS artifacts). Free space (docker system prune / old kind clusters) or point WORK / RUSTFS_DATA / XFS_LOOP_FILE at a bigger volume"
+ fi
+ log "disk headroom: $target has $((avail_kb / 1024 / 1024 / 1024))G free"
+ done
+ # Fail fast on busy host ports instead of a cryptic docker/kind bind error.
+ local port
+ for port in "$RUSTFS_PORT" "$RUSTFS_CONSOLE_PORT" "$SERVER_HOST_PORT" "$GATEWAY_HOST_PORT"; do
+ if host_port_busy "$port"; then
+ die "127.0.0.1:$port is already in use (check 'ss -ltnp' / 'docker ps'); free it, or set RUSTFS_PORT / RUSTFS_CONSOLE_PORT / SERVER_HOST_PORT / GATEWAY_HOST_PORT"
+ fi
+ done
+ ensure_image "$RUSTFS_IMAGE"
+ ensure_image "$RC_IMAGE"
+ pass "preflight"
+}
+
+sysctl_set() {
+ local current
+ current="$(sysctl -n fs.inotify.max_user_instances)"
+ [[ "$current" -ge "$INOTIFY_VALUE" ]] && return 0
+ echo "$current" > "$SYSCTL_BACKUP"
+ log "sysctl fs.inotify.max_user_instances: $current -> $INOTIFY_VALUE"
+ sudo sysctl -w fs.inotify.max_user_instances="$INOTIFY_VALUE" >/dev/null
+}
+
+sysctl_restore() {
+ [[ -f "$SYSCTL_BACKUP" ]] || return 0
+ local previous
+ previous="$(cat "$SYSCTL_BACKUP")"
+ log "restoring fs.inotify.max_user_instances -> $previous"
+ sudo sysctl -w fs.inotify.max_user_instances="$previous" >/dev/null || true
+ rm -f "$SYSCTL_BACKUP"
+}
+
+# --- stage: fast-sandbox checkout @ master ---------------------------------------------
+
+# Scratch Go tool run inside the fast-sandbox module (imports
+# internal/registryconfig); removed before dirty checks and on down.
+FSB_GEN_DIR="$FSB_DIR/.fast-sandbox-env-gen"
+
+ensure_fsb() {
+ if [[ ! -d "$FSB_DIR/.git" ]]; then
+ log "cloning $FSB_REPO into $FSB_DIR"
+ git clone "$FSB_REPO" "$FSB_DIR" || die "clone failed; check network"
+ fi
+ rm -rf "$FSB_GEN_DIR"
+ [[ -z "$(git -C "$FSB_DIR" status --porcelain)" ]] \
+ || die "fast-sandbox checkout at $FSB_DIR has local changes; delete it to re-clone or point FSB_DIR at a clean checkout"
+ # Raw SHA fetch needs allow-reachable-sha1-in-want; full fetch otherwise.
+ if ! git -C "$FSB_DIR" fetch -q origin "$FSB_COMMIT" 2>/dev/null; then
+ git -C "$FSB_DIR" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' \
+ || die "git fetch failed for $FSB_REPO"
+ fi
+ git -C "$FSB_DIR" rev-parse --verify --quiet "$FSB_COMMIT^{commit}" >/dev/null \
+ || die "pinned commit $FSB_COMMIT is not reachable from $FSB_REPO"
+ if [[ "$(git -C "$FSB_DIR" rev-parse HEAD)" != "$FSB_COMMIT" ]]; then
+ git -C "$FSB_DIR" clean -ffdx
+ fi
+ git -C "$FSB_DIR" checkout --force -q "$FSB_COMMIT" \
+ || die "git checkout $FSB_COMMIT failed"
+ [[ "$(git -C "$FSB_DIR" rev-parse HEAD)" == "$FSB_COMMIT" ]] \
+ || die "fast-sandbox checkout is not at the pinned commit $FSB_COMMIT"
+ log "fast-sandbox @ pinned $(git -C "$FSB_DIR" rev-parse --short HEAD) (manifests/third-party/fast-sandbox.commit)"
+ pass "fast-sandbox checkout ready"
+}
+
+# --- stage: images ---------------------------------------------------------------------
+
+build_images() {
+ log "building fast-sandbox images (pinned commit, firecracker scope) via manifests/release/build-fast-sandbox.sh"
+ # Same checkout and defaults as the standalone builder: env and published
+ # build paths cannot drift.
+ FSB_SRC_DIR="$FSB_DIR" "$OSB_ROOT/manifests/release/build-fast-sandbox.sh" \
+ || die "fast-sandbox image build failed"
+ log "building the OpenSandbox egress image ($IMG_EGRESS)"
+ # Context is the repo root (COPYs components/egress + components/internal).
+ # shellcheck disable=SC2086
+ docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
+ -f "$OSB_ROOT/components/egress/Dockerfile" -t "$IMG_EGRESS" "$OSB_ROOT" >/dev/null \
+ || die "egress image build failed"
+ log "building the OpenSandbox server image ($IMG_SERVER)"
+ # Context is server/ (self-contained Dockerfile).
+ # shellcheck disable=SC2086
+ docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
+ -f "$OSB_ROOT/server/Dockerfile" -t "$IMG_SERVER" "$OSB_ROOT/server" >/dev/null \
+ || die "server image build failed"
+ log "building the OpenSandbox ingress image ($IMG_INGRESS)"
+ # shellcheck disable=SC2086
+ docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
+ -f "$OSB_ROOT/components/ingress/Dockerfile" -t "$IMG_INGRESS" "$OSB_ROOT" >/dev/null \
+ || die "ingress image build failed"
+ pass "images built (fast-sandbox + OpenSandbox)"
+}
+
+# --- stage: XFS StateRoot (reflink CoW per-sandbox rootfs) ------------------------------
+
+XFS_STATEROOT="${XFS_STATEROOT:-1}"
+XFS_LOOP_FILE="${XFS_LOOP_FILE:-$WORK/fast-sandbox.img}"
+XFS_SIZE="${XFS_SIZE:-24G}"
+XFS_MOUNT_POINT="${XFS_MOUNT_POINT:-/var/lib/fast-sandbox}"
+
+ensure_xfsprogs() {
+ command -v mkfs.xfs >/dev/null 2>&1 && return 0
+ log "installing xfsprogs (mkfs.xfs)"
+ if command -v apt-get >/dev/null 2>&1; then
+ sudo_ apt-get install -y xfsprogs >/dev/null
+ elif command -v yum >/dev/null 2>&1; then
+ sudo_ yum install -y xfsprogs >/dev/null
+ else
+ die "xfsprogs not installed and no supported package manager"
+ fi
+}
+
+stateroot_xfs_up() {
+ [[ "$XFS_STATEROOT" == 1 ]] || {
+ sudo_ mkdir -p "$XFS_MOUNT_POINT"
+ log "XFS StateRoot disabled (XFS_STATEROOT=0); per-sandbox rootfs pays a full copy"
+ return 0
+ }
+ if findmnt -no FSTYPE "$XFS_MOUNT_POINT" 2>/dev/null | grep -qx xfs; then
+ # Leftover from a run killed before teardown: scrub, and recreate the
+ # image if still below the agent's min-free floor (it strips the kvm
+ # node label and template builds stop scheduling).
+ stateroot_scrub
+ if (( $(stateroot_free_gib) >= XFS_MIN_FREE_GIB )); then
+ pass "XFS StateRoot ready (reflink CoW rootfs, scrubbed)"
+ return 0
+ fi
+ log "StateRoot below ${XFS_MIN_FREE_GIB}GiB free after scrub; recreating the XFS image"
+ sudo_ umount "$XFS_MOUNT_POINT" || die "unmount $XFS_MOUNT_POINT failed"
+ rm -f "$XFS_LOOP_FILE"
+ fi
+ ensure_xfsprogs
+ if [[ ! -f "$XFS_LOOP_FILE" ]]; then
+ log "creating sparse XFS image $XFS_LOOP_FILE (virtual $XFS_SIZE)"
+ truncate -s "$XFS_SIZE" "$XFS_LOOP_FILE"
+ sudo_ mkfs.xfs -f "$XFS_LOOP_FILE" >/dev/null 2>&1 || die "mkfs.xfs failed on $XFS_LOOP_FILE"
+ fi
+ sudo_ mkdir -p "$XFS_MOUNT_POINT"
+ sudo_ mount -o noatime "$XFS_LOOP_FILE" "$XFS_MOUNT_POINT" \
+ || die "mount $XFS_LOOP_FILE at $XFS_MOUNT_POINT failed (loop support? try XFS_STATEROOT=0)"
+ local a b
+ a="$XFS_MOUNT_POINT/.reflink-a"
+ b="$XFS_MOUNT_POINT/.reflink-b"
+ printf 'probe' | sudo_ tee "$a" >/dev/null
+ if sudo_ cp --reflink=always "$a" "$b"; then
+ sudo_ rm -f "$a" "$b"
+ pass "XFS StateRoot ready (reflink CoW rootfs)"
+ else
+ sudo_ rm -f "$a" "$b"
+ die "reflink probe failed on $XFS_MOUNT_POINT (CoW rootfs would not work)"
+ fi
+}
+
+# Purge per-node runtime caches in the mounted StateRoot (shared by down()
+# and the reuse branch of stateroot_xfs_up); snapshots/ leaks across runs.
+stateroot_scrub() {
+ local node_dir
+ for node_dir in "$XFS_MOUNT_POINT"/*/; do
+ [[ -d "${node_dir}firecracker" ]] || continue
+ log "purging node runtime cache under ${node_dir%/}"
+ sudo_ rm -rf "${node_dir}firecracker/images" \
+ "${node_dir}firecracker/agent" \
+ "${node_dir}firecracker/jails" \
+ "${node_dir}firecracker/cache" \
+ "${node_dir}firecracker/snapshots" 2>/dev/null || true
+ done
+}
+
+# Free space on the mounted StateRoot, in GiB (GNU df).
+stateroot_free_gib() {
+ df -BG --output=avail "$XFS_MOUNT_POINT" 2>/dev/null | tail -1 | tr -dc '0-9' || echo 0
+}
+XFS_MIN_FREE_GIB="${XFS_MIN_FREE_GIB:-12}"
+
+stateroot_xfs_down() {
+ [[ "$XFS_STATEROOT" == 1 ]] || return 0
+ if findmnt -no SOURCE "$XFS_MOUNT_POINT" 2>/dev/null | grep -q "$(basename "$XFS_LOOP_FILE")"; then
+ log "unmounting XFS StateRoot $XFS_MOUNT_POINT"
+ sudo_ umount "$XFS_MOUNT_POINT"
+ fi
+ rm -f "$XFS_LOOP_FILE"
+}
+
+# --- stage: kind cluster -----------------------------------------------------------------
+
+render_kind_config() { # > $GEN_DIR/kind-cluster.yaml
+ local src="$MANIFESTS_DIR/cluster/kind-cluster.yaml" out="$GEN_DIR/kind-cluster.yaml"
+ mkdir -p "$GEN_DIR"
+ cp "$src" "$out"
+ if [[ "$KIND_SINGLE" == "1" ]]; then
+ # Strip the worker node: cache-only topology, no peer traffic.
+ awk '/^- role: worker/ {exit} {print}' "$out" > "$out.tmp" && mv "$out.tmp" "$out"
+ log "single-node topology (KIND_SINGLE=1: no worker, no peer traffic)"
+ fi
+ if [[ -n "${DOCKER_MIRROR:-}" ]]; then
+ # Host-specific, hence opt-in; one TOML endpoint ARRAY (repeated
+ # endpoint keys would override each other).
+ local block_file="$GEN_DIR/docker-mirror-block.yaml" mirror trimmed endpoints=""
+ IFS=',' read -ra mirrors <<<"$DOCKER_MIRROR"
+ for mirror in "${mirrors[@]}"; do
+ trimmed="$(printf '%s' "$mirror" | tr -d '[:space:]')"
+ [[ -n "$trimmed" ]] || continue
+ [[ -z "$endpoints" ]] && endpoints="\"$trimmed\"" || endpoints+=", \"$trimmed\""
+ done
+ [[ -n "$endpoints" ]] || die "DOCKER_MIRROR produced no usable endpoints"
+ {
+ echo 'containerdConfigPatches:'
+ echo '- |-'
+ echo ' [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]'
+ echo " endpoint = [$endpoints]"
+ } > "$block_file"
+ awk -v block_file="$block_file" '
+ NR == FNR { block = block $0 "\n"; next }
+ /^nodes:/ && !done { printf "%s", block; done = 1 }
+ { print }
+ ' "$block_file" "$out" > "$out.tmp" && mv "$out.tmp" "$out"
+ rm -f "$block_file"
+ log "docker.io containerd mirrors injected: $endpoints"
+ fi
+ # extraPortMappings publish the server/gateway on the host; they exist
+ # only at cluster creation, so a cluster reused without them stays
+ # cluster-internal.
+ local ports_file="$GEN_DIR/osb-ports-block.yaml"
+ {
+ echo ' extraPortMappings:'
+ echo " - containerPort: $SERVER_NODEPORT"
+ echo " hostPort: $SERVER_HOST_PORT"
+ echo ' listenAddress: 127.0.0.1'
+ echo ' protocol: TCP'
+ echo " - containerPort: $GATEWAY_NODEPORT"
+ echo " hostPort: $GATEWAY_HOST_PORT"
+ echo ' listenAddress: 127.0.0.1'
+ echo ' protocol: TCP'
+ } > "$ports_file"
+ awk -v block_file="$ports_file" '
+ NR == FNR { block = block $0 "\n"; next }
+ /^- role: control-plane/ && !done { print; printf "%s", block; done = 1; next }
+ { print }
+ ' "$ports_file" "$out" > "$out.tmp" && mv "$out.tmp" "$out"
+ rm -f "$ports_file"
+ log "server on 127.0.0.1:$SERVER_HOST_PORT, gateway on 127.0.0.1:$GATEWAY_HOST_PORT (NodePorts $SERVER_NODEPORT/$GATEWAY_NODEPORT)"
+}
+
+kind_up() {
+ local create_args=() kind_config="$GEN_DIR/kind-cluster.yaml"
+ [[ -f "$kind_config" ]] || die "kind config not rendered ($kind_config)"
+ [[ "$KIND_RETAIN" == 1 ]] && create_args+=(--retain)
+ if [[ -n "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]]; then
+ log "cluster $KIND_CLUSTER already exists; reusing (run down first for a clean rebuild)"
+ else
+ if [[ -n "${KIND_NODE_IMAGE:-}" ]]; then
+ log "pulling kind node image $KIND_NODE_IMAGE (this can take minutes)"
+ ensure_image "$KIND_NODE_IMAGE" || die "kind node image unavailable locally and pull failed (KIND_NODE_IMAGE=$KIND_NODE_IMAGE)"
+ create_args+=(--image "$KIND_NODE_IMAGE")
+ else
+ log "creating cluster (pulling kindest/node may take minutes; set KIND_NODE_IMAGE to a mirror if it fails)"
+ fi
+ kind create cluster --name "$KIND_CLUSTER" \
+ ${create_args+"${create_args[@]}"} --config "$kind_config" > "$LOGS_DIR/kind-create.log" 2>&1 || {
+ local rc=$?
+ tail -40 "$LOGS_DIR/kind-create.log" >&2 || true
+ log "kind create failed (full log: $LOGS_DIR/kind-create.log)" || true
+ return "$rc" # through ERR, so diagnostics + --auto-clean run
+ }
+ pass "kind cluster created"
+ fi
+ local node
+ for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
+ docker exec "$node" sh -c 'test -e /dev/kvm' || die "KVM not visible inside the kind node container $node"
+ log "node $node: /dev/kvm visible"
+ done
+ if [[ "$KIND_SINGLE" != "1" ]]; then
+ # Untaint the control plane: agent/fastlet/builder must schedule on
+ # BOTH nodes for P2P peer traffic.
+ kubectl taint nodes --all node-role.kubernetes.io/control-plane- >/dev/null 2>&1 || true
+ log "control-plane taint removed (both nodes schedulable for the P2P topology)"
+ fi
+ pass "kind cluster ready (KVM passthrough + labels on every node)"
+}
+
+# --- stage: RustFS + credentials ------------------------------------------------------------
+
+rustfs_up() {
+ docker rm -f "$RUSTFS_CONTAINER" >/dev/null 2>&1 || true
+ # RustFS runs as UID/GID 10001 and owns the bind-mounted store.
+ sudo_ rm -rf "$RUSTFS_DATA"
+ mkdir -p "$RUSTFS_DATA"
+ sudo_ chown -R 10001:10001 "$RUSTFS_DATA"
+ mkdir -p "$RC_CONFIG_DIR"
+ chmod 700 "$RC_CONFIG_DIR"
+ local net
+ net="$(kind_network)"
+ # On the kind network: pods reach the container IP directly (no
+ # docker-proxy/hairpin issues); 127.0.0.1 publishes serve the host.
+ docker run -d --name "$RUSTFS_CONTAINER" --network "$net" \
+ -p 127.0.0.1:"$RUSTFS_PORT":"$RUSTFS_CONTAINER_PORT" -p 127.0.0.1:"$RUSTFS_CONSOLE_PORT":9001 \
+ -e RUSTFS_ACCESS_KEY="$RUSTFS_AK" -e RUSTFS_SECRET_KEY="$RUSTFS_SK" \
+ -e RUSTFS_ADDRESS=":$RUSTFS_CONTAINER_PORT" \
+ -e RUSTFS_CONSOLE_ADDRESS=:9001 -e RUSTFS_CONSOLE_ENABLE=true \
+ -v "$RUSTFS_DATA:/data" \
+ "$RUSTFS_IMAGE" >/dev/null
+ local attempt
+ for attempt in $(seq 1 30); do
+ if curl -fsS "http://127.0.0.1:$RUSTFS_PORT/health" >/dev/null 2>&1; then break; fi
+ sleep 1
+ [[ "$attempt" == 30 ]] && die "RustFS did not become healthy"
+ done
+ rc alias set chain "http://127.0.0.1:$RUSTFS_PORT" "$RUSTFS_AK" "$RUSTFS_SK" >/dev/null 2>&1 \
+ || die "could not configure the RustFS rc alias"
+ for attempt in $(seq 1 30); do
+ if rc ls chain/ >/dev/null 2>&1; then break; fi
+ sleep 1
+ [[ "$attempt" == 30 ]] && die "RustFS S3 API did not become ready for authenticated bucket listing"
+ done
+ rc mb "chain/$RUSTFS_BUCKET" >/dev/null
+ pass "RustFS up (bucket=$RUSTFS_BUCKET)"
+}
+
+resolve_rustfs_endpoint() {
+ if [[ -n "$RUSTFS_ENDPOINT" ]]; then
+ log "RustFS endpoint (env): $RUSTFS_ENDPOINT"
+ else
+ local net ips ip
+ net="$(kind_network)"
+ ips="$(docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{$v.IPAddress}} {{end}}' "$RUSTFS_CONTAINER")"
+ ip="$(printf '%s' "$ips" | tr ' ' '\n' | grep -A1 -x "^$net$" | tail -1)"
+ [[ -n "$ip" ]] || die "could not find the RustFS IP on network $net (inspect: $ips)"
+ # Container port, not the host publish: kind-network clients hit the
+ # container IP directly.
+ RUSTFS_ENDPOINT="http://$ip:$RUSTFS_CONTAINER_PORT"
+ log "RustFS endpoint (kind network IP): $RUSTFS_ENDPOINT"
+ fi
+ local net
+ net="$(kind_network)"
+ docker run --rm --network "$net" --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
+ -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
+ -e http_proxy= -e https_proxy= -e all_proxy= \
+ -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" alias set kind-rustfs \
+ "$RUSTFS_ENDPOINT" "$RUSTFS_AK" "$RUSTFS_SK" >/dev/null 2>&1 \
+ || die "RustFS unreachable from the kind network at $RUSTFS_ENDPOINT (override RUSTFS_ENDPOINT)"
+ docker run --rm --network "$net" --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
+ -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
+ -e http_proxy= -e https_proxy= -e all_proxy= \
+ -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" ls "kind-rustfs/$RUSTFS_BUCKET" >/dev/null 2>&1 \
+ || die "RustFS authenticated bucket access failed from the kind network at $RUSTFS_ENDPOINT"
+ pass "RustFS reachable from the kind network"
+}
+
+# gen_registry builds the agent registry JSON via fast-sandbox's
+# registryconfig package; the write pair covers checkpoint publication.
+gen_registry() { # host username password endpoint [write-username write-password] > registry.json
+ mkdir -p "$FSB_GEN_DIR"
+ cat > "$FSB_GEN_DIR/gen-registry.go" <<'EOF'
+package main
+
+import (
+ "fmt"
+ "os"
+
+ "fast-sandbox/internal/registryconfig"
+)
+
+func main() {
+ if len(os.Args) != 5 && len(os.Args) != 7 {
+ fmt.Fprintln(os.Stderr, "usage: gen-registry <host> <username> <password> <endpoint> [write-username write-password]")
+ os.Exit(1)
+ }
+ credential := registryconfig.Credential{
+ Host: os.Args[1], Username: os.Args[2], Password: os.Args[3], Endpoint: os.Args[4],
+ }
+ if len(os.Args) == 7 {
+ // Optional publish (write) pair: empty keeps the store read-only.
+ credential.WriteUsername, credential.WritePassword = os.Args[5], os.Args[6]
+ }
+ compiled, err := registryconfig.NewCompiled([]registryconfig.Credential{credential})
+ if err != nil {
+ fmt.Fprintln(os.Stderr, err)
+ os.Exit(1)
+ }
+ payload, err := compiled.Marshal()
+ if err != nil {
+ fmt.Fprintln(os.Stderr, err)
+ os.Exit(1)
+ }
+ os.Stdout.Write(payload)
+}
+EOF
+ (cd "$FSB_DIR" && GOTOOLCHAIN=local go run .fast-sandbox-env-gen/gen-registry.go "$@")
+}
+
+credentials_up() {
+ # Namespaces exist even after a partial up (resume support).
+ kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ kubectl create namespace "$RESOURCE_NS" --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ local host
+ host="${RUSTFS_ENDPOINT#http://}"
+ host="${host#https://}"
+ # Publish credentials for the builder Pod (dataplane namespace).
+ kubectl -n "$RESOURCE_NS" create secret generic sandbox-oss-credentials \
+ --from-literal=accessKeyId="$RUSTFS_AK" \
+ --from-literal=secretAccessKey="$RUSTFS_SK" \
+ --from-literal=endpoint="$RUSTFS_ENDPOINT" \
+ --from-literal=region=us-east-1 \
+ --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ # Agent pull+publish credentials (the write pair covers checkpoints).
+ gen_registry "$host" "$RUSTFS_AK" "$RUSTFS_SK" "$RUSTFS_ENDPOINT" "$RUSTFS_AK" "$RUSTFS_SK" \
+ > "$WORK/agent-registry.json"
+ jq -e '.credentials[0].writeUsername' "$WORK/agent-registry.json" >/dev/null \
+ || die "generated agent registry carries no write credential"
+ kubectl -n "$NS" create secret generic fast-sandbox-agent-registry \
+ --from-file=registry.json="$WORK/agent-registry.json" \
+ --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ # Pull credentials for the fastlet; the artifact-store ConfigMap is
+ # rendered by charts/fast-sandbox at install time.
+ kubectl -n "$RESOURCE_NS" create secret docker-registry registry-rustfs \
+ --docker-server="$host" --docker-username="$RUSTFS_AK" --docker-password="$RUSTFS_SK" \
+ --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ kubectl -n "$RESOURCE_NS" create configmap fast-sandbox-registry \
+ --from-literal="registries.yaml=registries:
+ - host: $host
+ secretRef:
+ name: registry-rustfs
+" \
+ --dry-run=client -o yaml | kubectl apply -f - >/dev/null
+ pass "credentials written (publish/pull)"
+}
+
+# --- stage: control plane -------------------------------------------------------------------
+
+# helm is only a renderer; the cluster keeps no helm release state, so
+# re-runs stay plain idempotent `kubectl apply`.
+helm_render() { # release chart ns out [set-args...]
+ local release="$1" chart="$2" ns="$3" out="$4"
+ shift 4
+ helm template "$release" "$chart" --namespace "$ns" "$@" > "$out" \
+ || die "helm template $chart failed"
+}
+
+apply_ns() { # ns -> ensure the namespace exists (idempotent)
+ kubectl create namespace "$1" --dry-run=client -o yaml 2>/dev/null |
+ kubectl apply -f - >/dev/null
+}
+
+control_plane_up() {
+ # Charts are the source of truth (CRDs + RBAC + control plane), replacing
+ # the fast-sandbox checkout's kustomize applies and env-owned node manifests.
+ apply_ns "$NS"
+ helm_render fsb-base "$OSB_ROOT/manifests/charts/base" "$NS" "$GEN_DIR/fsb-base.yaml"
+ kubectl apply -f "$GEN_DIR/fsb-base.yaml" >/dev/null
+ helm_render fast-sandbox "$OSB_ROOT/manifests/charts/fast-sandbox" "$NS" \
+ "$GEN_DIR/fast-sandbox.yaml" \
+ --set controller.image.repository="$(image_repo "$IMG_CONTROLLER")" \
+ --set controller.image.tag="$(image_tag "$IMG_CONTROLLER")" \
+ --set controller.sandboxtemplateBuilderImage="$IMG_BUILDER" \
+ --set controller.fastletProxyImage="$IMG_FASTLET_PROXY" \
+ --set janitor.image.repository="$(image_repo "$IMG_JANITOR")" \
+ --set janitor.image.tag="$(image_tag "$IMG_JANITOR")" \
+ --set runtime.image.repository="$(image_repo "$IMG_RUNTIME")" \
+ --set runtime.image.tag="$(image_tag "$IMG_RUNTIME")" \
+ --set artifactStore.store="s3://$RUSTFS_BUCKET/publish" \
+ --set artifactStore.endpoint="$RUSTFS_ENDPOINT"
+ kubectl apply -f "$GEN_DIR/fast-sandbox.yaml" >/dev/null
+ local image
+ for image in "$IMG_CONTROLLER" "$IMG_FASTLET" "$IMG_FASTLET_PROXY" \
+ "$IMG_JANITOR" "$IMG_RUNTIME" "$IMG_EGRESS"; do
+ kind load docker-image "$image" --name "$KIND_CLUSTER" >/dev/null
+ done
+ wait_for "controller deployment ready" 120 \
+ kubectl -n "$NS" rollout status deploy/fast-sandbox-controller --timeout=10s
+ local crd
+ for crd in sandboxpools sandboxtemplates sandboxes sandboxsnapshots; do
+ kubectl get crd "$crd.sandbox.fast.io" >/dev/null 2>&1 || die "CRD $crd missing"
+ done
+ kubectl get crd batchsandboxes.sandbox.opensandbox.io >/dev/null 2>&1 \
+ || die "CRD batchsandboxes.sandbox.opensandbox.io missing"
+ pass "CRDs + control plane ready (charts @ pinned $(git -C "$FSB_DIR" rev-parse --short HEAD))"
+}
+
+# --- stage: firecracker runtime (node readiness + DART P2P) -----------------------------------
+
+runtime_node_labeled() { # node -> 0 when the agent applied both labels + condition
+ local node="$1"
+ kubectl get node "$node" -o json 2>/dev/null | jq -e \
+ --arg fc "$FC_NODE_LABEL" --arg kvm "$KVM_NODE_LABEL" '
+ (.metadata.labels[$fc] == "true") and
+ (.metadata.labels[$kvm] == "true") and
+ ([(.status.conditions // [])[]?
+ | select(.type == "FirecrackerReady" and .status == "True")] | length > 0)
+ ' >/dev/null
+}
+
+runtime_pods() {
+ kubectl -n "$NS" get pods -l component=firecracker-runtime -o jsonpath='{.items[*].metadata.name}' 2>/dev/null
+}
+
+dart_roster_ready() { # pod expected-members
+ local pod="$1" expected="$2" members
+ members="$(kubectl exec -n "$NS" "$pod" -- sh -c \
+ 'curl -fsS --noproxy "*" http://127.0.0.1:8147/admin/members' 2>/dev/null || true)"
+ [[ "$(printf '%s' "$members" | grep -o '"id":' | wc -l | tr -d ' ')" == "$expected" ]]
+}
+
+runtime_up() {
+ # DaemonSet + dart Service ship with charts/fast-sandbox; credentials land
+ # in credentials_up before this stage.
+ wait_for "firecracker-runtime DaemonSet ready" 120 \
+ kubectl -n "$NS" rollout status daemonset/firecracker-runtime --timeout=10s
+
+ # Positive wiring assertions: DART admin plane up, agent p2pUp=true
+ # (a missing P2P daemon only degrades pulls to direct S3).
+ local pod uid node pods
+ pods="$(runtime_pods)"
+ for pod in $pods; do
+ uid="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.metadata.uid}')"
+ node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}')"
+ wait_for "dart admin /healthz on $node" 30 \
+ kubectl exec -n "$NS" "$pod" -- sh -c \
+ "curl -fsS --noproxy '*' http://127.0.0.1:8147/healthz | grep -q ok"
+ wait_for "agent health p2pUp on $node" 30 \
+ kubectl exec -n "$NS" "$pod" -- sh -c \
+ "curl -fsS --noproxy '*' --unix-socket /run/fast-sandbox/firecracker/runtime.sock -H 'Content-Type: application/json' -d '{\"podUid\":\"$uid\",\"namespace\":\"$NS\"}' http://firecracker-agent/v1/health | grep -q '\"p2pUp\":true'"
+ log "dart: $node dart pid=$(kubectl exec -n "$NS" "$pod" -- sh -c 'pgrep -x dart')"
+ done
+ # P2P roster: every daemon must see every peer before any pull, so later
+ # nodes are fed by peers instead of the origin.
+ local expected_members
+ expected_members="$(printf '%s' "$pods" | wc -w | tr -d ' ')"
+ for pod in $pods; do
+ node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}')"
+ wait_for "dart roster full on $node ($expected_members members)" 90 \
+ dart_roster_ready "$pod" "$expected_members"
+ done
+ # The readiness loop applies labels + FirecrackerReady itself.
+ for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
+ wait_for "node $node labeled + FirecrackerReady" 120 \
+ runtime_node_labeled "$node"
+ done
+ pass "firecracker-runtime healthy + DART roster=$expected_members + nodes FirecrackerReady"
+}
+
+# --- stage (opensandbox CLI): SandboxTemplate golden image ---------------------
+
+wait_succeeded() { # description attempts probe probe_failed
+ local description="$1" attempts="$2" probe="$3" probe_failed="$4" attempt=0
+ while ! "$probe" >/dev/null 2>&1; do
+ if "$probe_failed" >/dev/null 2>&1; then
+ failure_dump "template-failed"
+ fail "$description (template entered Failed)"
+ fi
+ attempt=$((attempt + 1))
+ if [[ "$attempt" -ge "$attempts" ]]; then
+ failure_dump "template-timeout"
+ fail "$description (after $attempts attempts)"
+ fi
+ sleep 2
+ done
+ pass "$description"
+}
+
+# Template build via the CLI (server /templates API behind it): the row is
+# projected onto a SandboxTemplate CRD; the build runs in fast-sandbox, so
+# the builder image must be in the cluster.
+
+_template_phase() {
+ osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.phase // empty'
+}
+
+template_succeeded() {
+ [[ "$(_template_phase)" == "Succeeded" ]]
+}
+
+template_failed() {
+ local phase message
+ phase="$(_template_phase)"
+ [[ "$phase" == "Failed" ]] || return 1
+ message="$(osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.message // empty')"
+ log "template Failed: ${message:-<no message>}"
+ return 0
+}
+
+template_up() {
+ log "building the sandboxtemplate-builder image"
+ # shellcheck disable=SC2086
+ docker build ${DOCKER_BUILD_FLAGS:-} --quiet -t "$IMG_BUILDER" \
+ -f "$FSB_DIR/build/Dockerfile.sandboxtemplate-builder" "$FSB_DIR" >/dev/null \
+ || die "sandboxtemplate-builder image build failed"
+ kind load docker-image "$IMG_BUILDER" --name "$KIND_CLUSTER" >/dev/null
+ local created
+ local -a tmpl_args=(
+ template create
+ --image "$SBX_IMAGE"
+ --publish "s3://$RUSTFS_BUCKET/publish"
+ --format native
+ --resource cpu=1 --resource memory=512Mi --resource disk=2Gi
+ --warmup-seconds 15
+ --metadata origin=fast-sandbox-env
+ )
+ log "verify: creating the golden-image template via the opensandbox CLI (image=$SBX_IMAGE)"
+ created="$(osb_api "${tmpl_args[@]}" -o json 2>/dev/null)" \
+ || fail "template create failed against $SERVER_URL: $(osb_api "${tmpl_args[@]}" 2>&1 >/dev/null | head -c 400)"
+ TEMPLATE_ID="$(printf '%s' "$created" | jq -r '.template_id')"
+ printf '%s' "$TEMPLATE_ID" > "$WORK/template-id"
+ [[ -n "$TEMPLATE_ID" && "$TEMPLATE_ID" != "null" ]] || fail "template create response carried no templateId"
+ log "template id=$TEMPLATE_ID"
+ wait_succeeded "template phase=Succeeded" 300 template_succeeded template_failed
+ local manifest_ref
+ manifest_ref="$(osb_api template get "$TEMPLATE_ID" -o json | jq -r '.status.manifest_ref // empty')"
+ [[ -n "$manifest_ref" ]] || fail "template manifestRef is empty"
+ log "template manifestRef: $manifest_ref"
+ pass "SandboxTemplate Succeeded + artifacts published (via opensandbox CLI)"
+}
+
+# --- stage: SandboxPool (egress attached, P2P spread) --------------------------------------------
+
+pool_pods() {
+ kubectl -n "$RESOURCE_NS" get pods \
+ -l "app=sandbox-fastlet,fast-sandbox.io/pool=$POOL_NAME" \
+ -o jsonpath='{.items[*].metadata.name}' 2>/dev/null
+}
+
+first_pool_pod() {
+ kubectl -n "$RESOURCE_NS" get pods \
+ -l "app=sandbox-fastlet,fast-sandbox.io/pool=$POOL_NAME" \
+ -o jsonpath='{.items[0].metadata.name}' 2>/dev/null
+}
+
+fastlet_pods_ready() {
+ local ready=0 pod
+ for pod in $(pool_pods); do
+ if kubectl -n "$RESOURCE_NS" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null | grep -q True; then
+ ready=$((ready + 1))
+ fi
+ done
+ [[ "$ready" -ge "$POOL_MIN" ]]
+}
+
+egress_containers_ready() {
+ local pods pod
+ pods="$(pool_pods)"
+ [[ -n "$pods" ]] || return 1
+ for pod in $pods; do
+ kubectl -n "$RESOURCE_NS" get pod "$pod" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"="}{.ready}{" "}{end}' 2>/dev/null \
+ | grep -q 'egress=true' || return 1
+ done
+}
+
+# GET /_fastlet/v1/actions/status must echo the shared apiVersion, ready=true
+# and a non-empty instanceId; the handler binds Pod loopback only, so the
+# probe runs inside the egress container.
+egress_status_ready() {
+ local pod out
+ pod="$(first_pool_pod)"
+ [[ -n "$pod" ]] || return 1
+ out="$(kubectl -n "$RESOURCE_NS" exec "pod/$pod" -c egress -- \
+ curl -fsS -m 5 "http://127.0.0.1:18080/_fastlet/v1/actions/status" 2>/dev/null)" || return 1
+ [[ "$out" == *'"apiVersion":"sandbox.fast.io/actions/v1"'* && "$out" == *'"ready":true'* && "$out" == *'"instanceId":'* ]]
+}
+
+pool_status_ready() {
+ local ready
+ ready="$(kubectl_get "sandboxpool/$POOL_NAME" '{.status.readyPods}' 2>/dev/null || true)"
+ [[ "$ready" =~ ^[0-9]+$ ]] && (( ready >= POOL_MIN ))
+}
+
+pool_condition_true() { # condition-type
+ [[ "$(kubectl_get "sandboxpool/$POOL_NAME" "{.status.conditions[?(@.type==\"$1\")].status}" 2>/dev/null)" == "True" ]]
+}
+
+warm_images_ready() {
+ kubectl -n "$RESOURCE_NS" get sandboxpool "$POOL_NAME" -o jsonpath='{.status.warmImages[*].cachedFastlets}' 2>/dev/null | grep -qv '^0*$'
+}
+
+render_pool() { # > $GEN_DIR/firecracker-egress-pool.yaml
+ local src="$MANIFESTS_DIR/pool/firecracker-egress-pool.yaml" out="$GEN_DIR/firecracker-egress-pool.yaml"
+ mkdir -p "$GEN_DIR"
+ # Quoted tokens in the template keep rendered scalars' natural types
+ # (poolMin stays an integer); awk for sed portability.
+ awk -v fastlet="$IMG_FASTLET" -v egress="$IMG_EGRESS" \
+ -v pool_min="$POOL_MIN" -v pool_max="$POOL_MAX" \
+ -v max_per_pod="$MAX_SANDBOXES_PER_POD" \
+ -v warm="$WARM_IMAGES" -v image="$TEMPLATE_ID" '
+ { gsub(/"@FASTLET_IMAGE@"/, fastlet)
+ gsub(/"@EGRESS_IMAGE@"/, egress)
+ gsub(/"@POOL_MIN@"/, pool_min)
+ gsub(/"@POOL_MAX@"/, pool_max)
+ gsub(/maxSandboxesPerPod: 5/, "maxSandboxesPerPod: " max_per_pod) }
+ /^# @WARM_IMAGES@$/ {
+ if (warm == "1") printf " warmImages:\n - %s\n", image
+ next }
+ { print }
+ ' "$src" > "$out"
+ # Leftover-token guard: match only value positions (key: ...@T@...),
+ # never the header comments that document the tokens themselves.
+ if grep -Eq '^[[:space:]]*[A-Za-z][A-Za-z0-9]*:.*@[A-Z_]+@' "$out"; then
+ die "unrendered token left in $out"
+ fi
+}
+
+pool_up() {
+ render_pool
+ log "applying SandboxPool $POOL_NAME (runtime=firecracker, egress attached, poolMin=$POOL_MIN)"
+ kubectl apply -f "$GEN_DIR/firecracker-egress-pool.yaml" >/dev/null
+ wait_for "fastlet pods Ready (poolMin=$POOL_MIN)" 300 fastlet_pods_ready
+ wait_for "egress container ready in every fastlet pod" 180 egress_containers_ready
+ wait_for "pool condition RuntimeReady=True" 120 pool_condition_true RuntimeReady
+ wait_for "pool condition InfraReady=True" 120 pool_condition_true InfraReady
+ wait_for "egress actions endpoint answering (/_fastlet/v1/actions/status)" 60 egress_status_ready
+ if [[ "$WARM_IMAGES" == "1" ]]; then
+ wait_for "pool warmImages Ready" 300 warm_images_ready
+ p2p_evidence "warm preheat"
+ pass "fastlet Running + warmImages Ready (P2P evidence captured)"
+ else
+ pass "fastlet Running, on-demand pull (default: first sandbox pulls through DART)"
+ fi
+}
+
+# Assert the P2P outcome from DART block counters: the artifact set is
+# pulled ~once per 4MiB block from the origin cluster-wide, and extra nodes
+# must be fed by peers, not the origin.
+p2p_evidence() { # description
+ local description="$1"
+ local pods pod manifest_ref manifest_key build_dir expected_blocks=0 stat_json
+ local origin_total=0 peer_total=0 cache_total=0 size source value active_nodes=0 node_total
+ manifest_ref="$(osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.manifest_ref // empty')"
+ manifest_key="${manifest_ref#s3://$RUSTFS_BUCKET/}"
+ build_dir="$(dirname "$manifest_key")"
+ local object
+ for object in rootfs.ext4 vmstate.snap memory.snap; do
+ stat_json="$(rc stat --json "chain/$RUSTFS_BUCKET/$build_dir/$object" 2>&1)" \
+ || die "rc stat failed for published $object (publish incomplete?): $stat_json"
+ size="$(jq -er '.size_bytes | numbers' <<<"$stat_json" 2>/dev/null)" \
+ || die "rc stat returned no numeric size_bytes for published $object"
+ expected_blocks=$((expected_blocks + (size + 4194303) / 4194304))
+ done
+ pods="$(runtime_pods)"
+ for pod in $pods; do
+ node_total=0
+ while read -r source value; do
+ case "$source" in
+ origin) origin_total=$((origin_total + value)); node_total=$((node_total + value)) ;;
+ peer) peer_total=$((peer_total + value)); node_total=$((node_total + value)) ;;
+ cache) cache_total=$((cache_total + value)); node_total=$((node_total + value)) ;;
+ esac
+ done < <(dart_source_counters "$pod")
+ [[ "$node_total" -gt 0 ]] && active_nodes=$((active_nodes + 1))
+ done
+ log "p2p evidence ($description): expected origin=$expected_blocks blocks; cluster origin=$origin_total peer=$peer_total cache=$cache_total active-nodes=$active_nodes"
+ [[ "$origin_total" -ge "$expected_blocks" ]] || fail "cluster origin $origin_total < expected $expected_blocks blocks"
+ [[ "$origin_total" -le $((expected_blocks + 4)) ]] \
+ || fail "origin amplified: $origin_total > $((expected_blocks + 4)): pulls were not deduplicated by DART"
+ if [[ "$active_nodes" -ge 2 ]]; then
+ [[ "$peer_total" -gt 0 ]] || fail "no peer traffic across $active_nodes nodes: the second node was not served by the peer"
+ pass "P2P evidence ($description): origin ~1 fetch per block (cluster=$origin_total/$expected_blocks), peer=$peer_total, nodes=$active_nodes"
+ else
+ pass "P2P evidence ($description): origin ~1 fetch per block (cluster=$origin_total/$expected_blocks) on $active_nodes node (no peer needed)"
+ fi
+}
+
+dart_source_counters() { # pod -> "<source> <value>" lines
+ local pod="$1"
+ kubectl exec -n "$NS" "$pod" -- sh -c 'curl -fsS --noproxy "*" http://127.0.0.1:8147/metrics' 2>/dev/null \
+ | awk '/^dart_block_source_total\{source="(cache|peer|origin)"\}/ {
+ match($0, /source="[^"]+"/); s = substr($0, RSTART + 8, RLENGTH - 9)
+ match($0, /} [0-9]+$/); print s, substr($0, RSTART + 2)
+ }'
+}
+
+# --- stage: idle-pause Redis (OSEP-0024 profile) --------------------------------
+
+redis_up() {
+ # The sweeper connects at startup, so Redis must precede the server.
+ [[ "$IDLE_PAUSE" == "1" ]] || { log "idle-pause profile off (IDLE_PAUSE=0); skipping Redis"; rm -f "$WORK/idle-pause"; return 0; }
+ touch "$WORK/idle-pause"
+ mkdir -p "$GEN_DIR"
+ sed "s|image: redis:7-alpine|image: $REDIS_IMAGE|" \
+ "$MANIFESTS_DIR/idle-pause/redis.yaml" > "$GEN_DIR/idle-pause-redis.yaml"
+ kubectl apply -n "$OSB_NS" -f "$GEN_DIR/idle-pause-redis.yaml" >/dev/null
+ wait_for "idle-pause Redis deployment ready" 120 \
+ kubectl -n "$OSB_NS" rollout status deploy/idle-pause-redis --timeout=10s
+ pass "idle-pause Redis up ($REDIS_DSN)"
+}
+
+# --- stage: OpenSandbox server + ingress gateway --------------------------------
+
+# Per-workdir key (not per-run): re-applies must keep old routes verifiable.
+osb_signing_key() {
+ if [[ ! -s "$SIGNING_KEY_FILE" ]]; then
+ openssl rand -base64 32 | tr -d '\n' > "$SIGNING_KEY_FILE"
+ fi
+ cat "$SIGNING_KEY_FILE"
+}
+
+# Server config.toml (fsb runtime) with workdir tokens; the gateway [ingress]
+# block is appended by charts/server from server.gateway.* values.
+render_server_config() { # > $GEN_DIR/osb-server-config.toml
+ mkdir -p "$GEN_DIR"
+ awk -v api_key="$SERVER_API_KEY" \
+ -v fastpath="$FASTPATH_ENDPOINT" -v fsb_ns="$RESOURCE_NS" -v pool="$POOL_NAME" \
+ -v execd="$EXECD" '
+ { gsub(/@SERVER_API_KEY@/, api_key)
+ gsub(/@SIGNING_KEY@/, signing_key)
+ gsub(/@FASTPATH_ENDPOINT@/, fastpath)
+ gsub(/@FSB_NAMESPACE@/, fsb_ns)
+ gsub(/@POOL_NAME@/, pool)
+ gsub(/@EXECD_IMAGE@/, execd)
+ gsub(/@GATEWAY_ADDRESS@/, gateway)
+ print }
+ ' <<'TOML' > "$GEN_DIR/osb-server-config.toml"
+[server]
+host = "0.0.0.0"
+port = 80
+api_key = "@SERVER_API_KEY@"
+
+[log]
+level = "INFO"
+
+[runtime]
+type = "kubernetes"
+execd_image = "@EXECD_IMAGE@"
+
+[kubernetes]
+# One block for CR reads + fsb settings; sandboxes live in the pool's
+# namespace (poolRef resolves) and get the execd image injected by the server.
+namespace = "@FSB_NAMESPACE@"
+fastpath_endpoint = "@FASTPATH_ENDPOINT@"
+fastpath_resource_pool = "@POOL_NAME@"
+fastpath_wait_ready_seconds = 30.0
+template_s3_publish_secret = "sandbox-oss-credentials"
+TOML
+ if [[ "$IDLE_PAUSE" == "1" ]]; then
+ # Dotted keys nest the redis sub-table.
+ cat >> "$GEN_DIR/osb-server-config.toml" <<TOML
+
+[automation.idle_pause]
+enabled = true
+sweep_interval_seconds = 5
+activity_ttl_seconds = 1800
+redis.dsn = "$REDIS_DSN"
+redis.lock_ttl_seconds = 30
+TOML
+ fi
+ if grep -Eq '@[A-Z_]+@' "$GEN_DIR/osb-server-config.toml"; then
+ die "unrendered token left in $GEN_DIR/osb-server-config.toml"
+ fi
+}
+
+opensandbox_up() {
+ kind load docker-image "$IMG_SERVER" --name "$KIND_CLUSTER" >/dev/null
+ kind load docker-image "$IMG_INGRESS" --name "$KIND_CLUSTER" >/dev/null
+ render_server_config
+ apply_ns "$OSB_NS"
+ # charts/server has the fsb RBAC built in; the config carries the wiring.
+ helm_render opensandbox-server "$OSB_ROOT/manifests/charts/server" "$OSB_NS" \
+ "$GEN_DIR/osb-server.yaml" \
+ --set server.image.repository="$(image_repo "$IMG_SERVER")" \
+ --set server.image.tag="$(image_tag "$IMG_SERVER")" \
+ --set server.service.type=NodePort \
+ --set server.service.nodePort="$SERVER_NODEPORT" \
+ --set server.resources.requests.cpu=250m \
+ --set server.resources.requests.memory=512Mi \
+ --set server.resources.limits.cpu=1 \
+ --set server.resources.limits.memory=2Gi \
+ --set-file configToml="$GEN_DIR/osb-server-config.toml" \
+ --set server.gateway.enabled=true \
+ --set server.gateway.host="$GATEWAY_ADDRESS" \
+ --set server.gateway.gatewayRouteMode=header \
+ --set server.gateway.secureAccess.activeKey=a \
+ --set "server.gateway.secureAccess.keys[0].key_id=a" \
+ --set "server.gateway.secureAccess.keys[0].key=$(osb_signing_key)"
+ kubectl apply -f "$GEN_DIR/osb-server.yaml" >/dev/null
+ # fast-sandbox provider via FastPath, same signing key as the server.
+ local gateway_idle_args=()
+ if [[ "$IDLE_PAUSE" == "1" ]]; then
+ gateway_idle_args=(
+ --set gateway.activity.enabled=true
+ --set gateway.activity.redisDsn="$REDIS_DSN"
+ --set gateway.activity.minInterval=5s
+ --set gateway.activity.ttlSeconds=1800
+ --set gateway.wake.enabled=true
+ --set gateway.wake.parkBudget="${IDLE_PAUSE_PARK_BUDGET:-5s}"
+ )
+ fi
+ helm_render opensandbox-ingress-gateway "$OSB_ROOT/manifests/charts/ingress-gateway" "$OSB_NS" \
+ "$GEN_DIR/osb-ingress-gateway.yaml" \
+ --set gateway.image.repository="$(image_repo "$IMG_INGRESS")" \
+ --set gateway.image.tag="$(image_tag "$IMG_INGRESS")" \
+ --set gateway.replicaCount=1 \
+ --set gateway.providerType=fast-sandbox \
+ --set gateway.dataplaneNamespace="$NS" \
+ --set gateway.fastpathEndpoint="$FASTPATH_ENDPOINT" \
+ --set "gateway.secureAccess.keys[0].key_id=a" \
+ --set "gateway.secureAccess.keys[0].key=$(osb_signing_key)" \
+ --set gateway.service.type=NodePort \
+ --set gateway.service.nodePort="$GATEWAY_NODEPORT" \
+ --set gateway.resources.requests.cpu=100m \
+ --set gateway.resources.requests.memory=128Mi \
+ --set gateway.resources.limits.cpu=1 \
+ --set gateway.resources.limits.memory=1Gi \
+ "${gateway_idle_args[@]}"
+ kubectl apply -f "$GEN_DIR/osb-ingress-gateway.yaml" >/dev/null
+ wait_for "server deployment ready" 180 \
+ kubectl -n "$OSB_NS" rollout status deploy/opensandbox-server --timeout=10s
+ wait_for "ingress gateway deployment ready" 180 \
+ kubectl -n "$OSB_NS" rollout status deploy/opensandbox-ingress-gateway --timeout=10s
+ # A ready gateway proves FastPath reachability: it fails startup without
+ # the gRPC connection.
+ wait_for "server /health on 127.0.0.1:$SERVER_HOST_PORT" 60 \
+ curl -fsS -m 5 "$SERVER_URL/health"
+ wait_for "gateway /status.ok on $GATEWAY_ADDRESS" 60 \
+ curl -fsS -m 5 --noproxy "*" "$GATEWAY_URL/status.ok"
+ pass "server + ingress gateway up (fsb runtime, gateway routes signed with key 'a')"
+}
+dart_metrics_summary() {
+ local pods pod node metrics
+ pods="$(runtime_pods 2>/dev/null || true)"
+ [[ -n "$pods" ]] || { echo " (no agent pods)"; return 0; }
+ for pod in $pods; do
+ node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)"
+ metrics="$(kubectl exec -n "$NS" "$pod" -- sh -c 'curl -fsS --noproxy "*" http://127.0.0.1:8147/metrics' 2>/dev/null || true)"
+ echo " $node:"
+ if [[ -z "$metrics" ]]; then
+ echo " (DART metrics unreachable)"
+ continue
+ fi
+ printf '%s\n' "$metrics" | grep -E '^dart_block_source_total\{source="(cache|peer|origin)"\}' \
+ | sed 's/^/ /' || true
+ done
+}
+
+status() {
+ log "status: RustFS"
+ docker ps --filter "name=$RUSTFS_CONTAINER" --format '{{.Names}} {{.Status}}' 2>/dev/null || true
+ echo
+ log "status: kind cluster / nodes"
+ if kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null; then
+ kubectl get nodes -o wide
+ else
+ log "kind cluster $KIND_CLUSTER: down"
+ return 0
+ fi
+ echo
+ log "status: pods ($NS)"
+ if [[ "$IDLE_PAUSE" == "1" ]]; then
+ log "status: idle-pause Redis ($OSB_NS)"
+ kubectl -n "$OSB_NS" get deploy,svc -l app=idle-pause-redis -o wide 2>/dev/null || true
+ echo
+ fi
+ kubectl -n "$NS" get pods -o wide
+ kubectl -n "$RESOURCE_NS" get pods -o wide
+ echo
+ log "status: SandboxPool"
+ kubectl -n "$RESOURCE_NS" get sandboxpool \
+ -o custom-columns='NAME:.metadata.name,RUNTIME:.spec.runtime,READY:.status.readyPods,CAPACITY:.spec.capacity.poolMin,WARM_IMAGES:.status.warmImages' 2>/dev/null || true
+ echo
+ log "status: fastlet pods (egress sidecar)"
+ kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet \
+ -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,CONTAINERS:.status.containerStatuses[*].name,READY:.status.containerStatuses[*].ready' 2>/dev/null || true
+ echo
+ log "status: DART P2P (block_source cache/peer/origin per node)"
+ dart_metrics_summary || true
+ echo
+ log "status: OpenSandbox ($OSB_NS)"
+ if kubectl get namespace "$OSB_NS" >/dev/null 2>&1; then
+ kubectl -n "$OSB_NS" get pods -o wide
+ printf ' server health: %s\n' "$(curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 && echo OK || echo unreachable)"
+ printf ' gateway health: %s\n' "$(curl -fsS -m 5 "$GATEWAY_URL/status.ok" >/dev/null 2>&1 && echo OK || echo unreachable)"
+ printf ' server URL: %s (header OPEN-SANDBOX-API-KEY: %s)\n' "$SERVER_URL" "$SERVER_API_KEY"
+ printf ' gateway URL: %s (header routing, signed f1.* scopes)\n' "$GATEWAY_URL"
+ else
+ echo " (not deployed)"
+ fi
+}
+
+env_summary() {
+ highlight "== environment summary =="
+ printf ' %-22s %s\n' "kind cluster" "$KIND_CLUSTER ($(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ') nodes)"
+ printf ' %-22s %s\n' "fast-sandbox" "pinned $(git -C "$FSB_DIR" rev-parse --short HEAD 2>/dev/null || echo "$FSB_COMMIT") ($FSB_DIR)"
+ printf ' %-22s %s\n' "RustFS endpoint" "$RUSTFS_ENDPOINT"
+ printf ' %-22s %s\n' "pool" "$POOL_NAME (runtime=firecracker, poolMin=$POOL_MIN, egress=$IMG_EGRESS)"
+ printf ' %-22s %s\n' "P2P" "DART daemons=$(printf '%s' "$(runtime_pods)" | wc -w | tr -d ' ') (on-demand pulls: cache -> peer -> origin)"
+ printf ' %-22s %s\n' "template" "${TEMPLATE_ID:-n/a} ($(if [[ -n "$TEMPLATE_ID" ]]; then _template_phase || echo unknown; else echo "not built"; fi))"
+ printf ' %-22s %s\n' "StateRoot fs" "$(findmnt -no FSTYPE "$XFS_MOUNT_POINT" 2>/dev/null || echo 'plain directory (full copy per sandbox)')"
+ printf ' %-22s %s\n' "server" "$IMG_SERVER -> $SERVER_URL (fsb runtime)"
+ printf ' %-22s %s\n' "ingress gateway" "$IMG_INGRESS -> $GATEWAY_URL (fsb provider, header mode)"
+ printf ' %-22s %s\n' "fastpath" "$FASTPATH_ENDPOINT"
+ printf ' %-22s %s\n' "logs" "$LOGS_DIR"
+}
+
+# --- down ------------------------------------------------------------------------------------
+
+down() {
+ log "down: teardown"
+ if kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null; then
+ kind delete cluster --name "$KIND_CLUSTER" > "$LOGS_DIR/kind-delete.log" 2>&1 || true
+ fi
+ [[ -z "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]] \
+ || fail "kind cluster $KIND_CLUSTER still exists after delete"
+ docker rm -f "$RUSTFS_CONTAINER" >/dev/null 2>&1 || true
+ [[ -z "$(docker ps -a --filter "name=$RUSTFS_CONTAINER" --format '{{.Names}}' || true)" ]] \
+ || fail "RustFS container still present"
+ # Server/gateway die with the cluster; only the signing key outlives it.
+ rm -f "$WORK/agent-registry.json" "$SIGNING_KEY_FILE"
+ rm -rf "$GEN_DIR" "$FSB_GEN_DIR"
+ # A stale profile marker would let idle-e2e run against a stack whose
+ # Redis/sweeper are gone.
+ rm -f "$WORK/idle-pause"
+ # RustFS data is owned by UID/GID 10001; leftovers pollute the host and
+ # later docker build contexts.
+ sudo_ rm -rf "$RUSTFS_DATA"
+ sudo_ rm -rf "$RC_CONFIG_DIR"
+ sysctl_restore
+ stateroot_xfs_down
+ # A committed DART cache is FINAL (never refreshed): purge it, or a
+ # rebuilt SandboxTemplate stays ignored when the StateRoot survives.
+ stateroot_scrub
+ pass "host cleanup complete"
+}
+
+# --- bring-up pipeline --------------------------------------------------------------
+
+# Provisioning stages only (no verify stages): the entry script chains these
+# with the verify/*.sh suites so `up` reports one unified stage summary.
+env_up_stages() {
+ run_stage "preflight + tooling" preflight
+ run_stage "opensandbox CLI (osb, latest from PyPI)" ensure_osb
+ run_stage "sysctl (fs.inotify)" sysctl_set
+ run_stage "fast-sandbox checkout @ pinned commit" ensure_fsb
+ run_stage "build images (fast-sandbox + OpenSandbox)" build_images
+ run_stage "XFS StateRoot (reflink)" stateroot_xfs_up
+ run_stage "render kind config" render_kind_config
+ run_stage "kind cluster (KVM passthrough + labels)" kind_up
+ run_stage "RustFS + bucket" rustfs_up
+ run_stage "RustFS endpoint (kind network)" resolve_rustfs_endpoint
+ run_stage "CRDs + control plane" control_plane_up
+ run_stage "credentials (publish/pull)" credentials_up
+ run_stage "firecracker runtime readiness + DART (P2P)" runtime_up
+ run_stage "idle-pause Redis (OSEP-0024 profile)" redis_up
+ run_stage "OpenSandbox server + ingress gateway" opensandbox_up
+ run_stage "SandboxTemplate build (opensandbox CLI)" template_up
+ run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
+}
+
+# Shared `up` prelude: output tee'd into $WORK/run.log, environment
+# snapshot, and leftover cleanup. Used by the standalone env_up and by the
+# entry script (which chains the verify stages after env_up_stages).
+env_up_prelude() {
+ exec > >(tee -a "$WORK/run.log") 2>&1
+ log "=== fast-sandbox-env up ($(date -u +%FT%TZ)) ==="
+ {
+ echo "environment snapshot ($(date -u +%FT%TZ))"
+ command -v kind >/dev/null && kind --version
+ kubectl version --client 2>/dev/null | head -1
+ go version
+ docker --version
+ echo "cluster=$KIND_CLUSTER single=$KIND_SINGLE rustfs=$RUSTFS_IMAGE port=$RUSTFS_PORT bucket=$RUSTFS_BUCKET"
+ echo "sbxImage=$SBX_IMAGE execd=$EXECD warmImages=$WARM_IMAGES"
+ echo "pool=$POOL_NAME poolMin=$POOL_MIN egress=$IMG_EGRESS"
+ echo "server=$IMG_SERVER ingress=$IMG_INGRESS fastpath=$FASTPATH_ENDPOINT"
+ echo "images: controller=$IMG_CONTROLLER runtime=$IMG_RUNTIME"
+ } > "$LOGS_DIR/environment.txt" 2>&1 || true
+ if [[ -n "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]] \
+ || docker ps -a --format '{{.Names}}' | grep -qx "$RUSTFS_CONTAINER"; then
+ if [[ "$SKIP_LEFTOVER_CLEAN" == 1 ]]; then
+ log "leftover resources detected; aborting (SKIP_LEFTOVER_CLEAN=1). Run 'integration.sh down' first"
+ exit 1
+ fi
+ log "leftover resources detected; cleaning and rebuilding"
+ down
+ fi
+}
+
+# Standalone `up`: provision everything, no verify stages.
+env_up() {
+ env_up_prelude
+ trap 'on_error up' ERR
+ env_up_stages
+ trap - ERR
+ stage_summary
+ env_summary
+ highlight "== up complete: server=$SERVER_URL (header OPEN-SANDBOX-API-KEY: $SERVER_API_KEY), gateway=$GATEWAY_URL =="
+}
+
+env_usage() {
+ cat <<'EOUT'
+usage: env.sh {up|down|status|pool}
+
+ up provision the environment (no verify stages): fast-sandbox@pinned-commit
+ images, two-node kind cluster (KVM), RustFS, control plane,
+ firecracker runtime readiness + DART (P2P), SandboxTemplate golden
+ image, firecracker-egress-pool (egress attached), the source-built
+ OpenSandbox server + ingress gateway
+ pool re-apply only the SandboxPool (after editing manifests/pool/)
+ status nodes / pods / pool / DART P2P counters / RustFS / OpenSandbox health
+ down teardown: kind cluster + RustFS + sysctl + XFS StateRoot + caches
+
+ The entry script (integration.sh) chains `up` with the verify stages;
+ see its header for the full action set and environment overrides.
+EOUT
+ exit 1
+}
+
+# Standalone dispatch (skipped when sourced by the entry script).
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ case "${1:-}" in
+ up) env_up ;;
+ pool)
+ exec > >(tee -a "$WORK/run.log") 2>&1
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ trap 'on_error pool' ERR
+ run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
+ trap - ERR
+ env_summary
+ ;;
+ status) status ;;
+ down) down ;;
+ *) env_usage ;;
+ esac
+fi
diff --git a/scripts/fast-sandbox-env/integration-env.sh b/scripts/fast-sandbox-env/integration-env.sh
deleted file mode 100755
index 199ee3b96..000000000
--- a/scripts/fast-sandbox-env/integration-env.sh
+++ /dev/null
@@ -1,2386 +0,0 @@
-#!/usr/bin/env bash
-# Copyright 2026 The OpenSandbox Authors
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# integration-env.sh — one-command fast-sandbox integration environment,
-# driven from the OpenSandbox repository.
-#
-# On a bare-metal Linux KVM host it builds the full stack — fast-sandbox at
-# the commit pinned in manifests/third-party/fast-sandbox.commit, a two-node
-# kind cluster with KVM passthrough, the OpenSandbox Helm charts (base CRDs,
-# fast-sandbox control plane, server, ingress-gateway), a RustFS artifact
-# store, and the firecracker-egress-pool SandboxPool with the OpenSandbox
-# egress sidecar — then verifies end to end: sandbox create via the
-# opensandbox CLI, execd /ping through the signed gateway route, delete,
-# a pause/resume round trip, and a snapshot -> restore-from-snapshotId boot.
-#
-# All Kubernetes resources come from the charts via `helm template` + plain
-# `kubectl apply` (no helm release state); the only env-owned manifests are
-# the kind cluster config and the SandboxPool. The verify stages drive the
-# server through the published opensandbox CLI ("osb", installed fresh from
-# PyPI; OSB_BIN overrides); raw curl remains only for the calls the CLI
-# cannot express (networkpolicy replace PUT, metadata merge-patch, snapshot
-# re-entry fence) plus the gateway /ping probes.
-#
-# Usage:
-# ./scripts/fast-sandbox-env/integration-env.sh up # full environment + pool + server/ingress + verify
-# ./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
-# ./scripts/fast-sandbox-env/integration-env.sh status # component/pool/DART/OpenSandbox health
-# ./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
-# ./scripts/fast-sandbox-env/integration-env.sh up --auto-clean # down on failure
-#
-# Environment overrides (all optional):
-# WORK workspace root (default /data/fast-sandbox-env when /data exists, else $PWD/.fast-sandbox-env)
-# FSB_DIR fast-sandbox checkout (default $WORK/fast-sandbox; source pinned in manifests/third-party/fast-sandbox.commit)
-# KIND_CLUSTER / KIND_NODE_IMAGE / KIND_RETAIN / KIND_SINGLE
-# DOCKER_MIRROR comma list injected as docker.io containerd mirrors
-# RUSTFS_PORT / RUSTFS_CONSOLE_PORT / RUSTFS_AK / RUSTFS_SK / RUSTFS_IMAGE / RC_IMAGE / RUSTFS_ENDPOINT
-# IMAGE_<NAME> fast-sandbox component image tags
-# EGRESS_IMAGE egress image tag (default docker.io/opensandbox/egress:latest)
-# SERVER_IMAGE / INGRESS_IMAGE OpenSandbox server/ingress image tags
-# SERVER_HOST_PORT / GATEWAY_HOST_PORT host-side publishes (default 18080/18081)
-# WARM_IMAGES=1 preheat pool warmImages (default: on-demand first-sandbox pull)
-# SBX_IMAGE / EXECD template build inputs (default opensandbox/fsb-sandbox-golden:latest / opensandbox/execd:latest)
-# POOL_MIN / POOL_MAX pool capacity (default 2/2; auto 1/1 when KIND_SINGLE=1)
-# MAX_SANDBOXES_PER_POD per-fastlet sandbox capacity (default 8)
-# XFS_STATEROOT / XFS_SIZE reflink StateRoot on/off and virtual size
-# OSB_PACKAGE / OSB_BIN opensandbox CLI for the verify stages (latest from PyPI / preinstalled binary)
-# SKIP_TOOL_INSTALL / SKIP_LEFTOVER_CLEAN / INOTIFY_VALUE
-# IDLE_PAUSE=1 OSEP-0024 idle-pause/wake profile: Redis, server
-# [automation.idle_pause], gateway activity+wake flags,
-# idle-pause verify stage
-# IDLE_PAUSE_ROUNDS auto-pause/wake rounds in the verify stage and idle-e2e (default 3)
-# IDLE_PAUSE_PARK_BUDGET gateway wake park budget (default 5s; set e.g. 100ms to
-# force every wake through the marked-503 + client-retry path)
-# REDIS_IMAGE Redis image for the idle-pause profile (default redis:7-alpine)
-#
-# Every stage logs to $WORK/logs/; failures dump component logs to
-# logs/failure-<task>-<ts>.txt before exiting (never silently).
-
-# -E (errtrace): stage functions must inherit the ERR trap, or on_error
-# (and its failure dump) never fires — every stage runs inside a function.
-set -euEo pipefail
-ENV_PID=$BASHPID
-
-OSB_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-MANIFESTS_DIR="$SCRIPT_DIR/manifests"
-# Heavy run-state belongs on a data volume, not the repo/root disk.
-if [[ -d /data ]] && [[ -z "${WORK:-}" ]]; then
- WORK="/data/fast-sandbox-env"
-else
- WORK="${WORK:-$PWD/.fast-sandbox-env}"
-fi
-LOGS_DIR="$WORK/logs"
-GEN_DIR="$WORK/gen"
-
-# Env-owned fast-sandbox clone; the source is exactly the commit pinned in
-# manifests/third-party/fast-sandbox.commit (no ref override — bump the pin).
-FSB_DIR="${FSB_DIR:-$WORK/fast-sandbox}"
-FSB_REPO="$(sed -n 's/^repo:[[:space:]]*//p' "$OSB_ROOT/manifests/third-party/fast-sandbox.commit")"
-FSB_COMMIT="$(sed -n 's/^commit:[[:space:]]*//p' "$OSB_ROOT/manifests/third-party/fast-sandbox.commit")"
-
-KIND_CLUSTER="${KIND_CLUSTER:-fast-sandbox-integration}"
-KIND_SINGLE="${KIND_SINGLE:-0}"
-KIND_RETAIN="${KIND_RETAIN:-0}"
-# Control plane + node runtime (controller/fastpath, firecracker-runtime,
-# agent credentials, artifact-store config) ...
-NS="opensandbox-system"
-# ... while the SandboxPool/Template/Sandbox resources and the fastlet and
-# builder Pods they spawn live in the dataplane namespace.
-RESOURCE_NS="opensandbox-dataplane"
-
-RUSTFS_IMAGE="${RUSTFS_IMAGE:-rustfs/rustfs:latest}"
-RC_IMAGE="${RC_IMAGE:-rustfs/rc:latest}"
-RUSTFS_PORT="${RUSTFS_PORT:-19000}"
-# In-container ports are fixed (9000 S3 / 9001 console); these vars only move
-# the host-side 127.0.0.1 publishes (defaults avoid 9000/9001 collisions).
-RUSTFS_CONTAINER_PORT=9000
-RUSTFS_CONSOLE_PORT="${RUSTFS_CONSOLE_PORT:-19001}"
-RUSTFS_AK="${RUSTFS_AK:-integration-env}"
-RUSTFS_SK="${RUSTFS_SK:-integration-env-secret}"
-RUSTFS_BUCKET="sandbox-images"
-RUSTFS_CONTAINER="${RUSTFS_CONTAINER:-fast-sandbox-env-rustfs}"
-RUSTFS_DATA="$WORK/rustfs-data"
-RUSTFS_ENDPOINT="${RUSTFS_ENDPOINT:-}" # auto-derived from the kind network
-RC_CONFIG_DIR="$WORK/rc-config"
-
-SBX_IMAGE="${SBX_IMAGE:-opensandbox/fsb-sandbox-golden:latest}"
-EXECD="${EXECD:-opensandbox/execd:latest}"
-# WARM_IMAGES=1 preheats pool warmImages (keyed by template id); the default
-# is the on-demand first-sandbox pull through DART.
-WARM_IMAGES="${WARM_IMAGES:-0}"
-
-POOL_NAME="${POOL_NAME:-firecracker-egress-pool}"
-MAX_SANDBOXES_PER_POD="${MAX_SANDBOXES_PER_POD:-8}"
-if [[ -n "${POOL_MIN:-}" && -n "${POOL_MAX:-}" ]]; then
- :
-elif [[ "$KIND_SINGLE" == "1" ]]; then
- # Cache-only topology: a single fastlet is enough (no peer traffic).
- POOL_MIN="${POOL_MIN:-1}"
- POOL_MAX="${POOL_MAX:-1}"
-else
- POOL_MIN="${POOL_MIN:-2}"
- POOL_MAX="${POOL_MAX:-2}"
-fi
-
-# Image tags (env-overridable per component).
-IMG_CONTROLLER="${IMAGE_CONTROLLER:-opensandbox/fsb-controller:dev}"
-IMG_FASTLET="${IMAGE_FASTLET:-opensandbox/fsb-fastlet:dev}"
-IMG_FASTLET_PROXY="${IMAGE_FASTLET_PROXY:-opensandbox/fsb-fastlet-proxy:dev}"
-IMG_JANITOR="${IMAGE_JANITOR:-opensandbox/fsb-janitor:dev}"
-IMG_BUILDER="${IMAGE_BUILDER:-opensandbox/fsb-sandboxtemplate-builder:dev}"
-IMG_RUNTIME="${IMAGE_RUNTIME:-opensandbox/fsb-firecracker-runtime:dev}"
-IMG_EGRESS="${EGRESS_IMAGE:-docker.io/opensandbox/egress:latest}"
-
-image_repo() { printf '%s' "${1%:*}"; }
-image_tag() { printf '%s' "${1##*:}"; }
-
-# --- OpenSandbox server + ingress gateway (source-built; always deployed) ------
-
-OSB_NS="opensandbox-system"
-IMG_SERVER="${SERVER_IMAGE:-docker.io/opensandbox/server:env}"
-IMG_INGRESS="${INGRESS_IMAGE:-docker.io/opensandbox/ingress:env}"
-FASTPATH_ENDPOINT="fast-sandbox-fastpath.opensandbox-system.svc:9090"
-SERVER_API_KEY="fast-sandbox-env"
-
-# OSEP-0024 profile: Redis + server [automation.idle_pause] + gateway activity/wake flags.
-IDLE_PAUSE="${IDLE_PAUSE:-0}"
-REDIS_IMAGE="${REDIS_IMAGE:-redis:7-alpine}"
-REDIS_RELEASE="idle-pause-redis"
-REDIS_DSN="redis://$REDIS_RELEASE.$OSB_NS.svc.cluster.local:6379/0"
-# Per-workdir f1.* route-scope signing key: re-applies must keep old routes
-# verifiable.
-SIGNING_KEY_FILE="$WORK/opensandbox-signing-key"
-# Host publish via kind extraPortMappings (loopback only; overridable on
-# port collisions).
-SERVER_HOST_PORT="${SERVER_HOST_PORT:-18080}"
-GATEWAY_HOST_PORT="${GATEWAY_HOST_PORT:-18081}"
-SERVER_NODEPORT=30880
-GATEWAY_NODEPORT=30881
-GATEWAY_ADDRESS="127.0.0.1:$GATEWAY_HOST_PORT"
-SERVER_URL="http://127.0.0.1:$SERVER_HOST_PORT"
-GATEWAY_URL="http://127.0.0.1:$GATEWAY_HOST_PORT"
-
-# Applied by the firecracker-runtime readiness loop itself (never manually).
-KVM_NODE_LABEL="sandbox.fast.io/kvm"
-FC_NODE_LABEL="fast-sandbox.io/firecracker-node"
-
-INOTIFY_VALUE="${INOTIFY_VALUE:-8192}"
-SYSCTL_BACKUP="$WORK/sysctl-backup"
-SKIP_TOOL_INSTALL="${SKIP_TOOL_INSTALL:-0}"
-SKIP_LEFTOVER_CLEAN="${SKIP_LEFTOVER_CLEAN:-0}"
-KIND_VERSION="${KIND_VERSION:-v0.24.0}"
-KUBECTL_VERSION="${KUBECTL_VERSION:-v1.31.0}"
-HELM_VERSION="${HELM_VERSION:-v3.16.4}"
-
-# opensandbox CLI for the verify stages; installed by ensure_osb.
-OSB_PACKAGE="${OSB_PACKAGE:-opensandbox-cli}"
-OSB_BIN="${OSB_BIN:-}"
-OSB_VENV="$WORK/tools/osb-venv"
-
-# Internal goproxy mirrors can 500 on shared hosts; direct VCS just works there.
-FSB_GOPROXY="${FSB_GOPROXY:-direct}"
-export GOPROXY="$FSB_GOPROXY"
-
-AUTO_CLEAN=0
-ACTION=""
-
-# --- logging / stage machinery --------------------------------------------------
-
-log() { printf '\033[1;34m[fast-sandbox-env]\033[0m %s\n' "$*" | tee -a "$WORK/run.log" >&2; }
-die() { printf '\033[1;31m[fast-sandbox-env] ERROR:\033[0m %s\n' "$*" >&2; exit 1; }
-pass() { printf '\033[1;32m[fast-sandbox-env] PASS\033[0m %s\n' "$*" | tee -a "$WORK/run.log" >&2; }
-# fail dumps before exiting: an explicit `exit` does not fire the ERR trap.
-fail() { printf '\033[1;31m[fast-sandbox-env] FAIL\033[0m %s\n' "$*" >&2; failure_dump "${STAGE_CUR_LABEL:-$ACTION}" || true; exit 1; }
-highlight() { printf '\033[1;36m%s\033[0m\n' "$*"; }
-
-now_ms() { date +%s%N; } # GNU date (Linux); the script targets a Linux KVM host
-ms2s() { awk -v ms="$1" 'BEGIN { printf "%.1f", ms / 1000 }'; }
-
-declare -a STAGE_ORDER=()
-declare -a STAGE_MS_LIST=()
-STAGE_CUR=""
-STAGE_CUR_LABEL=""
-STAGE_START_NS=0
-STAGE_N=0
-
-stage_begin() {
- STAGE_N=$((STAGE_N + 1))
- STAGE_CUR="$1"
- STAGE_CUR_LABEL="$(printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_')"
- STAGE_START_NS="$(now_ms)"
- printf '\n\033[1;36m==> [%d] %s\033[0m\n' "$STAGE_N" "$1"
-}
-
-stage_done() {
- local ms
- ms=$(( ($(now_ms) - STAGE_START_NS) / 1000000 ))
- STAGE_ORDER+=("$STAGE_CUR")
- STAGE_MS_LIST+=("$ms")
- printf '\033[1;32m OK in %ss\033[0m %s\n' "$(ms2s "$ms")" "${1:-}"
-}
-
-run_stage() {
- local description="$1" func="$2"
- shift 2
- stage_begin "$description"
- "$func" "$@"
- stage_done
-}
-
-stage_summary() {
- local index name ms total=0
- highlight "== stage timings =="
- printf ' \033[1m%-46s %10s\033[0m\n' "stage" "duration"
- for index in "${!STAGE_ORDER[@]}"; do
- name="${STAGE_ORDER[$index]}"
- ms="${STAGE_MS_LIST[$index]}"
- total=$((total + ms))
- printf ' %-46s %9ss\n' "$name" "$(ms2s "$ms")"
- done
- printf ' \033[1m%-46s %9ss\033[0m\n' "TOTAL" "$(ms2s "$total")"
-}
-
-wait_for() { # description attempts command [args...]
- local description="$1" attempts="$2" attempt=0
- shift 2
- while ! "$@" >/dev/null 2>&1; do
- attempt=$((attempt + 1))
- if [[ "$attempt" -ge "$attempts" ]]; then
- fail "$description (after $attempts attempts)"
- fi
- sleep 2
- done
- pass "$description"
-}
-
-kubectl_get() { kubectl -n "$RESOURCE_NS" get "$1" -o jsonpath="$2"; }
-
-sudo_() { if [[ "$(id -u)" == 0 ]]; then "$@"; else sudo "$@"; fi; }
-
-# --- helpers ---------------------------------------------------------------------
-
-kind_node() { kind get nodes --name "$KIND_CLUSTER" 2>/dev/null | head -1; }
-
-kind_network() { # docker network of the first node container
- local node
- node="$(kind_node)" || return 1
- docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$node" | tr ' ' '\n' | grep -v '^$' | head -1
-}
-
-rc() {
- docker run --rm --network host --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
- -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
- -e http_proxy= -e https_proxy= -e all_proxy= \
- -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" "$@"
-}
-
-# --- failure dump ------------------------------------------------------------------
-
-on_error() {
- local rc=$? task="$1"
- # Subshells inherit ERR: propagate the error instead of tearing down
- # main-process resources; ignore SIGPIPE (cancelled runners).
- trap - ERR
- [[ "$BASHPID" == "$ENV_PID" ]] || exit "$rc"
- trap '' PIPE
- # Dump before --auto-clean teardown destroys the evidence.
- failure_dump "$task" || true
- if [[ "$AUTO_CLEAN" == 1 ]]; then
- log "$ACTION failed at $task; --auto-clean: running down" || true
- (down) >/dev/null 2>&1 || true
- fi
- printf '\033[1;31m[fast-sandbox-env] FAILED at %s; dump: %s\033[0m\n' \
- "$task" "$LOGS_DIR/failure-$task-*.txt" >&2 || true
- exit "$rc"
-}
-
-failure_dump() {
- local task="$1"
- local dump="$LOGS_DIR/failure-$task-$(date +%s).txt"
- mkdir -p "$LOGS_DIR"
- {
- echo "=== fast-sandbox-env failure: $task ($(date -u +%FT%TZ)) ==="
- env | grep -E '^(KIND_(CLUSTER|SINGLE|RETAIN|NODE_IMAGE)=|FSB_DIR=|SBX_IMAGE=|IMG_[A-Z_]+=|EGRESS_IMAGE=|EXECD=|WORK=|POOL_(NAME|MIN|MAX)=|INGRESS_IMAGE=|WARM_IMAGES=|XFS_(STATEROOT|SIZE)=|RUSTFS_(IMAGE|PORT|CONSOLE_PORT|BUCKET|CONTAINER|DATA|ENDPOINT)=|RC_IMAGE=)' || true
- echo "--- fast-sandbox checkout ---"
- git -C "$FSB_DIR" rev-parse HEAD 2>&1 || true
- echo "--- kind-create.log (tail) ---"
- tail -40 "$LOGS_DIR/kind-create.log" 2>&1 || true
- echo "--- nodes ---"
- kubectl get nodes -o wide 2>&1 || true
- echo "--- pods ---"
- kubectl get pods -n "$NS" -o wide 2>&1 || true
- echo "--- controller logs (tail) ---"
- kubectl logs -n "$NS" deploy/fast-sandbox-controller --tail=80 2>&1 || true
- echo "--- firecracker-runtime logs (tail) ---"
- kubectl logs -n "$NS" daemonset/firecracker-runtime --all-containers --tail=80 2>&1 || true
- # --all-containers: the label selector alone logs only the first
- # container and drops egress. Deep tail: failure loops flood the
- # window and bury the pre-incident history.
- echo "--- fastlet pod logs, every container (deep tail) ---"
- kubectl logs -n "$RESOURCE_NS" -l app=sandbox-fastlet \
- --all-containers --prefix --timestamps --tail=15000 2>&1 || true
- echo "--- fastlet pod logs, previous container instances (crashes) ---"
- for pod in $(kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet -o name 2>/dev/null); do
- for container in $(kubectl -n "$RESOURCE_NS" get "$pod" -o jsonpath='{.spec.containers[*].name}' 2>/dev/null); do
- prev="$(kubectl -n "$RESOURCE_NS" logs "$pod" -c "$container" --previous --timestamps --tail=200 2>&1 || true)"
- case "$prev" in *"no previous container"*) continue ;; esac
- echo "### ${pod##*/} container=$container (previous)"
- printf '%s\n' "$prev"
- done
- done
- echo "--- builder pods + logs (tail) ---"
- kubectl get pods -n "$RESOURCE_NS" -l sandbox.fast.io/sandboxtemplate --show-labels 2>&1 || true
- kubectl logs -n "$RESOURCE_NS" -l sandbox.fast.io/sandboxtemplate --tail=80 2>&1 || true
- echo "--- SandboxTemplates (status carries the build failure reason) ---"
- kubectl get sandboxtemplates -n "$RESOURCE_NS" -o yaml 2>&1 || true
- # Sandbox CR states: a pause/snapshot/resume failure shows up as a
- # stuck or failed status condition here, with the phase the poll
- # never reached.
- echo "--- Sandbox CRs (status only) ---"
- kubectl get sandboxes -n "$RESOURCE_NS" -o json 2>/dev/null \
- | jq -c '.items[] | {name: .metadata.name, phase: .status.phase, conditions: .status.conditions, infra: .status.infraComponents}' 2>&1 || true
- # A mount-namespace mismatch (driver staging path invisible to
- # egress) shows up as ENOENT on the egress side only.
- for pod in $(kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet -o name 2>/dev/null); do
- echo "### ${pod##*/}: snapshot staging visibility"
- echo "-- fastlet view:"
- kubectl -n "$RESOURCE_NS" exec "$pod" -c fastlet -- \
- ls -la /var/lib/fast-sandbox/firecracker/snapshots 2>&1 | tail -8 || true
- echo "-- egress view:"
- kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- \
- ls -la /var/lib/fast-sandbox/firecracker/snapshots 2>&1 | tail -8 || true
- done
- echo "--- recent events ($RESOURCE_NS) ---"
- kubectl get events -n "$RESOURCE_NS" --sort-by=.lastTimestamp 2>&1 | tail -30 || true
- echo "--- OpenSandbox pods ($OSB_NS) ---"
- kubectl get pods -n "$OSB_NS" -o wide 2>&1 || true
- echo "--- server logs (tail) ---"
- kubectl logs -n "$OSB_NS" deploy/opensandbox-server --tail=80 2>&1 || true
- echo "--- ingress gateway logs (tail) ---"
- kubectl logs -n "$OSB_NS" deploy/opensandbox-ingress-gateway --tail=80 2>&1 || true
- echo "--- pool ---"
- kubectl get sandboxpool -n "$RESOURCE_NS" -o yaml 2>&1 || true
- echo "--- rustfs docker logs (tail) ---"
- docker logs "$RUSTFS_CONTAINER" --tail=80 2>&1 || true
- } > "$dump" 2>&1 || true
- log "failure dump: $dump"
-}
-
-# --- stage: preflight + tooling ------------------------------------------------------
-
-install_release_binary() { # name version url
- local name="$1" version="$2" url="$3" tmp
- log "installing $name $version -> /usr/local/bin/$name"
- tmp="$(mktemp -d)"
- curl -fL --retry 3 -o "$tmp/$name" "$url" \
- || die "download $name failed ($url); install it manually or retry"
- sudo_ install -m 0755 "$tmp/$name" "/usr/local/bin/$name"
- rm -rf "$tmp"
-}
-
-ensure_tool() { # name
- local name="$1"
- if command -v "$name" >/dev/null 2>&1; then
- return 0
- fi
- if [[ "$SKIP_TOOL_INSTALL" == 1 ]]; then
- die "$name is required (SKIP_TOOL_INSTALL=1: install it manually)"
- fi
- case "$name" in
- kind)
- install_release_binary kind "$KIND_VERSION" \
- "https://github.com/kubernetes-sigs/kind/releases/download/$KIND_VERSION/kind-linux-amd64"
- ;;
- kubectl)
- install_release_binary kubectl "${KUBECTL_VERSION#v}" \
- "https://dl.k8s.io/release/$KUBECTL_VERSION/bin/linux/amd64/kubectl"
- ;;
- helm)
- local tmp
- log "installing helm $HELM_VERSION -> /usr/local/bin/helm"
- tmp="$(mktemp -d)"
- curl -fL --retry 3 -o "$tmp/helm.tgz" \
- "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \
- || die "download helm failed; install it manually or retry"
- sudo_ tar -xzf "$tmp/helm.tgz" -C "$tmp" linux-amd64/helm
- sudo_ install -m 0755 "$tmp/linux-amd64/helm" /usr/local/bin/helm
- rm -rf "$tmp"
- ;;
- jq)
- log "installing jq via package manager"
- if command -v apt-get >/dev/null 2>&1; then
- sudo_ apt-get install -y jq >/dev/null
- elif command -v yum >/dev/null 2>&1; then
- sudo_ yum install -y jq >/dev/null
- elif command -v apk >/dev/null 2>&1; then
- sudo_ apk add --no-cache jq >/dev/null
- else
- die "no supported package manager to install jq; install it manually"
- fi
- ;;
- *)
- die "$name is required; install it manually"
- ;;
- esac
- command -v "$name" >/dev/null 2>&1 || die "$name installation failed"
-}
-
-ensure_osb() {
- if [[ -n "$OSB_BIN" ]]; then
- [[ -x "$OSB_BIN" ]] || die "OSB_BIN=$OSB_BIN is not executable"
- log "using osb from OSB_BIN: $OSB_BIN"
- else
- if [[ "$SKIP_TOOL_INSTALL" == 1 ]] && [[ ! -x "$OSB_VENV/bin/osb" ]]; then
- die "osb is required (SKIP_TOOL_INSTALL=1: install opensandbox-cli manually or set OSB_BIN)"
- fi
- command -v uv >/dev/null 2>&1 \
- || die "uv is required to install the opensandbox CLI (or set OSB_BIN)"
- log "installing the latest $OSB_PACKAGE into $OSB_VENV (uv)"
- : > "$LOGS_DIR/osb-install.log"
- rm -rf "$OSB_VENV"
- # Fresh venv at its final path: venv console-script shebangs embed the
- # interpreter's absolute path, so create-elsewhere-and-move breaks them.
- uv venv --python 3.12 "$OSB_VENV" >>"$LOGS_DIR/osb-install.log" 2>&1 \
- || { tail -n 20 "$LOGS_DIR/osb-install.log" >&2 || true; die "uv venv --python 3.12 failed (full log: $LOGS_DIR/osb-install.log; or set OSB_BIN)"; }
- uv pip install --python "$OSB_VENV/bin/python" --upgrade "$OSB_PACKAGE" \
- >>"$LOGS_DIR/osb-install.log" 2>&1 \
- || { tail -n 20 "$LOGS_DIR/osb-install.log" >&2 || true; die "uv pip install --upgrade $OSB_PACKAGE failed (index unreachable? full log: $LOGS_DIR/osb-install.log; or set OSB_BIN)"; }
- OSB_BIN="$OSB_VENV/bin/osb"
- fi
- log "osb version: $("$OSB_BIN" --version 2>/dev/null | tail -n1 || echo unknown)"
- pass "opensandbox CLI ready ($OSB_BIN)"
-}
-
-host_port_busy() { # port -> 0 when something already listens on 127.0.0.1:<port>
- (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null
-}
-
-# Pull only when absent locally (*_IMAGE overrides cover private registries).
-ensure_image() { # image -> pull only when missing locally
- docker image inspect "$1" >/dev/null 2>&1 && return 0
- docker pull -q "$1" >/dev/null || die "image $1 is not available locally and the pull failed (pre-load it with docker load, or override the *_IMAGE variable)"
-}
-
-preflight() {
- [[ "$(uname -s)" == "Linux" ]] \
- || die "this environment requires a Linux host with KVM (run it on the remote development VM)"
- command -v docker >/dev/null || die "docker is required"
- command -v go >/dev/null || die "go is required (>=1.25, used by make images and gen-registry)"
- ensure_tool kind
- ensure_tool kubectl
- ensure_tool helm
- ensure_tool jq
- docker info >/dev/null 2>&1 || die "docker daemon is not reachable"
- local cgver
- cgver="$(docker info --format '{{.CgroupVersion}}' 2>/dev/null || true)"
- log "docker cgroup version=$cgver"
- # kind requires cgroup v2; v1 hosts fail kubepods cgroup creation.
- if [[ "$cgver" == "1" ]]; then
- die "docker cgroup Version is 1; kind requires cgroup v2. Enable it with the kernel cmdline 'systemd.unified_cgroup_hierarchy=1' and reboot"
- fi
- [[ -e /dev/kvm ]] || die "/dev/kvm is missing on this host (KVM required)"
- # Fail fast per heavy-data target instead of ENOSPC mid-run.
- local min_free_kb=$((20 * 1024 * 1024)) avail_kb target xfs_dir
- xfs_dir="${XFS_LOOP_FILE%/*}"
- mkdir -p "$WORK" "$RUSTFS_DATA" "$xfs_dir" 2>/dev/null || true
- local -a targets=("$WORK" "$RUSTFS_DATA" "$xfs_dir")
- for target in "${targets[@]}"; do
- avail_kb="$(df -Pk "$target" 2>/dev/null | awk 'NR==2 {print $4}')"
- [[ "$avail_kb" =~ ^[0-9]+$ ]] || die "cannot determine free disk space on $target"
- if (( avail_kb < min_free_kb )); then
- die "$target has $((avail_kb / 1024 / 1024))G free; at least 20G is required (built images, XFS StateRoot, RustFS artifacts). Free space (docker system prune / old kind clusters) or point WORK / RUSTFS_DATA / XFS_LOOP_FILE at a bigger volume"
- fi
- log "disk headroom: $target has $((avail_kb / 1024 / 1024 / 1024))G free"
- done
- # Fail fast on busy host ports instead of a cryptic docker/kind bind error.
- local port
- for port in "$RUSTFS_PORT" "$RUSTFS_CONSOLE_PORT" "$SERVER_HOST_PORT" "$GATEWAY_HOST_PORT"; do
- if host_port_busy "$port"; then
- die "127.0.0.1:$port is already in use (check 'ss -ltnp' / 'docker ps'); free it, or set RUSTFS_PORT / RUSTFS_CONSOLE_PORT / SERVER_HOST_PORT / GATEWAY_HOST_PORT"
- fi
- done
- ensure_image "$RUSTFS_IMAGE"
- ensure_image "$RC_IMAGE"
- pass "preflight"
-}
-
-sysctl_set() {
- local current
- current="$(sysctl -n fs.inotify.max_user_instances)"
- [[ "$current" -ge "$INOTIFY_VALUE" ]] && return 0
- echo "$current" > "$SYSCTL_BACKUP"
- log "sysctl fs.inotify.max_user_instances: $current -> $INOTIFY_VALUE"
- sudo sysctl -w fs.inotify.max_user_instances="$INOTIFY_VALUE" >/dev/null
-}
-
-sysctl_restore() {
- [[ -f "$SYSCTL_BACKUP" ]] || return 0
- local previous
- previous="$(cat "$SYSCTL_BACKUP")"
- log "restoring fs.inotify.max_user_instances -> $previous"
- sudo sysctl -w fs.inotify.max_user_instances="$previous" >/dev/null || true
- rm -f "$SYSCTL_BACKUP"
-}
-
-# --- stage: fast-sandbox checkout @ master ---------------------------------------------
-
-# Scratch Go tool run inside the fast-sandbox module (imports
-# internal/registryconfig); removed before dirty checks and on down.
-FSB_GEN_DIR="$FSB_DIR/.fast-sandbox-env-gen"
-
-ensure_fsb() {
- if [[ ! -d "$FSB_DIR/.git" ]]; then
- log "cloning $FSB_REPO into $FSB_DIR"
- git clone "$FSB_REPO" "$FSB_DIR" || die "clone failed; check network"
- fi
- rm -rf "$FSB_GEN_DIR"
- [[ -z "$(git -C "$FSB_DIR" status --porcelain)" ]] \
- || die "fast-sandbox checkout at $FSB_DIR has local changes; delete it to re-clone or point FSB_DIR at a clean checkout"
- # Raw SHA fetch needs allow-reachable-sha1-in-want; full fetch otherwise.
- if ! git -C "$FSB_DIR" fetch -q origin "$FSB_COMMIT" 2>/dev/null; then
- git -C "$FSB_DIR" fetch -q origin '+refs/heads/*:refs/remotes/origin/*' \
- || die "git fetch failed for $FSB_REPO"
- fi
- git -C "$FSB_DIR" rev-parse --verify --quiet "$FSB_COMMIT^{commit}" >/dev/null \
- || die "pinned commit $FSB_COMMIT is not reachable from $FSB_REPO"
- if [[ "$(git -C "$FSB_DIR" rev-parse HEAD)" != "$FSB_COMMIT" ]]; then
- git -C "$FSB_DIR" clean -ffdx
- fi
- git -C "$FSB_DIR" checkout --force -q "$FSB_COMMIT" \
- || die "git checkout $FSB_COMMIT failed"
- [[ "$(git -C "$FSB_DIR" rev-parse HEAD)" == "$FSB_COMMIT" ]] \
- || die "fast-sandbox checkout is not at the pinned commit $FSB_COMMIT"
- log "fast-sandbox @ pinned $(git -C "$FSB_DIR" rev-parse --short HEAD) (manifests/third-party/fast-sandbox.commit)"
- pass "fast-sandbox checkout ready"
-}
-
-# --- stage: images ---------------------------------------------------------------------
-
-build_images() {
- log "building fast-sandbox images (pinned commit, firecracker scope) via manifests/release/build-fast-sandbox.sh"
- # Same checkout and defaults as the standalone builder: env and published
- # build paths cannot drift.
- FSB_SRC_DIR="$FSB_DIR" "$OSB_ROOT/manifests/release/build-fast-sandbox.sh" \
- || die "fast-sandbox image build failed"
- log "building the OpenSandbox egress image ($IMG_EGRESS)"
- # Context is the repo root (COPYs components/egress + components/internal).
- # shellcheck disable=SC2086
- docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
- -f "$OSB_ROOT/components/egress/Dockerfile" -t "$IMG_EGRESS" "$OSB_ROOT" >/dev/null \
- || die "egress image build failed"
- log "building the OpenSandbox server image ($IMG_SERVER)"
- # Context is server/ (self-contained Dockerfile).
- # shellcheck disable=SC2086
- docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
- -f "$OSB_ROOT/server/Dockerfile" -t "$IMG_SERVER" "$OSB_ROOT/server" >/dev/null \
- || die "server image build failed"
- log "building the OpenSandbox ingress image ($IMG_INGRESS)"
- # shellcheck disable=SC2086
- docker build ${DOCKER_BUILD_FLAGS:-} --quiet \
- -f "$OSB_ROOT/components/ingress/Dockerfile" -t "$IMG_INGRESS" "$OSB_ROOT" >/dev/null \
- || die "ingress image build failed"
- pass "images built (fast-sandbox + OpenSandbox)"
-}
-
-# --- stage: XFS StateRoot (reflink CoW per-sandbox rootfs) ------------------------------
-
-XFS_STATEROOT="${XFS_STATEROOT:-1}"
-XFS_LOOP_FILE="${XFS_LOOP_FILE:-$WORK/fast-sandbox.img}"
-XFS_SIZE="${XFS_SIZE:-24G}"
-XFS_MOUNT_POINT="${XFS_MOUNT_POINT:-/var/lib/fast-sandbox}"
-
-ensure_xfsprogs() {
- command -v mkfs.xfs >/dev/null 2>&1 && return 0
- log "installing xfsprogs (mkfs.xfs)"
- if command -v apt-get >/dev/null 2>&1; then
- sudo_ apt-get install -y xfsprogs >/dev/null
- elif command -v yum >/dev/null 2>&1; then
- sudo_ yum install -y xfsprogs >/dev/null
- else
- die "xfsprogs not installed and no supported package manager"
- fi
-}
-
-stateroot_xfs_up() {
- [[ "$XFS_STATEROOT" == 1 ]] || {
- sudo_ mkdir -p "$XFS_MOUNT_POINT"
- log "XFS StateRoot disabled (XFS_STATEROOT=0); per-sandbox rootfs pays a full copy"
- return 0
- }
- if findmnt -no FSTYPE "$XFS_MOUNT_POINT" 2>/dev/null | grep -qx xfs; then
- # Leftover from a run killed before teardown: scrub, and recreate the
- # image if still below the agent's min-free floor (it strips the kvm
- # node label and template builds stop scheduling).
- stateroot_scrub
- if (( $(stateroot_free_gib) >= XFS_MIN_FREE_GIB )); then
- pass "XFS StateRoot ready (reflink CoW rootfs, scrubbed)"
- return 0
- fi
- log "StateRoot below ${XFS_MIN_FREE_GIB}GiB free after scrub; recreating the XFS image"
- sudo_ umount "$XFS_MOUNT_POINT" || die "unmount $XFS_MOUNT_POINT failed"
- rm -f "$XFS_LOOP_FILE"
- fi
- ensure_xfsprogs
- if [[ ! -f "$XFS_LOOP_FILE" ]]; then
- log "creating sparse XFS image $XFS_LOOP_FILE (virtual $XFS_SIZE)"
- truncate -s "$XFS_SIZE" "$XFS_LOOP_FILE"
- sudo_ mkfs.xfs -f "$XFS_LOOP_FILE" >/dev/null 2>&1 || die "mkfs.xfs failed on $XFS_LOOP_FILE"
- fi
- sudo_ mkdir -p "$XFS_MOUNT_POINT"
- sudo_ mount -o noatime "$XFS_LOOP_FILE" "$XFS_MOUNT_POINT" \
- || die "mount $XFS_LOOP_FILE at $XFS_MOUNT_POINT failed (loop support? try XFS_STATEROOT=0)"
- local a b
- a="$XFS_MOUNT_POINT/.reflink-a"
- b="$XFS_MOUNT_POINT/.reflink-b"
- printf 'probe' | sudo_ tee "$a" >/dev/null
- if sudo_ cp --reflink=always "$a" "$b"; then
- sudo_ rm -f "$a" "$b"
- pass "XFS StateRoot ready (reflink CoW rootfs)"
- else
- sudo_ rm -f "$a" "$b"
- die "reflink probe failed on $XFS_MOUNT_POINT (CoW rootfs would not work)"
- fi
-}
-
-# Purge per-node runtime caches in the mounted StateRoot (shared by down()
-# and the reuse branch of stateroot_xfs_up); snapshots/ leaks across runs.
-stateroot_scrub() {
- local node_dir
- for node_dir in "$XFS_MOUNT_POINT"/*/; do
- [[ -d "${node_dir}firecracker" ]] || continue
- log "purging node runtime cache under ${node_dir%/}"
- sudo_ rm -rf "${node_dir}firecracker/images" \
- "${node_dir}firecracker/agent" \
- "${node_dir}firecracker/jails" \
- "${node_dir}firecracker/cache" \
- "${node_dir}firecracker/snapshots" 2>/dev/null || true
- done
-}
-
-# Free space on the mounted StateRoot, in GiB (GNU df).
-stateroot_free_gib() {
- df -BG --output=avail "$XFS_MOUNT_POINT" 2>/dev/null | tail -1 | tr -dc '0-9' || echo 0
-}
-XFS_MIN_FREE_GIB="${XFS_MIN_FREE_GIB:-12}"
-
-stateroot_xfs_down() {
- [[ "$XFS_STATEROOT" == 1 ]] || return 0
- if findmnt -no SOURCE "$XFS_MOUNT_POINT" 2>/dev/null | grep -q "$(basename "$XFS_LOOP_FILE")"; then
- log "unmounting XFS StateRoot $XFS_MOUNT_POINT"
- sudo_ umount "$XFS_MOUNT_POINT"
- fi
- rm -f "$XFS_LOOP_FILE"
-}
-
-# --- stage: kind cluster -----------------------------------------------------------------
-
-render_kind_config() { # > $GEN_DIR/kind-cluster.yaml
- local src="$MANIFESTS_DIR/cluster/kind-cluster.yaml" out="$GEN_DIR/kind-cluster.yaml"
- mkdir -p "$GEN_DIR"
- cp "$src" "$out"
- if [[ "$KIND_SINGLE" == "1" ]]; then
- # Strip the worker node: cache-only topology, no peer traffic.
- awk '/^- role: worker/ {exit} {print}' "$out" > "$out.tmp" && mv "$out.tmp" "$out"
- log "single-node topology (KIND_SINGLE=1: no worker, no peer traffic)"
- fi
- if [[ -n "${DOCKER_MIRROR:-}" ]]; then
- # Host-specific, hence opt-in; one TOML endpoint ARRAY (repeated
- # endpoint keys would override each other).
- local block_file="$GEN_DIR/docker-mirror-block.yaml" mirror trimmed endpoints=""
- IFS=',' read -ra mirrors <<<"$DOCKER_MIRROR"
- for mirror in "${mirrors[@]}"; do
- trimmed="$(printf '%s' "$mirror" | tr -d '[:space:]')"
- [[ -n "$trimmed" ]] || continue
- [[ -z "$endpoints" ]] && endpoints="\"$trimmed\"" || endpoints+=", \"$trimmed\""
- done
- [[ -n "$endpoints" ]] || die "DOCKER_MIRROR produced no usable endpoints"
- {
- echo 'containerdConfigPatches:'
- echo '- |-'
- echo ' [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]'
- echo " endpoint = [$endpoints]"
- } > "$block_file"
- awk -v block_file="$block_file" '
- NR == FNR { block = block $0 "\n"; next }
- /^nodes:/ && !done { printf "%s", block; done = 1 }
- { print }
- ' "$block_file" "$out" > "$out.tmp" && mv "$out.tmp" "$out"
- rm -f "$block_file"
- log "docker.io containerd mirrors injected: $endpoints"
- fi
- # extraPortMappings publish the server/gateway on the host; they exist
- # only at cluster creation, so a cluster reused without them stays
- # cluster-internal.
- local ports_file="$GEN_DIR/osb-ports-block.yaml"
- {
- echo ' extraPortMappings:'
- echo " - containerPort: $SERVER_NODEPORT"
- echo " hostPort: $SERVER_HOST_PORT"
- echo ' listenAddress: 127.0.0.1'
- echo ' protocol: TCP'
- echo " - containerPort: $GATEWAY_NODEPORT"
- echo " hostPort: $GATEWAY_HOST_PORT"
- echo ' listenAddress: 127.0.0.1'
- echo ' protocol: TCP'
- } > "$ports_file"
- awk -v block_file="$ports_file" '
- NR == FNR { block = block $0 "\n"; next }
- /^- role: control-plane/ && !done { print; printf "%s", block; done = 1; next }
- { print }
- ' "$ports_file" "$out" > "$out.tmp" && mv "$out.tmp" "$out"
- rm -f "$ports_file"
- log "server on 127.0.0.1:$SERVER_HOST_PORT, gateway on 127.0.0.1:$GATEWAY_HOST_PORT (NodePorts $SERVER_NODEPORT/$GATEWAY_NODEPORT)"
-}
-
-kind_up() {
- local create_args=() kind_config="$GEN_DIR/kind-cluster.yaml"
- [[ -f "$kind_config" ]] || die "kind config not rendered ($kind_config)"
- [[ "$KIND_RETAIN" == 1 ]] && create_args+=(--retain)
- if [[ -n "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]]; then
- log "cluster $KIND_CLUSTER already exists; reusing (run down first for a clean rebuild)"
- else
- if [[ -n "${KIND_NODE_IMAGE:-}" ]]; then
- log "pulling kind node image $KIND_NODE_IMAGE (this can take minutes)"
- ensure_image "$KIND_NODE_IMAGE" || die "kind node image unavailable locally and pull failed (KIND_NODE_IMAGE=$KIND_NODE_IMAGE)"
- create_args+=(--image "$KIND_NODE_IMAGE")
- else
- log "creating cluster (pulling kindest/node may take minutes; set KIND_NODE_IMAGE to a mirror if it fails)"
- fi
- kind create cluster --name "$KIND_CLUSTER" \
- ${create_args+"${create_args[@]}"} --config "$kind_config" > "$LOGS_DIR/kind-create.log" 2>&1 || {
- local rc=$?
- tail -40 "$LOGS_DIR/kind-create.log" >&2 || true
- log "kind create failed (full log: $LOGS_DIR/kind-create.log)" || true
- return "$rc" # through ERR, so diagnostics + --auto-clean run
- }
- pass "kind cluster created"
- fi
- local node
- for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
- docker exec "$node" sh -c 'test -e /dev/kvm' || die "KVM not visible inside the kind node container $node"
- log "node $node: /dev/kvm visible"
- done
- if [[ "$KIND_SINGLE" != "1" ]]; then
- # Untaint the control plane: agent/fastlet/builder must schedule on
- # BOTH nodes for P2P peer traffic.
- kubectl taint nodes --all node-role.kubernetes.io/control-plane- >/dev/null 2>&1 || true
- log "control-plane taint removed (both nodes schedulable for the P2P topology)"
- fi
- pass "kind cluster ready (KVM passthrough + labels on every node)"
-}
-
-# --- stage: RustFS + credentials ------------------------------------------------------------
-
-rustfs_up() {
- docker rm -f "$RUSTFS_CONTAINER" >/dev/null 2>&1 || true
- # RustFS runs as UID/GID 10001 and owns the bind-mounted store.
- sudo_ rm -rf "$RUSTFS_DATA"
- mkdir -p "$RUSTFS_DATA"
- sudo_ chown -R 10001:10001 "$RUSTFS_DATA"
- mkdir -p "$RC_CONFIG_DIR"
- chmod 700 "$RC_CONFIG_DIR"
- local net
- net="$(kind_network)"
- # On the kind network: pods reach the container IP directly (no
- # docker-proxy/hairpin issues); 127.0.0.1 publishes serve the host.
- docker run -d --name "$RUSTFS_CONTAINER" --network "$net" \
- -p 127.0.0.1:"$RUSTFS_PORT":"$RUSTFS_CONTAINER_PORT" -p 127.0.0.1:"$RUSTFS_CONSOLE_PORT":9001 \
- -e RUSTFS_ACCESS_KEY="$RUSTFS_AK" -e RUSTFS_SECRET_KEY="$RUSTFS_SK" \
- -e RUSTFS_ADDRESS=":$RUSTFS_CONTAINER_PORT" \
- -e RUSTFS_CONSOLE_ADDRESS=:9001 -e RUSTFS_CONSOLE_ENABLE=true \
- -v "$RUSTFS_DATA:/data" \
- "$RUSTFS_IMAGE" >/dev/null
- local attempt
- for attempt in $(seq 1 30); do
- if curl -fsS "http://127.0.0.1:$RUSTFS_PORT/health" >/dev/null 2>&1; then break; fi
- sleep 1
- [[ "$attempt" == 30 ]] && die "RustFS did not become healthy"
- done
- rc alias set chain "http://127.0.0.1:$RUSTFS_PORT" "$RUSTFS_AK" "$RUSTFS_SK" >/dev/null 2>&1 \
- || die "could not configure the RustFS rc alias"
- for attempt in $(seq 1 30); do
- if rc ls chain/ >/dev/null 2>&1; then break; fi
- sleep 1
- [[ "$attempt" == 30 ]] && die "RustFS S3 API did not become ready for authenticated bucket listing"
- done
- rc mb "chain/$RUSTFS_BUCKET" >/dev/null
- pass "RustFS up (bucket=$RUSTFS_BUCKET)"
-}
-
-resolve_rustfs_endpoint() {
- if [[ -n "$RUSTFS_ENDPOINT" ]]; then
- log "RustFS endpoint (env): $RUSTFS_ENDPOINT"
- else
- local net ips ip
- net="$(kind_network)"
- ips="$(docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{$v.IPAddress}} {{end}}' "$RUSTFS_CONTAINER")"
- ip="$(printf '%s' "$ips" | tr ' ' '\n' | grep -A1 -x "^$net$" | tail -1)"
- [[ -n "$ip" ]] || die "could not find the RustFS IP on network $net (inspect: $ips)"
- # Container port, not the host publish: kind-network clients hit the
- # container IP directly.
- RUSTFS_ENDPOINT="http://$ip:$RUSTFS_CONTAINER_PORT"
- log "RustFS endpoint (kind network IP): $RUSTFS_ENDPOINT"
- fi
- local net
- net="$(kind_network)"
- docker run --rm --network "$net" --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
- -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
- -e http_proxy= -e https_proxy= -e all_proxy= \
- -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" alias set kind-rustfs \
- "$RUSTFS_ENDPOINT" "$RUSTFS_AK" "$RUSTFS_SK" >/dev/null 2>&1 \
- || die "RustFS unreachable from the kind network at $RUSTFS_ENDPOINT (override RUSTFS_ENDPOINT)"
- docker run --rm --network "$net" --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
- -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
- -e http_proxy= -e https_proxy= -e all_proxy= \
- -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" ls "kind-rustfs/$RUSTFS_BUCKET" >/dev/null 2>&1 \
- || die "RustFS authenticated bucket access failed from the kind network at $RUSTFS_ENDPOINT"
- pass "RustFS reachable from the kind network"
-}
-
-# gen_registry builds the agent registry JSON via fast-sandbox's
-# registryconfig package; the write pair covers checkpoint publication.
-gen_registry() { # host username password endpoint [write-username write-password] > registry.json
- mkdir -p "$FSB_GEN_DIR"
- cat > "$FSB_GEN_DIR/gen-registry.go" <<'EOF'
-package main
-
-import (
- "fmt"
- "os"
-
- "fast-sandbox/internal/registryconfig"
-)
-
-func main() {
- if len(os.Args) != 5 && len(os.Args) != 7 {
- fmt.Fprintln(os.Stderr, "usage: gen-registry <host> <username> <password> <endpoint> [write-username write-password]")
- os.Exit(1)
- }
- credential := registryconfig.Credential{
- Host: os.Args[1], Username: os.Args[2], Password: os.Args[3], Endpoint: os.Args[4],
- }
- if len(os.Args) == 7 {
- // Optional publish (write) pair: empty keeps the store read-only.
- credential.WriteUsername, credential.WritePassword = os.Args[5], os.Args[6]
- }
- compiled, err := registryconfig.NewCompiled([]registryconfig.Credential{credential})
- if err != nil {
- fmt.Fprintln(os.Stderr, err)
- os.Exit(1)
- }
- payload, err := compiled.Marshal()
- if err != nil {
- fmt.Fprintln(os.Stderr, err)
- os.Exit(1)
- }
- os.Stdout.Write(payload)
-}
-EOF
- (cd "$FSB_DIR" && GOTOOLCHAIN=local go run .fast-sandbox-env-gen/gen-registry.go "$@")
-}
-
-credentials_up() {
- # Namespaces exist even after a partial up (resume support).
- kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- kubectl create namespace "$RESOURCE_NS" --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- local host
- host="${RUSTFS_ENDPOINT#http://}"
- host="${host#https://}"
- # Publish credentials for the builder Pod (dataplane namespace).
- kubectl -n "$RESOURCE_NS" create secret generic sandbox-oss-credentials \
- --from-literal=accessKeyId="$RUSTFS_AK" \
- --from-literal=secretAccessKey="$RUSTFS_SK" \
- --from-literal=endpoint="$RUSTFS_ENDPOINT" \
- --from-literal=region=us-east-1 \
- --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- # Agent pull+publish credentials (the write pair covers checkpoints).
- gen_registry "$host" "$RUSTFS_AK" "$RUSTFS_SK" "$RUSTFS_ENDPOINT" "$RUSTFS_AK" "$RUSTFS_SK" \
- > "$WORK/agent-registry.json"
- jq -e '.credentials[0].writeUsername' "$WORK/agent-registry.json" >/dev/null \
- || die "generated agent registry carries no write credential"
- kubectl -n "$NS" create secret generic fast-sandbox-agent-registry \
- --from-file=registry.json="$WORK/agent-registry.json" \
- --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- # Pull credentials for the fastlet; the artifact-store ConfigMap is
- # rendered by charts/fast-sandbox at install time.
- kubectl -n "$RESOURCE_NS" create secret docker-registry registry-rustfs \
- --docker-server="$host" --docker-username="$RUSTFS_AK" --docker-password="$RUSTFS_SK" \
- --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- kubectl -n "$RESOURCE_NS" create configmap fast-sandbox-registry \
- --from-literal="registries.yaml=registries:
- - host: $host
- secretRef:
- name: registry-rustfs
-" \
- --dry-run=client -o yaml | kubectl apply -f - >/dev/null
- pass "credentials written (publish/pull)"
-}
-
-# --- stage: control plane -------------------------------------------------------------------
-
-# helm is only a renderer; the cluster keeps no helm release state, so
-# re-runs stay plain idempotent `kubectl apply`.
-helm_render() { # release chart ns out [set-args...]
- local release="$1" chart="$2" ns="$3" out="$4"
- shift 4
- helm template "$release" "$chart" --namespace "$ns" "$@" > "$out" \
- || die "helm template $chart failed"
-}
-
-apply_ns() { # ns -> ensure the namespace exists (idempotent)
- kubectl create namespace "$1" --dry-run=client -o yaml 2>/dev/null |
- kubectl apply -f - >/dev/null
-}
-
-control_plane_up() {
- # Charts are the source of truth (CRDs + RBAC + control plane), replacing
- # the fast-sandbox checkout's kustomize applies and env-owned node manifests.
- apply_ns "$NS"
- helm_render fsb-base "$OSB_ROOT/manifests/charts/base" "$NS" "$GEN_DIR/fsb-base.yaml"
- kubectl apply -f "$GEN_DIR/fsb-base.yaml" >/dev/null
- helm_render fast-sandbox "$OSB_ROOT/manifests/charts/fast-sandbox" "$NS" \
- "$GEN_DIR/fast-sandbox.yaml" \
- --set controller.image.repository="$(image_repo "$IMG_CONTROLLER")" \
- --set controller.image.tag="$(image_tag "$IMG_CONTROLLER")" \
- --set controller.sandboxtemplateBuilderImage="$IMG_BUILDER" \
- --set controller.fastletProxyImage="$IMG_FASTLET_PROXY" \
- --set janitor.image.repository="$(image_repo "$IMG_JANITOR")" \
- --set janitor.image.tag="$(image_tag "$IMG_JANITOR")" \
- --set runtime.image.repository="$(image_repo "$IMG_RUNTIME")" \
- --set runtime.image.tag="$(image_tag "$IMG_RUNTIME")" \
- --set artifactStore.store="s3://$RUSTFS_BUCKET/publish" \
- --set artifactStore.endpoint="$RUSTFS_ENDPOINT"
- kubectl apply -f "$GEN_DIR/fast-sandbox.yaml" >/dev/null
- local image
- for image in "$IMG_CONTROLLER" "$IMG_FASTLET" "$IMG_FASTLET_PROXY" \
- "$IMG_JANITOR" "$IMG_RUNTIME" "$IMG_EGRESS"; do
- kind load docker-image "$image" --name "$KIND_CLUSTER" >/dev/null
- done
- wait_for "controller deployment ready" 120 \
- kubectl -n "$NS" rollout status deploy/fast-sandbox-controller --timeout=10s
- local crd
- for crd in sandboxpools sandboxtemplates sandboxes sandboxsnapshots; do
- kubectl get crd "$crd.sandbox.fast.io" >/dev/null 2>&1 || die "CRD $crd missing"
- done
- kubectl get crd batchsandboxes.sandbox.opensandbox.io >/dev/null 2>&1 \
- || die "CRD batchsandboxes.sandbox.opensandbox.io missing"
- pass "CRDs + control plane ready (charts @ pinned $(git -C "$FSB_DIR" rev-parse --short HEAD))"
-}
-
-# --- stage: firecracker runtime (node readiness + DART P2P) -----------------------------------
-
-runtime_node_labeled() { # node -> 0 when the agent applied both labels + condition
- local node="$1"
- kubectl get node "$node" -o json 2>/dev/null | jq -e \
- --arg fc "$FC_NODE_LABEL" --arg kvm "$KVM_NODE_LABEL" '
- (.metadata.labels[$fc] == "true") and
- (.metadata.labels[$kvm] == "true") and
- ([(.status.conditions // [])[]?
- | select(.type == "FirecrackerReady" and .status == "True")] | length > 0)
- ' >/dev/null
-}
-
-runtime_pods() {
- kubectl -n "$NS" get pods -l component=firecracker-runtime -o jsonpath='{.items[*].metadata.name}' 2>/dev/null
-}
-
-dart_roster_ready() { # pod expected-members
- local pod="$1" expected="$2" members
- members="$(kubectl exec -n "$NS" "$pod" -- sh -c \
- 'curl -fsS --noproxy "*" http://127.0.0.1:8147/admin/members' 2>/dev/null || true)"
- [[ "$(printf '%s' "$members" | grep -o '"id":' | wc -l | tr -d ' ')" == "$expected" ]]
-}
-
-runtime_up() {
- # DaemonSet + dart Service ship with charts/fast-sandbox; credentials land
- # in credentials_up before this stage.
- wait_for "firecracker-runtime DaemonSet ready" 120 \
- kubectl -n "$NS" rollout status daemonset/firecracker-runtime --timeout=10s
-
- # Positive wiring assertions: DART admin plane up, agent p2pUp=true
- # (a missing P2P daemon only degrades pulls to direct S3).
- local pod uid node pods
- pods="$(runtime_pods)"
- for pod in $pods; do
- uid="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.metadata.uid}')"
- node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}')"
- wait_for "dart admin /healthz on $node" 30 \
- kubectl exec -n "$NS" "$pod" -- sh -c \
- "curl -fsS --noproxy '*' http://127.0.0.1:8147/healthz | grep -q ok"
- wait_for "agent health p2pUp on $node" 30 \
- kubectl exec -n "$NS" "$pod" -- sh -c \
- "curl -fsS --noproxy '*' --unix-socket /run/fast-sandbox/firecracker/runtime.sock -H 'Content-Type: application/json' -d '{\"podUid\":\"$uid\",\"namespace\":\"$NS\"}' http://firecracker-agent/v1/health | grep -q '\"p2pUp\":true'"
- log "dart: $node dart pid=$(kubectl exec -n "$NS" "$pod" -- sh -c 'pgrep -x dart')"
- done
- # P2P roster: every daemon must see every peer before any pull, so later
- # nodes are fed by peers instead of the origin.
- local expected_members
- expected_members="$(printf '%s' "$pods" | wc -w | tr -d ' ')"
- for pod in $pods; do
- node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}')"
- wait_for "dart roster full on $node ($expected_members members)" 90 \
- dart_roster_ready "$pod" "$expected_members"
- done
- # The readiness loop applies labels + FirecrackerReady itself.
- for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
- wait_for "node $node labeled + FirecrackerReady" 120 \
- runtime_node_labeled "$node"
- done
- pass "firecracker-runtime healthy + DART roster=$expected_members + nodes FirecrackerReady"
-}
-
-# --- stage (opensandbox CLI): SandboxTemplate golden image ---------------------
-
-wait_succeeded() { # description attempts probe probe_failed
- local description="$1" attempts="$2" probe="$3" probe_failed="$4" attempt=0
- while ! "$probe" >/dev/null 2>&1; do
- if "$probe_failed" >/dev/null 2>&1; then
- failure_dump "template-failed"
- fail "$description (template entered Failed)"
- fi
- attempt=$((attempt + 1))
- if [[ "$attempt" -ge "$attempts" ]]; then
- failure_dump "template-timeout"
- fail "$description (after $attempts attempts)"
- fi
- sleep 2
- done
- pass "$description"
-}
-
-# Template build via the CLI (server /templates API behind it): the row is
-# projected onto a SandboxTemplate CRD; the build runs in fast-sandbox, so
-# the builder image must be in the cluster.
-TEMPLATE_ID=""
-
-_template_phase() {
- osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.phase // empty'
-}
-
-template_succeeded() {
- [[ "$(_template_phase)" == "Succeeded" ]]
-}
-
-template_failed() {
- local phase message
- phase="$(_template_phase)"
- [[ "$phase" == "Failed" ]] || return 1
- message="$(osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.message // empty')"
- log "template Failed: ${message:-<no message>}"
- return 0
-}
-
-template_up() {
- log "building the sandboxtemplate-builder image"
- # shellcheck disable=SC2086
- docker build ${DOCKER_BUILD_FLAGS:-} --quiet -t "$IMG_BUILDER" \
- -f "$FSB_DIR/build/Dockerfile.sandboxtemplate-builder" "$FSB_DIR" >/dev/null \
- || die "sandboxtemplate-builder image build failed"
- kind load docker-image "$IMG_BUILDER" --name "$KIND_CLUSTER" >/dev/null
- local created
- local -a tmpl_args=(
- template create
- --image "$SBX_IMAGE"
- --publish "s3://$RUSTFS_BUCKET/publish"
- --format native
- --resource cpu=1 --resource memory=512Mi --resource disk=2Gi
- --warmup-seconds 15
- --metadata origin=fast-sandbox-env
- )
- log "verify: creating the golden-image template via the opensandbox CLI (image=$SBX_IMAGE)"
- created="$(osb_api "${tmpl_args[@]}" -o json 2>/dev/null)" \
- || fail "template create failed against $SERVER_URL: $(osb_api "${tmpl_args[@]}" 2>&1 >/dev/null | head -c 400)"
- TEMPLATE_ID="$(printf '%s' "$created" | jq -r '.template_id')"
- printf '%s' "$TEMPLATE_ID" > "$WORK/template-id"
- [[ -n "$TEMPLATE_ID" && "$TEMPLATE_ID" != "null" ]] || fail "template create response carried no templateId"
- log "template id=$TEMPLATE_ID"
- wait_succeeded "template phase=Succeeded" 300 template_succeeded template_failed
- local manifest_ref
- manifest_ref="$(osb_api template get "$TEMPLATE_ID" -o json | jq -r '.status.manifest_ref // empty')"
- [[ -n "$manifest_ref" ]] || fail "template manifestRef is empty"
- log "template manifestRef: $manifest_ref"
- pass "SandboxTemplate Succeeded + artifacts published (via opensandbox CLI)"
-}
-
-# --- stage: SandboxPool (egress attached, P2P spread) --------------------------------------------
-
-pool_pods() {
- kubectl -n "$RESOURCE_NS" get pods \
- -l "app=sandbox-fastlet,fast-sandbox.io/pool=$POOL_NAME" \
- -o jsonpath='{.items[*].metadata.name}' 2>/dev/null
-}
-
-first_pool_pod() {
- kubectl -n "$RESOURCE_NS" get pods \
- -l "app=sandbox-fastlet,fast-sandbox.io/pool=$POOL_NAME" \
- -o jsonpath='{.items[0].metadata.name}' 2>/dev/null
-}
-
-fastlet_pods_ready() {
- local ready=0 pod
- for pod in $(pool_pods); do
- if kubectl -n "$RESOURCE_NS" get pod "$pod" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null | grep -q True; then
- ready=$((ready + 1))
- fi
- done
- [[ "$ready" -ge "$POOL_MIN" ]]
-}
-
-egress_containers_ready() {
- local pods pod
- pods="$(pool_pods)"
- [[ -n "$pods" ]] || return 1
- for pod in $pods; do
- kubectl -n "$RESOURCE_NS" get pod "$pod" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"="}{.ready}{" "}{end}' 2>/dev/null \
- | grep -q 'egress=true' || return 1
- done
-}
-
-# GET /_fastlet/v1/actions/status must echo the shared apiVersion, ready=true
-# and a non-empty instanceId; the handler binds Pod loopback only, so the
-# probe runs inside the egress container.
-egress_status_ready() {
- local pod out
- pod="$(first_pool_pod)"
- [[ -n "$pod" ]] || return 1
- out="$(kubectl -n "$RESOURCE_NS" exec "pod/$pod" -c egress -- \
- curl -fsS -m 5 "http://127.0.0.1:18080/_fastlet/v1/actions/status" 2>/dev/null)" || return 1
- [[ "$out" == *'"apiVersion":"sandbox.fast.io/actions/v1"'* && "$out" == *'"ready":true'* && "$out" == *'"instanceId":'* ]]
-}
-
-pool_status_ready() {
- local ready
- ready="$(kubectl_get "sandboxpool/$POOL_NAME" '{.status.readyPods}' 2>/dev/null || true)"
- [[ "$ready" =~ ^[0-9]+$ ]] && (( ready >= POOL_MIN ))
-}
-
-pool_condition_true() { # condition-type
- [[ "$(kubectl_get "sandboxpool/$POOL_NAME" "{.status.conditions[?(@.type==\"$1\")].status}" 2>/dev/null)" == "True" ]]
-}
-
-warm_images_ready() {
- kubectl -n "$RESOURCE_NS" get sandboxpool "$POOL_NAME" -o jsonpath='{.status.warmImages[*].cachedFastlets}' 2>/dev/null | grep -qv '^0*$'
-}
-
-render_pool() { # > $GEN_DIR/firecracker-egress-pool.yaml
- local src="$MANIFESTS_DIR/pool/firecracker-egress-pool.yaml" out="$GEN_DIR/firecracker-egress-pool.yaml"
- mkdir -p "$GEN_DIR"
- # Quoted tokens in the template keep rendered scalars' natural types
- # (poolMin stays an integer); awk for sed portability.
- awk -v fastlet="$IMG_FASTLET" -v egress="$IMG_EGRESS" \
- -v pool_min="$POOL_MIN" -v pool_max="$POOL_MAX" \
- -v max_per_pod="$MAX_SANDBOXES_PER_POD" \
- -v warm="$WARM_IMAGES" -v image="$TEMPLATE_ID" '
- { gsub(/"@FASTLET_IMAGE@"/, fastlet)
- gsub(/"@EGRESS_IMAGE@"/, egress)
- gsub(/"@POOL_MIN@"/, pool_min)
- gsub(/"@POOL_MAX@"/, pool_max)
- gsub(/maxSandboxesPerPod: 5/, "maxSandboxesPerPod: " max_per_pod) }
- /^# @WARM_IMAGES@$/ {
- if (warm == "1") printf " warmImages:\n - %s\n", image
- next }
- { print }
- ' "$src" > "$out"
- # Leftover-token guard: match only value positions (key: ...@T@...),
- # never the header comments that document the tokens themselves.
- if grep -Eq '^[[:space:]]*[A-Za-z][A-Za-z0-9]*:.*@[A-Z_]+@' "$out"; then
- die "unrendered token left in $out"
- fi
-}
-
-pool_up() {
- render_pool
- log "applying SandboxPool $POOL_NAME (runtime=firecracker, egress attached, poolMin=$POOL_MIN)"
- kubectl apply -f "$GEN_DIR/firecracker-egress-pool.yaml" >/dev/null
- wait_for "fastlet pods Ready (poolMin=$POOL_MIN)" 300 fastlet_pods_ready
- wait_for "egress container ready in every fastlet pod" 180 egress_containers_ready
- wait_for "pool condition RuntimeReady=True" 120 pool_condition_true RuntimeReady
- wait_for "pool condition InfraReady=True" 120 pool_condition_true InfraReady
- wait_for "egress actions endpoint answering (/_fastlet/v1/actions/status)" 60 egress_status_ready
- if [[ "$WARM_IMAGES" == "1" ]]; then
- wait_for "pool warmImages Ready" 300 warm_images_ready
- p2p_evidence "warm preheat"
- pass "fastlet Running + warmImages Ready (P2P evidence captured)"
- else
- pass "fastlet Running, on-demand pull (default: first sandbox pulls through DART)"
- fi
-}
-
-# Assert the P2P outcome from DART block counters: the artifact set is
-# pulled ~once per 4MiB block from the origin cluster-wide, and extra nodes
-# must be fed by peers, not the origin.
-p2p_evidence() { # description
- local description="$1"
- local pods pod manifest_ref manifest_key build_dir expected_blocks=0 stat_json
- local origin_total=0 peer_total=0 cache_total=0 size source value active_nodes=0 node_total
- manifest_ref="$(osb_api template get "$TEMPLATE_ID" -o json 2>/dev/null | jq -r '.status.manifest_ref // empty')"
- manifest_key="${manifest_ref#s3://$RUSTFS_BUCKET/}"
- build_dir="$(dirname "$manifest_key")"
- local object
- for object in rootfs.ext4 vmstate.snap memory.snap; do
- stat_json="$(rc stat --json "chain/$RUSTFS_BUCKET/$build_dir/$object" 2>&1)" \
- || die "rc stat failed for published $object (publish incomplete?): $stat_json"
- size="$(jq -er '.size_bytes | numbers' <<<"$stat_json" 2>/dev/null)" \
- || die "rc stat returned no numeric size_bytes for published $object"
- expected_blocks=$((expected_blocks + (size + 4194303) / 4194304))
- done
- pods="$(runtime_pods)"
- for pod in $pods; do
- node_total=0
- while read -r source value; do
- case "$source" in
- origin) origin_total=$((origin_total + value)); node_total=$((node_total + value)) ;;
- peer) peer_total=$((peer_total + value)); node_total=$((node_total + value)) ;;
- cache) cache_total=$((cache_total + value)); node_total=$((node_total + value)) ;;
- esac
- done < <(dart_source_counters "$pod")
- [[ "$node_total" -gt 0 ]] && active_nodes=$((active_nodes + 1))
- done
- log "p2p evidence ($description): expected origin=$expected_blocks blocks; cluster origin=$origin_total peer=$peer_total cache=$cache_total active-nodes=$active_nodes"
- [[ "$origin_total" -ge "$expected_blocks" ]] || fail "cluster origin $origin_total < expected $expected_blocks blocks"
- [[ "$origin_total" -le $((expected_blocks + 4)) ]] \
- || fail "origin amplified: $origin_total > $((expected_blocks + 4)): pulls were not deduplicated by DART"
- if [[ "$active_nodes" -ge 2 ]]; then
- [[ "$peer_total" -gt 0 ]] || fail "no peer traffic across $active_nodes nodes: the second node was not served by the peer"
- pass "P2P evidence ($description): origin ~1 fetch per block (cluster=$origin_total/$expected_blocks), peer=$peer_total, nodes=$active_nodes"
- else
- pass "P2P evidence ($description): origin ~1 fetch per block (cluster=$origin_total/$expected_blocks) on $active_nodes node (no peer needed)"
- fi
-}
-
-dart_source_counters() { # pod -> "<source> <value>" lines
- local pod="$1"
- kubectl exec -n "$NS" "$pod" -- sh -c 'curl -fsS --noproxy "*" http://127.0.0.1:8147/metrics' 2>/dev/null \
- | awk '/^dart_block_source_total\{source="(cache|peer|origin)"\}/ {
- match($0, /source="[^"]+"/); s = substr($0, RSTART + 8, RLENGTH - 9)
- match($0, /} [0-9]+$/); print s, substr($0, RSTART + 2)
- }'
-}
-
-# --- stage: idle-pause Redis (OSEP-0024 profile) --------------------------------
-
-redis_up() {
- # The sweeper connects at startup, so Redis must precede the server.
- [[ "$IDLE_PAUSE" == "1" ]] || { log "idle-pause profile off (IDLE_PAUSE=0); skipping Redis"; rm -f "$WORK/idle-pause"; return 0; }
- touch "$WORK/idle-pause"
- mkdir -p "$GEN_DIR"
- sed "s|image: redis:7-alpine|image: $REDIS_IMAGE|" \
- "$MANIFESTS_DIR/idle-pause/redis.yaml" > "$GEN_DIR/idle-pause-redis.yaml"
- kubectl apply -n "$OSB_NS" -f "$GEN_DIR/idle-pause-redis.yaml" >/dev/null
- wait_for "idle-pause Redis deployment ready" 120 \
- kubectl -n "$OSB_NS" rollout status deploy/idle-pause-redis --timeout=10s
- pass "idle-pause Redis up ($REDIS_DSN)"
-}
-
-# --- stage: OpenSandbox server + ingress gateway --------------------------------
-
-# Per-workdir key (not per-run): re-applies must keep old routes verifiable.
-osb_signing_key() {
- if [[ ! -s "$SIGNING_KEY_FILE" ]]; then
- openssl rand -base64 32 | tr -d '\n' > "$SIGNING_KEY_FILE"
- fi
- cat "$SIGNING_KEY_FILE"
-}
-
-# Server config.toml (fsb runtime) with workdir tokens; the gateway [ingress]
-# block is appended by charts/server from server.gateway.* values.
-render_server_config() { # > $GEN_DIR/osb-server-config.toml
- mkdir -p "$GEN_DIR"
- awk -v api_key="$SERVER_API_KEY" \
- -v fastpath="$FASTPATH_ENDPOINT" -v fsb_ns="$RESOURCE_NS" -v pool="$POOL_NAME" \
- -v execd="$EXECD" '
- { gsub(/@SERVER_API_KEY@/, api_key)
- gsub(/@SIGNING_KEY@/, signing_key)
- gsub(/@FASTPATH_ENDPOINT@/, fastpath)
- gsub(/@FSB_NAMESPACE@/, fsb_ns)
- gsub(/@POOL_NAME@/, pool)
- gsub(/@EXECD_IMAGE@/, execd)
- gsub(/@GATEWAY_ADDRESS@/, gateway)
- print }
- ' <<'TOML' > "$GEN_DIR/osb-server-config.toml"
-[server]
-host = "0.0.0.0"
-port = 80
-api_key = "@SERVER_API_KEY@"
-
-[log]
-level = "INFO"
-
-[runtime]
-type = "kubernetes"
-execd_image = "@EXECD_IMAGE@"
-
-[kubernetes]
-# One block for CR reads + fsb settings; sandboxes live in the pool's
-# namespace (poolRef resolves) and get the execd image injected by the server.
-namespace = "@FSB_NAMESPACE@"
-fastpath_endpoint = "@FASTPATH_ENDPOINT@"
-fastpath_resource_pool = "@POOL_NAME@"
-fastpath_wait_ready_seconds = 30.0
-template_s3_publish_secret = "sandbox-oss-credentials"
-TOML
- if [[ "$IDLE_PAUSE" == "1" ]]; then
- # Dotted keys nest the redis sub-table.
- cat >> "$GEN_DIR/osb-server-config.toml" <<TOML
-
-[automation.idle_pause]
-enabled = true
-sweep_interval_seconds = 5
-activity_ttl_seconds = 1800
-redis.dsn = "$REDIS_DSN"
-redis.lock_ttl_seconds = 30
-TOML
- fi
- if grep -Eq '@[A-Z_]+@' "$GEN_DIR/osb-server-config.toml"; then
- die "unrendered token left in $GEN_DIR/osb-server-config.toml"
- fi
-}
-
-opensandbox_up() {
- kind load docker-image "$IMG_SERVER" --name "$KIND_CLUSTER" >/dev/null
- kind load docker-image "$IMG_INGRESS" --name "$KIND_CLUSTER" >/dev/null
- render_server_config
- apply_ns "$OSB_NS"
- # charts/server has the fsb RBAC built in; the config carries the wiring.
- helm_render opensandbox-server "$OSB_ROOT/manifests/charts/server" "$OSB_NS" \
- "$GEN_DIR/osb-server.yaml" \
- --set server.image.repository="$(image_repo "$IMG_SERVER")" \
- --set server.image.tag="$(image_tag "$IMG_SERVER")" \
- --set server.service.type=NodePort \
- --set server.service.nodePort="$SERVER_NODEPORT" \
- --set server.resources.requests.cpu=250m \
- --set server.resources.requests.memory=512Mi \
- --set server.resources.limits.cpu=1 \
- --set server.resources.limits.memory=2Gi \
- --set-file configToml="$GEN_DIR/osb-server-config.toml" \
- --set server.gateway.enabled=true \
- --set server.gateway.host="$GATEWAY_ADDRESS" \
- --set server.gateway.gatewayRouteMode=header \
- --set server.gateway.secureAccess.activeKey=a \
- --set "server.gateway.secureAccess.keys[0].key_id=a" \
- --set "server.gateway.secureAccess.keys[0].key=$(osb_signing_key)"
- kubectl apply -f "$GEN_DIR/osb-server.yaml" >/dev/null
- # fast-sandbox provider via FastPath, same signing key as the server.
- local gateway_idle_args=()
- if [[ "$IDLE_PAUSE" == "1" ]]; then
- gateway_idle_args=(
- --set gateway.activity.enabled=true
- --set gateway.activity.redisDsn="$REDIS_DSN"
- --set gateway.activity.minInterval=5s
- --set gateway.activity.ttlSeconds=1800
- --set gateway.wake.enabled=true
- --set gateway.wake.parkBudget="${IDLE_PAUSE_PARK_BUDGET:-5s}"
- )
- fi
- helm_render opensandbox-ingress-gateway "$OSB_ROOT/manifests/charts/ingress-gateway" "$OSB_NS" \
- "$GEN_DIR/osb-ingress-gateway.yaml" \
- --set gateway.image.repository="$(image_repo "$IMG_INGRESS")" \
- --set gateway.image.tag="$(image_tag "$IMG_INGRESS")" \
- --set gateway.replicaCount=1 \
- --set gateway.providerType=fast-sandbox \
- --set gateway.dataplaneNamespace="$NS" \
- --set gateway.fastpathEndpoint="$FASTPATH_ENDPOINT" \
- --set "gateway.secureAccess.keys[0].key_id=a" \
- --set "gateway.secureAccess.keys[0].key=$(osb_signing_key)" \
- --set gateway.service.type=NodePort \
- --set gateway.service.nodePort="$GATEWAY_NODEPORT" \
- --set gateway.resources.requests.cpu=100m \
- --set gateway.resources.requests.memory=128Mi \
- --set gateway.resources.limits.cpu=1 \
- --set gateway.resources.limits.memory=1Gi \
- "${gateway_idle_args[@]}"
- kubectl apply -f "$GEN_DIR/osb-ingress-gateway.yaml" >/dev/null
- wait_for "server deployment ready" 180 \
- kubectl -n "$OSB_NS" rollout status deploy/opensandbox-server --timeout=10s
- wait_for "ingress gateway deployment ready" 180 \
- kubectl -n "$OSB_NS" rollout status deploy/opensandbox-ingress-gateway --timeout=10s
- # A ready gateway proves FastPath reachability: it fails startup without
- # the gRPC connection.
- wait_for "server /health on 127.0.0.1:$SERVER_HOST_PORT" 60 \
- curl -fsS -m 5 "$SERVER_URL/health"
- wait_for "gateway /status.ok on $GATEWAY_ADDRESS" 60 \
- curl -fsS -m 5 --noproxy "*" "$GATEWAY_URL/status.ok"
- pass "server + ingress gateway up (fsb runtime, gateway routes signed with key 'a')"
-}
-
-# Raw curl, kept for the calls the CLI cannot express (networkpolicy
-# replace PUT, metadata merge-patch, snapshot re-entry fence).
-server_api() { # method path [json-body]
- local method="$1" path="$2" body="${3:-}"
- if [[ -n "$body" ]]; then
- curl -fsS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
- -H "Content-Type: application/json" -d "$body" "$SERVER_URL$path"
- else
- curl -fsS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" "$SERVER_URL$path"
- fi
-}
-
-# opensandbox CLI pointed at the lifecycle server; --no-color keeps
-# -o json parseable.
-osb_api() {
- "$OSB_BIN" --no-color --domain "127.0.0.1:$SERVER_HOST_PORT" --protocol http \
- --api-key "$SERVER_API_KEY" --request-timeout 60 "$@"
-}
-
-# The server assigns the sandbox id (CreateSandboxRequest carries none);
-# the verify probes share it through VERIFY_ID.
-VERIFY_ID=""
-# The snapshot verify stage shares the created snapshot ids through
-# SNAPSHOT_ID / SNAPSHOT_ID2.
-SNAPSHOT_ID=""
-SNAPSHOT_ID2=""
-# A re-entry snapshot accepted during fence cache lag (202) is tracked here
-# so its terminal outcome is asserted and it is cleaned up.
-SNAPSHOT_EXTRA=""
-
-verify_sandbox_gone() {
- ! osb_api sandbox get "$VERIFY_ID" -o json >/dev/null 2>&1
-}
-
-verify_policy_enforced() {
- local out
- out="$(server_api GET "/sandboxes/$VERIFY_ID/networkpolicy" 2>/dev/null)" || return 1
- [[ "$(printf '%s' "$out" | jq -r '.mode // empty')" == "enforcing" ]] || return 1
- [[ "$(printf '%s' "$out" | jq -r '.policy.egress[0].target // empty')" == "example.com" ]]
-}
-
-verify_policy_updated() {
- local out
- out="$(server_api GET "/sandboxes/$VERIFY_ID/networkpolicy" 2>/dev/null)" || return 1
- [[ "$(printf '%s' "$out" | jq -r '.mode // empty')" == "enforcing" ]] || return 1
- [[ "$(printf '%s' "$out" | jq -r '.policy.egress[0].target // empty')" == "github.com" ]]
-}
-
-# Full wire-up: CLI create (fsb) -> FastPath -> fastlet -> firecracker
-# sandbox -> signed gateway route -> fastlet-proxy -> guest execd /ping ->
-# delete. One sandbox per call; availability is measured from the client
-# (poll /ping until 200, wall-clock budget — each route poll is a CLI call,
-# ~0.5s startup), not from CR status.
-verify_one_sandbox() { # <label> <ping-budget-ms>
- local label="$1" budget="$2" created t0 t1 t2 route code deadline
- local policy_file="$GEN_DIR/verify-policy.json"
- t0="$(now_ms)"
- # Egress policy rides on every create (binding -> nft rules), so the
- # chain is exercised each time; /ping is inbound and unaffected.
- jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "example.com"}, {action: "allow", target: "*.opensandbox.ai"}]}' \
- > "$policy_file"
- log "verify ($label): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
- created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
- --metadata origin=fast-sandbox-env-verify \
- --network-policy-file "$policy_file" \
- --skip-health-check -o json 2>/dev/null)" \
- || fail "osb sandbox create failed against $SERVER_URL: $(osb_api sandbox create \
- --template "$TEMPLATE_ID" --timeout 1h --skip-health-check 2>&1 >/dev/null | head -c 400)"
- VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
- [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
- t1="$(now_ms)"
- # Availability = /ping 200 through the signed gateway route (port 44772).
- deadline=$(( t1 + budget * 1000000 ))
- while :; do
- route="$(osb_api sandbox endpoint "$VERIFY_ID" --port 44772 -o json 2>/dev/null \
- | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
- if [[ -n "$route" ]]; then
- code="$(curl -sS -m 5 -o /dev/null -w '%{http_code}' \
- -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" 2>/dev/null || true)"
- [[ "$code" == "200" ]] && break
- fi
- if (( $(now_ms) >= deadline )); then
- fail "execd /ping did not return 200 within ${budget}ms (last code=${code:-none}, route=${route:-none})"
- fi
- sleep 0.2
- done
- t2="$(now_ms)"
- # jq filter in a variable: literal parens trip older bash's $( ) parser.
- local state raw raw_filter
- raw_filter='{rt:.status.runtime.state,dp:.status.dataPlane.state,infra:[.status.infraComponents[]?|{n:.name,s:.state}],bind:[.status.actionBindings[]?|{h:.handler,s:.state}],ready:(.status.conditions[]?|select(.type=="Ready")|.status)}'
- state="$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')"
- raw="$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" -o json 2>/dev/null | jq -c "$raw_filter")"
- log "verify ($label): $VERIFY_ID access via ingress gateway: curl -H \"OpenSandbox-Ingress-To: $route\" $GATEWAY_URL/ping"
- log "verify ($label): $VERIFY_ID create $(( (t1 - t0) / 1000000 ))ms, create->execd /ping 200 $(( (t2 - t1) / 1000000 ))ms, total $(( (t2 - t0) / 1000000 ))ms"
- log "verify ($label): CR at ping: server=$state raw=${raw:-unreachable}"
- pass "execd /ping 200 through the signed gateway route (44772)"
- wait_for "egress policy enforcing (networkPolicy -> egress action binding -> nft)" 120 verify_policy_enforced
- # Policy UPDATE path (raw PUT: the CLI only has merge-style patches).
- local put_body
- put_body="$(jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "github.com"}]}')"
- server_api PUT "/sandboxes/$VERIFY_ID/networkpolicy" "$put_body" >/dev/null \
- || fail "PUT networkpolicy failed for $VERIFY_ID: $(printf '%s' "$put_body" | head -c 200)"
- wait_for "policy update converged (PUT -> ReplaceActionBindings -> egress)" 120 verify_policy_updated
- # Re-sample the CR state: an early Failed observation may converge late.
- local state_final
- state_final="$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')"
- log "verify ($label): $VERIFY_ID CR state final: ${state_final:-unknown}"
-}
-
-# Remaining lifecycle surface on one sandbox: get, list, metadata merge-patch
-# (raw API — no CLI command), renew via the CLI (TTL-based).
-verify_lifecycle_ops() { # <sandbox-id>
- local id="$1" out expected_expires renewed_expires
- out="$(osb_api sandbox get "$id" -o json 2>/dev/null)" || fail "osb sandbox get $id failed"
- [[ "$(printf '%s' "$out" | jq -r '.id')" == "$id" ]] || fail "GET returned wrong id: $out"
- pass "lifecycle: osb sandbox get"
-
- osb_api sandbox list --page 1 --page-size 50 -o json > "$WORK/last-list.json" 2>"$WORK/last-list.err" \
- || fail "osb sandbox list failed: $(head -c 400 "$WORK/last-list.err" 2>/dev/null)"
- out="$(cat "$WORK/last-list.json")"
- [[ "$(printf '%s' "$out" | jq -r --arg id "$id" '.items[]?.id | select(. == $id)' | head -1)" == "$id" ]] \
- || fail "list does not contain $id: $(printf '%s' "$out" | jq -c '.pagination')"
- pass "lifecycle: osb sandbox list (contains the verify sandbox)"
-
- # JSON Merge Patch (RFC 7396): non-null upserts, null deletes.
- server_api PATCH "/sandboxes/$id/metadata" '{"env":"verify","stage":"lifecycle-ops"}' >/dev/null \
- || fail "PATCH metadata upsert failed"
- out="$(osb_api sandbox get "$id" -o json 2>/dev/null)"
- [[ "$(printf '%s' "$out" | jq -r '.metadata.env')" == "verify" \
- && "$(printf '%s' "$out" | jq -r '.metadata.stage')" == "lifecycle-ops" ]] \
- || fail "metadata upsert not visible: $(printf '%s' "$out" | jq -c '.metadata')"
- server_api PATCH "/sandboxes/$id/metadata" '{"stage":null}' >/dev/null \
- || fail "PATCH metadata delete failed"
- out="$(osb_api sandbox get "$id" -o json 2>/dev/null)"
- [[ "$(printf '%s' "$out" | jq -r '.metadata.stage')" == "null" \
- && "$(printf '%s' "$out" | jq -r '.metadata.env')" == "verify" ]] \
- || fail "metadata delete not visible: $(printf '%s' "$out" | jq -c '.metadata')"
- pass "lifecycle: PATCH metadata (upsert + null-delete via JSON Merge Patch)"
-
- # CLI renew takes a TTL: assert the expiration advanced and is visible.
- expected_expires="$(osb_api sandbox get "$id" -o json 2>/dev/null | jq -r '.expires_at')"
- out="$(osb_api sandbox renew "$id" --timeout 2h -o json 2>"$WORK/last-renew.err")" \
- || fail "osb sandbox renew failed: $(head -c 400 "$WORK/last-renew.err" 2>/dev/null)"
- renewed_expires="$(printf '%s' "$out" | jq -r '.expires_at')"
- [[ -n "$renewed_expires" && "$renewed_expires" != "null" && "$renewed_expires" != "$expected_expires" ]] \
- || fail "renewed expiresAt did not advance: $expected_expires -> $renewed_expires"
- local got_expires
- got_expires="$(osb_api sandbox get "$id" -o json 2>/dev/null | jq -r '.expires_at')"
- # Same instant, two renderings (renew panel str() vs RFC3339 dump):
- # compare epoch seconds, falling back to string equality.
- if [[ "$got_expires" != "$renewed_expires" ]] \
- && [[ "$(date -u -d "$renewed_expires" +%s 2>/dev/null || true)" \
- != "$(date -u -d "$got_expires" +%s 2>/dev/null || true)" ]]; then
- fail "renewed expiresAt not visible: expected $renewed_expires got $got_expires"
- fi
- pass "lifecycle: osb sandbox renew 2h ($expected_expires -> $renewed_expires)"
-}
-
-opensandbox_verify() {
- # Cold create pulls the golden image through DART (slowest; generous
- # budget). Warm creates: the second may still pull on the other node via
- # its DART peer; after both nodes cache the set, creates are sub-second.
- local ids=() id label index
- verify_one_sandbox "cold #1" 600000
- ids+=("$VERIFY_ID")
- verify_one_sandbox "warm #2" 600000
- ids+=("$VERIFY_ID")
- for index in 3 4 5 6; do
- verify_one_sandbox "warm #$index" 120000
- ids+=("$VERIFY_ID")
- done
- pass "end-to-end: opensandbox CLI -> server -> FastPath -> fastlet -> sandbox execd -> gateway route OK (6 sandboxes)"
- verify_lifecycle_ops "${ids[0]}"
- for id in "${ids[@]}"; do
- VERIFY_ID="$id"
- osb_api sandbox kill "$id" >/dev/null 2>&1 \
- || log "verify cleanup: kill failed; remove $id manually"
- done
- for id in "${ids[@]}"; do
- VERIFY_ID="$id"
- wait_for "verify sandbox $id deleted" 120 verify_sandbox_gone
- done
- pass "verify sandboxes cleaned up"
-}
-
-# --- stage: pause / resume (opensandbox CLI -> FastPath checkpoint) -------------
-
-# Fresh signed route + one GET: doubles as the "runtime actually serving" probe.
-execd_ping_ok() {
- local route code
- route="$(osb_api sandbox endpoint "$VERIFY_ID" --port 44772 -o json 2>/dev/null \
- | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
- [[ -n "$route" ]] || return 1
- code="$(curl -sS -m 5 -o /dev/null -w '%{http_code}' \
- -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" 2>/dev/null || true)"
- [[ "$code" == "200" ]]
-}
-
-sandbox_state_is() { # <state>
- [[ "$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')" == "$1" ]]
-}
-
-sandbox_running() { sandbox_state_is Running; }
-sandbox_paused() { sandbox_state_is Paused; }
-sandbox_state_get() {
- osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty'
-}
-
-pause_resume_verify() {
- # Accepted + poll: Paused == checkpoint durable + capacity released;
- # resume advances the route generation, so /ping needs a fresh route.
- local created t0 t1 t2 t3
- log "verify (pause): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
- t0="$(now_ms)"
- created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
- --metadata origin=fast-sandbox-env-pause --skip-health-check -o json 2>/dev/null)" \
- || fail "osb sandbox create failed against $SERVER_URL"
- VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
- [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
- wait_for "pause target Running" 150 sandbox_running
- t1="$(now_ms)"
- log "verify (pause): $VERIFY_ID create->Running $(( (t1 - t0) / 1000000 ))ms"
-
- wait_for "pre-pause execd /ping 200 through the gateway" 100 execd_ping_ok
-
- log "verify (pause): osb sandbox pause $VERIFY_ID"
- osb_api sandbox pause "$VERIFY_ID" >/dev/null 2>&1 \
- || fail "pause failed for $VERIFY_ID"
- # Paused is durable-first: the checkpoint must be complete in the artifact
- # store before the state is reported; the dump itself keeps serving.
- wait_for "poll GET until Paused (checkpoint durable, capacity released)" 240 sandbox_paused
- t2="$(now_ms)"
- log "verify (pause): $VERIFY_ID pause->Paused (checkpoint durable) $(( (t2 - t1) / 1000000 ))ms"
- if execd_ping_ok; then
- fail "paused sandbox still serves /ping through the gateway"
- fi
- pass "paused: execd /ping no longer served (runtime released, signed route rejected at the gateway)"
-
- log "verify (pause): osb sandbox resume $VERIFY_ID"
- t2="$(now_ms)"
- osb_api sandbox resume "$VERIFY_ID" --skip-health-check >/dev/null 2>&1 \
- || fail "resume failed for $VERIFY_ID"
- wait_for "poll GET until Running (checkpoint restored)" 240 sandbox_running
- t3="$(now_ms)"
- log "verify (pause): $VERIFY_ID resume->Running (checkpoint restored) $(( (t3 - t2) / 1000000 ))ms"
- wait_for "post-resume execd /ping 200 through a fresh route" 100 execd_ping_ok
- pass "pause/resume round-trip timings: pause->Paused $(( (t2 - t1) / 1000000 ))ms, resume->Running $(( (t3 - t2) / 1000000 ))ms (opensandbox CLI -> FastPath -> artifact store)"
-
- osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
- || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
- wait_for "pause/resume sandbox deleted" 120 verify_sandbox_gone
-}
-
-# --- stage: vault lifecycle (OSEP-0022: egress-held vault over the relay) --------
-
-# Status-code-returning variant of server_api for negative-path assertions
-# (curl -f hides the status).
-server_api_status() { # method path [json-body]
- local method="$1" path="$2" body="${3:-}" out
- if [[ -n "$body" ]]; then
- out="$(curl -sS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
- -H "Content-Type: application/json" -d "$body" -w '\n%{http_code}' "$SERVER_URL$path" 2>/dev/null)"
- else
- out="$(curl -sS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
- -w '\n%{http_code}' "$SERVER_URL$path" 2>/dev/null)"
- fi
- printf '%s' "${out##*$'\n'}"
-}
-
-# The vault stage's sandbox must ride the subject-scoped egress log markers
-# (export/import/registration), so every log assertion goes through this.
-vault_log_contains() { # <fixed-substring>
- kubectl -n "$RESOURCE_NS" logs -l app=sandbox-fastlet -c egress --tail=800 2>/dev/null \
- | grep -qF "$1"
-}
-
-vault_revision_is() { # <expected-revision>
- local out
- out="$(server_api GET "/sandboxes/$VERIFY_ID/credential-vault" 2>/dev/null)" || return 1
- [[ "$(printf '%s' "$out" | jq -r '.revision // empty')" == "$1" ]]
-}
-
-vault_lifecycle_verify() {
- local created vault_body ack rev sandbox_uid
- log "verify (vault): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
- # The policy rides the create: a policy-less create produces no egress
- # binding at all, so the subject would never register.
- jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "ext.test"}]}' \
- > "$GEN_DIR/vault-policy.json"
- created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
- --metadata origin=fast-sandbox-env-vault \
- --network-policy-file "$GEN_DIR/vault-policy.json" \
- --skip-health-check -o json 2>/dev/null)" \
- || fail "vault verify: sandbox create failed"
- VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
- wait_for "vault sandbox Running" 150 sandbox_running
- sandbox_uid="$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" -o jsonpath='{.metadata.uid}' 2>/dev/null)"
- [[ -n "$sandbox_uid" ]] || fail "vault verify: sandbox CR carried no uid"
- # Registered and activated before the vault POST, else the relay 404s.
- # (The "(state=active)" re-delivery only follows a policy UPDATE; a
- # create-carried policy activates without it.)
- wait_for "vault subject registered on egress" 120 \
- vault_log_contains "subject s-$sandbox_uid: data-plane-ready, policy active"
-
- # 1) create-only: ack carries the revision; duplicate POST is 409.
- vault_body="$(jq -n '{credentials: [{name: "tok", source: {type: "inline", value: "sekrit-vault-e2e"}}], bindings: [{name: "ext", match: {hosts: ["ext.test"]}, auth: {type: "apiKey", name: "X-Api-Key", credential: "tok"}}]}')"
- ack="$(server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" \
- || fail "vault verify: POST credential-vault failed"
- rev="$(printf '%s' "$ack" | jq -r '.revision // empty')"
- [[ "$rev" =~ ^[0-9]+$ ]] || fail "vault verify: create ack carried no revision: $(printf '%s' "$ack" | head -c 200)"
- pass "vault created over the relay (ack revision $rev)"
- if server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body" >/dev/null 2>&1; then
- fail "vault verify: duplicate POST succeeded (create-only must 409)"
- fi
- [[ "$(server_api_status POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" == "409" ]] \
- || fail "vault verify: duplicate POST was not 409"
- pass "duplicate POST rejected (create-only 409)"
-
- # 2) pause/resume carries the real vault: export -> bundle -> import,
- # with the relay GET reporting the same revision.
- osb_api sandbox pause "$VERIFY_ID" >/dev/null 2>&1 || fail "vault verify: pause failed"
- wait_for "vault sandbox Paused (checkpoint durable)" 240 sandbox_paused
- wait_for "vault exported into the checkpoint bundle" 120 \
- vault_log_contains "subject s-$sandbox_uid: vault exported to snapshot bundle"
- pass "pause exported the vault bundle (subject sealed)"
- osb_api sandbox resume "$VERIFY_ID" --skip-health-check >/dev/null 2>&1 || fail "vault verify: resume failed"
- wait_for "vault sandbox Running (restored)" 240 sandbox_running
- # The pause's REMOVE_BINDING purges the subject, so the restore import
- # lands through the unregistered branch and the file reloads at the
- # SET_BINDING replay (the reload line carries the restored revision).
- wait_for "vault bundle reloaded at re-registration" 120 \
- vault_log_contains "subject s-$sandbox_uid: vault reloaded from file"
- wait_for "vault revision preserved across pause/resume" 120 vault_revision_is "$rev"
- pass "vault survived pause/resume (revision $rev preserved)"
-
- # 3) egress-restart recovery: crash the worker (fresh instanceId ->
- # SET_BINDING replay -> vault file reload by incarnation).
- local pod restarted=0
- for pod in $(pool_pods); do
- if kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- sh -c 'command -v pkill >/dev/null' 2>/dev/null; then
- kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- pkill -f '/opt/opensandbox-egress/egress' >/dev/null 2>&1 || true
- restarted=1
- else
- log "verify (vault): egress restart skipped on ${pod##*/} (pkill unavailable)"
- fi
- done
- if [[ "$restarted" == 1 ]]; then
- wait_for "pool egress containers back up" 180 egress_containers_ready
- wait_for "vault reloaded after egress restart" 240 vault_revision_is "$rev"
- pass "vault reloaded after egress restart (revision $rev preserved)"
- fi
-
- # 4) delete + re-create.
- server_api DELETE "/sandboxes/$VERIFY_ID/credential-vault" >/dev/null \
- || fail "vault verify: DELETE failed"
- [[ "$(server_api_status GET "/sandboxes/$VERIFY_ID/credential-vault")" == "404" ]] \
- || fail "vault verify: GET after DELETE was not 404"
- pass "vault deleted (GET 404)"
- ack="$(server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" \
- || fail "vault verify: re-create after DELETE failed"
- rev="$(printf '%s' "$ack" | jq -r '.revision // empty')"
- [[ "$rev" == "1" ]] || fail "vault verify: fresh vault revision is $rev, expected 1"
- pass "vault re-created (fresh revision $rev)"
-
- osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
- || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
- wait_for "vault sandbox deleted" 120 verify_sandbox_gone
-}
-
-
-
-# --- stage: idle-pause / wake-on-access verify (OSEP-0024 profile) --------------
-
-# Poll the gateway until /ping returns 200 (marked 503s tolerated; the
-# resume is never canceled). Fails on budget exhaustion.
-idle_wake_wait() { # <label> [sandbox_id]
- local label="$1" id="${2:-$VERIFY_ID}" code
- local deadline=$(( $(now_ms) + 600000000000 )) wake_seen=0 next_state_log=0
- while :; do
- code="$(idle_ping_code "$id")"
- case "$code" in
- 200) break ;;
- wake-503) wake_seen=1 ;;
- esac
- if (( $(now_ms) >= next_state_log )); then
- log "verify (idle-pause): $label still waiting (last=$code, state=$(sandbox_state_get))"
- next_state_log=$(( $(now_ms) + 30000000000 ))
- fi
- [[ "$(now_ms)" -ge "$deadline" ]] && fail "$label: wake /ping never returned 200 (last=$code)"
- sleep 1
- done
-}
-
-# Gateway answer for one /ping attempt: "200", "wake-503", or "other-<code>".
-idle_ping_code() { # [sandbox_id] (default $VERIFY_ID)
- local id="${1:-$VERIFY_ID}" route headers code marker
- route="$(osb_api sandbox endpoint "$id" --port 44772 -o json 2>/dev/null \
- | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
- [[ -n "$route" ]] || { echo "noroute"; return 0; }
- headers="$(mktemp)"
- curl -sS -m 10 -D "$headers" -o /dev/null \
- -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" >/dev/null 2>&1 || true
- code="$(tr -d '\r' < "$headers" | head -1 | awk '{print $2}')" || true
- marker="$(tr -d '\r' < "$headers" | grep -i '^OpenSandbox-Wake:' | awk '{print $2}')" || true
- rm -f "$headers"
- if [[ "$code" == "200" ]]; then
- echo "200"
- elif [[ "$code" == "503" && "$marker" == "expired" ]]; then
- echo "wake-503"
- else
- echo "other-${code:-none}"
- fi
-}
-
-idle_pause_verify() {
- # Raw curl create: the CLI has no lifecycle.idlePolicy parameter yet.
- [[ "$IDLE_PAUSE" == "1" ]] || { log "idle-pause profile off (IDLE_PAUSE=0); skipping verify"; return 0; }
- local created t0 t1 code i
- local x=30
- local rounds="${IDLE_PAUSE_ROUNDS:-3}"
- [[ "$rounds" =~ ^[1-9][0-9]*$ ]] || die "IDLE_PAUSE_ROUNDS must be a positive integer (got $rounds)"
- log "verify (idle-pause): creating an opted-in sandbox (idleTimeoutSeconds=$x, rounds=$rounds) via the lifecycle API"
- t0="$(now_ms)"
- created="$(server_api POST /sandboxes "$(jq -n \
- --arg template "$TEMPLATE_ID" \
- --argjson timeout 3600 --argjson x "$x" \
- '{templateId: $template, timeout: $timeout,
- lifecycle: {idlePolicy: {idleTimeoutSeconds: $x, onIdle: "pause", wakeOnAccess: true}},
- metadata: {origin: "fast-sandbox-env-idle-pause"}}')" 2>/dev/null)" \
- || fail "idle-pause create failed against $SERVER_URL"
- VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
- [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
- wait_for "idle-pause target Running" 300 sandbox_running
- log "verify (idle-pause): $VERIFY_ID create->Running $(( ($(now_ms) - t0) / 1000000 ))ms"
-
- wait_for "pre-idle execd /ping 200 through the gateway (records activity)" 100 execd_ping_ok
-
- local -a pause_ms=() wake_ms=() attempts=()
- for ((i = 1; i <= rounds; i++)); do
- t0="$(now_ms)"
- wait_for "round $i/$rounds auto-paused within X + sweep + checkpoint" 240 sandbox_paused
- pause_ms+=("$(( ($(now_ms) - t0) / 1000000 ))")
- pass "round $i/$rounds: auto-paused in ${pause_ms[-1]}ms"
-
- t1="$(now_ms)"
- idle_wake_wait "round $i wake"
- wake_ms+=("$(( ($(now_ms) - t1) / 1000000 ))")
- attempts+=("$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" \
- -o jsonpath='{.status.placement.attempt}' 2>/dev/null || echo '?')")
- pass "round $i/$rounds: woke in ${wake_ms[-1]}ms (placement.attempt=${attempts[-1]})"
- done
-
- local joined_pause joined_wake joined_att
- printf -v joined_pause '%sms/' "${pause_ms[@]}"
- printf -v joined_wake '%sms/' "${wake_ms[@]}"
- printf -v joined_att '%s/' "${attempts[@]}"
- pass "idle-pause ${rounds} rounds: pause=${joined_pause%?} wake=${joined_wake%?} placement.attempt=${joined_att%?}"
-
- local main_id="$VERIFY_ID" created2 id2 payload
- log "verify (idle-pause): scenario wakeOnAccess=false"
- created2="$(server_api POST /sandboxes "$(jq -n \
- --arg template "$TEMPLATE_ID" --argjson timeout 3600 --argjson x "$x" \
- '{templateId: $template, timeout: $timeout,
- lifecycle: {idlePolicy: {idleTimeoutSeconds: $x, onIdle: "pause", wakeOnAccess: false}},
- metadata: {origin: "fast-sandbox-env-idle-pause"}}')" 2>/dev/null)" \
- || fail "wakeOnAccess=false create failed"
- id2="$(printf '%s' "$created2" | jq -r '.id')"
- [[ -n "$id2" && "$id2" != "null" ]] || fail "wakeOnAccess=false create carried no id"
- VERIFY_ID="$id2"
- wait_for "wakeOnAccess=false target Running" 300 sandbox_running
- wait_for "wakeOnAccess=false auto-paused" 240 sandbox_paused
- code="$(idle_ping_code "$id2")"
- [[ "$code" == "200" || "$code" == "wake-503" ]] \
- && fail "wakeOnAccess=false sandbox was woken by traffic (code=$code)"
- sleep 10
- sandbox_paused || fail "wakeOnAccess=false sandbox changed state without resume API"
- osb_api sandbox resume "$id2" --skip-health-check >/dev/null 2>&1 || fail "manual resume failed"
- wait_for "wakeOnAccess=false resumed manually" 240 sandbox_running
- pass "wakeOnAccess=false: traffic never woke it, manual resume did"
- osb_api sandbox kill "$id2" >/dev/null 2>&1 || true
- wait_for "wakeOnAccess=false sandbox deleted" 120 verify_sandbox_gone
- VERIFY_ID="$main_id"
-
- log "verify (idle-pause): scenario redis down"
- # The previous scenario may have left it Paused.
- idle_wake_wait "pre-redis-down wake"
- t0="$(now_ms)"
- kubectl -n "$OSB_NS" scale deploy/idle-pause-redis --replicas=0 >/dev/null
- while sandbox_running; do
- (( $(now_ms) - t0 >= 90000000000 )) && break
- sleep 5
- done
- sandbox_running || fail "sandbox paused while Redis was scaled down"
- pass "redis down: no pauses while Redis unreachable"
- kubectl -n "$OSB_NS" scale deploy/idle-pause-redis --replicas=1 >/dev/null
- wait_for "redis back ready" 120 \
- kubectl -n "$OSB_NS" rollout status deploy/idle-pause-redis --timeout=10s
- wait_for "tombstone pause after redis recovery" 300 sandbox_paused
- pass "redis recovery: missing-key tombstone paused after the full idle window"
-
- idle_wake_wait "post-flushall wake"
- t0="$(now_ms)"
- kubectl -n "$OSB_NS" exec deploy/idle-pause-redis -- redis-cli FLUSHALL >/dev/null 2>&1 \
- || fail "FLUSHALL failed"
- while :; do
- if sandbox_paused; then
- (( $(now_ms) - t0 < (x - 5) * 1000000000 )) \
- && fail "FLUSHALL accelerated the pause ($(( ($(now_ms) - t0) / 1000000 ))ms after flush)"
- break
- fi
- (( $(now_ms) - t0 >= 300000000000 )) && fail "no pause within budget after FLUSHALL"
- sleep 2
- done
- pass "flushall: pause delayed to a full idle window after data loss ($(( ($(now_ms) - t0) / 1000000 ))ms)"
-
-
- osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
- || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
- wait_for "idle-pause sandbox deleted" 120 verify_sandbox_gone
-}
-
-# --- stage: snapshot (opensandbox CLI -> SandboxSnapshot CR -> restore) ----------
-
-# Ready once the server watcher converges the row from the SandboxSnapshot
-# CR (Succeeded + template index published).
-snapshot_ready() { # <snapshot-id>
- [[ "$(osb_api snapshot get "$1" -o json 2>/dev/null | jq -r '.status.state // empty')" == "Ready" ]]
-}
-
-# Terminal (Ready or Failed): the pause-window fence resolves any
-# accepted-but-conflicting snapshot one way or the other.
-snapshot_terminal() { # <snapshot-id>
- local state
- state="$(osb_api snapshot get "$1" -o json 2>/dev/null | jq -r '.status.state // empty')"
- [[ "$state" == "Ready" || "$state" == "Failed" ]]
-}
-
-snapshot_verify() {
- # Full round trip on the live stack: snapshot a Running sandbox, watch it
- # to Ready, restore a NEW sandbox from the snapshotId and boot it. Also
- # covers re-entry fencing, source-sandbox survival, and a repeat snapshot.
- local created out source_id snapshot_id snapshot_id2 restore_id reentry_out reentry_code
- local t0 t1 t2 t3 t4
- t0="$(now_ms)"
- log "verify (snapshot): creating the source sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
- created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
- --metadata origin=fast-sandbox-env-snapshot --skip-health-check -o json 2>/dev/null)" \
- || fail "osb sandbox create failed against $SERVER_URL"
- source_id="$(printf '%s' "$created" | jq -r '.id')"
- [[ -n "$source_id" && "$source_id" != "null" ]] || fail "create response carried no id"
- VERIFY_ID="$source_id"
- wait_for "snapshot source sandbox Running" 300 sandbox_running
- wait_for "pre-snapshot execd /ping 200 through the gateway" 100 execd_ping_ok
- t1="$(now_ms)"
- log "verify (snapshot): source sandbox $source_id create->Running $(( (t1 - t0) / 1000000 ))ms"
-
- # 1. Snapshot create returns Creating; the row converges from the
- # SandboxSnapshot CR via the server watcher.
- log "verify (snapshot): osb snapshot create $source_id --name env-verify"
- t1="$(now_ms)"
- out="$(osb_api snapshot create "$source_id" --name env-verify -o json 2>/dev/null)" \
- || fail "snapshot create failed for $source_id"
- SNAPSHOT_ID="$(printf '%s' "$out" | jq -r '.id')"
- [[ -n "$SNAPSHOT_ID" && "$SNAPSHOT_ID" != "null" ]] || fail "snapshot create carried no id"
- [[ "$(printf '%s' "$out" | jq -r '.status')" == "Creating" ]] \
- || fail "snapshot create did not return Creating: $(printf '%s' "$out" | head -c 300)"
-
- # 2. Re-entry during the dump window: FastPath fences with 409 once the
- # CR is cache-visible; with cache lag it is accepted (202) and the
- # pause window resolves it to a terminal phase later.
- reentry_out="$(curl -sS -m 60 -w '\n%{http_code}' -X POST \
- -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" -H "Content-Type: application/json" \
- -d '{"name":"env-verify-reentry"}' "$SERVER_URL/sandboxes/$source_id/snapshots" 2>/dev/null || true)"
- reentry_code="$(printf '%s' "$reentry_out" | tail -n1)"
- reentry_out="$(printf '%s' "$reentry_out" | sed '$d')"
- case "$reentry_code" in
- 409)
- pass "snapshot: re-entry rejected by the fence (409)"
- ;;
- 202)
- SNAPSHOT_EXTRA="$(printf '%s' "$reentry_out" | jq -r '.id' 2>/dev/null || true)"
- [[ -n "$SNAPSHOT_EXTRA" && "$SNAPSHOT_EXTRA" != "null" ]] \
- || fail "re-entry snapshot POST returned 202 without an id: $(printf '%s' "$reentry_out" | head -c 300)"
- log "verify (snapshot): re-entry accepted during fence cache lag ($SNAPSHOT_EXTRA); terminal outcome asserted below"
- ;;
- *)
- fail "re-entry snapshot POST returned unexpected HTTP ${reentry_code:-none}: $(printf '%s' "$reentry_out" | head -c 300)"
- ;;
- esac
-
- wait_for "poll osb snapshot get $SNAPSHOT_ID until Ready (watcher -> SandboxSnapshot CR -> store index)" 300 snapshot_ready "$SNAPSHOT_ID"
- t2="$(now_ms)"
- log "verify (snapshot): $SNAPSHOT_ID snapshot create->Ready $(( (t2 - t1) / 1000000 ))ms"
- pass "snapshot: create -> Creating -> watcher -> Ready"
-
- # 3. Source survival: Running + /ping again after the terminal phase.
- VERIFY_ID="$source_id"
- wait_for "source sandbox Running again after the snapshot" 120 sandbox_running
- wait_for "source sandbox execd /ping 200 after the snapshot" 100 execd_ping_ok
- pass "snapshot: source sandbox survived (Running + /ping 200)"
-
- # 3b. An accepted re-entry snapshot must reach a terminal phase; a stuck
- # one would block all future snapshots via the re-entry fence.
- if [[ -n "$SNAPSHOT_EXTRA" ]]; then
- wait_for "re-entry snapshot $SNAPSHOT_EXTRA reaches a terminal phase" 150 snapshot_terminal "$SNAPSHOT_EXTRA"
- log "verify (snapshot): re-entry snapshot $SNAPSHOT_EXTRA terminal: $(osb_api snapshot get "$SNAPSHOT_EXTRA" -o json 2>/dev/null | jq -r '.status.state')"
- pass "snapshot: accepted re-entry snapshot resolved to a terminal phase"
- fi
-
- # 4. Second snapshot once the first is terminal: fresh id.
- t3="$(now_ms)"
- out="$(osb_api snapshot create "$source_id" --name env-verify-2 -o json 2>/dev/null)" \
- || fail "second snapshot create failed for $source_id"
- snapshot_id2="$(printf '%s' "$out" | jq -r '.id')"
- [[ -n "$snapshot_id2" && "$snapshot_id2" != "null" && "$snapshot_id2" != "$SNAPSHOT_ID" ]] \
- || fail "second snapshot did not produce a distinct id: $snapshot_id2"
- SNAPSHOT_ID2="$snapshot_id2"
- wait_for "poll osb snapshot get $snapshot_id2 until Ready" 300 snapshot_ready "$snapshot_id2"
- t4="$(now_ms)"
- log "verify (snapshot): $snapshot_id2 second snapshot create->Ready $(( (t4 - t3) / 1000000 ))ms"
- pass "snapshot: repeated snapshot of the same sandbox -> Ready (distinct id)"
-
- # 5. Listing: sandboxId scoping returns both required snapshots, both
- # Ready (an accepted re-entry snapshot may add a third, terminal row).
- out="$(osb_api snapshot list --sandbox-id "$source_id" --page-size 50 -o json 2>"$WORK/last-snapshot-list.err")" \
- || fail "snapshot list failed for $source_id: $(head -c 400 "$WORK/last-snapshot-list.err" 2>/dev/null)"
- [[ "$(printf '%s' "$out" | jq -r --arg id "$SNAPSHOT_ID" --arg id2 "$snapshot_id2" \
- '[.items[] | select((.id == $id or .id == $id2) and .status.state == "Ready")] | length')" == "2" ]] \
- || fail "snapshot list does not contain both required snapshots as Ready: $(printf '%s' "$out" | head -c 400)"
- if [[ -n "$SNAPSHOT_EXTRA" ]]; then
- [[ "$(printf '%s' "$out" | jq -r --arg id "$SNAPSHOT_EXTRA" \
- '[.items[] | select(.id == $id)] | length')" == "1" ]] \
- || fail "accepted re-entry snapshot $SNAPSHOT_EXTRA missing from the list"
- fi
- pass "snapshot: list scoped by sandboxId contains the snapshots (Ready)"
-
- # 6. Restore: the row resolves to the published template index key; the
- # resource flags restate the pool profile (firecracker pool).
- local restored
- log "verify (snapshot): osb sandbox create --snapshot-id $SNAPSHOT_ID (pool profile restated)"
- t3="$(now_ms)"
- restored="$(osb_api sandbox create --snapshot-id "$SNAPSHOT_ID" --timeout 1h \
- --resource cpu=1 --resource memory=512Mi --resource pids=128 \
- --skip-health-check -o json 2>/dev/null)" \
- || fail "osb sandbox create (snapshotId=$SNAPSHOT_ID) failed"
- restore_id="$(printf '%s' "$restored" | jq -r '.id')"
- [[ -n "$restore_id" && "$restore_id" != "null" ]] || fail "restore create carried no id"
- VERIFY_ID="$restore_id"
- wait_for "restored sandbox Running" 600 sandbox_running
- t4="$(now_ms)"
- wait_for "restored execd /ping 200 through the gateway" 300 execd_ping_ok
- log "verify (snapshot): restored $restore_id create->Running $(( (t4 - t3) / 1000000 ))ms"
- pass "snapshot: restore -> sandbox boots the published artifact set -> execd /ping OK"
-
- # 7. Cleanup; the server forwards artifact deletion per snapshot row.
- osb_api sandbox kill "$restore_id" >/dev/null 2>&1 \
- || log "verify cleanup: kill $restore_id failed"
- wait_for "restored sandbox deleted" 120 verify_sandbox_gone
- VERIFY_ID="$source_id"
- osb_api sandbox kill "$source_id" >/dev/null 2>&1 \
- || log "verify cleanup: kill $source_id failed"
- wait_for "snapshot source sandbox deleted" 120 verify_sandbox_gone
- for snapshot_id in "$SNAPSHOT_ID" "$SNAPSHOT_ID2" "$SNAPSHOT_EXTRA"; do
- [[ -n "$snapshot_id" ]] || continue
- osb_api snapshot delete "$snapshot_id" >/dev/null 2>&1 \
- || log "verify cleanup: snapshot delete $snapshot_id failed"
- done
- pass "snapshot: cleanup (restored + source sandboxes, both snapshot rows)"
-}
-
-# --- stage: python sdk e2e (tests/python/tests/test_fsb_e2e.py) -----------------
-
-sdk_e2e() {
- kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
- || die "cluster $KIND_CLUSTER is not up (run up first)"
- curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 \
- || die "server $SERVER_URL is not reachable (run up first)"
- command -v uv >/dev/null 2>&1 \
- || die "uv is required on PATH (https://docs.astral.sh/uv/ — curl -LsSf https://astral.sh/uv/install.sh | sh)"
- local template_id
- template_id="$(cat "$WORK/template-id" 2>/dev/null || true)"
- [[ -n "$template_id" ]] || die "no template id at $WORK/template-id (run up first; template_up stores it there)"
- export OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$template_id"
- export OPENSANDBOX_TEST_DOMAIN="127.0.0.1:$SERVER_HOST_PORT"
- export OPENSANDBOX_TEST_PROTOCOL="http"
- export OPENSANDBOX_TEST_API_KEY="$SERVER_API_KEY"
- log "sdk e2e: template=$template_id server=$SERVER_URL -> tests/python/tests/test_fsb_e2e.py"
- cd "$OSB_ROOT/tests/python" || die "tests/python not found under $OSB_ROOT"
- # No exec: the ERR trap (and the failure dump) must survive a pytest
- # failure, and exec replaces the shell.
- uv run pytest tests/test_fsb_e2e.py
-}
-
-idle_e2e() {
- if [[ "$IDLE_PAUSE" != "1" && ! -f "$WORK/idle-pause" ]]; then
- die "idle-e2e requires the idle-pause profile (IDLE_PAUSE=1 up)"
- fi
- kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
- || die "cluster $KIND_CLUSTER is not up (run up first)"
- curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 \
- || die "server $SERVER_URL is not reachable (run up first)"
- command -v uv >/dev/null 2>&1 \
- || die "uv is required on PATH (https://docs.astral.sh/uv/ — curl -LsSf https://astral.sh/uv/install.sh | sh)"
- local template_id
- template_id="$(cat "$WORK/template-id" 2>/dev/null || true)"
- [[ -n "$template_id" ]] || die "no template id at $WORK/template-id (run up first; template_up stores it there)"
- export OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$template_id"
- export OPENSANDBOX_TEST_DOMAIN="127.0.0.1:$SERVER_HOST_PORT"
- export OPENSANDBOX_TEST_PROTOCOL="http"
- export OPENSANDBOX_TEST_API_KEY="$SERVER_API_KEY"
- export OPENSANDBOX_TEST_IDLE_PAUSE="1"
- export OPENSANDBOX_TEST_IDLE_ROUNDS="${IDLE_PAUSE_ROUNDS:-3}"
- log "sdk e2e (idle-pause): template=$template_id server=$SERVER_URL rounds=$OPENSANDBOX_TEST_IDLE_ROUNDS -> tests/python/tests/test_fsb_idle_pause_e2e.py"
- cd "$OSB_ROOT/tests/python" || die "tests/python not found under $OSB_ROOT"
- uv run pytest tests/test_fsb_idle_pause_e2e.py
-}
-
-# --- status / summary ---------------------------------------------------------------------
-
-dart_metrics_summary() {
- local pods pod node metrics
- pods="$(runtime_pods 2>/dev/null || true)"
- [[ -n "$pods" ]] || { echo " (no agent pods)"; return 0; }
- for pod in $pods; do
- node="$(kubectl -n "$NS" get pod "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)"
- metrics="$(kubectl exec -n "$NS" "$pod" -- sh -c 'curl -fsS --noproxy "*" http://127.0.0.1:8147/metrics' 2>/dev/null || true)"
- echo " $node:"
- if [[ -z "$metrics" ]]; then
- echo " (DART metrics unreachable)"
- continue
- fi
- printf '%s\n' "$metrics" | grep -E '^dart_block_source_total\{source="(cache|peer|origin)"\}' \
- | sed 's/^/ /' || true
- done
-}
-
-status() {
- log "status: RustFS"
- docker ps --filter "name=$RUSTFS_CONTAINER" --format '{{.Names}} {{.Status}}' 2>/dev/null || true
- echo
- log "status: kind cluster / nodes"
- if kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null; then
- kubectl get nodes -o wide
- else
- log "kind cluster $KIND_CLUSTER: down"
- return 0
- fi
- echo
- log "status: pods ($NS)"
- if [[ "$IDLE_PAUSE" == "1" ]]; then
- log "status: idle-pause Redis ($OSB_NS)"
- kubectl -n "$OSB_NS" get deploy,svc -l app=idle-pause-redis -o wide 2>/dev/null || true
- echo
- fi
- kubectl -n "$NS" get pods -o wide
- kubectl -n "$RESOURCE_NS" get pods -o wide
- echo
- log "status: SandboxPool"
- kubectl -n "$RESOURCE_NS" get sandboxpool \
- -o custom-columns='NAME:.metadata.name,RUNTIME:.spec.runtime,READY:.status.readyPods,CAPACITY:.spec.capacity.poolMin,WARM_IMAGES:.status.warmImages' 2>/dev/null || true
- echo
- log "status: fastlet pods (egress sidecar)"
- kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet \
- -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,CONTAINERS:.status.containerStatuses[*].name,READY:.status.containerStatuses[*].ready' 2>/dev/null || true
- echo
- log "status: DART P2P (block_source cache/peer/origin per node)"
- dart_metrics_summary || true
- echo
- log "status: OpenSandbox ($OSB_NS)"
- if kubectl get namespace "$OSB_NS" >/dev/null 2>&1; then
- kubectl -n "$OSB_NS" get pods -o wide
- printf ' server health: %s\n' "$(curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 && echo OK || echo unreachable)"
- printf ' gateway health: %s\n' "$(curl -fsS -m 5 "$GATEWAY_URL/status.ok" >/dev/null 2>&1 && echo OK || echo unreachable)"
- printf ' server URL: %s (header OPEN-SANDBOX-API-KEY: %s)\n' "$SERVER_URL" "$SERVER_API_KEY"
- printf ' gateway URL: %s (header routing, signed f1.* scopes)\n' "$GATEWAY_URL"
- else
- echo " (not deployed)"
- fi
-}
-
-env_summary() {
- highlight "== environment summary =="
- printf ' %-22s %s\n' "kind cluster" "$KIND_CLUSTER ($(kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ') nodes)"
- printf ' %-22s %s\n' "fast-sandbox" "pinned $(git -C "$FSB_DIR" rev-parse --short HEAD 2>/dev/null || echo "$FSB_COMMIT") ($FSB_DIR)"
- printf ' %-22s %s\n' "RustFS endpoint" "$RUSTFS_ENDPOINT"
- printf ' %-22s %s\n' "pool" "$POOL_NAME (runtime=firecracker, poolMin=$POOL_MIN, egress=$IMG_EGRESS)"
- printf ' %-22s %s\n' "P2P" "DART daemons=$(printf '%s' "$(runtime_pods)" | wc -w | tr -d ' ') (on-demand pulls: cache -> peer -> origin)"
- printf ' %-22s %s\n' "template" "${TEMPLATE_ID:-n/a} ($(if [[ -n "$TEMPLATE_ID" ]]; then _template_phase || echo unknown; else echo "not built"; fi))"
- printf ' %-22s %s\n' "StateRoot fs" "$(findmnt -no FSTYPE "$XFS_MOUNT_POINT" 2>/dev/null || echo 'plain directory (full copy per sandbox)')"
- printf ' %-22s %s\n' "server" "$IMG_SERVER -> $SERVER_URL (fsb runtime)"
- printf ' %-22s %s\n' "ingress gateway" "$IMG_INGRESS -> $GATEWAY_URL (fsb provider, header mode)"
- printf ' %-22s %s\n' "fastpath" "$FASTPATH_ENDPOINT"
- printf ' %-22s %s\n' "logs" "$LOGS_DIR"
-}
-
-# --- down ------------------------------------------------------------------------------------
-
-down() {
- log "down: teardown"
- if kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null; then
- kind delete cluster --name "$KIND_CLUSTER" > "$LOGS_DIR/kind-delete.log" 2>&1 || true
- fi
- [[ -z "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]] \
- || fail "kind cluster $KIND_CLUSTER still exists after delete"
- docker rm -f "$RUSTFS_CONTAINER" >/dev/null 2>&1 || true
- [[ -z "$(docker ps -a --filter "name=$RUSTFS_CONTAINER" --format '{{.Names}}' || true)" ]] \
- || fail "RustFS container still present"
- # Server/gateway die with the cluster; only the signing key outlives it.
- rm -f "$WORK/agent-registry.json" "$SIGNING_KEY_FILE"
- rm -rf "$GEN_DIR" "$FSB_GEN_DIR"
- # A stale profile marker would let idle-e2e run against a stack whose
- # Redis/sweeper are gone.
- rm -f "$WORK/idle-pause"
- # RustFS data is owned by UID/GID 10001; leftovers pollute the host and
- # later docker build contexts.
- sudo_ rm -rf "$RUSTFS_DATA"
- sudo_ rm -rf "$RC_CONFIG_DIR"
- sysctl_restore
- stateroot_xfs_down
- # A committed DART cache is FINAL (never refreshed): purge it, or a
- # rebuilt SandboxTemplate stays ignored when the StateRoot survives.
- stateroot_scrub
- pass "host cleanup complete"
-}
-
-# --- main --------------------------------------------------------------------------------------
-
-usage() {
- cat <<'EOF'
-usage: integration-env.sh [--auto-clean] {up|down|status|pool|sdk-e2e|idle-e2e}
-
- up initialize the full environment: fast-sandbox@pinned-commit images,
- two-node kind cluster (KVM), RustFS, control plane, firecracker
- firecracker runtime readiness + DART (P2P), SandboxTemplate golden
- image, firecracker-egress-pool (egress attached), the
- source-built OpenSandbox server + ingress gateway, and
- end-to-end verifies driven by the opensandbox CLI
- (create -> gateway route -> execd /ping, plus a pause/resume
- round-trip through the checkpoint). The CLI (osb) is installed
- fresh from PyPI unless OSB_BIN points at an existing binary.
- With IDLE_PAUSE=1: Redis + server [automation.idle_pause] + gateway
- activity/wake flags, plus the idle-pause/wake verify stage
- (IDLE_PAUSE_ROUNDS cycles, per-round timing summary).
- pool re-apply only the SandboxPool (after editing manifests/pool/)
- status nodes / pods / pool / DART P2P counters / RustFS / OpenSandbox health
- sdk-e2e run the Python SDK e2e suite (tests/python/tests/test_fsb_e2e.py)
- against the live stack; requires `up` (template id at
- $WORK/template-id) and uv on PATH. Extra pytest args go through
- PYTEST_ADDOPTS.
- idle-e2e run the OSEP-0024 idle-pause/wake SDK suite
- (tests/python/tests/test_fsb_idle_pause_e2e.py); requires an
- `up` with IDLE_PAUSE=1 and uv on PATH.
- down teardown: kind cluster + RustFS + sysctl + XFS StateRoot + caches
-
- --auto-clean on up failure, run down automatically before dumping logs
-
- The RustFS migration replaces MINIO_* and MC_IMAGE overrides. Before
- upgrading an existing environment, run `down` with the previous script.
-
-Notable env overrides: WORK, FSB_DIR, KIND_CLUSTER, KIND_SINGLE,
-DOCKER_MIRROR, RUSTFS_*, RC_IMAGE, EGRESS_IMAGE, SERVER_IMAGE, INGRESS_IMAGE,
-FSB_GOPROXY (default direct; set e.g. https://mirrors.aliyun.com/goproxy/,direct
-when the host cannot reach module VCS hosts directly),
-IMAGE_<COMPONENT>, POOL_MIN/POOL_MAX, WARM_IMAGES=1, SBX_IMAGE, EXECD,
-XFS_STATEROOT=0, OSB_PACKAGE/OSB_BIN, SKIP_TOOL_INSTALL=1, SKIP_LEFTOVER_CLEAN=1,
-IDLE_PAUSE=1 (OSEP-0024 idle-pause/wake profile; REDIS_IMAGE overrides the
-Redis image). See the header of this script.
-EOF
- exit 1
-}
-
-# Allow focused tests to load the stage functions without starting the environment.
-[[ "${BASH_SOURCE[0]}" == "$0" ]] || return 0
-
-for arg in "$@"; do
- case "$arg" in
- --auto-clean) AUTO_CLEAN=1 ;;
- up|down|status|pool|sdk-e2e|idle-e2e) ACTION="$arg" ;;
- *) usage ;;
- esac
-done
-[[ -n "$ACTION" ]] || usage
-
-mkdir -p "$WORK" "$LOGS_DIR"
-
-case "$ACTION" in
- up)
- exec > >(tee -a "$WORK/run.log") 2>&1
- log "=== fast-sandbox-env up ($(date -u +%FT%TZ)) ==="
- {
- echo "environment snapshot ($(date -u +%FT%TZ))"
- command -v kind >/dev/null && kind --version
- kubectl version --client 2>/dev/null | head -1
- go version
- docker --version
- echo "cluster=$KIND_CLUSTER single=$KIND_SINGLE rustfs=$RUSTFS_IMAGE port=$RUSTFS_PORT bucket=$RUSTFS_BUCKET"
- echo "sbxImage=$SBX_IMAGE execd=$EXECD warmImages=$WARM_IMAGES"
- echo "pool=$POOL_NAME poolMin=$POOL_MIN egress=$IMG_EGRESS"
- echo "server=$IMG_SERVER ingress=$IMG_INGRESS fastpath=$FASTPATH_ENDPOINT"
- echo "images: controller=$IMG_CONTROLLER runtime=$IMG_RUNTIME"
- } > "$LOGS_DIR/environment.txt" 2>&1 || true
- if [[ -n "$(kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" || true)" ]] \
- || docker ps -a --format '{{.Names}}' | grep -qx "$RUSTFS_CONTAINER"; then
- if [[ "$SKIP_LEFTOVER_CLEAN" == 1 ]]; then
- log "leftover resources detected; aborting (SKIP_LEFTOVER_CLEAN=1). Run 'integration-env.sh down' first"
- exit 1
- fi
- log "leftover resources detected; cleaning and rebuilding"
- down
- fi
- trap 'on_error up' ERR
- run_stage "preflight + tooling" preflight
- run_stage "opensandbox CLI (osb, latest from PyPI)" ensure_osb
- run_stage "sysctl (fs.inotify)" sysctl_set
- run_stage "fast-sandbox checkout @ pinned commit" ensure_fsb
- run_stage "build images (fast-sandbox + OpenSandbox)" build_images
- run_stage "XFS StateRoot (reflink)" stateroot_xfs_up
- run_stage "render kind config" render_kind_config
- run_stage "kind cluster (KVM passthrough + labels)" kind_up
- run_stage "RustFS + bucket" rustfs_up
- run_stage "RustFS endpoint (kind network)" resolve_rustfs_endpoint
- run_stage "CRDs + control plane" control_plane_up
- run_stage "credentials (publish/pull)" credentials_up
- run_stage "firecracker runtime readiness + DART (P2P)" runtime_up
- run_stage "idle-pause Redis (OSEP-0024 profile)" redis_up
- run_stage "OpenSandbox server + ingress gateway" opensandbox_up
- run_stage "SandboxTemplate build (opensandbox CLI)" template_up
- run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
- run_stage "end-to-end verify (CLI create -> gateway -> execd /ping)" opensandbox_verify
- run_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verify
- run_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify
- run_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verify
- run_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify
- trap - ERR
- stage_summary
- env_summary
- highlight "== up complete: server=$SERVER_URL (header OPEN-SANDBOX-API-KEY: $SERVER_API_KEY), gateway=$GATEWAY_URL =="
- ;;
- pool)
- exec > >(tee -a "$WORK/run.log") 2>&1
- kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
- || die "cluster $KIND_CLUSTER is not up (run up first)"
- trap 'on_error pool' ERR
- run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
- trap - ERR
- env_summary
- ;;
- status)
- status
- ;;
- sdk-e2e)
- kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
- || die "cluster $KIND_CLUSTER is not up (run up first)"
- trap 'on_error sdk-e2e' ERR
- sdk_e2e
- trap - ERR
- ;;
- idle-e2e)
- kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
- || die "cluster $KIND_CLUSTER is not up (run up first)"
- trap 'on_error idle-e2e' ERR
- idle_e2e
- trap - ERR
- ;;
- down)
- down
- ;;
-esac
diff --git a/scripts/fast-sandbox-env/integration.sh b/scripts/fast-sandbox-env/integration.sh
new file mode 100644
index 000000000..ff4a3c760
--- /dev/null
+++ b/scripts/fast-sandbox-env/integration.sh
@@ -0,0 +1,254 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# integration.sh — one-command fast-sandbox integration environment,
+# driven from the OpenSandbox repository.
+#
+# On a bare-metal Linux KVM host it builds the full stack — fast-sandbox at
+# the commit pinned in manifests/third-party/fast-sandbox.commit, a two-node
+# kind cluster with KVM passthrough, the OpenSandbox Helm charts (base CRDs,
+# fast-sandbox control plane, server, ingress-gateway), a RustFS artifact
+# store, and the firecracker-egress-pool SandboxPool with the OpenSandbox
+# egress sidecar — then verifies end to end: sandbox create via the
+# opensandbox CLI, execd /ping through the signed gateway route, delete,
+# a pause/resume round trip, a credential-vault relay lifecycle, the
+# OSEP-0024 idle-pause/wake profile (IDLE_PAUSE=1), and a snapshot ->
+# restore-from-snapshotId boot.
+#
+# All Kubernetes resources come from the charts via `helm template` + plain
+# `kubectl apply` (no helm release state); the only env-owned manifests are
+# the kind cluster config and the SandboxPool. The verify stages drive the
+# server through the published opensandbox CLI ("osb", installed fresh from
+# PyPI; OSB_BIN overrides); raw curl remains only for the calls the CLI
+# cannot express (networkpolicy replace PUT, metadata merge-patch, snapshot
+# re-entry fence) plus the gateway /ping probes.
+#
+# The script is split for maintainability; this file is the entry that
+# sources and chains the parts (a single process, so the ERR trap /
+# failure-dump / --auto-clean contract and the unified stage summary keep
+# working across all stages):
+#
+# lib.sh shared config, logging/stage machinery, failure dump,
+# tooling, server/CLI helpers (sourced by every part)
+# env.sh environment lifecycle: env_up_stages (preflight ->
+# SandboxPool), down, status, pool
+# verify/sandbox.sh end-to-end sandbox verify + egress policy + lifecycle ops
+# verify/pause-resume.sh pause/resume round trip
+# verify/vault.sh credential-vault relay lifecycle (OSEP-0022)
+# verify/idle-pause.sh idle-pause / wake-on-access (OSEP-0024, IDLE_PAUSE=1)
+# verify/snapshot.sh snapshot -> restore round trip
+#
+# env.sh and each verify script also run standalone against a live stack.
+#
+# Usage:
+# ./scripts/fast-sandbox-env/integration.sh up # full environment + pool + server/ingress + verify
+# ./scripts/fast-sandbox-env/integration.sh pool # re-apply the pool only
+# ./scripts/fast-sandbox-env/integration.sh status # component/pool/DART/OpenSandbox health
+# ./scripts/fast-sandbox-env/integration.sh down # teardown, host left clean
+# ./scripts/fast-sandbox-env/integration.sh up --auto-clean # down on failure
+#
+# Environment overrides (all optional):
+# WORK workspace root (default /data/fast-sandbox-env when /data exists, else $PWD/.fast-sandbox-env)
+# FSB_DIR fast-sandbox checkout (default $WORK/fast-sandbox; source pinned in manifests/third-party/fast-sandbox.commit)
+# KIND_CLUSTER / KIND_NODE_IMAGE / KIND_RETAIN / KIND_SINGLE
+# DOCKER_MIRROR comma list injected as docker.io containerd mirrors
+# RUSTFS_PORT / RUSTFS_CONSOLE_PORT / RUSTFS_AK / RUSTFS_SK / RUSTFS_IMAGE / RC_IMAGE / RUSTFS_ENDPOINT
+# IMAGE_<NAME> fast-sandbox component image tags
+# EGRESS_IMAGE egress image tag (default docker.io/opensandbox/egress:latest)
+# SERVER_IMAGE / INGRESS_IMAGE OpenSandbox server/ingress image tags
+# SERVER_HOST_PORT / GATEWAY_HOST_PORT host-side publishes (default 18080/18081)
+# WARM_IMAGES=1 preheat pool warmImages (default: on-demand first-sandbox pull)
+# SBX_IMAGE / EXECD template build inputs (default opensandbox/fsb-sandbox-golden:latest / opensandbox/execd:latest)
+# POOL_MIN / POOL_MAX pool capacity (default 2/2; auto 1/1 when KIND_SINGLE=1)
+# MAX_SANDBOXES_PER_POD per-fastlet sandbox capacity (default 8)
+# XFS_STATEROOT / XFS_SIZE reflink StateRoot on/off and virtual size
+# OSB_PACKAGE / OSB_BIN opensandbox CLI for the verify stages (latest from PyPI / preinstalled binary)
+# SKIP_TOOL_INSTALL / SKIP_LEFTOVER_CLEAN / INOTIFY_VALUE
+# IDLE_PAUSE=1 OSEP-0024 idle-pause/wake profile: Redis, server
+# [automation.idle_pause], gateway activity+wake flags,
+# idle-pause verify stage
+# IDLE_PAUSE_ROUNDS auto-pause/wake rounds in the verify stage and idle-e2e (default 3)
+# IDLE_PAUSE_PARK_BUDGET gateway wake park budget (default 5s; set e.g. 100ms to
+# force every wake through the marked-503 + client-retry path)
+# REDIS_IMAGE Redis image for the idle-pause profile (default redis:7-alpine)
+#
+# Every stage logs to $WORK/logs/; failures dump component logs to
+# logs/failure-<task>-<ts>.txt before exiting (never silently).
+
+# Everything below sources into this shell: one process for the whole `up`,
+# so stage timings, the ERR trap and --auto-clean stay unified.
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib.sh"
+source "$SCRIPT_DIR/env.sh"
+source "$SCRIPT_DIR/verify/sandbox.sh"
+source "$SCRIPT_DIR/verify/pause-resume.sh"
+source "$SCRIPT_DIR/verify/vault.sh"
+source "$SCRIPT_DIR/verify/idle-pause.sh"
+source "$SCRIPT_DIR/verify/snapshot.sh"
+
+
+# --- stage: python sdk e2e (tests/python/tests/test_fsb_e2e.py) -----------------
+
+sdk_e2e() {
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 \
+ || die "server $SERVER_URL is not reachable (run up first)"
+ command -v uv >/dev/null 2>&1 \
+ || die "uv is required on PATH (https://docs.astral.sh/uv/ — curl -LsSf https://astral.sh/uv/install.sh | sh)"
+ local template_id
+ template_id="$(cat "$WORK/template-id" 2>/dev/null || true)"
+ [[ -n "$template_id" ]] || die "no template id at $WORK/template-id (run up first; template_up stores it there)"
+ export OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$template_id"
+ export OPENSANDBOX_TEST_DOMAIN="127.0.0.1:$SERVER_HOST_PORT"
+ export OPENSANDBOX_TEST_PROTOCOL="http"
+ export OPENSANDBOX_TEST_API_KEY="$SERVER_API_KEY"
+ log "sdk e2e: template=$template_id server=$SERVER_URL -> tests/python/tests/test_fsb_e2e.py"
+ cd "$OSB_ROOT/tests/python" || die "tests/python not found under $OSB_ROOT"
+ # No exec: the ERR trap (and the failure dump) must survive a pytest
+ # failure, and exec replaces the shell.
+ uv run pytest tests/test_fsb_e2e.py
+}
+
+idle_e2e() {
+ if [[ "$IDLE_PAUSE" != "1" && ! -f "$WORK/idle-pause" ]]; then
+ die "idle-e2e requires the idle-pause profile (IDLE_PAUSE=1 up)"
+ fi
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 \
+ || die "server $SERVER_URL is not reachable (run up first)"
+ command -v uv >/dev/null 2>&1 \
+ || die "uv is required on PATH (https://docs.astral.sh/uv/ — curl -LsSf https://astral.sh/uv/install.sh | sh)"
+ local template_id
+ template_id="$(cat "$WORK/template-id" 2>/dev/null || true)"
+ [[ -n "$template_id" ]] || die "no template id at $WORK/template-id (run up first; template_up stores it there)"
+ export OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$template_id"
+ export OPENSANDBOX_TEST_DOMAIN="127.0.0.1:$SERVER_HOST_PORT"
+ export OPENSANDBOX_TEST_PROTOCOL="http"
+ export OPENSANDBOX_TEST_API_KEY="$SERVER_API_KEY"
+ export OPENSANDBOX_TEST_IDLE_PAUSE="1"
+ export OPENSANDBOX_TEST_IDLE_ROUNDS="${IDLE_PAUSE_ROUNDS:-3}"
+ log "sdk e2e (idle-pause): template=$template_id server=$SERVER_URL rounds=$OPENSANDBOX_TEST_IDLE_ROUNDS -> tests/python/tests/test_fsb_idle_pause_e2e.py"
+ cd "$OSB_ROOT/tests/python" || die "tests/python not found under $OSB_ROOT"
+ uv run pytest tests/test_fsb_idle_pause_e2e.py
+}
+
+
+usage() {
+ cat <<'EOUT'
+usage: integration.sh [--auto-clean] {up|down|status|pool|sdk-e2e|idle-e2e}
+
+ up initialize the full environment and run every verify stage:
+ fast-sandbox@pinned-commit images, two-node kind cluster (KVM),
+ RustFS, control plane, firecracker runtime readiness + DART (P2P),
+ SandboxTemplate golden image, firecracker-egress-pool (egress
+ attached), the source-built OpenSandbox server + ingress gateway,
+ then the verify stages (see verify/): end-to-end create ->
+ gateway route -> execd /ping, pause/resume, credential-vault
+ relay lifecycle, idle-pause/wake (IDLE_PAUSE=1) and the snapshot
+ -> restore round trip. The CLI (osb) is installed fresh from
+ PyPI unless OSB_BIN points at an existing binary. With
+ IDLE_PAUSE=1: Redis + server [automation.idle_pause] + gateway
+ activity/wake flags (IDLE_PAUSE_ROUNDS cycles, per-round timing
+ summary).
+ pool re-apply only the SandboxPool (after editing manifests/pool/)
+ status nodes / pods / pool / DART P2P counters / RustFS / OpenSandbox health
+ sdk-e2e run the Python SDK e2e suite (tests/python/tests/test_fsb_e2e.py)
+ against the live stack; requires `up` (template id at
+ $WORK/template-id) and uv on PATH. Extra pytest args go through
+ PYTEST_ADDOPTS.
+ idle-e2e run the OSEP-0024 idle-pause/wake SDK suite
+ (tests/python/tests/test_fsb_idle_pause_e2e.py); requires an
+ `up` with IDLE_PAUSE=1 and uv on PATH.
+ down teardown: kind cluster + RustFS + sysctl + XFS StateRoot + caches
+
+ --auto-clean on up failure, run down automatically before dumping logs
+
+ Environment pieces can also be driven directly: env.sh
+ {up|down|status|pool} (provisioning without verifies) and the
+ verify/*.sh scripts (one verify stage each, against a live stack).
+
+ The RustFS migration replaces MINIO_* and MC_IMAGE overrides. Before
+ upgrading an existing environment, run `down` with the previous script.
+
+Notable env overrides: WORK, FSB_DIR, KIND_CLUSTER, KIND_SINGLE,
+DOCKER_MIRROR, RUSTFS_*, RC_IMAGE, EGRESS_IMAGE, SERVER_IMAGE, INGRESS_IMAGE,
+FSB_GOPROXY (default direct; set e.g. https://mirrors.aliyun.com/goproxy/,direct
+when the host cannot reach module VCS hosts directly),
+IMAGE_<COMPONENT>, POOL_MIN/POOL_MAX, WARM_IMAGES=1, SBX_IMAGE, EXECD,
+XFS_STATEROOT=0, OSB_PACKAGE/OSB_BIN, SKIP_TOOL_INSTALL=1, SKIP_LEFTOVER_CLEAN=1,
+IDLE_PAUSE=1 (OSEP-0024 idle-pause/wake profile; REDIS_IMAGE overrides the
+Redis image). See the header of this script.
+EOUT
+ exit 1
+}
+
+# Allow focused tests to load the stage functions without starting the environment.
+[[ "${BASH_SOURCE[0]}" == "$0" ]] || return 0
+
+for arg in "$@"; do
+ case "$arg" in
+ --auto-clean) AUTO_CLEAN=1 ;;
+ up|down|status|pool|sdk-e2e|idle-e2e) ACTION="$arg" ;;
+ *) usage ;;
+ esac
+done
+[[ -n "$ACTION" ]] || usage
+
+case "$ACTION" in
+ up)
+ env_up_prelude
+ trap 'on_error up' ERR
+ env_up_stages
+ run_stage "end-to-end verify (CLI create -> gateway -> execd /ping)" opensandbox_verify
+ run_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verify
+ run_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify
+ run_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verify
+ run_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify
+ trap - ERR
+ stage_summary
+ env_summary
+ highlight "== up complete: server=$SERVER_URL (header OPEN-SANDBOX-API-KEY: $SERVER_API_KEY), gateway=$GATEWAY_URL =="
+ ;;
+ pool)
+ exec > >(tee -a "$WORK/run.log") 2>&1
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ trap 'on_error pool' ERR
+ run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
+ trap - ERR
+ env_summary
+ ;;
+ status)
+ status
+ ;;
+ sdk-e2e)
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ trap 'on_error sdk-e2e' ERR
+ sdk_e2e
+ trap - ERR
+ ;;
+ idle-e2e)
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ trap 'on_error idle-e2e' ERR
+ idle_e2e
+ trap - ERR
+ ;;
+ down)
+ down
+ ;;
+esac
diff --git a/scripts/fast-sandbox-env/lib.sh b/scripts/fast-sandbox-env/lib.sh
new file mode 100644
index 000000000..901136ca4
--- /dev/null
+++ b/scripts/fast-sandbox-env/lib.sh
@@ -0,0 +1,508 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# lib.sh — shared base for the fast-sandbox integration environment scripts
+# (sourced, never executed): configuration variables, logging + stage
+# machinery, the ERR/failure-dump/--auto-clean contract, tool installation,
+# and the raw server / osb CLI helpers every verify script shares.
+#
+# Layout: integration.sh (entry/orchestration), env.sh (environment
+# up/down/status/pool), verify/*.sh (one script per verify stage).
+# Usage docs: integration.sh.
+
+# Source-once guard: re-sourcing must not reset run state.
+[[ -n "${_FSB_ENV_LIB_SOURCED:-}" ]] && return 0
+_FSB_ENV_LIB_SOURCED=1
+
+# -E (errtrace): stage functions must inherit the ERR trap, or on_error
+# (and its failure dump) never fires — every stage runs inside a function.
+set -euEo pipefail
+ENV_PID=$BASHPID
+
+OSB_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+MANIFESTS_DIR="$SCRIPT_DIR/manifests"
+# Heavy run-state belongs on a data volume, not the repo/root disk.
+if [[ -d /data ]] && [[ -z "${WORK:-}" ]]; then
+ WORK="/data/fast-sandbox-env"
+else
+ WORK="${WORK:-$PWD/.fast-sandbox-env}"
+fi
+LOGS_DIR="$WORK/logs"
+GEN_DIR="$WORK/gen"
+
+# Env-owned fast-sandbox clone; the source is exactly the commit pinned in
+# manifests/third-party/fast-sandbox.commit (no ref override — bump the pin).
+FSB_DIR="${FSB_DIR:-$WORK/fast-sandbox}"
+FSB_REPO="$(sed -n 's/^repo:[[:space:]]*//p' "$OSB_ROOT/manifests/third-party/fast-sandbox.commit")"
+FSB_COMMIT="$(sed -n 's/^commit:[[:space:]]*//p' "$OSB_ROOT/manifests/third-party/fast-sandbox.commit")"
+
+KIND_CLUSTER="${KIND_CLUSTER:-fast-sandbox-integration}"
+KIND_SINGLE="${KIND_SINGLE:-0}"
+KIND_RETAIN="${KIND_RETAIN:-0}"
+# Control plane + node runtime (controller/fastpath, firecracker-runtime,
+# agent credentials, artifact-store config) ...
+NS="opensandbox-system"
+# ... while the SandboxPool/Template/Sandbox resources and the fastlet and
+# builder Pods they spawn live in the dataplane namespace.
+RESOURCE_NS="opensandbox-dataplane"
+
+RUSTFS_IMAGE="${RUSTFS_IMAGE:-rustfs/rustfs:latest}"
+RC_IMAGE="${RC_IMAGE:-rustfs/rc:latest}"
+RUSTFS_PORT="${RUSTFS_PORT:-19000}"
+# In-container ports are fixed (9000 S3 / 9001 console); these vars only move
+# the host-side 127.0.0.1 publishes (defaults avoid 9000/9001 collisions).
+RUSTFS_CONTAINER_PORT=9000
+RUSTFS_CONSOLE_PORT="${RUSTFS_CONSOLE_PORT:-19001}"
+RUSTFS_AK="${RUSTFS_AK:-integration-env}"
+RUSTFS_SK="${RUSTFS_SK:-integration-env-secret}"
+RUSTFS_BUCKET="sandbox-images"
+RUSTFS_CONTAINER="${RUSTFS_CONTAINER:-fast-sandbox-env-rustfs}"
+RUSTFS_DATA="$WORK/rustfs-data"
+RUSTFS_ENDPOINT="${RUSTFS_ENDPOINT:-}" # auto-derived from the kind network
+RC_CONFIG_DIR="$WORK/rc-config"
+
+SBX_IMAGE="${SBX_IMAGE:-opensandbox/fsb-sandbox-golden:latest}"
+EXECD="${EXECD:-opensandbox/execd:latest}"
+# WARM_IMAGES=1 preheats pool warmImages (keyed by template id); the default
+# is the on-demand first-sandbox pull through DART.
+WARM_IMAGES="${WARM_IMAGES:-0}"
+
+POOL_NAME="${POOL_NAME:-firecracker-egress-pool}"
+MAX_SANDBOXES_PER_POD="${MAX_SANDBOXES_PER_POD:-8}"
+if [[ -n "${POOL_MIN:-}" && -n "${POOL_MAX:-}" ]]; then
+ :
+elif [[ "$KIND_SINGLE" == "1" ]]; then
+ # Cache-only topology: a single fastlet is enough (no peer traffic).
+ POOL_MIN="${POOL_MIN:-1}"
+ POOL_MAX="${POOL_MAX:-1}"
+else
+ POOL_MIN="${POOL_MIN:-2}"
+ POOL_MAX="${POOL_MAX:-2}"
+fi
+
+# Image tags (env-overridable per component).
+IMG_CONTROLLER="${IMAGE_CONTROLLER:-opensandbox/fsb-controller:dev}"
+IMG_FASTLET="${IMAGE_FASTLET:-opensandbox/fsb-fastlet:dev}"
+IMG_FASTLET_PROXY="${IMAGE_FASTLET_PROXY:-opensandbox/fsb-fastlet-proxy:dev}"
+IMG_JANITOR="${IMAGE_JANITOR:-opensandbox/fsb-janitor:dev}"
+IMG_BUILDER="${IMAGE_BUILDER:-opensandbox/fsb-sandboxtemplate-builder:dev}"
+IMG_RUNTIME="${IMAGE_RUNTIME:-opensandbox/fsb-firecracker-runtime:dev}"
+IMG_EGRESS="${EGRESS_IMAGE:-docker.io/opensandbox/egress:latest}"
+
+image_repo() { printf '%s' "${1%:*}"; }
+image_tag() { printf '%s' "${1##*:}"; }
+
+# --- OpenSandbox server + ingress gateway (source-built; always deployed) ------
+
+OSB_NS="opensandbox-system"
+IMG_SERVER="${SERVER_IMAGE:-docker.io/opensandbox/server:env}"
+IMG_INGRESS="${INGRESS_IMAGE:-docker.io/opensandbox/ingress:env}"
+FASTPATH_ENDPOINT="fast-sandbox-fastpath.opensandbox-system.svc:9090"
+SERVER_API_KEY="fast-sandbox-env"
+
+# OSEP-0024 profile: Redis + server [automation.idle_pause] + gateway activity/wake flags.
+IDLE_PAUSE="${IDLE_PAUSE:-0}"
+REDIS_IMAGE="${REDIS_IMAGE:-redis:7-alpine}"
+REDIS_RELEASE="idle-pause-redis"
+REDIS_DSN="redis://$REDIS_RELEASE.$OSB_NS.svc.cluster.local:6379/0"
+# Per-workdir f1.* route-scope signing key: re-applies must keep old routes
+# verifiable.
+SIGNING_KEY_FILE="$WORK/opensandbox-signing-key"
+# Host publish via kind extraPortMappings (loopback only; overridable on
+# port collisions).
+SERVER_HOST_PORT="${SERVER_HOST_PORT:-18080}"
+GATEWAY_HOST_PORT="${GATEWAY_HOST_PORT:-18081}"
+SERVER_NODEPORT=30880
+GATEWAY_NODEPORT=30881
+GATEWAY_ADDRESS="127.0.0.1:$GATEWAY_HOST_PORT"
+SERVER_URL="http://127.0.0.1:$SERVER_HOST_PORT"
+GATEWAY_URL="http://127.0.0.1:$GATEWAY_HOST_PORT"
+
+# Applied by the firecracker-runtime readiness loop itself (never manually).
+KVM_NODE_LABEL="sandbox.fast.io/kvm"
+FC_NODE_LABEL="fast-sandbox.io/firecracker-node"
+
+INOTIFY_VALUE="${INOTIFY_VALUE:-8192}"
+SYSCTL_BACKUP="$WORK/sysctl-backup"
+SKIP_TOOL_INSTALL="${SKIP_TOOL_INSTALL:-0}"
+SKIP_LEFTOVER_CLEAN="${SKIP_LEFTOVER_CLEAN:-0}"
+KIND_VERSION="${KIND_VERSION:-v0.24.0}"
+KUBECTL_VERSION="${KUBECTL_VERSION:-v1.31.0}"
+HELM_VERSION="${HELM_VERSION:-v3.16.4}"
+
+# opensandbox CLI for the verify stages; installed by ensure_osb.
+OSB_PACKAGE="${OSB_PACKAGE:-opensandbox-cli}"
+OSB_BIN="${OSB_BIN:-}"
+OSB_VENV="$WORK/tools/osb-venv"
+
+# Internal goproxy mirrors can 500 on shared hosts; direct VCS just works there.
+FSB_GOPROXY="${FSB_GOPROXY:-direct}"
+export GOPROXY="$FSB_GOPROXY"
+
+AUTO_CLEAN=0
+ACTION=""
+
+# --- logging / stage machinery --------------------------------------------------
+
+log() { printf '\033[1;34m[fast-sandbox-env]\033[0m %s\n' "$*" | tee -a "$WORK/run.log" >&2; }
+die() { printf '\033[1;31m[fast-sandbox-env] ERROR:\033[0m %s\n' "$*" >&2; exit 1; }
+pass() { printf '\033[1;32m[fast-sandbox-env] PASS\033[0m %s\n' "$*" | tee -a "$WORK/run.log" >&2; }
+# fail dumps before exiting: an explicit `exit` does not fire the ERR trap.
+fail() { printf '\033[1;31m[fast-sandbox-env] FAIL\033[0m %s\n' "$*" >&2; failure_dump "${STAGE_CUR_LABEL:-$ACTION}" || true; exit 1; }
+highlight() { printf '\033[1;36m%s\033[0m\n' "$*"; }
+
+now_ms() { date +%s%N; } # GNU date (Linux); the script targets a Linux KVM host
+ms2s() { awk -v ms="$1" 'BEGIN { printf "%.1f", ms / 1000 }'; }
+
+declare -a STAGE_ORDER=()
+declare -a STAGE_MS_LIST=()
+STAGE_CUR=""
+STAGE_CUR_LABEL=""
+STAGE_START_NS=0
+STAGE_N=0
+
+stage_begin() {
+ STAGE_N=$((STAGE_N + 1))
+ STAGE_CUR="$1"
+ STAGE_CUR_LABEL="$(printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_')"
+ STAGE_START_NS="$(now_ms)"
+ printf '\n\033[1;36m==> [%d] %s\033[0m\n' "$STAGE_N" "$1"
+}
+
+stage_done() {
+ local ms
+ ms=$(( ($(now_ms) - STAGE_START_NS) / 1000000 ))
+ STAGE_ORDER+=("$STAGE_CUR")
+ STAGE_MS_LIST+=("$ms")
+ printf '\033[1;32m OK in %ss\033[0m %s\n' "$(ms2s "$ms")" "${1:-}"
+}
+
+run_stage() {
+ local description="$1" func="$2"
+ shift 2
+ stage_begin "$description"
+ "$func" "$@"
+ stage_done
+}
+
+stage_summary() {
+ local index name ms total=0
+ highlight "== stage timings =="
+ printf ' \033[1m%-46s %10s\033[0m\n' "stage" "duration"
+ for index in "${!STAGE_ORDER[@]}"; do
+ name="${STAGE_ORDER[$index]}"
+ ms="${STAGE_MS_LIST[$index]}"
+ total=$((total + ms))
+ printf ' %-46s %9ss\n' "$name" "$(ms2s "$ms")"
+ done
+ printf ' \033[1m%-46s %9ss\033[0m\n' "TOTAL" "$(ms2s "$total")"
+}
+
+wait_for() { # description attempts command [args...]
+ local description="$1" attempts="$2" attempt=0
+ shift 2
+ while ! "$@" >/dev/null 2>&1; do
+ attempt=$((attempt + 1))
+ if [[ "$attempt" -ge "$attempts" ]]; then
+ fail "$description (after $attempts attempts)"
+ fi
+ sleep 2
+ done
+ pass "$description"
+}
+
+kubectl_get() { kubectl -n "$RESOURCE_NS" get "$1" -o jsonpath="$2"; }
+
+sudo_() { if [[ "$(id -u)" == 0 ]]; then "$@"; else sudo "$@"; fi; }
+
+# --- helpers ---------------------------------------------------------------------
+
+kind_node() { kind get nodes --name "$KIND_CLUSTER" 2>/dev/null | head -1; }
+
+kind_network() { # docker network of the first node container
+ local node
+ node="$(kind_node)" || return 1
+ docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' "$node" | tr ' ' '\n' | grep -v '^$' | head -1
+}
+
+rc() {
+ docker run --rm --network host --user "$(id -u):$(id -g)" -e RC_CONFIG_DIR=/config \
+ -e HTTP_PROXY= -e HTTPS_PROXY= -e ALL_PROXY= \
+ -e http_proxy= -e https_proxy= -e all_proxy= \
+ -v "$RC_CONFIG_DIR:/config" "$RC_IMAGE" "$@"
+}
+
+# --- failure dump ------------------------------------------------------------------
+
+on_error() {
+ local rc=$? task="$1"
+ # Subshells inherit ERR: propagate the error instead of tearing down
+ # main-process resources; ignore SIGPIPE (cancelled runners).
+ trap - ERR
+ [[ "$BASHPID" == "$ENV_PID" ]] || exit "$rc"
+ trap '' PIPE
+ # Dump before --auto-clean teardown destroys the evidence.
+ failure_dump "$task" || true
+ # down lives in env.sh; standalone verify scripts have no teardown to run.
+ if [[ "$AUTO_CLEAN" == 1 ]] && declare -F down >/dev/null; then
+ log "$ACTION failed at $task; --auto-clean: running down" || true
+ (down) >/dev/null 2>&1 || true
+ fi
+ printf '\033[1;31m[fast-sandbox-env] FAILED at %s; dump: %s\033[0m\n' \
+ "$task" "$LOGS_DIR/failure-$task-*.txt" >&2 || true
+ exit "$rc"
+}
+
+failure_dump() {
+ local task="$1"
+ local dump="$LOGS_DIR/failure-$task-$(date +%s).txt"
+ mkdir -p "$LOGS_DIR"
+ {
+ echo "=== fast-sandbox-env failure: $task ($(date -u +%FT%TZ)) ==="
+ env | grep -E '^(KIND_(CLUSTER|SINGLE|RETAIN|NODE_IMAGE)=|FSB_DIR=|SBX_IMAGE=|IMG_[A-Z_]+=|EGRESS_IMAGE=|EXECD=|WORK=|POOL_(NAME|MIN|MAX)=|INGRESS_IMAGE=|WARM_IMAGES=|XFS_(STATEROOT|SIZE)=|RUSTFS_(IMAGE|PORT|CONSOLE_PORT|BUCKET|CONTAINER|DATA|ENDPOINT)=|RC_IMAGE=)' || true
+ echo "--- fast-sandbox checkout ---"
+ git -C "$FSB_DIR" rev-parse HEAD 2>&1 || true
+ echo "--- kind-create.log (tail) ---"
+ tail -40 "$LOGS_DIR/kind-create.log" 2>&1 || true
+ echo "--- nodes ---"
+ kubectl get nodes -o wide 2>&1 || true
+ echo "--- pods ---"
+ kubectl get pods -n "$NS" -o wide 2>&1 || true
+ echo "--- controller logs (tail) ---"
+ kubectl logs -n "$NS" deploy/fast-sandbox-controller --tail=80 2>&1 || true
+ echo "--- firecracker-runtime logs (tail) ---"
+ kubectl logs -n "$NS" daemonset/firecracker-runtime --all-containers --tail=80 2>&1 || true
+ # --all-containers: the label selector alone logs only the first
+ # container and drops egress. Deep tail: failure loops flood the
+ # window and bury the pre-incident history.
+ echo "--- fastlet pod logs, every container (deep tail) ---"
+ kubectl logs -n "$RESOURCE_NS" -l app=sandbox-fastlet \
+ --all-containers --prefix --timestamps --tail=15000 2>&1 || true
+ echo "--- fastlet pod logs, previous container instances (crashes) ---"
+ for pod in $(kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet -o name 2>/dev/null); do
+ for container in $(kubectl -n "$RESOURCE_NS" get "$pod" -o jsonpath='{.spec.containers[*].name}' 2>/dev/null); do
+ prev="$(kubectl -n "$RESOURCE_NS" logs "$pod" -c "$container" --previous --timestamps --tail=200 2>&1 || true)"
+ case "$prev" in *"no previous container"*) continue ;; esac
+ echo "### ${pod##*/} container=$container (previous)"
+ printf '%s\n' "$prev"
+ done
+ done
+ echo "--- builder pods + logs (tail) ---"
+ kubectl get pods -n "$RESOURCE_NS" -l sandbox.fast.io/sandboxtemplate --show-labels 2>&1 || true
+ kubectl logs -n "$RESOURCE_NS" -l sandbox.fast.io/sandboxtemplate --tail=80 2>&1 || true
+ echo "--- SandboxTemplates (status carries the build failure reason) ---"
+ kubectl get sandboxtemplates -n "$RESOURCE_NS" -o yaml 2>&1 || true
+ # Sandbox CR states: a pause/snapshot/resume failure shows up as a
+ # stuck or failed status condition here, with the phase the poll
+ # never reached.
+ echo "--- Sandbox CRs (status only) ---"
+ kubectl get sandboxes -n "$RESOURCE_NS" -o json 2>/dev/null \
+ | jq -c '.items[] | {name: .metadata.name, phase: .status.phase, conditions: .status.conditions, infra: .status.infraComponents}' 2>&1 || true
+ # A mount-namespace mismatch (driver staging path invisible to
+ # egress) shows up as ENOENT on the egress side only.
+ for pod in $(kubectl -n "$RESOURCE_NS" get pods -l app=sandbox-fastlet -o name 2>/dev/null); do
+ echo "### ${pod##*/}: snapshot staging visibility"
+ echo "-- fastlet view:"
+ kubectl -n "$RESOURCE_NS" exec "$pod" -c fastlet -- \
+ ls -la /var/lib/fast-sandbox/firecracker/snapshots 2>&1 | tail -8 || true
+ echo "-- egress view:"
+ kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- \
+ ls -la /var/lib/fast-sandbox/firecracker/snapshots 2>&1 | tail -8 || true
+ done
+ echo "--- recent events ($RESOURCE_NS) ---"
+ kubectl get events -n "$RESOURCE_NS" --sort-by=.lastTimestamp 2>&1 | tail -30 || true
+ echo "--- OpenSandbox pods ($OSB_NS) ---"
+ kubectl get pods -n "$OSB_NS" -o wide 2>&1 || true
+ echo "--- server logs (tail) ---"
+ kubectl logs -n "$OSB_NS" deploy/opensandbox-server --tail=80 2>&1 || true
+ echo "--- ingress gateway logs (tail) ---"
+ kubectl logs -n "$OSB_NS" deploy/opensandbox-ingress-gateway --tail=80 2>&1 || true
+ echo "--- pool ---"
+ kubectl get sandboxpool -n "$RESOURCE_NS" -o yaml 2>&1 || true
+ echo "--- rustfs docker logs (tail) ---"
+ docker logs "$RUSTFS_CONTAINER" --tail=80 2>&1 || true
+ } > "$dump" 2>&1 || true
+ log "failure dump: $dump"
+}
+
+# --- stage: preflight + tooling ------------------------------------------------------
+
+install_release_binary() { # name version url
+ local name="$1" version="$2" url="$3" tmp
+ log "installing $name $version -> /usr/local/bin/$name"
+ tmp="$(mktemp -d)"
+ curl -fL --retry 3 -o "$tmp/$name" "$url" \
+ || die "download $name failed ($url); install it manually or retry"
+ sudo_ install -m 0755 "$tmp/$name" "/usr/local/bin/$name"
+ rm -rf "$tmp"
+}
+
+ensure_tool() { # name
+ local name="$1"
+ if command -v "$name" >/dev/null 2>&1; then
+ return 0
+ fi
+ if [[ "$SKIP_TOOL_INSTALL" == 1 ]]; then
+ die "$name is required (SKIP_TOOL_INSTALL=1: install it manually)"
+ fi
+ case "$name" in
+ kind)
+ install_release_binary kind "$KIND_VERSION" \
+ "https://github.com/kubernetes-sigs/kind/releases/download/$KIND_VERSION/kind-linux-amd64"
+ ;;
+ kubectl)
+ install_release_binary kubectl "${KUBECTL_VERSION#v}" \
+ "https://dl.k8s.io/release/$KUBECTL_VERSION/bin/linux/amd64/kubectl"
+ ;;
+ helm)
+ local tmp
+ log "installing helm $HELM_VERSION -> /usr/local/bin/helm"
+ tmp="$(mktemp -d)"
+ curl -fL --retry 3 -o "$tmp/helm.tgz" \
+ "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \
+ || die "download helm failed; install it manually or retry"
+ sudo_ tar -xzf "$tmp/helm.tgz" -C "$tmp" linux-amd64/helm
+ sudo_ install -m 0755 "$tmp/linux-amd64/helm" /usr/local/bin/helm
+ rm -rf "$tmp"
+ ;;
+ jq)
+ log "installing jq via package manager"
+ if command -v apt-get >/dev/null 2>&1; then
+ sudo_ apt-get install -y jq >/dev/null
+ elif command -v yum >/dev/null 2>&1; then
+ sudo_ yum install -y jq >/dev/null
+ elif command -v apk >/dev/null 2>&1; then
+ sudo_ apk add --no-cache jq >/dev/null
+ else
+ die "no supported package manager to install jq; install it manually"
+ fi
+ ;;
+ *)
+ die "$name is required; install it manually"
+ ;;
+ esac
+ command -v "$name" >/dev/null 2>&1 || die "$name installation failed"
+}
+
+ensure_osb() {
+ if [[ -n "$OSB_BIN" ]]; then
+ [[ -x "$OSB_BIN" ]] || die "OSB_BIN=$OSB_BIN is not executable"
+ log "using osb from OSB_BIN: $OSB_BIN"
+ else
+ if [[ "$SKIP_TOOL_INSTALL" == 1 ]] && [[ ! -x "$OSB_VENV/bin/osb" ]]; then
+ die "osb is required (SKIP_TOOL_INSTALL=1: install opensandbox-cli manually or set OSB_BIN)"
+ fi
+ command -v uv >/dev/null 2>&1 \
+ || die "uv is required to install the opensandbox CLI (or set OSB_BIN)"
+ log "installing the latest $OSB_PACKAGE into $OSB_VENV (uv)"
+ : > "$LOGS_DIR/osb-install.log"
+ rm -rf "$OSB_VENV"
+ # Fresh venv at its final path: venv console-script shebangs embed the
+ # interpreter's absolute path, so create-elsewhere-and-move breaks them.
+ uv venv --python 3.12 "$OSB_VENV" >>"$LOGS_DIR/osb-install.log" 2>&1 \
+ || { tail -n 20 "$LOGS_DIR/osb-install.log" >&2 || true; die "uv venv --python 3.12 failed (full log: $LOGS_DIR/osb-install.log; or set OSB_BIN)"; }
+ uv pip install --python "$OSB_VENV/bin/python" --upgrade "$OSB_PACKAGE" \
+ >>"$LOGS_DIR/osb-install.log" 2>&1 \
+ || { tail -n 20 "$LOGS_DIR/osb-install.log" >&2 || true; die "uv pip install --upgrade $OSB_PACKAGE failed (index unreachable? full log: $LOGS_DIR/osb-install.log; or set OSB_BIN)"; }
+ OSB_BIN="$OSB_VENV/bin/osb"
+ fi
+ log "osb version: $("$OSB_BIN" --version 2>/dev/null | tail -n1 || echo unknown)"
+ pass "opensandbox CLI ready ($OSB_BIN)"
+}
+
+host_port_busy() { # port -> 0 when something already listens on 127.0.0.1:<port>
+ (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null
+}
+
+# Pull only when absent locally (*_IMAGE overrides cover private registries).
+ensure_image() { # image -> pull only when missing locally
+ docker image inspect "$1" >/dev/null 2>&1 && return 0
+ docker pull -q "$1" >/dev/null || die "image $1 is not available locally and the pull failed (pre-load it with docker load, or override the *_IMAGE variable)"
+}
+# Raw curl, kept for the calls the CLI cannot express (networkpolicy
+# replace PUT, metadata merge-patch, snapshot re-entry fence).
+server_api() { # method path [json-body]
+ local method="$1" path="$2" body="${3:-}"
+ if [[ -n "$body" ]]; then
+ curl -fsS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
+ -H "Content-Type: application/json" -d "$body" "$SERVER_URL$path"
+ else
+ curl -fsS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" "$SERVER_URL$path"
+ fi
+}
+
+# opensandbox CLI pointed at the lifecycle server; --no-color keeps
+# -o json parseable.
+osb_api() {
+ "$OSB_BIN" --no-color --domain "127.0.0.1:$SERVER_HOST_PORT" --protocol http \
+ --api-key "$SERVER_API_KEY" --request-timeout 60 "$@"
+}
+
+# The server assigns the sandbox id (CreateSandboxRequest carries none);
+# the verify probes share it through VERIFY_ID.
+VERIFY_ID=""
+# The golden-image template id: template_up sets it and persists it at
+# $WORK/template-id; the verify scripts resolve it through
+# require_template_id when run standalone.
+TEMPLATE_ID=""
+
+verify_sandbox_gone() {
+ ! osb_api sandbox get "$VERIFY_ID" -o json >/dev/null 2>&1
+}
+
+# Fresh signed route + one GET: doubles as the "runtime actually serving" probe.
+execd_ping_ok() {
+ local route code
+ route="$(osb_api sandbox endpoint "$VERIFY_ID" --port 44772 -o json 2>/dev/null \
+ | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
+ [[ -n "$route" ]] || return 1
+ code="$(curl -sS -m 5 -o /dev/null -w '%{http_code}' \
+ -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" 2>/dev/null || true)"
+ [[ "$code" == "200" ]]
+}
+
+sandbox_state_is() { # <state>
+ [[ "$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')" == "$1" ]]
+}
+
+sandbox_running() { sandbox_state_is Running; }
+sandbox_paused() { sandbox_state_is Paused; }
+sandbox_state_get() {
+ osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty'
+}
+
+# --- standalone-verify helpers ------------------------------------------------------
+
+# Locate the osb CLI without installing it (verify scripts run against an
+# already-provisioned stack; only env `up` installs).
+resolve_osb() {
+ if [[ -z "$OSB_BIN" ]]; then
+ OSB_BIN="$OSB_VENV/bin/osb"
+ fi
+ [[ -x "$OSB_BIN" ]] \
+ || die "osb not found at $OSB_BIN (run 'integration.sh up' first, or set OSB_BIN)"
+ log "osb version: $("$OSB_BIN" --version 2>/dev/null | tail -n1 || echo unknown)"
+}
+
+# Resolve the golden-image template id for standalone verify runs.
+require_template_id() {
+ [[ -n "${TEMPLATE_ID:-}" ]] || TEMPLATE_ID="$(cat "$WORK/template-id" 2>/dev/null || true)"
+ [[ -n "$TEMPLATE_ID" ]] \
+ || die "no template id at $WORK/template-id (run 'integration.sh up' first)"
+}
+
+# Workspace directories must exist before any log()/stage call.
+mkdir -p "$WORK" "$LOGS_DIR" "$GEN_DIR"
diff --git a/scripts/fast-sandbox-env/manifests/cluster/kind-cluster.yaml b/scripts/fast-sandbox-env/manifests/cluster/kind-cluster.yaml
index 75041ab9f..6cee66bcd 100644
--- a/scripts/fast-sandbox-env/manifests/cluster/kind-cluster.yaml
+++ b/scripts/fast-sandbox-env/manifests/cluster/kind-cluster.yaml
@@ -12,7 +12,7 @@
# /sys/.../kvm KVM access inside the container
# /dev/net/tun TAP devices for the per-clone netns data plane
# /dev/shm snapshot memory staging (default 64MiB is too small)
-# /var/lib/fast-sandbox StateRoot; prepared by integration-env.sh up
+# /var/lib/fast-sandbox StateRoot; prepared by integration.sh up
# (XFS loop image when XFS_STATEROOT=1 so
# per-sandbox rootfs copies are CoW reflinks)
#
@@ -33,10 +33,10 @@
# Registry mirrors: this file assumes direct Docker Hub access (the
# installer pulls curlimages/curl, alpine). On blocked hosts set
# DOCKER_MIRROR="https://mirror.example.com[,https://fallback]" and the
-# render step in integration-env.sh injects a containerdConfigPatches
+# render step in integration.sh injects a containerdConfigPatches
# block for docker.io before `kind create`.
#
-# Usage (normally through integration-env.sh up):
+# Usage (normally through integration.sh up):
# kind create cluster --config kind-cluster.yaml
# kubectl label node <node> sandbox.fast.io/kvm=true
# kubectl label node <node> fast-sandbox.io/firecracker-node=true
diff --git a/scripts/fast-sandbox-env/manifests/idle-pause/redis.yaml b/scripts/fast-sandbox-env/manifests/idle-pause/redis.yaml
index 58232704a..5d22d8be8 100644
--- a/scripts/fast-sandbox-env/manifests/idle-pause/redis.yaml
+++ b/scripts/fast-sandbox-env/manifests/idle-pause/redis.yaml
@@ -13,7 +13,7 @@
# limitations under the License.
#
# OSEP-0024 profile: ephemeral Redis for activity keys and pause locks.
-# Applied into the opensandbox-system namespace by integration-env.sh.
+# Applied into the opensandbox-system namespace by integration.sh.
apiVersion: apps/v1
kind: Deployment
metadata:
diff --git a/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml b/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml
index 9974da180..e1f0d1622 100644
--- a/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml
+++ b/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml
@@ -28,7 +28,7 @@
# No sandboxes are created by the environment itself — apply-driven
# capacity only.
#
-# Tokens rendered by integration-env.sh (pool stage, into $WORK/gen):
+# Tokens rendered by integration.sh (pool stage, into $WORK/gen):
# "@FASTLET_IMAGE@" "@EGRESS_IMAGE@" "@POOL_MIN@" "@POOL_MAX@"
# # @WARM_IMAGES@ -> warmImages block when WARM_IMAGES=1
# (quoted in the template so the file stays valid YAML; the render step
diff --git a/scripts/fast-sandbox-env/verify/idle-pause.sh b/scripts/fast-sandbox-env/verify/idle-pause.sh
new file mode 100755
index 000000000..dd2c2d127
--- /dev/null
+++ b/scripts/fast-sandbox-env/verify/idle-pause.sh
@@ -0,0 +1,189 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# verify/idle-pause.sh — idle-pause / wake-on-access verify (OSEP-0024 profile)
+#
+# Auto-pause after the idle window, wake on the next gateway request, the wakeOnAccess=false compatibility invariant, and pause behavior while Redis is down / flushed.
+# Runs as part of `integration.sh up` (the entry chains every verify
+# script) or standalone against a live stack from `env.sh up`.
+# Requires the idle-pause profile: IDLE_PAUSE=1 up (or the $WORK/idle-pause marker it leaves).
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
+
+# --- stage: idle-pause / wake-on-access verify (OSEP-0024 profile) --------------
+
+# Poll the gateway until /ping returns 200 (marked 503s tolerated; the
+# resume is never canceled). Fails on budget exhaustion.
+idle_wake_wait() { # <label> [sandbox_id]
+ local label="$1" id="${2:-$VERIFY_ID}" code
+ local deadline=$(( $(now_ms) + 600000000000 )) wake_seen=0 next_state_log=0
+ while :; do
+ code="$(idle_ping_code "$id")"
+ case "$code" in
+ 200) break ;;
+ wake-503) wake_seen=1 ;;
+ esac
+ if (( $(now_ms) >= next_state_log )); then
+ log "verify (idle-pause): $label still waiting (last=$code, state=$(sandbox_state_get))"
+ next_state_log=$(( $(now_ms) + 30000000000 ))
+ fi
+ [[ "$(now_ms)" -ge "$deadline" ]] && fail "$label: wake /ping never returned 200 (last=$code)"
+ sleep 1
+ done
+}
+
+# Gateway answer for one /ping attempt: "200", "wake-503", or "other-<code>".
+idle_ping_code() { # [sandbox_id] (default $VERIFY_ID)
+ local id="${1:-$VERIFY_ID}" route headers code marker
+ route="$(osb_api sandbox endpoint "$id" --port 44772 -o json 2>/dev/null \
+ | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
+ [[ -n "$route" ]] || { echo "noroute"; return 0; }
+ headers="$(mktemp)"
+ curl -sS -m 10 -D "$headers" -o /dev/null \
+ -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" >/dev/null 2>&1 || true
+ code="$(tr -d '\r' < "$headers" | head -1 | awk '{print $2}')" || true
+ marker="$(tr -d '\r' < "$headers" | grep -i '^OpenSandbox-Wake:' | awk '{print $2}')" || true
+ rm -f "$headers"
+ if [[ "$code" == "200" ]]; then
+ echo "200"
+ elif [[ "$code" == "503" && "$marker" == "expired" ]]; then
+ echo "wake-503"
+ else
+ echo "other-${code:-none}"
+ fi
+}
+
+idle_pause_verify() {
+ # Raw curl create: the CLI has no lifecycle.idlePolicy parameter yet.
+ # Marker-aware so a standalone run against an IDLE_PAUSE=1 stack works
+ # without re-exporting the flag (down() removes the marker).
+ [[ "$IDLE_PAUSE" == "1" || -f "$WORK/idle-pause" ]] \
+ || { log "idle-pause profile off (IDLE_PAUSE=0); skipping verify"; return 0; }
+ local created t0 t1 code i
+ local x=30
+ local rounds="${IDLE_PAUSE_ROUNDS:-3}"
+ [[ "$rounds" =~ ^[1-9][0-9]*$ ]] || die "IDLE_PAUSE_ROUNDS must be a positive integer (got $rounds)"
+ log "verify (idle-pause): creating an opted-in sandbox (idleTimeoutSeconds=$x, rounds=$rounds) via the lifecycle API"
+ t0="$(now_ms)"
+ created="$(server_api POST /sandboxes "$(jq -n \
+ --arg template "$TEMPLATE_ID" \
+ --argjson timeout 3600 --argjson x "$x" \
+ '{templateId: $template, timeout: $timeout,
+ lifecycle: {idlePolicy: {idleTimeoutSeconds: $x, onIdle: "pause", wakeOnAccess: true}},
+ metadata: {origin: "fast-sandbox-env-idle-pause"}}')" 2>/dev/null)" \
+ || fail "idle-pause create failed against $SERVER_URL"
+ VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
+ [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
+ wait_for "idle-pause target Running" 300 sandbox_running
+ log "verify (idle-pause): $VERIFY_ID create->Running $(( ($(now_ms) - t0) / 1000000 ))ms"
+
+ wait_for "pre-idle execd /ping 200 through the gateway (records activity)" 100 execd_ping_ok
+
+ local -a pause_ms=() wake_ms=() attempts=()
+ for ((i = 1; i <= rounds; i++)); do
+ t0="$(now_ms)"
+ wait_for "round $i/$rounds auto-paused within X + sweep + checkpoint" 240 sandbox_paused
+ pause_ms+=("$(( ($(now_ms) - t0) / 1000000 ))")
+ pass "round $i/$rounds: auto-paused in ${pause_ms[-1]}ms"
+
+ t1="$(now_ms)"
+ idle_wake_wait "round $i wake"
+ wake_ms+=("$(( ($(now_ms) - t1) / 1000000 ))")
+ attempts+=("$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" \
+ -o jsonpath='{.status.placement.attempt}' 2>/dev/null || echo '?')")
+ pass "round $i/$rounds: woke in ${wake_ms[-1]}ms (placement.attempt=${attempts[-1]})"
+ done
+
+ local joined_pause joined_wake joined_att
+ printf -v joined_pause '%sms/' "${pause_ms[@]}"
+ printf -v joined_wake '%sms/' "${wake_ms[@]}"
+ printf -v joined_att '%s/' "${attempts[@]}"
+ pass "idle-pause ${rounds} rounds: pause=${joined_pause%?} wake=${joined_wake%?} placement.attempt=${joined_att%?}"
+
+ local main_id="$VERIFY_ID" created2 id2 payload
+ log "verify (idle-pause): scenario wakeOnAccess=false"
+ created2="$(server_api POST /sandboxes "$(jq -n \
+ --arg template "$TEMPLATE_ID" --argjson timeout 3600 --argjson x "$x" \
+ '{templateId: $template, timeout: $timeout,
+ lifecycle: {idlePolicy: {idleTimeoutSeconds: $x, onIdle: "pause", wakeOnAccess: false}},
+ metadata: {origin: "fast-sandbox-env-idle-pause"}}')" 2>/dev/null)" \
+ || fail "wakeOnAccess=false create failed"
+ id2="$(printf '%s' "$created2" | jq -r '.id')"
+ [[ -n "$id2" && "$id2" != "null" ]] || fail "wakeOnAccess=false create carried no id"
+ VERIFY_ID="$id2"
+ wait_for "wakeOnAccess=false target Running" 300 sandbox_running
+ wait_for "wakeOnAccess=false auto-paused" 240 sandbox_paused
+ code="$(idle_ping_code "$id2")"
+ [[ "$code" == "200" || "$code" == "wake-503" ]] \
+ && fail "wakeOnAccess=false sandbox was woken by traffic (code=$code)"
+ sleep 10
+ sandbox_paused || fail "wakeOnAccess=false sandbox changed state without resume API"
+ osb_api sandbox resume "$id2" --skip-health-check >/dev/null 2>&1 || fail "manual resume failed"
+ wait_for "wakeOnAccess=false resumed manually" 240 sandbox_running
+ pass "wakeOnAccess=false: traffic never woke it, manual resume did"
+ osb_api sandbox kill "$id2" >/dev/null 2>&1 || true
+ wait_for "wakeOnAccess=false sandbox deleted" 120 verify_sandbox_gone
+ VERIFY_ID="$main_id"
+
+ log "verify (idle-pause): scenario redis down"
+ # The previous scenario may have left it Paused.
+ idle_wake_wait "pre-redis-down wake"
+ t0="$(now_ms)"
+ kubectl -n "$OSB_NS" scale deploy/idle-pause-redis --replicas=0 >/dev/null
+ while sandbox_running; do
+ (( $(now_ms) - t0 >= 90000000000 )) && break
+ sleep 5
+ done
+ sandbox_running || fail "sandbox paused while Redis was scaled down"
+ pass "redis down: no pauses while Redis unreachable"
+ kubectl -n "$OSB_NS" scale deploy/idle-pause-redis --replicas=1 >/dev/null
+ wait_for "redis back ready" 120 \
+ kubectl -n "$OSB_NS" rollout status deploy/idle-pause-redis --timeout=10s
+ wait_for "tombstone pause after redis recovery" 300 sandbox_paused
+ pass "redis recovery: missing-key tombstone paused after the full idle window"
+
+ idle_wake_wait "post-flushall wake"
+ t0="$(now_ms)"
+ kubectl -n "$OSB_NS" exec deploy/idle-pause-redis -- redis-cli FLUSHALL >/dev/null 2>&1 \
+ || fail "FLUSHALL failed"
+ while :; do
+ if sandbox_paused; then
+ (( $(now_ms) - t0 < (x - 5) * 1000000000 )) \
+ && fail "FLUSHALL accelerated the pause ($(( ($(now_ms) - t0) / 1000000 ))ms after flush)"
+ break
+ fi
+ (( $(now_ms) - t0 >= 300000000000 )) && fail "no pause within budget after FLUSHALL"
+ sleep 2
+ done
+ pass "flushall: pause delayed to a full idle window after data loss ($(( ($(now_ms) - t0) / 1000000 ))ms)"
+
+
+ osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
+ || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
+ wait_for "idle-pause sandbox deleted" 120 verify_sandbox_gone
+}
+
+# --- standalone entry ----------------------------------------------------------------
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ # Same profile check as the entry script's idle-e2e action: the marker is
+ # left by an IDLE_PAUSE=1 up and removed by down.
+ if [[ "$IDLE_PAUSE" != "1" && ! -f "$WORK/idle-pause" ]]; then
+ die "idle-pause verify requires the idle-pause profile (IDLE_PAUSE=1 up)"
+ fi
+ resolve_osb
+ require_template_id
+ trap 'on_error idle-pause-verify' ERR
+ run_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verify
+fi
diff --git a/scripts/fast-sandbox-env/verify/pause-resume.sh b/scripts/fast-sandbox-env/verify/pause-resume.sh
new file mode 100755
index 000000000..cab25a7ee
--- /dev/null
+++ b/scripts/fast-sandbox-env/verify/pause-resume.sh
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# verify/pause-resume.sh — pause/resume round trip (CLI -> FastPath checkpoint)
+#
+# Pause until the checkpoint is durable and the signed route stops serving, resume back to Running, and /ping through a fresh route.
+# Runs as part of `integration.sh up` (the entry chains every verify
+# script) or standalone against a live stack from `env.sh up`.
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
+
+pause_resume_verify() {
+ # Accepted + poll: Paused == checkpoint durable + capacity released;
+ # resume advances the route generation, so /ping needs a fresh route.
+ local created t0 t1 t2 t3
+ log "verify (pause): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
+ t0="$(now_ms)"
+ created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
+ --metadata origin=fast-sandbox-env-pause --skip-health-check -o json 2>/dev/null)" \
+ || fail "osb sandbox create failed against $SERVER_URL"
+ VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
+ [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
+ wait_for "pause target Running" 150 sandbox_running
+ t1="$(now_ms)"
+ log "verify (pause): $VERIFY_ID create->Running $(( (t1 - t0) / 1000000 ))ms"
+
+ wait_for "pre-pause execd /ping 200 through the gateway" 100 execd_ping_ok
+
+ log "verify (pause): osb sandbox pause $VERIFY_ID"
+ osb_api sandbox pause "$VERIFY_ID" >/dev/null 2>&1 \
+ || fail "pause failed for $VERIFY_ID"
+ # Paused is durable-first: the checkpoint must be complete in the artifact
+ # store before the state is reported; the dump itself keeps serving.
+ wait_for "poll GET until Paused (checkpoint durable, capacity released)" 240 sandbox_paused
+ t2="$(now_ms)"
+ log "verify (pause): $VERIFY_ID pause->Paused (checkpoint durable) $(( (t2 - t1) / 1000000 ))ms"
+ if execd_ping_ok; then
+ fail "paused sandbox still serves /ping through the gateway"
+ fi
+ pass "paused: execd /ping no longer served (runtime released, signed route rejected at the gateway)"
+
+ log "verify (pause): osb sandbox resume $VERIFY_ID"
+ t2="$(now_ms)"
+ osb_api sandbox resume "$VERIFY_ID" --skip-health-check >/dev/null 2>&1 \
+ || fail "resume failed for $VERIFY_ID"
+ wait_for "poll GET until Running (checkpoint restored)" 240 sandbox_running
+ t3="$(now_ms)"
+ log "verify (pause): $VERIFY_ID resume->Running (checkpoint restored) $(( (t3 - t2) / 1000000 ))ms"
+ wait_for "post-resume execd /ping 200 through a fresh route" 100 execd_ping_ok
+ pass "pause/resume round-trip timings: pause->Paused $(( (t2 - t1) / 1000000 ))ms, resume->Running $(( (t3 - t2) / 1000000 ))ms (opensandbox CLI -> FastPath -> artifact store)"
+
+ osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
+ || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
+ wait_for "pause/resume sandbox deleted" 120 verify_sandbox_gone
+}
+
+# --- standalone entry ----------------------------------------------------------------
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ resolve_osb
+ require_template_id
+ trap 'on_error pause-resume' ERR
+ run_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verify
+fi
diff --git a/scripts/fast-sandbox-env/verify/sandbox.sh b/scripts/fast-sandbox-env/verify/sandbox.sh
new file mode 100755
index 000000000..5d376ffd8
--- /dev/null
+++ b/scripts/fast-sandbox-env/verify/sandbox.sh
@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# verify/sandbox.sh — end-to-end sandbox verify (CLI create -> gateway -> execd /ping)
+#
+# Cold + warm sandbox creates through the full wire-up, egress policy enforcement and update convergence, and the remaining lifecycle surface (get/list/metadata/renew).
+# Runs as part of `integration.sh up` (the entry chains every verify
+# script) or standalone against a live stack from `env.sh up`.
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
+
+verify_policy_enforced() {
+ local out
+ out="$(server_api GET "/sandboxes/$VERIFY_ID/networkpolicy" 2>/dev/null)" || return 1
+ [[ "$(printf '%s' "$out" | jq -r '.mode // empty')" == "enforcing" ]] || return 1
+ [[ "$(printf '%s' "$out" | jq -r '.policy.egress[0].target // empty')" == "example.com" ]]
+}
+
+verify_policy_updated() {
+ local out
+ out="$(server_api GET "/sandboxes/$VERIFY_ID/networkpolicy" 2>/dev/null)" || return 1
+ [[ "$(printf '%s' "$out" | jq -r '.mode // empty')" == "enforcing" ]] || return 1
+ [[ "$(printf '%s' "$out" | jq -r '.policy.egress[0].target // empty')" == "github.com" ]]
+}
+
+# Full wire-up: CLI create (fsb) -> FastPath -> fastlet -> firecracker
+# sandbox -> signed gateway route -> fastlet-proxy -> guest execd /ping ->
+# delete. One sandbox per call; availability is measured from the client
+# (poll /ping until 200, wall-clock budget — each route poll is a CLI call,
+# ~0.5s startup), not from CR status.
+verify_one_sandbox() { # <label> <ping-budget-ms>
+ local label="$1" budget="$2" created t0 t1 t2 route code deadline
+ local policy_file="$GEN_DIR/verify-policy.json"
+ t0="$(now_ms)"
+ # Egress policy rides on every create (binding -> nft rules), so the
+ # chain is exercised each time; /ping is inbound and unaffected.
+ jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "example.com"}, {action: "allow", target: "*.opensandbox.ai"}]}' \
+ > "$policy_file"
+ log "verify ($label): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
+ created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
+ --metadata origin=fast-sandbox-env-verify \
+ --network-policy-file "$policy_file" \
+ --skip-health-check -o json 2>/dev/null)" \
+ || fail "osb sandbox create failed against $SERVER_URL: $(osb_api sandbox create \
+ --template "$TEMPLATE_ID" --timeout 1h --skip-health-check 2>&1 >/dev/null | head -c 400)"
+ VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
+ [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "create response carried no id"
+ t1="$(now_ms)"
+ # Availability = /ping 200 through the signed gateway route (port 44772).
+ deadline=$(( t1 + budget * 1000000 ))
+ while :; do
+ route="$(osb_api sandbox endpoint "$VERIFY_ID" --port 44772 -o json 2>/dev/null \
+ | jq -r '.headers["OpenSandbox-Ingress-To"] // empty')"
+ if [[ -n "$route" ]]; then
+ code="$(curl -sS -m 5 -o /dev/null -w '%{http_code}' \
+ -H "OpenSandbox-Ingress-To: $route" "$GATEWAY_URL/ping" 2>/dev/null || true)"
+ [[ "$code" == "200" ]] && break
+ fi
+ if (( $(now_ms) >= deadline )); then
+ fail "execd /ping did not return 200 within ${budget}ms (last code=${code:-none}, route=${route:-none})"
+ fi
+ sleep 0.2
+ done
+ t2="$(now_ms)"
+ # jq filter in a variable: literal parens trip older bash's $( ) parser.
+ local state raw raw_filter
+ raw_filter='{rt:.status.runtime.state,dp:.status.dataPlane.state,infra:[.status.infraComponents[]?|{n:.name,s:.state}],bind:[.status.actionBindings[]?|{h:.handler,s:.state}],ready:(.status.conditions[]?|select(.type=="Ready")|.status)}'
+ state="$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')"
+ raw="$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" -o json 2>/dev/null | jq -c "$raw_filter")"
+ log "verify ($label): $VERIFY_ID access via ingress gateway: curl -H \"OpenSandbox-Ingress-To: $route\" $GATEWAY_URL/ping"
+ log "verify ($label): $VERIFY_ID create $(( (t1 - t0) / 1000000 ))ms, create->execd /ping 200 $(( (t2 - t1) / 1000000 ))ms, total $(( (t2 - t0) / 1000000 ))ms"
+ log "verify ($label): CR at ping: server=$state raw=${raw:-unreachable}"
+ pass "execd /ping 200 through the signed gateway route (44772)"
+ wait_for "egress policy enforcing (networkPolicy -> egress action binding -> nft)" 120 verify_policy_enforced
+ # Policy UPDATE path (raw PUT: the CLI only has merge-style patches).
+ local put_body
+ put_body="$(jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "github.com"}]}')"
+ server_api PUT "/sandboxes/$VERIFY_ID/networkpolicy" "$put_body" >/dev/null \
+ || fail "PUT networkpolicy failed for $VERIFY_ID: $(printf '%s' "$put_body" | head -c 200)"
+ wait_for "policy update converged (PUT -> ReplaceActionBindings -> egress)" 120 verify_policy_updated
+ # Re-sample the CR state: an early Failed observation may converge late.
+ local state_final
+ state_final="$(osb_api sandbox get "$VERIFY_ID" -o json 2>/dev/null | jq -r '.status.state // empty')"
+ log "verify ($label): $VERIFY_ID CR state final: ${state_final:-unknown}"
+}
+
+# Remaining lifecycle surface on one sandbox: get, list, metadata merge-patch
+# (raw API — no CLI command), renew via the CLI (TTL-based).
+verify_lifecycle_ops() { # <sandbox-id>
+ local id="$1" out expected_expires renewed_expires
+ out="$(osb_api sandbox get "$id" -o json 2>/dev/null)" || fail "osb sandbox get $id failed"
+ [[ "$(printf '%s' "$out" | jq -r '.id')" == "$id" ]] || fail "GET returned wrong id: $out"
+ pass "lifecycle: osb sandbox get"
+
+ osb_api sandbox list --page 1 --page-size 50 -o json > "$WORK/last-list.json" 2>"$WORK/last-list.err" \
+ || fail "osb sandbox list failed: $(head -c 400 "$WORK/last-list.err" 2>/dev/null)"
+ out="$(cat "$WORK/last-list.json")"
+ [[ "$(printf '%s' "$out" | jq -r --arg id "$id" '.items[]?.id | select(. == $id)' | head -1)" == "$id" ]] \
+ || fail "list does not contain $id: $(printf '%s' "$out" | jq -c '.pagination')"
+ pass "lifecycle: osb sandbox list (contains the verify sandbox)"
+
+ # JSON Merge Patch (RFC 7396): non-null upserts, null deletes.
+ server_api PATCH "/sandboxes/$id/metadata" '{"env":"verify","stage":"lifecycle-ops"}' >/dev/null \
+ || fail "PATCH metadata upsert failed"
+ out="$(osb_api sandbox get "$id" -o json 2>/dev/null)"
+ [[ "$(printf '%s' "$out" | jq -r '.metadata.env')" == "verify" \
+ && "$(printf '%s' "$out" | jq -r '.metadata.stage')" == "lifecycle-ops" ]] \
+ || fail "metadata upsert not visible: $(printf '%s' "$out" | jq -c '.metadata')"
+ server_api PATCH "/sandboxes/$id/metadata" '{"stage":null}' >/dev/null \
+ || fail "PATCH metadata delete failed"
+ out="$(osb_api sandbox get "$id" -o json 2>/dev/null)"
+ [[ "$(printf '%s' "$out" | jq -r '.metadata.stage')" == "null" \
+ && "$(printf '%s' "$out" | jq -r '.metadata.env')" == "verify" ]] \
+ || fail "metadata delete not visible: $(printf '%s' "$out" | jq -c '.metadata')"
+ pass "lifecycle: PATCH metadata (upsert + null-delete via JSON Merge Patch)"
+
+ # CLI renew takes a TTL: assert the expiration advanced and is visible.
+ expected_expires="$(osb_api sandbox get "$id" -o json 2>/dev/null | jq -r '.expires_at')"
+ out="$(osb_api sandbox renew "$id" --timeout 2h -o json 2>"$WORK/last-renew.err")" \
+ || fail "osb sandbox renew failed: $(head -c 400 "$WORK/last-renew.err" 2>/dev/null)"
+ renewed_expires="$(printf '%s' "$out" | jq -r '.expires_at')"
+ [[ -n "$renewed_expires" && "$renewed_expires" != "null" && "$renewed_expires" != "$expected_expires" ]] \
+ || fail "renewed expiresAt did not advance: $expected_expires -> $renewed_expires"
+ local got_expires
+ got_expires="$(osb_api sandbox get "$id" -o json 2>/dev/null | jq -r '.expires_at')"
+ # Same instant, two renderings (renew panel str() vs RFC3339 dump):
+ # compare epoch seconds, falling back to string equality.
+ if [[ "$got_expires" != "$renewed_expires" ]] \
+ && [[ "$(date -u -d "$renewed_expires" +%s 2>/dev/null || true)" \
+ != "$(date -u -d "$got_expires" +%s 2>/dev/null || true)" ]]; then
+ fail "renewed expiresAt not visible: expected $renewed_expires got $got_expires"
+ fi
+ pass "lifecycle: osb sandbox renew 2h ($expected_expires -> $renewed_expires)"
+}
+
+opensandbox_verify() {
+ # Cold create pulls the golden image through DART (slowest; generous
+ # budget). Warm creates: the second may still pull on the other node via
+ # its DART peer; after both nodes cache the set, creates are sub-second.
+ local ids=() id label index
+ verify_one_sandbox "cold #1" 600000
+ ids+=("$VERIFY_ID")
+ verify_one_sandbox "warm #2" 600000
+ ids+=("$VERIFY_ID")
+ for index in 3 4 5 6; do
+ verify_one_sandbox "warm #$index" 120000
+ ids+=("$VERIFY_ID")
+ done
+ pass "end-to-end: opensandbox CLI -> server -> FastPath -> fastlet -> sandbox execd -> gateway route OK (6 sandboxes)"
+ verify_lifecycle_ops "${ids[0]}"
+ for id in "${ids[@]}"; do
+ VERIFY_ID="$id"
+ osb_api sandbox kill "$id" >/dev/null 2>&1 \
+ || log "verify cleanup: kill failed; remove $id manually"
+ done
+ for id in "${ids[@]}"; do
+ VERIFY_ID="$id"
+ wait_for "verify sandbox $id deleted" 120 verify_sandbox_gone
+ done
+ pass "verify sandboxes cleaned up"
+}
+
+# --- standalone entry ----------------------------------------------------------------
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ resolve_osb
+ require_template_id
+ trap 'on_error verify-sandbox' ERR
+ run_stage "end-to-end verify (CLI create -> gateway -> execd /ping)" opensandbox_verify
+fi
diff --git a/scripts/fast-sandbox-env/verify/snapshot.sh b/scripts/fast-sandbox-env/verify/snapshot.sh
new file mode 100755
index 000000000..33ea00932
--- /dev/null
+++ b/scripts/fast-sandbox-env/verify/snapshot.sh
@@ -0,0 +1,188 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# verify/snapshot.sh — snapshot round trip (CLI -> SandboxSnapshot CR -> restore)
+#
+# Snapshot a Running sandbox to Ready (re-entry fencing included), keep the source serving, restore a new sandbox from the snapshot id, list, and clean up.
+# Runs as part of `integration.sh up` (the entry chains every verify
+# script) or standalone against a live stack from `env.sh up`.
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
+
+# The snapshot verify stage shares the created snapshot ids through
+# SNAPSHOT_ID / SNAPSHOT_ID2.
+SNAPSHOT_ID=""
+SNAPSHOT_ID2=""
+# A re-entry snapshot accepted during fence cache lag (202) is tracked here
+# so its terminal outcome is asserted and it is cleaned up.
+SNAPSHOT_EXTRA=""
+
+# --- stage: snapshot (opensandbox CLI -> SandboxSnapshot CR -> restore) ----------
+
+# Ready once the server watcher converges the row from the SandboxSnapshot
+# CR (Succeeded + template index published).
+snapshot_ready() { # <snapshot-id>
+ [[ "$(osb_api snapshot get "$1" -o json 2>/dev/null | jq -r '.status.state // empty')" == "Ready" ]]
+}
+
+# Terminal (Ready or Failed): the pause-window fence resolves any
+# accepted-but-conflicting snapshot one way or the other.
+snapshot_terminal() { # <snapshot-id>
+ local state
+ state="$(osb_api snapshot get "$1" -o json 2>/dev/null | jq -r '.status.state // empty')"
+ [[ "$state" == "Ready" || "$state" == "Failed" ]]
+}
+
+snapshot_verify() {
+ # Full round trip on the live stack: snapshot a Running sandbox, watch it
+ # to Ready, restore a NEW sandbox from the snapshotId and boot it. Also
+ # covers re-entry fencing, source-sandbox survival, and a repeat snapshot.
+ local created out source_id snapshot_id snapshot_id2 restore_id reentry_out reentry_code
+ local t0 t1 t2 t3 t4
+ t0="$(now_ms)"
+ log "verify (snapshot): creating the source sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
+ created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
+ --metadata origin=fast-sandbox-env-snapshot --skip-health-check -o json 2>/dev/null)" \
+ || fail "osb sandbox create failed against $SERVER_URL"
+ source_id="$(printf '%s' "$created" | jq -r '.id')"
+ [[ -n "$source_id" && "$source_id" != "null" ]] || fail "create response carried no id"
+ VERIFY_ID="$source_id"
+ wait_for "snapshot source sandbox Running" 300 sandbox_running
+ wait_for "pre-snapshot execd /ping 200 through the gateway" 100 execd_ping_ok
+ t1="$(now_ms)"
+ log "verify (snapshot): source sandbox $source_id create->Running $(( (t1 - t0) / 1000000 ))ms"
+
+ # 1. Snapshot create returns Creating; the row converges from the
+ # SandboxSnapshot CR via the server watcher.
+ log "verify (snapshot): osb snapshot create $source_id --name env-verify"
+ t1="$(now_ms)"
+ out="$(osb_api snapshot create "$source_id" --name env-verify -o json 2>/dev/null)" \
+ || fail "snapshot create failed for $source_id"
+ SNAPSHOT_ID="$(printf '%s' "$out" | jq -r '.id')"
+ [[ -n "$SNAPSHOT_ID" && "$SNAPSHOT_ID" != "null" ]] || fail "snapshot create carried no id"
+ [[ "$(printf '%s' "$out" | jq -r '.status')" == "Creating" ]] \
+ || fail "snapshot create did not return Creating: $(printf '%s' "$out" | head -c 300)"
+
+ # 2. Re-entry during the dump window: FastPath fences with 409 once the
+ # CR is cache-visible; with cache lag it is accepted (202) and the
+ # pause window resolves it to a terminal phase later.
+ reentry_out="$(curl -sS -m 60 -w '\n%{http_code}' -X POST \
+ -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" -H "Content-Type: application/json" \
+ -d '{"name":"env-verify-reentry"}' "$SERVER_URL/sandboxes/$source_id/snapshots" 2>/dev/null || true)"
+ reentry_code="$(printf '%s' "$reentry_out" | tail -n1)"
+ reentry_out="$(printf '%s' "$reentry_out" | sed '$d')"
+ case "$reentry_code" in
+ 409)
+ pass "snapshot: re-entry rejected by the fence (409)"
+ ;;
+ 202)
+ SNAPSHOT_EXTRA="$(printf '%s' "$reentry_out" | jq -r '.id' 2>/dev/null || true)"
+ [[ -n "$SNAPSHOT_EXTRA" && "$SNAPSHOT_EXTRA" != "null" ]] \
+ || fail "re-entry snapshot POST returned 202 without an id: $(printf '%s' "$reentry_out" | head -c 300)"
+ log "verify (snapshot): re-entry accepted during fence cache lag ($SNAPSHOT_EXTRA); terminal outcome asserted below"
+ ;;
+ *)
+ fail "re-entry snapshot POST returned unexpected HTTP ${reentry_code:-none}: $(printf '%s' "$reentry_out" | head -c 300)"
+ ;;
+ esac
+
+ wait_for "poll osb snapshot get $SNAPSHOT_ID until Ready (watcher -> SandboxSnapshot CR -> store index)" 300 snapshot_ready "$SNAPSHOT_ID"
+ t2="$(now_ms)"
+ log "verify (snapshot): $SNAPSHOT_ID snapshot create->Ready $(( (t2 - t1) / 1000000 ))ms"
+ pass "snapshot: create -> Creating -> watcher -> Ready"
+
+ # 3. Source survival: Running + /ping again after the terminal phase.
+ VERIFY_ID="$source_id"
+ wait_for "source sandbox Running again after the snapshot" 120 sandbox_running
+ wait_for "source sandbox execd /ping 200 after the snapshot" 100 execd_ping_ok
+ pass "snapshot: source sandbox survived (Running + /ping 200)"
+
+ # 3b. An accepted re-entry snapshot must reach a terminal phase; a stuck
+ # one would block all future snapshots via the re-entry fence.
+ if [[ -n "$SNAPSHOT_EXTRA" ]]; then
+ wait_for "re-entry snapshot $SNAPSHOT_EXTRA reaches a terminal phase" 150 snapshot_terminal "$SNAPSHOT_EXTRA"
+ log "verify (snapshot): re-entry snapshot $SNAPSHOT_EXTRA terminal: $(osb_api snapshot get "$SNAPSHOT_EXTRA" -o json 2>/dev/null | jq -r '.status.state')"
+ pass "snapshot: accepted re-entry snapshot resolved to a terminal phase"
+ fi
+
+ # 4. Second snapshot once the first is terminal: fresh id.
+ t3="$(now_ms)"
+ out="$(osb_api snapshot create "$source_id" --name env-verify-2 -o json 2>/dev/null)" \
+ || fail "second snapshot create failed for $source_id"
+ snapshot_id2="$(printf '%s' "$out" | jq -r '.id')"
+ [[ -n "$snapshot_id2" && "$snapshot_id2" != "null" && "$snapshot_id2" != "$SNAPSHOT_ID" ]] \
+ || fail "second snapshot did not produce a distinct id: $snapshot_id2"
+ SNAPSHOT_ID2="$snapshot_id2"
+ wait_for "poll osb snapshot get $snapshot_id2 until Ready" 300 snapshot_ready "$snapshot_id2"
+ t4="$(now_ms)"
+ log "verify (snapshot): $snapshot_id2 second snapshot create->Ready $(( (t4 - t3) / 1000000 ))ms"
+ pass "snapshot: repeated snapshot of the same sandbox -> Ready (distinct id)"
+
+ # 5. Listing: sandboxId scoping returns both required snapshots, both
+ # Ready (an accepted re-entry snapshot may add a third, terminal row).
+ out="$(osb_api snapshot list --sandbox-id "$source_id" --page-size 50 -o json 2>"$WORK/last-snapshot-list.err")" \
+ || fail "snapshot list failed for $source_id: $(head -c 400 "$WORK/last-snapshot-list.err" 2>/dev/null)"
+ [[ "$(printf '%s' "$out" | jq -r --arg id "$SNAPSHOT_ID" --arg id2 "$snapshot_id2" \
+ '[.items[] | select((.id == $id or .id == $id2) and .status.state == "Ready")] | length')" == "2" ]] \
+ || fail "snapshot list does not contain both required snapshots as Ready: $(printf '%s' "$out" | head -c 400)"
+ if [[ -n "$SNAPSHOT_EXTRA" ]]; then
+ [[ "$(printf '%s' "$out" | jq -r --arg id "$SNAPSHOT_EXTRA" \
+ '[.items[] | select(.id == $id)] | length')" == "1" ]] \
+ || fail "accepted re-entry snapshot $SNAPSHOT_EXTRA missing from the list"
+ fi
+ pass "snapshot: list scoped by sandboxId contains the snapshots (Ready)"
+
+ # 6. Restore: the row resolves to the published template index key; the
+ # resource flags restate the pool profile (firecracker pool).
+ local restored
+ log "verify (snapshot): osb sandbox create --snapshot-id $SNAPSHOT_ID (pool profile restated)"
+ t3="$(now_ms)"
+ restored="$(osb_api sandbox create --snapshot-id "$SNAPSHOT_ID" --timeout 1h \
+ --resource cpu=1 --resource memory=512Mi --resource pids=128 \
+ --skip-health-check -o json 2>/dev/null)" \
+ || fail "osb sandbox create (snapshotId=$SNAPSHOT_ID) failed"
+ restore_id="$(printf '%s' "$restored" | jq -r '.id')"
+ [[ -n "$restore_id" && "$restore_id" != "null" ]] || fail "restore create carried no id"
+ VERIFY_ID="$restore_id"
+ wait_for "restored sandbox Running" 600 sandbox_running
+ t4="$(now_ms)"
+ wait_for "restored execd /ping 200 through the gateway" 300 execd_ping_ok
+ log "verify (snapshot): restored $restore_id create->Running $(( (t4 - t3) / 1000000 ))ms"
+ pass "snapshot: restore -> sandbox boots the published artifact set -> execd /ping OK"
+
+ # 7. Cleanup; the server forwards artifact deletion per snapshot row.
+ osb_api sandbox kill "$restore_id" >/dev/null 2>&1 \
+ || log "verify cleanup: kill $restore_id failed"
+ wait_for "restored sandbox deleted" 120 verify_sandbox_gone
+ VERIFY_ID="$source_id"
+ osb_api sandbox kill "$source_id" >/dev/null 2>&1 \
+ || log "verify cleanup: kill $source_id failed"
+ wait_for "snapshot source sandbox deleted" 120 verify_sandbox_gone
+ for snapshot_id in "$SNAPSHOT_ID" "$SNAPSHOT_ID2" "$SNAPSHOT_EXTRA"; do
+ [[ -n "$snapshot_id" ]] || continue
+ osb_api snapshot delete "$snapshot_id" >/dev/null 2>&1 \
+ || log "verify cleanup: snapshot delete $snapshot_id failed"
+ done
+ pass "snapshot: cleanup (restored + source sandboxes, both snapshot rows)"
+}
+
+# --- standalone entry ----------------------------------------------------------------
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ resolve_osb
+ require_template_id
+ trap 'on_error snapshot-verify' ERR
+ run_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify
+fi
diff --git a/scripts/fast-sandbox-env/verify/vault.sh b/scripts/fast-sandbox-env/verify/vault.sh
new file mode 100755
index 000000000..c827a5860
--- /dev/null
+++ b/scripts/fast-sandbox-env/verify/vault.sh
@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+# Copyright 2026 The OpenSandbox Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# verify/vault.sh — credential-vault lifecycle over the server relay (OSEP-0022)
+#
+# Vault create (create-only 409), export into the checkpoint bundle on pause, reload + revision preservation across resume and an egress crash, and delete/re-create.
+# Runs as part of `integration.sh up` (the entry chains every verify
+# script) or standalone against a live stack from `env.sh up`.
+
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
+
+# --- stage: vault lifecycle (OSEP-0022: egress-held vault over the relay) --------
+
+# Status-code-returning variant of server_api for negative-path assertions
+# (curl -f hides the status).
+server_api_status() { # method path [json-body]
+ local method="$1" path="$2" body="${3:-}" out
+ if [[ -n "$body" ]]; then
+ out="$(curl -sS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
+ -H "Content-Type: application/json" -d "$body" -w '\n%{http_code}' "$SERVER_URL$path" 2>/dev/null)"
+ else
+ out="$(curl -sS -m 60 -X "$method" -H "OPEN-SANDBOX-API-KEY: $SERVER_API_KEY" \
+ -w '\n%{http_code}' "$SERVER_URL$path" 2>/dev/null)"
+ fi
+ printf '%s' "${out##*$'\n'}"
+}
+
+# The vault stage's sandbox must ride the subject-scoped egress log markers
+# (export/import/registration), so every log assertion goes through this.
+vault_log_contains() { # <fixed-substring>
+ kubectl -n "$RESOURCE_NS" logs -l app=sandbox-fastlet -c egress --tail=800 2>/dev/null \
+ | grep -qF "$1"
+}
+
+vault_revision_is() { # <expected-revision>
+ local out
+ out="$(server_api GET "/sandboxes/$VERIFY_ID/credential-vault" 2>/dev/null)" || return 1
+ [[ "$(printf '%s' "$out" | jq -r '.revision // empty')" == "$1" ]]
+}
+
+vault_lifecycle_verify() {
+ local created vault_body ack rev sandbox_uid
+ log "verify (vault): creating a sandbox via the opensandbox CLI (templateId=$TEMPLATE_ID)"
+ # The policy rides the create: a policy-less create produces no egress
+ # binding at all, so the subject would never register.
+ jq -n '{defaultAction: "deny", egress: [{action: "allow", target: "ext.test"}]}' \
+ > "$GEN_DIR/vault-policy.json"
+ created="$(osb_api sandbox create --template "$TEMPLATE_ID" --timeout 1h \
+ --metadata origin=fast-sandbox-env-vault \
+ --network-policy-file "$GEN_DIR/vault-policy.json" \
+ --skip-health-check -o json 2>/dev/null)" \
+ || fail "vault verify: sandbox create failed"
+ VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"
+ wait_for "vault sandbox Running" 150 sandbox_running
+ sandbox_uid="$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" -o jsonpath='{.metadata.uid}' 2>/dev/null)"
+ [[ -n "$sandbox_uid" ]] || fail "vault verify: sandbox CR carried no uid"
+ # Registered and activated before the vault POST, else the relay 404s.
+ # (The "(state=active)" re-delivery only follows a policy UPDATE; a
+ # create-carried policy activates without it.)
+ wait_for "vault subject registered on egress" 120 \
+ vault_log_contains "subject s-$sandbox_uid: data-plane-ready, policy active"
+
+ # 1) create-only: ack carries the revision; duplicate POST is 409.
+ vault_body="$(jq -n '{credentials: [{name: "tok", source: {type: "inline", value: "sekrit-vault-e2e"}}], bindings: [{name: "ext", match: {hosts: ["ext.test"]}, auth: {type: "apiKey", name: "X-Api-Key", credential: "tok"}}]}')"
+ ack="$(server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" \
+ || fail "vault verify: POST credential-vault failed"
+ rev="$(printf '%s' "$ack" | jq -r '.revision // empty')"
+ [[ "$rev" =~ ^[0-9]+$ ]] || fail "vault verify: create ack carried no revision: $(printf '%s' "$ack" | head -c 200)"
+ pass "vault created over the relay (ack revision $rev)"
+ if server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body" >/dev/null 2>&1; then
+ fail "vault verify: duplicate POST succeeded (create-only must 409)"
+ fi
+ [[ "$(server_api_status POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" == "409" ]] \
+ || fail "vault verify: duplicate POST was not 409"
+ pass "duplicate POST rejected (create-only 409)"
+
+ # 2) pause/resume carries the real vault: export -> bundle -> import,
+ # with the relay GET reporting the same revision.
+ osb_api sandbox pause "$VERIFY_ID" >/dev/null 2>&1 || fail "vault verify: pause failed"
+ wait_for "vault sandbox Paused (checkpoint durable)" 240 sandbox_paused
+ wait_for "vault exported into the checkpoint bundle" 120 \
+ vault_log_contains "subject s-$sandbox_uid: vault exported to snapshot bundle"
+ pass "pause exported the vault bundle (subject sealed)"
+ osb_api sandbox resume "$VERIFY_ID" --skip-health-check >/dev/null 2>&1 || fail "vault verify: resume failed"
+ wait_for "vault sandbox Running (restored)" 240 sandbox_running
+ # The pause's REMOVE_BINDING purges the subject, so the restore import
+ # lands through the unregistered branch and the file reloads at the
+ # SET_BINDING replay (the reload line carries the restored revision).
+ wait_for "vault bundle reloaded at re-registration" 120 \
+ vault_log_contains "subject s-$sandbox_uid: vault reloaded from file"
+ wait_for "vault revision preserved across pause/resume" 120 vault_revision_is "$rev"
+ pass "vault survived pause/resume (revision $rev preserved)"
+
+ # 3) egress-restart recovery: crash the worker (fresh instanceId ->
+ # SET_BINDING replay -> vault file reload by incarnation).
+ local pod restarted=0
+ for pod in $(pool_pods); do
+ if kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- sh -c 'command -v pkill >/dev/null' 2>/dev/null; then
+ kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- pkill -f '/opt/opensandbox-egress/egress' >/dev/null 2>&1 || true
+ restarted=1
+ else
+ log "verify (vault): egress restart skipped on ${pod##*/} (pkill unavailable)"
+ fi
+ done
+ if [[ "$restarted" == 1 ]]; then
+ wait_for "pool egress containers back up" 180 egress_containers_ready
+ wait_for "vault reloaded after egress restart" 240 vault_revision_is "$rev"
+ pass "vault reloaded after egress restart (revision $rev preserved)"
+ fi
+
+ # 4) delete + re-create.
+ server_api DELETE "/sandboxes/$VERIFY_ID/credential-vault" >/dev/null \
+ || fail "vault verify: DELETE failed"
+ [[ "$(server_api_status GET "/sandboxes/$VERIFY_ID/credential-vault")" == "404" ]] \
+ || fail "vault verify: GET after DELETE was not 404"
+ pass "vault deleted (GET 404)"
+ ack="$(server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" \
+ || fail "vault verify: re-create after DELETE failed"
+ rev="$(printf '%s' "$ack" | jq -r '.revision // empty')"
+ [[ "$rev" == "1" ]] || fail "vault verify: fresh vault revision is $rev, expected 1"
+ pass "vault re-created (fresh revision $rev)"
+
+ osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \
+ || log "verify cleanup: kill failed; remove $VERIFY_ID manually"
+ wait_for "vault sandbox deleted" 120 verify_sandbox_gone
+}
+
+# --- standalone entry ----------------------------------------------------------------
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+ resolve_osb
+ require_template_id
+ trap 'on_error vault-verify' ERR
+ run_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify
+fi
diff --git a/scripts/tests/test_fast_sandbox_integration_env.py b/scripts/tests/test_fast_sandbox_integration_env.py
index b110c9664..d2abda6dd 100644
--- a/scripts/tests/test_fast_sandbox_integration_env.py
+++ b/scripts/tests/test_fast_sandbox_integration_env.py
@@ -21,7 +21,7 @@ import tempfile
import unittest
-SCRIPT = Path(__file__).resolve().parents[1] / "fast-sandbox-env/integration-env.sh"
+SCRIPT = Path(__file__).resolve().parents[1] / "fast-sandbox-env/integration.sh"
SETUP = r'''
source "$SCRIPT"
mkdir -p "$LOGS_DIR" "$GEN_DIR"
diff --git a/tests/python/README.md b/tests/python/README.md
index 9c6b553a5..afe03d191 100644
--- a/tests/python/README.md
+++ b/tests/python/README.md
@@ -34,10 +34,10 @@ Credential Vault E2E tests require a reachable target service and `OPENSANDBOX_C
### Fast-sandbox (fsb) integration env
-`tests/test_fsb_e2e.py` converts the HTTP verify stages of `scripts/fast-sandbox-env/integration-env.sh` (template create → gateway ping → networkpolicy PATCH/DELETE convergence, lifecycle ops, pause/resume, public snapshot round trip) into Python SDK calls. It is skipped unless the fsb stack is up and `OPENSANDBOX_TEST_FSB_TEMPLATE_ID` points at the golden-image template the env script builds, and it is excluded from the default `make test` run:
+`tests/test_fsb_e2e.py` converts the HTTP verify stages of `scripts/fast-sandbox-env/integration.sh` (template create → gateway ping → networkpolicy PATCH/DELETE convergence, lifecycle ops, pause/resume, public snapshot round trip) into Python SDK calls. It is skipped unless the fsb stack is up and `OPENSANDBOX_TEST_FSB_TEMPLATE_ID` points at the golden-image template the env script builds, and it is excluded from the default `make test` run:
```bash
-./scripts/fast-sandbox-env/integration-env.sh up # brings up the stack, prints/keeps the template id
+./scripts/fast-sandbox-env/integration.sh up # brings up the stack, prints/keeps the template id
cd tests/python
OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$(cat "$WORK/template-id")" make test-fsb
```
diff --git a/tests/python/tests/test_fsb_e2e.py b/tests/python/tests/test_fsb_e2e.py
index 2fa0d7673..9f610772f 100644
--- a/tests/python/tests/test_fsb_e2e.py
+++ b/tests/python/tests/test_fsb_e2e.py
@@ -16,7 +16,7 @@
"""E2E coverage for the fast-sandbox (fsb) integration environment.
Mirrors the HTTP verify stages of
-``scripts/fast-sandbox-env/integration-env.sh`` using the Python SDK
+``scripts/fast-sandbox-env/integration.sh`` using the Python SDK
instead of raw curl: template catalog visibility, template-based sandbox
creation through the signed gateway route, networkpolicy convergence
(PATCH merge + DELETE), lifecycle ops (get/list/metadata/renew),
@@ -27,7 +27,7 @@ Requires the fsb integration stack to be up plus
``OPENSANDBOX_TEST_FSB_TEMPLATE_ID`` (the golden-image template built by
the env script's ``template_up`` stage):
- ./scripts/fast-sandbox-env/integration-env.sh up
+ ./scripts/fast-sandbox-env/integration.sh up
cd tests/python
OPENSANDBOX_TEST_FSB_TEMPLATE_ID="$(cat "$WORK/template-id")" \
uv run pytest tests/test_fsb_e2e.py
diff --git a/tests/python/tests/test_fsb_idle_pause_e2e.py b/tests/python/tests/test_fsb_idle_pause_e2e.py
index d4155d4f1..3ccd4e4cb 100644
--- a/tests/python/tests/test_fsb_idle_pause_e2e.py
+++ b/tests/python/tests/test_fsb_idle_pause_e2e.py
@@ -15,8 +15,8 @@
#
"""OSEP-0024 idle-pause / wake-on-access E2E (fsb integration env).
-Requires ``IDLE_PAUSE=1 integration-env.sh up`` plus
-``OPENSANDBOX_TEST_FSB_TEMPLATE_ID``; ``integration-env.sh idle-e2e``
+Requires ``IDLE_PAUSE=1 integration.sh up`` plus
+``OPENSANDBOX_TEST_FSB_TEMPLATE_ID``; ``integration.sh idle-e2e``
exports both. Skipped otherwise and excluded from ``make test``.
"""