Skip to content

Secure, Fast, and Extensible Sandbox runtime for AI agents.

ci(fast-sandbox): split the template verify matrix into per-stack steps

高然01ac4c8f6fda
diff --git a/.github/workflows/fast-sandbox-test.yml b/.github/workflows/fast-sandbox-test.yml
index f8ad406ad..3589b4ad5 100644
--- a/.github/workflows/fast-sandbox-test.yml
+++ b/.github/workflows/fast-sandbox-test.yml
@@ -34,7 +34,6 @@ jobs:
needs: changes
if: needs.changes.outputs.relevant == 'true'
runs-on: kvm
- # The verify matrix runs twice when OCI_STACK=1 (native + overlaybd).
timeout-minutes: 240
env:
# Full-stack profile (single round while scenarios stabilize).
@@ -77,9 +76,18 @@ jobs:
sed -i "s/^commit:.*/commit: $FSB_COMMIT_OVERRIDE/" manifests/third-party/fast-sandbox.commit
manifests/release/build-fast-sandbox.sh --no-build --sync-crds
- - name: Run integration tests
+ - name: Run integration tests (environment init)
run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh up --auto-clean
+ - name: Run native template verify
+ timeout-minutes: 75
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh verify-native
+
+ - name: Run overlaybd template verify
+ # Builds the overlaybd template first; no-op when OCI_STACK=0.
+ timeout-minutes: 90
+ run: sudo -E env "PATH=$PATH" ./scripts/fast-sandbox-env/integration-env.sh verify-overlaybd
+
- name: Run base SDK e2e
# Runs once per template (native + overlaybd).
timeout-minutes: 60
diff --git a/scripts/fast-sandbox-env/README.md b/scripts/fast-sandbox-env/README.md
index d63c7d16b..15e3f6e06 100644
--- a/scripts/fast-sandbox-env/README.md
+++ b/scripts/fast-sandbox-env/README.md
@@ -1,22 +1,24 @@
# fast-sandbox integration environment
-One-command OpenSandbox ecosystem integration environment driven from the OpenSandbox repository: a two-node kind cluster (KVM passthrough) running the fast-sandbox Firecracker chain from the source pinned in `manifests/third-party/fast-sandbox.commit`, with the source-built OpenSandbox **egress** sidecar attached to the default pool and the source-built OpenSandbox **server** (fsb runtime) and **ingress gateway** wired on top — verified end to end on every `up`.
+One-command OpenSandbox ecosystem integration environment driven from the OpenSandbox repository: a two-node kind cluster (KVM passthrough) running the fast-sandbox Firecracker chain from the source pinned in `manifests/third-party/fast-sandbox.commit`, with the source-built OpenSandbox **egress** sidecar attached to the default pool and the source-built OpenSandbox **server** (fsb runtime) and **ingress gateway** wired on top — verified end to end by the `verify-native` / `verify-overlaybd` actions.
All Kubernetes resources ship from the OpenSandbox Helm charts (`manifests/charts`): `base` (CRDs + RBAC), `fast-sandbox` (control plane, janitor, node installer, runtime-agent + DART), `server` and `ingress-gateway`. The only env-owned manifests left are the kind cluster config and the SandboxPool resource.
-The last `up` stages drive the server through the opensandbox CLI (`osb`, installed fresh from PyPI into the workspace unless `OSB_BIN` is set): create a sandbox from the golden-image template, reach the in-sandbox execd `/ping` through the signed gateway route, and delete it again. Raw HTTP calls remain only where the CLI has no equivalent (networkpolicy replace, metadata merge-patch, snapshot re-entry fence). The environment is all-or-nothing: running fast-sandbox without the OpenSandbox layers is not a supported shape of this script.
+The `verify-native` / `verify-overlaybd` actions drive the server through the opensandbox CLI (`osb`, installed fresh from PyPI into the workspace by `up` unless `OSB_BIN` is set): create a sandbox from the golden-image template, reach the in-sandbox execd `/ping` through the signed gateway route, and delete it again. Raw HTTP calls remain only where the CLI has no equivalent (networkpolicy replace, metadata merge-patch, snapshot re-entry fence). The environment is all-or-nothing: running fast-sandbox without the OpenSandbox layers is not a supported shape of this script.
Requires a bare-metal Linux host with KVM (`/dev/kvm`), Docker, Go ≥ 1.25, cgroup v2, and `sudo` for the XFS StateRoot loop mount and sysctl bump. The CLI install requires `uv` (it manages its own ≥ 3.10 interpreter), or set `OSB_BIN` to a preinstalled `osb` binary. Full operational notes (topology, latency characteristics, production caveats): `fast-sandbox/docs/guides/firecracker-integration-env.md`.
## Usage
```bash
-./scripts/fast-sandbox-env/integration-env.sh up # full stack + end-to-end + pause/resume verify
-./scripts/fast-sandbox-env/integration-env.sh status # component / pool / DART / OpenSandbox health
-./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
-./scripts/fast-sandbox-env/integration-env.sh sdk-e2e # Python SDK e2e suite against the live stack
-./scripts/fast-sandbox-env/integration-env.sh idle-e2e # OSEP-0024 idle-pause/wake suite (needs IDLE_PAUSE=1 up)
-./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
+./scripts/fast-sandbox-env/integration-env.sh up # environment: stack + pool + server/ingress + native template
+./scripts/fast-sandbox-env/integration-env.sh verify-native # verify matrix on the native template (needs up)
+./scripts/fast-sandbox-env/integration-env.sh verify-overlaybd # overlaybd template build + verify matrix (needs up)
+./scripts/fast-sandbox-env/integration-env.sh status # component / pool / DART / OpenSandbox health
+./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
+./scripts/fast-sandbox-env/integration-env.sh sdk-e2e # Python SDK e2e suite against the live stack
+./scripts/fast-sandbox-env/integration-env.sh idle-e2e # OSEP-0024 idle-pause/wake suite (needs IDLE_PAUSE=1 up)
+./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
```
### Idle-pause / wake-on-access profile (OSEP-0024)
@@ -25,11 +27,11 @@ Requires a bare-metal Linux host with KVM (`/dev/kvm`), Docker, Go ≥ 1.25, cgr
### OverlayBD OCI profile (default on)
-`OCI_STACK` defaults to 1: one cluster serves both template paths. The env generates a local CA + TLS certificate and deploys an in-cluster `registry:2` (fsb-vm and overlaybd require HTTPS), bakes the CA into the `fsb-fastlet`, `fsb-firecracker-runtime` and sandboxtemplate-builder images, installs the `fast-sandbox` chart's `runtime.oci` node stack, wires the registry credentials and the server's `template_oci_publish_secret`, and renders the pool fastlet with `FAST_SANDBOX_OCI_ENABLED=true` plus the shared tools/socket//dev mounts. An overlaybd template is built beside the native one and the whole verify matrix runs twice — once per template (on the overlaybd pass, template-mode create resolves `status.manifestRef`, boot is ublk on-demand, and snapshots publish/restore via `oci://` manifests). Nodes need kernel ublk support (`CONFIG_BLK_DEV_UBLK`; `modprobe ublk_drv` when built as a module). `OCI_STACK=0` restores the native-only environment.
+`OCI_STACK` defaults to 1: one cluster serves both template paths. The env generates a local CA + TLS certificate and deploys an in-cluster `registry:2` (fsb-vm and overlaybd require HTTPS), bakes the CA into the `fsb-fastlet`, `fsb-firecracker-runtime` and sandboxtemplate-builder images, installs the `fast-sandbox` chart's `runtime.oci` node stack, wires the registry credentials and the server's `template_oci_publish_secret`, and renders the pool fastlet with `FAST_SANDBOX_OCI_ENABLED=true` plus the shared tools/socket//dev mounts. The `verify-overlaybd` action builds an overlaybd template (published to the in-cluster registry) and runs the whole verify matrix against it — the same assertions as `verify-native`, but on the overlaybd path (template-mode create resolves `status.manifestRef`, boot is ublk on-demand, and snapshots publish/restore via `oci://` manifests). Nodes need kernel ublk support (`CONFIG_BLK_DEV_UBLK`; `modprobe ublk_drv` when built as a module). `OCI_STACK=0` restores the native-only environment (`verify-overlaybd` becomes a no-op).
After `up`, point any OpenSandbox SDK at `http://127.0.0.1:18080` with the `OPEN-SANDBOX-API-KEY: fast-sandbox-env` header; sandbox endpoints are signed `f1.*` header routes served by the gateway at `http://127.0.0.1:18081`. The installed CLI works against the same stack, e.g. `OPEN_SANDBOX_DOMAIN=127.0.0.1:18080 OPEN_SANDBOX_API_KEY=fast-sandbox-env osb sandbox list`.
-The `up` verify stages are also available as Python SDK e2e tests (template create → gateway ping → networkpolicy convergence, lifecycle ops, pause/resume, snapshot round trip):
+The `verify-native` / `verify-overlaybd` stages are also available as Python SDK e2e tests (template create → gateway ping → networkpolicy convergence, lifecycle ops, pause/resume, snapshot round trip):
```bash
./scripts/fast-sandbox-env/integration-env.sh sdk-e2e # needs uv on PATH
@@ -62,7 +64,7 @@ SDK ──header──> ingress gateway (source-built, --provider-type=fast-sand
## Layout
```
-integration-env.sh entrypoint: up / down / status / pool
+integration-env.sh entrypoint: up / verify-native / verify-overlaybd / down / status / pool
architecture.svg how the environment works (topology, pipeline,
P2P delivery, egress actions channel)
manifests/
@@ -80,9 +82,13 @@ Everything else comes from the OpenSandbox Helm charts (`manifests/charts`), ren
| `server` | lifecycle server: fsb runtime config (rendered `configToml`), NodePort |
| `ingress-gateway` | fsb provider + FastPath + shared signing key, NodePort |
-## Stage order (up)
+## Stage order
-preflight → sysctl → fast-sandbox checkout (pinned commit) → build images (6 fast-sandbox images via `manifests/release/build-fast-sandbox.sh` + egress + server + ingress) → XFS StateRoot → kind cluster + node labels (host ports 18080/18081) → RustFS → charts rendered + applied: base + fast-sandbox (CRDs, RBAC, control plane, installer, agent) → credentials → installer/agent roster asserted → charts rendered + applied: server + ingress gateway → SandboxTemplate golden image **built through the server `POST /templates` API** → SandboxPool (fastlet Ready, egress Ready, pool conditions, actions protocol check) → end-to-end verify (templateId create → signed gateway route → execd `/ping` → delete) → pause/resume round-trip (POST pause → poll `Paused` + route released → POST resume → poll `Running` → fresh-route `/ping` → delete).
+`up`: preflight → sysctl → fast-sandbox checkout (pinned commit) → build images (6 fast-sandbox images via `manifests/release/build-fast-sandbox.sh` + egress + server + ingress) → XFS StateRoot → kind cluster + node labels (host ports 18080/18081) → RustFS → charts rendered + applied: base + fast-sandbox (CRDs, RBAC, control plane, installer, agent) → credentials → installer/agent roster asserted → charts rendered + applied: server + ingress gateway → SandboxTemplate golden image **built through the server `POST /templates` API** → SandboxPool (fastlet Ready, egress Ready, pool conditions, actions protocol check).
+
+`verify-native`: end-to-end verify (templateId create → signed gateway route → execd `/ping` → delete) → pause/resume round-trip (POST pause → poll `Paused` + route released → POST resume → poll `Running` → fresh-route `/ping` → delete) → vault lifecycle → idle-pause/wake (IDLE_PAUSE profile) → snapshot round trip.
+
+`verify-overlaybd`: overlaybd SandboxTemplate build (oci:// publication) → the same verify matrix against the overlaybd template.
Every stage logs to `$WORK/logs/`; failures dump component logs to `logs/failure-<task>-<ts>.txt`.
diff --git a/scripts/fast-sandbox-env/integration-env.sh b/scripts/fast-sandbox-env/integration-env.sh
index aa2eaf95a..6ad71412e 100755
--- a/scripts/fast-sandbox-env/integration-env.sh
+++ b/scripts/fast-sandbox-env/integration-env.sh
@@ -34,10 +34,12 @@
# re-entry fence) plus the gateway /ping probes.
#
# Usage:
-# ./scripts/fast-sandbox-env/integration-env.sh up # full environment + pool + server/ingress + verify
-# ./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
-# ./scripts/fast-sandbox-env/integration-env.sh status # component/pool/DART/OpenSandbox health
-# ./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
+# ./scripts/fast-sandbox-env/integration-env.sh up # environment: stack + pool + server/ingress + native template
+# ./scripts/fast-sandbox-env/integration-env.sh verify-native # verify matrix on the native template (needs up)
+# ./scripts/fast-sandbox-env/integration-env.sh verify-overlaybd # overlaybd template build + verify matrix (needs up)
+# ./scripts/fast-sandbox-env/integration-env.sh pool # re-apply the pool only
+# ./scripts/fast-sandbox-env/integration-env.sh status # component/pool/DART/OpenSandbox health
+# ./scripts/fast-sandbox-env/integration-env.sh down # teardown, host left clean
# ./scripts/fast-sandbox-env/integration-env.sh up --auto-clean # down on failure
#
# Environment overrides (all optional):
@@ -1196,9 +1198,6 @@ template_up() {
[[ -n "$manifest_ref" ]] || fail "template manifestRef is empty"
log "template manifestRef: $manifest_ref"
pass "SandboxTemplate Succeeded + artifacts published (via opensandbox CLI)"
- if [[ "$OCI_STACK" == 1 ]]; then
- template_up_overlaybd
- fi
return 0
}
@@ -2711,6 +2710,42 @@ idle_e2e() {
uv run pytest tests/test_fsb_idle_pause_e2e.py
}
+# --- verify actions (native / overlaybd) --------------------------------------
+
+require_live_stack() {
+ kind get clusters 2>/dev/null | grep -x "$KIND_CLUSTER" >/dev/null \
+ || die "cluster $KIND_CLUSTER is not up (run up first)"
+ curl -fsS -m 5 "$SERVER_URL/health" >/dev/null 2>&1 \
+ || die "server $SERVER_URL is not reachable (run up first)"
+ [[ -n "$OSB_BIN" ]] || OSB_BIN="$OSB_VENV/bin/osb"
+ [[ -x "$OSB_BIN" ]] || die "osb not found at $OSB_BIN (run up first, or set OSB_BIN)"
+}
+
+verify_native() {
+ require_live_stack
+ TEMPLATE_ID="$(cat "$WORK/template-id" 2>/dev/null || true)"
+ [[ -n "$TEMPLATE_ID" ]] || die "no template id at $WORK/template-id (run up first; template_up stores it there)"
+ run_stage "end-to-end verify (API create -> gateway -> execd /ping)" opensandbox_verify
+ run_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verify
+ run_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify
+ run_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verify
+ run_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify
+}
+
+verify_overlaybd() {
+ require_live_stack
+ [[ "$OCI_STACK" == 1 ]] || {
+ log "OCI_STACK=0: overlaybd disabled; nothing to verify"
+ return 0
+ }
+ run_stage "SandboxTemplate build (overlaybd, oci:// publication)" template_up_overlaybd
+ run_stage "end-to-end verify (overlaybd template)" opensandbox_verify "$OCI_TEMPLATE_ID"
+ run_stage "pause/resume verify (overlaybd template)" pause_resume_verify "$OCI_TEMPLATE_ID"
+ run_stage "vault lifecycle verify (overlaybd template)" vault_lifecycle_verify "$OCI_TEMPLATE_ID"
+ run_stage "idle-pause / wake-on-access verify (overlaybd template)" idle_pause_verify "$OCI_TEMPLATE_ID"
+ run_stage "snapshot verify (overlaybd template)" snapshot_verify "$OCI_TEMPLATE_ID"
+}
+
# --- status / summary ---------------------------------------------------------------------
dart_metrics_summary() {
@@ -2838,20 +2873,21 @@ down() {
usage() {
cat <<'EOF'
-usage: integration-env.sh [--auto-clean] {up|down|status|pool|sdk-e2e|idle-e2e}
+usage: integration-env.sh [--auto-clean] {up|verify-native|verify-overlaybd|down|status|pool|sdk-e2e|idle-e2e}
- up initialize the full environment: fast-sandbox@pinned-commit images,
+ up initialize the environment: fast-sandbox@pinned-commit images,
two-node kind cluster (KVM), RustFS, control plane, firecracker
- firecracker runtime readiness + DART (P2P), SandboxTemplate golden
- image, firecracker-egress-pool (egress attached), the
- source-built OpenSandbox server + ingress gateway, and
- end-to-end verifies driven by the opensandbox CLI
- (create -> gateway route -> execd /ping, plus a pause/resume
- round-trip through the checkpoint). The CLI (osb) is installed
- fresh from PyPI unless OSB_BIN points at an existing binary.
- With IDLE_PAUSE=1: Redis + server [automation.idle_pause] + gateway
- activity/wake flags, plus the idle-pause/wake verify stage
- (IDLE_PAUSE_ROUNDS cycles, per-round timing summary).
+ runtime readiness + DART (P2P), the native SandboxTemplate golden
+ image, firecracker-egress-pool (egress attached), and the
+ source-built OpenSandbox server + ingress gateway. With
+ IDLE_PAUSE=1: Redis + server [automation.idle_pause] + gateway
+ activity/wake flags.
+ verify-native run the verify matrix against the native template
+ (create -> gateway route -> execd /ping, pause/resume,
+ vault, idle-pause, snapshot); requires `up`
+ verify-overlaybd build the overlaybd SandboxTemplate (oci:// publication)
+ and run the same verify matrix against it; requires `up`
+ (no-op when OCI_STACK=0)
pool re-apply only the SandboxPool (after editing manifests/pool/)
status nodes / pods / pool / DART P2P counters / RustFS / OpenSandbox health
sdk-e2e run the Python SDK e2e suite (tests/python/tests/test_fsb_e2e.py)
@@ -2888,7 +2924,7 @@ EOF
for arg in "$@"; do
case "$arg" in
--auto-clean) AUTO_CLEAN=1 ;;
- up|down|status|pool|sdk-e2e|idle-e2e) ACTION="$arg" ;;
+ up|down|status|pool|sdk-e2e|idle-e2e|verify-native|verify-overlaybd) ACTION="$arg" ;;
*) usage ;;
esac
done
@@ -2942,26 +2978,11 @@ case "$ACTION" in
run_stage "OpenSandbox server + ingress gateway" opensandbox_up
run_stage "SandboxTemplate build (opensandbox CLI)" template_up
run_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_up
- run_stage "end-to-end verify (API create -> gateway -> execd /ping)" opensandbox_verify
- run_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verify
- run_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify
- run_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verify
- run_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify
- if [[ "$OCI_STACK" == 1 ]]; then
- # Full matrix second pass: every verify stage re-runs against the
- # overlaybd template — same assertions on the on-demand boot
- # path (ublk attach, OCI snapshot publication, manifest_ref
- # restore). Native keeps its own pass above.
- run_stage "end-to-end verify (overlaybd template)" opensandbox_verify "$OCI_TEMPLATE_ID"
- run_stage "pause/resume verify (overlaybd template)" pause_resume_verify "$OCI_TEMPLATE_ID"
- run_stage "vault lifecycle verify (overlaybd template)" vault_lifecycle_verify "$OCI_TEMPLATE_ID"
- run_stage "idle-pause / wake-on-access verify (overlaybd template)" idle_pause_verify "$OCI_TEMPLATE_ID"
- run_stage "snapshot verify (overlaybd template)" snapshot_verify "$OCI_TEMPLATE_ID"
- fi
trap - ERR
stage_summary
env_summary
highlight "== up complete: server=$SERVER_URL (header OPEN-SANDBOX-API-KEY: $SERVER_API_KEY), gateway=$GATEWAY_URL =="
+ highlight "== next: verify-native, verify-overlaybd, sdk-e2e, idle-e2e =="
;;
pool)
exec > >(tee -a "$WORK/run.log") 2>&1
@@ -2989,6 +3010,22 @@ case "$ACTION" in
idle_e2e
trap - ERR
;;
+ verify-native)
+ exec > >(tee -a "$WORK/run.log") 2>&1
+ log "=== fast-sandbox-env verify-native ($(date -u +%FT%TZ)) ==="
+ trap 'on_error verify-native' ERR
+ verify_native
+ trap - ERR
+ stage_summary
+ ;;
+ verify-overlaybd)
+ exec > >(tee -a "$WORK/run.log") 2>&1
+ log "=== fast-sandbox-env verify-overlaybd ($(date -u +%FT%TZ)) ==="
+ trap 'on_error verify-overlaybd' ERR
+ verify_overlaybd
+ trap - ERR
+ stage_summary
+ ;;
down)
down
;;