Skip to content

Secure, Fast, and Extensible Sandbox runtime for AI agents.

docs(test): detailed coverage headers on every verify script

高然af4ba5597cd9
diff --git a/scripts/fast-sandbox-env/verify/idle-pause.sh b/scripts/fast-sandbox-env/verify/idle-pause.sh
index dd2c2d127..02805da70 100755
--- a/scripts/fast-sandbox-env/verify/idle-pause.sh
+++ b/scripts/fast-sandbox-env/verify/idle-pause.sh
@@ -15,10 +15,38 @@
# verify/idle-pause.sh — idle-pause / wake-on-access verify (OSEP-0024 profile)
#
-# Auto-pause after the idle window, wake on the next gateway request, the wakeOnAccess=false compatibility invariant, and pause behavior while Redis is down / flushed.
+# Requires the idle-pause profile: IDLE_PAUSE=1 up (or the $WORK/idle-pause
+# marker it leaves) — Redis + the server [automation.idle_pause] sweeper +
+# the gateway activity/wake flags. The opted-in sandbox is created through
+# the raw lifecycle API because the CLI has no lifecycle.idlePolicy yet;
+# idleTimeoutSeconds=30 (the server rejects X < 30) with onIdle=pause.
+#
+# Coverage:
+#
+# 1. Auto-pause + wake-on-access, IDLE_PAUSE_ROUNDS rounds (default 3):
+# with zero traffic the sandbox must reach Paused within X + sweep
+# interval + checkpoint window (240s per round); the next gateway
+# request must then WAKE it — traffic is driven at the paused
+# sandbox until /ping is served again, with no resume API call
+# (10min wake budget, marked 503s tolerated). Per-round pause/wake
+# latencies and the CR placement.attempt counter are logged.
+# 2. wakeOnAccess=false invariant: such a sandbox is auto-paused as
+# well, but gateway traffic must NEVER wake it (a 200 — or even a
+# wake-503 — after traffic fails the round) and the state must stay
+# Paused until a MANUAL resume brings it back to Running.
+# 3. Redis outage: scaling idle-pause-redis to 0 must suppress pausing
+# entirely — a sandbox must not be paused while the sweeper cannot
+# reach Redis. After Redis is scaled back, the missing-activity
+# tombstone produces a pause only after a full idle window.
+# 4. Redis data loss: FLUSHALL must not accelerate the pause — activity
+# keys disappearing must be treated as "activity unknown", so the
+# pause happens only after a complete fresh idle window (a pause
+# within X-5s of the flush fails the round; the pause must still
+# arrive within a 300s budget).
+# 5. Cleanup: kill the sandbox and wait until GET 404s.
+#
# Runs as part of `integration.sh up` (the entry chains every verify
# script) or standalone against a live stack from `env.sh up`.
-# Requires the idle-pause profile: IDLE_PAUSE=1 up (or the $WORK/idle-pause marker it leaves).
source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"
# --- stage: idle-pause / wake-on-access verify (OSEP-0024 profile) --------------
diff --git a/scripts/fast-sandbox-env/verify/pause-resume.sh b/scripts/fast-sandbox-env/verify/pause-resume.sh
index cab25a7ee..30a70a2ac 100755
--- a/scripts/fast-sandbox-env/verify/pause-resume.sh
+++ b/scripts/fast-sandbox-env/verify/pause-resume.sh
@@ -15,7 +15,26 @@
# verify/pause-resume.sh — pause/resume round trip (CLI -> FastPath checkpoint)
#
-# Pause until the checkpoint is durable and the signed route stops serving, resume back to Running, and /ping through a fresh route.
+# Coverage:
+#
+# 1. Baseline: create a sandbox via `osb sandbox create` (template,
+# skip-health-check), poll it to Running, and confirm execd /ping
+# returns 200 through a freshly resolved signed gateway route.
+# 2. Pause semantics (durable-first): POST pause, then poll GET until
+# state=Paused. Paused means the checkpoint is fully durable in the
+# artifact store AND the fastlet capacity is released — not merely
+# that the call was accepted. The pause -> Paused latency is logged.
+# 3. Route release (negative assertion): after Paused, the OLD signed
+# route must no longer serve /ping — the gateway must reject a
+# route whose sandbox is not running. A still-serving paused
+# sandbox fails the verify.
+# 4. Resume: POST resume, poll GET back to Running (checkpoint
+# restored from the artifact store), then /ping must return 200
+# through a FRESH route — resume advances the route generation, so
+# re-resolving the endpoint is mandatory and the old route staying
+# valid would indicate a generation bug.
+# 5. Cleanup: kill the sandbox and wait until GET 404s.
+#
# Runs as part of `integration.sh up` (the entry chains every verify
# script) or standalone against a live stack from `env.sh up`.
diff --git a/scripts/fast-sandbox-env/verify/sandbox.sh b/scripts/fast-sandbox-env/verify/sandbox.sh
index 5d376ffd8..d1828f961 100755
--- a/scripts/fast-sandbox-env/verify/sandbox.sh
+++ b/scripts/fast-sandbox-env/verify/sandbox.sh
@@ -15,7 +15,33 @@
# verify/sandbox.sh — end-to-end sandbox verify (CLI create -> gateway -> execd /ping)
#
-# Cold + warm sandbox creates through the full wire-up, egress policy enforcement and update convergence, and the remaining lifecycle surface (get/list/metadata/renew).
+# Coverage:
+#
+# 1. Full create wire-up, 6 sandboxes: `osb sandbox create` -> lifecycle
+# server (fsb runtime) -> FastPath gRPC -> fastlet -> firecracker
+# microVM -> signed gateway route -> fastlet-proxy -> guest execd
+# /ping 200. Cold #1 pulls the golden image set through DART (600s
+# ping budget); warm #2 may still pull on the peer node via its DART
+# peer (600s); warm #3-6 hit the node-local caches (120s budgets).
+# Each create logs the create latency, the create -> /ping-200
+# latency, and the Sandbox CR state at ping time (runtime/dataPlane
+# states, infraComponents, actionBindings, Ready condition).
+# 2. Egress policy enforcement on every create: each create carries a
+# default-deny policy (allow example.com + *.opensandbox.ai), so the
+# networkPolicy -> egress action binding -> nft chain must converge
+# and GET /networkpolicy must report mode=enforcing with the
+# example.com rule (120s budget) before the sandbox counts as done.
+# 3. Policy update path (raw PUT; the CLI only has merge-style patches):
+# PUT /networkpolicy with a github.com allow rule -> ReplaceActionBindings
+# -> the egress sidecar hot-swaps the nft rules -> the policy reports
+# the new target as enforcing (120s budget).
+# 4. Lifecycle surface on the first sandbox: GET returns the created id;
+# the paginated list contains it; PATCH /metadata exercises JSON Merge
+# Patch (RFC 7396) — non-null upsert and null-delete; renew 2h
+# advances expiresAt and the advanced value is visible via GET
+# (compared as epoch seconds, with string equality as fallback).
+# 5. Cleanup: kill all six sandboxes and wait until each GET 404s.
+#
# Runs as part of `integration.sh up` (the entry chains every verify
# script) or standalone against a live stack from `env.sh up`.
diff --git a/scripts/fast-sandbox-env/verify/snapshot.sh b/scripts/fast-sandbox-env/verify/snapshot.sh
index 33ea00932..fe44ad2c2 100755
--- a/scripts/fast-sandbox-env/verify/snapshot.sh
+++ b/scripts/fast-sandbox-env/verify/snapshot.sh
@@ -15,7 +15,34 @@
# verify/snapshot.sh — snapshot round trip (CLI -> SandboxSnapshot CR -> restore)
#
-# Snapshot a Running sandbox to Ready (re-entry fencing included), keep the source serving, restore a new sandbox from the snapshot id, list, and clean up.
+# Coverage:
+#
+# 1. Snapshot create: a Running, /ping-healthy source sandbox is
+# snapshotted via `osb snapshot create`; the POST returns
+# status=Creating and the server row converges from the
+# SandboxSnapshot CR via its watcher to Ready (300s budget).
+# 2. Re-entry fencing during the dump window: a second snapshot POST
+# for the same sandbox must be fenced with 409 — or, during fence
+# cache lag, accepted with 202 and then resolved to a terminal
+# phase (a stuck accepted snapshot would block all future
+# snapshots through the fence).
+# 3. Source survival: once the snapshot is terminal, the pause window
+# must be released — the source sandbox is Running and /ping-200
+# again.
+# 4. Repeat snapshot: a second snapshot of the same source after the
+# first is terminal must produce a distinct id and reach Ready.
+# 5. Listing: `osb snapshot list --sandbox-id` must contain both
+# required snapshots (and any accepted re-entry one) as Ready.
+# 6. Restore: `osb sandbox create --snapshot-id` (resource flags
+# restating the pool profile) boots the artifact set published at
+# dump time — the restored sandbox reaches Running and serves
+# execd /ping through the gateway.
+# 7. Cleanup: kill the restored and source sandboxes and delete every
+# snapshot row (the server forwards per-row artifact deletion).
+#
+# The snapshot ids are shared through SNAPSHOT_ID / SNAPSHOT_ID2 /
+# SNAPSHOT_EXTRA.
+#
# Runs as part of `integration.sh up` (the entry chains every verify
# script) or standalone against a live stack from `env.sh up`.
diff --git a/scripts/fast-sandbox-env/verify/vault.sh b/scripts/fast-sandbox-env/verify/vault.sh
index c827a5860..349668f4c 100755
--- a/scripts/fast-sandbox-env/verify/vault.sh
+++ b/scripts/fast-sandbox-env/verify/vault.sh
@@ -15,7 +15,32 @@
# verify/vault.sh — credential-vault lifecycle over the server relay (OSEP-0022)
#
-# Vault create (create-only 409), export into the checkpoint bundle on pause, reload + revision preservation across resume and an egress crash, and delete/re-create.
+# Coverage (all assertions on the live egress sidecar via the fastlet
+# Actions channel; log markers are read from the pool's egress containers):
+#
+# 1. Subject registration: the sandbox is created with a create-carried
+# default-deny egress policy (allow ext.test) — a policy-less create
+# produces no egress binding at all, so the subject would never
+# register. The verify waits for the egress log marker
+# "subject s-<uid>: data-plane-ready, policy active".
+# 2. Vault create over the server relay: POST
+# /sandboxes/{id}/credential-vault (inline credential `tok` + an
+# apiKey binding for host ext.test) must ack with a numeric revision.
+# 3. Create-only invariant: a duplicate POST must be rejected with 409.
+# 4. Pause/resume vault transfer: pause exports the vault into the
+# checkpoint bundle ("vault exported to snapshot bundle" marker);
+# the pause's REMOVE_BINDING purges the subject, so on resume the
+# import lands through the unregistered branch and the file reloads
+# at the SET_BINDING replay ("vault reloaded from file" marker). The
+# relay GET must report the SAME revision across the round trip.
+# 5. Egress crash recovery: the egress process is pkilled in every pool
+# pod; when the containers come back (fresh instanceId -> SET_BINDING
+# replay -> vault file reload by incarnation) the relay must still
+# report the original revision. Skipped on pods without pkill.
+# 6. Delete/re-create: DELETE -> GET 404; a fresh POST starts again at
+# revision 1.
+# 7. Cleanup: kill the sandbox and wait until GET 404s.
+#
# Runs as part of `integration.sh up` (the entry chains every verify
# script) or standalone against a live stack from `env.sh up`.