Secure, Fast, and Extensible Sandbox runtime for AI agents.
docs(test): detailed coverage headers on every verify script
高然af4ba5597cd9
diff --git a/scripts/fast-sandbox-env/verify/idle-pause.sh b/scripts/fast-sandbox-env/verify/idle-pause.shindex dd2c2d127..02805da70 100755--- a/scripts/fast-sandbox-env/verify/idle-pause.sh+++ b/scripts/fast-sandbox-env/verify/idle-pause.sh@@ -15,10 +15,38 @@# verify/idle-pause.sh — idle-pause / wake-on-access verify (OSEP-0024 profile)#-# Auto-pause after the idle window, wake on the next gateway request, the wakeOnAccess=false compatibility invariant, and pause behavior while Redis is down / flushed.+# Requires the idle-pause profile: IDLE_PAUSE=1 up (or the $WORK/idle-pause+# marker it leaves) — Redis + the server [automation.idle_pause] sweeper ++# the gateway activity/wake flags. The opted-in sandbox is created through+# the raw lifecycle API because the CLI has no lifecycle.idlePolicy yet;+# idleTimeoutSeconds=30 (the server rejects X < 30) with onIdle=pause.+#+# Coverage:+#+# 1. Auto-pause + wake-on-access, IDLE_PAUSE_ROUNDS rounds (default 3):+# with zero traffic the sandbox must reach Paused within X + sweep+# interval + checkpoint window (240s per round); the next gateway+# request must then WAKE it — traffic is driven at the paused+# sandbox until /ping is served again, with no resume API call+# (10min wake budget, marked 503s tolerated). Per-round pause/wake+# latencies and the CR placement.attempt counter are logged.+# 2. wakeOnAccess=false invariant: such a sandbox is auto-paused as+# well, but gateway traffic must NEVER wake it (a 200 — or even a+# wake-503 — after traffic fails the round) and the state must stay+# Paused until a MANUAL resume brings it back to Running.+# 3. Redis outage: scaling idle-pause-redis to 0 must suppress pausing+# entirely — a sandbox must not be paused while the sweeper cannot+# reach Redis. After Redis is scaled back, the missing-activity+# tombstone produces a pause only after a full idle window.+# 4. Redis data loss: FLUSHALL must not accelerate the pause — activity+# keys disappearing must be treated as "activity unknown", so the+# pause happens only after a complete fresh idle window (a pause+# within X-5s of the flush fails the round; the pause must still+# arrive within a 300s budget).+# 5. Cleanup: kill the sandbox and wait until GET 404s.+## Runs as part of `integration.sh up` (the entry chains every verify# script) or standalone against a live stack from `env.sh up`.-# Requires the idle-pause profile: IDLE_PAUSE=1 up (or the $WORK/idle-pause marker it leaves).source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/lib.sh"# --- stage: idle-pause / wake-on-access verify (OSEP-0024 profile) --------------diff --git a/scripts/fast-sandbox-env/verify/pause-resume.sh b/scripts/fast-sandbox-env/verify/pause-resume.shindex cab25a7ee..30a70a2ac 100755--- a/scripts/fast-sandbox-env/verify/pause-resume.sh+++ b/scripts/fast-sandbox-env/verify/pause-resume.sh@@ -15,7 +15,26 @@# verify/pause-resume.sh — pause/resume round trip (CLI -> FastPath checkpoint)#-# Pause until the checkpoint is durable and the signed route stops serving, resume back to Running, and /ping through a fresh route.+# Coverage:+#+# 1. Baseline: create a sandbox via `osb sandbox create` (template,+# skip-health-check), poll it to Running, and confirm execd /ping+# returns 200 through a freshly resolved signed gateway route.+# 2. Pause semantics (durable-first): POST pause, then poll GET until+# state=Paused. Paused means the checkpoint is fully durable in the+# artifact store AND the fastlet capacity is released — not merely+# that the call was accepted. The pause -> Paused latency is logged.+# 3. Route release (negative assertion): after Paused, the OLD signed+# route must no longer serve /ping — the gateway must reject a+# route whose sandbox is not running. A still-serving paused+# sandbox fails the verify.+# 4. Resume: POST resume, poll GET back to Running (checkpoint+# restored from the artifact store), then /ping must return 200+# through a FRESH route — resume advances the route generation, so+# re-resolving the endpoint is mandatory and the old route staying+# valid would indicate a generation bug.+# 5. Cleanup: kill the sandbox and wait until GET 404s.+## Runs as part of `integration.sh up` (the entry chains every verify# script) or standalone against a live stack from `env.sh up`.diff --git a/scripts/fast-sandbox-env/verify/sandbox.sh b/scripts/fast-sandbox-env/verify/sandbox.shindex 5d376ffd8..d1828f961 100755--- a/scripts/fast-sandbox-env/verify/sandbox.sh+++ b/scripts/fast-sandbox-env/verify/sandbox.sh@@ -15,7 +15,33 @@# verify/sandbox.sh — end-to-end sandbox verify (CLI create -> gateway -> execd /ping)#-# Cold + warm sandbox creates through the full wire-up, egress policy enforcement and update convergence, and the remaining lifecycle surface (get/list/metadata/renew).+# Coverage:+#+# 1. Full create wire-up, 6 sandboxes: `osb sandbox create` -> lifecycle+# server (fsb runtime) -> FastPath gRPC -> fastlet -> firecracker+# microVM -> signed gateway route -> fastlet-proxy -> guest execd+# /ping 200. Cold #1 pulls the golden image set through DART (600s+# ping budget); warm #2 may still pull on the peer node via its DART+# peer (600s); warm #3-6 hit the node-local caches (120s budgets).+# Each create logs the create latency, the create -> /ping-200+# latency, and the Sandbox CR state at ping time (runtime/dataPlane+# states, infraComponents, actionBindings, Ready condition).+# 2. Egress policy enforcement on every create: each create carries a+# default-deny policy (allow example.com + *.opensandbox.ai), so the+# networkPolicy -> egress action binding -> nft chain must converge+# and GET /networkpolicy must report mode=enforcing with the+# example.com rule (120s budget) before the sandbox counts as done.+# 3. Policy update path (raw PUT; the CLI only has merge-style patches):+# PUT /networkpolicy with a github.com allow rule -> ReplaceActionBindings+# -> the egress sidecar hot-swaps the nft rules -> the policy reports+# the new target as enforcing (120s budget).+# 4. Lifecycle surface on the first sandbox: GET returns the created id;+# the paginated list contains it; PATCH /metadata exercises JSON Merge+# Patch (RFC 7396) — non-null upsert and null-delete; renew 2h+# advances expiresAt and the advanced value is visible via GET+# (compared as epoch seconds, with string equality as fallback).+# 5. Cleanup: kill all six sandboxes and wait until each GET 404s.+## Runs as part of `integration.sh up` (the entry chains every verify# script) or standalone against a live stack from `env.sh up`.diff --git a/scripts/fast-sandbox-env/verify/snapshot.sh b/scripts/fast-sandbox-env/verify/snapshot.shindex 33ea00932..fe44ad2c2 100755--- a/scripts/fast-sandbox-env/verify/snapshot.sh+++ b/scripts/fast-sandbox-env/verify/snapshot.sh@@ -15,7 +15,34 @@# verify/snapshot.sh — snapshot round trip (CLI -> SandboxSnapshot CR -> restore)#-# Snapshot a Running sandbox to Ready (re-entry fencing included), keep the source serving, restore a new sandbox from the snapshot id, list, and clean up.+# Coverage:+#+# 1. Snapshot create: a Running, /ping-healthy source sandbox is+# snapshotted via `osb snapshot create`; the POST returns+# status=Creating and the server row converges from the+# SandboxSnapshot CR via its watcher to Ready (300s budget).+# 2. Re-entry fencing during the dump window: a second snapshot POST+# for the same sandbox must be fenced with 409 — or, during fence+# cache lag, accepted with 202 and then resolved to a terminal+# phase (a stuck accepted snapshot would block all future+# snapshots through the fence).+# 3. Source survival: once the snapshot is terminal, the pause window+# must be released — the source sandbox is Running and /ping-200+# again.+# 4. Repeat snapshot: a second snapshot of the same source after the+# first is terminal must produce a distinct id and reach Ready.+# 5. Listing: `osb snapshot list --sandbox-id` must contain both+# required snapshots (and any accepted re-entry one) as Ready.+# 6. Restore: `osb sandbox create --snapshot-id` (resource flags+# restating the pool profile) boots the artifact set published at+# dump time — the restored sandbox reaches Running and serves+# execd /ping through the gateway.+# 7. Cleanup: kill the restored and source sandboxes and delete every+# snapshot row (the server forwards per-row artifact deletion).+#+# The snapshot ids are shared through SNAPSHOT_ID / SNAPSHOT_ID2 /+# SNAPSHOT_EXTRA.+## Runs as part of `integration.sh up` (the entry chains every verify# script) or standalone against a live stack from `env.sh up`.diff --git a/scripts/fast-sandbox-env/verify/vault.sh b/scripts/fast-sandbox-env/verify/vault.shindex c827a5860..349668f4c 100755--- a/scripts/fast-sandbox-env/verify/vault.sh+++ b/scripts/fast-sandbox-env/verify/vault.sh@@ -15,7 +15,32 @@# verify/vault.sh — credential-vault lifecycle over the server relay (OSEP-0022)#-# Vault create (create-only 409), export into the checkpoint bundle on pause, reload + revision preservation across resume and an egress crash, and delete/re-create.+# Coverage (all assertions on the live egress sidecar via the fastlet+# Actions channel; log markers are read from the pool's egress containers):+#+# 1. Subject registration: the sandbox is created with a create-carried+# default-deny egress policy (allow ext.test) — a policy-less create+# produces no egress binding at all, so the subject would never+# register. The verify waits for the egress log marker+# "subject s-<uid>: data-plane-ready, policy active".+# 2. Vault create over the server relay: POST+# /sandboxes/{id}/credential-vault (inline credential `tok` + an+# apiKey binding for host ext.test) must ack with a numeric revision.+# 3. Create-only invariant: a duplicate POST must be rejected with 409.+# 4. Pause/resume vault transfer: pause exports the vault into the+# checkpoint bundle ("vault exported to snapshot bundle" marker);+# the pause's REMOVE_BINDING purges the subject, so on resume the+# import lands through the unregistered branch and the file reloads+# at the SET_BINDING replay ("vault reloaded from file" marker). The+# relay GET must report the SAME revision across the round trip.+# 5. Egress crash recovery: the egress process is pkilled in every pool+# pod; when the containers come back (fresh instanceId -> SET_BINDING+# replay -> vault file reload by incarnation) the relay must still+# report the original revision. Skipped on pods without pkill.+# 6. Delete/re-create: DELETE -> GET 404; a fresh POST starts again at+# revision 1.+# 7. Cleanup: kill the sandbox and wait until GET 404s.+## Runs as part of `integration.sh up` (the entry chains every verify# script) or standalone against a live stack from `env.sh up`.