Secure, Fast, and Extensible Sandbox runtime for AI agents.
test(env): exercise the full MITM CA chain in the integration environment (OSEP-0022)
高然0d0e4d52675d
diff --git a/scripts/fast-sandbox-env/README.md b/scripts/fast-sandbox-env/README.mdindex 15e3f6e06..5b1b1e29a 100644--- a/scripts/fast-sandbox-env/README.md+++ b/scripts/fast-sandbox-env/README.md@@ -6,6 +6,16 @@ All Kubernetes resources ship from the OpenSandbox Helm charts (`manifests/chartThe `verify-native` / `verify-overlaybd` actions drive the server through the opensandbox CLI (`osb`, installed fresh from PyPI into the workspace by `up` unless `OSB_BIN` is set): create a sandbox from the golden-image template, reach the in-sandbox execd `/ping` through the signed gateway route, and delete it again. Raw HTTP calls remain only where the CLI has no equivalent (networkpolicy replace, metadata merge-patch, snapshot re-entry fence). The environment is all-or-nothing: running fast-sandbox without the OpenSandbox layers is not a supported shape of this script.+### MITM CA chain (OSEP-0022, always on)++The environment exercises the full MITM trust-anchor chain against the real components — no simulation:++1. **CA material** — a self-signed CA (fresh per run) is provisioned as the versioned Secret `opensandbox-mitm-ca-v1` in the dataplane namespace, per the ops-runbook shape collapsed into one namespace: keypair entries (`tls.crt` + `tls.key`) for the egress seed mount, certificate-only entry (`ca.crt`) for the template bake. The stored certificate is fingerprint-checked against the generated one (content, not name).+2. **Template bake** — the rendered server `config.toml` sets `kubernetes.template_mitm_ca_secret`, so the server fills `spec.egressMitmCertificate` on the template create and the builder installs the certificate into the guest trust store, stamping `lineage.egressMitmCA.certSha256` into the published artifact manifest. The template stage asserts the CR field and the lineage fingerprint.+3. **Pool pinning** — the pool's egress container mounts the Secret read-only as its CA seed; the MITM addon starts with the pinned CA (fail-closed, never generates) and exposes its fingerprint on `/healthz`. The pool stage asserts the fingerprint equals the pinned one.+4. **Injection e2e** — a dedicated verify stage creates a sandbox with a policy allowing the CA-signed TLS echo endpoint (a container on the kind network reflecting request headers), first proves the credential-free passthrough (no `X-Api-Key`), then binds a vault credential to the SNI (`X-Api-Key` via the relay) and asserts the guest's own `curl --resolve … https://echo.test` response reflects the injected header — TLS verified against the template-baked CA on every leg, no `-k` anywhere. A pause/resume round trip then repeats the request: trust and vault travel with the snapshot.+5. **Pinning assertion** — template manifest lineage fingerprint == egress `/healthz` fingerprint == the pinned Secret's fingerprint, asserted three-way.+Requires a bare-metal Linux host with KVM (`/dev/kvm`), Docker, Go ≥ 1.25, cgroup v2, and `sudo` for the XFS StateRoot loop mount and sysctl bump. The CLI install requires `uv` (it manages its own ≥ 3.10 interpreter), or set `OSB_BIN` to a preinstalled `osb` binary. Full operational notes (topology, latency characteristics, production caveats): `fast-sandbox/docs/guides/firecracker-integration-env.md`.## Usage@@ -56,7 +66,7 @@ SDK ──header──> ingress gateway (source-built, --provider-type=fast-sand## Defaults-- **Pool with egress**: `firecracker-egress-pool` (firecracker runtime, `poolMin=2`) carries the OpenSandbox egress container in every fastlet Pod netns — fast-sandbox profile, `dns+nft` mode — wired through the Sandbox Actions channel (`infraComponents` host-process entry + `actionHandlers` egress@18080 with the runtime-ready / data-plane-ready hooks).+- **Pool with egress**: `firecracker-egress-pool` (firecracker runtime, `poolMin=2`) carries the OpenSandbox egress container in every fastlet Pod netns — fast-sandbox profile, `dns+nft` mode, MITM addon pinned to the CA seed mount — wired through the Sandbox Actions channel (`infraComponents` host-process entry + `actionHandlers` egress@18080 with the runtime-ready / data-plane-ready hooks).- **P2P by default**: two kind nodes, each running the runtime-agent with a node-local DART daemon (peer discovery through the headless `dart` Service); the pool spreads one fastlet per node via podAntiAffinity. Each node binds its **own** state-root subdirectory (`/var/lib/fast-sandbox/control-plane`, `/var/lib/fast-sandbox/worker`) at the same container path, so one node's committed image cache never satisfies another node's pull — artifact delivery flows `cache -> peer -> origin` with the origin fetched ~once per 4MiB block cluster-wide on every node's first create. `KIND_SINGLE=1` falls back to one node (cache-only).- **On-demand loading**: the pool has no `warmImages`; the verify sandbox's create pulls the golden snapshot set through DART. `WARM_IMAGES=1` preheats instead.- **fast-sandbox @ pinned commit**: env-owned clone at `$WORK/fast-sandbox`, cloned on first `up` and checked out at the pinned commit afterwards; relocate it with `FSB_DIR`. The source itself has no override — bump the commit in `manifests/third-party/fast-sandbox.commit`.@@ -70,7 +80,8 @@ architecture.svg how the environment works (topology, pipeline,manifests/cluster/kind-cluster.yaml two-node kind cluster, KVM/tun/shm mounts,per-node state-root subdirectories- pool/firecracker-egress-pool.yaml SandboxPool: egress attached + P2P spread+ pool/firecracker-egress-pool.yaml SandboxPool: egress attached (MITM CA+ seed mount) + P2P spread```Everything else comes from the OpenSandbox Helm charts (`manifests/charts`), rendered with `helm template` from the workdir's values (image tags, artifact-store endpoint, FastPath endpoint, signing key, NodePorts) and applied with plain `kubectl apply` — the cluster keeps no helm state:@@ -84,9 +95,9 @@ Everything else comes from the OpenSandbox Helm charts (`manifests/charts`), ren## Stage order-`up`: preflight → sysctl → fast-sandbox checkout (pinned commit) → build images (6 fast-sandbox images via `manifests/release/build-fast-sandbox.sh` + egress + server + ingress) → XFS StateRoot → kind cluster + node labels (host ports 18080/18081) → RustFS → charts rendered + applied: base + fast-sandbox (CRDs, RBAC, control plane, installer, agent) → credentials → installer/agent roster asserted → charts rendered + applied: server + ingress gateway → SandboxTemplate golden image **built through the server `POST /templates` API** → SandboxPool (fastlet Ready, egress Ready, pool conditions, actions protocol check).+`up`: preflight → sysctl → fast-sandbox checkout (pinned commit) → build images (6 fast-sandbox images via `manifests/release/build-fast-sandbox.sh` + egress + server + ingress) → XFS StateRoot → kind cluster + node labels (host ports 18080/18081) → RustFS → charts rendered + applied: base + fast-sandbox (CRDs, RBAC, control plane, installer, agent) → credentials → **MITM CA material (Secret + echo leaf cert)** → installer/agent roster asserted → charts rendered + applied: server + ingress gateway → **TLS echo endpoint (kind-network container)** → SandboxTemplate golden image **built through the server `POST /templates` API, CA baked** → SandboxPool (fastlet Ready, egress Ready, pool conditions, actions protocol check, **pinned CA fingerprint**).-`verify-native`: end-to-end verify (templateId create → signed gateway route → execd `/ping` → delete) → pause/resume round-trip (POST pause → poll `Paused` + route released → POST resume → poll `Running` → fresh-route `/ping` → delete) → vault lifecycle → idle-pause/wake (IDLE_PAUSE profile) → snapshot round trip.+`verify-native`: end-to-end verify (templateId create → signed gateway route → execd `/ping` → delete) → pause/resume round-trip (POST pause → poll `Paused` + route released → POST resume → poll `Running` → fresh-route `/ping` → delete) → vault lifecycle → **MITM CA chain verify (pinning three-way, passthrough negative control, SNI credential injection, post-resume injection)** → idle-pause/wake (IDLE_PAUSE profile) → snapshot round trip.`verify-overlaybd`: overlaybd SandboxTemplate build (oci:// publication) → the same verify matrix against the overlaybd template.diff --git a/scripts/fast-sandbox-env/integration-env.sh b/scripts/fast-sandbox-env/integration-env.shindex 1e252bbe6..6256173ea 100755--- a/scripts/fast-sandbox-env/integration-env.sh+++ b/scripts/fast-sandbox-env/integration-env.sh@@ -21,9 +21,16 @@# kind cluster with KVM passthrough, the OpenSandbox Helm charts (base CRDs,# fast-sandbox control plane, server, ingress-gateway), a RustFS artifact# store, and the firecracker-egress-pool SandboxPool with the OpenSandbox-# egress sidecar — then verifies end to end: sandbox create via the-# opensandbox CLI, execd /ping through the signed gateway route, delete,-# a pause/resume round trip, and a snapshot -> restore-from-snapshotId boot.+# egress sidecar — and the verify-native matrix then proves the stack end to+# end: sandbox create via the opensandbox CLI, execd /ping through the signed+# gateway route, delete, a pause/resume round trip, a snapshot ->+# restore-from-snapshotId boot, the egress-held credential vault lifecycle+# (OSEP-0022 FastPath relay), and the full MITM CA chain (OSEP-0022 trust+# anchor): a self-signed CA is provisioned as the versioned Secret, the+# server bakes it into the golden image, the pool pins the keypair into+# mitmdump, and a guest HTTPS request proves the template↔pool fingerprint+# pinning plus per-subject credential injection against a CA-signed TLS echo+# endpoint.## All Kubernetes resources come from the charts via `helm template` + plain# `kubectl apply` (no helm release state); the only env-owned manifests are@@ -168,6 +175,26 @@ OCI_REGISTRY_REPO="sandbox/templates"OCI_REGISTRY_HOST="registry.${RESOURCE_NS}.svc.cluster.local"CA_DIR="$WORK/registry-ca"+# OSEP-0022 MITM trust anchor — always on in this environment: a self-signed+# CA is provisioned as the versioned Secret (ops-runbook shape), the server+# bakes it into the golden-image template, the egress seeds it into mitmdump,+# and a dedicated verify stage proves the whole chain (fingerprint pinning ++# per-subject credential injection over guest HTTPS). A broken chain fails+# closed and observably (MITM addon down on /healthz, template bake, pinning+# check) instead of silently degrading.+MITM_CA_SECRET="${MITM_CA_SECRET:-opensandbox-mitm-ca-v1}"+MITM_CA_DIR="$WORK/mitm-ca"+MITM_CA_FINGERPRINT_FILE="$MITM_CA_DIR/fingerprint"+MITM_CA_FINGERPRINT=""+# TLS echo endpoint (certificate signed by the pinned CA) as a container on+# the kind network (same shape as RustFS): the guest reaches it by IP, the+# vault binding matches by SNI.+MITM_ECHO_CONTAINER="${MITM_ECHO_CONTAINER:-fast-sandbox-env-mitm-echo}"+MITM_ECHO_DOMAIN="${MITM_ECHO_DOMAIN:-echo.test}"+MITM_ECHO_PORT=443+MITM_ECHO_IP=""+MITM_ECHO_NET=""+image_repo() { printf '%s' "${1%:*}"; }image_tag() { printf '%s' "${1##*:}"; }@@ -341,7 +368,7 @@ failure_dump() {mkdir -p "$LOGS_DIR"{echo "=== fast-sandbox-env failure: $task ($(date -u +%FT%TZ)) ==="- env | grep -E '^(KIND_(CLUSTER|SINGLE|RETAIN|NODE_IMAGE)=|FSB_DIR=|SBX_IMAGE=|IMG_[A-Z_]+=|EGRESS_IMAGE=|EXECD=|WORK=|POOL_(NAME|MIN|MAX)=|INGRESS_IMAGE=|WARM_IMAGES=|XFS_(STATEROOT|SIZE)=|RUSTFS_(IMAGE|PORT|CONSOLE_PORT|BUCKET|CONTAINER|DATA|ENDPOINT)=|RC_IMAGE=|OCI_STACK=)' || true+ env | grep -E '^(KIND_(CLUSTER|SINGLE|RETAIN|NODE_IMAGE)=|FSB_DIR=|SBX_IMAGE=|IMG_[A-Z_]+=|EGRESS_IMAGE=|EXECD=|WORK=|POOL_(NAME|MIN|MAX)=|INGRESS_IMAGE=|WARM_IMAGES=|XFS_(STATEROOT|SIZE)=|RUSTFS_(IMAGE|PORT|CONSOLE_PORT|BUCKET|CONTAINER|DATA|ENDPOINT)=|RC_IMAGE=|OCI_STACK=|MITM_(CA_SECRET|CA_DIR|ECHO_CONTAINER|ECHO_DOMAIN)=)' || trueecho "--- fast-sandbox checkout ---"git -C "$FSB_DIR" rev-parse HEAD 2>&1 || trueecho "--- kind-create.log (tail) ---"@@ -396,6 +423,63 @@ failure_dump() {kubectl -n "$RESOURCE_NS" exec "$pod" -c egress -- \ls -la /var/lib/fast-sandbox/firecracker/snapshots 2>&1 | tail -8 || truedone+ # MITM CA chain (OSEP-0022) failure scene: the addon-down reason+ # lives on /healthz; seed/confdir visibility separates "Secret never+ # mounted" from "mitmdump failed to start". The echo container and+ # the guest curl stderr are only capturable here (down deletes both).+ echo "--- MITM CA chain diagnostics ---"+ if docker ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$MITM_ECHO_CONTAINER"; then+ echo "-- echo container --"+ docker ps -a --filter "name=$MITM_ECHO_CONTAINER" --format '{{.Names}} {{.Status}}' 2>&1 || true+ docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}}={{$v.IPAddress}} {{end}}' "$MITM_ECHO_CONTAINER" 2>&1 || true+ docker logs "$MITM_ECHO_CONTAINER" --tail=50 2>&1 || true+ fi+ if [[ -d "$MITM_CA_DIR" ]]; then+ echo "-- pinned CA material (names + fingerprint only, never key bytes) --"+ ls -l "$MITM_CA_DIR" 2>&1 || true+ echo "pinned fingerprint: $(cat "$MITM_CA_FINGERPRINT_FILE" 2>/dev/null || echo missing)"+ fi+ if [[ -f "$LOGS_DIR/mitm-ca-openssl.err" ]]; then+ echo "-- openssl stderr (last MITM CA stage failure) --"+ tail -20 "$LOGS_DIR/mitm-ca-openssl.err" 2>&1 || true+ fi+ if [[ -f "$LOGS_DIR/mitm-echo-probe.err" ]]; then+ echo "-- echo endpoint probe stderr --"+ tail -20 "$LOGS_DIR/mitm-echo-probe.err" 2>&1 || true+ fi+ if [[ -s "$MITM_CA_DIR/tls.crt" && -s "$MITM_CA_DIR/echo.crt" ]]; then+ echo "-- certificate extensions (public material) --"+ for cert in tls echo; do+ openssl x509 -in "$MITM_CA_DIR/$cert.crt" -noout -subject -issuer -text 2>/dev/null \+ | grep -E "subject=|issuer=|Basic Constraints|CA:|Key Usage|Alternative|Extended" || true+ done+ fi+ if [[ -f "$LOGS_DIR/mitm-guest-curl.err" ]]; then+ echo "-- last guest curl stderr ($LOGS_DIR/mitm-guest-curl.err) --"+ tail -20 "$LOGS_DIR/mitm-guest-curl.err" 2>&1 || true+ fi+ local mitm_pod+ mitm_pod="$(first_pool_pod 2>/dev/null || true)"+ if [[ -n "$mitm_pod" ]]; then+ echo "-- egress MITM surfaces (pod ${mitm_pod##*/}) --"+ kubectl -n "$RESOURCE_NS" exec "pod/$mitm_pod" -c egress -- \+ curl -fsS -m 5 "http://127.0.0.1:18080/healthz" 2>&1 || true+ echo+ kubectl -n "$RESOURCE_NS" exec "pod/$mitm_pod" -c egress -- \+ curl -fsS -m 5 "http://127.0.0.1:18080/_fastlet/v1/actions/status" 2>&1 || true+ echo+ echo "-- egress seed mount + mitmdump confdir visibility --"+ kubectl -n "$RESOURCE_NS" exec "pod/$mitm_pod" -c egress -- \+ ls -la /var/egress/mitm-ca-seed 2>&1 || true+ kubectl -n "$RESOURCE_NS" exec "pod/$mitm_pod" -c egress -- \+ ls -la /var/lib/mitmproxy/.mitmproxy 2>&1 || true+ kubectl -n "$RESOURCE_NS" exec "pod/$mitm_pod" -c egress -- \+ pgrep -af mitmdump 2>&1 || true+ fi+ if [[ -f "$GEN_DIR/mitm-ca-policy.json" ]]; then+ echo "-- mitm-ca verify sandbox policy --"+ cat "$GEN_DIR/mitm-ca-policy.json" 2>&1 || true+ fiecho "--- recent events ($RESOURCE_NS) ---"kubectl get events -n "$RESOURCE_NS" --sort-by=.lastTimestamp 2>&1 | tail -30 || trueecho "--- OpenSandbox pods ($OSB_NS) ---"@@ -512,6 +596,7 @@ preflight() {|| die "this environment requires a Linux host with KVM (run it on the remote development VM)"command -v docker >/dev/null || die "docker is required"command -v go >/dev/null || die "go is required (>=1.25, used by make images and gen-registry)"+ command -v openssl >/dev/null || die "openssl is required (signing key + pinned MITM CA generation)"ensure_tool kindensure_tool kubectlensure_tool helm@@ -997,6 +1082,221 @@ credentials_up() {pass "credentials written (publish/pull)"}+# --- stage: MITM CA material (OSEP-0022 trust anchor) -------------------------++# mitm_ca_fingerprint <cert.pem> -> lowercase hex SHA-256 over the DER — the+# exact fingerprint form the egress /healthz and the template manifest+# lineage expose.+mitm_ca_fingerprint() {+ openssl x509 -in "$1" -noout -fingerprint -sha256 2>/dev/null | sed 's/^.*=//' | tr -d ':' | tr 'A-Z' 'a-z'+}++load_mitm_ca_fingerprint() {+ if [[ -z "$MITM_CA_FINGERPRINT" && -s "$MITM_CA_FINGERPRINT_FILE" ]]; then+ MITM_CA_FINGERPRINT="$(cat "$MITM_CA_FINGERPRINT_FILE")"+ fi+}++mitm_ca_up() {+ local stored_fp openssl_err+ openssl_err="$LOGS_DIR/mitm-ca-openssl.err"+ mkdir -p "$MITM_CA_DIR"+ # Toolchain evidence: the chain guard depends on the host openssl, so a+ # failure must always carry the version + stderr (never swallowed).+ openssl version >> "$LOGS_DIR/environment.txt" 2>&1 || true+ log "openssl: $(command -v openssl) ($(openssl version 2>&1))"+ # Fresh material every run: the env owns no CA state worth carrying+ # across runs (the template that bakes it is rebuilt each time anyway);+ # down() removes the directory so a stale CA can never resurface.+ rm -f "$MITM_CA_DIR"/tls.crt "$MITM_CA_DIR"/tls.key "$MITM_CA_DIR"/tls.srl \+ "$MITM_CA_DIR"/ca.cnf "$MITM_CA_DIR"/echo.crt "$MITM_CA_DIR"/echo.key "$MITM_CA_DIR"/echo.csr "$MITM_CA_DIR"/echo.ext \+ "$MITM_CA_FINGERPRINT_FILE"+ # Versioned, immutable-by-convention CA (the v1 of opensandbox-mitm-ca-*):+ # a real signing CA (keyCertSign), what the egress seed validator demands.+ # Extensions ride an explicit config file (-config + -extensions), not+ # -addext: the runner-family openssl builds seen in CI (1.1.1k-FIPS)+ # have verifier quirks, so generation sticks to the most portable form+ # (config sections work identically from 1.0.2 through 3.x). Every+ # openssl call captures stderr into $openssl_err so a failure carries+ # the toolchain's own diagnosis, not just our message.+ cat > "$MITM_CA_DIR/ca.cnf" <<CNF+[req]+distinguished_name = dn+prompt = no++[dn]+CN = $MITM_CA_SECRET++[v3_ca]+basicConstraints = critical, CA:TRUE+keyUsage = critical, keyCertSign, cRLSign+subjectKeyIdentifier = hash+authorityKeyIdentifier = keyid:always, issuer++[v3_echo]+subjectAltName = DNS:$MITM_ECHO_DOMAIN+keyUsage = critical, digitalSignature, keyEncipherment+extendedKeyUsage = serverAuth+subjectKeyIdentifier = hash+authorityKeyIdentifier = keyid, issuer+CNF+ if ! openssl req -x509 -new -newkey rsa:2048 -nodes -days 2 \+ -keyout "$MITM_CA_DIR/tls.key" -out "$MITM_CA_DIR/tls.crt" \+ -config "$MITM_CA_DIR/ca.cnf" -extensions v3_ca 2>"$openssl_err"; then+ die "pinned MITM CA generation failed (openssl $(openssl version 2>&1)): $(tail -3 "$openssl_err" 2>/dev/null | tr '\n' ' ')"+ fi+ # Generator self-check: a build that silently drops the CA constraint+ # would seed cleanly and fail every downstream TLS handshake with an+ # opaque issuer-lookup error — refuse it here, with the evidence.+ if ! openssl x509 -in "$MITM_CA_DIR/tls.crt" -noout -text 2>/dev/null | grep -q "CA:TRUE"; then+ die "generated CA certificate lacks BasicConstraints CA:TRUE (openssl $(openssl version 2>&1) misbehaved): $(openssl x509 -in "$MITM_CA_DIR/tls.crt" -noout -text 2>/dev/null | grep -A1 -E 'Basic Constraints|Key Usage' | tr '\n' ' ')"+ fi+ log "pinned CA extensions: $(openssl x509 -in "$MITM_CA_DIR/tls.crt" -noout -text 2>/dev/null | grep -A1 -E 'Basic Constraints|Key Usage' | tr '\n' ';' | tr -s ' ')"+ MITM_CA_FINGERPRINT="$(mitm_ca_fingerprint "$MITM_CA_DIR/tls.crt")"+ [[ "$MITM_CA_FINGERPRINT" =~ ^[0-9a-f]{64}$ ]] \+ || die "generated MITM CA fingerprint is malformed: $MITM_CA_FINGERPRINT"+ printf '%s' "$MITM_CA_FINGERPRINT" > "$MITM_CA_FINGERPRINT_FILE"+ # Leaf certificate for the TLS echo endpoint, signed by the pinned CA —+ # both trust legs of the chain test terminate at it (the guest trusts it+ # through the MITM-signed certificate, mitmdump trusts it upstream).+ if ! openssl req -new -newkey rsa:2048 -nodes \+ -keyout "$MITM_CA_DIR/echo.key" -out "$MITM_CA_DIR/echo.csr" \+ -subj "/CN=$MITM_ECHO_DOMAIN" 2>>"$openssl_err"; then+ die "echo certificate request generation failed: $(tail -3 "$openssl_err" 2>/dev/null | tr '\n' ' ')"+ fi+ if ! openssl x509 -req -in "$MITM_CA_DIR/echo.csr" \+ -CA "$MITM_CA_DIR/tls.crt" -CAkey "$MITM_CA_DIR/tls.key" -CAcreateserial \+ -out "$MITM_CA_DIR/echo.crt" -days 2 \+ -extfile "$MITM_CA_DIR/ca.cnf" -extensions v3_echo 2>>"$openssl_err"; then+ die "echo certificate signing failed: $(tail -3 "$openssl_err" 2>/dev/null | tr '\n' ' ')"+ fi+ # Structural chain check (issuer == CA subject): deliberately NOT+ # `openssl verify`, whose trust-store behavior diverges across the+ # openssl builds seen on runners (1.1.1k-FIPS failed a textbook-valid+ # chain with error 20). The chain itself is genuinely verified later at+ # runtime — the guest curls without -k against the baked CA, and+ # mitmdump verifies the upstream against the pinned CA — so this early+ # check only needs to catch a mis-signed leaf, which a DN comparison+ # does on every openssl version.+ local leaf_issuer ca_subject+ leaf_issuer="$(openssl x509 -in "$MITM_CA_DIR/echo.crt" -noout -issuer 2>>"$openssl_err" | sed 's/^issuer=//')"+ ca_subject="$(openssl x509 -in "$MITM_CA_DIR/tls.crt" -noout -subject 2>>"$openssl_err" | sed 's/^subject=//')"+ [[ -n "$leaf_issuer" && "$leaf_issuer" == "$ca_subject" ]] \+ || die "echo certificate issuer does not match the pinned CA subject: issuer='$leaf_issuer' subject='$ca_subject'"+ # Ops-runbook distribution, collapsed into one namespace here (the pool+ # and template-build namespaces coincide): the keypair entries feed the+ # egress seed mount, the certificate-only entry (ca.crt) feeds the server+ # template bake. A real deployment keeps the certificate-only copy in the+ # template namespace and never carries the key.+ kubectl -n "$RESOURCE_NS" create secret generic "$MITM_CA_SECRET" \+ --from-file=tls.crt="$MITM_CA_DIR/tls.crt" \+ --from-file=tls.key="$MITM_CA_DIR/tls.key" \+ --from-file=ca.crt="$MITM_CA_DIR/tls.crt" \+ --dry-run=client -o yaml | kubectl apply -f - >/dev/null+ # Content, not name: the stored certificate must fingerprint identically+ # (the same-named-copy-wrong-content drift the design calls out).+ stored_fp="$(kubectl -n "$RESOURCE_NS" get secret "$MITM_CA_SECRET" \+ -o jsonpath='{.data.tls\.crt}' | base64 -d | mitm_ca_fingerprint /dev/stdin)"+ [[ "$stored_fp" == "$MITM_CA_FINGERPRINT" ]] \+ || die "Secret $MITM_CA_SECRET certificate fingerprint mismatch: stored=$stored_fp pinned=$MITM_CA_FINGERPRINT"+ pass "pinned MITM CA provisioned ($MITM_CA_SECRET, fingerprint ${MITM_CA_FINGERPRINT:0:16}…)"+}++# --- stage: TLS echo endpoint (certificate signed by the pinned CA) -----------++# The echo reflects the request headers it observed as the response body —+# the injected credential header must appear there, which is the per-subject+# injection assertion. Runs as a plain container on the kind network (same+# shape as RustFS): reuses the locally built egress image (python3) so no+# new image dependency is introduced.+mitm_echo_up() {+ local net script probe+ net="$(kind_network)"+ mkdir -p "$GEN_DIR"+ script="$GEN_DIR/mitm-echo-server.py"+ cat > "$script" <<'PY'+import http.server+import json+import ssl+import sys++port, cert, key = int(sys.argv[1]), sys.argv[2], sys.argv[3]+++class Handler(http.server.BaseHTTPRequestHandler):+ def _echo(self):+ body = json.dumps({k.lower(): v for k, v in self.headers.items()}).encode()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ do_GET = _echo+ do_POST = _echo++ def log_message(self, *args):+ pass+++httpd = http.server.ThreadingHTTPServer(("0.0.0.0", port), Handler)+ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)+ctx.load_cert_chain(cert, key)+httpd.socket = ctx.wrap_socket(httpd.socket, server_side=True)+httpd.serve_forever()+PY+ docker rm -f "$MITM_ECHO_CONTAINER" >/dev/null 2>&1 || true+ # --entrypoint: the egress image's supervisor ENTRYPOINT must not wrap+ # this process; root (image default) binds :443.+ docker run -d --name "$MITM_ECHO_CONTAINER" --network "$net" \+ --entrypoint python3 \+ -v "$MITM_CA_DIR/echo.crt":/etc/mitm-echo/tls.crt:ro \+ -v "$MITM_CA_DIR/echo.key":/etc/mitm-echo/tls.key:ro \+ -v "$script":/srv/mitm-echo-server.py:ro \+ "$IMG_EGRESS" /srv/mitm-echo-server.py "$MITM_ECHO_PORT" \+ /etc/mitm-echo/tls.crt /etc/mitm-echo/tls.key >/dev/null \+ || die "TLS echo container failed to start"+ MITM_ECHO_IP="$(docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$v.IPAddress}}{{end}}' "$MITM_ECHO_CONTAINER")"+ MITM_ECHO_NET="$net"+ [[ -n "$MITM_ECHO_IP" ]] || die "could not resolve the echo container IP on network $net"+ wait_for "TLS echo endpoint answering on $MITM_ECHO_IP:$MITM_ECHO_PORT" 30 mitm_echo_ready+ probe="$(mitm_echo_headers 'X-Mitm-Echo-Probe: env-up')"+ # Whitespace-tolerant: the body is python json.dumps output (": " separators).+ printf '%s' "$probe" | grep -Eq '"x-mitm-echo-probe":[[:space:]]*"env-up"' \+ || die "echo endpoint did not reflect headers: $probe"+ pass "TLS echo endpoint up ($MITM_ECHO_DOMAIN @ $MITM_ECHO_IP:$MITM_ECHO_PORT, cert signed by the pinned CA)"+}++mitm_echo_ready() {+ [[ -n "$MITM_ECHO_IP" && -n "$MITM_ECHO_NET" ]] || return 1+ mitm_echo_curl >/dev/null 2>&1+}++# mitm_echo_curl [extra-header]: HTTPS probe of the echo endpoint from a+# container ON the kind network (the resolve_rustfs_endpoint precedent) —+# the host->docker-bridge route is not a path this environment relies on+# anywhere else. Runs the locally built egress image's curl, verifying the+# chain against the pinned CA (probe stderr lands in+# $LOGS_DIR/mitm-echo-probe.err for the failure dump).+mitm_echo_curl() {+ local extra=()+ [[ -n "${1:-}" ]] && extra=(-H "$1")+ docker run --rm --network "$MITM_ECHO_NET" \+ --entrypoint curl \+ -v "$MITM_CA_DIR/tls.crt":/probe/ca.crt:ro \+ -e HOME=/tmp \+ "$IMG_EGRESS" \+ "${extra[@]}" \+ -fsS -m 10 --noproxy '*' --cacert /probe/ca.crt \+ --resolve "$MITM_ECHO_DOMAIN:$MITM_ECHO_PORT:$MITM_ECHO_IP" \+ "https://$MITM_ECHO_DOMAIN:$MITM_ECHO_PORT/echo" 2>"$LOGS_DIR/mitm-echo-probe.err"+}++# mitm_echo_headers [extra-header] -> reflected request headers (JSON body).+mitm_echo_headers() {+ mitm_echo_curl "${1:-}"+}+# --- stage: control plane -------------------------------------------------------------------# helm is only a renderer; the cluster keeps no helm release state, so@@ -1197,6 +1497,24 @@ template_up() {manifest_ref="$(osb_api template get "$TEMPLATE_ID" -o json | jq -r '.status.manifest_ref // empty')"[[ -n "$manifest_ref" ]] || fail "template manifestRef is empty"log "template manifestRef: $manifest_ref"+ # OSEP-0022 CA wiring: the server filled spec.egressMitmCertificate from+ # template_mitm_ca_secret (first real exercise of the server-side fill),+ # and the builder stamped the baked certificate's fingerprint into the+ # published manifest lineage.+ local cert_ref lineage_fp+ cert_ref="$(kubectl -n "$RESOURCE_NS" get sandboxtemplate "$TEMPLATE_ID" \+ -o jsonpath='{.spec.egressMitmCertificate.secretRef.name}' 2>/dev/null)"+ [[ "$cert_ref" == "$MITM_CA_SECRET" ]] \+ || fail "template CR carries no egressMitmCertificate secretRef (got '$cert_ref', want $MITM_CA_SECRET)"+ local cert_key+ cert_key="$(kubectl -n "$RESOURCE_NS" get sandboxtemplate "$TEMPLATE_ID" \+ -o jsonpath='{.spec.egressMitmCertificate.key}' 2>/dev/null)"+ [[ "$cert_key" == "ca.crt" ]] \+ || fail "template CR egressMitmCertificate.key is '$cert_key', want ca.crt (certificate entry only)"+ lineage_fp="$(template_mitm_lineage_fingerprint)"+ [[ "$lineage_fp" == "$MITM_CA_FINGERPRINT" ]] \+ || fail "template manifest lineage egressMitmCA.certSha256 is '${lineage_fp:-empty}', want pinned $MITM_CA_FINGERPRINT"+ log "template baked CA: secret/$cert_ref (ca.crt), lineage fingerprint ${lineage_fp:0:16}…"pass "SandboxTemplate Succeeded + artifacts published (via opensandbox CLI)"return 0}@@ -1497,6 +1815,19 @@ bake_registry_ca_into_images() {verify_image_ca_bake "$IMG_RUNTIME"}+# template_mitm_lineage_fingerprint [template-id] -> the egressMitmCA.certSha256+# the builder stamped into the published artifact-set manifest (empty when the+# manifest or the lineage entry is missing).+template_mitm_lineage_fingerprint() {+ local tid manifest_ref manifest_json+ tid="${1:-$TEMPLATE_ID}"+ manifest_ref="$(osb_api template get "$tid" -o json 2>/dev/null | jq -r '.status.manifest_ref // empty')"+ [[ -n "$manifest_ref" ]] || return 0+ manifest_json="$(rc cat "chain/$RUSTFS_BUCKET/$(dirname "${manifest_ref#s3://$RUSTFS_BUCKET/}")/manifest.json" 2>/dev/null)" || return 0+ printf '%s' "$manifest_json" | jq -r '.lineage.egressMitmCA.certificates[0].certSha256 // empty' 2>/dev/null+}++# --- stage: SandboxPool (egress attached, P2P spread) --------------------------------------------pool_pods() {kubectl -n "$RESOURCE_NS" get pods \@@ -1600,11 +1931,13 @@ render_pool() { # > $GEN_DIR/firecracker-egress-pool.yamlawk -v fastlet="$IMG_FASTLET" -v egress="$IMG_EGRESS" \-v pool_min="$POOL_MIN" -v pool_max="$POOL_MAX" \-v max_per_pod="$MAX_SANDBOXES_PER_POD" \- -v warm="$WARM_IMAGES" -v image="$TEMPLATE_ID" '+ -v warm="$WARM_IMAGES" -v image="$TEMPLATE_ID" \+ -v mitm_ca="$MITM_CA_SECRET" '{ gsub(/"@FASTLET_IMAGE@"/, fastlet)gsub(/"@EGRESS_IMAGE@"/, egress)gsub(/"@POOL_MIN@"/, pool_min)gsub(/"@POOL_MAX@"/, pool_max)+ gsub(/"@MITM_CA_SECRET@"/, mitm_ca)gsub(/maxSandboxesPerPod: 5/, "maxSandboxesPerPod: " max_per_pod) }/^# @WARM_IMAGES@$/ {if (warm == "1") printf " warmImages:\n - %s\n", image@@ -1669,14 +2002,17 @@ VOLUMES}pool_up() {+ load_mitm_ca_fingerprint+ [[ -n "$MITM_CA_FINGERPRINT" ]] || die "no pinned MITM CA fingerprint at $MITM_CA_FINGERPRINT_FILE (run up first; mitm_ca_up stores it there)"render_pool- log "applying SandboxPool $POOL_NAME (runtime=firecracker, egress attached, poolMin=$POOL_MIN)"+ log "applying SandboxPool $POOL_NAME (runtime=firecracker, egress attached, MITM CA pinned, poolMin=$POOL_MIN)"kubectl apply -f "$GEN_DIR/firecracker-egress-pool.yaml" >/dev/nullwait_for "fastlet pods Ready (poolMin=$POOL_MIN)" 300 fastlet_pods_readywait_for "egress container ready in every fastlet pod" 180 egress_containers_readywait_for "pool condition RuntimeReady=True" 120 pool_condition_true RuntimeReadywait_for "pool condition InfraReady=True" 120 pool_condition_true InfraReadywait_for "egress actions endpoint answering (/_fastlet/v1/actions/status)" 60 egress_status_ready+ wait_for "egress MITM addon up with the pinned CA fingerprint" 120 egress_mitm_ca_pinnedif [[ "$WARM_IMAGES" == "1" ]]; thenwait_for "pool warmImages Ready" 300 warm_images_readyp2p_evidence "warm preheat"@@ -1686,6 +2022,27 @@ pool_up() {fi}+# egress_mitm_ca_fingerprint -> the pinned CA fingerprint the egress exposes+# on /healthz (empty while the MITM addon is down/pending or not yet up).+egress_mitm_ca_fingerprint() {+ local pod out+ pod="$(first_pool_pod)"+ [[ -n "$pod" ]] || return 1+ out="$(kubectl -n "$RESOURCE_NS" exec "pod/$pod" -c egress -- \+ curl -fsS -m 5 "http://127.0.0.1:18080/healthz" 2>/dev/null)" || return 1+ printf '%s' "$out" | sed -n 's/.*"mitmCaFingerprint":"\([0-9a-f]*\)".*/\1/p'+}++# Content pinning (OSEP-0022): the egress seed's fingerprint must equal the+# pinned CA's — a same-named Secret copy carrying different (but valid)+# material fails here loudly instead of degrading into opaque guest TLS+# failures.+egress_mitm_ca_pinned() {+ local fp+ fp="$(egress_mitm_ca_fingerprint)"+ [[ -n "$fp" && "$fp" == "$MITM_CA_FINGERPRINT" ]]+}+# Assert the P2P outcome from DART block counters: the artifact set is# pulled ~once per 4MiB block from the origin cluster-wide, and extra nodes# must be fed by peers, not the origin.@@ -1768,7 +2125,10 @@ render_server_config() { # > $GEN_DIR/osb-server-config.tomlmkdir -p "$GEN_DIR"awk -v api_key="$SERVER_API_KEY" \-v fastpath="$FASTPATH_ENDPOINT" -v fsb_ns="$RESOURCE_NS" -v pool="$POOL_NAME" \- -v execd="$EXECD" -v oci_secret="$OCI_REGISTRY_SECRET" -v oci_stack="$OCI_STACK" '+ awk -v api_key="$SERVER_API_KEY" \+ -v fastpath="$FASTPATH_ENDPOINT" -v fsb_ns="$RESOURCE_NS" -v pool="$POOL_NAME" \+ -v execd="$EXECD" -v oci_secret="$OCI_REGISTRY_SECRET" -v oci_stack="$OCI_STACK" \+ -v mitm_ca="$MITM_CA_SECRET" '{ if ($0 == "# @OCI_PUBLISH_SECRET@") {# emitted in-place so the key stays inside [kubernetes].if (oci_stack == "1")@@ -1780,6 +2140,7 @@ render_server_config() { # > $GEN_DIR/osb-server-config.tomlgsub(/@FSB_NAMESPACE@/, fsb_ns)gsub(/@POOL_NAME@/, pool)gsub(/@EXECD_IMAGE@/, execd)+ gsub(/@MITM_CA_SECRET@/, mitm_ca)gsub(/@GATEWAY_ADDRESS@/, gateway)print }' <<'TOML' > "$GEN_DIR/osb-server-config.toml"@@ -1804,6 +2165,12 @@ fastpath_resource_pool = "@POOL_NAME@"fastpath_wait_ready_seconds = 30.0template_s3_publish_secret = "sandbox-oss-credentials"# @OCI_PUBLISH_SECRET@+# OSEP-0022 MITM trust anchor: server-created SandboxTemplates bake+# spec.egressMitmCertificate from the pinned CA Secret (certificate entry+# only, never the private key), so the builder installs it into the guest+# trust store at build time.+template_mitm_ca_secret = "@MITM_CA_SECRET@"+template_mitm_ca_secret_key = "ca.crt"TOMLif [[ "$IDLE_PAUSE" == "1" ]]; then# Dotted keys nest the redis sub-table.@@ -2341,6 +2708,124 @@ vault_lifecycle_verify() {wait_for "vault sandbox deleted" 120 verify_sandbox_gone}+# --- stage: MITM CA chain verify (OSEP-0022: baked CA + injection + pinning) ----++# Guest-side HTTPS request to the echo endpoint via `osb command run` (the+# only sandbox-reachable surface). `curl --resolve` pins the SNI to the echo+# domain — the domain is deliberately unresolvable DNS, and the policy+# carries the IP-allow rule the resolved flow needs; the egress DNATs the+# allowed flow to the shared mitmdump, which matches the vault binding by+# SNI. TLS verification runs against the template-baked CA (no -k anywhere).+# The guest-side stderr (curl/CLI error text — the first diagnostic for a+# broken TLS chain) is kept per attempt at $LOGS_DIR/mitm-guest-curl.err for+# the failure dump.+mitm_ca_guest_echo() {+ osb_api command run "$VERIFY_ID" -- \+ "curl -fsS -m 20 --resolve $MITM_ECHO_DOMAIN:$MITM_ECHO_PORT:$MITM_ECHO_IP https://$MITM_ECHO_DOMAIN:$MITM_ECHO_PORT/echo" \+ 2>"$LOGS_DIR/mitm-guest-curl.err"+}++# osb_command_succeeds <command>: wait_for probe variant — the guest command+# must exit 0 (a failed execd command makes the CLI exit non-zero). stderr+# lands in the same per-attempt diagnostics file as mitm_ca_guest_echo.+osb_command_succeeds() {+ osb_api command run "$VERIFY_ID" -- "$1" >/dev/null 2>"$LOGS_DIR/mitm-guest-curl.err"+}++# mitm_ca_guest_header_is <key> <value>: the echo reflected the exact header.+# Scans line-by-line: the CLI stream may carry non-JSON framing.+mitm_ca_guest_header_is() { # <lowercase-key> <expected-value>+ local out line+ out="$(mitm_ca_guest_echo)" || return 1+ while IFS= read -r line; do+ [[ -n "$line" ]] || continue+ printf '%s' "$line" | jq -e --arg k "$1" --arg v "$2" '.[$k] == $v' >/dev/null 2>&1 && return 0+ done <<EOF+$out+EOF+ return 1+}++mitm_ca_chain_verify() {+ # Optional template override (the overlaybd second pass); default native.+ local template_id="${1:-$TEMPLATE_ID}"+ local created sandbox_uid ack echo_secret+ log "verify (mitm-ca): three-way fingerprint pinning (template lineage == egress /healthz == pinned Secret)"+ local healthz_fp lineage_fp+ healthz_fp="$(egress_mitm_ca_fingerprint)"+ [[ "$healthz_fp" == "$MITM_CA_FINGERPRINT" ]] \+ || fail "mitm-ca verify: egress /healthz fingerprint is '${healthz_fp:-empty}', want pinned $MITM_CA_FINGERPRINT"+ lineage_fp="$(template_mitm_lineage_fingerprint "$template_id")"+ [[ "$lineage_fp" == "$MITM_CA_FINGERPRINT" ]] \+ || fail "mitm-ca verify: template manifest lineage fingerprint is '${lineage_fp:-empty}', want pinned $MITM_CA_FINGERPRINT"+ pass "template↔pool CA pinning: manifest lineage == egress /healthz == pinned (${MITM_CA_FINGERPRINT:0:16}…)"++ # Sandbox policy: the domain allow covers the vault binding (binding+ # validation requires the host be policy-allowed), the IP allow covers+ # the --resolve flow at the nft layer (no DNS query ever happens).+ jq -n --arg ip "$MITM_ECHO_IP" --arg domain "$MITM_ECHO_DOMAIN" \+ '{defaultAction: "deny", egress: [{action: "allow", target: $ip}, {action: "allow", target: $domain}]}' \+ > "$GEN_DIR/mitm-ca-policy.json"+ log "verify (mitm-ca): creating a sandbox via the opensandbox CLI (templateId=$template_id, echo=$MITM_ECHO_IP)"+ created="$(osb_api sandbox create --template "$template_id" --timeout 1h \+ --metadata origin=fast-sandbox-env-mitm-ca \+ --network-policy-file "$GEN_DIR/mitm-ca-policy.json" \+ --skip-health-check -o json 2>/dev/null)" \+ || fail "mitm-ca verify: sandbox create failed"+ VERIFY_ID="$(printf '%s' "$created" | jq -r '.id')"+ [[ -n "$VERIFY_ID" && "$VERIFY_ID" != "null" ]] || fail "mitm-ca verify: create response carried no id"+ wait_for "mitm-ca sandbox Running" 300 sandbox_running+ sandbox_uid="$(kubectl -n "$RESOURCE_NS" get sandbox "$VERIFY_ID" -o jsonpath='{.metadata.uid}' 2>/dev/null)"+ [[ -n "$sandbox_uid" ]] || fail "mitm-ca verify: sandbox CR carried no uid"+ wait_for "mitm-ca subject registered on egress" 120 \+ vault_log_contains "subject s-$sandbox_uid: data-plane-ready, policy active"++ # Negative control BEFORE the vault exists: the flow is allowed and+ # passes through the shared mitmdump opaquely (unbound SNI), TLS is+ # verified end-to-end against the template-baked CA — and no credential+ # header appears. Proves the header in the positive case comes from the+ # vault, not from anything ambient.+ wait_for "pre-vault guest HTTPS reaches the echo (passthrough, baked-CA verified)" 120 \+ osb_command_succeeds "curl -fsS -m 20 --resolve $MITM_ECHO_DOMAIN:$MITM_ECHO_PORT:$MITM_ECHO_IP https://$MITM_ECHO_DOMAIN:$MITM_ECHO_PORT/echo"+ if mitm_ca_guest_echo | grep -qi 'x-api-key'; then+ fail "mitm-ca verify: x-api-key present before any vault binding existed"+ fi+ pass "pre-vault: guest HTTPS passthrough works, no credential injected"++ # Per-subject vault binding: SNI echo.test -> X-Api-Key.+ echo_secret="mitm-ca-e2e-$(openssl rand -hex 8)"+ local vault_body+ vault_body="$(jq -n --arg v "$echo_secret" --arg domain "$MITM_ECHO_DOMAIN" \+ '{credentials: [{name: "tok", source: {type: "inline", value: $v}}],+ bindings: [{name: "echo", match: {hosts: [$domain]}, auth: {type: "apiKey", name: "X-Api-Key", credential: "tok"}}]}')"+ ack="$(server_api POST "/sandboxes/$VERIFY_ID/credential-vault" "$vault_body")" \+ || fail "mitm-ca verify: POST credential-vault failed"+ [[ "$(printf '%s' "$ack" | jq -r '.revision // empty')" =~ ^[0-9]+$ ]] \+ || fail "mitm-ca verify: vault create ack carried no revision: $(printf '%s' "$ack" | head -c 200)"+ pass "vault bound to SNI $MITM_ECHO_DOMAIN (X-Api-Key) over the relay"++ wait_for "guest HTTPS reflects the injected X-Api-Key (MITM decrypt + vault by SNI)" 120 \+ mitm_ca_guest_header_is "x-api-key" "$echo_secret"+ pass "credential injection observed by the echo endpoint on a guest-originated HTTPS request"++ # Pause/resume: trust travels with the snapshot (the baked CA is part of+ # the rootfs) and the vault is carried in the bundle — the same request+ # must keep working end to end after restore.+ osb_api sandbox pause "$VERIFY_ID" >/dev/null 2>&1 || fail "mitm-ca verify: pause failed"+ wait_for "mitm-ca sandbox Paused (checkpoint durable)" 240 sandbox_paused+ osb_api sandbox resume "$VERIFY_ID" --skip-health-check >/dev/null 2>&1 || fail "mitm-ca verify: resume failed"+ wait_for "mitm-ca sandbox Running (restored)" 240 sandbox_running+ wait_for "restored subject re-registered on egress" 120 \+ vault_log_contains "subject s-$sandbox_uid: vault reloaded from file"+ wait_for "post-resume guest HTTPS still MITM-injected (baked CA + vault survive the snapshot)" 180 \+ mitm_ca_guest_header_is "x-api-key" "$echo_secret"+ pass "paused/restored sandbox keeps passing TLS verification against the baked CA (injection intact)"++ osb_api sandbox kill "$VERIFY_ID" >/dev/null 2>&1 \+ || log "verify cleanup: kill failed; remove $VERIFY_ID manually"+ wait_for "mitm-ca sandbox deleted" 120 verify_sandbox_gone+}+# --- stage: idle-pause / wake-on-access verify (OSEP-0024 profile) --------------@@ -2728,6 +3213,7 @@ verify_native() {run_stage "end-to-end verify (API create -> gateway -> execd /ping)" opensandbox_verifyrun_stage "pause/resume verify (CLI -> FastPath checkpoint)" pause_resume_verifyrun_stage "vault lifecycle verify (CLI -> relay -> egress-held vault)" vault_lifecycle_verify+ run_stage "MITM CA chain verify (baked CA -> injection -> pinning)" mitm_ca_chain_verifyrun_stage "idle-pause / wake-on-access verify (OSEP-0024 profile)" idle_pause_verifyrun_stage "snapshot verify (CLI -> SandboxSnapshot -> restore)" snapshot_verify}@@ -2830,6 +3316,7 @@ env_summary() {printf ' %-22s %s\n' "fast-sandbox" "pinned $(git -C "$FSB_DIR" rev-parse --short HEAD 2>/dev/null || echo "$FSB_COMMIT") ($FSB_DIR)"printf ' %-22s %s\n' "RustFS endpoint" "$RUSTFS_ENDPOINT"printf ' %-22s %s\n' "pool" "$POOL_NAME (runtime=firecracker, poolMin=$POOL_MIN, egress=$IMG_EGRESS)"+ printf ' %-22s %s\n' "MITM CA" "$MITM_CA_SECRET (fingerprint ${MITM_CA_FINGERPRINT:0:16}…, echo=https://$MITM_ECHO_DOMAIN via $MITM_ECHO_IP:$MITM_ECHO_PORT)"printf ' %-22s %s\n' "P2P" "DART daemons=$(printf '%s' "$(runtime_pods)" | wc -w | tr -d ' ') (on-demand pulls: cache -> peer -> origin)"printf ' %-22s %s\n' "template" "${TEMPLATE_ID:-n/a} ($(if [[ -n "$TEMPLATE_ID" ]]; then _template_phase || echo unknown; else echo "not built"; fi))"printf ' %-22s %s\n' "StateRoot fs" "$(findmnt -no FSTYPE "$XFS_MOUNT_POINT" 2>/dev/null || echo 'plain directory (full copy per sandbox)')"@@ -2851,6 +3338,12 @@ down() {docker rm -f "$RUSTFS_CONTAINER" >/dev/null 2>&1 || true[[ -z "$(docker ps -a --filter "name=$RUSTFS_CONTAINER" --format '{{.Names}}' || true)" ]] \|| fail "RustFS container still present"+ # TLS echo endpoint + its CA material die with the run (regenerated fresh+ # on every up; a stale CA must never resurface).+ docker rm -f "$MITM_ECHO_CONTAINER" >/dev/null 2>&1 || true+ [[ -z "$(docker ps -a --filter "name=$MITM_ECHO_CONTAINER" --format '{{.Names}}' || true)" ]] \+ || fail "MITM echo container still present"+ rm -rf "$MITM_CA_DIR"# Server/gateway die with the cluster; only the signing key outlives it.rm -f "$WORK/agent-registry.json" "$SIGNING_KEY_FILE"rm -rf "$GEN_DIR" "$FSB_GEN_DIR"@@ -2976,9 +3469,11 @@ case "$ACTION" inrun_stage "OCI registry (in-cluster, TLS)" registry_uprun_stage "CRDs + control plane" control_plane_uprun_stage "credentials (publish/pull)" credentials_up+ run_stage "MITM CA material (OSEP-0022 trust anchor)" mitm_ca_uprun_stage "firecracker runtime readiness + DART (P2P)" runtime_uprun_stage "idle-pause Redis (OSEP-0024 profile)" redis_uprun_stage "OpenSandbox server + ingress gateway" opensandbox_up+ run_stage "TLS echo endpoint (cert signed by the pinned CA)" mitm_echo_uprun_stage "SandboxTemplate build (opensandbox CLI)" template_uprun_stage "SandboxPool $POOL_NAME (egress + P2P)" pool_uptrap - ERRdiff --git a/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml b/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yamlindex 4bcb5b549..95769451e 100644--- a/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml+++ b/scripts/fast-sandbox-env/manifests/pool/firecracker-egress-pool.yaml@@ -30,6 +30,7 @@## Tokens rendered by integration-env.sh (pool stage, into $WORK/gen):# "@FASTLET_IMAGE@" "@EGRESS_IMAGE@" "@POOL_MIN@" "@POOL_MAX@"+# "@MITM_CA_SECRET@" the pool-pinned MITM CA Secret (seed mount)# # @WARM_IMAGES@ -> warmImages block when WARM_IMAGES=1# # @OCI_FASTLET_ENV@ / # @OCI_FASTLET_MOUNTS@ / # @OCI_FASTLET_VOLUMES@# -> the fastlet OCI env item + shared mounts/volumes when OCI_STACK=1@@ -90,6 +91,15 @@ spec:# transfer. Lives on the same Pod-scoped volume as the vault files.- name: OPENSANDBOX_EGRESS_POLICY_DIRvalue: /var/lib/opensandbox-egress/policy+ # MITM trust anchor (OSEP-0022): enable the shared mitmdump and pin+ # it to the seed-mounted CA (below). The extra-CA file makes the+ # upstream leg verify against the pinned CA as well.+ - name: OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT+ value: "true"+ - name: OPENSANDBOX_EGRESS_MITM_CA_SEED_DIR+ value: /var/egress/mitm-ca-seed+ - name: OPENSANDBOX_EGRESS_MITMPROXY_UPSTREAM_EXTRA_CA+ value: /var/egress/mitm-ca-seed/tls.crtvolumeMounts:- name: egress-vault# Plaintext v1 posture: the vault volume joins the credential@@ -101,6 +111,15 @@ spec:# same runtime-injected volume so the path resolves here.- name: firecracker-statemountPath: /var/lib/fast-sandbox/firecracker+ # MITM trust anchor (OSEP-0022): the pool-pinned CA keypair rides a+ # read-only seed mount; egress copies it idempotently into the+ # mitmdump confdir and never generates a CA (missing/unloadable seed+ # fails the MITM addon closed, surfaced on /healthz). The extra-CA+ # env pins the mitmdump->upstream leg to the same CA (the echo+ # upstream presents a certificate signed by it).+ - name: mitm-ca-seed+ mountPath: /var/egress/mitm-ca-seed+ readOnly: true# Egress installs nft/iptables rules (DNS redirect + per-subject# sets) and marks sockets; NET_ADMIN covers nft/iptables/SO_MARK,# NET_RAW covers the raw sockets used for DNS response spoofing.@@ -129,6 +148,20 @@ spec:- name: egress-vaultemptyDir: {}# @OCI_FASTLET_VOLUMES@+ # Pool-pinned MITM CA keypair (OSEP-0022, ops-runbook shape): the+ # versioned Secret copy lives in the pool namespace with the keypair+ # entries the seed mount expects (tls.crt + tls.key). In this env the+ # template-build namespace coincides with the pool namespace, so the+ # same Secret also carries the certificate-only entry (ca.crt) the+ # server references for the template bake.+ - name: mitm-ca-seed+ secret:+ secretName: "@MITM_CA_SECRET@"+ items:+ - key: tls.crt+ path: tls.crt+ - key: tls.key+ path: tls.keynodeSelector:fast-sandbox.io/firecracker-node: "true"# P2P topology: spread the poolMin fastlets one per node so each