Secure, Fast, and Extensible Sandbox runtime for AI agents.
Update gVisor notes and references.
Etienne Perot642e8dd89cc4
diff --git a/docs/architecture/network/egress.md b/docs/architecture/network/egress.mdindex 89dee2791..d924c2e2d 100644--- a/docs/architecture/network/egress.md+++ b/docs/architecture/network/egress.md@@ -46,7 +46,7 @@ Current limits: TCP only (no UDP); at most 256 ports per rule; domain targets ca| Two layers, two costs | DNS filtering alone is cheap and universal but only gates names. `dns+nft` adds packet-level default-deny with DNS-learned, leased exceptions. || Encrypted DNS is an escape hatch | DoT (853) is always dropped, and blocking DoH over 443 is a one-switch option. || Fail-closed over best-effort | If redirect rules cannot be installed, the sidecar exits instead of running as silent pass-through; a supervisor restarts it with backoff and a crash-loop breaker. |-| Portable by fallback, honest about limits | Redirects install via iptables where available and fall back to native nftables (Firecracker-style guest kernels). gVisor's netstack cannot support this — Kata is the supported secure runtime — and transparent mesh sidecars conflict by construction. |+| Portable by fallback, honest about limits | Redirects install via iptables where available and fall back to native nftables (Firecracker-style guest kernels). gVisor does not support the nftables features either path needs, so neither works there — Kata is the supported secure runtime — and transparent mesh sidecars conflict by construction. |## Transparent TLS interception (L7)diff --git a/docs/guides/egress-upstream-proxy.md b/docs/guides/egress-upstream-proxy.mdindex 8afdf52a6..e391248fc 100644--- a/docs/guides/egress-upstream-proxy.md+++ b/docs/guides/egress-upstream-proxy.md@@ -77,7 +77,7 @@ Compatibility reflects current upstream `main` only:| Docker (`runc`) | Supported | Requires `[docker] network_mode = "bridge"`; use `ca_cert_path` only when a private CA is required. || Kubernetes (`runc`) | Supported | Non-pooled AgentSandbox and BatchSandbox workloads; use `ca_secret_name` only when a private CA is required. || Kubernetes (`Kata`) | Supported | Accepted by egress validation; validate the selected RuntimeClass, CNI, and kernel in your deployment. |-| gVisor (`secure_runtime.type = "gvisor"`) | Rejected | Sandbox creation returns HTTP 400 because gVisor netstack lacks the required iptables nat redirect. |+| gVisor (`secure_runtime.type = "gvisor"`) | Rejected | Sandbox creation returns HTTP 400 because gVisor does not support the nftables interface the egress sidecar uses for its redirect and policy rules. || Kubernetes Pool Mode | Unsupported | Rejects per-request `networkPolicy`; cannot attach per-allocation egress sidecars. || Fast Sandbox | Unsupported | Current fast-sandbox profile rejects `networkPolicy` when `upstream_proxy` is set. |diff --git a/docs/guides/secure-container.md b/docs/guides/secure-container.mdindex 33878c4e0..6482bf4b3 100644--- a/docs/guides/secure-container.md+++ b/docs/guides/secure-container.md@@ -1,6 +1,6 @@---title: Secure Container Runtime-description: Use gVisor, Kata Containers, and Firecracker microVMs for hardware-level sandbox isolation.+description: Use gVisor, Kata Containers, and Firecracker microVMs for stronger sandbox isolation.---# Secure Container Runtime Guide@@ -47,7 +47,7 @@ OpenSandbox is designed to execute untrusted code generated by AI models (ClaudeOpenSandbox supports the following secure runtime types through server-level configuration:-- `"gvisor"` - Google gVisor with runsc+- `"gvisor"` - gVisor with runsc- `"kata"` - Kata Containers with QEMU hypervisor (default)- `"firecracker"` - Kata Containers with Firecracker hypervisor- `""` (empty) - Standard runc (default, no secure runtime)@@ -372,6 +372,12 @@ Use the `runsc install` command to automatically configure Docker daemon:sudo runsc install```+To keep Docker snapshots working (see the warning below), pass `--overlay2=none` through to runsc:++```bash+sudo runsc install -- --overlay2=none+```+Or manually edit `/etc/docker/daemon.json`:```json@@ -380,8 +386,6 @@ Or manually edit `/etc/docker/daemon.json`:"runsc": {"path": "/usr/bin/runsc","runtimeArgs": [- "--platform=systrap",- "--network=host","--overlay2=none"]}@@ -601,7 +605,7 @@ Kubernetes mode supports secure runtimes through RuntimeClass resources.#### Step 1: Install gVisor Components on All Nodes-For Kubernetes with containerd, you need to install **two** components:+For Kubernetes with containerd, every node needs **two** binaries, both part of the `runsc` package:1. **runsc** - the gVisor OCI runtime2. **containerd-shim-runsc-v1** - the containerd shim for gVisor@@ -613,8 +617,8 @@ echo "deb [signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://stosudo tee /etc/apt/sources.list.d/gvisor.listsudo apt-get update-# Install both gVisor components-sudo apt-get install -y runsc containerd-shim-runsc-v1+# Install gVisor+sudo apt-get install -y runsc# Verify installationrunsc --version@@ -624,20 +628,32 @@ containerd-shim-runsc-v1 --version#### Step 2: Configure containerd-Edit `/etc/containerd/config.toml`:+Edit `/etc/containerd/config.toml` (containerd 1.x, config `version = 2`):```toml[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]- runtime_type = "io.containerd.runsc.v1"- [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]- TypeUrl = "io.containerd.runsc.v1.options"- ConfigPath = "/etc/containerd/runsc.toml"+ runtime_type = "io.containerd.runsc.v1"+```++For containerd 2.x (config `version = 3`), the section is `[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runsc]`.++To pass flags to runsc, point the shim at a config file and put the flags under `[runsc_config]`. The shim silently ignores other section names. For example:++```toml+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]+ runtime_type = "io.containerd.runsc.v1"+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]+ TypeUrl = "io.containerd.runsc.v1.options"+ ConfigPath = "/etc/containerd/runsc.toml"``````bashsudo tee /etc/containerd/runsc.toml > /dev/null <<'EOF'-[runsc]- platform = "ptrace"+[runsc_config]+ # Each key is passed to runsc as --<key>=<value>.+ # For example, here is how to enable nvproxy for GPU functionality.+ # This is equivalent to `--nvproxy=true`.+ nvproxy = "true"EOF```@@ -656,9 +672,6 @@ kind: RuntimeClassmetadata:name: gvisorhandler: runsc-scheduling:- nodeSelector:- kubernetes.io/arch: amd64``````bash@@ -863,7 +876,7 @@ If validation fails, the server refuses to start with a clear error message.| Production AI Code Execution | gVisor | Good balance of security and performance || High-Security Requirements | Kata (QEMU) | Maximum isolation, full compatibility || High-Density Multi-Tenant | Kata (Firecracker) | Minimal memory overhead per sandbox |-| Untrusted Network Code | gVisor or Kata | Syscall filtering prevents network attacks |+| Untrusted Network Code | gVisor or Kata | Network traffic is handled by gVisor's netstack or a guest Linux kernel, not the host's TCP/IP stack |---@@ -899,7 +912,7 @@ kubectl apply -f gvisor-runtimeclass.yaml**Cause**: gVisor doesn't implement all syscalls. Some applications may not be compatible.-**Solution**: Check the [gVisor compatibility guide](https://gvisor.dev/docs/user_guide/compatibility/). Try using Kata (QEMU) which has better compatibility.+**Solution**: Check the [gVisor compatibility guide](https://gvisor.dev/application-compatibility/). To find the failing syscall, temporarily add `--debug --debug-log=/tmp/runsc/ --strace` to the runsc flags and look for `unsupported syscall` messages in the logs ([gVisor debugging guide](https://gvisor.dev/docs/user_guide/debugging/)). If the workload needs unimplemented kernel features, use Kata (QEMU), which runs a full Linux guest kernel.#### 4. Pod Stuck in ContainerCreating@@ -938,12 +951,13 @@ iptables v1.8.9 (legacy): can't initialize iptables table 'nat': Table does not| Feature | runc | gVisor | Kata (QEMU) | Kata (CLH) | Kata (FC) ||---------|------|--------|-------------|------------|-----------|| Syscall Compatibility | Full | Partial | Full | Full | Limited |-| GPU Support | Yes | No | Yes | Yes | No |+| GPU Support | Yes | Yes\* | Yes | Yes | No || IPv6 | Yes | Yes | Yes | Yes | Yes |-| Privileged Mode | Yes | No | Yes | Yes | No |+| Privileged Mode | Yes | Yes | Yes | Yes | No || Docker Volume | Yes | Yes | Yes | Yes | Yes |-| Systemd | Yes | No | Yes | Yes | No |-| iptables `nat` table (egress sidecar) | Yes | **No** | Yes | Yes | Yes |+| Systemd | Yes | Yes\* | Yes | Yes | No |++`*`: Requires configuration.### Getting Helpdiff --git a/kubernetes/Makefile b/kubernetes/Makefileindex bd20026d1..a0aac5217 100644--- a/kubernetes/Makefile+++ b/kubernetes/Makefile@@ -186,7 +186,7 @@ setup-gvisor: download-gvisor ## Set up Kind cluster with gVisor (runsc) for e2e"$(KIND)" create cluster --config -@echo "Creating runsc.toml on Kind nodes..."@for node in $$(docker ps --filter "name=$(GVISOR_KIND_CLUSTER)-" --format "{{.Names}}"); do \- docker exec $$node sh -c 'mkdir -p /etc/containerd && echo "[runsc]" > /etc/containerd/runsc.toml && echo " platform = \"ptrace\"" >> /etc/containerd/runsc.toml'; \+ docker exec $$node sh -c 'mkdir -p /etc/containerd && echo "[runsc_config]" > /etc/containerd/runsc.toml'; \done.PHONY: cleanup-gvisordiff --git a/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go b/kubernetes/test/e2e_runtime/gvisor/gvisor_test.goindex 51ad30381..ef920968a 100644--- a/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go+++ b/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go@@ -139,7 +139,7 @@ spec:}})- It("should fail to start the egress sidecar under gVisor due to missing iptables nat table", func() {+ It("should fail to start the egress sidecar under gVisor due to missing nftables support", func() {egressImage := os.Getenv("EGRESS_IMG")if egressImage == "" {Skip("EGRESS_IMG not set; skipping gVisor + egress incompatibility test")@@ -187,7 +187,7 @@ spec:g.Expect(output).NotTo(Equal("0"), "egress container should exit with non-zero code")}, 2*time.Minute).Should(Succeed())- By("verifying egress logs mention iptables nat failure")+ By("verifying egress logs mention the nftables failure")output, err := runKubectl("logs", podName, "-n", testNamespace, "-c", "egress")Expect(err).NotTo(HaveOccurred())Expect(output).To(SatisfyAny(diff --git a/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml b/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yamlindex 62e2e4e63..e8f26c81d 100644--- a/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml+++ b/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml@@ -3,6 +3,3 @@ kind: RuntimeClassmetadata:name: gvisorhandler: runsc-scheduling:- nodeSelector:- kubernetes.io/arch: amd64diff --git a/oseps/0004-secure-container-runtime.md b/oseps/0004-secure-container-runtime.mdindex 7757e5d1f..10f18f58a 100644--- a/oseps/0004-secure-container-runtime.md+++ b/oseps/0004-secure-container-runtime.md@@ -443,8 +443,9 @@ Create the gVisor configuration file:```bashsudo tee /etc/containerd/runsc.toml > /dev/null <<'EOF'-[runsc]- platform = "ptrace"+[runsc_config]+ # Example configuration flag to enable GPU support:+ nvproxy = "true"EOF```diff --git a/oseps/0013-isolated-execution-api.md b/oseps/0013-isolated-execution-api.mdindex d2d7d964d..b4ccb59c1 100644--- a/oseps/0013-isolated-execution-api.md+++ b/oseps/0013-isolated-execution-api.md@@ -183,7 +183,7 @@ execd proxies filesystem operations outside the bwrap namespace, simulating the| Mount order sensitivity — argv builder is a bug-prone area | Fixed segment order with unit test coverage for every segment || seccomp blocklist only — does not prevent 0-day syscalls | Defense-in-depth; primary boundary remains namespace isolation || No per-execution cgroup — CPU/memory/pids quota falls to outer Pod | Accept for v1; cgroup wrapper planned for v2 |-| gVisor mount restrictions — commit unavailable | Capabilities endpoint marks `commit_supported = false`; diff and isolated execution still work |+| Mount restrictions — commit unavailable | Capabilities endpoint marks `commit_supported = false`; diff and isolated execution still work || Overlay upper not persistent across Pods | Upper lives on Pod filesystem; cross-Pod persistence requires caller-side upload to object storage |## Design Details@@ -341,7 +341,7 @@ AWS_*, ALI_*, ALIYUN_*, K8S_*, KUBE_*- `available = false`: all `/v1/isolated/*` write operations return `503 Service Unavailable`. Callers should fall back to the existing API or error. **SDK does not silently fall back.**- `persist.available = false`: `--isolation-upper-root` emptyDir not mounted. diff/commit/`persist.enabled=true` unavailable; isolated execution still works (upper on tmpfs).-- `commit_supported = false`: no mount permission (typical in gVisor). diff still works; commit returns 503.+- `commit_supported = false`: no mount permission (can be caused by missing `CAP_SYS_ADMIN` or a restrictive seccomp/AppArmor profile). diff still works; commit returns 503.### 6. Isolator Interface@@ -434,7 +434,7 @@ Handles whiteout (character device 0,0 → delete corresponding workspace path)v1 supports only `strategy = overwrite`. `skip-existing`, `fail-on-conflict`, and selective commit (path whitelist) are deferred to v2.-gVisor typically cannot mount overlayfs — commit is marked `commit_supported = false` in capabilities; diff and isolated execution remain available.+Where overlayfs cannot be mounted (either missing `CAP_SYS_ADMIN` or a restrictive seccomp/AppArmor profile), commit is marked `commit_supported = false` in capabilities; diff and isolated execution remain available.Diff output: `GET .../diff` streams tar.gz via `Transfer-Encoding: chunked`. execd flag `--isolation-diff-max-bytes` (default 4 GiB) rejects oversized downloads with 413.@@ -528,7 +528,7 @@ Cost: execd binary grows by ~800KB (negligible).- `/tmp` isolation: files written to `/tmp` in one session are invisible to another.- Filesystem proxy: upload → download round-trip through overlay; delete creates whiteout; search merges upper + lower.- `persist.enabled = false`: upper destroyed on session close; diff returns 404.-- `commit_supported = false` (simulated gVisor): commit returns 503; diff works.+- `commit_supported = false` (simulated mount denial): commit returns 503; diff works.- Multi-session: two concurrent sessions have independent namespaces.- Idle GC: session auto-destroyed after timeout; `run` resets timer.@@ -565,7 +565,7 @@ cgroup provides resource limits but not filesystem or PID isolation. bwrap namesbwrap natively supports `--unshare-user` with `--uid`/`--gid` for user namespace UID mapping. This would allow execd to run as non-root. Rejected because:-- Nested user namespaces require CRI/runtime support (`user.max_user_namespaces`, runtime seccomp profiles allowing `clone(CLONE_NEWUSER)`). gVisor and hardened runtimes often restrict this.+- Nested user namespaces require CRI/runtime support (`user.max_user_namespaces`, runtime seccomp profiles allowing `clone(CLONE_NEWUSER)`). Hardened runtimes often restrict this.- User namespace UID mapping is virtual — all sessions map to the same host UID, so file permissions do not provide real isolation between sessions.- Real setuid provides stronger guarantees: different sessions have different host UIDs, so file permission checks are enforced by the kernel at the host level.- The tradeoff (execd must run as root) is acceptable because execd already runs in a dedicated sandbox container, not on shared infrastructure.diff --git a/server/opensandbox_server/services/runtime_resolver.py b/server/opensandbox_server/services/runtime_resolver.pyindex 88a28e36e..3bf2cd41e 100644--- a/server/opensandbox_server/services/runtime_resolver.py+++ b/server/opensandbox_server/services/runtime_resolver.py@@ -258,7 +258,7 @@ def _warn_gvisor_egress_incompatibility(config: "AppConfig") -> None:if config.secure_runtime and config.secure_runtime.type == "gvisor" and egress_image:logger.warning("gVisor runtime is configured with egress sidecar image. "- "The egress sidecar's iptables nat-based DNS redirect is incompatible with gVisor. "+ "The egress sidecar's nftables-based DNS redirect and policy rules are incompatible with gVisor. ""Sandboxes created with network_policy will be rejected at creation time.")diff --git a/server/opensandbox_server/services/validators.py b/server/opensandbox_server/services/validators.pyindex 922f8d1f8..f85c5c86a 100644--- a/server/opensandbox_server/services/validators.py+++ b/server/opensandbox_server/services/validators.py@@ -611,10 +611,7 @@ def ensure_egress_runtime_compatible(effective_runtime_class: Optional[str] = None,) -> None:"""- Reject network_policy when the secure runtime lacks iptables nat table support.-- gVisor's netstack does not implement the iptables nat table, which the egress- sidecar requires for DNS redirect (REDIRECT target on port 53).+ Reject network_policy when the secure runtime cannot run the egress sidecar."""if not network_policy:return@@ -632,7 +629,7 @@ def ensure_egress_runtime_compatible("code": SandboxErrorCodes.INVALID_PARAMETER,"message": (f"networkPolicy is not compatible with runtime '{runtime_type}': "- f"gVisor does not support the iptables nat table required by the egress sidecar. "+ f"gVisor does not support the nftables rules required by the egress sidecar. "f"Use a compatible runtime (e.g. kata) or remove networkPolicy."),},diff --git a/server/tests/test_runtime_resolver.py b/server/tests/test_runtime_resolver.pyindex 1c76c1ee0..0098af854 100644--- a/server/tests/test_runtime_resolver.py+++ b/server/tests/test_runtime_resolver.py@@ -190,7 +190,7 @@ async def test_validate_startup_warns_gvisor_with_egress() -> None:await validate_secure_runtime_on_startup(config, k8s_client=k8s_client)mock_logger.warning.assert_called_once()- assert "iptables nat" in mock_logger.warning.call_args[0][0]+ assert "nftables" in mock_logger.warning.call_args[0][0]@pytest.mark.asyncio