Skip to content

Secure, Fast, and Extensible Sandbox runtime for AI agents.

Update gVisor notes and references.

Etienne Perot642e8dd89cc4
diff --git a/docs/architecture/network/egress.md b/docs/architecture/network/egress.md
index 89dee2791..d924c2e2d 100644
--- a/docs/architecture/network/egress.md
+++ b/docs/architecture/network/egress.md
@@ -46,7 +46,7 @@ Current limits: TCP only (no UDP); at most 256 ports per rule; domain targets ca
| Two layers, two costs | DNS filtering alone is cheap and universal but only gates names. `dns+nft` adds packet-level default-deny with DNS-learned, leased exceptions. |
| Encrypted DNS is an escape hatch | DoT (853) is always dropped, and blocking DoH over 443 is a one-switch option. |
| Fail-closed over best-effort | If redirect rules cannot be installed, the sidecar exits instead of running as silent pass-through; a supervisor restarts it with backoff and a crash-loop breaker. |
-| Portable by fallback, honest about limits | Redirects install via iptables where available and fall back to native nftables (Firecracker-style guest kernels). gVisor's netstack cannot support this — Kata is the supported secure runtime — and transparent mesh sidecars conflict by construction. |
+| Portable by fallback, honest about limits | Redirects install via iptables where available and fall back to native nftables (Firecracker-style guest kernels). gVisor does not support the nftables features either path needs, so neither works there — Kata is the supported secure runtime — and transparent mesh sidecars conflict by construction. |
## Transparent TLS interception (L7)
diff --git a/docs/guides/egress-upstream-proxy.md b/docs/guides/egress-upstream-proxy.md
index 8afdf52a6..e391248fc 100644
--- a/docs/guides/egress-upstream-proxy.md
+++ b/docs/guides/egress-upstream-proxy.md
@@ -77,7 +77,7 @@ Compatibility reflects current upstream `main` only:
| Docker (`runc`) | Supported | Requires `[docker] network_mode = "bridge"`; use `ca_cert_path` only when a private CA is required. |
| Kubernetes (`runc`) | Supported | Non-pooled AgentSandbox and BatchSandbox workloads; use `ca_secret_name` only when a private CA is required. |
| Kubernetes (`Kata`) | Supported | Accepted by egress validation; validate the selected RuntimeClass, CNI, and kernel in your deployment. |
-| gVisor (`secure_runtime.type = "gvisor"`) | Rejected | Sandbox creation returns HTTP 400 because gVisor netstack lacks the required iptables nat redirect. |
+| gVisor (`secure_runtime.type = "gvisor"`) | Rejected | Sandbox creation returns HTTP 400 because gVisor does not support the nftables interface the egress sidecar uses for its redirect and policy rules. |
| Kubernetes Pool Mode | Unsupported | Rejects per-request `networkPolicy`; cannot attach per-allocation egress sidecars. |
| Fast Sandbox | Unsupported | Current fast-sandbox profile rejects `networkPolicy` when `upstream_proxy` is set. |
diff --git a/docs/guides/secure-container.md b/docs/guides/secure-container.md
index 33878c4e0..6482bf4b3 100644
--- a/docs/guides/secure-container.md
+++ b/docs/guides/secure-container.md
@@ -1,6 +1,6 @@
---
title: Secure Container Runtime
-description: Use gVisor, Kata Containers, and Firecracker microVMs for hardware-level sandbox isolation.
+description: Use gVisor, Kata Containers, and Firecracker microVMs for stronger sandbox isolation.
---
# Secure Container Runtime Guide
@@ -47,7 +47,7 @@ OpenSandbox is designed to execute untrusted code generated by AI models (Claude
OpenSandbox supports the following secure runtime types through server-level configuration:
-- `"gvisor"` - Google gVisor with runsc
+- `"gvisor"` - gVisor with runsc
- `"kata"` - Kata Containers with QEMU hypervisor (default)
- `"firecracker"` - Kata Containers with Firecracker hypervisor
- `""` (empty) - Standard runc (default, no secure runtime)
@@ -372,6 +372,12 @@ Use the `runsc install` command to automatically configure Docker daemon:
sudo runsc install
```
+To keep Docker snapshots working (see the warning below), pass `--overlay2=none` through to runsc:
+
+```bash
+sudo runsc install -- --overlay2=none
+```
+
Or manually edit `/etc/docker/daemon.json`:
```json
@@ -380,8 +386,6 @@ Or manually edit `/etc/docker/daemon.json`:
"runsc": {
"path": "/usr/bin/runsc",
"runtimeArgs": [
- "--platform=systrap",
- "--network=host",
"--overlay2=none"
]
}
@@ -601,7 +605,7 @@ Kubernetes mode supports secure runtimes through RuntimeClass resources.
#### Step 1: Install gVisor Components on All Nodes
-For Kubernetes with containerd, you need to install **two** components:
+For Kubernetes with containerd, every node needs **two** binaries, both part of the `runsc` package:
1. **runsc** - the gVisor OCI runtime
2. **containerd-shim-runsc-v1** - the containerd shim for gVisor
@@ -613,8 +617,8 @@ echo "deb [signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://sto
sudo tee /etc/apt/sources.list.d/gvisor.list
sudo apt-get update
-# Install both gVisor components
-sudo apt-get install -y runsc containerd-shim-runsc-v1
+# Install gVisor
+sudo apt-get install -y runsc
# Verify installation
runsc --version
@@ -624,20 +628,32 @@ containerd-shim-runsc-v1 --version
#### Step 2: Configure containerd
-Edit `/etc/containerd/config.toml`:
+Edit `/etc/containerd/config.toml` (containerd 1.x, config `version = 2`):
```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
- runtime_type = "io.containerd.runsc.v1"
- [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
- TypeUrl = "io.containerd.runsc.v1.options"
- ConfigPath = "/etc/containerd/runsc.toml"
+ runtime_type = "io.containerd.runsc.v1"
+```
+
+For containerd 2.x (config `version = 3`), the section is `[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runsc]`.
+
+To pass flags to runsc, point the shim at a config file and put the flags under `[runsc_config]`. The shim silently ignores other section names. For example:
+
+```toml
+[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
+ runtime_type = "io.containerd.runsc.v1"
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
+ TypeUrl = "io.containerd.runsc.v1.options"
+ ConfigPath = "/etc/containerd/runsc.toml"
```
```bash
sudo tee /etc/containerd/runsc.toml > /dev/null <<'EOF'
-[runsc]
- platform = "ptrace"
+[runsc_config]
+ # Each key is passed to runsc as --<key>=<value>.
+ # For example, here is how to enable nvproxy for GPU functionality.
+ # This is equivalent to `--nvproxy=true`.
+ nvproxy = "true"
EOF
```
@@ -656,9 +672,6 @@ kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
-scheduling:
- nodeSelector:
- kubernetes.io/arch: amd64
```
```bash
@@ -863,7 +876,7 @@ If validation fails, the server refuses to start with a clear error message.
| Production AI Code Execution | gVisor | Good balance of security and performance |
| High-Security Requirements | Kata (QEMU) | Maximum isolation, full compatibility |
| High-Density Multi-Tenant | Kata (Firecracker) | Minimal memory overhead per sandbox |
-| Untrusted Network Code | gVisor or Kata | Syscall filtering prevents network attacks |
+| Untrusted Network Code | gVisor or Kata | Network traffic is handled by gVisor's netstack or a guest Linux kernel, not the host's TCP/IP stack |
---
@@ -899,7 +912,7 @@ kubectl apply -f gvisor-runtimeclass.yaml
**Cause**: gVisor doesn't implement all syscalls. Some applications may not be compatible.
-**Solution**: Check the [gVisor compatibility guide](https://gvisor.dev/docs/user_guide/compatibility/). Try using Kata (QEMU) which has better compatibility.
+**Solution**: Check the [gVisor compatibility guide](https://gvisor.dev/application-compatibility/). To find the failing syscall, temporarily add `--debug --debug-log=/tmp/runsc/ --strace` to the runsc flags and look for `unsupported syscall` messages in the logs ([gVisor debugging guide](https://gvisor.dev/docs/user_guide/debugging/)). If the workload needs unimplemented kernel features, use Kata (QEMU), which runs a full Linux guest kernel.
#### 4. Pod Stuck in ContainerCreating
@@ -938,12 +951,13 @@ iptables v1.8.9 (legacy): can't initialize iptables table 'nat': Table does not
| Feature | runc | gVisor | Kata (QEMU) | Kata (CLH) | Kata (FC) |
|---------|------|--------|-------------|------------|-----------|
| Syscall Compatibility | Full | Partial | Full | Full | Limited |
-| GPU Support | Yes | No | Yes | Yes | No |
+| GPU Support | Yes | Yes\* | Yes | Yes | No |
| IPv6 | Yes | Yes | Yes | Yes | Yes |
-| Privileged Mode | Yes | No | Yes | Yes | No |
+| Privileged Mode | Yes | Yes | Yes | Yes | No |
| Docker Volume | Yes | Yes | Yes | Yes | Yes |
-| Systemd | Yes | No | Yes | Yes | No |
-| iptables `nat` table (egress sidecar) | Yes | **No** | Yes | Yes | Yes |
+| Systemd | Yes | Yes\* | Yes | Yes | No |
+
+`*`: Requires configuration.
### Getting Help
diff --git a/kubernetes/Makefile b/kubernetes/Makefile
index bd20026d1..a0aac5217 100644
--- a/kubernetes/Makefile
+++ b/kubernetes/Makefile
@@ -186,7 +186,7 @@ setup-gvisor: download-gvisor ## Set up Kind cluster with gVisor (runsc) for e2e
"$(KIND)" create cluster --config -
@echo "Creating runsc.toml on Kind nodes..."
@for node in $$(docker ps --filter "name=$(GVISOR_KIND_CLUSTER)-" --format "{{.Names}}"); do \
- docker exec $$node sh -c 'mkdir -p /etc/containerd && echo "[runsc]" > /etc/containerd/runsc.toml && echo " platform = \"ptrace\"" >> /etc/containerd/runsc.toml'; \
+ docker exec $$node sh -c 'mkdir -p /etc/containerd && echo "[runsc_config]" > /etc/containerd/runsc.toml'; \
done
.PHONY: cleanup-gvisor
diff --git a/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go b/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go
index 51ad30381..ef920968a 100644
--- a/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go
+++ b/kubernetes/test/e2e_runtime/gvisor/gvisor_test.go
@@ -139,7 +139,7 @@ spec:
}
})
- It("should fail to start the egress sidecar under gVisor due to missing iptables nat table", func() {
+ It("should fail to start the egress sidecar under gVisor due to missing nftables support", func() {
egressImage := os.Getenv("EGRESS_IMG")
if egressImage == "" {
Skip("EGRESS_IMG not set; skipping gVisor + egress incompatibility test")
@@ -187,7 +187,7 @@ spec:
g.Expect(output).NotTo(Equal("0"), "egress container should exit with non-zero code")
}, 2*time.Minute).Should(Succeed())
- By("verifying egress logs mention iptables nat failure")
+ By("verifying egress logs mention the nftables failure")
output, err := runKubectl("logs", podName, "-n", testNamespace, "-c", "egress")
Expect(err).NotTo(HaveOccurred())
Expect(output).To(SatisfyAny(
diff --git a/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml b/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml
index 62e2e4e63..e8f26c81d 100644
--- a/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml
+++ b/kubernetes/test/e2e_runtime/gvisor/testdata/runtimeclass.yaml
@@ -3,6 +3,3 @@ kind: RuntimeClass
metadata:
name: gvisor
handler: runsc
-scheduling:
- nodeSelector:
- kubernetes.io/arch: amd64
diff --git a/oseps/0004-secure-container-runtime.md b/oseps/0004-secure-container-runtime.md
index 7757e5d1f..10f18f58a 100644
--- a/oseps/0004-secure-container-runtime.md
+++ b/oseps/0004-secure-container-runtime.md
@@ -443,8 +443,9 @@ Create the gVisor configuration file:
```bash
sudo tee /etc/containerd/runsc.toml > /dev/null <<'EOF'
-[runsc]
- platform = "ptrace"
+[runsc_config]
+ # Example configuration flag to enable GPU support:
+ nvproxy = "true"
EOF
```
diff --git a/oseps/0013-isolated-execution-api.md b/oseps/0013-isolated-execution-api.md
index d2d7d964d..b4ccb59c1 100644
--- a/oseps/0013-isolated-execution-api.md
+++ b/oseps/0013-isolated-execution-api.md
@@ -183,7 +183,7 @@ execd proxies filesystem operations outside the bwrap namespace, simulating the
| Mount order sensitivity — argv builder is a bug-prone area | Fixed segment order with unit test coverage for every segment |
| seccomp blocklist only — does not prevent 0-day syscalls | Defense-in-depth; primary boundary remains namespace isolation |
| No per-execution cgroup — CPU/memory/pids quota falls to outer Pod | Accept for v1; cgroup wrapper planned for v2 |
-| gVisor mount restrictions — commit unavailable | Capabilities endpoint marks `commit_supported = false`; diff and isolated execution still work |
+| Mount restrictions — commit unavailable | Capabilities endpoint marks `commit_supported = false`; diff and isolated execution still work |
| Overlay upper not persistent across Pods | Upper lives on Pod filesystem; cross-Pod persistence requires caller-side upload to object storage |
## Design Details
@@ -341,7 +341,7 @@ AWS_*, ALI_*, ALIYUN_*, K8S_*, KUBE_*
- `available = false`: all `/v1/isolated/*` write operations return `503 Service Unavailable`. Callers should fall back to the existing API or error. **SDK does not silently fall back.**
- `persist.available = false`: `--isolation-upper-root` emptyDir not mounted. diff/commit/`persist.enabled=true` unavailable; isolated execution still works (upper on tmpfs).
-- `commit_supported = false`: no mount permission (typical in gVisor). diff still works; commit returns 503.
+- `commit_supported = false`: no mount permission (can be caused by missing `CAP_SYS_ADMIN` or a restrictive seccomp/AppArmor profile). diff still works; commit returns 503.
### 6. Isolator Interface
@@ -434,7 +434,7 @@ Handles whiteout (character device 0,0 → delete corresponding workspace path)
v1 supports only `strategy = overwrite`. `skip-existing`, `fail-on-conflict`, and selective commit (path whitelist) are deferred to v2.
-gVisor typically cannot mount overlayfs — commit is marked `commit_supported = false` in capabilities; diff and isolated execution remain available.
+Where overlayfs cannot be mounted (either missing `CAP_SYS_ADMIN` or a restrictive seccomp/AppArmor profile), commit is marked `commit_supported = false` in capabilities; diff and isolated execution remain available.
Diff output: `GET .../diff` streams tar.gz via `Transfer-Encoding: chunked`. execd flag `--isolation-diff-max-bytes` (default 4 GiB) rejects oversized downloads with 413.
@@ -528,7 +528,7 @@ Cost: execd binary grows by ~800KB (negligible).
- `/tmp` isolation: files written to `/tmp` in one session are invisible to another.
- Filesystem proxy: upload → download round-trip through overlay; delete creates whiteout; search merges upper + lower.
- `persist.enabled = false`: upper destroyed on session close; diff returns 404.
-- `commit_supported = false` (simulated gVisor): commit returns 503; diff works.
+- `commit_supported = false` (simulated mount denial): commit returns 503; diff works.
- Multi-session: two concurrent sessions have independent namespaces.
- Idle GC: session auto-destroyed after timeout; `run` resets timer.
@@ -565,7 +565,7 @@ cgroup provides resource limits but not filesystem or PID isolation. bwrap names
bwrap natively supports `--unshare-user` with `--uid`/`--gid` for user namespace UID mapping. This would allow execd to run as non-root. Rejected because:
-- Nested user namespaces require CRI/runtime support (`user.max_user_namespaces`, runtime seccomp profiles allowing `clone(CLONE_NEWUSER)`). gVisor and hardened runtimes often restrict this.
+- Nested user namespaces require CRI/runtime support (`user.max_user_namespaces`, runtime seccomp profiles allowing `clone(CLONE_NEWUSER)`). Hardened runtimes often restrict this.
- User namespace UID mapping is virtual — all sessions map to the same host UID, so file permissions do not provide real isolation between sessions.
- Real setuid provides stronger guarantees: different sessions have different host UIDs, so file permission checks are enforced by the kernel at the host level.
- The tradeoff (execd must run as root) is acceptable because execd already runs in a dedicated sandbox container, not on shared infrastructure.
diff --git a/server/opensandbox_server/services/runtime_resolver.py b/server/opensandbox_server/services/runtime_resolver.py
index 88a28e36e..3bf2cd41e 100644
--- a/server/opensandbox_server/services/runtime_resolver.py
+++ b/server/opensandbox_server/services/runtime_resolver.py
@@ -258,7 +258,7 @@ def _warn_gvisor_egress_incompatibility(config: "AppConfig") -> None:
if config.secure_runtime and config.secure_runtime.type == "gvisor" and egress_image:
logger.warning(
"gVisor runtime is configured with egress sidecar image. "
- "The egress sidecar's iptables nat-based DNS redirect is incompatible with gVisor. "
+ "The egress sidecar's nftables-based DNS redirect and policy rules are incompatible with gVisor. "
"Sandboxes created with network_policy will be rejected at creation time."
)
diff --git a/server/opensandbox_server/services/validators.py b/server/opensandbox_server/services/validators.py
index 922f8d1f8..f85c5c86a 100644
--- a/server/opensandbox_server/services/validators.py
+++ b/server/opensandbox_server/services/validators.py
@@ -611,10 +611,7 @@ def ensure_egress_runtime_compatible(
effective_runtime_class: Optional[str] = None,
) -> None:
"""
- Reject network_policy when the secure runtime lacks iptables nat table support.
-
- gVisor's netstack does not implement the iptables nat table, which the egress
- sidecar requires for DNS redirect (REDIRECT target on port 53).
+ Reject network_policy when the secure runtime cannot run the egress sidecar.
"""
if not network_policy:
return
@@ -632,7 +629,7 @@ def ensure_egress_runtime_compatible(
"code": SandboxErrorCodes.INVALID_PARAMETER,
"message": (
f"networkPolicy is not compatible with runtime '{runtime_type}': "
- f"gVisor does not support the iptables nat table required by the egress sidecar. "
+ f"gVisor does not support the nftables rules required by the egress sidecar. "
f"Use a compatible runtime (e.g. kata) or remove networkPolicy."
),
},
diff --git a/server/tests/test_runtime_resolver.py b/server/tests/test_runtime_resolver.py
index 1c76c1ee0..0098af854 100644
--- a/server/tests/test_runtime_resolver.py
+++ b/server/tests/test_runtime_resolver.py
@@ -190,7 +190,7 @@ async def test_validate_startup_warns_gvisor_with_egress() -> None:
await validate_secure_runtime_on_startup(config, k8s_client=k8s_client)
mock_logger.warning.assert_called_once()
- assert "iptables nat" in mock_logger.warning.call_args[0][0]
+ assert "nftables" in mock_logger.warning.call_args[0][0]
@pytest.mark.asyncio