Skip to content

Agent Mesh

nano-init: route the sandbox instead of asking it to cooperate

Antonio Ojea3520b2b3d368
diff --git a/.goreleaser.yaml b/.goreleaser.yaml
index df1d8120..1671e44e 100644
--- a/.goreleaser.yaml
+++ b/.goreleaser.yaml
@@ -72,7 +72,10 @@ builds:
ldflags:
- -s -w
- id: nano-init
- main: ./cmd/nano-init
+ main: .
+ # nano-init is its own module: it carries a userspace TCP stack, and that
+ # dependency has no business in the graph every other binary builds from.
+ dir: ./cmd/nano-init
binary: nano-init
env:
- CGO_ENABLED=0
diff --git a/Dockerfile.nano-init b/Dockerfile.nano-init
index caf9c8a8..1e8685d8 100644
--- a/Dockerfile.nano-init
+++ b/Dockerfile.nano-init
@@ -2,15 +2,13 @@
FROM golang:1.26.6 AS builder
RUN apt-get update && apt-get install -y gcc libc-dev
WORKDIR /app
-COPY go.mod go.sum ./
+COPY cmd/nano-init/go.mod cmd/nano-init/go.sum ./
RUN go mod download
-COPY . .
-RUN gcc -shared -fPIC -ldl -o libinterceptor.so cmd/nano-init/interceptor/interceptor.c
-RUN CGO_ENABLED=0 GOOS=linux go build -buildvcs=false -o nano-init ./cmd/nano-init
+COPY cmd/nano-init/ ./
+RUN CGO_ENABLED=0 GOOS=linux go build -buildvcs=false -ldflags="-s -w" -o nano-init .
# Stage 2: Final
FROM gcr.io/distroless/base:nonroot
COPY --from=builder --chown=nonroot:nonroot /app/nano-init /
-COPY --from=builder --chown=nonroot:nonroot /app/libinterceptor.so /opt/sam/libinterceptor.so
USER nonroot:nonroot
ENTRYPOINT ["/nano-init"]
diff --git a/Makefile b/Makefile
index 59b44b46..830417e4 100644
--- a/Makefile
+++ b/Makefile
@@ -1,8 +1,6 @@
REPO_ROOT:=${CURDIR}
OUT_DIR=$(REPO_ROOT)/bin
-# libinterceptor.so is declared before "build" for its file-based prerequisite
-# rule; without this, plain "make" would default to that first rule instead.
.DEFAULT_GOAL := build
# disable CGO by default for static binaries
@@ -24,19 +22,17 @@ ANDROID_CC_ARM64=$(ANDROID_NDK_TOOLCHAIN)/aarch64-linux-android30-clang
ANDROID_CC_X86_64=$(ANDROID_NDK_TOOLCHAIN)/x86_64-linux-android30-clang
-$(OUT_DIR)/libinterceptor.so: cmd/nano-init/interceptor/interceptor.c
- mkdir -p "$(OUT_DIR)"
- gcc -shared -fPIC -ldl -o "$(OUT_DIR)/libinterceptor.so" cmd/nano-init/interceptor/interceptor.c
-
-build: $(OUT_DIR)/libinterceptor.so
+build:
go build -v -o "$(OUT_DIR)/sam-node" ./cmd/sam-node
go build -v -o "$(OUT_DIR)/sam-control-plane" ./cmd/sam-control-plane
go build -v -o "$(OUT_DIR)/sam-router" ./cmd/sam-router
go build -v -o "$(OUT_DIR)/mcp-client" ./cmd/mcp-client
- go build -v -o "$(OUT_DIR)/nano-init" ./cmd/nano-init
go build -v -o "$(OUT_DIR)/sam-box" ./cmd/sam-box
go build -v -o "$(OUT_DIR)/sam-bench" ./cmd/sam-bench
go build -v -o "$(OUT_DIR)/sam-console" ./cmd/sam-console
+ # nano-init is a separate module: it carries a userspace TCP stack, which
+ # has no business in the dependency graph every other binary builds from.
+ go -C cmd/nano-init build -v -o "$(OUT_DIR)/nano-init" .
.PHONY: mobile-ffi-host mobile-ffi-android mobile-ffi-android-x86_64 mobile-ffi-ios mobile-ffi mobile-app-apk mobile-app-apk-emulator
@@ -134,6 +130,7 @@ kind-e2e-mesh: build
test:
CGO_ENABLED=1 go test -v -race -count 1 $(if $(WHAT),-run $(WHAT)) ./...
+ go -C cmd/nano-init test -race -count 1 $(if $(WHAT),-run $(WHAT)) ./...
.PHONY: test-python test-python-e2e
test-python:
diff --git a/cmd/nano-init/.gitignore b/cmd/nano-init/.gitignore
new file mode 100644
index 00000000..8f3b2a4c
--- /dev/null
+++ b/cmd/nano-init/.gitignore
@@ -0,0 +1,2 @@
+# `go build` in this directory drops the binary here; it belongs in bin/.
+/nano-init
diff --git a/cmd/nano-init/README.md b/cmd/nano-init/README.md
index a96b6c70..a4db038f 100644
--- a/cmd/nano-init/README.md
+++ b/cmd/nano-init/README.md
@@ -1,34 +1,82 @@
# nano-init
-`nano-init` is a minimal init system designed to run as PID 1 in a container, while transparently proxying HTTP/HTTPS traffic to a Unix Domain Socket (UDS).
+PID 1 in an agent sandbox. It gives the sandbox one route, which leads to the
+boundary, and then gets out of the agent's way.
## What it does
-1. **PID 1 Duties (Zombie Reaping):** It correctly handles `SIGCHLD` to reap any zombie child processes spawned during the container's lifecycle.
-2. **UDS to TCP Proxy:** It spins up a transparent TCP listener on a random localhost port and securely proxies all traffic over a given Unix Domain Socket.
-3. **Process Management:** It wraps your target application (e.g., an agent), automatically injecting the standard `HTTP_PROXY` and `HTTPS_PROXY` environment variables pointing to the dynamic TCP port it opened.
-4. **Signal Propagation:** It intercepts termination signals (`SIGINT`, `SIGTERM`, `SIGQUIT`) and gracefully propagates them to the child process group to orchestrate clean shutdowns.
-5. **Exit Code Passthrough:** When the target process exits, `nano-init` catches its exit code and exits with the same code.
+1. **Builds the only way out.** Creates `tun0` over netlink, gives it a
+ link-local address, and makes it the default route. There is no other
+ interface in the sandbox, so this is not the preferred path out; it is the
+ only one.
+2. **Carries a TCP stack.** Terminates the sandbox's TCP in userspace via
+ gVisor (through `tun2socks`) and opens a SOCKS5 flow to the boundary for
+ each connection.
+3. **Keeps the name.** Answers DNS with a placeholder address per name and
+ remembers the pairing, so what reaches the boundary is `mesh.sam.alt` rather
+ than an address. The boundary chooses a provider from the name, which is the
+ entire reason the name has to survive the trip.
+4. **PID 1 duties.** Reaps orphans, propagates `SIGINT`/`SIGTERM`/`SIGQUIT` to
+ the child's process group, and exits with the agent's own status.
-## Why it is needed
+It is a separate Go module. A userspace TCP stack is a large dependency and has
+no business in the graph every other SAM binary builds from.
-Many applications and standard libraries lack native support for using a Unix Domain Socket as an HTTP/HTTPS proxy. By running `nano-init` as the entrypoint, the containerized application can simply use standard TCP-based `HTTP_PROXY` environment variables to securely route traffic out through the node's Unix Domain Socket, without needing any code modifications or complex network configurations.
+## What it deliberately does not do
-### Architecture & Rationale
+It does not touch the agent. No `HTTP_PROXY` in its environment, no CA bundle
+injected, nothing preloaded into its address space.
-When deploying agent sandboxes, runtimes like gVisor, Kata Containers, or Docker with `network:none` are used to completely isolate the network namespace. In this isolated environment, we securely bind mount a Unix Domain Socket into the sandbox. `nano-init` then acts as the primary init process, forwarding all the agent's HTTP traffic to that bind-mounted socket.
+That is a reversal. This program used to do all three, and argued for it: route
+everything through an HTTP proxy, the reasoning went, because HTTP has
+well-established ways to assert identity, and supporting arbitrary L3/L4 would
+mean building a network stack.
+
+The objection is not that it was inelegant. It is that **every one of those
+mechanisms is a request for the agent's cooperation.** `HTTP_PROXY` works if
+the client library reads it. `LD_PRELOAD` works if the binary has a dynamic
+loader. Both are outside the boundary the moment an agent uses a library that
+ignores the convention, spawns a subprocess that clears its environment, or
+speaks something that is not HTTP. An agent that has to cooperate with its own
+confinement is not confined — and an agent driven by a model, acting on text it
+did not write, is exactly the case where you cannot assume cooperation.
+
+Routing does not ask. The cost is the network stack the old rationale wanted to
+avoid, which is why this uses gVisor's rather than writing one: retransmission,
+windowing and teardown are easy to get subtly wrong, and the symptom is tail
+latency under load.
+
+Name resolution is the one piece that looks like the old design and is not. The
+resolver here is a convenience for clients that look a name up before
+connecting; it is not a control. An agent that ignores it and hardcodes another
+resolver has its packets routed through the tun regardless, and reaches exactly
+what policy allows.
-This approach is highly opinionated by design:
-* **Simplified Auth/Authz:** HTTP has a long, well-established list of mechanisms for authentication and authorization. Funneling traffic through an HTTP proxy simplifies how we assert identity and enforce access control for agents.
-* **Scalability vs L3/L4:** Supporting arbitrary L3 or L4 protocols directly is complex and difficult to scale securely. By enforcing HTTP proxying at the boundary, we avoid building a custom L3/L4 stack and instead leverage our existing network architecture to provide a highly scalable authentication and authorization framework.
## Usage
```bash
-./nano-init <uds-path> <cmd> [args...]
+nano-init run <boundary-socket> <cmd> [args...]
```
-Example:
+```bash
+nano-init run /run/agent.sock python agent.py "summarise the open issues"
+```
+
+Needs `NET_ADMIN` and `/dev/net/tun` to build the tun. In a container:
```bash
-./nano-init /var/run/sam.sock ./my-agent --flag1 value
+docker run --rm --network none \
+ --cap-add NET_ADMIN --device /dev/net/tun \
+ -v /run/sam/agent.sock:/run/agent.sock \
+ my-agent-image
```
+
+`nano-init copy <dest>` writes the binary somewhere else, for building a sandbox
+image that has nothing else in it.
+
+## See also
+
+- [Running agents on SAM](https://sam-mesh.dev/docs/user/running-agents/) — the
+ full picture, including the microVM arrangement
+- [Agent architecture](https://sam-mesh.dev/docs/agent-architecture/) — why the
+ boundary speaks SOCKS5
diff --git a/cmd/nano-init/boundary.go b/cmd/nano-init/boundary.go
new file mode 100644
index 00000000..67e9d369
--- /dev/null
+++ b/cmd/nano-init/boundary.go
@@ -0,0 +1,197 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "encoding/binary"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "strconv"
+ "time"
+
+ M "github.com/xjasonlyu/tun2socks/v2/metadata"
+)
+
+// Everything the sandbox sends leaves through here, as a SOCKS5 CONNECT naming
+// a destination. The boundary on the far side decides whether that destination
+// is allowed; this end decides nothing, which is deliberate. A sandbox that
+// could rule on its own traffic would be ruling on behalf of the agent it
+// contains.
+
+const (
+ socks5Version = 0x05
+ authNone = 0x00
+ cmdConnect = 0x01
+ atypIPv4 = 0x01
+ atypDomain = 0x03
+ atypIPv6 = 0x04
+ replySucceeded = 0x00
+
+ boundaryDialTimeout = 30 * time.Second
+)
+
+// boundaryProxy carries sandbox flows to sam-box, restoring the name the agent
+// asked for. It satisfies tun2socks's proxy.Proxy.
+type boundaryProxy struct {
+ socket string
+ resolver *resolver
+}
+
+// DialContext opens one flow to the boundary.
+func (b *boundaryProxy) DialContext(ctx context.Context, meta *M.Metadata) (net.Conn, error) {
+ // The address gvisor saw is a placeholder standing for a name, unless the
+ // agent dialled a literal address. Either way what arrives at the boundary
+ // is what the agent actually asked for.
+ host := meta.DstIP.String()
+ if name, ok := b.resolver.nameFor(meta.DstIP); ok {
+ host = name
+ }
+
+ conn, err := dialBoundary(ctx, b.socket)
+ if err != nil {
+ return nil, fmt.Errorf("dial boundary %s: %w", b.socket, err)
+ }
+
+ if deadline, ok := ctx.Deadline(); ok {
+ _ = conn.SetDeadline(deadline)
+ }
+ if err := connect(conn, host, meta.DstPort); err != nil {
+ _ = conn.Close()
+ return nil, err
+ }
+ // The handshake deadline must not become the flow's deadline: an agent
+ // holding a long-lived stream would otherwise be cut off mid-conversation.
+ _ = conn.SetDeadline(time.Time{})
+
+ return conn, nil
+}
+
+// DialUDP refuses. The boundary does not offer UDP ASSOCIATE, by design: a
+// datagram path would be a second way out with its own rules, and one way out
+// is the property the whole arrangement rests on.
+func (b *boundaryProxy) DialUDP(*M.Metadata) (net.PacketConn, error) {
+ return nil, errors.New("nano-init: the boundary carries TCP only")
+}
+
+// connect performs the SOCKS5 handshake for one destination.
+func connect(conn net.Conn, host string, port uint16) error {
+ if _, err := conn.Write([]byte{socks5Version, 1, authNone}); err != nil {
+ return fmt.Errorf("socks greeting: %w", err)
+ }
+
+ greeting := make([]byte, 2)
+ if _, err := io.ReadFull(conn, greeting); err != nil {
+ return fmt.Errorf("socks greeting reply: %w", err)
+ }
+ if greeting[0] != socks5Version || greeting[1] != authNone {
+ return fmt.Errorf("socks greeting refused: %v", greeting)
+ }
+
+ request, err := connectRequest(host, port)
+ if err != nil {
+ return err
+ }
+ if _, err := conn.Write(request); err != nil {
+ return fmt.Errorf("socks connect: %w", err)
+ }
+
+ return readReply(conn)
+}
+
+// connectRequest encodes a CONNECT for a name or an address.
+func connectRequest(host string, port uint16) ([]byte, error) {
+ request := []byte{socks5Version, cmdConnect, 0x00}
+
+ switch ip := net.ParseIP(host); {
+ case ip == nil:
+ // A name. This is the case that matters: the boundary resolves it.
+ if len(host) > 255 {
+ return nil, fmt.Errorf("destination name too long: %d bytes", len(host))
+ }
+ request = append(request, atypDomain, byte(len(host)))
+ request = append(request, host...)
+ case ip.To4() != nil:
+ request = append(request, atypIPv4)
+ request = append(request, ip.To4()...)
+ default:
+ request = append(request, atypIPv6)
+ request = append(request, ip.To16()...)
+ }
+
+ return binary.BigEndian.AppendUint16(request, port), nil
+}
+
+// readReply consumes the SOCKS5 reply, including the bound address the
+// boundary reports but nothing here needs.
+func readReply(conn net.Conn) error {
+ head := make([]byte, 4)
+ if _, err := io.ReadFull(conn, head); err != nil {
+ return fmt.Errorf("socks reply: %w", err)
+ }
+ if head[0] != socks5Version {
+ return fmt.Errorf("socks reply version %d", head[0])
+ }
+ if head[1] != replySucceeded {
+ return fmt.Errorf("boundary refused: %s", replyMessage(head[1]))
+ }
+
+ var skip int
+ switch head[3] {
+ case atypIPv4:
+ skip = net.IPv4len
+ case atypIPv6:
+ skip = net.IPv6len
+ case atypDomain:
+ length := make([]byte, 1)
+ if _, err := io.ReadFull(conn, length); err != nil {
+ return fmt.Errorf("socks reply address: %w", err)
+ }
+ skip = int(length[0])
+ default:
+ return fmt.Errorf("socks reply address type %d", head[3])
+ }
+
+ if _, err := io.ReadFull(conn, make([]byte, skip+2)); err != nil {
+ return fmt.Errorf("socks reply address: %w", err)
+ }
+ return nil
+}
+
+// replyMessage turns a SOCKS5 failure into something an agent's logs can be
+// read against. "Not allowed" is a policy decision and has to be legible as
+// one, rather than looking like the mesh being broken.
+func replyMessage(code byte) string {
+ switch code {
+ case 0x02:
+ return "not allowed by policy"
+ case 0x03:
+ return "network unreachable"
+ case 0x04:
+ return "host unreachable"
+ case 0x05:
+ return "connection refused"
+ case 0x06:
+ return "TTL expired"
+ case 0x07:
+ return "command not supported"
+ case 0x08:
+ return "address type not supported"
+ default:
+ return "general failure (" + strconv.Itoa(int(code)) + ")"
+ }
+}
diff --git a/cmd/nano-init/copy.go b/cmd/nano-init/copy.go
new file mode 100644
index 00000000..b33e0415
--- /dev/null
+++ b/cmd/nano-init/copy.go
@@ -0,0 +1,38 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "io"
+ "os"
+)
+
+func copyFile(src, dest string) error {
+ in, err := os.Open(src) // #nosec G304 -- the caller names its own binary
+ if err != nil {
+ return err
+ }
+ defer func() { _ = in.Close() }()
+
+ out, err := os.OpenFile(dest, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755) // #nosec G302,G304
+ if err != nil {
+ return err
+ }
+ if _, err := io.Copy(out, in); err != nil {
+ _ = out.Close()
+ return err
+ }
+ return out.Close()
+}
diff --git a/cmd/nano-init/go.mod b/cmd/nano-init/go.mod
new file mode 100644
index 00000000..6e77a65c
--- /dev/null
+++ b/cmd/nano-init/go.mod
@@ -0,0 +1,34 @@
+module github.com/google/sam/cmd/nano-init
+
+go 1.26.5
+
+require (
+ github.com/vishvananda/netlink v1.3.1
+ github.com/xjasonlyu/tun2socks/v2 v2.7.0
+ golang.org/x/net v0.58.0
+ golang.org/x/sys v0.47.0
+)
+
+require (
+ github.com/ajg/form v1.7.1 // indirect
+ github.com/docker/go-units v0.5.0 // indirect
+ github.com/go-chi/chi/v5 v5.3.0 // indirect
+ github.com/go-chi/cors v1.2.2 // indirect
+ github.com/go-chi/render v1.0.3 // indirect
+ github.com/go-gost/relay v0.6.1 // indirect
+ github.com/google/btree v1.1.3 // indirect
+ github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
+ github.com/google/uuid v1.6.0 // indirect
+ github.com/gorilla/schema v1.4.1 // indirect
+ github.com/gorilla/websocket v1.5.3 // indirect
+ github.com/vishvananda/netns v0.0.5 // indirect
+ go.uber.org/atomic v1.11.0 // indirect
+ go.uber.org/multierr v1.11.0 // indirect
+ go.uber.org/zap v1.28.0 // indirect
+ golang.org/x/crypto v0.55.0 // indirect
+ golang.org/x/exp v0.0.0-20260611194520-c48552f49976 // indirect
+ golang.org/x/time v0.15.0 // indirect
+ golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
+ golang.zx2c4.com/wireguard v0.0.0-20260522210424-ecfc5a8d5446 // indirect
+ gvisor.dev/gvisor v0.0.0-20260701204157-69c2d17aea96 // indirect
+)
diff --git a/cmd/nano-init/go.sum b/cmd/nano-init/go.sum
new file mode 100644
index 00000000..c5b18ad4
--- /dev/null
+++ b/cmd/nano-init/go.sum
@@ -0,0 +1,67 @@
+github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
+github.com/ajg/form v1.7.1 h1:OsnBDzTkrWdrxvEnO68I72ZVGJGNaMwPhoAm0V+llgc=
+github.com/ajg/form v1.7.1/go.mod h1:HL757PzLyNkj5AIfptT6L+iGNeXTlnrr/oDePGc/y7Q=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
+github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
+github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
+github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
+github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
+github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
+github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
+github.com/go-chi/render v1.0.3 h1:AsXqd2a1/INaIfUSKq3G5uA8weYx20FOsM7uSoCyyt4=
+github.com/go-chi/render v1.0.3/go.mod h1:/gr3hVkmYR0YlEy3LxCuVRFzEu9Ruok+gFqbIofjao0=
+github.com/go-gost/relay v0.6.1 h1:7SqnHFbY8x/DzvjpK03a5zcVH9+TbJAcnW/RT6s1ecc=
+github.com/go-gost/relay v0.6.1/go.mod h1:Dku0f5sfjOClrZFiDmQUrYYJ4uof7rnkCUBfsl0PSAI=
+github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
+github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
+github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
+github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gorilla/schema v1.4.1 h1:jUg5hUjCSDZpNGLuXQOgIWGdlgrIdYvgQ0wZtdK1M3E=
+github.com/gorilla/schema v1.4.1/go.mod h1:Dg5SSm5PV60mhF2NFaTV1xuYYj8tV8NOPRo4FggUMnM=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
+github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
+github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
+github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
+github.com/xjasonlyu/tun2socks/v2 v2.7.0 h1:fYEN0Q1sSanuoID8xvUTsMODbKLJJH/J5ywYoMRxIPw=
+github.com/xjasonlyu/tun2socks/v2 v2.7.0/go.mod h1:XHgJ3wbs63mZxGbJawt6jjNX65aPDl+Qhn2ePPEE4KI=
+go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
+go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
+go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
+go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
+go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo=
+go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q=
+go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
+go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
+golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
+golang.org/x/exp v0.0.0-20260611194520-c48552f49976 h1:X8Hz2ImujgbmetVuW+w2YkyZChE3cBpZi2P158rTG9M=
+golang.org/x/exp v0.0.0-20260611194520-c48552f49976/go.mod h1:vnf4pv9iKZXY58sQE1L86zmNWJ4159e1RkcWiLCkeEY=
+golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
+golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
+golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
+golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
+golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
+golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
+golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 h1:B82qJJgjvYKsXS9jeunTOisW56dUokqW/FOteYJJ/yg=
+golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2/go.mod h1:deeaetjYA+DHMHg+sMSMI58GrEteJUUzzw7en6TJQcI=
+golang.zx2c4.com/wireguard v0.0.0-20260522210424-ecfc5a8d5446 h1:cqHQ3AycTHvM2R7ikgyX57D+XvtcSnGylsLkOVhta/w=
+golang.zx2c4.com/wireguard v0.0.0-20260522210424-ecfc5a8d5446/go.mod h1:rpwXGsirqLqN2L0JDJQlwOboGHmptD5ZD6T2VmcqhTw=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gvisor.dev/gvisor v0.0.0-20260701204157-69c2d17aea96 h1:LZXOf4NwvAwUy/eI8P+Y3DUdYLyaCEDNPEjsL2OP2ro=
+gvisor.dev/gvisor v0.0.0-20260701204157-69c2d17aea96/go.mod h1:8aLQqUBHDH8fY5y60lzmwDpMMbQCcT3EBfoSwhfaGCY=
diff --git a/cmd/nano-init/interceptor/interceptor.c b/cmd/nano-init/interceptor/interceptor.c
deleted file mode 100644
index e9e25ce3..00000000
--- a/cmd/nano-init/interceptor/interceptor.c
+++ /dev/null
@@ -1,60 +0,0 @@
-#define _GNU_SOURCE
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <dlfcn.h>
-#include <sys/socket.h>
-#include <netinet/in.h>
-#include <arpa/inet.h>
-#include <errno.h>
-
-// Function pointer to the original connect syscall
-static int (*original_connect)(int sockfd, const struct sockaddr *addr, socklen_t addrlen) = NULL;
-
-static void init(void) __attribute__((constructor));
-static void init(void) {
- original_connect = dlsym(RTLD_NEXT, "connect");
-}
-
-int connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen) {
- if (!original_connect) {
- fprintf(stderr, "[interceptor] Error: original connect not initialized\n");
- errno = EFAULT;
- return -1;
- }
-
- struct sockaddr_storage modified_addr;
- const struct sockaddr *final_addr = addr;
-
- if (addr && (addr->sa_family == AF_INET || addr->sa_family == AF_INET6)) {
- char *proxy_port_str = getenv("SAM_PROXY_PORT");
- if (proxy_port_str) {
- int proxy_port = atoi(proxy_port_str);
- if (proxy_port > 0 && proxy_port < 65536) {
- if (addr->sa_family == AF_INET && addrlen >= sizeof(struct sockaddr_in)) {
- struct sockaddr_in *addr4 = (struct sockaddr_in *)addr;
- uint16_t port = ntohs(addr4->sin_port);
- if (port == 80 || port == 443) {
- memcpy(&modified_addr, addr, sizeof(struct sockaddr_in));
- struct sockaddr_in *m_addr4 = (struct sockaddr_in *)&modified_addr;
- m_addr4->sin_port = htons(proxy_port);
- m_addr4->sin_addr.s_addr = htonl(INADDR_LOOPBACK);
- final_addr = (const struct sockaddr *)&modified_addr;
- }
- } else if (addr->sa_family == AF_INET6 && addrlen >= sizeof(struct sockaddr_in6)) {
- struct sockaddr_in6 *addr6 = (struct sockaddr_in6 *)addr;
- uint16_t port = ntohs(addr6->sin6_port);
- if (port == 80 || port == 443) {
- memcpy(&modified_addr, addr, sizeof(struct sockaddr_in6));
- struct sockaddr_in6 *m_addr6 = (struct sockaddr_in6 *)&modified_addr;
- m_addr6->sin6_port = htons(proxy_port);
- m_addr6->sin6_addr = in6addr_loopback;
- final_addr = (const struct sockaddr *)&modified_addr;
- }
- }
- }
- }
- }
-
- return original_connect(sockfd, final_addr, addrlen);
-}
diff --git a/cmd/nano-init/main.go b/cmd/nano-init/main.go
index 4a003c77..484db41a 100644
--- a/cmd/nano-init/main.go
+++ b/cmd/nano-init/main.go
@@ -12,150 +12,203 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+// Command nano-init is PID 1 in an agent sandbox.
+//
+// It gives the sandbox one route, which leads to the boundary, and then gets
+// out of the agent's way.
+//
+// It used to do the opposite. It rewrote /etc/resolv.conf to point at a DNS
+// server it ran itself, answered lookups with addresses it invented, injected
+// HTTP_PROXY and friends into the agent's environment, and preloaded a shared
+// object into the agent's address space to catch the connections that got past
+// all that. Every one of those asks the agent to cooperate, and an agent that
+// has to cooperate with its own confinement is not confined: the next library
+// that ignores the proxy variables, the next subprocess that clears its
+// environment, the next static binary with no loader to preload into, each one
+// was outside the boundary.
+//
+// Routing does not ask. There is no interface in this sandbox except the tun,
+// and the tun goes to the boundary, so an agent that ignores every convention
+// here still reaches only what policy allowed. The resolver that remains is a
+// convenience for clients that look a name up before connecting, not a control:
+// an agent that resolves some other way is routed through the tun regardless.
+//
+// The TCP stack is gVisor's, via tun2socks. Writing one would mean writing
+// retransmission, windowing and teardown, and getting those subtly wrong shows
+// up as tail latency under load, which is exactly where this has to be
+// trusted.
package main
import (
"context"
"errors"
"fmt"
- "io"
"log"
"net"
- "net/http"
"os"
"os/exec"
"os/signal"
- "runtime"
- "strconv"
- "strings"
"syscall"
"time"
- "golang.org/x/net/dns/dnsmessage"
+ "github.com/vishvananda/netlink"
+ "github.com/xjasonlyu/tun2socks/v2/engine"
+ "github.com/xjasonlyu/tun2socks/v2/tunnel"
+)
+
+const (
+ tunName = "tun0"
+
+ // Addresses inside the sandbox are link-local because that is what these
+ // addresses are for: RFC 3927 describes a single link with no router, and
+ // a tun to the boundary is exactly that. A sandbox numbered out of
+ // 10.0.0.0/8 will eventually be deployed somewhere that already uses it.
+ tunIP = "169.254.1.1"
+ tunAddr = tunIP + "/30"
+
+ // The resolver answers on the tun's own address, which is the one address
+ // this sandbox is certain to have. Nothing outside can reach it: a
+ // link-local address is not routed anywhere.
+ dnsAddr = tunIP + ":53"
+
+ // Disjoint from tunAddr: an overlap would hand out the interface's own
+ // address as a name's answer, which fails in a way nobody enjoys reading.
+ virtualPool = "169.254.64.0/18"
)
func main() {
if len(os.Args) < 2 {
- printUsageAndExit()
+ usage()
}
- subCmd := os.Args[1]
- switch subCmd {
+ switch os.Args[1] {
case "copy":
if len(os.Args) != 3 {
- log.Fatalf("Usage: %s copy <dest>", os.Args[0])
+ log.Fatalf("usage: %s copy <dest>", os.Args[0])
}
src, err := os.Executable()
if err != nil {
src = "/nano-init"
}
- dest := os.Args[2]
- log.Printf("Copying %s to %s...", src, dest)
- if err := copyFile(src, dest); err != nil {
- log.Fatalf("Failed to copy binary: %v", err)
+ if err := copyFile(src, os.Args[2]); err != nil {
+ log.Fatalf("copy binary: %v", err)
}
- log.Println("Copy complete.")
- return
case "run":
if len(os.Args) < 4 {
- log.Fatalf("Usage: %s run <uds-path> <cmd> [args...]", os.Args[0])
+ log.Fatalf("usage: %s run <boundary-socket> <cmd> [args...]", os.Args[0])
}
- udsPath := os.Args[2]
- cmdName := os.Args[3]
- cmdArgs := os.Args[4:]
-
- runProxy(udsPath, cmdName, cmdArgs)
+ run(os.Args[2], os.Args[3], os.Args[4:])
default:
- printUsageAndExit()
+ usage()
}
}
-func printUsageAndExit() {
- log.Fatalf("Usage:\n %s copy <dest>\n %s run <uds-path> <cmd> [args...]", os.Args[0], os.Args[0])
+func usage() {
+ log.Fatalf("usage:\n %s copy <dest>\n %s run <boundary-socket> <cmd> [args...]",
+ os.Args[0], os.Args[0])
}
-func runProxy(udsPath, cmdName string, cmdArgs []string) {
+// run wires the sandbox up and hands it to the agent.
+func run(boundarySocket, cmdName string, cmdArgs []string) {
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
defer cancel()
- log.Printf("Bootstrapping CA certificate from UDS: %s", udsPath)
- if err := bootstrapCA(ctx, udsPath); err != nil {
- log.Fatalf("Bootstrap CA failed: %v", err)
+ if err := checkBoundary(boundarySocket); err != nil {
+ log.Fatalf("this sandbox has no way out: %v", err)
}
- log.Println("Bootstrap CA successfully written to /tmp/ephemeral_ca.pem")
- // 1b. Bootstrap Interceptor
- log.Println("Attempting to bootstrap socket interceptor from UDS gateway...")
- var bootstrappedInterceptor string
- if interceptor, err := bootstrapInterceptor(ctx, udsPath); err != nil {
- log.Printf("Warning: failed to bootstrap socket interceptor: %v. Running without transparent interception.", err)
- } else {
- bootstrappedInterceptor = interceptor
- log.Printf("Socket interceptor successfully written to %s", bootstrappedInterceptor)
+ names, err := newResolver(virtualPool)
+ if err != nil {
+ log.Fatalf("resolver: %v", err)
}
-
- // 2. DNS Spoofer
- dnsPort := os.Getenv("SAM_DNS_PORT")
- if dnsPort == "" {
- dnsPort = "53"
+ if err := setupNetwork(ctx, boundarySocket, names); err != nil {
+ log.Fatalf("set up sandbox network: %v", err)
}
- dnsAddr := "127.0.0.1:" + dnsPort
- if err := startDNSSpoofer(ctx, dnsAddr); err != nil {
- log.Fatalf("Failed to start DNS spoofer: %v", err)
+
+ os.Exit(runAgent(ctx, cancel, cmdName, cmdArgs))
+}
+
+// setupNetwork builds the only route out of the sandbox.
+//
+// This talks netlink rather than shelling out to `ip`, and carries its own TCP
+// stack rather than running a separate binary, so a sandbox image can be the
+// agent and nothing else. That is not tidiness: image size is what decides how
+// many agents fit on a host.
+func setupNetwork(ctx context.Context, boundarySocket string, names *resolver) error {
+ // As PID 1 in a microVM nothing else has done this, and a sandbox without
+ // loopback breaks things that have no business caring about the network.
+ if lo, err := netlink.LinkByName("lo"); err == nil {
+ _ = netlink.LinkSetUp(lo)
}
- log.Printf("DNS spoofer listening on UDP %s", dnsAddr)
- // Overwrite /etc/resolv.conf
- if err := os.WriteFile("/etc/resolv.conf", []byte("nameserver 127.0.0.1\n"), 0644); err != nil {
- log.Printf("Warning: failed to overwrite /etc/resolv.conf: %v", err)
- } else {
- log.Println("Overwrote /etc/resolv.conf to nameserver 127.0.0.1")
+ tun := &netlink.Tuntap{
+ LinkAttrs: netlink.LinkAttrs{Name: tunName},
+ Mode: netlink.TUNTAP_MODE_TUN,
+ }
+ if err := netlink.LinkAdd(tun); err != nil {
+ return fmt.Errorf("create %s: %w", tunName, err)
+ }
+ if err := netlink.LinkSetUp(tun); err != nil {
+ return fmt.Errorf("bring up %s: %w", tunName, err)
}
- // 3. TCP Forwarding
- listener4, err := net.Listen("tcp", "127.0.0.1:0")
+ addr, err := netlink.ParseAddr(tunAddr)
if err != nil {
- log.Fatalf("Failed to bind random TCP port: %v", err)
+ return fmt.Errorf("parse %s: %w", tunAddr, err)
+ }
+ if err := netlink.AddrAdd(tun, addr); err != nil {
+ return fmt.Errorf("address %s: %w", tunName, err)
}
- defer func() { _ = listener4.Close() }()
- assignedProxyPort := listener4.Addr().(*net.TCPAddr).Port
- log.Printf("Blind UDS-forwarder listening on 127.0.0.1:%d", assignedProxyPort)
- var listener6 net.Listener
- if l6, err := net.Listen("tcp", fmt.Sprintf("[::1]:%d", assignedProxyPort)); err == nil {
- listener6 = l6
- defer func() { _ = listener6.Close() }()
- log.Printf("Blind UDS-forwarder listening on [::1]:%d", assignedProxyPort)
- } else {
- log.Printf("Warning: failed to bind TCP port on IPv6 loopback: %v. Running IPv4-only forwarder.", err)
+ // A device route with no gateway: nothing on the far side of this link has
+ // an address worth naming, and everything goes the same way regardless.
+ // The destination has to be spelled out rather than left nil, which
+ // netlink reads as "no route specified at all".
+ if err := netlink.RouteAdd(&netlink.Route{
+ LinkIndex: tun.Attrs().Index,
+ Scope: netlink.SCOPE_LINK,
+ Dst: &net.IPNet{IP: net.IPv4zero, Mask: net.CIDRMask(0, 32)},
+ }); err != nil {
+ return fmt.Errorf("default route via %s: %w", tunName, err)
}
- serveListener := func(l net.Listener) {
- for {
- conn, err := l.Accept()
- if err != nil {
- select {
- case <-ctx.Done():
- return
- default:
- log.Printf("TCP Accept error on %s: %v", l.Addr(), err)
- time.Sleep(50 * time.Millisecond)
- continue
- }
- }
- go handleConnection(conn, udsPath)
- }
+ dns, err := net.ListenPacket("udp", dnsAddr)
+ if err != nil {
+ return fmt.Errorf("resolver on %s: %w", dnsAddr, err)
}
+ go names.serveDNS(dns)
+ go func() {
+ <-ctx.Done()
+ _ = dns.Close()
+ }()
- go serveListener(listener4)
- if listener6 != nil {
- go serveListener(listener6)
+ if err := os.WriteFile("/etc/resolv.conf", []byte("nameserver "+tunIP+"\n"), 0o644); err != nil {
+ // Not fatal: resolution is a convenience here, not the control.
+ log.Printf("could not write /etc/resolv.conf, name resolution may fail: %v", err)
}
- // 4. Execution of the target process
+ // direct:// is a placeholder the engine insists on. The real dialer is
+ // installed below, before the agent exists to send anything, so nothing
+ // can take the placeholder path.
+ engine.Insert(&engine.Key{
+ Device: "tun://" + tunName,
+ Proxy: "direct://",
+ LogLevel: "warn",
+ })
+ engine.Start()
+
+ tunnel.T().SetProxy(&boundaryProxy{socket: boundarySocket, resolver: names})
+ return nil
+}
+
+// runAgent starts the agent and reports the exit status it should be judged by.
+func runAgent(ctx context.Context, cancel context.CancelFunc, cmdName string, cmdArgs []string) int {
cmd := exec.CommandContext(ctx, cmdName, cmdArgs...)
+ cmd.Stdout, cmd.Stderr, cmd.Stdin = os.Stdout, os.Stderr, os.Stdin
+ cmd.Env = os.Environ() // Nothing injected: the agent is not configured, it is routed.
+ cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
if cmd.Process == nil {
return nil
@@ -164,260 +217,38 @@ func runProxy(udsPath, cmdName string, cmdArgs []string) {
}
cmd.WaitDelay = 5 * time.Second
- cmd.Stdout = os.Stdout
- cmd.Stderr = os.Stderr
- cmd.Stdin = os.Stdin
-
- // Inject environment variables
- interceptorPath := os.Getenv("SAM_INTERCEPTOR_PATH")
- if interceptorPath == "" {
- interceptorPath = bootstrappedInterceptor
- }
- cmd.Env = buildAgentEnv(assignedProxyPort, "/tmp/ephemeral_ca.pem", interceptorPath)
-
- cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
-
if err := cmd.Start(); err != nil {
- log.Fatalf("Failed to start agent: %v", err)
+ log.Fatalf("start agent: %v", err)
}
- agentExitStatus := make(chan syscall.WaitStatus, 1)
- setupPID1Duties(cmd.Process.Pid, agentExitStatus)
+ // As PID 1 this process inherits every orphan in the sandbox, so it has to
+ // reap them or the guest fills with zombies. Reaping also means Wait can
+ // lose the race for the agent's own status, hence the channel.
+ agentExit := make(chan syscall.WaitStatus, 1)
+ reapChildren(cmd.Process.Pid, agentExit)
waitErr := cmd.Wait()
cancel()
- var status syscall.WaitStatus
- hasStatus := false
-
if waitErr != nil && errors.Is(waitErr, syscall.ECHILD) {
- status = <-agentExitStatus
- hasStatus = true
- }
-
- if hasStatus {
+ status := <-agentExit
if status.Signaled() {
- os.Exit(128 + int(status.Signal()))
+ return 128 + int(status.Signal())
}
- os.Exit(status.ExitStatus())
+ return status.ExitStatus()
}
-
if waitErr != nil {
- if exitError, ok := waitErr.(*exec.ExitError); ok {
- os.Exit(exitError.ExitCode())
+ var exitErr *exec.ExitError
+ if errors.As(waitErr, &exitErr) {
+ return exitErr.ExitCode()
}
- os.Exit(1)
- }
- os.Exit(0)
-}
-
-func bootstrapCA(ctx context.Context, udsPath string) error {
- client := &http.Client{
- Transport: &http.Transport{
- DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
- return net.Dial("unix", udsPath)
- },
- },
- }
- req, err := http.NewRequestWithContext(ctx, "GET", "http://localhost/internal/bootstrap/ca.crt", nil)
- if err != nil {
- return err
- }
- resp, err := client.Do(req)
- if err != nil {
- return err
- }
- defer func() { _ = resp.Body.Close() }()
-
- if resp.StatusCode != http.StatusOK {
- return fmt.Errorf("unexpected status: %s", resp.Status)
- }
-
- caBytes, err := io.ReadAll(resp.Body)
- if err != nil {
- return err
- }
-
- return os.WriteFile("/tmp/ephemeral_ca.pem", caBytes, 0644)
-}
-
-func bootstrapInterceptor(ctx context.Context, udsPath string) (string, error) {
- client := &http.Client{
- Transport: &http.Transport{
- DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
- return net.Dial("unix", udsPath)
- },
- },
- Timeout: 5 * time.Second,
- }
-
- arch := runtime.GOARCH
- libc := detectLibc()
-
- urlStr := fmt.Sprintf("http://localhost/internal/bootstrap/libinterceptor.so?arch=%s&libc=%s", arch, libc)
- req, err := http.NewRequestWithContext(ctx, "GET", urlStr, nil)
- if err != nil {
- return "", err
- }
-
- resp, err := client.Do(req)
- if err != nil {
- return "", err
+ return 1
}
- defer func() { _ = resp.Body.Close() }()
-
- if resp.StatusCode != http.StatusOK {
- return "", fmt.Errorf("server returned status: %s", resp.Status)
- }
-
- soBytes, err := io.ReadAll(resp.Body)
- if err != nil {
- return "", err
- }
-
- targetPath := "/tmp/libinterceptor.so"
- if err := os.WriteFile(targetPath, soBytes, 0755); err != nil {
- return "", err
- }
-
- return targetPath, nil
+ return 0
}
-func detectLibc() string {
- paths := []string{
- "/lib/ld-musl-x86_64.so.1",
- "/lib/ld-musl-aarch64.so.1",
- "/lib/ld-musl-arm.so.1",
- }
- for _, p := range paths {
- if _, err := os.Stat(p); err == nil {
- return "musl"
- }
- }
-
- // Also scan directories as fallback
- if files, err := os.ReadDir("/lib"); err == nil {
- for _, f := range files {
- if strings.Contains(strings.ToLower(f.Name()), "musl") {
- return "musl"
- }
- }
- }
- if files, err := os.ReadDir("/lib64"); err == nil {
- for _, f := range files {
- if strings.Contains(strings.ToLower(f.Name()), "musl") {
- return "musl"
- }
- }
- }
-
- return "glibc"
-}
-
-func startDNSSpoofer(ctx context.Context, dnsAddr string) error {
- addr, err := net.ResolveUDPAddr("udp", dnsAddr)
- if err != nil {
- return err
- }
- conn, err := net.ListenUDP("udp", addr)
- if err != nil {
- return err
- }
- go func() {
- <-ctx.Done()
- _ = conn.Close()
- }()
-
- go func() {
- buf := make([]byte, 512)
- for {
- n, remoteAddr, err := conn.ReadFromUDP(buf)
- if err != nil {
- select {
- case <-ctx.Done():
- return
- default:
- log.Printf("DNS Server read error: %v", err)
- time.Sleep(50 * time.Millisecond)
- continue
- }
- }
-
- packet := make([]byte, n)
- copy(packet, buf[:n])
-
- go func(p []byte, rAddr *net.UDPAddr) {
- var msg dnsmessage.Message
- if err := msg.Unpack(p); err != nil {
- return
- }
-
- resp := dnsmessage.Message{
- Header: dnsmessage.Header{
- ID: msg.ID,
- Response: true,
- Authoritative: true,
- },
- Questions: msg.Questions,
- }
-
- if len(msg.Questions) > 0 {
- q := msg.Questions[0]
- switch q.Type {
- case dnsmessage.TypeA:
- resp.Answers = append(resp.Answers, dnsmessage.Resource{
- Header: dnsmessage.ResourceHeader{Name: q.Name, Type: dnsmessage.TypeA, Class: dnsmessage.ClassINET, TTL: 300},
- Body: &dnsmessage.AResource{A: [4]byte{127, 0, 0, 1}},
- })
- case dnsmessage.TypeAAAA:
- ip := net.ParseIP("::1").To16()
- var aaaa [16]byte
- copy(aaaa[:], ip)
- resp.Answers = append(resp.Answers, dnsmessage.Resource{
- Header: dnsmessage.ResourceHeader{Name: q.Name, Type: dnsmessage.TypeAAAA, Class: dnsmessage.ClassINET, TTL: 300},
- Body: &dnsmessage.AAAAResource{AAAA: aaaa},
- })
- }
- }
-
- packed, err := resp.Pack()
- if err == nil {
- _, _ = conn.WriteToUDP(packed, rAddr)
- }
- }(packet, remoteAddr)
- }
- }()
-
- return nil
-}
-
-func handleConnection(client net.Conn, udsPath string) {
- defer func() { _ = client.Close() }()
-
- uds, err := net.DialTimeout("unix", udsPath, 5*time.Second)
- if err != nil {
- log.Printf("UDS Dial error: %v", err)
- return
- }
- defer func() { _ = uds.Close() }()
-
- cp := func(dst, src net.Conn, done chan<- struct{}) {
- _, _ = io.Copy(dst, src)
- if cw, ok := dst.(interface{ CloseWrite() error }); ok {
- _ = cw.CloseWrite()
- }
- done <- struct{}{}
- }
-
- done := make(chan struct{}, 2)
- go cp(uds, client, done)
- go cp(client, uds, done)
-
- <-done
- <-done
-}
-
-func setupPID1Duties(agentPid int, exitChan chan<- syscall.WaitStatus) {
+// reapChildren collects orphans and remembers the agent's own status.
+func reapChildren(agentPid int, exitChan chan<- syscall.WaitStatus) {
sigCh := make(chan os.Signal, 10)
signal.Notify(sigCh, syscall.SIGCHLD)
@@ -427,7 +258,7 @@ func setupPID1Duties(agentPid int, exitChan chan<- syscall.WaitStatus) {
var status syscall.WaitStatus
pid, err := syscall.Wait4(-1, &status, syscall.WNOHANG, nil)
if pid <= 0 || err != nil {
- break
+ return
}
if pid == agentPid {
select {
@@ -437,67 +268,11 @@ func setupPID1Duties(agentPid int, exitChan chan<- syscall.WaitStatus) {
}
}
}
- // Initial reap pass to catch any processes that exited before signal.Notify was registered
+ // Once before waiting on signals, to catch anything that exited
+ // between Start and Notify.
reap()
for range sigCh {
reap()
}
}()
}
-
-func buildAgentEnv(assignedPort int, caPath string, interceptorPath string) []string {
- proxyURL := fmt.Sprintf("http://127.0.0.1:%d", assignedPort)
-
- overrides := map[string]string{
- "SSL_CERT_FILE": caPath,
- "REQUESTS_CA_BUNDLE": caPath,
- "NODE_EXTRA_CA_CERTS": caPath,
- "HTTP_PROXY": proxyURL,
- "HTTPS_PROXY": proxyURL,
- "ALL_PROXY": proxyURL,
- "http_proxy": proxyURL,
- "https_proxy": proxyURL,
- "all_proxy": proxyURL,
- "SAM_PROXY_PORT": strconv.Itoa(assignedPort),
- }
-
- if interceptorPath != "" {
- overrides["LD_PRELOAD"] = interceptorPath
- }
-
- var result []string
- for _, env := range os.Environ() {
- parts := strings.SplitN(env, "=", 2)
- if len(parts) != 2 {
- continue
- }
- key := parts[0]
- if _, exists := overrides[key]; exists {
- continue
- }
- result = append(result, env)
- }
-
- for k, v := range overrides {
- result = append(result, fmt.Sprintf("%s=%s", k, v))
- }
-
- return result
-}
-
-func copyFile(src, dest string) error {
- in, err := os.Open(src)
- if err != nil {
- return err
- }
- defer in.Close() //nolint:errcheck
- out, err := os.OpenFile(dest, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0755)
- if err != nil {
- return err
- }
- if _, err = io.Copy(out, in); err != nil {
- out.Close() //nolint:errcheck
- return err
- }
- return out.Close()
-}
diff --git a/cmd/nano-init/main_test.go b/cmd/nano-init/main_test.go
index a719d0be..3c0572c1 100644
--- a/cmd/nano-init/main_test.go
+++ b/cmd/nano-init/main_test.go
@@ -15,393 +15,297 @@
package main
import (
- "context"
- "io"
- "net"
- "net/http"
+ "net/netip"
"os"
"path/filepath"
"strings"
"testing"
- "time"
"golang.org/x/net/dns/dnsmessage"
)
-func TestBootstrapCA(t *testing.T) {
- tempDir := t.TempDir()
- udsPath := filepath.Join(tempDir, "mock-sam-box.sock")
-
- // Start a mock HTTP server on a Unix Domain Socket
- listener, err := net.Listen("unix", udsPath)
+func TestTheSameNameAlwaysGetsTheSameAddress(t *testing.T) {
+ // Clients cache. An agent that resolves once, holds the answer, and
+ // connects later must arrive at the same place, or a long-lived agent
+ // starts failing for reasons nothing in its own behaviour explains.
+ r, err := newResolver("169.254.64.0/18")
if err != nil {
- t.Fatalf("Failed to listen on UDS: %v", err)
+ t.Fatalf("newResolver: %v", err)
}
- defer func() { _ = listener.Close() }()
-
- mockCACert := []byte("fake-ca-cert-data")
- srv := &http.Server{
- Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- if r.Method == "GET" && r.URL.Path == "/internal/bootstrap/ca.crt" {
- w.WriteHeader(http.StatusOK)
- _, _ = w.Write(mockCACert)
- } else {
- w.WriteHeader(http.StatusNotFound)
- }
- }),
+ first, err := r.assign("mesh.sam.alt")
+ if err != nil {
+ t.Fatalf("assign: %v", err)
}
- go func() {
- _ = srv.Serve(listener)
- }()
- defer func() { _ = srv.Close() }()
-
- ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
- defer cancel()
-
- // Clean target file if it already exists
- _ = os.Remove("/tmp/ephemeral_ca.pem")
- defer func() { _ = os.Remove("/tmp/ephemeral_ca.pem") }()
-
- if err := bootstrapCA(ctx, udsPath); err != nil {
- t.Fatalf("bootstrapCA failed: %v", err)
+ again, err := r.assign("mesh.sam.alt")
+ if err != nil {
+ t.Fatalf("assign: %v", err)
}
-
- data, err := os.ReadFile("/tmp/ephemeral_ca.pem")
+ other, err := r.assign("calc.mcp.sam.alt")
if err != nil {
- t.Fatalf("Failed to read bootstrap CA: %v", err)
+ t.Fatalf("assign: %v", err)
}
- if string(data) != string(mockCACert) {
- t.Errorf("Expected cert data %q, got %q", string(mockCACert), string(data))
+ if first != again {
+ t.Errorf("same name got %v then %v", first, again)
+ }
+ if first == other {
+ t.Errorf("two names share the address %v", first)
}
}
-func TestDNSSpoofer(t *testing.T) {
- ctx, cancel := context.WithCancel(context.Background())
- defer cancel()
-
- dnsAddr := "127.0.0.1:10053"
- if err := startDNSSpoofer(ctx, dnsAddr); err != nil {
- t.Fatalf("Failed to start DNS spoofer: %v", err)
+func TestTheAddressLeadsBackToTheName(t *testing.T) {
+ // This is the whole point of the resolver: the boundary must be told a
+ // name, because mesh.sam.alt has no address and the boundary is what
+ // chooses a provider for it.
+ r, err := newResolver("169.254.64.0/18")
+ if err != nil {
+ t.Fatalf("newResolver: %v", err)
}
- // Create UDP connection to make queries
- conn, err := net.Dial("udp", dnsAddr)
- if err != nil {
- t.Fatalf("Failed to dial DNS spoofer: %v", err)
- }
- defer func() { _ = conn.Close() }()
-
- // Query A record
- msgA := dnsmessage.Message{
- Header: dnsmessage.Header{
- ID: 1234,
- OpCode: 0,
- RecursionDesired: true,
- },
- Questions: []dnsmessage.Question{
- {
- Name: dnsmessage.MustNewName("api.github.com."),
- Type: dnsmessage.TypeA,
- Class: dnsmessage.ClassINET,
- },
- },
- }
- packedA, _ := msgA.Pack()
- _, _ = conn.Write(packedA)
-
- buf := make([]byte, 512)
- _ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
- n, err := conn.Read(buf)
+ addr, err := r.assign("mesh.sam.alt")
if err != nil {
- t.Fatalf("Failed to read DNS reply: %v", err)
+ t.Fatalf("assign: %v", err)
}
- var respA dnsmessage.Message
- if err := respA.Unpack(buf[:n]); err != nil {
- t.Fatalf("Failed to unpack DNS response: %v", err)
+ name, ok := r.nameFor(addr)
+ if !ok || name != "mesh.sam.alt" {
+ t.Errorf("nameFor(%v) = %q, %v; want mesh.sam.alt", addr, name, ok)
}
- if len(respA.Answers) == 0 {
- t.Fatalf("No answers in DNS A response")
+ // An address nobody handed out is not an error: an agent may dial a
+ // literal address, and whether that is allowed is policy's business.
+ if _, ok := r.nameFor(netip.MustParseAddr("93.184.216.34")); ok {
+ t.Error("an address that was never assigned resolved to a name")
}
+}
- answerA := respA.Answers[0]
- if answerA.Header.Type != dnsmessage.TypeA {
- t.Fatalf("Expected A record answer, got %v", answerA.Header.Type)
+func TestTheAddressPoolIsBounded(t *testing.T) {
+ // A /30 holds four addresses, and the network address is skipped. An
+ // agent resolving endlessly must be refused rather than handed something
+ // outside the pool that would then be routed somewhere real.
+ r, err := newResolver("169.254.64.0/30")
+ if err != nil {
+ t.Fatalf("newResolver: %v", err)
}
- bodyVarA := answerA.Body
- aRes, ok := bodyVarA.(*dnsmessage.AResource)
- if !ok {
- t.Fatalf("Expected AResource body type")
+ var lastErr error
+ for i := range 10 {
+ if _, err := r.assign(strings.Repeat("a", i+1) + ".example"); err != nil {
+ lastErr = err
+ break
+ }
}
-
- expectedA := [4]byte{127, 0, 0, 1}
- if aRes.A != expectedA {
- t.Errorf("Expected A record %v, got %v", expectedA, aRes.A)
+ if lastErr == nil {
+ t.Error("the pool handed out more addresses than it contains")
}
+}
- // Query AAAA record
- msgAAAA := dnsmessage.Message{
- Header: dnsmessage.Header{
- ID: 5678,
- OpCode: 0,
- RecursionDesired: true,
- },
- Questions: []dnsmessage.Question{
- {
- Name: dnsmessage.MustNewName("api.github.com."),
- Type: dnsmessage.TypeAAAA,
- Class: dnsmessage.ClassINET,
- },
- },
+func TestAQueriesAreAnsweredWithAPlaceholder(t *testing.T) {
+ r, err := newResolver("169.254.64.0/18")
+ if err != nil {
+ t.Fatalf("newResolver: %v", err)
}
- packedAAAA, _ := msgAAAA.Pack()
- _, _ = conn.Write(packedAAAA)
- _ = conn.SetReadDeadline(time.Now().Add(2 * time.Second))
- n, err = conn.Read(buf)
+ query, err := buildQuery("mesh.sam.alt.", dnsmessage.TypeA)
if err != nil {
- t.Fatalf("Failed to read DNS reply: %v", err)
+ t.Fatalf("buildQuery: %v", err)
}
- var respAAAA dnsmessage.Message
- if err := respAAAA.Unpack(buf[:n]); err != nil {
- t.Fatalf("Failed to unpack DNS response: %v", err)
+ reply, err := r.answer(query)
+ if err != nil {
+ t.Fatalf("answer: %v", err)
}
- if len(respAAAA.Answers) == 0 {
- t.Fatalf("No answers in DNS AAAA response")
+ var parser dnsmessage.Parser
+ if _, err := parser.Start(reply); err != nil {
+ t.Fatalf("parse reply: %v", err)
}
-
- answerAAAA := respAAAA.Answers[0]
- if answerAAAA.Header.Type != dnsmessage.TypeAAAA {
- t.Fatalf("Expected AAAA record answer, got %v", answerAAAA.Header.Type)
+ if err := parser.SkipAllQuestions(); err != nil {
+ t.Fatalf("skip questions: %v", err)
}
-
- bodyVarAAAA := answerAAAA.Body
- aaaaRes, ok := bodyVarAAAA.(*dnsmessage.AAAAResource)
- if !ok {
- t.Fatalf("Expected AAAAResource body type")
+ answer, err := parser.AnswerHeader()
+ if err != nil {
+ t.Fatalf("answer header: %v", err)
}
-
- expectedAAAA := net.ParseIP("::1").To16()
- var expectedAAAABytes [16]byte
- copy(expectedAAAABytes[:], expectedAAAA)
-
- if aaaaRes.AAAA != expectedAAAABytes {
- t.Errorf("Expected AAAA record %v, got %v", expectedAAAABytes, aaaaRes.AAAA)
+ if answer.Type != dnsmessage.TypeA {
+ t.Fatalf("answer type = %v, want A", answer.Type)
}
-}
-func TestTCPForwarder(t *testing.T) {
- tempDir := t.TempDir()
- udsPath := filepath.Join(tempDir, "mock-uds.sock")
-
- // UDS Listener (representing gateway)
- udsListener, err := net.Listen("unix", udsPath)
+ resource, err := parser.AResource()
if err != nil {
- t.Fatalf("Failed to listen on UDS: %v", err)
+ t.Fatalf("A resource: %v", err)
}
- defer func() { _ = udsListener.Close() }()
-
- mockGatewayMsg := []byte("hello from gateway")
-
- go func() {
- conn, err := udsListener.Accept()
- if err != nil {
- return
- }
- defer func() { _ = conn.Close() }()
- _, _ = conn.Write(mockGatewayMsg)
- }()
+ got := netip.AddrFrom4(resource.A)
+ if name, ok := r.nameFor(got); !ok || name != "mesh.sam.alt" {
+ t.Errorf("answered %v, which maps to %q, %v", got, name, ok)
+ }
+}
- listener, err := net.Listen("tcp", "127.0.0.1:0")
+func TestAAAAQueriesAreAnsweredEmpty(t *testing.T) {
+ // A sandbox that got an AAAA answer would try IPv6 first and wait for it
+ // to fail, which reads as a slow mesh rather than an absent address family.
+ r, err := newResolver("169.254.64.0/18")
if err != nil {
- t.Fatalf("Failed to listen: %v", err)
+ t.Fatalf("newResolver: %v", err)
}
- defer func() { _ = listener.Close() }()
- go func() {
- for {
- conn, err := listener.Accept()
- if err != nil {
- return
- }
- go handleConnection(conn, udsPath)
- }
- }()
-
- // Dial forwarder
- conn, err := net.Dial("tcp", listener.Addr().String())
+ query, err := buildQuery("mesh.sam.alt.", dnsmessage.TypeAAAA)
if err != nil {
- t.Fatalf("Failed to dial forwarder: %v", err)
+ t.Fatalf("buildQuery: %v", err)
}
- defer func() { _ = conn.Close() }()
-
- reply := make([]byte, len(mockGatewayMsg))
- _, err = io.ReadFull(conn, reply)
+ reply, err := r.answer(query)
if err != nil {
- t.Fatalf("Failed to read from forwarder connection: %v", err)
+ t.Fatalf("answer: %v", err)
}
- if string(reply) != string(mockGatewayMsg) {
- t.Errorf("Expected message %q, got %q", string(mockGatewayMsg), string(reply))
+ var parser dnsmessage.Parser
+ if _, err := parser.Start(reply); err != nil {
+ t.Fatalf("parse reply: %v", err)
}
-}
-
-func TestBuildAgentEnv(t *testing.T) {
- caPath := "/tmp/test_ca.pem"
- interceptorPath := "/tmp/test_interceptor.so"
-
- env := buildAgentEnv(1234, caPath, interceptorPath)
-
- expected := map[string]string{
- "SSL_CERT_FILE": caPath,
- "REQUESTS_CA_BUNDLE": caPath,
- "NODE_EXTRA_CA_CERTS": caPath,
- "HTTP_PROXY": "http://127.0.0.1:1234",
- "HTTPS_PROXY": "http://127.0.0.1:1234",
- "ALL_PROXY": "http://127.0.0.1:1234",
- "http_proxy": "http://127.0.0.1:1234",
- "https_proxy": "http://127.0.0.1:1234",
- "all_proxy": "http://127.0.0.1:1234",
- "SAM_PROXY_PORT": "1234",
- "LD_PRELOAD": interceptorPath,
- }
-
- envMap := make(map[string]string)
- for _, kv := range env {
- parts := strings.SplitN(kv, "=", 2)
- if len(parts) == 2 {
- envMap[parts[0]] = parts[1]
- }
+ if err := parser.SkipAllQuestions(); err != nil {
+ t.Fatalf("skip questions: %v", err)
}
-
- for k, expectedVal := range expected {
- val, exists := envMap[k]
- if !exists {
- t.Errorf("Expected environment variable %s to be set", k)
- continue
- }
- if val != expectedVal {
- t.Errorf("Expected environment variable %s to be %q, got %q", k, expectedVal, val)
- }
+ if _, err := parser.AnswerHeader(); err == nil {
+ t.Error("AAAA query produced an answer record")
}
}
-func TestCopyFile(t *testing.T) {
- tempDir := t.TempDir()
- src := filepath.Join(tempDir, "source-file")
- dest := filepath.Join(tempDir, "dest-file")
-
- content := []byte("hello copy world")
- if err := os.WriteFile(src, content, 0755); err != nil {
- t.Fatalf("Failed to write source file: %v", err)
+func TestConnectSendsTheNameNotTheAddress(t *testing.T) {
+ // If this regresses, the boundary receives an address it cannot resolve a
+ // provider for, and every mesh name stops working while every literal
+ // address keeps working, which is a confusing way to lose the design.
+ request, err := connectRequest("mesh.sam.alt", 80)
+ if err != nil {
+ t.Fatalf("connectRequest: %v", err)
}
- if err := copyFile(src, dest); err != nil {
- t.Fatalf("copyFile failed: %v", err)
+ if request[3] != atypDomain {
+ t.Fatalf("address type = %d, want %d (domain)", request[3], atypDomain)
}
-
- data, err := os.ReadFile(dest)
- if err != nil {
- t.Fatalf("Failed to read dest file: %v", err)
+ length := int(request[4])
+ if got := string(request[5 : 5+length]); got != "mesh.sam.alt" {
+ t.Errorf("name = %q, want mesh.sam.alt", got)
}
-
- if string(data) != string(content) {
- t.Errorf("Expected content %q, got %q", string(content), string(data))
+ port := int(request[5+length])<<8 | int(request[6+length])
+ if port != 80 {
+ t.Errorf("port = %d, want 80", port)
}
+}
- fi, err := os.Stat(dest)
+func TestConnectPassesLiteralAddressesThrough(t *testing.T) {
+ // An agent that dials an address rather than a name is not doing anything
+ // forbidden; the boundary still decides. Encoding it as a name would make
+ // the boundary try to resolve "93.184.216.34" as a mesh service.
+ request, err := connectRequest("93.184.216.34", 443)
if err != nil {
- t.Fatalf("Stat dest failed: %v", err)
+ t.Fatalf("connectRequest: %v", err)
}
+ if request[3] != atypIPv4 {
+ t.Errorf("address type = %d, want %d (IPv4)", request[3], atypIPv4)
+ }
+}
- expectedMode := os.FileMode(0755)
- if fi.Mode().Perm() != expectedMode {
- t.Errorf("Expected permissions %v, got %v", expectedMode, fi.Mode().Perm())
+func TestConnectRefusesAnUnencodableName(t *testing.T) {
+ if _, err := connectRequest(strings.Repeat("a", 256), 80); err == nil {
+ t.Error("a name too long for the protocol was encoded anyway")
}
}
-func TestBootstrapInterceptor(t *testing.T) {
- tempDir := t.TempDir()
- udsPath := filepath.Join(tempDir, "mock-sam-box-interceptor.sock")
+func TestPolicyDenialIsLegibleAsPolicy(t *testing.T) {
+ // An operator reading an agent's logs has to be able to tell "you may not"
+ // from "the mesh is broken", because the two have nothing in common.
+ if got := replyMessage(0x02); !strings.Contains(got, "not allowed") {
+ t.Errorf("reply 0x02 reads as %q, which does not say it was policy", got)
+ }
+ if got := replyMessage(0x05); !strings.Contains(got, "refused") {
+ t.Errorf("reply 0x05 reads as %q", got)
+ }
+}
- listener, err := net.Listen("unix", udsPath)
+func TestTheAgentEnvironmentIsNotDoctored(t *testing.T) {
+ // The point of the rewrite: nano-init no longer reaches into the agent. If
+ // these come back, confinement has quietly become a request for the
+ // agent's cooperation again and every argument for the design stops
+ // holding.
+ source, err := os.ReadFile("main.go")
if err != nil {
- t.Fatalf("Failed to listen on UDS: %v", err)
+ t.Fatalf("read main.go: %v", err)
+ }
+ for _, forbidden := range []string{
+ "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY",
+ "LD_PRELOAD", "SSL_CERT_FILE", "REQUESTS_CA_BUNDLE",
+ } {
+ if strings.Contains(string(source), `"`+forbidden+`"`) {
+ t.Errorf("%s is being set for the agent again", forbidden)
+ }
}
- defer func() { _ = listener.Close() }()
-
- mockSO := []byte("fake-shared-library-so-data")
- var receivedArch, receivedLibc string
+}
- srv := &http.Server{
- Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- if r.Method == "GET" && r.URL.Path == "/internal/bootstrap/libinterceptor.so" {
- receivedArch = r.URL.Query().Get("arch")
- receivedLibc = r.URL.Query().Get("libc")
- w.WriteHeader(http.StatusOK)
- _, _ = w.Write(mockSO)
- } else {
- w.WriteHeader(http.StatusNotFound)
- }
- }),
+func TestTheBoundaryCanBeNamedEitherWay(t *testing.T) {
+ // A container dials a path and a microVM dials vsock. One binary serves
+ // both, and nothing else in the sandbox knows which kind it is, so this
+ // string is the entire difference between them.
+ if _, _, err := parseVsock("2:1080"); err != nil {
+ t.Errorf("parseVsock(2:1080): %v", err)
+ }
+ for _, bad := range []string{"2", "host:1080", "2:not-a-port", ""} {
+ if _, _, err := parseVsock(bad); err == nil {
+ t.Errorf("parseVsock(%q) was accepted", bad)
+ }
}
- go func() {
- _ = srv.Serve(listener)
- }()
- defer func() { _ = srv.Close() }()
- ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
- defer cancel()
+ // A missing socket has to be reported at startup. A sandbox that starts
+ // without a way out looks like a mesh outage on the agent's first call.
+ if err := checkBoundary(filepath.Join(t.TempDir(), "absent.sock")); err == nil {
+ t.Error("a boundary socket that does not exist was accepted")
+ }
+ if err := checkBoundary("vsock://2:1080"); err != nil {
+ t.Errorf("a well-formed vsock boundary was rejected: %v", err)
+ }
+}
- _ = os.Remove("/tmp/libinterceptor.so")
- defer func() { _ = os.Remove("/tmp/libinterceptor.so") }()
+func TestCopyFile(t *testing.T) {
+ dir := t.TempDir()
+ src := filepath.Join(dir, "src")
+ dest := filepath.Join(dir, "dest")
- soPath, err := bootstrapInterceptor(ctx, udsPath)
- if err != nil {
- t.Fatalf("bootstrapInterceptor failed: %v", err)
+ if err := os.WriteFile(src, []byte("binary"), 0o600); err != nil {
+ t.Fatalf("WriteFile: %v", err)
}
-
- if soPath != "/tmp/libinterceptor.so" {
- t.Errorf("Expected path '/tmp/libinterceptor.so', got %q", soPath)
+ if err := copyFile(src, dest); err != nil {
+ t.Fatalf("copyFile: %v", err)
}
- data, err := os.ReadFile("/tmp/libinterceptor.so")
+ got, err := os.ReadFile(dest)
if err != nil {
- t.Fatalf("Failed to read bootstrap SO: %v", err)
+ t.Fatalf("ReadFile: %v", err)
}
-
- if string(data) != string(mockSO) {
- t.Errorf("Expected SO data %q, got %q", string(mockSO), string(data))
+ if string(got) != "binary" {
+ t.Errorf("contents = %q, want %q", got, "binary")
}
- if receivedArch == "" {
- t.Errorf("Expected arch query parameter to be passed, but was empty")
+ info, err := os.Stat(dest)
+ if err != nil {
+ t.Fatalf("Stat: %v", err)
}
- if receivedLibc == "" {
- t.Errorf("Expected libc query parameter to be passed, but was empty")
+ // The copy has to be runnable; the exact bits are the umask's business.
+ if info.Mode().Perm()&0o100 == 0 {
+ t.Errorf("mode = %v, want the owner execute bit set", info.Mode().Perm())
}
+}
- // Test fallback / error case (server returns 404)
- errSrvListener, err := net.Listen("unix", filepath.Join(tempDir, "mock-error-sam-box.sock"))
- if err == nil {
- defer func() { _ = errSrvListener.Close() }()
- errSrv := &http.Server{
- Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
- w.WriteHeader(http.StatusNotFound)
- }),
- }
- go func() { _ = errSrv.Serve(errSrvListener) }()
- defer func() { _ = errSrv.Close() }()
-
- _, err = bootstrapInterceptor(ctx, errSrvListener.Addr().String())
- if err == nil {
- t.Errorf("Expected bootstrapInterceptor to return error when server returns 404")
- }
+func buildQuery(name string, qtype dnsmessage.Type) ([]byte, error) {
+ builder := dnsmessage.NewBuilder(nil, dnsmessage.Header{ID: 1234})
+ if err := builder.StartQuestions(); err != nil {
+ return nil, err
+ }
+ if err := builder.Question(dnsmessage.Question{
+ Name: dnsmessage.MustNewName(name),
+ Type: qtype,
+ Class: dnsmessage.ClassINET,
+ }); err != nil {
+ return nil, err
}
+ return builder.Finish()
}
diff --git a/cmd/nano-init/resolver.go b/cmd/nano-init/resolver.go
new file mode 100644
index 00000000..4f51c2e6
--- /dev/null
+++ b/cmd/nano-init/resolver.go
@@ -0,0 +1,178 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "fmt"
+ "net"
+ "net/netip"
+ "sync"
+
+ "golang.org/x/net/dns/dnsmessage"
+)
+
+// The sandbox has to keep hold of the name.
+//
+// An agent asks for mesh.sam.alt, which has no address anywhere: the boundary
+// answers it by choosing a provider, and choosing is the whole point. But an
+// agent opens a socket to an address, and the packets that arrive at the tun
+// carry an address and nothing else. Somewhere between the two the name has to
+// survive.
+//
+// So this hands out an address that means nothing except "the name I was asked
+// for", and remembers which. When the flow comes back the other way, the
+// address is turned back into the name and it is the name that reaches the
+// boundary. The addresses are never real, never routed and never leave the
+// sandbox.
+//
+// This is not the DNS interception that used to live here. That answered with
+// real-looking addresses and then relied on the agent honouring proxy
+// variables to catch the connection; an agent that ignored them was outside
+// the boundary. Here the address is a token in a table, the route is the only
+// route, and an agent that ignores everything still arrives at the same place.
+
+// resolver allocates a placeholder address per name and remembers the pairing.
+type resolver struct {
+ mu sync.RWMutex
+ pool netip.Prefix
+ next netip.Addr
+ names map[netip.Addr]string
+ addrs map[string]netip.Addr
+}
+
+func newResolver(pool string) (*resolver, error) {
+ prefix, err := netip.ParsePrefix(pool)
+ if err != nil {
+ return nil, fmt.Errorf("virtual address pool %q: %w", pool, err)
+ }
+ return &resolver{
+ pool: prefix,
+ // The network address itself is skipped: handing it out would make a
+ // name indistinguishable from "no name at all" in a packet capture.
+ next: prefix.Addr().Next(),
+ names: map[netip.Addr]string{},
+ addrs: map[string]netip.Addr{},
+ }, nil
+}
+
+// assign returns the placeholder for a name, allocating one if needed. The
+// same name always gets the same address, so an agent that resolves twice and
+// caches the first answer still reaches the same place.
+func (r *resolver) assign(name string) (netip.Addr, error) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+
+ if addr, ok := r.addrs[name]; ok {
+ return addr, nil
+ }
+ if !r.pool.Contains(r.next) {
+ return netip.Addr{}, fmt.Errorf("virtual address pool %s exhausted", r.pool)
+ }
+
+ addr := r.next
+ r.next = r.next.Next()
+ r.names[addr] = name
+ r.addrs[name] = addr
+ return addr, nil
+}
+
+// nameFor returns the name a placeholder stands for. An address that was never
+// handed out is not an error: an agent may dial a literal address, and that is
+// policy's business rather than something to fail here.
+func (r *resolver) nameFor(addr netip.Addr) (string, bool) {
+ r.mu.RLock()
+ defer r.mu.RUnlock()
+ name, ok := r.names[addr]
+ return name, ok
+}
+
+// serveDNS answers A queries with placeholders until ctx ends.
+//
+// Nothing here is a security control. An agent that ignores this resolver and
+// hardcodes another one still has its packets routed through the tun, and
+// still reaches only what policy allows. This exists so that ordinary clients,
+// which resolve before they connect, get an address to connect to.
+func (r *resolver) serveDNS(conn net.PacketConn) {
+ buf := make([]byte, 512)
+ for {
+ n, from, err := conn.ReadFrom(buf)
+ if err != nil {
+ return
+ }
+
+ reply, err := r.answer(buf[:n])
+ if err != nil {
+ continue
+ }
+ if _, err := conn.WriteTo(reply, from); err != nil {
+ return
+ }
+ }
+}
+
+// answer builds a reply for one query.
+func (r *resolver) answer(query []byte) ([]byte, error) {
+ var parser dnsmessage.Parser
+ header, err := parser.Start(query)
+ if err != nil {
+ return nil, err
+ }
+ question, err := parser.Question()
+ if err != nil {
+ return nil, err
+ }
+
+ builder := dnsmessage.NewBuilder(nil, dnsmessage.Header{
+ ID: header.ID,
+ Response: true,
+ Authoritative: true,
+ RCode: dnsmessage.RCodeSuccess,
+ })
+ builder.EnableCompression()
+ if err := builder.StartQuestions(); err != nil {
+ return nil, err
+ }
+ if err := builder.Question(question); err != nil {
+ return nil, err
+ }
+ if err := builder.StartAnswers(); err != nil {
+ return nil, err
+ }
+
+ // Only A is answered. A sandbox that got a AAAA answer would try IPv6
+ // first and wait for it to fail, which looks like a slow mesh.
+ if question.Type == dnsmessage.TypeA {
+ name := trimRoot(question.Name.String())
+ addr, err := r.assign(name)
+ if err != nil {
+ return nil, err
+ }
+ if err := builder.AResource(
+ dnsmessage.ResourceHeader{Name: question.Name, Class: question.Class, TTL: 1},
+ dnsmessage.AResource{A: addr.As4()},
+ ); err != nil {
+ return nil, err
+ }
+ }
+
+ return builder.Finish()
+}
+
+func trimRoot(name string) string {
+ if len(name) > 0 && name[len(name)-1] == '.' {
+ return name[:len(name)-1]
+ }
+ return name
+}
diff --git a/cmd/nano-init/vsock.go b/cmd/nano-init/vsock.go
new file mode 100644
index 00000000..804cb087
--- /dev/null
+++ b/cmd/nano-init/vsock.go
@@ -0,0 +1,116 @@
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+ "context"
+ "fmt"
+ "net"
+ "os"
+ "strconv"
+ "strings"
+
+ "golang.org/x/sys/unix"
+)
+
+// A sandbox reaches its boundary one of two ways, and the difference is the
+// only thing that distinguishes a container from a microVM here.
+//
+// A container gets the socket bind-mounted in, so it dials a path. A microVM
+// has no shared filesystem and no network device, so it dials vsock, and
+// Firecracker delivers that to a Unix socket on the host named
+// "<uds_path>_<port>". Both arrive at the same sam-box, which is why one
+// binary serves both and neither sandbox contains anything that knows which
+// kind it is beyond this string.
+
+const vsockScheme = "vsock://"
+
+// dialBoundary opens a connection to the boundary named by spec, which is
+// either "vsock://<cid>:<port>" or a Unix socket path.
+func dialBoundary(ctx context.Context, spec string) (net.Conn, error) {
+ if strings.HasPrefix(spec, vsockScheme) {
+ return dialVsock(ctx, strings.TrimPrefix(spec, vsockScheme))
+ }
+ return (&net.Dialer{Timeout: boundaryDialTimeout}).DialContext(ctx, "unix", spec)
+}
+
+// checkBoundary reports whether the boundary named by spec could plausibly be
+// reached, so a sandbox with no way out says so at startup rather than on the
+// agent's first request.
+func checkBoundary(spec string) error {
+ if strings.HasPrefix(spec, vsockScheme) {
+ if _, _, err := parseVsock(strings.TrimPrefix(spec, vsockScheme)); err != nil {
+ return err
+ }
+ // Whether the host is listening cannot be known until a connection is
+ // attempted, and attempting one here would consume a flow the agent
+ // has not asked for yet.
+ return nil
+ }
+ if _, err := os.Stat(spec); err != nil {
+ return fmt.Errorf("no boundary at %s: %w", spec, err)
+ }
+ return nil
+}
+
+func parseVsock(hostPort string) (cid, port uint32, err error) {
+ rawCID, rawPort, found := strings.Cut(hostPort, ":")
+ if !found {
+ return 0, 0, fmt.Errorf("vsock boundary %q is not <cid>:<port>", hostPort)
+ }
+
+ parsedCID, err := strconv.ParseUint(rawCID, 10, 32)
+ if err != nil {
+ return 0, 0, fmt.Errorf("vsock cid %q: %w", rawCID, err)
+ }
+ parsedPort, err := strconv.ParseUint(rawPort, 10, 32)
+ if err != nil {
+ return 0, 0, fmt.Errorf("vsock port %q: %w", rawPort, err)
+ }
+ return uint32(parsedCID), uint32(parsedPort), nil
+}
+
+// dialVsock opens an AF_VSOCK connection.
+//
+// This is done by hand rather than with a library because it is one socket, one
+// connect, and a library would be a dependency carried into every sandbox for
+// forty lines.
+func dialVsock(ctx context.Context, hostPort string) (net.Conn, error) {
+ cid, port, err := parseVsock(hostPort)
+ if err != nil {
+ return nil, err
+ }
+
+ fd, err := unix.Socket(unix.AF_VSOCK, unix.SOCK_STREAM|unix.SOCK_CLOEXEC, 0)
+ if err != nil {
+ return nil, fmt.Errorf("vsock socket: %w", err)
+ }
+
+ if err := unix.Connect(fd, &unix.SockaddrVM{CID: cid, Port: port}); err != nil {
+ _ = unix.Close(fd)
+ return nil, fmt.Errorf("vsock connect %d:%d: %w", cid, port, err)
+ }
+
+ // net.FileConn dups the descriptor and hands it to the runtime poller, so
+ // the original has to be closed or every flow leaks one.
+ file := os.NewFile(uintptr(fd), "vsock")
+ defer func() { _ = file.Close() }()
+
+ conn, err := net.FileConn(file)
+ if err != nil {
+ return nil, fmt.Errorf("vsock conn: %w", err)
+ }
+ return conn, nil
+}
diff --git a/development/examples/agent-harness/Dockerfile b/development/examples/agent-harness/Dockerfile
index 03bd9a3d..6d880b4a 100644
--- a/development/examples/agent-harness/Dockerfile
+++ b/development/examples/agent-harness/Dockerfile
@@ -1,31 +1,27 @@
-# A sandbox image holds the harness and the plumbing that gives it a route.
+# A sandbox image holds the harness and the init that gives it a route.
# No credentials, and no proxy configuration: the agent is unmodified and does
# not know it is confined.
+FROM golang:1.26.5 AS init
+WORKDIR /src
+COPY cmd/nano-init/go.mod cmd/nano-init/go.sum ./
+RUN go mod download
+COPY cmd/nano-init/ ./
+RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /nano-init .
+
FROM python:3.12-slim
-# socat bridges the boundary's Unix socket to a port, iproute2 builds the tun,
-# and tun2proxy makes that tun the way out. This is the same arrangement a
-# microVM runs, which is the point: the agent above it is identical either way.
-ARG TUN2PROXY_VERSION=v0.8.3
-RUN apt-get update \
- && apt-get install -y --no-install-recommends socat iproute2 ca-certificates curl unzip \
- && curl -sSL -o /tmp/tun2proxy.zip \
- "https://github.com/tun2proxy/tun2proxy/releases/download/${TUN2PROXY_VERSION}/tun2proxy-x86_64-unknown-linux-musl.zip" \
- && unzip -j /tmp/tun2proxy.zip tun2proxy-bin -d /usr/local/bin \
- && mv /usr/local/bin/tun2proxy-bin /usr/local/bin/tun2proxy \
- && chmod +x /usr/local/bin/tun2proxy \
- && apt-get purge -y curl unzip && apt-get autoremove -y \
- && rm -rf /var/lib/apt/lists/* /tmp/tun2proxy.zip
+# Nothing else is needed. nano-init carries its own TCP stack and speaks
+# netlink directly, so this image has no tun2proxy, no socat and no iproute2 —
+# which is the difference between a sandbox that is the agent and one that is
+# the agent plus a toolbox.
+COPY --from=init /nano-init /usr/local/bin/nano-init
WORKDIR /app
-
-COPY requirements.txt .
+COPY development/examples/agent-harness/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
-
-COPY agent.py entrypoint.sh ./
-RUN chmod +x entrypoint.sh
+COPY development/examples/agent-harness/agent.py .
# Building the tun needs NET_ADMIN, so this stays root. The isolation that
-# matters is the sandbox around the container, not the uid inside it; a
-# microVM makes the same trade with its own kernel.
-ENTRYPOINT ["./entrypoint.sh"]
+# matters is the sandbox around the container, not the uid inside it; a microVM
+# makes the same trade with its own kernel.
+ENTRYPOINT ["nano-init", "run", "/run/agent.sock", "python", "agent.py"]
diff --git a/development/examples/agent-harness/README.md b/development/examples/agent-harness/README.md
index 779a3ad6..f078034f 100644
--- a/development/examples/agent-harness/README.md
+++ b/development/examples/agent-harness/README.md
@@ -24,9 +24,12 @@ Full walkthrough: [Running agents on SAM](https://sam-mesh.dev/docs/user/running
| | |
| --- | --- |
| `agent.py` | The harness. Discovers tools, calls a model, runs the loop. |
-| `entrypoint.sh` | The sandbox's route: a tun that leads to the boundary and nowhere else. |
| `bundle.yaml` | Who this agent is and what it may reach. Read by `sam-box`, outside the sandbox. |
-| `Dockerfile` | The sandbox image: the harness and the plumbing that gives it a route. |
+| `Dockerfile` | The sandbox image: the harness, plus `nano-init` to give it a route. |
+
+The image contains no tun2proxy, no socat and no iproute2. `nano-init` carries
+its own TCP stack and speaks netlink directly, so a sandbox can be the agent
+and nothing else.
## Running it
diff --git a/development/examples/agent-harness/entrypoint.sh b/development/examples/agent-harness/entrypoint.sh
deleted file mode 100755
index ae58421b..00000000
--- a/development/examples/agent-harness/entrypoint.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/bin/sh
-# The sandbox's network, such as it is.
-#
-# This does the same job as the microVM's init: give the sandbox one route,
-# leading to the boundary, and then leave the agent alone. In particular it
-# sets no proxy variables. An agent that has to be told where its proxy is has
-# to cooperate with its own confinement, and the next library that ignores the
-# convention, or the next subprocess that clears the environment, is outside
-# it. Here the agent is unmodified and its traffic leaves through tun0 because
-# there is nowhere else for it to go.
-set -eu
-
-BOUNDARY_SOCKET="${BOUNDARY_SOCKET:-/run/agent.sock}"
-
-if [ ! -S "${BOUNDARY_SOCKET}" ]; then
- echo "no boundary socket at ${BOUNDARY_SOCKET}: this sandbox has no way out" >&2
- exit 1
-fi
-
-# HTTP clients and tun2proxy both want a host:port; the boundary is a Unix
-# socket. socat is plumbing only: it decides nothing, and every byte still
-# arrives at the same socket for the boundary to rule on.
-socat TCP-LISTEN:1080,bind=127.0.0.1,fork,reuseaddr "UNIX-CONNECT:${BOUNDARY_SOCKET}" &
-
-# Everything the sandbox is given an address for is link-local, because that is
-# what these addresses are: RFC 3927 describes a single link with no router,
-# which is exactly what a tun to the boundary is. Nothing here is routable, and
-# nothing here can be confused with a real destination — a sandbox numbered out
-# of 10.0.0.0/8 would eventually be deployed somewhere that already uses it.
-ip tuntap add dev tun0 mode tun
-ip link set dev tun0 up
-ip addr add 169.254.1.1/30 dev tun0
-ip route add default dev tun0
-
-# The resolver address is a fiction that never receives a packet: virtual DNS
-# answers in tun2proxy and hands the name to the boundary. That is the point —
-# mesh.sam.alt has no address, and the boundary is what turns a name into a
-# provider.
-echo "nameserver 169.254.1.53" > /etc/resolv.conf
-
-tun2proxy --tun tun0 --proxy socks5://127.0.0.1:1080 --dns virtual \
- --dns-addr 169.254.1.53 --virtual-dns-pool 169.254.64.0/18 &
-
-for _ in $(seq 1 100); do
- ip route show default | grep -q tun0 && break
- sleep 0.05
-done
-
-exec python agent.py "$@"
diff --git a/hack/lint.sh b/hack/lint.sh
index b7cc99c1..853757bf 100755
--- a/hack/lint.sh
+++ b/hack/lint.sh
@@ -23,6 +23,9 @@ REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
cd $REPO_ROOT
docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v2.11.4 golangci-lint run -v
+# nano-init is a separate module, so ./... above does not reach it.
+docker run --rm -v $(pwd):/app -w /app/cmd/nano-init golangci/golangci-lint:v2.11.4 golangci-lint run -v
+
# golangci-lint has no deadcode linter (removed upstream in v1.49) and its
# replacement, "unused", ignores exported identifiers. This catches exported
# code that is unreachable from every binary and test.