The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
test(sandbox): verify the systemd install path against a real systemd
shukieshah432a6cd98f83
diff --git a/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md b/.claude/skills/self-verify-beacon-in-sandbox/SKILL.mdindex a6a91695..6103bad9 100644--- a/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md+++ b/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md@@ -52,7 +52,7 @@ confusingly.```bashgo run ./cmd/beacon-sandbox run --scenario s02-bash-command # one scenario -- do this while iterating-go run ./cmd/beacon-sandbox run # the whole suite: all 7 scenarios, ~20 min+go run ./cmd/beacon-sandbox run # the whole suite, ~30 min```Pick the scenario matching what changed:@@ -63,8 +63,15 @@ Pick the scenario matching what changed:| File read or write signals | `s03-file-write` or `s04-file-read` || Prompt, session, token, or cost capture | `s01-hello` || Approval or permission handling | `s07-denied-tool` |+| `endpoint install`, config paths, service startup | `i01-install-supervised` |+| The systemd backend, unit files, Linux system mode | `i02-install-systemd` || Something broad, or preparing a PR | the whole suite (bare `run`) |+The `s0*` scenarios collect through `beacon ci exec`, a temporary collector. The `i0*` scenarios+install Beacon first, so they are the only ones that cover installation and service management.+`i02` runs inside a nested privileged container, because systemd will not start unless it is PID 1+and the sandbox provider's own init holds that slot — expect it to take several minutes longer.+Other flags: `--repeat N` to tell flaky from broken, `--keep-sandbox` to leave the instance up fordebugging.diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.mdindex c6ee9e70..5c120a25 100644--- a/beacon-sandbox/AGENTS.md+++ b/beacon-sandbox/AGENTS.md@@ -59,11 +59,13 @@ real money on the user's account, so:| File read or write signals | `--scenario s03-file-write` or `s04-file-read` || Prompt, session, token, or cost capture | `--scenario s01-hello` || Approval or permission handling | `--scenario s07-denied-tool` |+| `endpoint install`, config paths, service startup | `--scenario i01-install-supervised` |+| The systemd backend, unit files, Linux system mode | `--scenario i02-install-systemd` || Something broad, or preparing a PR | the whole suite (no `--scenario`) |```bashgo run ./cmd/beacon-sandbox run --scenario s02-bash-command # one scenario-go run ./cmd/beacon-sandbox run # the whole suite: all 7, ~20 min, ~$0.50+go run ./cmd/beacon-sandbox run # the whole suite, ~30 min, ~$0.70````run` with no `--scenario` is the suite command. Other flags: `--repeat N` to tell flaky fromdiff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.mdindex 6c14cd27..70601008 100644--- a/beacon-sandbox/README.md+++ b/beacon-sandbox/README.md@@ -27,7 +27,7 @@ cd cli/beacon && make build-linux-amd64 # the Beacon beincd ../../beacon-sandboxgo run ./cmd/beacon-sandbox doctor --fix # checks the setupgo run ./cmd/beacon-sandbox run --scenario s02-bash-command # one test, ~3 min-go run ./cmd/beacon-sandbox run # all seven, ~20 min+go run ./cmd/beacon-sandbox run # all of them, ~30 min````doctor` checks everything the tool needs and prints the exact fix for anything missing, so it is@@ -97,8 +97,12 @@ Other options are `drop-commands`, `drop-action:<action>`, and `plant-secret`. E## LimitationsModal only provides Linux x86 machines, so this **cannot test the macOS build** — launchd, the-signed installer, and notarization are out of reach. It also tests one collection path (the-temporary collector Beacon uses in CI and cloud agents) rather than a permanently installed Beacon,-and it confirms the thing it planted was recorded rather than proving nothing else was missed.+signed installer, and notarization are out of reach. It also confirms the thing it planted was+recorded rather than proving nothing else was missed.++The `i0*` tests install Beacon for real and cover installation and service management; the `s0*`+tests use a temporary collector instead, the path Beacon uses in CI and cloud agents. `i02` runs+inside a nested privileged container, since systemd only starts as PID 1 and Modal's own init holds+that slot, so it takes several minutes longer than the rest.The [full guide](https://docs.asymptotelabs.ai/contributing/beacon-sandbox) has the complete list.diff --git a/beacon-sandbox/image/image.go b/beacon-sandbox/image/image.goindex cfa9f693..3218bbaa 100644--- a/beacon-sandbox/image/image.go+++ b/beacon-sandbox/image/image.go@@ -37,6 +37,13 @@ type Spec struct {ClaudeVersion string// RepoRoot locates the Beacon build artifacts.RepoRoot string+ // WithDocker adds a Docker daemon, for scenarios that need a real systemd as PID 1.+ //+ // Modal cannot provide that directly -- its own init holds PID 1 in both lanes -- but systemd+ // can be PID 1 inside a privileged container, so systemd scenarios run nested. This layer is+ // opt-in because it is a large install that every other scenario would pay for and none of+ // them need.+ WithDocker bool}// LinuxArtifacts are the binaries the sandbox needs, as host paths.@@ -93,6 +100,19 @@ func Build(spec Spec, log func(string, ...any)) (sandbox.ImageSpec, error) {}pkgs := "curl ca-certificates git tar ripgrep procps sudo python3 jq"+ if spec.WithDocker {+ // docker.io rather than the upstream repo: it is one apt line, and the daemon only has+ // to be new enough to run a privileged container with a host cgroup namespace.+ //+ // systemd is installed here, in the image, even though nothing in the sandbox itself can+ // use it -- the provider's init owns PID 1. It is here because the nested container is+ // built by importing this very filesystem, and an image build is the only moment with+ // unrestricted network. Installing it later, inside the nested build, is what the first+ // attempt did: the pull succeeded because the daemon shares the sandbox's network, then+ // apt failed because a build container's traffic is NATed through docker0 and the+ // sandbox's egress allowlist does not cover it.+ pkgs += " docker.io iproute2 systemd systemd-sysv dbus"+ }layers := []string{"ENV DEBIAN_FRONTEND=noninteractive",@@ -112,6 +132,13 @@ func Build(spec Spec, log func(string, ...any)) (sandbox.ImageSpec, error) {AgentUser, WorkDir, WorkDir),}+ if spec.WithDocker {+ // Group membership is set at image build time; adding it later would need a re-login to+ // take effect, and the session runs as this user.+ layers = append(layers,+ fmt.Sprintf("RUN usermod -aG docker %s", AgentUser))+ }+return sandbox.ImageSpec{Base: spec.Base,Layers: layers,diff --git a/beacon-sandbox/runner/guest.go b/beacon-sandbox/runner/guest.gonew file mode 100644index 00000000..d7b0f4d4--- /dev/null+++ b/beacon-sandbox/runner/guest.go@@ -0,0 +1,335 @@+package runner++import (+ "context"+ "encoding/base64"+ "fmt"+ "path"+ "strings"+ "sync/atomic"+ "time"++ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox"+)++// guest is "the environment under test" -- where Beacon gets installed, where the agent session+// runs, and where the runtime log lands.+//+// For almost every scenario that is the sandbox instance itself. Scenarios that need a real+// systemd get a nested privileged container instead, because the sandbox provider's own init+// holds PID 1 and systemd refuses to start as anything else.+//+// Both sit behind one interface so everything above it -- canary, sentinel, quiescence,+// collection, verdict -- is byte-for-byte the same code. A systemd run is therefore judged by+// exactly the same rules as every other run, rather than by a parallel path that could drift into+// being more forgiving.+type guest interface {+ Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error)+ // Get copies a file out of the guest to a local path.+ Get(ctx context.Context, remotePath, localPath string) error+ // Describe names the environment for run metadata, so a verdict says where it ran.+ Describe() string+}++// directGuest runs commands in the sandbox instance itself.+type directGuest struct {+ p sandbox.Provider+ inst sandbox.Instance+}++func (g directGuest) Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error) {+ return g.p.Exec(ctx, g.inst, script, opts)+}++func (g directGuest) Get(ctx context.Context, remotePath, localPath string) error {+ return g.p.Get(ctx, g.inst, remotePath, localPath)+}++func (g directGuest) Describe() string { return "sandbox" }++// Names and paths for the nested lane. Fixed rather than random: a leftover container from an+// interrupted run is reclaimed by name on the next attempt instead of accumulating.+const (+ nestedContainer = "beacon-systemd"+ nestedImage = "beacon-systemd:local"+ // /var/tmp, not /tmp. Once systemd is PID 1 it may enable tmp.mount, which mounts a tmpfs+ // over /tmp -- a file placed there earlier is then shadowed and reads as missing. That+ // wasted real time in the packaging smoke test before it was understood.+ nestedStageDir = "/var/tmp/beacon-nested"+ // nestedStartMarker is how a script that ran is told apart from one that never did.+ nestedStartMarker = "__BEACON_NESTED_STARTED__"+)++// nestedGuest runs commands inside a privileged container where systemd is genuinely PID 1.+//+// Why nested at all: the sandbox provider's init occupies PID 1 in both of its lanes, and systemd+// exits immediately when it is not PID 1 ("Couldn't find an alternative telinit implementation to+// spawn"). A privileged container with a host cgroup namespace can hand PID 1 to systemd, so the+// systemd surface -- unit installation, `enable --now`, `Restart=always`, `systemctl is-active` --+// is exercised for real rather than against a stub.+type nestedGuest struct {+ p sandbox.Provider+ inst sandbox.Instance+ seq atomic.Int64+ // dockerVersion and systemdState are recorded so a run says what it ran on.+ dockerVersion string+ systemdState string+}++func (g *nestedGuest) Describe() string {+ return fmt.Sprintf("nested container %s (docker %s, systemd %s)",+ nestedContainer, g.dockerVersion, g.systemdState)+}++// Exec runs a script inside the container as the requested user.+//+// The script has to cross two shells: the sandbox's, then the container's. Escaping it twice is+// what broke repeatedly in the early probes -- anything holding a quote, a semicolon or a loop+// arrived mangled. So the script travels as base64, which no shell can misread, and is decoded+// into a file that the container executes directly. Nothing metacharacter-bearing ever appears on+// a command line.+func (g *nestedGuest) Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error) {+ n := g.seq.Add(1)+ stage := path.Join(nestedStageDir, fmt.Sprintf("step-%03d.sh", n))+ // The script announces itself on stderr before doing anything. Without that marker, a script+ // that never ran -- a failed copy, a stopped container -- looks exactly like a script that ran+ // and printed nothing, and the first nested run duly reported an empty `beacon version` and+ // carried on to fail somewhere less informative.+ enc := base64.StdEncoding.EncodeToString([]byte("echo " + nestedStartMarker + " >&2\n" + script))++ flags := []string{"exec"}+ if opts.User != "" {+ flags = append(flags, "-u", opts.User)+ }+ if opts.Dir != "" {+ flags = append(flags, "-w", opts.Dir)+ }+ if opts.HomeDir != "" {+ flags = append(flags, "-e", "HOME="+opts.HomeDir)+ }+ if len(opts.PathPrepend) > 0 {+ flags = append(flags, "-e", "PATH="+strings.Join(opts.PathPrepend, ":")++ ":/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")+ }+ // Named without a value on purpose: docker then takes the value from its own environment, so+ // the credential is inherited rather than written onto a command line. Passing `-e K=v` would+ // put the key in argv, which is precisely what the argv leak check exists to catch -- the+ // harness must not be the thing that leaks it.+ for _, k := range nestedPassThroughEnv {+ flags = append(flags, "-e", k)+ }+ flags = append(flags, nestedContainer, "sh", stage)++ wrapper := fmt.Sprintf("set -e\nmkdir -p %s\nprintf %%s '%s' | base64 -d > %s\n"++ "docker cp %s %s:%s >/dev/null\nexec docker %s\n",+ shq(nestedStageDir), enc, shq(stage),+ shq(stage), nestedContainer, shq(stage),+ strings.Join(quoteAll(flags), " "))++ // The docker CLI runs as root in the sandbox and needs its environment preserved, since that+ // is where the pass-through credentials live.+ timeout := opts.Timeout+ if timeout == 0 {+ timeout = 5 * time.Minute+ }+ r, err := g.p.Exec(ctx, g.inst, wrapper, sandbox.ExecOpts{+ User: "root", Dir: "/", Timeout: timeout, PreserveEnv: true,+ })+ if err != nil {+ return r, err+ }+ if !strings.Contains(r.Stderr, nestedStartMarker) {+ // Reported as an error rather than a non-zero result, because the two mean different+ // things to every caller: a non-zero result is a finding about Beacon, while this is the+ // harness failing to reach the container at all.+ return r, fmt.Errorf("the script never started inside %s (rc=%d): %s",+ nestedContainer, r.ExitCode, tailOf(r.Stdout+r.Stderr))+ }+ r.Stderr = strings.Replace(r.Stderr, nestedStartMarker+"\n", "", 1)+ return r, nil+}++// nestedPassThroughEnv are the variables the agent session needs inside the container. Only+// credentials Claude Code reads; nothing Beacon needs, because Beacon's configuration comes from+// the install it performs in there.+var nestedPassThroughEnv = []string{+ "ANTHROPIC_API_KEY",+ "ANTHROPIC_AUTH_TOKEN",+ "CLAUDE_CODE_OAUTH_TOKEN",+}++// Get copies a file out of the container, then out of the sandbox.+func (g *nestedGuest) Get(ctx context.Context, remotePath, localPath string) error {+ n := g.seq.Add(1)+ stage := path.Join(nestedStageDir, fmt.Sprintf("fetch-%03d", n))+ script := fmt.Sprintf("set -e\nmkdir -p %s\ndocker cp %s:%s %s\n",+ shq(nestedStageDir), nestedContainer, shq(remotePath), shq(stage))+ r, err := g.p.Exec(ctx, g.inst, script, sandbox.ExecOpts{+ User: "root", Dir: "/", Timeout: 2 * time.Minute,+ })+ if err != nil {+ return fmt.Errorf("docker cp %s out of the container: %w", remotePath, err)+ }+ if r.ExitCode != 0 {+ return fmt.Errorf("docker cp %s out of the container failed (rc=%d): %s",+ remotePath, r.ExitCode, oneLine(r.Stdout+r.Stderr))+ }+ return g.p.Get(ctx, g.inst, stage, localPath)+}++// startNested brings up dockerd and a systemd container, and refuses to return a guest that is+// not actually running systemd as PID 1.+//+// Every step reports what it saw. A half-started environment that silently degraded to "no+// service manager" would make a systemd scenario pass while testing the supervised backend, which+// is the one outcome worth going to some trouble to prevent.+func startNested(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,+ logf func(string, ...any)) (*nestedGuest, error) {++ root := sandbox.ExecOpts{User: "root", Dir: "/", Timeout: 10 * time.Minute, PreserveEnv: true}+ g := &nestedGuest{p: p, inst: inst}++ logf("starting dockerd (this scenario needs systemd as PID 1, which only a privileged container can provide)")+ r, err := p.Exec(ctx, inst, dockerdScript, root)+ if err != nil || r.ExitCode != 0 {+ return nil, fmt.Errorf("dockerd did not start (rc=%d): %v %s",+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))+ }+ g.dockerVersion = oneLine(r.Stdout)+ logf("dockerd ready: %s", g.dockerVersion)++ logf("building the systemd container image")+ r, err = p.Exec(ctx, inst, nestedBuildScript(), root)+ if err != nil || r.ExitCode != 0 {+ return nil, fmt.Errorf("building the systemd image failed (rc=%d): %v %s",+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))+ }++ logf("starting the systemd container")+ r, err = p.Exec(ctx, inst, nestedRunScript(), root)+ if err != nil || r.ExitCode != 0 {+ return nil, fmt.Errorf("the systemd container did not come up (rc=%d): %v %s",+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))+ }+ // The script prints PID1=<comm> STATE=<is-system-running> and exits non-zero unless PID 1 is+ // systemd, so reaching here means the requirement is met; the values are kept for metadata.+ for _, f := range strings.Fields(oneLine(r.Stdout)) {+ if v, ok := strings.CutPrefix(f, "STATE="); ok {+ g.systemdState = v+ }+ }+ logf("systemd is PID 1 in %s (%s)", nestedContainer, oneLine(r.Stdout))++ // docker cp refuses a destination whose parent directory does not exist, and the image import+ // excludes /var/tmp, so the container starts without one. Created once here rather than per+ // exec: if it is missing, every single command fails, so this is the right place to find out.+ r, err = p.Exec(ctx, inst, fmt.Sprintf("mkdir -p %s && docker exec %s mkdir -p %s",+ shq(nestedStageDir), nestedContainer, shq(nestedStageDir)), root)+ if err != nil || r.ExitCode != 0 {+ return nil, fmt.Errorf("could not create the staging directory %s (rc=%d): %v %s",+ nestedStageDir, r.ExitCode, err, tailOf(r.Stdout+r.Stderr))+ }+ return g, nil+}++// dockerdScript starts the daemon if it is not already up, and fails loudly with its log if not.+const dockerdScript = `set -u+if docker info >/dev/null 2>&1; then+ docker version --format '{{.Server.Version}}'+ exit 0+fi+(dockerd >/var/log/dockerd.log 2>&1 &)+i=0+while [ "$i" -lt 60 ]; do+ docker info >/dev/null 2>&1 && break+ i=$((i+1)); sleep 2+done+if ! docker info >/dev/null 2>&1; then+ echo "dockerd never became ready; last 30 lines of its log:"+ tail -30 /var/log/dockerd.log 2>/dev/null || echo "(no log)"+ exit 1+fi+docker version --format '{{.Server.Version}}'+`++// nestedBuildScript turns the sandbox's own filesystem into the container image.+//+// The obvious approach -- a Dockerfile that pulls a base image and installs systemd -- does not+// work here, and the way it fails is worth recording. The pull succeeds, because the daemon shares+// the sandbox's network and the registry is reachable. Then `apt-get` fails inside the build+// container, because a build container's traffic is NATed through docker0 and the sandbox's+// outbound allowlist has no rule for it. The first attempt spent a full sandbox launch discovering+// that from a truncated apt exit code.+//+// Importing the live root filesystem instead needs no network whatsoever, and it is strictly more+// faithful: the container gets the exact Beacon binary under test, the same pinned Claude Code, the+// same seeded git fixture, and the same `agent` account with the same uid. There is no second+// environment that could drift from the one every other scenario runs in.+//+// --one-file-system is deliberately not used: it would silently drop anything the provider mounts+// separately, and a missing binary would surface much later as a confusing exec failure. The+// exclusions are named instead, so what is left out is visible.+func nestedBuildScript() string {+ return fmt.Sprintf(`set -eu+if docker image inspect %[1]s >/dev/null 2>&1; then+ echo "reusing %[1]s"+ exit 0+fi+tar -cf - --numeric-owner --exclude=./proc --exclude=./sys --exclude=./dev --exclude=./run --exclude=./tmp --exclude=./var/lib/docker --exclude=./var/tmp --exclude=./mnt --exclude=./media --exclude=./lost+found -C / . 2>/dev/null | docker import -c 'CMD ["/sbin/init"]' - %[1]s+docker image inspect %[1]s >/dev/null+`, nestedImage)+}++// nestedRunScript starts the container and proves systemd owns PID 1 before anything is installed.+//+// The flags are not incidental. --privileged plus a host cgroup namespace and a writable+// /sys/fs/cgroup are what let systemd manage units at all; the two tmpfs mounts give it the+// runtime directories it expects to own. Drop any of them and systemd starts but cannot start+// services, which presents as a mystifying unit failure much later.+//+// --stop-signal is how systemd is asked to shut down cleanly. docker import cannot carry a+// STOPSIGNAL, so it is set here rather than in the image.+func nestedRunScript() string {+ return fmt.Sprintf(`set -u+docker rm -f %[1]s >/dev/null 2>&1 || true+docker run -d --name %[1]s --privileged --cgroupns=host --stop-signal=SIGRTMIN+3 -v /sys/fs/cgroup:/sys/fs/cgroup:rw --tmpfs /run --tmpfs /run/lock %[2]s >/dev/null || exit 1+state=""+i=0+while [ "$i" -lt 45 ]; do+ state="$(docker exec %[1]s systemctl is-system-running 2>/dev/null || true)"+ case "$state" in running|degraded) break ;; esac+ i=$((i+1)); sleep 2+done+pid1="$(docker exec %[1]s cat /proc/1/comm 2>/dev/null || true)"+echo "PID1=$pid1 STATE=$state"+if [ "$pid1" != systemd ]; then+ echo "PID 1 is not systemd, so this scenario would silently test the supervised backend instead:"+ docker logs %[1]s 2>&1 | tail -20+ exit 1+fi+`, nestedContainer, nestedImage)+}++// tailOf keeps the end of a command's output rather than its beginning.+//+// Build and daemon output is long, and the sentence that says what went wrong is the last one. The+// first version of this file reported these failures through oneLine, which truncates the head --+// so the report was pages of layer-download chatter and no error, and the actual cause took another+// full sandbox launch to find.+func tailOf(s string) string {+ lines := strings.Split(strings.TrimRight(s, "\n"), "\n")+ if len(lines) > 12 {+ lines = lines[len(lines)-12:]+ }+ return oneLine(strings.Join(lines, " | "))+}++// quoteAll shell-quotes each argument. Used for docker's own argv, which is built from values the+// harness controls but still passes through the sandbox's shell.+func quoteAll(args []string) []string {+ out := make([]string, len(args))+ for i, a := range args {+ out[i] = shq(a)+ }+ return out+}diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.goindex 0e9c7b17..b073e9c2 100644--- a/beacon-sandbox/runner/runner.go+++ b/beacon-sandbox/runner/runner.go@@ -120,6 +120,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts OptimgSpec, err := image.Build(image.Spec{RepoRoot: opts.RepoRoot,ClaudeVersion: opts.ClaudeVersion,+ WithDocker: sc.NeedsRealSystemd(),}, logf)if err != nil {return art, err@@ -131,7 +132,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Optart.Meta["image"] = img.Ref()logf("image %s", img.Ref())- inst, err := p.Launch(ctx, img, sandbox.LaunchSpec{+ launch := sandbox.LaunchSpec{CPU: 2,MemLimitMiB: 4096,Timeout: sc.Timeout() + 5*time.Minute, // headroom for setup and collection@@ -144,7 +145,21 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt// here, which is how the local-only posture gets asserted rather than assumed."api.anthropic.com", "statsig.anthropic.com", "*.anthropic.com",},- })+ }+ if sc.NeedsRealSystemd() {+ // The VM lane has a real kernel and cgroup2, which dockerd needs; the default lane does+ // not. Neither lane makes our entrypoint PID 1, hence the nested container -- see+ // startNested.+ launch.Lane = sandbox.LaneVM+ launch.CPU, launch.MemLimitMiB = 4, 8192 // dockerd + systemd + collector + the agent+ launch.Timeout = sc.Timeout() + 20*time.Minute+ // The allowlist is deliberately not widened. Building the nested image from the sandbox's+ // own filesystem needs no network, so these scenarios assert exactly the same egress+ // posture as every other one -- which is worth more than the convenience of a registry+ // pull would have been.+ art.Meta["lane"] = string(launch.Lane)+ }+ inst, err := p.Launch(ctx, img, launch)if err != nil {return art, fmt.Errorf("launch: %w", err)}@@ -162,20 +177,44 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt}// Finish the artifact layer: the snapshot captured the files, but chmod and symlinks- // have to happen in a live instance.+ // have to happen in a live instance. Always the sandbox itself, never the guest -- this+ // prepares the very tree the guest will mount.if _, err := p.Exec(ctx, inst, image.PostPushLayers(), sandbox.ExecOpts{User: "root", Timeout: time.Minute}); err != nil {return art, fmt.Errorf("prepare binaries: %w", err)}- if v, err := p.Exec(ctx, inst, "beacon version; claude --version; id -un; uname -m", agent); err == nil {- art.Meta["versions"] = oneLine(v.Stdout)- logf("versions: %s", art.Meta["versions"])+ // Everything the scenario itself does happens in the guest. For most scenarios that is the+ // sandbox; a scenario needing a real systemd gets a nested container instead. See guest.go.+ var g guest = directGuest{p: p, inst: inst}+ if sc.NeedsRealSystemd() {+ ng, err := startNested(ctx, p, inst, logf)+ if err != nil {+ return art, err+ }+ g = ng+ art.Meta["systemd_state"] = ng.systemdState+ art.Meta["docker_version"] = ng.dockerVersion+ }+ art.Meta["guest"] = g.Describe()++ // Not just recorded -- checked. An empty version line means beacon or claude is not reachable+ // in the guest, and every later failure is then a symptom of that rather than of Beacon. The+ // first nested run printed "versions:" with nothing after it and kept going.+ v, err := g.Exec(ctx, "beacon version; claude --version; id -un; uname -m", agent)+ if err != nil {+ return art, fmt.Errorf("could not run commands in the %s guest: %w", g.Describe(), err)+ }+ art.Meta["versions"] = oneLine(v.Stdout)+ logf("versions: %s", art.Meta["versions"])+ if !strings.Contains(v.Stdout, "beacon version") {+ return art, fmt.Errorf("the guest did not report a Beacon version, so the binary under "++ "test is not reachable there (rc=%d): %s", v.ExitCode, oneLine(v.Stdout+v.Stderr))}sentinel := scenario.Expand(sc.Sentinel, canary, "", image.WorkDir)if sc.Setup != "" {setup := scenario.Expand(sc.Setup, canary, sentinel, image.WorkDir)- if r, err := p.Exec(ctx, inst, setup, agent); err != nil || r.ExitCode != 0 {+ if r, err := g.Exec(ctx, setup, agent); err != nil || r.ExitCode != 0 {return art, fmt.Errorf("scenario setup failed (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stderr))}}@@ -199,14 +238,14 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt// collector actually receives telemetry. Nothing else in the suite can tell you that.var res sandbox.Resultif sc.Install != nil {- installedLog, err := doInstall(ctx, p, inst, sc, agent, &art, logf)+ installedLog, err := doInstall(ctx, g, sc, agent, &art, logf)if err != nil {return art, err}logPath = installedLoglogf("running session against the installed endpoint (budget $%.2f, timeout %s)",sc.Budget(), sc.Timeout())- res, err = p.Exec(ctx, inst, plainSessionScript(sc, prompt), sandbox.ExecOpts{+ res, err = g.Exec(ctx, plainSessionScript(sc, prompt), sandbox.ExecOpts{User: image.AgentUser, Dir: workDirFor(sc), HomeDir: image.AgentHome,PreserveEnv: true, PathPrepend: image.PathPrepend(), Timeout: sc.Timeout() + time.Minute,})@@ -217,7 +256,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt} else {logf("running session (budget $%.2f, timeout %s)", sc.Budget(), sc.Timeout())var err error- res, err = p.Exec(ctx, inst, sessionScript(sc, prompt, logPath), sandbox.ExecOpts{+ res, err = g.Exec(ctx, sessionScript(sc, prompt, logPath), sandbox.ExecOpts{User: image.AgentUser, Dir: workDirFor(sc), HomeDir: image.AgentHome,PreserveEnv: true, PathPrepend: image.PathPrepend(), Timeout: sc.Timeout() + time.Minute,})@@ -242,7 +281,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt// disabling exactly the signal that distinguishes a dead agent from a capture gap.outPath := shq(sessionFile(sc, "claude-out.json"))errPath := shq(sessionFile(sc, "claude-err.txt"))- if r, err := p.Exec(ctx, inst, "cat "+outPath+" 2>/dev/null || true", agent); err == nil {+ if r, err := g.Exec(ctx, "cat "+outPath+" 2>/dev/null || true", agent); err == nil {var result map[string]anyif json.Unmarshal([]byte(strings.TrimSpace(r.Stdout)), &result) == nil {art.SessionOK, art.SessionKnown = sessionOutcome(result)@@ -255,7 +294,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Optlogf("claude: %s", art.Meta["claude_result"])_ = os.WriteFile(filepath.Join(runDir, "claude-result.json"), []byte(r.Stdout), 0o600)} else {- if e, err2 := p.Exec(ctx, inst, "tail -c 800 "+errPath+" 2>/dev/null || true", agent); err2 == nil {+ if e, err2 := g.Exec(ctx, "tail -c 800 "+errPath+" 2>/dev/null || true", agent); err2 == nil {art.Meta["claude_stderr"] = oneLine(e.Stdout)}}@@ -273,7 +312,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt// sentinel file, which would have misreported the second as a broken probe.probe := fmt.Sprintf("if [ -f %[1]s ]; then echo __FOUND__; cat %[1]s; else echo __MISSING__; fi",shq(sentinel))- r, execErr := p.Exec(ctx, inst, probe, agent)+ r, execErr := g.Exec(ctx, probe, agent)art.SentinelProbed = execErr == nil &&(strings.Contains(r.Stdout, "__FOUND__") || strings.Contains(r.Stdout, "__MISSING__"))if !art.SentinelProbed {@@ -343,12 +382,12 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt// Wait for quiescence rather than sleeping a fixed amount: the collector batches at 5s// and Claude Code's metric interval defaults to 60s, so a fixed sleep is either wasteful// or wrong. Record how long it took -- that latency is itself a finding.- stable, waited := waitForQuiescence(ctx, p, inst, agent, logPath, logf)+ stable, waited := waitForQuiescence(ctx, g, agent, logPath, logf)art.Meta["quiescence_seconds"] = fmt.Sprintf("%.0f", waited.Seconds())art.Meta["quiescent"] = fmt.Sprintf("%v", stable)local := filepath.Join(runDir, "runtime.jsonl")- if err := p.Get(ctx, inst, logPath, local); err != nil {+ if err := g.Get(ctx, logPath, local); err != nil {return art, fmt.Errorf("collect runtime log: %w", err)}art.RuntimeLog = local@@ -417,7 +456,7 @@ func sessionScript(sc scenario.Scenario, prompt, logPath string) string {}// waitForQuiescence polls the log until its size stops changing, bounded.-func waitForQuiescence(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,+func waitForQuiescence(ctx context.Context, g guest,opts sandbox.ExecOpts, logPath string, logf func(string, ...any)) (bool, time.Duration) {const (@@ -430,7 +469,7 @@ func waitForQuiescence(ctx context.Context, p sandbox.Provider, inst sandbox.InsstableSince := time.Time{}for time.Since(start) < maxWait {- r, err := p.Exec(ctx, inst, "wc -c < "+shq(logPath)+" 2>/dev/null || echo 0", opts)+ r, err := g.Exec(ctx, "wc -c < "+shq(logPath)+" 2>/dev/null || echo 0", opts)if err == nil {cur := strings.TrimSpace(r.Stdout)if cur == last && cur != "0" {@@ -580,7 +619,7 @@ func sampleCountOf(verdict string) string {// The log path comes from `endpoint status --json` rather than being assumed, because the whole// point of an install scenario is to check what Beacon actually did — hardcoding the expected// path would make the scenario agree with itself instead of with Beacon.-func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,+func doInstall(ctx context.Context, g guest,sc scenario.Scenario, agent sandbox.ExecOpts, art *Artifacts, logf func(string, ...any)) (string, error) {in := sc.Install@@ -605,12 +644,12 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,// Dry run first. It is free, and it is where a plan that advertises the wrong service// manager for the host would show up — so a regression there needs no second scenario.- if r, err := p.Exec(ctx, inst, "beacon endpoint install "+scope+svc+" --harness claude --dry-run 2>&1", opts); err == nil {+ if r, err := g.Exec(ctx, "beacon endpoint install "+scope+svc+" --harness claude --dry-run 2>&1", opts); err == nil {art.Meta["install_dry_run"] = oneLine(r.Stdout)logf("dry-run plan: %s", art.Meta["install_dry_run"])}- r, err := p.Exec(ctx, inst, "beacon endpoint install "+scope+svc+" --harness claude 2>&1", opts)+ r, err := g.Exec(ctx, "beacon endpoint install "+scope+svc+" --harness claude 2>&1", opts)art.Meta["install_output"] = oneLine(r.Stdout + r.Stderr)if err != nil || r.ExitCode != 0 {logf("install FAILED rc=%d: %s", r.ExitCode, art.Meta["install_output"])@@ -619,13 +658,40 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,}logf("install ok: %s", art.Meta["install_output"])- logPath, err := installedLogPath(ctx, p, inst, scope, opts, in, art, logf)+ // A system install configures the *installing* user's agent runtime, which for a package+ // install is root -- not the person whose sessions matter. The package postinstall therefore+ // runs a second step to resolve that person and point their Claude Code at the collector, and+ // a system-mode scenario has to do the same or its capture assertions would be testing+ // nothing but root's unused settings file.+ //+ // SUDO_USER is how the real path identifies them (`sudo apt install ./beacon.deb`), so it is+ // what gets set here rather than inventing a flag that no user would pass.+ if system {+ cmd := fmt.Sprintf("SUDO_USER=%s beacon endpoint user-config repair-installed "++ "--system --harness claude --json 2>&1", image.AgentUser)+ u, err := g.Exec(ctx, cmd, opts)+ art.Meta["install_user_config"] = oneLine(u.Stdout + u.Stderr)+ if err != nil || u.ExitCode != 0 {+ return "", fmt.Errorf("configuring %s's agent runtime failed (rc=%d): %v %s",+ image.AgentUser, u.ExitCode, err, art.Meta["install_user_config"])+ }+ logf("user config: %s", art.Meta["install_user_config"])+ // A skip is not a pass. The command reports one rather than failing, because on a real+ // unattended host there may be nobody to configure -- but in this scenario there is, and+ // treating a skip as success would let a broken resolver produce a green run.+ if strings.Contains(art.Meta["install_user_config"], "skipped_reason") {+ return "", fmt.Errorf("user-config skipped instead of configuring %s: %s",+ image.AgentUser, art.Meta["install_user_config"])+ }+ }++ logPath, err := installedLogPath(ctx, g, scope, opts, in, art, logf)if err != nil {return "", err}// Doctor output is recorded whether or not it passes: on a failure it is the first thing// worth reading, and it costs one exec.- if d, err := p.Exec(ctx, inst, "beacon endpoint doctor "+scope+" --json 2>&1", opts); err == nil {+ if d, err := g.Exec(ctx, "beacon endpoint doctor "+scope+" --json 2>&1", opts); err == nil {art.Meta["install_doctor"] = oneLine(d.Stdout)}return logPath, nil@@ -636,11 +702,11 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,// A status that cannot be parsed is not fatal — the run can still proceed against the// documented default path — but ExpectStatusRunning is enforced, since a scenario asking for a// running service and silently accepting an unparseable status would verify nothing.-func installedLogPath(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,+func installedLogPath(ctx context.Context, g guest,scope string, opts sandbox.ExecOpts, in *scenario.Install, art *Artifacts,logf func(string, ...any)) (string, error) {- s, err := p.Exec(ctx, inst, "beacon endpoint status "+scope+" --json 2>&1", opts)+ s, err := g.Exec(ctx, "beacon endpoint status "+scope+" --json 2>&1", opts)if err == nil {art.Meta["install_status"] = oneLine(s.Stdout)var status struct {@@ -665,14 +731,19 @@ func installedLogPath(ctx context.Context, p sandbox.Provider, inst sandbox.Instreturn "", fmt.Errorf("service did not report running after install: %s",status.Service.Message)}+ if in.ExpectServiceKind != "" && status.Service.Kind != in.ExpectServiceKind {+ return "", fmt.Errorf("service backend is %q, want %q: the scenario would "++ "otherwise verify a different service manager than it claims to",+ status.Service.Kind, in.ExpectServiceKind)+ }if status.LogPath != "" {return status.LogPath, nil}}}- if in.ExpectStatusRunning {- return "", fmt.Errorf("could not read service status, so a running service cannot be " +- "confirmed; refusing to report a pass the run did not earn")+ if in.ExpectStatusRunning || in.ExpectServiceKind != "" {+ return "", fmt.Errorf("could not read service status, so neither a running service nor " ++ "the selected backend can be confirmed; refusing to report a pass the run did not earn")}logf("status unparseable; falling back to the documented default log path")if in.Mode == "system" {diff --git a/beacon-sandbox/scenario/scenario.go b/beacon-sandbox/scenario/scenario.goindex 4fad7540..812182ff 100644--- a/beacon-sandbox/scenario/scenario.go+++ b/beacon-sandbox/scenario/scenario.go@@ -52,6 +52,19 @@ type Install struct {Service string `yaml:"service,omitempty"`// ExpectStatusRunning requires the collector service to report running after install.ExpectStatusRunning bool `yaml:"expect_status_running,omitempty"`+ // ExpectServiceKind requires the endpoint to report this service backend after install.+ //+ // Which backend got selected is the substance of a service-manager scenario, and it is not+ // implied by "running": the supervised fallback also reports running, so a systemd scenario+ // on a host without systemd would otherwise pass while testing the wrong thing.+ ExpectServiceKind string `yaml:"expect_service_kind,omitempty"`+ // NeedsRealSystemd runs the install inside a nested privileged container where systemd is+ // genuinely PID 1.+ //+ // The sandbox provider cannot offer that directly: its own init holds PID 1, so systemctl+ // refuses. Rather than make a scenario describe VM lanes and Docker, it states the+ // requirement and the runner arranges it.+ NeedsRealSystemd bool `yaml:"needs_real_systemd,omitempty"`}// Scenario is one declarative case.@@ -127,6 +140,22 @@ func (s Scenario) Validate() error {default:return fmt.Errorf("%s: install.mode %q must be user or system", s.ID, s.Install.Mode)}+ // A systemd requirement with the supervised backend, or without system mode, would+ // silently exercise something other than what the scenario claims to test.+ if s.Install.NeedsRealSystemd && s.Install.Service == "none" {+ return fmt.Errorf("%s: needs_real_systemd with service=none is contradictory; "++ "the supervised backend does not use systemd", s.ID)+ }+ if s.Install.NeedsRealSystemd && s.Install.Mode != "system" {+ return fmt.Errorf("%s: needs_real_systemd requires mode=system; a user unit in a "++ "container has no logind session to attach to", s.ID)+ }+ switch s.Install.ExpectServiceKind {+ case "", "systemd", "launchd", "none":+ default:+ return fmt.Errorf("%s: install.expect_service_kind %q must be systemd, launchd or none",+ s.ID, s.Install.ExpectServiceKind)+ }switch s.Install.Service {case "", "auto", "systemd", "launchd", "none":default:@@ -273,3 +302,8 @@ func isFilenameByte(b byte) bool {}return false}++// NeedsRealSystemd reports whether this scenario must run somewhere systemd is genuinely PID 1.+func (s Scenario) NeedsRealSystemd() bool {+ return s.Install != nil && s.Install.NeedsRealSystemd+}diff --git a/beacon-sandbox/scenario/scenario_test.go b/beacon-sandbox/scenario/scenario_test.goindex 392cd3c7..641423b7 100644--- a/beacon-sandbox/scenario/scenario_test.go+++ b/beacon-sandbox/scenario/scenario_test.go@@ -347,3 +347,27 @@ func TestInstallModeAndServiceAreValidated(t *testing.T) {}}}++// A systemd requirement combined with the supervised backend, or without system mode, would+// silently exercise something other than what the scenario claims — and the run would pass,+// reporting coverage it does not have.+func TestNeedsRealSystemdRejectsContradictions(t *testing.T) {+ base := func(in *Install) Scenario {+ return Scenario{ID: "i", Prompt: "do a thing", Install: in,+ Expect: []Expect{{Action: "prompt.submitted", Why: "baseline"}}}+ }+ contradictory := []*Install{+ {NeedsRealSystemd: true, Mode: "system", Service: "none"},+ {NeedsRealSystemd: true, Mode: "user", Service: "systemd"},+ {NeedsRealSystemd: true}, // mode defaults to user+ }+ for _, in := range contradictory {+ if err := base(in).Validate(); err == nil {+ t.Errorf("%+v is contradictory and must be rejected", in)+ }+ }+ // The coherent combination must be accepted.+ if err := base(&Install{NeedsRealSystemd: true, Mode: "system", Service: "systemd"}).Validate(); err != nil {+ t.Errorf("system+systemd is the intended combination: %v", err)+ }+}diff --git a/beacon-sandbox/scenarios/i01-install-supervised.yaml b/beacon-sandbox/scenarios/i01-install-supervised.yamlindex 016dadd0..1b40cbb4 100644--- a/beacon-sandbox/scenarios/i01-install-supervised.yaml+++ b/beacon-sandbox/scenarios/i01-install-supervised.yaml@@ -18,6 +18,9 @@ install:mode: userservice: noneexpect_status_running: true+ # Asserted rather than assumed: if systemd detection ever started returning true in a container,+ # this scenario would silently stop covering the fallback backend it exists to cover.+ expect_service_kind: noneprompt: >-Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}}diff --git a/beacon-sandbox/scenarios/i02-install-systemd.yaml b/beacon-sandbox/scenarios/i02-install-systemd.yamlnew file mode 100644index 00000000..8a835c50--- /dev/null+++ b/beacon-sandbox/scenarios/i02-install-systemd.yaml@@ -0,0 +1,54 @@+id: i02-install-systemd+# The scenario a Linux user actually lives: a system-mode install managed by systemd, capturing a+# real agent session.+#+# i01 proves `beacon endpoint install` works and that an installed collector receives telemetry,+# but it runs the supervised backend -- the fallback for containers, where nothing restarts the+# collector and no unit file is involved. So it says nothing about the systemd path, which is what+# a package install on a Linux workstation uses.+#+# `needs_real_systemd` is what makes this possible. The sandbox provider's own init holds PID 1, and+# systemd exits immediately when it is not PID 1, so the runner starts a privileged container where+# systemd genuinely is PID 1 and performs the whole scenario in there. Everything else -- canary,+# sentinel, quiescence, verdict -- is the same code as every other scenario.+#+# What this proves that nothing else can: a systemd unit that systemd accepts and reports active,+# a collector started by that unit, the installing user's Claude Code pointed at it, and a live+# session's telemetry landing in the system runtime log.+install:+ mode: system+ service: systemd+ needs_real_systemd: true+ expect_status_running: true+ # Not redundant with the above. The supervised backend also reports running, so without this a+ # host where systemd detection failed would pass this scenario while testing the fallback.+ expect_service_kind: systemd++prompt: >-+ Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}}+sentinel: /home/agent/work/i02.sentinel+budget_usd: 1.0+timeout_seconds: 420++expect:+ - action: prompt.submitted+ contains: ["{{canary}}"]+ fields: ["prompt.text", "session.id", "harness.name"]+ why: >-+ The end of the chain: systemd started the collector, the install pointed the user's agent at+ it, and the session's prompt arrived. A failure here means one of those three links is+ broken, and the install output plus `systemctl status` in the run metadata says which.++ - action: token.usage+ fields: ["gen_ai.usage.input_tokens", "model"]+ why: >-+ Metrics travel a separate pipeline from logs, so a working prompt event does not imply a+ working metrics path through a unit-started collector.++ - action: command.executed+ contains: ["{{canary}}"]+ fields: ["command.command"]+ why: >-+ The detection surface that matters most, asserted through the systemd path. Non-optional on+ purpose: a systemd-managed collector is never allowed to look healthier than the supervised+ one for the same telemetry.diff --git a/docs/contributing/beacon-sandbox.mdx b/docs/contributing/beacon-sandbox.mdxindex 982ab19e..0c295115 100644--- a/docs/contributing/beacon-sandbox.mdx+++ b/docs/contributing/beacon-sandbox.mdx@@ -177,7 +177,7 @@ Tests encode what Beacon *should* do. When Beacon's behaviour legitimately impro```bashgo run ./cmd/beacon-sandbox run --scenario s01-hello # one test-go run ./cmd/beacon-sandbox run # all of them, ~20 min+go run ./cmd/beacon-sandbox run # all of them, ~30 min```| Test | Covers |@@ -189,9 +189,36 @@ go run ./cmd/beacon-sandbox run # all of them, ~20 min| `s05-repo-task` | A multi-step task in a git repository || `s06-subagent` | Sub-agent activity || `s07-denied-tool` | Approvals and denials |+| `i01-install-supervised` | `beacon endpoint install`, then a session against the installed collector |+| `i02-install-systemd` | A system-mode install managed by systemd, then a session against it |Use one test while you are iterating. Run all of them before opening a pull request.+The `s0*` tests collect through a temporary collector — the path Beacon uses in CI and cloud agents.+The `i0*` tests install Beacon properly first, so they cover things the others cannot: whether config+lands where Beacon says it did, whether the service actually starts, and whether a permanently+installed collector receives telemetry.++`i02` is the slowest test by some margin, and the reason is worth knowing. A system-mode install on+Linux is managed by systemd, and systemd only runs as process 1 — but Modal's own startup process+holds that slot, so systemd refuses to start. The way around it is a privileged container inside the+sandbox, where systemd *is* process 1, and running the whole test in there. The tool sets this up for+you when a test asks for it:++```yaml title="Asking for a real systemd"+install:+ mode: system+ service: systemd+ needs_real_systemd: true+ expect_status_running: true+ expect_service_kind: systemd+```++`expect_service_kind` is not redundant with `expect_status_running`. Beacon falls back to a plain+background collector when it finds no service manager, and that fallback also reports itself as+running — so without naming the backend you expect, a test meant to cover systemd could quietly pass+while covering the fallback instead.+## Commands| Command | What it does |@@ -252,7 +279,7 @@ go test ./...Modal only provides Linux machines, so **this cannot test the macOS build**. Anything macOS-specific — launchd, the signed installer, notarization — is out of reach.</Warning>-- **It tests one collection path.** Specifically the temporary-collector path Beacon uses in CI and cloud agents, not a permanently installed Beacon.+- **The systemd tests run nested.** `i02` runs inside a container within the sandbox, so it is slower and needs a few more minutes of patience than the others.- **It checks presence, not completeness.** It confirms the thing it planted was recorded. It cannot tell you whether Beacon missed something it never knew to look for.- **Linux x86 only.** Modal does not offer a choice of processor architecture.- **It costs money and takes minutes.** Each test is a real API session.