Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

test(sandbox): verify the systemd install path against a real systemd

shukieshah432a6cd98f83
diff --git a/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md b/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md
index a6a91695..6103bad9 100644
--- a/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md
+++ b/.claude/skills/self-verify-beacon-in-sandbox/SKILL.md
@@ -52,7 +52,7 @@ confusingly.
```bash
go run ./cmd/beacon-sandbox run --scenario s02-bash-command # one scenario -- do this while iterating
-go run ./cmd/beacon-sandbox run # the whole suite: all 7 scenarios, ~20 min
+go run ./cmd/beacon-sandbox run # the whole suite, ~30 min
```
Pick the scenario matching what changed:
@@ -63,8 +63,15 @@ Pick the scenario matching what changed:
| File read or write signals | `s03-file-write` or `s04-file-read` |
| Prompt, session, token, or cost capture | `s01-hello` |
| Approval or permission handling | `s07-denied-tool` |
+| `endpoint install`, config paths, service startup | `i01-install-supervised` |
+| The systemd backend, unit files, Linux system mode | `i02-install-systemd` |
| Something broad, or preparing a PR | the whole suite (bare `run`) |
+The `s0*` scenarios collect through `beacon ci exec`, a temporary collector. The `i0*` scenarios
+install Beacon first, so they are the only ones that cover installation and service management.
+`i02` runs inside a nested privileged container, because systemd will not start unless it is PID 1
+and the sandbox provider's own init holds that slot — expect it to take several minutes longer.
+
Other flags: `--repeat N` to tell flaky from broken, `--keep-sandbox` to leave the instance up for
debugging.
diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.md
index c6ee9e70..5c120a25 100644
--- a/beacon-sandbox/AGENTS.md
+++ b/beacon-sandbox/AGENTS.md
@@ -59,11 +59,13 @@ real money on the user's account, so:
| File read or write signals | `--scenario s03-file-write` or `s04-file-read` |
| Prompt, session, token, or cost capture | `--scenario s01-hello` |
| Approval or permission handling | `--scenario s07-denied-tool` |
+| `endpoint install`, config paths, service startup | `--scenario i01-install-supervised` |
+| The systemd backend, unit files, Linux system mode | `--scenario i02-install-systemd` |
| Something broad, or preparing a PR | the whole suite (no `--scenario`) |
```bash
go run ./cmd/beacon-sandbox run --scenario s02-bash-command # one scenario
-go run ./cmd/beacon-sandbox run # the whole suite: all 7, ~20 min, ~$0.50
+go run ./cmd/beacon-sandbox run # the whole suite, ~30 min, ~$0.70
```
`run` with no `--scenario` is the suite command. Other flags: `--repeat N` to tell flaky from
diff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.md
index 6c14cd27..70601008 100644
--- a/beacon-sandbox/README.md
+++ b/beacon-sandbox/README.md
@@ -27,7 +27,7 @@ cd cli/beacon && make build-linux-amd64 # the Beacon bein
cd ../../beacon-sandbox
go run ./cmd/beacon-sandbox doctor --fix # checks the setup
go run ./cmd/beacon-sandbox run --scenario s02-bash-command # one test, ~3 min
-go run ./cmd/beacon-sandbox run # all seven, ~20 min
+go run ./cmd/beacon-sandbox run # all of them, ~30 min
```
`doctor` checks everything the tool needs and prints the exact fix for anything missing, so it is
@@ -97,8 +97,12 @@ Other options are `drop-commands`, `drop-action:<action>`, and `plant-secret`. E
## Limitations
Modal only provides Linux x86 machines, so this **cannot test the macOS build** — launchd, the
-signed installer, and notarization are out of reach. It also tests one collection path (the
-temporary collector Beacon uses in CI and cloud agents) rather than a permanently installed Beacon,
-and it confirms the thing it planted was recorded rather than proving nothing else was missed.
+signed installer, and notarization are out of reach. It also confirms the thing it planted was
+recorded rather than proving nothing else was missed.
+
+The `i0*` tests install Beacon for real and cover installation and service management; the `s0*`
+tests use a temporary collector instead, the path Beacon uses in CI and cloud agents. `i02` runs
+inside a nested privileged container, since systemd only starts as PID 1 and Modal's own init holds
+that slot, so it takes several minutes longer than the rest.
The [full guide](https://docs.asymptotelabs.ai/contributing/beacon-sandbox) has the complete list.
diff --git a/beacon-sandbox/image/image.go b/beacon-sandbox/image/image.go
index cfa9f693..3218bbaa 100644
--- a/beacon-sandbox/image/image.go
+++ b/beacon-sandbox/image/image.go
@@ -37,6 +37,13 @@ type Spec struct {
ClaudeVersion string
// RepoRoot locates the Beacon build artifacts.
RepoRoot string
+ // WithDocker adds a Docker daemon, for scenarios that need a real systemd as PID 1.
+ //
+ // Modal cannot provide that directly -- its own init holds PID 1 in both lanes -- but systemd
+ // can be PID 1 inside a privileged container, so systemd scenarios run nested. This layer is
+ // opt-in because it is a large install that every other scenario would pay for and none of
+ // them need.
+ WithDocker bool
}
// LinuxArtifacts are the binaries the sandbox needs, as host paths.
@@ -93,6 +100,19 @@ func Build(spec Spec, log func(string, ...any)) (sandbox.ImageSpec, error) {
}
pkgs := "curl ca-certificates git tar ripgrep procps sudo python3 jq"
+ if spec.WithDocker {
+ // docker.io rather than the upstream repo: it is one apt line, and the daemon only has
+ // to be new enough to run a privileged container with a host cgroup namespace.
+ //
+ // systemd is installed here, in the image, even though nothing in the sandbox itself can
+ // use it -- the provider's init owns PID 1. It is here because the nested container is
+ // built by importing this very filesystem, and an image build is the only moment with
+ // unrestricted network. Installing it later, inside the nested build, is what the first
+ // attempt did: the pull succeeded because the daemon shares the sandbox's network, then
+ // apt failed because a build container's traffic is NATed through docker0 and the
+ // sandbox's egress allowlist does not cover it.
+ pkgs += " docker.io iproute2 systemd systemd-sysv dbus"
+ }
layers := []string{
"ENV DEBIAN_FRONTEND=noninteractive",
@@ -112,6 +132,13 @@ func Build(spec Spec, log func(string, ...any)) (sandbox.ImageSpec, error) {
AgentUser, WorkDir, WorkDir),
}
+ if spec.WithDocker {
+ // Group membership is set at image build time; adding it later would need a re-login to
+ // take effect, and the session runs as this user.
+ layers = append(layers,
+ fmt.Sprintf("RUN usermod -aG docker %s", AgentUser))
+ }
+
return sandbox.ImageSpec{
Base: spec.Base,
Layers: layers,
diff --git a/beacon-sandbox/runner/guest.go b/beacon-sandbox/runner/guest.go
new file mode 100644
index 00000000..d7b0f4d4
--- /dev/null
+++ b/beacon-sandbox/runner/guest.go
@@ -0,0 +1,335 @@
+package runner
+
+import (
+ "context"
+ "encoding/base64"
+ "fmt"
+ "path"
+ "strings"
+ "sync/atomic"
+ "time"
+
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox"
+)
+
+// guest is "the environment under test" -- where Beacon gets installed, where the agent session
+// runs, and where the runtime log lands.
+//
+// For almost every scenario that is the sandbox instance itself. Scenarios that need a real
+// systemd get a nested privileged container instead, because the sandbox provider's own init
+// holds PID 1 and systemd refuses to start as anything else.
+//
+// Both sit behind one interface so everything above it -- canary, sentinel, quiescence,
+// collection, verdict -- is byte-for-byte the same code. A systemd run is therefore judged by
+// exactly the same rules as every other run, rather than by a parallel path that could drift into
+// being more forgiving.
+type guest interface {
+ Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error)
+ // Get copies a file out of the guest to a local path.
+ Get(ctx context.Context, remotePath, localPath string) error
+ // Describe names the environment for run metadata, so a verdict says where it ran.
+ Describe() string
+}
+
+// directGuest runs commands in the sandbox instance itself.
+type directGuest struct {
+ p sandbox.Provider
+ inst sandbox.Instance
+}
+
+func (g directGuest) Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error) {
+ return g.p.Exec(ctx, g.inst, script, opts)
+}
+
+func (g directGuest) Get(ctx context.Context, remotePath, localPath string) error {
+ return g.p.Get(ctx, g.inst, remotePath, localPath)
+}
+
+func (g directGuest) Describe() string { return "sandbox" }
+
+// Names and paths for the nested lane. Fixed rather than random: a leftover container from an
+// interrupted run is reclaimed by name on the next attempt instead of accumulating.
+const (
+ nestedContainer = "beacon-systemd"
+ nestedImage = "beacon-systemd:local"
+ // /var/tmp, not /tmp. Once systemd is PID 1 it may enable tmp.mount, which mounts a tmpfs
+ // over /tmp -- a file placed there earlier is then shadowed and reads as missing. That
+ // wasted real time in the packaging smoke test before it was understood.
+ nestedStageDir = "/var/tmp/beacon-nested"
+ // nestedStartMarker is how a script that ran is told apart from one that never did.
+ nestedStartMarker = "__BEACON_NESTED_STARTED__"
+)
+
+// nestedGuest runs commands inside a privileged container where systemd is genuinely PID 1.
+//
+// Why nested at all: the sandbox provider's init occupies PID 1 in both of its lanes, and systemd
+// exits immediately when it is not PID 1 ("Couldn't find an alternative telinit implementation to
+// spawn"). A privileged container with a host cgroup namespace can hand PID 1 to systemd, so the
+// systemd surface -- unit installation, `enable --now`, `Restart=always`, `systemctl is-active` --
+// is exercised for real rather than against a stub.
+type nestedGuest struct {
+ p sandbox.Provider
+ inst sandbox.Instance
+ seq atomic.Int64
+ // dockerVersion and systemdState are recorded so a run says what it ran on.
+ dockerVersion string
+ systemdState string
+}
+
+func (g *nestedGuest) Describe() string {
+ return fmt.Sprintf("nested container %s (docker %s, systemd %s)",
+ nestedContainer, g.dockerVersion, g.systemdState)
+}
+
+// Exec runs a script inside the container as the requested user.
+//
+// The script has to cross two shells: the sandbox's, then the container's. Escaping it twice is
+// what broke repeatedly in the early probes -- anything holding a quote, a semicolon or a loop
+// arrived mangled. So the script travels as base64, which no shell can misread, and is decoded
+// into a file that the container executes directly. Nothing metacharacter-bearing ever appears on
+// a command line.
+func (g *nestedGuest) Exec(ctx context.Context, script string, opts sandbox.ExecOpts) (sandbox.Result, error) {
+ n := g.seq.Add(1)
+ stage := path.Join(nestedStageDir, fmt.Sprintf("step-%03d.sh", n))
+ // The script announces itself on stderr before doing anything. Without that marker, a script
+ // that never ran -- a failed copy, a stopped container -- looks exactly like a script that ran
+ // and printed nothing, and the first nested run duly reported an empty `beacon version` and
+ // carried on to fail somewhere less informative.
+ enc := base64.StdEncoding.EncodeToString([]byte("echo " + nestedStartMarker + " >&2\n" + script))
+
+ flags := []string{"exec"}
+ if opts.User != "" {
+ flags = append(flags, "-u", opts.User)
+ }
+ if opts.Dir != "" {
+ flags = append(flags, "-w", opts.Dir)
+ }
+ if opts.HomeDir != "" {
+ flags = append(flags, "-e", "HOME="+opts.HomeDir)
+ }
+ if len(opts.PathPrepend) > 0 {
+ flags = append(flags, "-e", "PATH="+strings.Join(opts.PathPrepend, ":")+
+ ":/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin")
+ }
+ // Named without a value on purpose: docker then takes the value from its own environment, so
+ // the credential is inherited rather than written onto a command line. Passing `-e K=v` would
+ // put the key in argv, which is precisely what the argv leak check exists to catch -- the
+ // harness must not be the thing that leaks it.
+ for _, k := range nestedPassThroughEnv {
+ flags = append(flags, "-e", k)
+ }
+ flags = append(flags, nestedContainer, "sh", stage)
+
+ wrapper := fmt.Sprintf("set -e\nmkdir -p %s\nprintf %%s '%s' | base64 -d > %s\n"+
+ "docker cp %s %s:%s >/dev/null\nexec docker %s\n",
+ shq(nestedStageDir), enc, shq(stage),
+ shq(stage), nestedContainer, shq(stage),
+ strings.Join(quoteAll(flags), " "))
+
+ // The docker CLI runs as root in the sandbox and needs its environment preserved, since that
+ // is where the pass-through credentials live.
+ timeout := opts.Timeout
+ if timeout == 0 {
+ timeout = 5 * time.Minute
+ }
+ r, err := g.p.Exec(ctx, g.inst, wrapper, sandbox.ExecOpts{
+ User: "root", Dir: "/", Timeout: timeout, PreserveEnv: true,
+ })
+ if err != nil {
+ return r, err
+ }
+ if !strings.Contains(r.Stderr, nestedStartMarker) {
+ // Reported as an error rather than a non-zero result, because the two mean different
+ // things to every caller: a non-zero result is a finding about Beacon, while this is the
+ // harness failing to reach the container at all.
+ return r, fmt.Errorf("the script never started inside %s (rc=%d): %s",
+ nestedContainer, r.ExitCode, tailOf(r.Stdout+r.Stderr))
+ }
+ r.Stderr = strings.Replace(r.Stderr, nestedStartMarker+"\n", "", 1)
+ return r, nil
+}
+
+// nestedPassThroughEnv are the variables the agent session needs inside the container. Only
+// credentials Claude Code reads; nothing Beacon needs, because Beacon's configuration comes from
+// the install it performs in there.
+var nestedPassThroughEnv = []string{
+ "ANTHROPIC_API_KEY",
+ "ANTHROPIC_AUTH_TOKEN",
+ "CLAUDE_CODE_OAUTH_TOKEN",
+}
+
+// Get copies a file out of the container, then out of the sandbox.
+func (g *nestedGuest) Get(ctx context.Context, remotePath, localPath string) error {
+ n := g.seq.Add(1)
+ stage := path.Join(nestedStageDir, fmt.Sprintf("fetch-%03d", n))
+ script := fmt.Sprintf("set -e\nmkdir -p %s\ndocker cp %s:%s %s\n",
+ shq(nestedStageDir), nestedContainer, shq(remotePath), shq(stage))
+ r, err := g.p.Exec(ctx, g.inst, script, sandbox.ExecOpts{
+ User: "root", Dir: "/", Timeout: 2 * time.Minute,
+ })
+ if err != nil {
+ return fmt.Errorf("docker cp %s out of the container: %w", remotePath, err)
+ }
+ if r.ExitCode != 0 {
+ return fmt.Errorf("docker cp %s out of the container failed (rc=%d): %s",
+ remotePath, r.ExitCode, oneLine(r.Stdout+r.Stderr))
+ }
+ return g.p.Get(ctx, g.inst, stage, localPath)
+}
+
+// startNested brings up dockerd and a systemd container, and refuses to return a guest that is
+// not actually running systemd as PID 1.
+//
+// Every step reports what it saw. A half-started environment that silently degraded to "no
+// service manager" would make a systemd scenario pass while testing the supervised backend, which
+// is the one outcome worth going to some trouble to prevent.
+func startNested(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
+ logf func(string, ...any)) (*nestedGuest, error) {
+
+ root := sandbox.ExecOpts{User: "root", Dir: "/", Timeout: 10 * time.Minute, PreserveEnv: true}
+ g := &nestedGuest{p: p, inst: inst}
+
+ logf("starting dockerd (this scenario needs systemd as PID 1, which only a privileged container can provide)")
+ r, err := p.Exec(ctx, inst, dockerdScript, root)
+ if err != nil || r.ExitCode != 0 {
+ return nil, fmt.Errorf("dockerd did not start (rc=%d): %v %s",
+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))
+ }
+ g.dockerVersion = oneLine(r.Stdout)
+ logf("dockerd ready: %s", g.dockerVersion)
+
+ logf("building the systemd container image")
+ r, err = p.Exec(ctx, inst, nestedBuildScript(), root)
+ if err != nil || r.ExitCode != 0 {
+ return nil, fmt.Errorf("building the systemd image failed (rc=%d): %v %s",
+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))
+ }
+
+ logf("starting the systemd container")
+ r, err = p.Exec(ctx, inst, nestedRunScript(), root)
+ if err != nil || r.ExitCode != 0 {
+ return nil, fmt.Errorf("the systemd container did not come up (rc=%d): %v %s",
+ r.ExitCode, err, tailOf(r.Stdout+r.Stderr))
+ }
+ // The script prints PID1=<comm> STATE=<is-system-running> and exits non-zero unless PID 1 is
+ // systemd, so reaching here means the requirement is met; the values are kept for metadata.
+ for _, f := range strings.Fields(oneLine(r.Stdout)) {
+ if v, ok := strings.CutPrefix(f, "STATE="); ok {
+ g.systemdState = v
+ }
+ }
+ logf("systemd is PID 1 in %s (%s)", nestedContainer, oneLine(r.Stdout))
+
+ // docker cp refuses a destination whose parent directory does not exist, and the image import
+ // excludes /var/tmp, so the container starts without one. Created once here rather than per
+ // exec: if it is missing, every single command fails, so this is the right place to find out.
+ r, err = p.Exec(ctx, inst, fmt.Sprintf("mkdir -p %s && docker exec %s mkdir -p %s",
+ shq(nestedStageDir), nestedContainer, shq(nestedStageDir)), root)
+ if err != nil || r.ExitCode != 0 {
+ return nil, fmt.Errorf("could not create the staging directory %s (rc=%d): %v %s",
+ nestedStageDir, r.ExitCode, err, tailOf(r.Stdout+r.Stderr))
+ }
+ return g, nil
+}
+
+// dockerdScript starts the daemon if it is not already up, and fails loudly with its log if not.
+const dockerdScript = `set -u
+if docker info >/dev/null 2>&1; then
+ docker version --format '{{.Server.Version}}'
+ exit 0
+fi
+(dockerd >/var/log/dockerd.log 2>&1 &)
+i=0
+while [ "$i" -lt 60 ]; do
+ docker info >/dev/null 2>&1 && break
+ i=$((i+1)); sleep 2
+done
+if ! docker info >/dev/null 2>&1; then
+ echo "dockerd never became ready; last 30 lines of its log:"
+ tail -30 /var/log/dockerd.log 2>/dev/null || echo "(no log)"
+ exit 1
+fi
+docker version --format '{{.Server.Version}}'
+`
+
+// nestedBuildScript turns the sandbox's own filesystem into the container image.
+//
+// The obvious approach -- a Dockerfile that pulls a base image and installs systemd -- does not
+// work here, and the way it fails is worth recording. The pull succeeds, because the daemon shares
+// the sandbox's network and the registry is reachable. Then `apt-get` fails inside the build
+// container, because a build container's traffic is NATed through docker0 and the sandbox's
+// outbound allowlist has no rule for it. The first attempt spent a full sandbox launch discovering
+// that from a truncated apt exit code.
+//
+// Importing the live root filesystem instead needs no network whatsoever, and it is strictly more
+// faithful: the container gets the exact Beacon binary under test, the same pinned Claude Code, the
+// same seeded git fixture, and the same `agent` account with the same uid. There is no second
+// environment that could drift from the one every other scenario runs in.
+//
+// --one-file-system is deliberately not used: it would silently drop anything the provider mounts
+// separately, and a missing binary would surface much later as a confusing exec failure. The
+// exclusions are named instead, so what is left out is visible.
+func nestedBuildScript() string {
+ return fmt.Sprintf(`set -eu
+if docker image inspect %[1]s >/dev/null 2>&1; then
+ echo "reusing %[1]s"
+ exit 0
+fi
+tar -cf - --numeric-owner --exclude=./proc --exclude=./sys --exclude=./dev --exclude=./run --exclude=./tmp --exclude=./var/lib/docker --exclude=./var/tmp --exclude=./mnt --exclude=./media --exclude=./lost+found -C / . 2>/dev/null | docker import -c 'CMD ["/sbin/init"]' - %[1]s
+docker image inspect %[1]s >/dev/null
+`, nestedImage)
+}
+
+// nestedRunScript starts the container and proves systemd owns PID 1 before anything is installed.
+//
+// The flags are not incidental. --privileged plus a host cgroup namespace and a writable
+// /sys/fs/cgroup are what let systemd manage units at all; the two tmpfs mounts give it the
+// runtime directories it expects to own. Drop any of them and systemd starts but cannot start
+// services, which presents as a mystifying unit failure much later.
+//
+// --stop-signal is how systemd is asked to shut down cleanly. docker import cannot carry a
+// STOPSIGNAL, so it is set here rather than in the image.
+func nestedRunScript() string {
+ return fmt.Sprintf(`set -u
+docker rm -f %[1]s >/dev/null 2>&1 || true
+docker run -d --name %[1]s --privileged --cgroupns=host --stop-signal=SIGRTMIN+3 -v /sys/fs/cgroup:/sys/fs/cgroup:rw --tmpfs /run --tmpfs /run/lock %[2]s >/dev/null || exit 1
+state=""
+i=0
+while [ "$i" -lt 45 ]; do
+ state="$(docker exec %[1]s systemctl is-system-running 2>/dev/null || true)"
+ case "$state" in running|degraded) break ;; esac
+ i=$((i+1)); sleep 2
+done
+pid1="$(docker exec %[1]s cat /proc/1/comm 2>/dev/null || true)"
+echo "PID1=$pid1 STATE=$state"
+if [ "$pid1" != systemd ]; then
+ echo "PID 1 is not systemd, so this scenario would silently test the supervised backend instead:"
+ docker logs %[1]s 2>&1 | tail -20
+ exit 1
+fi
+`, nestedContainer, nestedImage)
+}
+
+// tailOf keeps the end of a command's output rather than its beginning.
+//
+// Build and daemon output is long, and the sentence that says what went wrong is the last one. The
+// first version of this file reported these failures through oneLine, which truncates the head --
+// so the report was pages of layer-download chatter and no error, and the actual cause took another
+// full sandbox launch to find.
+func tailOf(s string) string {
+ lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
+ if len(lines) > 12 {
+ lines = lines[len(lines)-12:]
+ }
+ return oneLine(strings.Join(lines, " | "))
+}
+
+// quoteAll shell-quotes each argument. Used for docker's own argv, which is built from values the
+// harness controls but still passes through the sandbox's shell.
+func quoteAll(args []string) []string {
+ out := make([]string, len(args))
+ for i, a := range args {
+ out[i] = shq(a)
+ }
+ return out
+}
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.go
index 0e9c7b17..b073e9c2 100644
--- a/beacon-sandbox/runner/runner.go
+++ b/beacon-sandbox/runner/runner.go
@@ -120,6 +120,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
imgSpec, err := image.Build(image.Spec{
RepoRoot: opts.RepoRoot,
ClaudeVersion: opts.ClaudeVersion,
+ WithDocker: sc.NeedsRealSystemd(),
}, logf)
if err != nil {
return art, err
@@ -131,7 +132,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
art.Meta["image"] = img.Ref()
logf("image %s", img.Ref())
- inst, err := p.Launch(ctx, img, sandbox.LaunchSpec{
+ launch := sandbox.LaunchSpec{
CPU: 2,
MemLimitMiB: 4096,
Timeout: sc.Timeout() + 5*time.Minute, // headroom for setup and collection
@@ -144,7 +145,21 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
// here, which is how the local-only posture gets asserted rather than assumed.
"api.anthropic.com", "statsig.anthropic.com", "*.anthropic.com",
},
- })
+ }
+ if sc.NeedsRealSystemd() {
+ // The VM lane has a real kernel and cgroup2, which dockerd needs; the default lane does
+ // not. Neither lane makes our entrypoint PID 1, hence the nested container -- see
+ // startNested.
+ launch.Lane = sandbox.LaneVM
+ launch.CPU, launch.MemLimitMiB = 4, 8192 // dockerd + systemd + collector + the agent
+ launch.Timeout = sc.Timeout() + 20*time.Minute
+ // The allowlist is deliberately not widened. Building the nested image from the sandbox's
+ // own filesystem needs no network, so these scenarios assert exactly the same egress
+ // posture as every other one -- which is worth more than the convenience of a registry
+ // pull would have been.
+ art.Meta["lane"] = string(launch.Lane)
+ }
+ inst, err := p.Launch(ctx, img, launch)
if err != nil {
return art, fmt.Errorf("launch: %w", err)
}
@@ -162,20 +177,44 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
}
// Finish the artifact layer: the snapshot captured the files, but chmod and symlinks
- // have to happen in a live instance.
+ // have to happen in a live instance. Always the sandbox itself, never the guest -- this
+ // prepares the very tree the guest will mount.
if _, err := p.Exec(ctx, inst, image.PostPushLayers(), sandbox.ExecOpts{User: "root", Timeout: time.Minute}); err != nil {
return art, fmt.Errorf("prepare binaries: %w", err)
}
- if v, err := p.Exec(ctx, inst, "beacon version; claude --version; id -un; uname -m", agent); err == nil {
- art.Meta["versions"] = oneLine(v.Stdout)
- logf("versions: %s", art.Meta["versions"])
+ // Everything the scenario itself does happens in the guest. For most scenarios that is the
+ // sandbox; a scenario needing a real systemd gets a nested container instead. See guest.go.
+ var g guest = directGuest{p: p, inst: inst}
+ if sc.NeedsRealSystemd() {
+ ng, err := startNested(ctx, p, inst, logf)
+ if err != nil {
+ return art, err
+ }
+ g = ng
+ art.Meta["systemd_state"] = ng.systemdState
+ art.Meta["docker_version"] = ng.dockerVersion
+ }
+ art.Meta["guest"] = g.Describe()
+
+ // Not just recorded -- checked. An empty version line means beacon or claude is not reachable
+ // in the guest, and every later failure is then a symptom of that rather than of Beacon. The
+ // first nested run printed "versions:" with nothing after it and kept going.
+ v, err := g.Exec(ctx, "beacon version; claude --version; id -un; uname -m", agent)
+ if err != nil {
+ return art, fmt.Errorf("could not run commands in the %s guest: %w", g.Describe(), err)
+ }
+ art.Meta["versions"] = oneLine(v.Stdout)
+ logf("versions: %s", art.Meta["versions"])
+ if !strings.Contains(v.Stdout, "beacon version") {
+ return art, fmt.Errorf("the guest did not report a Beacon version, so the binary under "+
+ "test is not reachable there (rc=%d): %s", v.ExitCode, oneLine(v.Stdout+v.Stderr))
}
sentinel := scenario.Expand(sc.Sentinel, canary, "", image.WorkDir)
if sc.Setup != "" {
setup := scenario.Expand(sc.Setup, canary, sentinel, image.WorkDir)
- if r, err := p.Exec(ctx, inst, setup, agent); err != nil || r.ExitCode != 0 {
+ if r, err := g.Exec(ctx, setup, agent); err != nil || r.ExitCode != 0 {
return art, fmt.Errorf("scenario setup failed (rc=%d): %v %s", r.ExitCode, err, oneLine(r.Stderr))
}
}
@@ -199,14 +238,14 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
// collector actually receives telemetry. Nothing else in the suite can tell you that.
var res sandbox.Result
if sc.Install != nil {
- installedLog, err := doInstall(ctx, p, inst, sc, agent, &art, logf)
+ installedLog, err := doInstall(ctx, g, sc, agent, &art, logf)
if err != nil {
return art, err
}
logPath = installedLog
logf("running session against the installed endpoint (budget $%.2f, timeout %s)",
sc.Budget(), sc.Timeout())
- res, err = p.Exec(ctx, inst, plainSessionScript(sc, prompt), sandbox.ExecOpts{
+ res, err = g.Exec(ctx, plainSessionScript(sc, prompt), sandbox.ExecOpts{
User: image.AgentUser, Dir: workDirFor(sc), HomeDir: image.AgentHome,
PreserveEnv: true, PathPrepend: image.PathPrepend(), Timeout: sc.Timeout() + time.Minute,
})
@@ -217,7 +256,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
} else {
logf("running session (budget $%.2f, timeout %s)", sc.Budget(), sc.Timeout())
var err error
- res, err = p.Exec(ctx, inst, sessionScript(sc, prompt, logPath), sandbox.ExecOpts{
+ res, err = g.Exec(ctx, sessionScript(sc, prompt, logPath), sandbox.ExecOpts{
User: image.AgentUser, Dir: workDirFor(sc), HomeDir: image.AgentHome,
PreserveEnv: true, PathPrepend: image.PathPrepend(), Timeout: sc.Timeout() + time.Minute,
})
@@ -242,7 +281,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
// disabling exactly the signal that distinguishes a dead agent from a capture gap.
outPath := shq(sessionFile(sc, "claude-out.json"))
errPath := shq(sessionFile(sc, "claude-err.txt"))
- if r, err := p.Exec(ctx, inst, "cat "+outPath+" 2>/dev/null || true", agent); err == nil {
+ if r, err := g.Exec(ctx, "cat "+outPath+" 2>/dev/null || true", agent); err == nil {
var result map[string]any
if json.Unmarshal([]byte(strings.TrimSpace(r.Stdout)), &result) == nil {
art.SessionOK, art.SessionKnown = sessionOutcome(result)
@@ -255,7 +294,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
logf("claude: %s", art.Meta["claude_result"])
_ = os.WriteFile(filepath.Join(runDir, "claude-result.json"), []byte(r.Stdout), 0o600)
} else {
- if e, err2 := p.Exec(ctx, inst, "tail -c 800 "+errPath+" 2>/dev/null || true", agent); err2 == nil {
+ if e, err2 := g.Exec(ctx, "tail -c 800 "+errPath+" 2>/dev/null || true", agent); err2 == nil {
art.Meta["claude_stderr"] = oneLine(e.Stdout)
}
}
@@ -273,7 +312,7 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
// sentinel file, which would have misreported the second as a broken probe.
probe := fmt.Sprintf("if [ -f %[1]s ]; then echo __FOUND__; cat %[1]s; else echo __MISSING__; fi",
shq(sentinel))
- r, execErr := p.Exec(ctx, inst, probe, agent)
+ r, execErr := g.Exec(ctx, probe, agent)
art.SentinelProbed = execErr == nil &&
(strings.Contains(r.Stdout, "__FOUND__") || strings.Contains(r.Stdout, "__MISSING__"))
if !art.SentinelProbed {
@@ -343,12 +382,12 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
// Wait for quiescence rather than sleeping a fixed amount: the collector batches at 5s
// and Claude Code's metric interval defaults to 60s, so a fixed sleep is either wasteful
// or wrong. Record how long it took -- that latency is itself a finding.
- stable, waited := waitForQuiescence(ctx, p, inst, agent, logPath, logf)
+ stable, waited := waitForQuiescence(ctx, g, agent, logPath, logf)
art.Meta["quiescence_seconds"] = fmt.Sprintf("%.0f", waited.Seconds())
art.Meta["quiescent"] = fmt.Sprintf("%v", stable)
local := filepath.Join(runDir, "runtime.jsonl")
- if err := p.Get(ctx, inst, logPath, local); err != nil {
+ if err := g.Get(ctx, logPath, local); err != nil {
return art, fmt.Errorf("collect runtime log: %w", err)
}
art.RuntimeLog = local
@@ -417,7 +456,7 @@ func sessionScript(sc scenario.Scenario, prompt, logPath string) string {
}
// waitForQuiescence polls the log until its size stops changing, bounded.
-func waitForQuiescence(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
+func waitForQuiescence(ctx context.Context, g guest,
opts sandbox.ExecOpts, logPath string, logf func(string, ...any)) (bool, time.Duration) {
const (
@@ -430,7 +469,7 @@ func waitForQuiescence(ctx context.Context, p sandbox.Provider, inst sandbox.Ins
stableSince := time.Time{}
for time.Since(start) < maxWait {
- r, err := p.Exec(ctx, inst, "wc -c < "+shq(logPath)+" 2>/dev/null || echo 0", opts)
+ r, err := g.Exec(ctx, "wc -c < "+shq(logPath)+" 2>/dev/null || echo 0", opts)
if err == nil {
cur := strings.TrimSpace(r.Stdout)
if cur == last && cur != "0" {
@@ -580,7 +619,7 @@ func sampleCountOf(verdict string) string {
// The log path comes from `endpoint status --json` rather than being assumed, because the whole
// point of an install scenario is to check what Beacon actually did — hardcoding the expected
// path would make the scenario agree with itself instead of with Beacon.
-func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
+func doInstall(ctx context.Context, g guest,
sc scenario.Scenario, agent sandbox.ExecOpts, art *Artifacts, logf func(string, ...any)) (string, error) {
in := sc.Install
@@ -605,12 +644,12 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
// Dry run first. It is free, and it is where a plan that advertises the wrong service
// manager for the host would show up — so a regression there needs no second scenario.
- if r, err := p.Exec(ctx, inst, "beacon endpoint install "+scope+svc+" --harness claude --dry-run 2>&1", opts); err == nil {
+ if r, err := g.Exec(ctx, "beacon endpoint install "+scope+svc+" --harness claude --dry-run 2>&1", opts); err == nil {
art.Meta["install_dry_run"] = oneLine(r.Stdout)
logf("dry-run plan: %s", art.Meta["install_dry_run"])
}
- r, err := p.Exec(ctx, inst, "beacon endpoint install "+scope+svc+" --harness claude 2>&1", opts)
+ r, err := g.Exec(ctx, "beacon endpoint install "+scope+svc+" --harness claude 2>&1", opts)
art.Meta["install_output"] = oneLine(r.Stdout + r.Stderr)
if err != nil || r.ExitCode != 0 {
logf("install FAILED rc=%d: %s", r.ExitCode, art.Meta["install_output"])
@@ -619,13 +658,40 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
}
logf("install ok: %s", art.Meta["install_output"])
- logPath, err := installedLogPath(ctx, p, inst, scope, opts, in, art, logf)
+ // A system install configures the *installing* user's agent runtime, which for a package
+ // install is root -- not the person whose sessions matter. The package postinstall therefore
+ // runs a second step to resolve that person and point their Claude Code at the collector, and
+ // a system-mode scenario has to do the same or its capture assertions would be testing
+ // nothing but root's unused settings file.
+ //
+ // SUDO_USER is how the real path identifies them (`sudo apt install ./beacon.deb`), so it is
+ // what gets set here rather than inventing a flag that no user would pass.
+ if system {
+ cmd := fmt.Sprintf("SUDO_USER=%s beacon endpoint user-config repair-installed "+
+ "--system --harness claude --json 2>&1", image.AgentUser)
+ u, err := g.Exec(ctx, cmd, opts)
+ art.Meta["install_user_config"] = oneLine(u.Stdout + u.Stderr)
+ if err != nil || u.ExitCode != 0 {
+ return "", fmt.Errorf("configuring %s's agent runtime failed (rc=%d): %v %s",
+ image.AgentUser, u.ExitCode, err, art.Meta["install_user_config"])
+ }
+ logf("user config: %s", art.Meta["install_user_config"])
+ // A skip is not a pass. The command reports one rather than failing, because on a real
+ // unattended host there may be nobody to configure -- but in this scenario there is, and
+ // treating a skip as success would let a broken resolver produce a green run.
+ if strings.Contains(art.Meta["install_user_config"], "skipped_reason") {
+ return "", fmt.Errorf("user-config skipped instead of configuring %s: %s",
+ image.AgentUser, art.Meta["install_user_config"])
+ }
+ }
+
+ logPath, err := installedLogPath(ctx, g, scope, opts, in, art, logf)
if err != nil {
return "", err
}
// Doctor output is recorded whether or not it passes: on a failure it is the first thing
// worth reading, and it costs one exec.
- if d, err := p.Exec(ctx, inst, "beacon endpoint doctor "+scope+" --json 2>&1", opts); err == nil {
+ if d, err := g.Exec(ctx, "beacon endpoint doctor "+scope+" --json 2>&1", opts); err == nil {
art.Meta["install_doctor"] = oneLine(d.Stdout)
}
return logPath, nil
@@ -636,11 +702,11 @@ func doInstall(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
// A status that cannot be parsed is not fatal — the run can still proceed against the
// documented default path — but ExpectStatusRunning is enforced, since a scenario asking for a
// running service and silently accepting an unparseable status would verify nothing.
-func installedLogPath(ctx context.Context, p sandbox.Provider, inst sandbox.Instance,
+func installedLogPath(ctx context.Context, g guest,
scope string, opts sandbox.ExecOpts, in *scenario.Install, art *Artifacts,
logf func(string, ...any)) (string, error) {
- s, err := p.Exec(ctx, inst, "beacon endpoint status "+scope+" --json 2>&1", opts)
+ s, err := g.Exec(ctx, "beacon endpoint status "+scope+" --json 2>&1", opts)
if err == nil {
art.Meta["install_status"] = oneLine(s.Stdout)
var status struct {
@@ -665,14 +731,19 @@ func installedLogPath(ctx context.Context, p sandbox.Provider, inst sandbox.Inst
return "", fmt.Errorf("service did not report running after install: %s",
status.Service.Message)
}
+ if in.ExpectServiceKind != "" && status.Service.Kind != in.ExpectServiceKind {
+ return "", fmt.Errorf("service backend is %q, want %q: the scenario would "+
+ "otherwise verify a different service manager than it claims to",
+ status.Service.Kind, in.ExpectServiceKind)
+ }
if status.LogPath != "" {
return status.LogPath, nil
}
}
}
- if in.ExpectStatusRunning {
- return "", fmt.Errorf("could not read service status, so a running service cannot be " +
- "confirmed; refusing to report a pass the run did not earn")
+ if in.ExpectStatusRunning || in.ExpectServiceKind != "" {
+ return "", fmt.Errorf("could not read service status, so neither a running service nor " +
+ "the selected backend can be confirmed; refusing to report a pass the run did not earn")
}
logf("status unparseable; falling back to the documented default log path")
if in.Mode == "system" {
diff --git a/beacon-sandbox/scenario/scenario.go b/beacon-sandbox/scenario/scenario.go
index 4fad7540..812182ff 100644
--- a/beacon-sandbox/scenario/scenario.go
+++ b/beacon-sandbox/scenario/scenario.go
@@ -52,6 +52,19 @@ type Install struct {
Service string `yaml:"service,omitempty"`
// ExpectStatusRunning requires the collector service to report running after install.
ExpectStatusRunning bool `yaml:"expect_status_running,omitempty"`
+ // ExpectServiceKind requires the endpoint to report this service backend after install.
+ //
+ // Which backend got selected is the substance of a service-manager scenario, and it is not
+ // implied by "running": the supervised fallback also reports running, so a systemd scenario
+ // on a host without systemd would otherwise pass while testing the wrong thing.
+ ExpectServiceKind string `yaml:"expect_service_kind,omitempty"`
+ // NeedsRealSystemd runs the install inside a nested privileged container where systemd is
+ // genuinely PID 1.
+ //
+ // The sandbox provider cannot offer that directly: its own init holds PID 1, so systemctl
+ // refuses. Rather than make a scenario describe VM lanes and Docker, it states the
+ // requirement and the runner arranges it.
+ NeedsRealSystemd bool `yaml:"needs_real_systemd,omitempty"`
}
// Scenario is one declarative case.
@@ -127,6 +140,22 @@ func (s Scenario) Validate() error {
default:
return fmt.Errorf("%s: install.mode %q must be user or system", s.ID, s.Install.Mode)
}
+ // A systemd requirement with the supervised backend, or without system mode, would
+ // silently exercise something other than what the scenario claims to test.
+ if s.Install.NeedsRealSystemd && s.Install.Service == "none" {
+ return fmt.Errorf("%s: needs_real_systemd with service=none is contradictory; "+
+ "the supervised backend does not use systemd", s.ID)
+ }
+ if s.Install.NeedsRealSystemd && s.Install.Mode != "system" {
+ return fmt.Errorf("%s: needs_real_systemd requires mode=system; a user unit in a "+
+ "container has no logind session to attach to", s.ID)
+ }
+ switch s.Install.ExpectServiceKind {
+ case "", "systemd", "launchd", "none":
+ default:
+ return fmt.Errorf("%s: install.expect_service_kind %q must be systemd, launchd or none",
+ s.ID, s.Install.ExpectServiceKind)
+ }
switch s.Install.Service {
case "", "auto", "systemd", "launchd", "none":
default:
@@ -273,3 +302,8 @@ func isFilenameByte(b byte) bool {
}
return false
}
+
+// NeedsRealSystemd reports whether this scenario must run somewhere systemd is genuinely PID 1.
+func (s Scenario) NeedsRealSystemd() bool {
+ return s.Install != nil && s.Install.NeedsRealSystemd
+}
diff --git a/beacon-sandbox/scenario/scenario_test.go b/beacon-sandbox/scenario/scenario_test.go
index 392cd3c7..641423b7 100644
--- a/beacon-sandbox/scenario/scenario_test.go
+++ b/beacon-sandbox/scenario/scenario_test.go
@@ -347,3 +347,27 @@ func TestInstallModeAndServiceAreValidated(t *testing.T) {
}
}
}
+
+// A systemd requirement combined with the supervised backend, or without system mode, would
+// silently exercise something other than what the scenario claims — and the run would pass,
+// reporting coverage it does not have.
+func TestNeedsRealSystemdRejectsContradictions(t *testing.T) {
+ base := func(in *Install) Scenario {
+ return Scenario{ID: "i", Prompt: "do a thing", Install: in,
+ Expect: []Expect{{Action: "prompt.submitted", Why: "baseline"}}}
+ }
+ contradictory := []*Install{
+ {NeedsRealSystemd: true, Mode: "system", Service: "none"},
+ {NeedsRealSystemd: true, Mode: "user", Service: "systemd"},
+ {NeedsRealSystemd: true}, // mode defaults to user
+ }
+ for _, in := range contradictory {
+ if err := base(in).Validate(); err == nil {
+ t.Errorf("%+v is contradictory and must be rejected", in)
+ }
+ }
+ // The coherent combination must be accepted.
+ if err := base(&Install{NeedsRealSystemd: true, Mode: "system", Service: "systemd"}).Validate(); err != nil {
+ t.Errorf("system+systemd is the intended combination: %v", err)
+ }
+}
diff --git a/beacon-sandbox/scenarios/i01-install-supervised.yaml b/beacon-sandbox/scenarios/i01-install-supervised.yaml
index 016dadd0..1b40cbb4 100644
--- a/beacon-sandbox/scenarios/i01-install-supervised.yaml
+++ b/beacon-sandbox/scenarios/i01-install-supervised.yaml
@@ -18,6 +18,9 @@ install:
mode: user
service: none
expect_status_running: true
+ # Asserted rather than assumed: if systemd detection ever started returning true in a container,
+ # this scenario would silently stop covering the fallback backend it exists to cover.
+ expect_service_kind: none
prompt: >-
Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}}
diff --git a/beacon-sandbox/scenarios/i02-install-systemd.yaml b/beacon-sandbox/scenarios/i02-install-systemd.yaml
new file mode 100644
index 00000000..8a835c50
--- /dev/null
+++ b/beacon-sandbox/scenarios/i02-install-systemd.yaml
@@ -0,0 +1,54 @@
+id: i02-install-systemd
+# The scenario a Linux user actually lives: a system-mode install managed by systemd, capturing a
+# real agent session.
+#
+# i01 proves `beacon endpoint install` works and that an installed collector receives telemetry,
+# but it runs the supervised backend -- the fallback for containers, where nothing restarts the
+# collector and no unit file is involved. So it says nothing about the systemd path, which is what
+# a package install on a Linux workstation uses.
+#
+# `needs_real_systemd` is what makes this possible. The sandbox provider's own init holds PID 1, and
+# systemd exits immediately when it is not PID 1, so the runner starts a privileged container where
+# systemd genuinely is PID 1 and performs the whole scenario in there. Everything else -- canary,
+# sentinel, quiescence, verdict -- is the same code as every other scenario.
+#
+# What this proves that nothing else can: a systemd unit that systemd accepts and reports active,
+# a collector started by that unit, the installing user's Claude Code pointed at it, and a live
+# session's telemetry landing in the system runtime log.
+install:
+ mode: system
+ service: systemd
+ needs_real_systemd: true
+ expect_status_running: true
+ # Not redundant with the above. The supervised backend also reports running, so without this a
+ # host where systemd detection failed would pass this scenario while testing the fallback.
+ expect_service_kind: systemd
+
+prompt: >-
+ Run exactly this shell command and report its output: echo {{canary}} | tee {{sentinel}}
+sentinel: /home/agent/work/i02.sentinel
+budget_usd: 1.0
+timeout_seconds: 420
+
+expect:
+ - action: prompt.submitted
+ contains: ["{{canary}}"]
+ fields: ["prompt.text", "session.id", "harness.name"]
+ why: >-
+ The end of the chain: systemd started the collector, the install pointed the user's agent at
+ it, and the session's prompt arrived. A failure here means one of those three links is
+ broken, and the install output plus `systemctl status` in the run metadata says which.
+
+ - action: token.usage
+ fields: ["gen_ai.usage.input_tokens", "model"]
+ why: >-
+ Metrics travel a separate pipeline from logs, so a working prompt event does not imply a
+ working metrics path through a unit-started collector.
+
+ - action: command.executed
+ contains: ["{{canary}}"]
+ fields: ["command.command"]
+ why: >-
+ The detection surface that matters most, asserted through the systemd path. Non-optional on
+ purpose: a systemd-managed collector is never allowed to look healthier than the supervised
+ one for the same telemetry.
diff --git a/docs/contributing/beacon-sandbox.mdx b/docs/contributing/beacon-sandbox.mdx
index 982ab19e..0c295115 100644
--- a/docs/contributing/beacon-sandbox.mdx
+++ b/docs/contributing/beacon-sandbox.mdx
@@ -177,7 +177,7 @@ Tests encode what Beacon *should* do. When Beacon's behaviour legitimately impro
```bash
go run ./cmd/beacon-sandbox run --scenario s01-hello # one test
-go run ./cmd/beacon-sandbox run # all of them, ~20 min
+go run ./cmd/beacon-sandbox run # all of them, ~30 min
```
| Test | Covers |
@@ -189,9 +189,36 @@ go run ./cmd/beacon-sandbox run # all of them, ~20 min
| `s05-repo-task` | A multi-step task in a git repository |
| `s06-subagent` | Sub-agent activity |
| `s07-denied-tool` | Approvals and denials |
+| `i01-install-supervised` | `beacon endpoint install`, then a session against the installed collector |
+| `i02-install-systemd` | A system-mode install managed by systemd, then a session against it |
Use one test while you are iterating. Run all of them before opening a pull request.
+The `s0*` tests collect through a temporary collector — the path Beacon uses in CI and cloud agents.
+The `i0*` tests install Beacon properly first, so they cover things the others cannot: whether config
+lands where Beacon says it did, whether the service actually starts, and whether a permanently
+installed collector receives telemetry.
+
+`i02` is the slowest test by some margin, and the reason is worth knowing. A system-mode install on
+Linux is managed by systemd, and systemd only runs as process 1 — but Modal's own startup process
+holds that slot, so systemd refuses to start. The way around it is a privileged container inside the
+sandbox, where systemd *is* process 1, and running the whole test in there. The tool sets this up for
+you when a test asks for it:
+
+```yaml title="Asking for a real systemd"
+install:
+ mode: system
+ service: systemd
+ needs_real_systemd: true
+ expect_status_running: true
+ expect_service_kind: systemd
+```
+
+`expect_service_kind` is not redundant with `expect_status_running`. Beacon falls back to a plain
+background collector when it finds no service manager, and that fallback also reports itself as
+running — so without naming the backend you expect, a test meant to cover systemd could quietly pass
+while covering the fallback instead.
+
## Commands
| Command | What it does |
@@ -252,7 +279,7 @@ go test ./...
Modal only provides Linux machines, so **this cannot test the macOS build**. Anything macOS-specific — launchd, the signed installer, notarization — is out of reach.
</Warning>
-- **It tests one collection path.** Specifically the temporary-collector path Beacon uses in CI and cloud agents, not a permanently installed Beacon.
+- **The systemd tests run nested.** `i02` runs inside a container within the sandbox, so it is slower and needs a few more minutes of patience than the others.
- **It checks presence, not completeness.** It confirms the thing it planted was recorded. It cannot tell you whether Beacon missed something it never knew to look for.
- **Linux x86 only.** Modal does not offer a choice of processor architecture.
- **It costs money and takes minutes.** Each test is a real API session.