Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

fix(sandbox): uninstall the endpoint a scenario installed, on providers that are the host

shukieshah69a1d1ff18ed
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.go
index 0b9a065c..1eb524fc 100644
--- a/beacon-sandbox/runner/runner.go
+++ b/beacon-sandbox/runner/runner.go
@@ -276,6 +276,21 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Opt
var res sandbox.Result
if sc.Install != nil {
installedLog, err := doInstall(ctx, g, sc, privileged, agent, lay, sh, &art, logf)
+ // Uninstall afterwards when the sandbox is this machine, whether or not the install
+ // succeeded -- a partial install still holds the OTLP ports.
+ //
+ // A disposable per-scenario instance makes this unnecessary: Terminate throws the machine
+ // away and the next scenario starts clean. LocalExec has no machine to throw away, so an
+ // installed endpoint outlives the scenario that installed it, and the next install fails on
+ // "OTLP gRPC port 4317 and HTTP port 4318 are already in use". That is what happened: a
+ // system-mode scenario left its collector running and the user-mode scenario after it could
+ // not bind, so the suite's result depended on the order its scenarios happened to run in.
+ //
+ // Deferred rather than run inline because artifact collection happens further down this
+ // function and needs the log to still be there; --keep-logs is what makes that safe.
+ if local && !opts.KeepInstance {
+ defer teardownInstall(ctx, g, sc, privileged, logf)
+ }
if err != nil {
return art, err
}
@@ -825,3 +840,34 @@ func installedLogPath(ctx context.Context, g guest,
logf("status unparseable; falling back to the documented default log path")
return lay.DefaultLogPath(in.Mode == "system"), nil
}
+
+// teardownInstall removes an endpoint the scenario installed, so the next scenario on this machine
+// starts from the same state the first one did.
+//
+// Only reached when the provider mutates the host. Everywhere else the instance is discarded, which
+// is a stronger guarantee than any uninstall: it also takes back whatever the install touched that
+// uninstall does not know about.
+//
+// Failures are logged rather than returned. This runs after the scenario has been judged, so turning
+// a cleanup problem into a scenario failure would report the wrong thing -- but staying silent would
+// leave the *next* scenario to fail for a reason that has nothing to do with it, which is exactly the
+// confusion this function exists to prevent. Naming it here is what makes that traceable.
+func teardownInstall(ctx context.Context, g guest, sc scenario.Scenario,
+ privileged sandbox.ExecOpts, logf func(string, ...any)) {
+
+ scope := "--user"
+ if sc.Install != nil && sc.Install.Mode == "system" {
+ scope = "--system"
+ }
+ // --keep-logs because the runtime log is the artifact this whole run exists to collect, and the
+ // collection step runs after this one.
+ res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" --keep-logs 2>&1", privileged)
+ switch {
+ case err != nil:
+ logf("teardown: uninstall %s did not run: %v", scope, err)
+ case res.ExitCode != 0:
+ logf("teardown: uninstall %s exited %d: %s", scope, res.ExitCode, oneLine(res.Stdout))
+ default:
+ logf("teardown: uninstalled the %s endpoint", strings.TrimPrefix(scope, "--"))
+ }
+}
diff --git a/beacon-sandbox/scenarios/w01-install-service.yaml b/beacon-sandbox/scenarios/w01-install-service.yaml
index 7443fb15..9ddea89a 100644
--- a/beacon-sandbox/scenarios/w01-install-service.yaml
+++ b/beacon-sandbox/scenarios/w01-install-service.yaml
@@ -9,11 +9,11 @@ platform: windows
# so without it a host where SCM registration failed would pass while testing the wrong backend
# entirely -- the same trap the Linux systemd scenario documents.
#
-# Capture is asserted only optionally, and that is deliberate rather than lax. Hook installation on
-# Windows still emits a POSIX `VAR=value cmd` string that neither Windows shell accepts, so nothing
-# reaches the collector through hooks yet. Marking these required would make the scenario fail for
-# a known reason and stop it reporting the service facts it exists to establish. They become
-# required in the change that fixes the hook command format.
+# Capture is now required. It was optional while hook installation was believed to be broken on
+# Windows -- a belief a probe later disproved: Claude Code runs hook commands through Git Bash there,
+# which accepts the form Beacon was emitting. Once the command became one argv, a dispatched run of
+# this scenario captured 38 events, so there is no longer a known reason for these to fail and every
+# reason to gate on them.
prompt: >-
Run exactly this PowerShell command and then report its output verbatim:
Write-Output '{{canary}}' | Tee-Object -FilePath '{{sentinel}}'
@@ -30,16 +30,15 @@ install:
expect:
- action: prompt.submitted
contains: ["{{canary}}"]
- optional: true
why: >-
- Capture through an installed Windows endpoint. Optional until hook installation emits a
- command Windows can execute; the service assertions above are what this scenario gates on
- today, and they are checked before the session runs.
+ Capture through an installed Windows endpoint, which together with the service assertions
+ above is the whole claim: the collector is registered with the SCM, it is running, and events
+ written by hooks actually reach it.
- action: command.executed
fields: ["command.command"]
- optional: true
why: >-
- e.command.command is the detection surface for the whole rules/risky-command/ category. The
- exporter now reads it from Claude Code's PowerShell tool, so this is the assertion that will
- prove the fix end to end once hooks deliver events at all.
+ e.command.command is the detection surface for the whole rules/risky-command/ category, and
+ threat rules match scalar leaves, so an empty value disables every one of those rules with
+ nothing reporting it. This is the assertion that proves the exporter reads Claude Code's
+ PowerShell tool and not only bash.
diff --git a/beacon-sandbox/scenarios/w02-install-supervised.yaml b/beacon-sandbox/scenarios/w02-install-supervised.yaml
index affae5af..328b3f29 100644
--- a/beacon-sandbox/scenarios/w02-install-supervised.yaml
+++ b/beacon-sandbox/scenarios/w02-install-supervised.yaml
@@ -27,6 +27,10 @@ expect:
contains: ["{{canary}}"]
optional: true
why: >-
- Same reason as w01: hook installation on Windows does not yet emit a command Windows can
- execute, so capture is recorded rather than gated. The backend selection above is what this
- scenario asserts today.
+ Recorded rather than gated, and for a narrower reason than it used to be. w01 and w03 have both
+ captured through installed hooks in system mode, so the hook command itself works on Windows.
+ What no run has yet observed is capture in *user* mode: the only attempt failed before the
+ session on an OTLP port conflict with a system collector the previous scenario left running.
+ That is fixed in the runner rather than here, but this stays optional until a run has actually
+ demonstrated it -- promoting it on the strength of the system-mode result would be assuming the
+ thing this scenario exists to check.