The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(sandbox): uninstall the endpoint a scenario installed, on providers that are the host
shukieshah69a1d1ff18ed
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.goindex 0b9a065c..1eb524fc 100644--- a/beacon-sandbox/runner/runner.go+++ b/beacon-sandbox/runner/runner.go@@ -276,6 +276,21 @@ func Run(ctx context.Context, p sandbox.Provider, sc scenario.Scenario, opts Optvar res sandbox.Resultif sc.Install != nil {installedLog, err := doInstall(ctx, g, sc, privileged, agent, lay, sh, &art, logf)+ // Uninstall afterwards when the sandbox is this machine, whether or not the install+ // succeeded -- a partial install still holds the OTLP ports.+ //+ // A disposable per-scenario instance makes this unnecessary: Terminate throws the machine+ // away and the next scenario starts clean. LocalExec has no machine to throw away, so an+ // installed endpoint outlives the scenario that installed it, and the next install fails on+ // "OTLP gRPC port 4317 and HTTP port 4318 are already in use". That is what happened: a+ // system-mode scenario left its collector running and the user-mode scenario after it could+ // not bind, so the suite's result depended on the order its scenarios happened to run in.+ //+ // Deferred rather than run inline because artifact collection happens further down this+ // function and needs the log to still be there; --keep-logs is what makes that safe.+ if local && !opts.KeepInstance {+ defer teardownInstall(ctx, g, sc, privileged, logf)+ }if err != nil {return art, err}@@ -825,3 +840,34 @@ func installedLogPath(ctx context.Context, g guest,logf("status unparseable; falling back to the documented default log path")return lay.DefaultLogPath(in.Mode == "system"), nil}++// teardownInstall removes an endpoint the scenario installed, so the next scenario on this machine+// starts from the same state the first one did.+//+// Only reached when the provider mutates the host. Everywhere else the instance is discarded, which+// is a stronger guarantee than any uninstall: it also takes back whatever the install touched that+// uninstall does not know about.+//+// Failures are logged rather than returned. This runs after the scenario has been judged, so turning+// a cleanup problem into a scenario failure would report the wrong thing -- but staying silent would+// leave the *next* scenario to fail for a reason that has nothing to do with it, which is exactly the+// confusion this function exists to prevent. Naming it here is what makes that traceable.+func teardownInstall(ctx context.Context, g guest, sc scenario.Scenario,+ privileged sandbox.ExecOpts, logf func(string, ...any)) {++ scope := "--user"+ if sc.Install != nil && sc.Install.Mode == "system" {+ scope = "--system"+ }+ // --keep-logs because the runtime log is the artifact this whole run exists to collect, and the+ // collection step runs after this one.+ res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" --keep-logs 2>&1", privileged)+ switch {+ case err != nil:+ logf("teardown: uninstall %s did not run: %v", scope, err)+ case res.ExitCode != 0:+ logf("teardown: uninstall %s exited %d: %s", scope, res.ExitCode, oneLine(res.Stdout))+ default:+ logf("teardown: uninstalled the %s endpoint", strings.TrimPrefix(scope, "--"))+ }+}diff --git a/beacon-sandbox/scenarios/w01-install-service.yaml b/beacon-sandbox/scenarios/w01-install-service.yamlindex 7443fb15..9ddea89a 100644--- a/beacon-sandbox/scenarios/w01-install-service.yaml+++ b/beacon-sandbox/scenarios/w01-install-service.yaml@@ -9,11 +9,11 @@ platform: windows# so without it a host where SCM registration failed would pass while testing the wrong backend# entirely -- the same trap the Linux systemd scenario documents.#-# Capture is asserted only optionally, and that is deliberate rather than lax. Hook installation on-# Windows still emits a POSIX `VAR=value cmd` string that neither Windows shell accepts, so nothing-# reaches the collector through hooks yet. Marking these required would make the scenario fail for-# a known reason and stop it reporting the service facts it exists to establish. They become-# required in the change that fixes the hook command format.+# Capture is now required. It was optional while hook installation was believed to be broken on+# Windows -- a belief a probe later disproved: Claude Code runs hook commands through Git Bash there,+# which accepts the form Beacon was emitting. Once the command became one argv, a dispatched run of+# this scenario captured 38 events, so there is no longer a known reason for these to fail and every+# reason to gate on them.prompt: >-Run exactly this PowerShell command and then report its output verbatim:Write-Output '{{canary}}' | Tee-Object -FilePath '{{sentinel}}'@@ -30,16 +30,15 @@ install:expect:- action: prompt.submittedcontains: ["{{canary}}"]- optional: truewhy: >-- Capture through an installed Windows endpoint. Optional until hook installation emits a- command Windows can execute; the service assertions above are what this scenario gates on- today, and they are checked before the session runs.+ Capture through an installed Windows endpoint, which together with the service assertions+ above is the whole claim: the collector is registered with the SCM, it is running, and events+ written by hooks actually reach it.- action: command.executedfields: ["command.command"]- optional: truewhy: >-- e.command.command is the detection surface for the whole rules/risky-command/ category. The- exporter now reads it from Claude Code's PowerShell tool, so this is the assertion that will- prove the fix end to end once hooks deliver events at all.+ e.command.command is the detection surface for the whole rules/risky-command/ category, and+ threat rules match scalar leaves, so an empty value disables every one of those rules with+ nothing reporting it. This is the assertion that proves the exporter reads Claude Code's+ PowerShell tool and not only bash.diff --git a/beacon-sandbox/scenarios/w02-install-supervised.yaml b/beacon-sandbox/scenarios/w02-install-supervised.yamlindex affae5af..328b3f29 100644--- a/beacon-sandbox/scenarios/w02-install-supervised.yaml+++ b/beacon-sandbox/scenarios/w02-install-supervised.yaml@@ -27,6 +27,10 @@ expect:contains: ["{{canary}}"]optional: truewhy: >-- Same reason as w01: hook installation on Windows does not yet emit a command Windows can- execute, so capture is recorded rather than gated. The backend selection above is what this- scenario asserts today.+ Recorded rather than gated, and for a narrower reason than it used to be. w01 and w03 have both+ captured through installed hooks in system mode, so the hook command itself works on Windows.+ What no run has yet observed is capture in *user* mode: the only attempt failed before the+ session on an OTLP port conflict with a system collector the previous scenario left running.+ That is fixed in the runner rather than here, but this stays optional until a run has actually+ demonstrated it -- promoting it on the strength of the system-mode result would be assuming the+ thing this scenario exists to check.