The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(sandbox): do not carry one scenario's events into the next one's verdict
shukieshah3652d185069a
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.goindex 1eb524fc..84ca6f57 100644--- a/beacon-sandbox/runner/runner.go+++ b/beacon-sandbox/runner/runner.go@@ -859,9 +859,19 @@ func teardownInstall(ctx context.Context, g guest, sc scenario.Scenario,if sc.Install != nil && sc.Install.Mode == "system" {scope = "--system"}- // --keep-logs because the runtime log is the artifact this whole run exists to collect, and the- // collection step runs after this one.- res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" --keep-logs 2>&1", privileged)+ // The log is deliberately *not* kept.+ //+ // Collection happens inline, before any deferred function runs, so this cannot take away an+ // artifact the run still needs. Keeping it does active harm: every system-mode scenario installs+ // to the same %ProgramData% path and the collector appends, so a kept log carries the previous+ // scenario's events into the next one's verdict. Three scenarios in one suite reported 38, then+ // 87, then 129 events -- 38 + 49 is 87, which is the accumulation showing through rather than a+ // scenario capturing more.+ //+ // An expectation scoped by the run's canary survives that; one that only requires a field to be+ // present does not, and would pass on a leftover from the scenario before. Removing the log is+ // what makes each scenario judge its own run.+ res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" 2>&1", privileged)switch {case err != nil:logf("teardown: uninstall %s did not run: %v", scope, err)diff --git a/beacon-sandbox/scenarios/w03-hook-capture.yaml b/beacon-sandbox/scenarios/w03-hook-capture.yamlindex fa151f1e..b2238af9 100644--- a/beacon-sandbox/scenarios/w03-hook-capture.yaml+++ b/beacon-sandbox/scenarios/w03-hook-capture.yaml@@ -52,11 +52,16 @@ expect:only bash -- before that, this field was empty on Windows while every other field looked right.- action: file.modified+ contains: ["w03-written"]fields: ["file.path"]why: >-A second, differently-shaped signal. A capture path can be broken in ways that still deliverprompts -- one hook event configured correctly and the rest not -- so asserting only the prompt- would pass a half-installed endpoint.+ would pass a half-installed endpoint. Scoped to this scenario's filename rather than asserting+ the field alone: every system-mode scenario installs to the same log path, so an expectation+ that only requires a field to be present can be satisfied by the previous scenario's events.+ Teardown removes the log between scenarios, and this makes the assertion correct even when it+ does not.- action: approval.allowedfields: ["approval.decision"]diff --git a/beacon-sandbox/scenarios/w04-denied-tool.yaml b/beacon-sandbox/scenarios/w04-denied-tool.yamlindex 7b6c3689..e94249e3 100644--- a/beacon-sandbox/scenarios/w04-denied-tool.yaml+++ b/beacon-sandbox/scenarios/w04-denied-tool.yaml@@ -33,6 +33,13 @@ expect:filtering on category=approval misses them otherwise, which is the bug this assertion exists tokeep fixed.+ Nothing here ties the decision to this run's canary, because an approval event carries a tool+ decision rather than the prompt text. That makes this assertion dependent on the log holding+ only this scenario's events -- which is what teardown between scenarios provides, and which it+ did not, in the first suite run: three system-mode scenarios shared one appended log, so this+ could have been satisfied by the scenario before it. Worth knowing if it ever passes on a run+ where the prompt assertion above failed.+- action: approval.deniedoptional: truewhy: >-