Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

fix(sandbox): do not carry one scenario's events into the next one's verdict

shukieshah3652d185069a
diff --git a/beacon-sandbox/runner/runner.go b/beacon-sandbox/runner/runner.go
index 1eb524fc..84ca6f57 100644
--- a/beacon-sandbox/runner/runner.go
+++ b/beacon-sandbox/runner/runner.go
@@ -859,9 +859,19 @@ func teardownInstall(ctx context.Context, g guest, sc scenario.Scenario,
if sc.Install != nil && sc.Install.Mode == "system" {
scope = "--system"
}
- // --keep-logs because the runtime log is the artifact this whole run exists to collect, and the
- // collection step runs after this one.
- res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" --keep-logs 2>&1", privileged)
+ // The log is deliberately *not* kept.
+ //
+ // Collection happens inline, before any deferred function runs, so this cannot take away an
+ // artifact the run still needs. Keeping it does active harm: every system-mode scenario installs
+ // to the same %ProgramData% path and the collector appends, so a kept log carries the previous
+ // scenario's events into the next one's verdict. Three scenarios in one suite reported 38, then
+ // 87, then 129 events -- 38 + 49 is 87, which is the accumulation showing through rather than a
+ // scenario capturing more.
+ //
+ // An expectation scoped by the run's canary survives that; one that only requires a field to be
+ // present does not, and would pass on a leftover from the scenario before. Removing the log is
+ // what makes each scenario judge its own run.
+ res, err := g.Exec(ctx, "beacon endpoint uninstall "+scope+" 2>&1", privileged)
switch {
case err != nil:
logf("teardown: uninstall %s did not run: %v", scope, err)
diff --git a/beacon-sandbox/scenarios/w03-hook-capture.yaml b/beacon-sandbox/scenarios/w03-hook-capture.yaml
index fa151f1e..b2238af9 100644
--- a/beacon-sandbox/scenarios/w03-hook-capture.yaml
+++ b/beacon-sandbox/scenarios/w03-hook-capture.yaml
@@ -52,11 +52,16 @@ expect:
only bash -- before that, this field was empty on Windows while every other field looked right.
- action: file.modified
+ contains: ["w03-written"]
fields: ["file.path"]
why: >-
A second, differently-shaped signal. A capture path can be broken in ways that still deliver
prompts -- one hook event configured correctly and the rest not -- so asserting only the prompt
- would pass a half-installed endpoint.
+ would pass a half-installed endpoint. Scoped to this scenario's filename rather than asserting
+ the field alone: every system-mode scenario installs to the same log path, so an expectation
+ that only requires a field to be present can be satisfied by the previous scenario's events.
+ Teardown removes the log between scenarios, and this makes the assertion correct even when it
+ does not.
- action: approval.allowed
fields: ["approval.decision"]
diff --git a/beacon-sandbox/scenarios/w04-denied-tool.yaml b/beacon-sandbox/scenarios/w04-denied-tool.yaml
index 7b6c3689..e94249e3 100644
--- a/beacon-sandbox/scenarios/w04-denied-tool.yaml
+++ b/beacon-sandbox/scenarios/w04-denied-tool.yaml
@@ -33,6 +33,13 @@ expect:
filtering on category=approval misses them otherwise, which is the bug this assertion exists to
keep fixed.
+ Nothing here ties the decision to this run's canary, because an approval event carries a tool
+ decision rather than the prompt text. That makes this assertion dependent on the log holding
+ only this scenario's events -- which is what teardown between scenarios provides, and which it
+ did not, in the first suite run: three system-mode scenarios shared one appended log, so this
+ could have been satisfied by the scenario before it. Worth knowing if it ever passes on a run
+ where the prompt assertion above failed.
+
- action: approval.denied
optional: true
why: >-