Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

feat(sandbox): dispatch Windows runs with `run --provider github`

shukieshah787550b04644
diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.md
index 5c120a25..12f61c7f 100644
--- a/beacon-sandbox/AGENTS.md
+++ b/beacon-sandbox/AGENTS.md
@@ -10,9 +10,14 @@ documentation is in `README.md` and on the docs site; this file is the operating
```bash
cd beacon-sandbox
-go run ./cmd/beacon-sandbox doctor
+go run ./cmd/beacon-sandbox doctor # for a Linux run (the default)
+go run ./cmd/beacon-sandbox doctor --provider github # for a Windows run
```
+`--provider` decides which prerequisites are checked, because they differ: a Linux run needs a
+Modal account and a local Beacon build, a Windows run needs `gh` and a dispatchable workflow.
+Running the wrong set reports failures for things the chosen provider does not use.
+
`doctor` is free, needs no sandbox, and catches the failure modes that otherwise appear
mid-run as something else entirely. Add `--fix` to let it download the collector binary.
@@ -61,6 +66,7 @@ real money on the user's account, so:
| Approval or permission handling | `--scenario s07-denied-tool` |
| `endpoint install`, config paths, service startup | `--scenario i01-install-supervised` |
| The systemd backend, unit files, Linux system mode | `--scenario i02-install-systemd` |
+| Anything Windows | `--provider github --scenario w00-probe` |
| Something broad, or preparing a PR | the whole suite (no `--scenario`) |
```bash
@@ -201,9 +207,35 @@ Rules:
visible but not blocking.
- Validate without spending anything: `go test ./scenario/`.
+## Windows runs
+
+Windows scenarios do not run on Modal, which has no Windows machines. They are dispatched to a
+GitHub Actions Windows runner, which runs the harness against `--provider local` and uploads what
+it collected; this side downloads it and judges it with the same `check` package.
+
+```bash
+go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe
+```
+
+Four things to tell the user about a Windows run, because none of them are visible in the verdict's
+outcome line:
+
+- **A dispatch tests the pushed ref, not the working tree.** Uncommitted or unpushed work is
+ silently not under test. `doctor --provider github` reports this as `dispatch_ref`; treat it the
+ same way you treat `collector_freshness` — do what it asks before believing the result.
+- **The credential is not verifiable from here.** The agent authenticates from the workflow's
+ `windows-sandbox` environment secret, so the artifact leak check reports *unverified* rather than
+ clean. Say so; never paraphrase it as a pass.
+- **Host isolation is asserted, not compared.** The guest is the runner, so the before/after
+ comparison every Modal run gets cannot happen. Those runs carry a `safety.host_untouched` warning
+ naming their disposability evidence. That warning is expected and is not a failure — and it is
+ also not a clean bill of health.
+- **You cannot attach to a failing run.** Debug from the uploaded `runs/` artifacts and the job log.
+
## Environment constraints worth knowing
-- The sandbox is **Linux/amd64 only**. This cannot verify the macOS build.
+- The Modal sandbox is **Linux/amd64 only**. It cannot verify the macOS build, and Windows goes
+ through the dispatched path above.
- Claude Code must run as a **non-root** user; it refuses `--dangerously-skip-permissions` as root.
- Switching user with `su -` strips the injected credential (login shell resets the environment).
The code uses `su -p`; a test pins this.
diff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.md
index 70601008..72e91e16 100644
--- a/beacon-sandbox/README.md
+++ b/beacon-sandbox/README.md
@@ -55,8 +55,8 @@ passed. Silence from this tool is meant to mean "checked and clean".
| Command | What it does |
|---------|--------------|
-| `doctor [--fix] [--json]` | Check the setup and say how to fix what is missing |
-| `run [--scenario ID] [--repeat N] [--keep-sandbox]` | Start a sandbox, run a session, check results |
+| `doctor [--fix] [--json] [--provider NAME]` | Check the setup and say how to fix what is missing |
+| `run [--scenario ID] [--repeat N] [--keep-sandbox] [--provider NAME]` | Start a sandbox, run a session, check results |
| `verify <run-dir> [--mutate MODE]` | Re-check a saved run. Free and instant |
| `diff <before> <after>` | Compare what two runs captured |
| `clean` | Delete saved run output |
@@ -68,6 +68,33 @@ expects, rather than paying for another run.
Three ways to supply the API key, in the order the tool looks: `--modal-secret NAME`,
`--api-key-command CMD`, or `ANTHROPIC_API_KEY`.
+## Testing on Windows
+
+Modal has no Windows machines, so Windows scenarios run on a GitHub Actions Windows runner
+instead — which is already a disposable machine with a real Service Control Manager and
+administrator rights. You dispatch the run, it happens there, and the artifacts come back and are
+judged on your machine by the same code that judges a Modal run:
+
+```bash
+go run ./cmd/beacon-sandbox doctor --provider github # checks gh, the workflow, the secret
+go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe
+```
+
+Two things to know. A dispatch runs **the pushed ref**, not your working tree, so uncommitted work
+is not under test — `doctor` warns when that is the case, which is the Windows equivalent of the
+stale-collector trap below. And the agent's API key comes from the workflow's `windows-sandbox`
+environment secret rather than from your shell, so the artifact leak check has no value to search
+for and reports *unverified*, exactly as `--modal-secret` does. The in-runner argv scan still runs,
+because the key genuinely is present there.
+
+Scenarios say which platform they are written for (`platform: windows`), defaulting to Linux, and
+`run` reports how many it set aside rather than silently narrowing the suite.
+
+`--provider local` is what the workflow itself uses: it runs scenarios on the machine it is already
+on. It refuses to start unless that machine is a GitHub-hosted runner, because a scenario installs
+Beacon and registers services for real — on a workstation that is damage, not a test. Override with
+`--allow-host-mutation` only on a VM you are happy to lose.
+
## Layout
```
@@ -97,8 +124,11 @@ Other options are `drop-commands`, `drop-action:<action>`, and `plant-secret`. E
## Limitations
Modal only provides Linux x86 machines, so this **cannot test the macOS build** — launchd, the
-signed installer, and notarization are out of reach. It also confirms the thing it planted was
-recorded rather than proving nothing else was missed.
+signed installer, and notarization are out of reach. Windows is covered by dispatching to a CI
+runner instead (see above), which has its own limitation: the run happens somewhere you cannot
+attach to, so a scenario that fails opaquely is debugged from the uploaded artifacts and the job
+log rather than interactively. It also confirms the thing it planted was recorded rather than
+proving nothing else was missed.
The `i0*` tests install Beacon for real and cover installation and service management; the `s0*`
tests use a temporary collector instead, the path Beacon uses in CI and cloud agents. `i02` runs
diff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor.go b/beacon-sandbox/cmd/beacon-sandbox/doctor.go
index b41f35d1..1611ff13 100644
--- a/beacon-sandbox/cmd/beacon-sandbox/doctor.go
+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor.go
@@ -51,6 +51,8 @@ func cmdDoctor(args []string) error {
fix := fs.Bool("fix", false, "resolve what can be resolved automatically (downloads the collector)")
modalSecret := fs.String("modal-secret", "", "name of a Modal secret holding ANTHROPIC_API_KEY")
keyCommand := fs.String("api-key-command", "", "command whose stdout is the Anthropic API key")
+ provider := fs.String("provider", "modal", "check the prerequisites for this provider: modal, github, or local")
+ repo := fs.String("repo", "", "with --provider github, the owner/name that will be dispatched to")
fs.Parse(args)
root, err := repoRoot()
@@ -70,60 +72,75 @@ func cmdDoctor(args []string) error {
Fix: "install Go 1.24+ from https://go.dev/dl/"})
}
- // --- sandbox provider auth ---
- // Attempting a real connection is the only honest check: credentials can be present but
- // expired, and discovering that mid-run wastes a session.
- ctx := context.Background()
- if prov, err := sandbox.NewModal(ctx, appName); err == nil {
- prov.Close()
- add(doctorCheck{Name: "modal_auth", Status: statusOK, Detail: "authenticated"})
- } else {
- add(doctorCheck{Name: "modal_auth", Status: statusFail,
- Detail: firstLine(err.Error()),
- Fix: "pip install modal && modal token new (or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET)"})
- }
+ // The checks are per-provider, because a check that cannot apply must not be reported as a
+ // failure. Running the Modal checks before a dispatched Windows run would demand a Modal
+ // account and a Linux Beacon build for a run that uses neither, and `ready: false` on
+ // prerequisites the chosen provider does not have would train people to ignore the field.
+ switch *provider {
+ case "github":
+ for _, c := range githubChecks(root, *repo) {
+ add(c)
+ }
+ case "local":
+ for _, c := range localChecks() {
+ add(c)
+ }
+ default:
+ // --- sandbox provider auth ---
+ // Attempting a real connection is the only honest check: credentials can be present but
+ // expired, and discovering that mid-run wastes a session.
+ ctx := context.Background()
+ if prov, err := sandbox.NewModal(ctx, appName); err == nil {
+ prov.Close()
+ add(doctorCheck{Name: "modal_auth", Status: statusOK, Detail: "authenticated"})
+ } else {
+ add(doctorCheck{Name: "modal_auth", Status: statusFail,
+ Detail: firstLine(err.Error()),
+ Fix: "pip install modal && modal token new (or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET)"})
+ }
- // --- agent credential ---
- creds, credErr := credentials.Resolve(credentials.Options{
- ProviderSecretName: *modalSecret,
- KeyCommand: *keyCommand,
- })
- add(credentialCheck(creds, credErr, *keyCommand))
-
- // --- Beacon binary under test ---
- beaconPath := image.BeaconPath(root)
- if fi, err := os.Stat(beaconPath); err == nil && fi.Size() > 0 {
- add(doctorCheck{Name: "beacon_binary", Status: statusOK, Detail: beaconPath})
- } else {
- add(doctorCheck{Name: "beacon_binary", Status: statusFail,
- Detail: "missing " + beaconPath,
- Fix: image.BuildBeaconHint})
- }
+ // --- agent credential ---
+ creds, credErr := credentials.Resolve(credentials.Options{
+ ProviderSecretName: *modalSecret,
+ KeyCommand: *keyCommand,
+ })
+ add(credentialCheck(creds, credErr, *keyCommand))
+
+ // --- Beacon binary under test ---
+ beaconPath := image.BeaconPath(root)
+ if fi, err := os.Stat(beaconPath); err == nil && fi.Size() > 0 {
+ add(doctorCheck{Name: "beacon_binary", Status: statusOK, Detail: beaconPath})
+ } else {
+ add(doctorCheck{Name: "beacon_binary", Status: statusFail,
+ Detail: "missing " + beaconPath,
+ Fix: image.BuildBeaconHint})
+ }
- // --- collector ---
- collectorPath := image.CollectorPath(root)
- if fi, err := os.Stat(collectorPath); err == nil && fi.Size() > 0 {
- add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: collectorPath})
- } else if *fix {
- if p, err := image.EnsureCollector(root, func(f string, a ...any) {
- if !*asJSON {
- fmt.Printf(" "+f+"\n", a...)
+ // --- collector ---
+ collectorPath := image.CollectorPath(root)
+ if fi, err := os.Stat(collectorPath); err == nil && fi.Size() > 0 {
+ add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: collectorPath})
+ } else if *fix {
+ if p, err := image.EnsureCollector(root, func(f string, a ...any) {
+ if !*asJSON {
+ fmt.Printf(" "+f+"\n", a...)
+ }
+ }); err == nil {
+ add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: "downloaded to " + p})
+ } else {
+ add(doctorCheck{Name: "collector_binary", Status: statusFail, Detail: firstLine(err.Error()),
+ Fix: "see the build instructions in the error above"})
}
- }); err == nil {
- add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: "downloaded to " + p})
} else {
- add(doctorCheck{Name: "collector_binary", Status: statusFail, Detail: firstLine(err.Error()),
- Fix: "see the build instructions in the error above"})
+ add(doctorCheck{Name: "collector_binary", Status: statusFail,
+ Detail: "missing " + collectorPath,
+ Fix: "beacon-sandbox doctor --fix (downloads it from the latest release)"})
}
- } else {
- add(doctorCheck{Name: "collector_binary", Status: statusFail,
- Detail: "missing " + collectorPath,
- Fix: "beacon-sandbox doctor --fix (downloads it from the latest release)"})
- }
- // --- the trap: local exporter changes a downloaded collector will not contain ---
- stale, reason := image.CollectorIsStale(root)
- add(freshnessCheck(stale, reason))
+ // --- the trap: local exporter changes a downloaded collector will not contain ---
+ stale, reason := image.CollectorIsStale(root)
+ add(freshnessCheck(stale, reason))
+ }
res.Ready = true
for _, c := range res.Checks {
diff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor_github.go b/beacon-sandbox/cmd/beacon-sandbox/doctor_github.go
new file mode 100644
index 00000000..c6576cfb
--- /dev/null
+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor_github.go
@@ -0,0 +1,200 @@
+package main
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/dispatch"
+)
+
+// githubChecks are the prerequisites for a dispatched Windows run.
+//
+// Different in kind from the Modal ones, and the difference is worth stating: the workflow builds
+// both `beacon.exe` and `beacon-otelcol.exe` from the ref it runs, so the stale-binary trap that
+// `collector_freshness` guards cannot happen there. It is replaced by a sharper one -- a dispatch
+// tests the *pushed ref*, so uncommitted or unpushed work is silently not under test.
+func githubChecks(root, repo string) []doctorCheck {
+ var out []doctorCheck
+
+ if _, err := exec.LookPath("gh"); err != nil {
+ return append(out, doctorCheck{Name: "gh_cli", Status: statusFail,
+ Detail: "gh not found on PATH",
+ Fix: "install the GitHub CLI from https://cli.github.com/"})
+ }
+ if b, err := exec.Command("gh", "auth", "status").CombinedOutput(); err != nil {
+ out = append(out, doctorCheck{Name: "gh_auth", Status: statusFail,
+ Detail: firstLine(string(b)),
+ Fix: "gh auth login"})
+ } else {
+ out = append(out, doctorCheck{Name: "gh_auth", Status: statusOK, Detail: "authenticated"})
+ }
+
+ // The workflow has to exist on the *default branch* for workflow_dispatch to be offered at
+ // all, which is the one GitHub Actions rule most likely to be met with "but the file is right
+ // there". Checked locally first, then remotely, so the two failures are distinguishable.
+ wf := filepath.Join(root, ".github", "workflows", dispatch.Workflow)
+ if _, err := os.Stat(wf); err != nil {
+ out = append(out, doctorCheck{Name: "windows_workflow", Status: statusFail,
+ Detail: "missing " + wf,
+ Fix: "this checkout does not contain the Windows sandbox workflow; update it"})
+ } else {
+ out = append(out, workflowDispatchableCheck(repo))
+ }
+
+ out = append(out, dispatchRefCheck(root))
+ out = append(out, windowsSecretCheck(repo))
+ return out
+}
+
+// workflowDispatchableCheck asks GitHub whether the workflow can actually be dispatched.
+//
+// Queried against the repository rather than the checkout, so it catches the case where the file
+// exists locally but the branch carrying it has not reached the default branch -- GitHub only
+// offers workflow_dispatch for workflows present there, and `gh workflow run` then fails with a
+// bare "could not find any workflows".
+func workflowDispatchableCheck(repo string) doctorCheck {
+ args := []string{"workflow", "list", "--all", "--json", "name,state,path"}
+ if repo != "" {
+ args = append(args, "--repo", repo)
+ }
+ b, err := exec.Command("gh", args...).CombinedOutput()
+ if err != nil {
+ return doctorCheck{Name: "windows_workflow", Status: statusWarn,
+ Detail: "could not list the repository's workflows (" + firstLine(string(b)) + ")",
+ Fix: "check `gh workflow list` by hand; a dispatch will fail if the workflow is absent"}
+ }
+ var workflows []struct {
+ Name string `json:"name"`
+ State string `json:"state"`
+ Path string `json:"path"`
+ }
+ if err := json.Unmarshal(b, &workflows); err != nil {
+ return doctorCheck{Name: "windows_workflow", Status: statusWarn,
+ Detail: "could not parse `gh workflow list` output: " + firstLine(err.Error()),
+ Fix: "check `gh workflow list` by hand"}
+ }
+ for _, w := range workflows {
+ if filepath.Base(w.Path) != dispatch.Workflow {
+ continue
+ }
+ if !strings.EqualFold(w.State, "active") {
+ return doctorCheck{Name: "windows_workflow", Status: statusFail,
+ Detail: "the workflow is " + w.State,
+ Fix: "gh workflow enable " + dispatch.Workflow}
+ }
+ return doctorCheck{Name: "windows_workflow", Status: statusOK,
+ Detail: dispatch.Workflow + " is active and dispatchable"}
+ }
+ return doctorCheck{Name: "windows_workflow", Status: statusFail,
+ Detail: dispatch.Workflow + " exists in this checkout but the repository does not list it",
+ Fix: "GitHub only offers workflow_dispatch for workflows on the default branch; " +
+ "merge or push it there first"}
+}
+
+// dispatchRefCheck is the github provider's equivalent of the collector-freshness trap.
+//
+// A dispatch runs a *ref*, not the working tree, so local edits that have not been pushed are not
+// under test -- and the run will pass or fail on code the contributor is not looking at. That is
+// the same class of wasted investigation the stale-collector warning exists to prevent, so it gets
+// the same treatment: a warning naming exactly what is not covered, satisfiable by doing what it
+// asks.
+func dispatchRefCheck(root string) doctorCheck {
+ // Only the trees a Windows run actually exercises. Warning on unrelated edits would make the
+ // check fire constantly and be ignored.
+ watched := []string{"cli", "collector-builder", "beacon-sandbox", ".github/workflows"}
+ args := append([]string{"-C", root, "status", "--porcelain", "--"}, watched...)
+ b, err := exec.Command("git", args...).Output()
+ if err != nil {
+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,
+ Detail: "could not read git status, so it is unknown whether a dispatch would test your work",
+ Fix: "check `git status` by hand before relying on the result"}
+ }
+ var dirty []string
+ for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") {
+ if line = strings.TrimSpace(line); line != "" {
+ dirty = append(dirty, line)
+ }
+ }
+ if len(dirty) > 0 {
+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,
+ Detail: fmt.Sprintf("%d uncommitted change(s) under %s; a dispatched run tests the "+
+ "pushed ref, so these are NOT under test", len(dirty), strings.Join(watched, ", ")),
+ Fix: "commit and push, then dispatch with --ref <branch>"}
+ }
+
+ // Committed but unpushed is the subtler half: `git status` is clean, so everything looks fine,
+ // and the dispatch quietly runs the previous commit.
+ if b, err := exec.Command("git", "-C", root, "rev-list", "--count", "@{upstream}..HEAD").Output(); err == nil {
+ if n := strings.TrimSpace(string(b)); n != "" && n != "0" {
+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,
+ Detail: n + " commit(s) ahead of the upstream branch; a dispatch would run the " +
+ "pushed ref, which does not include them",
+ Fix: "git push, then dispatch with --ref <branch>"}
+ }
+ }
+ return doctorCheck{Name: "dispatch_ref", Status: statusOK,
+ Detail: "the working tree matches the pushed ref, so a dispatch tests what you are looking at"}
+}
+
+// windowsSecretCheck reports whether the agent credential the workflow needs is configured.
+//
+// Reported as a warning rather than a failure when it cannot be read: listing environment secrets
+// needs repository admin, so a contributor without it would otherwise be told the setup is broken
+// when it is merely unverifiable from here. The distinction is the usual one -- "I could not check"
+// must never render as either a pass or a failure.
+func windowsSecretCheck(repo string) doctorCheck {
+ const env = "windows-sandbox"
+ args := []string{"secret", "list", "--env", env}
+ if repo != "" {
+ args = append(args, "--repo", repo)
+ }
+ setCmd := "printf '%s' \"$ANTHROPIC_API_KEY\" | gh secret set ANTHROPIC_API_KEY --env " + env
+ b, err := exec.Command("gh", args...).CombinedOutput()
+ if err != nil {
+ // A 404 is a different problem from a permission error, and they need opposite actions:
+ // one means create the environment, the other means ask somebody who can see it. Reporting
+ // both as "could not check" would send a reader to the wrong place.
+ if strings.Contains(string(b), "404") {
+ return doctorCheck{Name: "windows_secret", Status: statusFail,
+ Detail: "the " + env + " environment does not exist, so the workflow cannot resolve its secret",
+ Fix: "gh api -X PUT repos/{owner}/{repo}/environments/" + env +
+ " then: " + setCmd}
+ }
+ return doctorCheck{Name: "windows_secret", Status: statusWarn,
+ Detail: "could not list the " + env + " environment's secrets (" + firstLine(string(b)) + ")",
+ Fix: "listing needs repo admin; if a run fails to authenticate, set it with: " + setCmd}
+ }
+ if !strings.Contains(string(b), "ANTHROPIC_API_KEY") {
+ return doctorCheck{Name: "windows_secret", Status: statusFail,
+ Detail: "the " + env + " environment has no ANTHROPIC_API_KEY, so the session cannot authenticate",
+ Fix: setCmd + " (use a dedicated, budget-capped key)"}
+ }
+ return doctorCheck{Name: "windows_secret", Status: statusOK,
+ Detail: "ANTHROPIC_API_KEY is set in the " + env + " environment"}
+}
+
+// localChecks are the prerequisites for running scenarios on this machine.
+//
+// Deliberately thin. The interesting question -- is this machine allowed to be mutated -- is
+// answered by the provider itself at construction time, and duplicating that judgment here would
+// give two places to disagree about it.
+func localChecks() []doctorCheck {
+ if os.Getenv("GITHUB_ACTIONS") != "true" {
+ return []doctorCheck{{Name: "disposable_host", Status: statusFail,
+ Detail: "not running inside GitHub Actions, so this machine cannot be shown to be disposable",
+ Fix: "use --provider github to dispatch to a runner, or pass --allow-host-mutation " +
+ "if this machine is genuinely throwaway"}}
+ }
+ env := os.Getenv("RUNNER_ENVIRONMENT")
+ if env != "" && env != "github-hosted" {
+ return []doctorCheck{{Name: "disposable_host", Status: statusWarn,
+ Detail: "this is a " + env + " runner, which persists between jobs, so an install here outlives the run",
+ Fix: "pass --allow-host-mutation only if this runner is genuinely disposable"}}
+ }
+ return []doctorCheck{{Name: "disposable_host", Status: statusOK,
+ Detail: "github-hosted ephemeral runner"}}
+}
diff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.go b/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.go
new file mode 100644
index 00000000..4d9be606
--- /dev/null
+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.go
@@ -0,0 +1,130 @@
+package main
+
+import (
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+// The disposability gate is the one thing standing between `--provider local` and a scenario
+// installing Beacon on somebody's workstation, so its three outcomes are pinned here.
+func TestLocalChecksRefuseAnUnprovableHost(t *testing.T) {
+ cases := []struct {
+ name string
+ githubEnv string
+ runnerEnv string
+ wantStatus checkStatus
+ }{
+ // Not CI at all: a developer's machine, and a scenario would install for real there.
+ {"workstation", "", "", statusFail},
+ // Self-hosted runners persist between jobs, so an install outlives the run -- the same
+ // escape the host guard exists to catch, just relocated.
+ {"self-hosted runner", "true", "self-hosted", statusWarn},
+ {"github-hosted runner", "true", "github-hosted", statusOK},
+ // An unreported environment is the older Actions behavior; treated as acceptable because
+ // GITHUB_ACTIONS is already the stronger signal.
+ {"unreported environment", "true", "", statusOK},
+ }
+ for _, c := range cases {
+ t.Setenv("GITHUB_ACTIONS", c.githubEnv)
+ t.Setenv("RUNNER_ENVIRONMENT", c.runnerEnv)
+
+ got := localChecks()
+ if len(got) != 1 {
+ t.Fatalf("%s: expected exactly one check, got %d", c.name, len(got))
+ }
+ if got[0].Status != c.wantStatus {
+ t.Errorf("%s: status = %q, want %q (detail: %s)",
+ c.name, got[0].Status, c.wantStatus, got[0].Detail)
+ }
+ // A failing or warning check with no fix is not actionable, which is the property the
+ // whole doctor output rests on.
+ if c.wantStatus != statusOK && got[0].Fix == "" {
+ t.Errorf("%s: a non-ok check must carry a fix", c.name)
+ }
+ }
+}
+
+// A dispatch runs a pushed ref, not the working tree. That makes uncommitted work silently not
+// under test -- the same class of wasted investigation the stale-collector warning prevents -- so
+// it has to be reported, and reported as satisfiable by doing what it asks.
+func TestDispatchRefCheckWarnsWhenTheWorkingTreeIsNotWhatWouldRun(t *testing.T) {
+ root := newTestRepo(t)
+
+ // Clean tree with no upstream: nothing is knowably missing from a dispatch.
+ if got := dispatchRefCheck(root); got.Status != statusOK {
+ t.Errorf("a clean tree should pass, got %q: %s", got.Status, got.Detail)
+ }
+
+ // An edit under a watched tree is not in the pushed ref.
+ if err := os.WriteFile(filepath.Join(root, "cli", "beacon", "main.go"),
+ []byte("package main // edited\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ got := dispatchRefCheck(root)
+ if got.Status != statusWarn {
+ t.Errorf("an uncommitted change under cli/ must warn, got %q: %s", got.Status, got.Detail)
+ }
+ if !strings.Contains(got.Detail, "NOT under test") {
+ t.Errorf("the warning must say what is not covered, got %q", got.Detail)
+ }
+ if got.Fix == "" {
+ t.Error("the warning must be satisfiable by doing what it asks")
+ }
+}
+
+// Warning on every edit anywhere would make the check fire constantly and be ignored, which is
+// worse than not having it. Only the trees a Windows run actually exercises count.
+func TestDispatchRefCheckIgnoresUnrelatedEdits(t *testing.T) {
+ root := newTestRepo(t)
+ if err := os.WriteFile(filepath.Join(root, "README.md"), []byte("# edited\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if got := dispatchRefCheck(root); got.Status != statusOK {
+ t.Errorf("an edit outside the watched trees must not warn, got %q: %s", got.Status, got.Detail)
+ }
+}
+
+// newTestRepo builds a committed git repo with the directory shape the check watches.
+func newTestRepo(t *testing.T) string {
+ t.Helper()
+ if _, err := exec.LookPath("git"); err != nil {
+ t.Skip("no git on PATH")
+ }
+ root := t.TempDir()
+ for _, dir := range []string{
+ filepath.Join("cli", "beacon"),
+ "collector-builder",
+ "beacon-sandbox",
+ filepath.Join(".github", "workflows"),
+ } {
+ if err := os.MkdirAll(filepath.Join(root, dir), 0o700); err != nil {
+ t.Fatal(err)
+ }
+ }
+ write := func(rel, body string) {
+ if err := os.WriteFile(filepath.Join(root, rel), []byte(body), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ write(filepath.Join("cli", "beacon", "main.go"), "package main\n")
+ write("README.md", "# fixture\n")
+
+ run := func(args ...string) {
+ cmd := exec.Command("git", append([]string{"-C", root}, args...)...)
+ // A contributor's global git config can set commit.gpgsign or a hooks path, either of
+ // which would fail the commit here for reasons unrelated to what is being tested.
+ cmd.Env = append(os.Environ(),
+ "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null",
+ "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@e", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@e")
+ if out, err := cmd.CombinedOutput(); err != nil {
+ t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)
+ }
+ }
+ run("init", "-q")
+ run("add", "-A")
+ run("commit", "-qm", "initial")
+ return root
+}
diff --git a/beacon-sandbox/cmd/beacon-sandbox/main.go b/beacon-sandbox/cmd/beacon-sandbox/main.go
index 54288b5f..97a5070c 100644
--- a/beacon-sandbox/cmd/beacon-sandbox/main.go
+++ b/beacon-sandbox/cmd/beacon-sandbox/main.go
@@ -21,6 +21,7 @@ import (
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/check"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/credentials"
+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/dispatch"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/runner"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox"
"github.com/asymptote-labs/agent-beacon/beacon-sandbox/scenario"
@@ -127,20 +128,31 @@ func cmdRun(args []string) error {
keyCommand := fs.String("api-key-command", "", "command whose stdout is the Anthropic API key (e.g. 'op read op://vault/anthropic/key')")
provider := fs.String("provider", "modal", "where scenarios run: modal (disposable Linux), github (dispatch to a Windows runner), or local (this machine)")
allowHostMutation := fs.Bool("allow-host-mutation", false, "with --provider local, run on a machine this tool cannot prove is disposable")
+ repo := fs.String("repo", "", "with --provider github, the owner/name to dispatch to (default: inferred from this checkout)")
+ ref := fs.String("ref", "", "with --provider github, the branch or tag to run (default: the workflow's default branch)")
fs.Parse(args)
root, err := repoRoot()
if err != nil {
return err
}
- creds, err := credentials.Resolve(credentials.Options{
- ProviderSecretName: *modalSecret,
- KeyCommand: *keyCommand,
- })
- if err != nil {
- return err
+ // A dispatched run authenticates the agent from the workflow's environment secret, so no local
+ // credential is needed or read. Resolving one anyway would fail on a machine that has none, and
+ // printing it would claim this process supplied a key it never sent.
+ var creds credentials.Resolved
+ if *provider == "github" {
+ fmt.Printf("credential: the %s workflow's environment secret (nothing is read locally)\n",
+ dispatch.Workflow)
+ } else {
+ creds, err = credentials.Resolve(credentials.Options{
+ ProviderSecretName: *modalSecret,
+ KeyCommand: *keyCommand,
+ })
+ if err != nil {
+ return err
+ }
+ fmt.Printf("credential: %s\n", creds.Describe())
}
- fmt.Printf("credential: %s\n", creds.Describe())
// A plain directory path, absolute or relative to the repo root. An earlier version
// rewrote any value without a path separator to the default directory, so `--suite core`
@@ -162,6 +174,16 @@ func cmdRun(args []string) error {
out = filepath.Join(root, "beacon-sandbox", "runs")
}
+ // A dispatched run is a different shape: the harness executes inside a CI job, and this side
+ // only starts it, waits, collects and judges. Handled before a Provider is opened because
+ // there is no machine here to drive.
+ if *provider == "github" {
+ return dispatchRun(sui, dispatchOptions{
+ repo: *repo, ref: *ref, scenario: *only, claudeVersion: *claudeVer,
+ outDir: out, suiteDir: scDir,
+ })
+ }
+
ctx := context.Background()
prov, closeProv, err := openProvider(ctx, *provider, *allowHostMutation)
if err != nil {
@@ -248,14 +270,82 @@ func openProvider(ctx context.Context, name string, allowHostMutation bool) (san
return nil, func() {}, err
}
return p, func() {}, nil
- case "github":
- return nil, func() {}, fmt.Errorf("--provider github dispatches a workflow rather than " +
- "driving a machine directly; it is not wired up yet")
default:
return nil, func() {}, fmt.Errorf("unknown --provider %q; want modal, github, or local", name)
}
}
+type dispatchOptions struct {
+ repo, ref, scenario, claudeVersion, outDir, suiteDir string
+}
+
+// dispatchRun starts the Windows workflow, waits for it, and judges what it collected.
+//
+// No credential is resolved here, and that is not an omission: the agent's key lives in the
+// workflow's environment secret and never reaches this process. The consequence is recorded
+// rather than hidden -- the artifact leak check has no value to search for, so it reports
+// unverified, exactly as the --modal-secret path does. The in-runner argv sampler still runs,
+// because there the key genuinely is present.
+func dispatchRun(sui scenario.Suite, opts dispatchOptions) error {
+ // Fail before spending a runner on a scenario that does not exist or is not a Windows one.
+ // The workflow would otherwise start, install an agent, and only then find nothing to run.
+ windows, _ := sui.For(scenario.PlatformWindows)
+ if len(windows.Scenarios) == 0 {
+ return fmt.Errorf("no windows scenarios in %s, so a dispatched run would assert nothing",
+ opts.suiteDir)
+ }
+ if opts.scenario != "" {
+ found := false
+ for _, sc := range windows.Scenarios {
+ if sc.ID == opts.scenario {
+ found = true
+ break
+ }
+ }
+ if !found {
+ return fmt.Errorf("no windows scenario with id %q in %s; --provider github runs "+
+ "windows scenarios", opts.scenario, opts.suiteDir)
+ }
+ }
+
+ res, dispatchErr := dispatch.Run(dispatch.Options{
+ Repo: opts.repo, Ref: opts.ref, Scenario: opts.scenario,
+ ClaudeVersion: opts.claudeVersion, OutDir: opts.outDir,
+ Log: func(f string, a ...any) { fmt.Printf(" "+f+"\n", a...) },
+ })
+ // Judge whatever came back before reporting the dispatch error. A failing job still uploads
+ // its artifacts, and those artifacts carry the host-state and argv findings that matter most
+ // precisely when something went wrong -- the same reason cmdRun folds them in on a failed run
+ // rather than replacing the verdict with a generic message.
+ root, err := repoRoot()
+ if err != nil {
+ return err
+ }
+ var verdicts []check.Verdict
+ for _, dir := range res.RunDirs {
+ v, err := judgeRunDir(root, dir, "")
+ if err != nil {
+ fmt.Printf(" could not judge %s: %v\n", dir, err)
+ continue
+ }
+ fmt.Print(indent(v.Report()))
+ verdicts = append(verdicts, v)
+ }
+ if dispatchErr != nil {
+ if len(verdicts) > 0 {
+ _ = summarize(verdicts)
+ }
+ return dispatchErr
+ }
+ // GitHub's own conclusion is reported alongside the verdicts rather than instead of them: a
+ // red job with a passing scenario means the workflow broke around the run, and a green job
+ // with a failing scenario is impossible by construction but worth noticing if it ever happens.
+ if res.Conclusion != "success" {
+ fmt.Printf(" the workflow run concluded %q: %s\n", res.Conclusion, res.URL)
+ }
+ return summarize(verdicts)
+}
+
// platformOf maps a provider's guest platform onto the scenario vocabulary.
func platformOf(p sandbox.Provider) scenario.Platform {
if p.Platform().IsWindows() {
@@ -418,63 +508,78 @@ func cmdVerify(args []string) error {
}
var verdicts []check.Verdict
for _, dir := range dirs {
- meta := map[string]string{}
- if b, err := os.ReadFile(filepath.Join(dir, "meta.json")); err == nil {
- _ = json.Unmarshal(b, &meta)
- }
- scID := meta["scenario"]
- if scID == "" {
- scID = filepath.Base(dir)
- }
- sc, err := findScenario(root, scID)
+ v, err := judgeRunDir(root, dir, *mutate)
if err != nil {
return err
}
-
- logPath := filepath.Join(dir, "runtime.jsonl")
- var planted string
- if *mutate != "" {
- logPath, planted, err = applyMutation(logPath, *mutate)
- if err != nil {
- return err
- }
- defer os.Remove(logPath)
- fmt.Printf("(mutation %q applied: a PASS becoming FAIL is the check working)\n", *mutate)
- }
-
- art := runner.Artifacts{RuntimeLog: logPath, Meta: meta}
- art.Canary = meta["canary"]
- // Sentinel state is not re-derivable offline, so trust what the run recorded -- including
- // the detail, which is the evidence a finding quotes. Dropping it made re-judged verdicts
- // show an empty detail for a sentinel the capture path had already described. Reported by
- // Cursor Bugbot.
- art.SentinelDetail = meta["sentinel_detail"]
- art.SentinelProbed, art.SentinelPresent = decodeSentinel(meta)
- art.SessionKnown, art.SessionOK = decodeSession(meta)
- creds := offlineCredential(meta)
- v, log := judge(sc, art, creds, planted)
fmt.Print(v.Report())
- // Refresh both artifacts, or neither. Only the fingerprint was rewritten before, so a
- // re-judge with changed expectations left verdict.json holding the old outcome while
- // fingerprint.json held the new one -- the two files on disk actively disagreeing about
- // the same run, with anything reading the verdict trusting a stale judgment. Reported by
- // Cursor Bugbot, and visible in this repo's own run directories.
- //
- // Neither is written for a mutated run: a self-test deliberately damages the input, so
- // persisting its verdict would overwrite the run's real record with a fabricated failure.
- if *mutate == "" {
- if err := v.WriteJSON(filepath.Join(dir, "verdict.json")); err != nil {
- fmt.Fprintf(os.Stderr, "warning: could not update verdict.json in %s: %v\n", dir, err)
- }
- if err := v.Fingerprint(log).WriteJSON(filepath.Join(dir, "fingerprint.json")); err != nil {
- fmt.Fprintf(os.Stderr, "warning: could not update fingerprint.json in %s: %v\n", dir, err)
- }
- }
verdicts = append(verdicts, v)
}
return summarize(verdicts)
}
+// judgeRunDir judges one collected run directory, offline.
+//
+// Shared by `verify` and by the dispatched-run path rather than duplicated, because the decoding
+// it performs carries several corrections that are easy to lose in a second copy: the sentinel and
+// session signals must decode as *unknown* rather than as known failures when the metadata
+// predates them, and the credential the leak check searches for must be provably the one the run
+// used or the check reports unverified. A second implementation would drift back into false greens.
+func judgeRunDir(root, dir, mutate string) (check.Verdict, error) {
+ meta := map[string]string{}
+ if b, err := os.ReadFile(filepath.Join(dir, "meta.json")); err == nil {
+ _ = json.Unmarshal(b, &meta)
+ }
+ scID := meta["scenario"]
+ if scID == "" {
+ scID = filepath.Base(dir)
+ }
+ sc, err := findScenario(root, scID)
+ if err != nil {
+ return check.Verdict{}, err
+ }
+
+ logPath := filepath.Join(dir, "runtime.jsonl")
+ var planted string
+ if mutate != "" {
+ logPath, planted, err = applyMutation(logPath, mutate)
+ if err != nil {
+ return check.Verdict{}, err
+ }
+ defer os.Remove(logPath)
+ fmt.Printf("(mutation %q applied: a PASS becoming FAIL is the check working)\n", mutate)
+ }
+
+ art := runner.Artifacts{RuntimeLog: logPath, Meta: meta}
+ art.Canary = meta["canary"]
+ // Sentinel state is not re-derivable offline, so trust what the run recorded -- including
+ // the detail, which is the evidence a finding quotes. Dropping it made re-judged verdicts
+ // show an empty detail for a sentinel the capture path had already described. Reported by
+ // Cursor Bugbot.
+ art.SentinelDetail = meta["sentinel_detail"]
+ art.SentinelProbed, art.SentinelPresent = decodeSentinel(meta)
+ art.SessionKnown, art.SessionOK = decodeSession(meta)
+ creds := offlineCredential(meta)
+ v, log := judge(sc, art, creds, planted)
+ // Refresh both artifacts, or neither. Only the fingerprint was rewritten before, so a
+ // re-judge with changed expectations left verdict.json holding the old outcome while
+ // fingerprint.json held the new one -- the two files on disk actively disagreeing about
+ // the same run, with anything reading the verdict trusting a stale judgment. Reported by
+ // Cursor Bugbot, and visible in this repo's own run directories.
+ //
+ // Neither is written for a mutated run: a self-test deliberately damages the input, so
+ // persisting its verdict would overwrite the run's real record with a fabricated failure.
+ if mutate == "" {
+ if err := v.WriteJSON(filepath.Join(dir, "verdict.json")); err != nil {
+ fmt.Fprintf(os.Stderr, "warning: could not update verdict.json in %s: %v\n", dir, err)
+ }
+ if err := v.Fingerprint(log).WriteJSON(filepath.Join(dir, "fingerprint.json")); err != nil {
+ fmt.Fprintf(os.Stderr, "warning: could not update fingerprint.json in %s: %v\n", dir, err)
+ }
+ }
+ return v, nil
+}
+
// cmdDiff compares two runs' capability fingerprints.
func cmdDiff(args []string) error {
fs := flag.NewFlagSet("diff", flag.ExitOnError)
diff --git a/beacon-sandbox/dispatch/dispatch.go b/beacon-sandbox/dispatch/dispatch.go
new file mode 100644
index 00000000..79be3d52
--- /dev/null
+++ b/beacon-sandbox/dispatch/dispatch.go
@@ -0,0 +1,324 @@
+// Package dispatch runs a scenario on a GitHub Actions Windows runner and brings the artifacts
+// home.
+//
+// It is not a sandbox.Provider, and that is deliberate. Provider models a machine you can Exec
+// against step by step; Actions is a batch substrate, and pretending otherwise would mean a
+// remote-control channel and a relay to host it. Instead the whole harness runs *inside* the job
+// against sandbox.LocalExec, and this package does the three things that have to happen outside
+// it: start the job, wait for it, and fetch what it collected.
+//
+// Judging deliberately stays on this side. check is a pure function of what is on disk, so a
+// dispatched Windows run is judged by the same code as a Modal Linux run -- and `verify`, `diff`
+// and `--mutate` keep working on the result without knowing where it came from.
+package dispatch
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strings"
+ "time"
+)
+
+// Workflow is the workflow file this package drives.
+const Workflow = "windows-sandbox.yml"
+
+// Options configures a dispatch.
+type Options struct {
+ // Repo is owner/name. Empty lets gh infer it from the checkout.
+ Repo string
+ // Ref is the branch or tag to run. Empty uses the current branch.
+ Ref string
+ // Scenario is the scenario id, or empty for every Windows scenario.
+ Scenario string
+ // ClaudeVersion pins the agent build.
+ ClaudeVersion string
+ // OutDir is where run directories are extracted, matching the local layout.
+ OutDir string
+ // PollInterval bounds how often the run is checked. Zero uses a sane default.
+ PollInterval time.Duration
+ // Timeout bounds the whole wait. Zero uses a sane default.
+ Timeout time.Duration
+ // Log receives progress lines.
+ Log func(string, ...any)
+}
+
+// Result describes a finished dispatch.
+type Result struct {
+ RunID string
+ URL string
+ // Conclusion is GitHub's own word for the outcome: success, failure, cancelled, ...
+ Conclusion string
+ // RunDirs are the extracted run directories, ready for the check package.
+ RunDirs []string
+}
+
+// Run dispatches the workflow, waits for it, and downloads its artifacts.
+func Run(opts Options) (Result, error) {
+ logf := opts.Log
+ if logf == nil {
+ logf = func(string, ...any) {}
+ }
+ if opts.PollInterval <= 0 {
+ opts.PollInterval = 15 * time.Second
+ }
+ if opts.Timeout <= 0 {
+ // The workflow's own timeout-minutes is 45; allow for queueing on top of it.
+ opts.Timeout = 60 * time.Minute
+ }
+ if err := requireGH(); err != nil {
+ return Result{}, err
+ }
+
+ // The newest run id *before* dispatching, so the new one can be identified without guessing.
+ // `gh workflow run` prints nothing machine-readable and returns no id, which is the whole
+ // difficulty here: polling for "the newest run" can otherwise latch onto a previous one that
+ // is still in progress and report its result as this dispatch's.
+ before, err := latestRunID(opts)
+ if err != nil {
+ return Result{}, err
+ }
+
+ args := []string{"workflow", "run", Workflow}
+ if opts.Repo != "" {
+ args = append(args, "--repo", opts.Repo)
+ }
+ if opts.Ref != "" {
+ args = append(args, "--ref", opts.Ref)
+ }
+ if opts.Scenario != "" {
+ args = append(args, "-f", "scenario="+opts.Scenario)
+ }
+ if opts.ClaudeVersion != "" {
+ args = append(args, "-f", "claude_version="+opts.ClaudeVersion)
+ }
+ if out, err := runGH(args...); err != nil {
+ return Result{}, fmt.Errorf("dispatch %s: %w\n%s", Workflow, err, out)
+ }
+ logf("dispatched %s; waiting for the run to appear", Workflow)
+
+ runID, err := awaitNewRun(opts, before, logf)
+ if err != nil {
+ return Result{}, err
+ }
+ res := Result{RunID: runID, URL: runURL(opts.Repo, runID)}
+ logf("run %s: %s", runID, res.URL)
+
+ conclusion, err := awaitCompletion(opts, runID, logf)
+ res.Conclusion = conclusion
+ if err != nil {
+ return res, err
+ }
+ logf("run %s concluded: %s", runID, conclusion)
+
+ dirs, err := download(opts, runID, logf)
+ res.RunDirs = dirs
+ if err != nil {
+ return res, err
+ }
+ // A concluded run with no artifacts is not a pass. The job uploads unconditionally, so this
+ // means it failed before collecting anything -- and reporting that as "nothing to judge, all
+ // clear" is exactly the false green the tool exists to prevent.
+ if len(dirs) == 0 {
+ return res, fmt.Errorf("run %s produced no run directories (%s), so there is nothing to "+
+ "judge; read the job log at %s", runID, conclusion, res.URL)
+ }
+ return res, nil
+}
+
+// requireGH fails early with the fix rather than letting exec produce "file not found".
+func requireGH() error {
+ if _, err := exec.LookPath("gh"); err != nil {
+ return fmt.Errorf("the GitHub CLI (gh) is required to dispatch a Windows run; " +
+ "install it and run `gh auth login`")
+ }
+ if out, err := runGH("auth", "status"); err != nil {
+ return fmt.Errorf("gh is not authenticated: %w\n%s", err, out)
+ }
+ return nil
+}
+
+func runGH(args ...string) (string, error) {
+ cmd := exec.Command("gh", args...)
+ out, err := cmd.CombinedOutput()
+ return string(out), err
+}
+
+// runIDPattern guards against a malformed id reaching a URL or a path. gh returns numeric ids;
+// anything else means the output shape changed and should be reported, not interpolated.
+var runIDPattern = regexp.MustCompile(`^[0-9]+$`)
+
+type ghRun struct {
+ DatabaseID int64 `json:"databaseId"`
+ Status string `json:"status"`
+ Conclusion string `json:"conclusion"`
+ URL string `json:"url"`
+}
+
+// listRuns returns the most recent runs of this workflow, newest first.
+func listRuns(opts Options, limit int) ([]ghRun, error) {
+ args := []string{"run", "list", "--workflow", Workflow,
+ "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url"}
+ if opts.Repo != "" {
+ args = append(args, "--repo", opts.Repo)
+ }
+ out, err := runGH(args...)
+ if err != nil {
+ return nil, fmt.Errorf("list runs of %s: %w\n%s", Workflow, err, out)
+ }
+ var runs []ghRun
+ if err := json.Unmarshal([]byte(out), &runs); err != nil {
+ return nil, fmt.Errorf("parse gh run list output: %w\n%s", err, out)
+ }
+ return runs, nil
+}
+
+func latestRunID(opts Options) (string, error) {
+ runs, err := listRuns(opts, 1)
+ if err != nil {
+ return "", err
+ }
+ if len(runs) == 0 {
+ // No history yet, which is normal on the first dispatch.
+ return "", nil
+ }
+ return fmt.Sprint(runs[0].DatabaseID), nil
+}
+
+// awaitNewRun waits for a run id that differs from the one seen before dispatching.
+//
+// Identifying the run by "newest that is not the one already there" rather than by "newest" is
+// what stops a still-running earlier dispatch being mistaken for this one -- which would report
+// somebody else's verdict as yours.
+func awaitNewRun(opts Options, before string, logf func(string, ...any)) (string, error) {
+ deadline := time.Now().Add(5 * time.Minute)
+ for time.Now().Before(deadline) {
+ runs, err := listRuns(opts, 5)
+ if err != nil {
+ return "", err
+ }
+ for _, r := range runs {
+ id := fmt.Sprint(r.DatabaseID)
+ if id != before && runIDPattern.MatchString(id) {
+ return id, nil
+ }
+ }
+ time.Sleep(5 * time.Second)
+ }
+ return "", fmt.Errorf("no new %s run appeared within 5 minutes of dispatching; check "+
+ "`gh run list --workflow %s`", Workflow, Workflow)
+}
+
+// awaitCompletion polls until the run finishes, returning GitHub's conclusion.
+//
+// A non-success conclusion is returned rather than raised: the job uploads artifacts even when
+// the scenario fails, and those artifacts are exactly what a maintainer needs. The caller decides
+// what a failure means after judging what came back.
+func awaitCompletion(opts Options, runID string, logf func(string, ...any)) (string, error) {
+ deadline := time.Now().Add(opts.Timeout)
+ lastStatus := ""
+ for time.Now().Before(deadline) {
+ runs, err := listRuns(opts, 20)
+ if err != nil {
+ return "", err
+ }
+ for _, r := range runs {
+ if fmt.Sprint(r.DatabaseID) != runID {
+ continue
+ }
+ if r.Status != lastStatus {
+ logf("run %s is %s", runID, r.Status)
+ lastStatus = r.Status
+ }
+ if r.Status == "completed" {
+ return r.Conclusion, nil
+ }
+ }
+ time.Sleep(opts.PollInterval)
+ }
+ return "", fmt.Errorf("run %s did not complete within %s; it may still be going at %s",
+ runID, opts.Timeout, runURL(opts.Repo, runID))
+}
+
+// download fetches the run's artifacts into OutDir and returns the extracted run directories.
+//
+// The workflow uploads `beacon-sandbox/runs/` as one artifact, so extraction yields the same
+// `<scenario>-<token>/` directories a local run would have produced. That sameness is the point:
+// check, verify and diff then work on a dispatched run without knowing it was dispatched.
+func download(opts Options, runID string, logf func(string, ...any)) ([]string, error) {
+ if err := os.MkdirAll(opts.OutDir, 0o700); err != nil {
+ return nil, err
+ }
+ // Into a staging directory first. gh extracts artifact contents directly into --dir, and
+ // unpacking straight into OutDir would interleave with existing run directories, making it
+ // impossible to tell which ones this dispatch produced.
+ stage, err := os.MkdirTemp(opts.OutDir, ".dispatch-"+runID+"-")
+ if err != nil {
+ return nil, err
+ }
+ defer os.RemoveAll(stage)
+
+ args := []string{"run", "download", runID, "--dir", stage}
+ if opts.Repo != "" {
+ args = append(args, "--repo", opts.Repo)
+ }
+ if out, err := runGH(args...); err != nil {
+ // A run that failed before uploading has no artifacts, and gh treats that as an error.
+ // Reported as "nothing collected" rather than as a download fault, because the two send a
+ // reader to completely different places.
+ if strings.Contains(out, "no artifacts") || strings.Contains(out, "not found") {
+ logf("run %s uploaded no artifacts", runID)
+ return nil, nil
+ }
+ return nil, fmt.Errorf("download artifacts of run %s: %w\n%s", runID, err, out)
+ }
+
+ dirs, err := findRunDirs(stage)
+ if err != nil {
+ return nil, err
+ }
+
+ var moved []string
+ for _, dir := range dirs {
+ dest := filepath.Join(opts.OutDir, filepath.Base(dir))
+ // Remove any previous extraction of the same run directory so a rename cannot fail
+ // halfway and leave two partial copies.
+ _ = os.RemoveAll(dest)
+ if err := os.Rename(dir, dest); err != nil {
+ return moved, fmt.Errorf("move collected run %s: %w", filepath.Base(dir), err)
+ }
+ moved = append(moved, dest)
+ logf("collected %s", dest)
+ }
+ return moved, nil
+}
+
+// findRunDirs locates collected run directories by looking for the runtime log, at any depth.
+//
+// By content rather than by path shape. The artifact name forms one directory level and the
+// uploaded path may add more, so hardcoding the nesting would break the first time the workflow's
+// upload path changed -- and it would break silently, as "the run collected nothing", which is the
+// one failure this tool must never report when something was in fact collected.
+func findRunDirs(stage string) ([]string, error) {
+ var dirs []string
+ err := filepath.WalkDir(stage, func(p string, d os.DirEntry, err error) error {
+ if err != nil || d.IsDir() || d.Name() != "runtime.jsonl" {
+ return err
+ }
+ dirs = append(dirs, filepath.Dir(p))
+ return nil
+ })
+ sort.Strings(dirs)
+ return dirs, err
+}
+
+func runURL(repo, runID string) string {
+ if repo == "" {
+ return "the run page (see `gh run view " + runID + " --web`)"
+ }
+ return fmt.Sprintf("https://github.com/%s/actions/runs/%s", repo, runID)
+}
diff --git a/beacon-sandbox/dispatch/dispatch_test.go b/beacon-sandbox/dispatch/dispatch_test.go
new file mode 100644
index 00000000..e3cffe54
--- /dev/null
+++ b/beacon-sandbox/dispatch/dispatch_test.go
@@ -0,0 +1,100 @@
+package dispatch
+
+import (
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "testing"
+)
+
+// Run directories are found by content, not by path shape. The artifact name forms one level and
+// the uploaded path may add more, so a hardcoded nesting would break the first time the workflow's
+// upload path changed -- and it would break as "the run collected nothing", which is the one thing
+// this tool must never report when something was in fact collected.
+func TestFindRunDirsLocatesLogsAtAnyDepth(t *testing.T) {
+ stage := t.TempDir()
+ want := []string{
+ filepath.Join(stage, "beacon-sandbox-runs", "w00-probe-abc12345"),
+ filepath.Join(stage, "beacon-sandbox-runs", "nested", "deeper", "w00-probe-def67890"),
+ }
+ for _, dir := range want {
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(dir, "runtime.jsonl"), []byte("{}\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ // A directory with other artifacts but no runtime log is not a run directory: judging it would
+ // fail on a missing log rather than reporting that nothing was collected.
+ decoy := filepath.Join(stage, "beacon-sandbox-runs", "no-log")
+ if err := os.MkdirAll(decoy, 0o700); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(decoy, "meta.json"), []byte("{}"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+
+ got, err := findRunDirs(stage)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != len(want) {
+ t.Fatalf("found %d run dirs, want %d: %v", len(got), len(want), got)
+ }
+ // Sorted, because filesystem walk order is not guaranteed and a run's output should not depend
+ // on it. Comparing against a sorted expectation is what pins that.
+ sort.Strings(want)
+ for i := range want {
+ if got[i] != want[i] {
+ t.Errorf("run dir %d = %q, want %q", i, got[i], want[i])
+ }
+ }
+ if !sort.StringsAreSorted(got) {
+ t.Errorf("results must be sorted so a run is reproducible, got %v", got)
+ }
+}
+
+// An empty staging directory must be an empty result rather than an error: a run that failed
+// before uploading is a real outcome the caller reports specifically.
+func TestFindRunDirsToleratesNothingCollected(t *testing.T) {
+ got, err := findRunDirs(t.TempDir())
+ if err != nil {
+ t.Fatalf("an empty staging directory must not error: %v", err)
+ }
+ if len(got) != 0 {
+ t.Errorf("expected no run dirs, got %v", got)
+ }
+}
+
+// gh returns numeric run ids. Anything else means the output shape changed, and interpolating it
+// into a URL or a filesystem path would be worse than reporting it.
+func TestRunIDPatternRejectsAnythingButDigits(t *testing.T) {
+ for _, ok := range []string{"1", "18234567890"} {
+ if !runIDPattern.MatchString(ok) {
+ t.Errorf("%q should be accepted as a run id", ok)
+ }
+ }
+ for _, bad := range []string{"", "abc", "12a", "../../etc", "12 34", "-1"} {
+ if runIDPattern.MatchString(bad) {
+ t.Errorf("%q must not be accepted as a run id", bad)
+ }
+ }
+}
+
+// The URL is what a reader follows to see why a run failed, so it must be a real link when the
+// repository is known and an honest instruction when it is not -- never a half-built URL.
+func TestRunURL(t *testing.T) {
+ got := runURL("Asymptote-Labs/agent-beacon", "42")
+ if got != "https://github.com/Asymptote-Labs/agent-beacon/actions/runs/42" {
+ t.Errorf("runURL with a repo = %q", got)
+ }
+ got = runURL("", "42")
+ if strings.Contains(got, "https://github.com//") {
+ t.Errorf("an unknown repo must not produce a malformed URL, got %q", got)
+ }
+ if !strings.Contains(got, "42") {
+ t.Errorf("the fallback must still identify the run, got %q", got)
+ }
+}