The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
feat(sandbox): dispatch Windows runs with `run --provider github`
shukieshah787550b04644
diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.mdindex 5c120a25..12f61c7f 100644--- a/beacon-sandbox/AGENTS.md+++ b/beacon-sandbox/AGENTS.md@@ -10,9 +10,14 @@ documentation is in `README.md` and on the docs site; this file is the operating```bashcd beacon-sandbox-go run ./cmd/beacon-sandbox doctor+go run ./cmd/beacon-sandbox doctor # for a Linux run (the default)+go run ./cmd/beacon-sandbox doctor --provider github # for a Windows run```+`--provider` decides which prerequisites are checked, because they differ: a Linux run needs a+Modal account and a local Beacon build, a Windows run needs `gh` and a dispatchable workflow.+Running the wrong set reports failures for things the chosen provider does not use.+`doctor` is free, needs no sandbox, and catches the failure modes that otherwise appearmid-run as something else entirely. Add `--fix` to let it download the collector binary.@@ -61,6 +66,7 @@ real money on the user's account, so:| Approval or permission handling | `--scenario s07-denied-tool` || `endpoint install`, config paths, service startup | `--scenario i01-install-supervised` || The systemd backend, unit files, Linux system mode | `--scenario i02-install-systemd` |+| Anything Windows | `--provider github --scenario w00-probe` || Something broad, or preparing a PR | the whole suite (no `--scenario`) |```bash@@ -201,9 +207,35 @@ Rules:visible but not blocking.- Validate without spending anything: `go test ./scenario/`.+## Windows runs++Windows scenarios do not run on Modal, which has no Windows machines. They are dispatched to a+GitHub Actions Windows runner, which runs the harness against `--provider local` and uploads what+it collected; this side downloads it and judges it with the same `check` package.++```bash+go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe+```++Four things to tell the user about a Windows run, because none of them are visible in the verdict's+outcome line:++- **A dispatch tests the pushed ref, not the working tree.** Uncommitted or unpushed work is+ silently not under test. `doctor --provider github` reports this as `dispatch_ref`; treat it the+ same way you treat `collector_freshness` — do what it asks before believing the result.+- **The credential is not verifiable from here.** The agent authenticates from the workflow's+ `windows-sandbox` environment secret, so the artifact leak check reports *unverified* rather than+ clean. Say so; never paraphrase it as a pass.+- **Host isolation is asserted, not compared.** The guest is the runner, so the before/after+ comparison every Modal run gets cannot happen. Those runs carry a `safety.host_untouched` warning+ naming their disposability evidence. That warning is expected and is not a failure — and it is+ also not a clean bill of health.+- **You cannot attach to a failing run.** Debug from the uploaded `runs/` artifacts and the job log.+## Environment constraints worth knowing-- The sandbox is **Linux/amd64 only**. This cannot verify the macOS build.+- The Modal sandbox is **Linux/amd64 only**. It cannot verify the macOS build, and Windows goes+ through the dispatched path above.- Claude Code must run as a **non-root** user; it refuses `--dangerously-skip-permissions` as root.- Switching user with `su -` strips the injected credential (login shell resets the environment).The code uses `su -p`; a test pins this.diff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.mdindex 70601008..72e91e16 100644--- a/beacon-sandbox/README.md+++ b/beacon-sandbox/README.md@@ -55,8 +55,8 @@ passed. Silence from this tool is meant to mean "checked and clean".| Command | What it does ||---------|--------------|-| `doctor [--fix] [--json]` | Check the setup and say how to fix what is missing |-| `run [--scenario ID] [--repeat N] [--keep-sandbox]` | Start a sandbox, run a session, check results |+| `doctor [--fix] [--json] [--provider NAME]` | Check the setup and say how to fix what is missing |+| `run [--scenario ID] [--repeat N] [--keep-sandbox] [--provider NAME]` | Start a sandbox, run a session, check results || `verify <run-dir> [--mutate MODE]` | Re-check a saved run. Free and instant || `diff <before> <after>` | Compare what two runs captured || `clean` | Delete saved run output |@@ -68,6 +68,33 @@ expects, rather than paying for another run.Three ways to supply the API key, in the order the tool looks: `--modal-secret NAME`,`--api-key-command CMD`, or `ANTHROPIC_API_KEY`.+## Testing on Windows++Modal has no Windows machines, so Windows scenarios run on a GitHub Actions Windows runner+instead — which is already a disposable machine with a real Service Control Manager and+administrator rights. You dispatch the run, it happens there, and the artifacts come back and are+judged on your machine by the same code that judges a Modal run:++```bash+go run ./cmd/beacon-sandbox doctor --provider github # checks gh, the workflow, the secret+go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe+```++Two things to know. A dispatch runs **the pushed ref**, not your working tree, so uncommitted work+is not under test — `doctor` warns when that is the case, which is the Windows equivalent of the+stale-collector trap below. And the agent's API key comes from the workflow's `windows-sandbox`+environment secret rather than from your shell, so the artifact leak check has no value to search+for and reports *unverified*, exactly as `--modal-secret` does. The in-runner argv scan still runs,+because the key genuinely is present there.++Scenarios say which platform they are written for (`platform: windows`), defaulting to Linux, and+`run` reports how many it set aside rather than silently narrowing the suite.++`--provider local` is what the workflow itself uses: it runs scenarios on the machine it is already+on. It refuses to start unless that machine is a GitHub-hosted runner, because a scenario installs+Beacon and registers services for real — on a workstation that is damage, not a test. Override with+`--allow-host-mutation` only on a VM you are happy to lose.+## Layout```@@ -97,8 +124,11 @@ Other options are `drop-commands`, `drop-action:<action>`, and `plant-secret`. E## LimitationsModal only provides Linux x86 machines, so this **cannot test the macOS build** — launchd, the-signed installer, and notarization are out of reach. It also confirms the thing it planted was-recorded rather than proving nothing else was missed.+signed installer, and notarization are out of reach. Windows is covered by dispatching to a CI+runner instead (see above), which has its own limitation: the run happens somewhere you cannot+attach to, so a scenario that fails opaquely is debugged from the uploaded artifacts and the job+log rather than interactively. It also confirms the thing it planted was recorded rather than+proving nothing else was missed.The `i0*` tests install Beacon for real and cover installation and service management; the `s0*`tests use a temporary collector instead, the path Beacon uses in CI and cloud agents. `i02` runsdiff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor.go b/beacon-sandbox/cmd/beacon-sandbox/doctor.goindex b41f35d1..1611ff13 100644--- a/beacon-sandbox/cmd/beacon-sandbox/doctor.go+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor.go@@ -51,6 +51,8 @@ func cmdDoctor(args []string) error {fix := fs.Bool("fix", false, "resolve what can be resolved automatically (downloads the collector)")modalSecret := fs.String("modal-secret", "", "name of a Modal secret holding ANTHROPIC_API_KEY")keyCommand := fs.String("api-key-command", "", "command whose stdout is the Anthropic API key")+ provider := fs.String("provider", "modal", "check the prerequisites for this provider: modal, github, or local")+ repo := fs.String("repo", "", "with --provider github, the owner/name that will be dispatched to")fs.Parse(args)root, err := repoRoot()@@ -70,60 +72,75 @@ func cmdDoctor(args []string) error {Fix: "install Go 1.24+ from https://go.dev/dl/"})}- // --- sandbox provider auth ---- // Attempting a real connection is the only honest check: credentials can be present but- // expired, and discovering that mid-run wastes a session.- ctx := context.Background()- if prov, err := sandbox.NewModal(ctx, appName); err == nil {- prov.Close()- add(doctorCheck{Name: "modal_auth", Status: statusOK, Detail: "authenticated"})- } else {- add(doctorCheck{Name: "modal_auth", Status: statusFail,- Detail: firstLine(err.Error()),- Fix: "pip install modal && modal token new (or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET)"})- }+ // The checks are per-provider, because a check that cannot apply must not be reported as a+ // failure. Running the Modal checks before a dispatched Windows run would demand a Modal+ // account and a Linux Beacon build for a run that uses neither, and `ready: false` on+ // prerequisites the chosen provider does not have would train people to ignore the field.+ switch *provider {+ case "github":+ for _, c := range githubChecks(root, *repo) {+ add(c)+ }+ case "local":+ for _, c := range localChecks() {+ add(c)+ }+ default:+ // --- sandbox provider auth ---+ // Attempting a real connection is the only honest check: credentials can be present but+ // expired, and discovering that mid-run wastes a session.+ ctx := context.Background()+ if prov, err := sandbox.NewModal(ctx, appName); err == nil {+ prov.Close()+ add(doctorCheck{Name: "modal_auth", Status: statusOK, Detail: "authenticated"})+ } else {+ add(doctorCheck{Name: "modal_auth", Status: statusFail,+ Detail: firstLine(err.Error()),+ Fix: "pip install modal && modal token new (or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET)"})+ }- // --- agent credential ---- creds, credErr := credentials.Resolve(credentials.Options{- ProviderSecretName: *modalSecret,- KeyCommand: *keyCommand,- })- add(credentialCheck(creds, credErr, *keyCommand))-- // --- Beacon binary under test ---- beaconPath := image.BeaconPath(root)- if fi, err := os.Stat(beaconPath); err == nil && fi.Size() > 0 {- add(doctorCheck{Name: "beacon_binary", Status: statusOK, Detail: beaconPath})- } else {- add(doctorCheck{Name: "beacon_binary", Status: statusFail,- Detail: "missing " + beaconPath,- Fix: image.BuildBeaconHint})- }+ // --- agent credential ---+ creds, credErr := credentials.Resolve(credentials.Options{+ ProviderSecretName: *modalSecret,+ KeyCommand: *keyCommand,+ })+ add(credentialCheck(creds, credErr, *keyCommand))++ // --- Beacon binary under test ---+ beaconPath := image.BeaconPath(root)+ if fi, err := os.Stat(beaconPath); err == nil && fi.Size() > 0 {+ add(doctorCheck{Name: "beacon_binary", Status: statusOK, Detail: beaconPath})+ } else {+ add(doctorCheck{Name: "beacon_binary", Status: statusFail,+ Detail: "missing " + beaconPath,+ Fix: image.BuildBeaconHint})+ }- // --- collector ---- collectorPath := image.CollectorPath(root)- if fi, err := os.Stat(collectorPath); err == nil && fi.Size() > 0 {- add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: collectorPath})- } else if *fix {- if p, err := image.EnsureCollector(root, func(f string, a ...any) {- if !*asJSON {- fmt.Printf(" "+f+"\n", a...)+ // --- collector ---+ collectorPath := image.CollectorPath(root)+ if fi, err := os.Stat(collectorPath); err == nil && fi.Size() > 0 {+ add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: collectorPath})+ } else if *fix {+ if p, err := image.EnsureCollector(root, func(f string, a ...any) {+ if !*asJSON {+ fmt.Printf(" "+f+"\n", a...)+ }+ }); err == nil {+ add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: "downloaded to " + p})+ } else {+ add(doctorCheck{Name: "collector_binary", Status: statusFail, Detail: firstLine(err.Error()),+ Fix: "see the build instructions in the error above"})}- }); err == nil {- add(doctorCheck{Name: "collector_binary", Status: statusOK, Detail: "downloaded to " + p})} else {- add(doctorCheck{Name: "collector_binary", Status: statusFail, Detail: firstLine(err.Error()),- Fix: "see the build instructions in the error above"})+ add(doctorCheck{Name: "collector_binary", Status: statusFail,+ Detail: "missing " + collectorPath,+ Fix: "beacon-sandbox doctor --fix (downloads it from the latest release)"})}- } else {- add(doctorCheck{Name: "collector_binary", Status: statusFail,- Detail: "missing " + collectorPath,- Fix: "beacon-sandbox doctor --fix (downloads it from the latest release)"})- }- // --- the trap: local exporter changes a downloaded collector will not contain ---- stale, reason := image.CollectorIsStale(root)- add(freshnessCheck(stale, reason))+ // --- the trap: local exporter changes a downloaded collector will not contain ---+ stale, reason := image.CollectorIsStale(root)+ add(freshnessCheck(stale, reason))+ }res.Ready = truefor _, c := range res.Checks {diff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor_github.go b/beacon-sandbox/cmd/beacon-sandbox/doctor_github.gonew file mode 100644index 00000000..c6576cfb--- /dev/null+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor_github.go@@ -0,0 +1,200 @@+package main++import (+ "encoding/json"+ "fmt"+ "os"+ "os/exec"+ "path/filepath"+ "strings"++ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/dispatch"+)++// githubChecks are the prerequisites for a dispatched Windows run.+//+// Different in kind from the Modal ones, and the difference is worth stating: the workflow builds+// both `beacon.exe` and `beacon-otelcol.exe` from the ref it runs, so the stale-binary trap that+// `collector_freshness` guards cannot happen there. It is replaced by a sharper one -- a dispatch+// tests the *pushed ref*, so uncommitted or unpushed work is silently not under test.+func githubChecks(root, repo string) []doctorCheck {+ var out []doctorCheck++ if _, err := exec.LookPath("gh"); err != nil {+ return append(out, doctorCheck{Name: "gh_cli", Status: statusFail,+ Detail: "gh not found on PATH",+ Fix: "install the GitHub CLI from https://cli.github.com/"})+ }+ if b, err := exec.Command("gh", "auth", "status").CombinedOutput(); err != nil {+ out = append(out, doctorCheck{Name: "gh_auth", Status: statusFail,+ Detail: firstLine(string(b)),+ Fix: "gh auth login"})+ } else {+ out = append(out, doctorCheck{Name: "gh_auth", Status: statusOK, Detail: "authenticated"})+ }++ // The workflow has to exist on the *default branch* for workflow_dispatch to be offered at+ // all, which is the one GitHub Actions rule most likely to be met with "but the file is right+ // there". Checked locally first, then remotely, so the two failures are distinguishable.+ wf := filepath.Join(root, ".github", "workflows", dispatch.Workflow)+ if _, err := os.Stat(wf); err != nil {+ out = append(out, doctorCheck{Name: "windows_workflow", Status: statusFail,+ Detail: "missing " + wf,+ Fix: "this checkout does not contain the Windows sandbox workflow; update it"})+ } else {+ out = append(out, workflowDispatchableCheck(repo))+ }++ out = append(out, dispatchRefCheck(root))+ out = append(out, windowsSecretCheck(repo))+ return out+}++// workflowDispatchableCheck asks GitHub whether the workflow can actually be dispatched.+//+// Queried against the repository rather than the checkout, so it catches the case where the file+// exists locally but the branch carrying it has not reached the default branch -- GitHub only+// offers workflow_dispatch for workflows present there, and `gh workflow run` then fails with a+// bare "could not find any workflows".+func workflowDispatchableCheck(repo string) doctorCheck {+ args := []string{"workflow", "list", "--all", "--json", "name,state,path"}+ if repo != "" {+ args = append(args, "--repo", repo)+ }+ b, err := exec.Command("gh", args...).CombinedOutput()+ if err != nil {+ return doctorCheck{Name: "windows_workflow", Status: statusWarn,+ Detail: "could not list the repository's workflows (" + firstLine(string(b)) + ")",+ Fix: "check `gh workflow list` by hand; a dispatch will fail if the workflow is absent"}+ }+ var workflows []struct {+ Name string `json:"name"`+ State string `json:"state"`+ Path string `json:"path"`+ }+ if err := json.Unmarshal(b, &workflows); err != nil {+ return doctorCheck{Name: "windows_workflow", Status: statusWarn,+ Detail: "could not parse `gh workflow list` output: " + firstLine(err.Error()),+ Fix: "check `gh workflow list` by hand"}+ }+ for _, w := range workflows {+ if filepath.Base(w.Path) != dispatch.Workflow {+ continue+ }+ if !strings.EqualFold(w.State, "active") {+ return doctorCheck{Name: "windows_workflow", Status: statusFail,+ Detail: "the workflow is " + w.State,+ Fix: "gh workflow enable " + dispatch.Workflow}+ }+ return doctorCheck{Name: "windows_workflow", Status: statusOK,+ Detail: dispatch.Workflow + " is active and dispatchable"}+ }+ return doctorCheck{Name: "windows_workflow", Status: statusFail,+ Detail: dispatch.Workflow + " exists in this checkout but the repository does not list it",+ Fix: "GitHub only offers workflow_dispatch for workflows on the default branch; " ++ "merge or push it there first"}+}++// dispatchRefCheck is the github provider's equivalent of the collector-freshness trap.+//+// A dispatch runs a *ref*, not the working tree, so local edits that have not been pushed are not+// under test -- and the run will pass or fail on code the contributor is not looking at. That is+// the same class of wasted investigation the stale-collector warning exists to prevent, so it gets+// the same treatment: a warning naming exactly what is not covered, satisfiable by doing what it+// asks.+func dispatchRefCheck(root string) doctorCheck {+ // Only the trees a Windows run actually exercises. Warning on unrelated edits would make the+ // check fire constantly and be ignored.+ watched := []string{"cli", "collector-builder", "beacon-sandbox", ".github/workflows"}+ args := append([]string{"-C", root, "status", "--porcelain", "--"}, watched...)+ b, err := exec.Command("git", args...).Output()+ if err != nil {+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,+ Detail: "could not read git status, so it is unknown whether a dispatch would test your work",+ Fix: "check `git status` by hand before relying on the result"}+ }+ var dirty []string+ for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") {+ if line = strings.TrimSpace(line); line != "" {+ dirty = append(dirty, line)+ }+ }+ if len(dirty) > 0 {+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,+ Detail: fmt.Sprintf("%d uncommitted change(s) under %s; a dispatched run tests the "++ "pushed ref, so these are NOT under test", len(dirty), strings.Join(watched, ", ")),+ Fix: "commit and push, then dispatch with --ref <branch>"}+ }++ // Committed but unpushed is the subtler half: `git status` is clean, so everything looks fine,+ // and the dispatch quietly runs the previous commit.+ if b, err := exec.Command("git", "-C", root, "rev-list", "--count", "@{upstream}..HEAD").Output(); err == nil {+ if n := strings.TrimSpace(string(b)); n != "" && n != "0" {+ return doctorCheck{Name: "dispatch_ref", Status: statusWarn,+ Detail: n + " commit(s) ahead of the upstream branch; a dispatch would run the " ++ "pushed ref, which does not include them",+ Fix: "git push, then dispatch with --ref <branch>"}+ }+ }+ return doctorCheck{Name: "dispatch_ref", Status: statusOK,+ Detail: "the working tree matches the pushed ref, so a dispatch tests what you are looking at"}+}++// windowsSecretCheck reports whether the agent credential the workflow needs is configured.+//+// Reported as a warning rather than a failure when it cannot be read: listing environment secrets+// needs repository admin, so a contributor without it would otherwise be told the setup is broken+// when it is merely unverifiable from here. The distinction is the usual one -- "I could not check"+// must never render as either a pass or a failure.+func windowsSecretCheck(repo string) doctorCheck {+ const env = "windows-sandbox"+ args := []string{"secret", "list", "--env", env}+ if repo != "" {+ args = append(args, "--repo", repo)+ }+ setCmd := "printf '%s' \"$ANTHROPIC_API_KEY\" | gh secret set ANTHROPIC_API_KEY --env " + env+ b, err := exec.Command("gh", args...).CombinedOutput()+ if err != nil {+ // A 404 is a different problem from a permission error, and they need opposite actions:+ // one means create the environment, the other means ask somebody who can see it. Reporting+ // both as "could not check" would send a reader to the wrong place.+ if strings.Contains(string(b), "404") {+ return doctorCheck{Name: "windows_secret", Status: statusFail,+ Detail: "the " + env + " environment does not exist, so the workflow cannot resolve its secret",+ Fix: "gh api -X PUT repos/{owner}/{repo}/environments/" + env ++ " then: " + setCmd}+ }+ return doctorCheck{Name: "windows_secret", Status: statusWarn,+ Detail: "could not list the " + env + " environment's secrets (" + firstLine(string(b)) + ")",+ Fix: "listing needs repo admin; if a run fails to authenticate, set it with: " + setCmd}+ }+ if !strings.Contains(string(b), "ANTHROPIC_API_KEY") {+ return doctorCheck{Name: "windows_secret", Status: statusFail,+ Detail: "the " + env + " environment has no ANTHROPIC_API_KEY, so the session cannot authenticate",+ Fix: setCmd + " (use a dedicated, budget-capped key)"}+ }+ return doctorCheck{Name: "windows_secret", Status: statusOK,+ Detail: "ANTHROPIC_API_KEY is set in the " + env + " environment"}+}++// localChecks are the prerequisites for running scenarios on this machine.+//+// Deliberately thin. The interesting question -- is this machine allowed to be mutated -- is+// answered by the provider itself at construction time, and duplicating that judgment here would+// give two places to disagree about it.+func localChecks() []doctorCheck {+ if os.Getenv("GITHUB_ACTIONS") != "true" {+ return []doctorCheck{{Name: "disposable_host", Status: statusFail,+ Detail: "not running inside GitHub Actions, so this machine cannot be shown to be disposable",+ Fix: "use --provider github to dispatch to a runner, or pass --allow-host-mutation " ++ "if this machine is genuinely throwaway"}}+ }+ env := os.Getenv("RUNNER_ENVIRONMENT")+ if env != "" && env != "github-hosted" {+ return []doctorCheck{{Name: "disposable_host", Status: statusWarn,+ Detail: "this is a " + env + " runner, which persists between jobs, so an install here outlives the run",+ Fix: "pass --allow-host-mutation only if this runner is genuinely disposable"}}+ }+ return []doctorCheck{{Name: "disposable_host", Status: statusOK,+ Detail: "github-hosted ephemeral runner"}}+}diff --git a/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.go b/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.gonew file mode 100644index 00000000..4d9be606--- /dev/null+++ b/beacon-sandbox/cmd/beacon-sandbox/doctor_github_test.go@@ -0,0 +1,130 @@+package main++import (+ "os"+ "os/exec"+ "path/filepath"+ "strings"+ "testing"+)++// The disposability gate is the one thing standing between `--provider local` and a scenario+// installing Beacon on somebody's workstation, so its three outcomes are pinned here.+func TestLocalChecksRefuseAnUnprovableHost(t *testing.T) {+ cases := []struct {+ name string+ githubEnv string+ runnerEnv string+ wantStatus checkStatus+ }{+ // Not CI at all: a developer's machine, and a scenario would install for real there.+ {"workstation", "", "", statusFail},+ // Self-hosted runners persist between jobs, so an install outlives the run -- the same+ // escape the host guard exists to catch, just relocated.+ {"self-hosted runner", "true", "self-hosted", statusWarn},+ {"github-hosted runner", "true", "github-hosted", statusOK},+ // An unreported environment is the older Actions behavior; treated as acceptable because+ // GITHUB_ACTIONS is already the stronger signal.+ {"unreported environment", "true", "", statusOK},+ }+ for _, c := range cases {+ t.Setenv("GITHUB_ACTIONS", c.githubEnv)+ t.Setenv("RUNNER_ENVIRONMENT", c.runnerEnv)++ got := localChecks()+ if len(got) != 1 {+ t.Fatalf("%s: expected exactly one check, got %d", c.name, len(got))+ }+ if got[0].Status != c.wantStatus {+ t.Errorf("%s: status = %q, want %q (detail: %s)",+ c.name, got[0].Status, c.wantStatus, got[0].Detail)+ }+ // A failing or warning check with no fix is not actionable, which is the property the+ // whole doctor output rests on.+ if c.wantStatus != statusOK && got[0].Fix == "" {+ t.Errorf("%s: a non-ok check must carry a fix", c.name)+ }+ }+}++// A dispatch runs a pushed ref, not the working tree. That makes uncommitted work silently not+// under test -- the same class of wasted investigation the stale-collector warning prevents -- so+// it has to be reported, and reported as satisfiable by doing what it asks.+func TestDispatchRefCheckWarnsWhenTheWorkingTreeIsNotWhatWouldRun(t *testing.T) {+ root := newTestRepo(t)++ // Clean tree with no upstream: nothing is knowably missing from a dispatch.+ if got := dispatchRefCheck(root); got.Status != statusOK {+ t.Errorf("a clean tree should pass, got %q: %s", got.Status, got.Detail)+ }++ // An edit under a watched tree is not in the pushed ref.+ if err := os.WriteFile(filepath.Join(root, "cli", "beacon", "main.go"),+ []byte("package main // edited\n"), 0o600); err != nil {+ t.Fatal(err)+ }+ got := dispatchRefCheck(root)+ if got.Status != statusWarn {+ t.Errorf("an uncommitted change under cli/ must warn, got %q: %s", got.Status, got.Detail)+ }+ if !strings.Contains(got.Detail, "NOT under test") {+ t.Errorf("the warning must say what is not covered, got %q", got.Detail)+ }+ if got.Fix == "" {+ t.Error("the warning must be satisfiable by doing what it asks")+ }+}++// Warning on every edit anywhere would make the check fire constantly and be ignored, which is+// worse than not having it. Only the trees a Windows run actually exercises count.+func TestDispatchRefCheckIgnoresUnrelatedEdits(t *testing.T) {+ root := newTestRepo(t)+ if err := os.WriteFile(filepath.Join(root, "README.md"), []byte("# edited\n"), 0o600); err != nil {+ t.Fatal(err)+ }+ if got := dispatchRefCheck(root); got.Status != statusOK {+ t.Errorf("an edit outside the watched trees must not warn, got %q: %s", got.Status, got.Detail)+ }+}++// newTestRepo builds a committed git repo with the directory shape the check watches.+func newTestRepo(t *testing.T) string {+ t.Helper()+ if _, err := exec.LookPath("git"); err != nil {+ t.Skip("no git on PATH")+ }+ root := t.TempDir()+ for _, dir := range []string{+ filepath.Join("cli", "beacon"),+ "collector-builder",+ "beacon-sandbox",+ filepath.Join(".github", "workflows"),+ } {+ if err := os.MkdirAll(filepath.Join(root, dir), 0o700); err != nil {+ t.Fatal(err)+ }+ }+ write := func(rel, body string) {+ if err := os.WriteFile(filepath.Join(root, rel), []byte(body), 0o600); err != nil {+ t.Fatal(err)+ }+ }+ write(filepath.Join("cli", "beacon", "main.go"), "package main\n")+ write("README.md", "# fixture\n")++ run := func(args ...string) {+ cmd := exec.Command("git", append([]string{"-C", root}, args...)...)+ // A contributor's global git config can set commit.gpgsign or a hooks path, either of+ // which would fail the commit here for reasons unrelated to what is being tested.+ cmd.Env = append(os.Environ(),+ "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null",+ "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@e", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@e")+ if out, err := cmd.CombinedOutput(); err != nil {+ t.Fatalf("git %s: %v\n%s", strings.Join(args, " "), err, out)+ }+ }+ run("init", "-q")+ run("add", "-A")+ run("commit", "-qm", "initial")+ return root+}diff --git a/beacon-sandbox/cmd/beacon-sandbox/main.go b/beacon-sandbox/cmd/beacon-sandbox/main.goindex 54288b5f..97a5070c 100644--- a/beacon-sandbox/cmd/beacon-sandbox/main.go+++ b/beacon-sandbox/cmd/beacon-sandbox/main.go@@ -21,6 +21,7 @@ import ("github.com/asymptote-labs/agent-beacon/beacon-sandbox/check""github.com/asymptote-labs/agent-beacon/beacon-sandbox/credentials"+ "github.com/asymptote-labs/agent-beacon/beacon-sandbox/dispatch""github.com/asymptote-labs/agent-beacon/beacon-sandbox/runner""github.com/asymptote-labs/agent-beacon/beacon-sandbox/sandbox""github.com/asymptote-labs/agent-beacon/beacon-sandbox/scenario"@@ -127,20 +128,31 @@ func cmdRun(args []string) error {keyCommand := fs.String("api-key-command", "", "command whose stdout is the Anthropic API key (e.g. 'op read op://vault/anthropic/key')")provider := fs.String("provider", "modal", "where scenarios run: modal (disposable Linux), github (dispatch to a Windows runner), or local (this machine)")allowHostMutation := fs.Bool("allow-host-mutation", false, "with --provider local, run on a machine this tool cannot prove is disposable")+ repo := fs.String("repo", "", "with --provider github, the owner/name to dispatch to (default: inferred from this checkout)")+ ref := fs.String("ref", "", "with --provider github, the branch or tag to run (default: the workflow's default branch)")fs.Parse(args)root, err := repoRoot()if err != nil {return err}- creds, err := credentials.Resolve(credentials.Options{- ProviderSecretName: *modalSecret,- KeyCommand: *keyCommand,- })- if err != nil {- return err+ // A dispatched run authenticates the agent from the workflow's environment secret, so no local+ // credential is needed or read. Resolving one anyway would fail on a machine that has none, and+ // printing it would claim this process supplied a key it never sent.+ var creds credentials.Resolved+ if *provider == "github" {+ fmt.Printf("credential: the %s workflow's environment secret (nothing is read locally)\n",+ dispatch.Workflow)+ } else {+ creds, err = credentials.Resolve(credentials.Options{+ ProviderSecretName: *modalSecret,+ KeyCommand: *keyCommand,+ })+ if err != nil {+ return err+ }+ fmt.Printf("credential: %s\n", creds.Describe())}- fmt.Printf("credential: %s\n", creds.Describe())// A plain directory path, absolute or relative to the repo root. An earlier version// rewrote any value without a path separator to the default directory, so `--suite core`@@ -162,6 +174,16 @@ func cmdRun(args []string) error {out = filepath.Join(root, "beacon-sandbox", "runs")}+ // A dispatched run is a different shape: the harness executes inside a CI job, and this side+ // only starts it, waits, collects and judges. Handled before a Provider is opened because+ // there is no machine here to drive.+ if *provider == "github" {+ return dispatchRun(sui, dispatchOptions{+ repo: *repo, ref: *ref, scenario: *only, claudeVersion: *claudeVer,+ outDir: out, suiteDir: scDir,+ })+ }+ctx := context.Background()prov, closeProv, err := openProvider(ctx, *provider, *allowHostMutation)if err != nil {@@ -248,14 +270,82 @@ func openProvider(ctx context.Context, name string, allowHostMutation bool) (sanreturn nil, func() {}, err}return p, func() {}, nil- case "github":- return nil, func() {}, fmt.Errorf("--provider github dispatches a workflow rather than " +- "driving a machine directly; it is not wired up yet")default:return nil, func() {}, fmt.Errorf("unknown --provider %q; want modal, github, or local", name)}}+type dispatchOptions struct {+ repo, ref, scenario, claudeVersion, outDir, suiteDir string+}++// dispatchRun starts the Windows workflow, waits for it, and judges what it collected.+//+// No credential is resolved here, and that is not an omission: the agent's key lives in the+// workflow's environment secret and never reaches this process. The consequence is recorded+// rather than hidden -- the artifact leak check has no value to search for, so it reports+// unverified, exactly as the --modal-secret path does. The in-runner argv sampler still runs,+// because there the key genuinely is present.+func dispatchRun(sui scenario.Suite, opts dispatchOptions) error {+ // Fail before spending a runner on a scenario that does not exist or is not a Windows one.+ // The workflow would otherwise start, install an agent, and only then find nothing to run.+ windows, _ := sui.For(scenario.PlatformWindows)+ if len(windows.Scenarios) == 0 {+ return fmt.Errorf("no windows scenarios in %s, so a dispatched run would assert nothing",+ opts.suiteDir)+ }+ if opts.scenario != "" {+ found := false+ for _, sc := range windows.Scenarios {+ if sc.ID == opts.scenario {+ found = true+ break+ }+ }+ if !found {+ return fmt.Errorf("no windows scenario with id %q in %s; --provider github runs "++ "windows scenarios", opts.scenario, opts.suiteDir)+ }+ }++ res, dispatchErr := dispatch.Run(dispatch.Options{+ Repo: opts.repo, Ref: opts.ref, Scenario: opts.scenario,+ ClaudeVersion: opts.claudeVersion, OutDir: opts.outDir,+ Log: func(f string, a ...any) { fmt.Printf(" "+f+"\n", a...) },+ })+ // Judge whatever came back before reporting the dispatch error. A failing job still uploads+ // its artifacts, and those artifacts carry the host-state and argv findings that matter most+ // precisely when something went wrong -- the same reason cmdRun folds them in on a failed run+ // rather than replacing the verdict with a generic message.+ root, err := repoRoot()+ if err != nil {+ return err+ }+ var verdicts []check.Verdict+ for _, dir := range res.RunDirs {+ v, err := judgeRunDir(root, dir, "")+ if err != nil {+ fmt.Printf(" could not judge %s: %v\n", dir, err)+ continue+ }+ fmt.Print(indent(v.Report()))+ verdicts = append(verdicts, v)+ }+ if dispatchErr != nil {+ if len(verdicts) > 0 {+ _ = summarize(verdicts)+ }+ return dispatchErr+ }+ // GitHub's own conclusion is reported alongside the verdicts rather than instead of them: a+ // red job with a passing scenario means the workflow broke around the run, and a green job+ // with a failing scenario is impossible by construction but worth noticing if it ever happens.+ if res.Conclusion != "success" {+ fmt.Printf(" the workflow run concluded %q: %s\n", res.Conclusion, res.URL)+ }+ return summarize(verdicts)+}+// platformOf maps a provider's guest platform onto the scenario vocabulary.func platformOf(p sandbox.Provider) scenario.Platform {if p.Platform().IsWindows() {@@ -418,63 +508,78 @@ func cmdVerify(args []string) error {}var verdicts []check.Verdictfor _, dir := range dirs {- meta := map[string]string{}- if b, err := os.ReadFile(filepath.Join(dir, "meta.json")); err == nil {- _ = json.Unmarshal(b, &meta)- }- scID := meta["scenario"]- if scID == "" {- scID = filepath.Base(dir)- }- sc, err := findScenario(root, scID)+ v, err := judgeRunDir(root, dir, *mutate)if err != nil {return err}-- logPath := filepath.Join(dir, "runtime.jsonl")- var planted string- if *mutate != "" {- logPath, planted, err = applyMutation(logPath, *mutate)- if err != nil {- return err- }- defer os.Remove(logPath)- fmt.Printf("(mutation %q applied: a PASS becoming FAIL is the check working)\n", *mutate)- }-- art := runner.Artifacts{RuntimeLog: logPath, Meta: meta}- art.Canary = meta["canary"]- // Sentinel state is not re-derivable offline, so trust what the run recorded -- including- // the detail, which is the evidence a finding quotes. Dropping it made re-judged verdicts- // show an empty detail for a sentinel the capture path had already described. Reported by- // Cursor Bugbot.- art.SentinelDetail = meta["sentinel_detail"]- art.SentinelProbed, art.SentinelPresent = decodeSentinel(meta)- art.SessionKnown, art.SessionOK = decodeSession(meta)- creds := offlineCredential(meta)- v, log := judge(sc, art, creds, planted)fmt.Print(v.Report())- // Refresh both artifacts, or neither. Only the fingerprint was rewritten before, so a- // re-judge with changed expectations left verdict.json holding the old outcome while- // fingerprint.json held the new one -- the two files on disk actively disagreeing about- // the same run, with anything reading the verdict trusting a stale judgment. Reported by- // Cursor Bugbot, and visible in this repo's own run directories.- //- // Neither is written for a mutated run: a self-test deliberately damages the input, so- // persisting its verdict would overwrite the run's real record with a fabricated failure.- if *mutate == "" {- if err := v.WriteJSON(filepath.Join(dir, "verdict.json")); err != nil {- fmt.Fprintf(os.Stderr, "warning: could not update verdict.json in %s: %v\n", dir, err)- }- if err := v.Fingerprint(log).WriteJSON(filepath.Join(dir, "fingerprint.json")); err != nil {- fmt.Fprintf(os.Stderr, "warning: could not update fingerprint.json in %s: %v\n", dir, err)- }- }verdicts = append(verdicts, v)}return summarize(verdicts)}+// judgeRunDir judges one collected run directory, offline.+//+// Shared by `verify` and by the dispatched-run path rather than duplicated, because the decoding+// it performs carries several corrections that are easy to lose in a second copy: the sentinel and+// session signals must decode as *unknown* rather than as known failures when the metadata+// predates them, and the credential the leak check searches for must be provably the one the run+// used or the check reports unverified. A second implementation would drift back into false greens.+func judgeRunDir(root, dir, mutate string) (check.Verdict, error) {+ meta := map[string]string{}+ if b, err := os.ReadFile(filepath.Join(dir, "meta.json")); err == nil {+ _ = json.Unmarshal(b, &meta)+ }+ scID := meta["scenario"]+ if scID == "" {+ scID = filepath.Base(dir)+ }+ sc, err := findScenario(root, scID)+ if err != nil {+ return check.Verdict{}, err+ }++ logPath := filepath.Join(dir, "runtime.jsonl")+ var planted string+ if mutate != "" {+ logPath, planted, err = applyMutation(logPath, mutate)+ if err != nil {+ return check.Verdict{}, err+ }+ defer os.Remove(logPath)+ fmt.Printf("(mutation %q applied: a PASS becoming FAIL is the check working)\n", mutate)+ }++ art := runner.Artifacts{RuntimeLog: logPath, Meta: meta}+ art.Canary = meta["canary"]+ // Sentinel state is not re-derivable offline, so trust what the run recorded -- including+ // the detail, which is the evidence a finding quotes. Dropping it made re-judged verdicts+ // show an empty detail for a sentinel the capture path had already described. Reported by+ // Cursor Bugbot.+ art.SentinelDetail = meta["sentinel_detail"]+ art.SentinelProbed, art.SentinelPresent = decodeSentinel(meta)+ art.SessionKnown, art.SessionOK = decodeSession(meta)+ creds := offlineCredential(meta)+ v, log := judge(sc, art, creds, planted)+ // Refresh both artifacts, or neither. Only the fingerprint was rewritten before, so a+ // re-judge with changed expectations left verdict.json holding the old outcome while+ // fingerprint.json held the new one -- the two files on disk actively disagreeing about+ // the same run, with anything reading the verdict trusting a stale judgment. Reported by+ // Cursor Bugbot, and visible in this repo's own run directories.+ //+ // Neither is written for a mutated run: a self-test deliberately damages the input, so+ // persisting its verdict would overwrite the run's real record with a fabricated failure.+ if mutate == "" {+ if err := v.WriteJSON(filepath.Join(dir, "verdict.json")); err != nil {+ fmt.Fprintf(os.Stderr, "warning: could not update verdict.json in %s: %v\n", dir, err)+ }+ if err := v.Fingerprint(log).WriteJSON(filepath.Join(dir, "fingerprint.json")); err != nil {+ fmt.Fprintf(os.Stderr, "warning: could not update fingerprint.json in %s: %v\n", dir, err)+ }+ }+ return v, nil+}+// cmdDiff compares two runs' capability fingerprints.func cmdDiff(args []string) error {fs := flag.NewFlagSet("diff", flag.ExitOnError)diff --git a/beacon-sandbox/dispatch/dispatch.go b/beacon-sandbox/dispatch/dispatch.gonew file mode 100644index 00000000..79be3d52--- /dev/null+++ b/beacon-sandbox/dispatch/dispatch.go@@ -0,0 +1,324 @@+// Package dispatch runs a scenario on a GitHub Actions Windows runner and brings the artifacts+// home.+//+// It is not a sandbox.Provider, and that is deliberate. Provider models a machine you can Exec+// against step by step; Actions is a batch substrate, and pretending otherwise would mean a+// remote-control channel and a relay to host it. Instead the whole harness runs *inside* the job+// against sandbox.LocalExec, and this package does the three things that have to happen outside+// it: start the job, wait for it, and fetch what it collected.+//+// Judging deliberately stays on this side. check is a pure function of what is on disk, so a+// dispatched Windows run is judged by the same code as a Modal Linux run -- and `verify`, `diff`+// and `--mutate` keep working on the result without knowing where it came from.+package dispatch++import (+ "encoding/json"+ "fmt"+ "os"+ "os/exec"+ "path/filepath"+ "regexp"+ "sort"+ "strings"+ "time"+)++// Workflow is the workflow file this package drives.+const Workflow = "windows-sandbox.yml"++// Options configures a dispatch.+type Options struct {+ // Repo is owner/name. Empty lets gh infer it from the checkout.+ Repo string+ // Ref is the branch or tag to run. Empty uses the current branch.+ Ref string+ // Scenario is the scenario id, or empty for every Windows scenario.+ Scenario string+ // ClaudeVersion pins the agent build.+ ClaudeVersion string+ // OutDir is where run directories are extracted, matching the local layout.+ OutDir string+ // PollInterval bounds how often the run is checked. Zero uses a sane default.+ PollInterval time.Duration+ // Timeout bounds the whole wait. Zero uses a sane default.+ Timeout time.Duration+ // Log receives progress lines.+ Log func(string, ...any)+}++// Result describes a finished dispatch.+type Result struct {+ RunID string+ URL string+ // Conclusion is GitHub's own word for the outcome: success, failure, cancelled, ...+ Conclusion string+ // RunDirs are the extracted run directories, ready for the check package.+ RunDirs []string+}++// Run dispatches the workflow, waits for it, and downloads its artifacts.+func Run(opts Options) (Result, error) {+ logf := opts.Log+ if logf == nil {+ logf = func(string, ...any) {}+ }+ if opts.PollInterval <= 0 {+ opts.PollInterval = 15 * time.Second+ }+ if opts.Timeout <= 0 {+ // The workflow's own timeout-minutes is 45; allow for queueing on top of it.+ opts.Timeout = 60 * time.Minute+ }+ if err := requireGH(); err != nil {+ return Result{}, err+ }++ // The newest run id *before* dispatching, so the new one can be identified without guessing.+ // `gh workflow run` prints nothing machine-readable and returns no id, which is the whole+ // difficulty here: polling for "the newest run" can otherwise latch onto a previous one that+ // is still in progress and report its result as this dispatch's.+ before, err := latestRunID(opts)+ if err != nil {+ return Result{}, err+ }++ args := []string{"workflow", "run", Workflow}+ if opts.Repo != "" {+ args = append(args, "--repo", opts.Repo)+ }+ if opts.Ref != "" {+ args = append(args, "--ref", opts.Ref)+ }+ if opts.Scenario != "" {+ args = append(args, "-f", "scenario="+opts.Scenario)+ }+ if opts.ClaudeVersion != "" {+ args = append(args, "-f", "claude_version="+opts.ClaudeVersion)+ }+ if out, err := runGH(args...); err != nil {+ return Result{}, fmt.Errorf("dispatch %s: %w\n%s", Workflow, err, out)+ }+ logf("dispatched %s; waiting for the run to appear", Workflow)++ runID, err := awaitNewRun(opts, before, logf)+ if err != nil {+ return Result{}, err+ }+ res := Result{RunID: runID, URL: runURL(opts.Repo, runID)}+ logf("run %s: %s", runID, res.URL)++ conclusion, err := awaitCompletion(opts, runID, logf)+ res.Conclusion = conclusion+ if err != nil {+ return res, err+ }+ logf("run %s concluded: %s", runID, conclusion)++ dirs, err := download(opts, runID, logf)+ res.RunDirs = dirs+ if err != nil {+ return res, err+ }+ // A concluded run with no artifacts is not a pass. The job uploads unconditionally, so this+ // means it failed before collecting anything -- and reporting that as "nothing to judge, all+ // clear" is exactly the false green the tool exists to prevent.+ if len(dirs) == 0 {+ return res, fmt.Errorf("run %s produced no run directories (%s), so there is nothing to "++ "judge; read the job log at %s", runID, conclusion, res.URL)+ }+ return res, nil+}++// requireGH fails early with the fix rather than letting exec produce "file not found".+func requireGH() error {+ if _, err := exec.LookPath("gh"); err != nil {+ return fmt.Errorf("the GitHub CLI (gh) is required to dispatch a Windows run; " ++ "install it and run `gh auth login`")+ }+ if out, err := runGH("auth", "status"); err != nil {+ return fmt.Errorf("gh is not authenticated: %w\n%s", err, out)+ }+ return nil+}++func runGH(args ...string) (string, error) {+ cmd := exec.Command("gh", args...)+ out, err := cmd.CombinedOutput()+ return string(out), err+}++// runIDPattern guards against a malformed id reaching a URL or a path. gh returns numeric ids;+// anything else means the output shape changed and should be reported, not interpolated.+var runIDPattern = regexp.MustCompile(`^[0-9]+$`)++type ghRun struct {+ DatabaseID int64 `json:"databaseId"`+ Status string `json:"status"`+ Conclusion string `json:"conclusion"`+ URL string `json:"url"`+}++// listRuns returns the most recent runs of this workflow, newest first.+func listRuns(opts Options, limit int) ([]ghRun, error) {+ args := []string{"run", "list", "--workflow", Workflow,+ "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url"}+ if opts.Repo != "" {+ args = append(args, "--repo", opts.Repo)+ }+ out, err := runGH(args...)+ if err != nil {+ return nil, fmt.Errorf("list runs of %s: %w\n%s", Workflow, err, out)+ }+ var runs []ghRun+ if err := json.Unmarshal([]byte(out), &runs); err != nil {+ return nil, fmt.Errorf("parse gh run list output: %w\n%s", err, out)+ }+ return runs, nil+}++func latestRunID(opts Options) (string, error) {+ runs, err := listRuns(opts, 1)+ if err != nil {+ return "", err+ }+ if len(runs) == 0 {+ // No history yet, which is normal on the first dispatch.+ return "", nil+ }+ return fmt.Sprint(runs[0].DatabaseID), nil+}++// awaitNewRun waits for a run id that differs from the one seen before dispatching.+//+// Identifying the run by "newest that is not the one already there" rather than by "newest" is+// what stops a still-running earlier dispatch being mistaken for this one -- which would report+// somebody else's verdict as yours.+func awaitNewRun(opts Options, before string, logf func(string, ...any)) (string, error) {+ deadline := time.Now().Add(5 * time.Minute)+ for time.Now().Before(deadline) {+ runs, err := listRuns(opts, 5)+ if err != nil {+ return "", err+ }+ for _, r := range runs {+ id := fmt.Sprint(r.DatabaseID)+ if id != before && runIDPattern.MatchString(id) {+ return id, nil+ }+ }+ time.Sleep(5 * time.Second)+ }+ return "", fmt.Errorf("no new %s run appeared within 5 minutes of dispatching; check "++ "`gh run list --workflow %s`", Workflow, Workflow)+}++// awaitCompletion polls until the run finishes, returning GitHub's conclusion.+//+// A non-success conclusion is returned rather than raised: the job uploads artifacts even when+// the scenario fails, and those artifacts are exactly what a maintainer needs. The caller decides+// what a failure means after judging what came back.+func awaitCompletion(opts Options, runID string, logf func(string, ...any)) (string, error) {+ deadline := time.Now().Add(opts.Timeout)+ lastStatus := ""+ for time.Now().Before(deadline) {+ runs, err := listRuns(opts, 20)+ if err != nil {+ return "", err+ }+ for _, r := range runs {+ if fmt.Sprint(r.DatabaseID) != runID {+ continue+ }+ if r.Status != lastStatus {+ logf("run %s is %s", runID, r.Status)+ lastStatus = r.Status+ }+ if r.Status == "completed" {+ return r.Conclusion, nil+ }+ }+ time.Sleep(opts.PollInterval)+ }+ return "", fmt.Errorf("run %s did not complete within %s; it may still be going at %s",+ runID, opts.Timeout, runURL(opts.Repo, runID))+}++// download fetches the run's artifacts into OutDir and returns the extracted run directories.+//+// The workflow uploads `beacon-sandbox/runs/` as one artifact, so extraction yields the same+// `<scenario>-<token>/` directories a local run would have produced. That sameness is the point:+// check, verify and diff then work on a dispatched run without knowing it was dispatched.+func download(opts Options, runID string, logf func(string, ...any)) ([]string, error) {+ if err := os.MkdirAll(opts.OutDir, 0o700); err != nil {+ return nil, err+ }+ // Into a staging directory first. gh extracts artifact contents directly into --dir, and+ // unpacking straight into OutDir would interleave with existing run directories, making it+ // impossible to tell which ones this dispatch produced.+ stage, err := os.MkdirTemp(opts.OutDir, ".dispatch-"+runID+"-")+ if err != nil {+ return nil, err+ }+ defer os.RemoveAll(stage)++ args := []string{"run", "download", runID, "--dir", stage}+ if opts.Repo != "" {+ args = append(args, "--repo", opts.Repo)+ }+ if out, err := runGH(args...); err != nil {+ // A run that failed before uploading has no artifacts, and gh treats that as an error.+ // Reported as "nothing collected" rather than as a download fault, because the two send a+ // reader to completely different places.+ if strings.Contains(out, "no artifacts") || strings.Contains(out, "not found") {+ logf("run %s uploaded no artifacts", runID)+ return nil, nil+ }+ return nil, fmt.Errorf("download artifacts of run %s: %w\n%s", runID, err, out)+ }++ dirs, err := findRunDirs(stage)+ if err != nil {+ return nil, err+ }++ var moved []string+ for _, dir := range dirs {+ dest := filepath.Join(opts.OutDir, filepath.Base(dir))+ // Remove any previous extraction of the same run directory so a rename cannot fail+ // halfway and leave two partial copies.+ _ = os.RemoveAll(dest)+ if err := os.Rename(dir, dest); err != nil {+ return moved, fmt.Errorf("move collected run %s: %w", filepath.Base(dir), err)+ }+ moved = append(moved, dest)+ logf("collected %s", dest)+ }+ return moved, nil+}++// findRunDirs locates collected run directories by looking for the runtime log, at any depth.+//+// By content rather than by path shape. The artifact name forms one directory level and the+// uploaded path may add more, so hardcoding the nesting would break the first time the workflow's+// upload path changed -- and it would break silently, as "the run collected nothing", which is the+// one failure this tool must never report when something was in fact collected.+func findRunDirs(stage string) ([]string, error) {+ var dirs []string+ err := filepath.WalkDir(stage, func(p string, d os.DirEntry, err error) error {+ if err != nil || d.IsDir() || d.Name() != "runtime.jsonl" {+ return err+ }+ dirs = append(dirs, filepath.Dir(p))+ return nil+ })+ sort.Strings(dirs)+ return dirs, err+}++func runURL(repo, runID string) string {+ if repo == "" {+ return "the run page (see `gh run view " + runID + " --web`)"+ }+ return fmt.Sprintf("https://github.com/%s/actions/runs/%s", repo, runID)+}diff --git a/beacon-sandbox/dispatch/dispatch_test.go b/beacon-sandbox/dispatch/dispatch_test.gonew file mode 100644index 00000000..e3cffe54--- /dev/null+++ b/beacon-sandbox/dispatch/dispatch_test.go@@ -0,0 +1,100 @@+package dispatch++import (+ "os"+ "path/filepath"+ "sort"+ "strings"+ "testing"+)++// Run directories are found by content, not by path shape. The artifact name forms one level and+// the uploaded path may add more, so a hardcoded nesting would break the first time the workflow's+// upload path changed -- and it would break as "the run collected nothing", which is the one thing+// this tool must never report when something was in fact collected.+func TestFindRunDirsLocatesLogsAtAnyDepth(t *testing.T) {+ stage := t.TempDir()+ want := []string{+ filepath.Join(stage, "beacon-sandbox-runs", "w00-probe-abc12345"),+ filepath.Join(stage, "beacon-sandbox-runs", "nested", "deeper", "w00-probe-def67890"),+ }+ for _, dir := range want {+ if err := os.MkdirAll(dir, 0o700); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(filepath.Join(dir, "runtime.jsonl"), []byte("{}\n"), 0o600); err != nil {+ t.Fatal(err)+ }+ }+ // A directory with other artifacts but no runtime log is not a run directory: judging it would+ // fail on a missing log rather than reporting that nothing was collected.+ decoy := filepath.Join(stage, "beacon-sandbox-runs", "no-log")+ if err := os.MkdirAll(decoy, 0o700); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(filepath.Join(decoy, "meta.json"), []byte("{}"), 0o600); err != nil {+ t.Fatal(err)+ }++ got, err := findRunDirs(stage)+ if err != nil {+ t.Fatal(err)+ }+ if len(got) != len(want) {+ t.Fatalf("found %d run dirs, want %d: %v", len(got), len(want), got)+ }+ // Sorted, because filesystem walk order is not guaranteed and a run's output should not depend+ // on it. Comparing against a sorted expectation is what pins that.+ sort.Strings(want)+ for i := range want {+ if got[i] != want[i] {+ t.Errorf("run dir %d = %q, want %q", i, got[i], want[i])+ }+ }+ if !sort.StringsAreSorted(got) {+ t.Errorf("results must be sorted so a run is reproducible, got %v", got)+ }+}++// An empty staging directory must be an empty result rather than an error: a run that failed+// before uploading is a real outcome the caller reports specifically.+func TestFindRunDirsToleratesNothingCollected(t *testing.T) {+ got, err := findRunDirs(t.TempDir())+ if err != nil {+ t.Fatalf("an empty staging directory must not error: %v", err)+ }+ if len(got) != 0 {+ t.Errorf("expected no run dirs, got %v", got)+ }+}++// gh returns numeric run ids. Anything else means the output shape changed, and interpolating it+// into a URL or a filesystem path would be worse than reporting it.+func TestRunIDPatternRejectsAnythingButDigits(t *testing.T) {+ for _, ok := range []string{"1", "18234567890"} {+ if !runIDPattern.MatchString(ok) {+ t.Errorf("%q should be accepted as a run id", ok)+ }+ }+ for _, bad := range []string{"", "abc", "12a", "../../etc", "12 34", "-1"} {+ if runIDPattern.MatchString(bad) {+ t.Errorf("%q must not be accepted as a run id", bad)+ }+ }+}++// The URL is what a reader follows to see why a run failed, so it must be a real link when the+// repository is known and an honest instruction when it is not -- never a half-built URL.+func TestRunURL(t *testing.T) {+ got := runURL("Asymptote-Labs/agent-beacon", "42")+ if got != "https://github.com/Asymptote-Labs/agent-beacon/actions/runs/42" {+ t.Errorf("runURL with a repo = %q", got)+ }+ got = runURL("", "42")+ if strings.Contains(got, "https://github.com//") {+ t.Errorf("an unknown repo must not produce a malformed URL, got %q", got)+ }+ if !strings.Contains(got, "42") {+ t.Errorf("the fallback must still identify the run, got %q", got)+ }+}