The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu
Go agent-ai-cliagent-harnessesagent-securitybrowser-extensionciclaude-codecloudcodexcursordetection-engineeringjevknowledge-basememorymobile-device-managementobservabilitysecuritysecurity-information-and-event-managementsecurity-toolstelemetrytraces
fix(sandbox): match each Windows dispatch to its own workflow run (#681)
Claude4271846e287e
diff --git a/.github/workflows/windows-sandbox.yml b/.github/workflows/windows-sandbox.ymlindex 1388a1f2..3755defb 100644--- a/.github/workflows/windows-sandbox.yml+++ b/.github/workflows/windows-sandbox.yml@@ -12,6 +12,12 @@# reachable from untrusted code, so there is no pull_request trigger here and there must never be# a pull_request_target one.name: Windows sandbox+# The correlation id is what lets `beacon-sandbox run --provider github` recognize the run its own+# dispatch created. `gh workflow run` returns no run id, and two dispatches made moments apart are+# otherwise indistinguishable: each caller would attach to whichever run appeared first (#681).+# Manual dispatches leave it empty and keep the plain workflow name as their title. It is only ever+# rendered into this title, never into a script, so it needs no validation of its own.+run-name: ${{ inputs.correlation_id && format('Windows sandbox [{0}]', inputs.correlation_id) || 'Windows sandbox' }}on:workflow_dispatch:@@ -25,6 +31,11 @@ on:required: falsedefault: "2.1.220"type: string+ correlation_id:+ description: "Set by beacon-sandbox to find this dispatch's run; leave empty when dispatching by hand"+ required: false+ default: ""+ type: stringpermissions:contents: readdiff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.mdindex bbd26066..0df8eac3 100644--- a/beacon-sandbox/AGENTS.md+++ b/beacon-sandbox/AGENTS.md@@ -218,6 +218,15 @@ it collected; this side downloads it and judges it with the same `check` packagego run ./cmd/beacon-sandbox run --provider github --scenario w00-probe```+Concurrent dispatches never mix up their results. Each one sends a random `correlation_id`+input, the workflow stamps it into the run's title, and the dispatch follows only the run carrying+it; runs without it are waited past. If none carrying it appears, the dispatch fails with an error+naming the runs it saw rather than judging one of them. A branch cut before the workflow declared+that input is refused with a request to rebase onto main. The runs still execute one at a time+(the workflow's concurrency group), and GitHub keeps only one pending run per group, so a third+concurrent dispatch cancels the one queued before it; that caller reports its own run as+`cancelled` with nothing collected, which is a reason to re-run, not a Beacon failure.+Four things to tell the user about a Windows run, because none of them are visible in the verdict'soutcome line:diff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.mdindex 72e91e16..bd08f081 100644--- a/beacon-sandbox/README.md+++ b/beacon-sandbox/README.md@@ -87,6 +87,12 @@ environment secret rather than from your shell, so the artifact leak check has nfor and reports *unverified*, exactly as `--modal-secret` does. The in-runner argv scan still runs,because the key genuinely is present there.+Several dispatches can run at once without mixing up their results. Each sends a random+`correlation_id` that the workflow puts in the run's title, and each follows only the run that+carries its own id, so one maintainer's verdict is never built from another's artifacts. The+workflow reads that input from the dispatched branch, so a branch cut before it existed needs a+rebase onto main first.+Scenarios say which platform they are written for (`platform: windows`), defaulting to Linux, and`run` reports how many it set aside rather than silently narrowing the suite.diff --git a/beacon-sandbox/dispatch/dispatch.go b/beacon-sandbox/dispatch/dispatch.goindex 5376ccb9..673896b6 100644--- a/beacon-sandbox/dispatch/dispatch.go+++ b/beacon-sandbox/dispatch/dispatch.go@@ -13,6 +13,8 @@package dispatchimport (+ "crypto/rand"+ "encoding/hex""encoding/json""fmt""os"@@ -27,6 +29,11 @@ import (// Workflow is the workflow file this package drives.const Workflow = "windows-sandbox.yml"+// correlationInput is the workflow_dispatch input that ties a run to the dispatch that created it.+// The workflow puts it in its run-name, so the run's display title carries it from the moment the+// run exists.+const correlationInput = "correlation_id"+// Options configures a dispatch.type Options struct {// Repo is owner/name. Empty lets gh infer it from the checkout.@@ -51,6 +58,11 @@ type Options struct {Timeout time.Duration// Log receives progress lines.Log func(string, ...any)++ // newRunPoll and newRunTimeout bound the wait for the dispatched run to appear. Zero uses the+ // defaults; they are unexported because only tests have a reason to shorten them.+ newRunPoll time.Duration+ newRunTimeout time.Duration}// Result describes a finished dispatch.@@ -76,6 +88,12 @@ func Run(opts Options) (Result, error) {// The workflow's own timeout-minutes is 45; allow for queueing on top of it.opts.Timeout = 60 * time.Minute}+ if opts.newRunPoll <= 0 {+ opts.newRunPoll = 5 * time.Second+ }+ if opts.newRunTimeout <= 0 {+ opts.newRunTimeout = 5 * time.Minute+ }if err := requireGH(); err != nil {return Result{}, err}@@ -92,10 +110,22 @@ func Run(opts Options) (Result, error) {}logf("dispatching against ref %s", opts.Ref)- // The newest run id *before* dispatching, so the new one can be identified without guessing.// `gh workflow run` prints nothing machine-readable and returns no id, which is the whole- // difficulty here: polling for "the newest run" can otherwise latch onto a previous one that- // is still in progress and report its result as this dispatch's.+ // difficulty here: the run this dispatch creates has to be recognized among runs other people+ // and other invocations create at the same time. So every dispatch carries an id of its own,+ // the workflow puts it in the run's title, and only a run bearing it is accepted as ours.+ //+ // Timing alone cannot do this. An earlier version took the earliest run created after a+ // snapshot of the newest id, and two callers that snapshot before either dispatch lands then+ // both pick the first run to appear: one judges the other's artifacts as its own verdict while+ // its own run goes unobserved (#681).+ correlationID, err := newCorrelationID()+ if err != nil {+ return Result{}, err+ }+ // The newest run id before dispatching is kept as a floor. It is no longer what identifies+ // our run, but a run older than the dispatch can never be ours, and the floor is what lets a+ // timeout report which runs appeared meanwhile without carrying our id.before, err := latestRunID(opts)if err != nil {return Result{}, err@@ -114,12 +144,22 @@ func Run(opts Options) (Result, error) {if opts.ClaudeVersion != "" {args = append(args, "-f", "claude_version="+opts.ClaudeVersion)}+ args = append(args, "-f", correlationInput+"="+correlationID)if out, err := runGH(args...); err != nil {+ // GitHub reads the workflow from the dispatched ref, so a branch cut before the workflow+ // declared the correlation input rejects it. Refused rather than retried without it: an+ // uncorrelated dispatch is exactly the one that can attach to someone else's run.+ if strings.Contains(out, "Unexpected inputs") && strings.Contains(out, correlationInput) {+ return Result{}, fmt.Errorf("dispatch %s: the workflow on ref %s does not declare the "++ "%s input, so its run could not be told apart from a concurrent dispatch's; "++ "merge or rebase onto main so the ref carries the current workflow: %w\n%s",+ Workflow, opts.Ref, correlationInput, err, out)+ }return Result{}, fmt.Errorf("dispatch %s: %w\n%s", Workflow, err, out)}- logf("dispatched %s; waiting for the run to appear", Workflow)+ logf("dispatched %s (correlation id %s); waiting for the run to appear", Workflow, correlationID)- runID, err := awaitNewRun(opts, before, logf)+ runID, err := awaitNewRun(opts, before, correlationID, logf)if err != nil {return Result{}, err}@@ -200,17 +240,43 @@ type ghRun struct {Status string `json:"status"`Conclusion string `json:"conclusion"`URL string `json:"url"`+ // DisplayTitle is the run's rendered run-name, which carries the correlation id.+ DisplayTitle string `json:"displayTitle"`+}++// newCorrelationID returns an id no other dispatch will carry.+//+// Random rather than derived from the time or the process: two callers on different machines can+// start in the same instant, and the id has to differ precisely then.+func newCorrelationID() (string, error) {+ var b [12]byte+ if _, err := rand.Read(b[:]); err != nil {+ return "", fmt.Errorf("generate a dispatch correlation id: %w", err)+ }+ return hex.EncodeToString(b[:]), nil+}++// correlationMarker is how the workflow's run-name renders a correlation id in the run's title.+// Bracketed so the match is on the whole id, never on a fragment of a longer word.+func correlationMarker(id string) string {+ return "[" + id + "]"}// listRuns returns the most recent runs of this workflow, newest first.func listRuns(opts Options, limit int) ([]ghRun, error) {args := []string{"run", "list", "--workflow", Workflow,- "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url"}+ "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url,displayTitle"}if opts.Repo != "" {args = append(args, "--repo", opts.Repo)}out, err := runGH(args...)if err != nil {+ // displayTitle is what carries the correlation id, and a gh too old to know the field+ // rejects the whole listing. Said plainly, because gh's own message does not suggest it.+ if strings.Contains(out, "Unknown JSON field") && strings.Contains(out, "displayTitle") {+ return nil, fmt.Errorf("list runs of %s: this gh does not support the displayTitle "++ "field that identifies a dispatch's run; upgrade gh: %w\n%s", Workflow, err, out)+ }return nil, fmt.Errorf("list runs of %s: %w\n%s", Workflow, err, out)}var runs []ghRun@@ -235,26 +301,27 @@ func latestRunID(opts Options) (int64, error) {return maxRunID(runs), nil}-// awaitNewRun waits for a run created after the dispatch.+// awaitNewRun waits for the run this dispatch created, identified by its correlation id.//-// Compared numerically, and that is the whole correctness of this function. GitHub run ids-// increase monotonically, so "created after ours" is exactly "id greater than the newest id that-// existed before we dispatched".+// A run is ours only if its title carries our id; being new is necessary but not sufficient. Two+// earlier rules each produced a confident wrong answer rather than an error://-// An earlier version accepted the first listed run whose id merely *differed* from that one, which-// is wrong in a way that produces a confident wrong answer rather than an error: the listing is-// newest-first, the freshly dispatched run usually does not exist yet on the first poll, so the-// second-newest *historical* run satisfied "differs" and was returned. The dispatch then waited on,-// downloaded and judged a previous run's artifacts and reported that stale verdict as this run's-// result -- the precise confusion this function exists to prevent. Reported by Cursor Bugbot.+// - Accepting the first listed run whose id merely differed from the pre-dispatch newest. The+// listing is newest-first and the fresh run usually does not exist on the first poll, so a+// *historical* run was returned and its stale verdict reported as this run's. Reported by+// Cursor Bugbot. The floor below still rules that out.+// - Accepting the earliest run above that floor. Two callers that snapshot the same floor before+// either dispatch lands both select the first new run, so one of them judges the other's+// artifacts while its own run is never matched (#681).//-// The smallest qualifying id is taken rather than the newest, so that if another dispatch lands-// while we poll, we follow the earlier one -- ours.-func awaitNewRun(opts Options, before int64, logf func(string, ...any)) (string, error) {- deadline := time.Now().Add(5 * time.Minute)+// Runs that appear without our id are someone else's and are waited past, not taken. If our id+// never shows up, the error names them rather than guessing which, if any, was ours.+func awaitNewRun(opts Options, before int64, correlationID string, logf func(string, ...any)) (string, error) {+ deadline := time.Now().Add(opts.newRunTimeout)var lastErr error+ var others []int64for time.Now().Before(deadline) {- runs, err := listRuns(opts, 10)+ runs, err := listRuns(opts, 20)if err != nil {// Transient rather than fatal. A single failed API call used to abandon a run that was// proceeding perfectly well, and the dispatch reported failure while the workflow it had@@ -262,38 +329,85 @@ func awaitNewRun(opts Options, before int64, logf func(string, ...any)) (string,// error kept so a persistent outage still reports the real cause rather than a timeout.lastErr = errlogf("could not list runs (%v); retrying", firstLine(err.Error()))- time.Sleep(5 * time.Second)+ time.Sleep(opts.newRunPoll)continue}lastErr = nil- if found := pickNewRun(runs, before); found > 0 {+ found, err := pickOwnRun(runs, before, correlationID)+ if err != nil {+ return "", err+ }+ if found > 0 {return fmt.Sprint(found), nil}- time.Sleep(5 * time.Second)+ others = newerRuns(runs, before)+ time.Sleep(opts.newRunPoll)}if lastErr != nil {- return "", fmt.Errorf("could not reach the GitHub API while waiting for a new %s run: %w",- Workflow, lastErr)- }- return "", fmt.Errorf("no new %s run appeared within 5 minutes of dispatching; check "+- "`gh run list --workflow %s`", Workflow, Workflow)+ return "", fmt.Errorf("could not reach the GitHub API while waiting for the %s run with "++ "correlation id %s: %w", Workflow, correlationID, lastErr)+ }+ if len(others) > 0 {+ // Explicitly ambiguous rather than resolved by timing. One of these may be ours with a+ // title the workflow failed to stamp, or all of them may be other dispatches while ours is+ // still not listed; nothing here can tell which, and picking one is how a caller ends up+ // judging another invocation's artifacts.+ return "", fmt.Errorf("no %s run carrying correlation id %s appeared within %s of "++ "dispatching, but %d other run(s) did (%s); refusing to guess which, if any, is this "++ "dispatch's. Check `gh run list --workflow %s` and that the dispatched ref's workflow "++ "puts inputs.%s in its run-name", Workflow, correlationID, opts.newRunTimeout,+ len(others), joinIDs(others), Workflow, correlationInput)+ }+ return "", fmt.Errorf("no new %s run appeared within %s of dispatching; check "++ "`gh run list --workflow %s`", Workflow, opts.newRunTimeout, Workflow)}-// pickNewRun returns the earliest run created after the floor, or 0 if none has appeared.+// pickOwnRun returns the run created after the floor whose title carries the correlation id, or+// 0 if it has not appeared yet.//-// Separated from the polling loop so the selection rule -- the part that was wrong -- is testable-// without a GitHub account or a live workflow.-func pickNewRun(runs []ghRun, before int64) int64 {- var found int64+// Separated from the polling loop so the selection rule -- the part that was wrong, twice -- is+// testable without a GitHub account or a live workflow. More than one run carrying the id is+// reported rather than resolved: the id is random per dispatch, so that means something other+// than this process used it, and either run could be the one whose verdict we would report.+func pickOwnRun(runs []ghRun, before int64, correlationID string) (int64, error) {+ marker := correlationMarker(correlationID)+ var matches []int64for _, r := range runs {- if r.DatabaseID <= before {+ if r.DatabaseID <= before || !strings.Contains(r.DisplayTitle, marker) {continue}- if found == 0 || r.DatabaseID < found {- found = r.DatabaseID+ matches = append(matches, r.DatabaseID)+ }+ switch len(matches) {+ case 0:+ return 0, nil+ case 1:+ return matches[0], nil+ }+ sort.Slice(matches, func(i, j int) bool { return matches[i] < matches[j] })+ return 0, fmt.Errorf("%d %s runs (%s) carry correlation id %s, which should belong to exactly "++ "one dispatch; refusing to guess which one to judge", len(matches), Workflow,+ joinIDs(matches), correlationID)+}++// newerRuns lists the runs created after the floor, oldest first, for an ambiguity report.+func newerRuns(runs []ghRun, before int64) []int64 {+ var ids []int64+ for _, r := range runs {+ if r.DatabaseID > before {+ ids = append(ids, r.DatabaseID)}}- return found+ sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] })+ return ids+}++func joinIDs(ids []int64) string {+ parts := make([]string, len(ids))+ for i, id := range ids {+ parts[i] = fmt.Sprint(id)+ }+ return strings.Join(parts, ", ")}// maxRunID is the floor "created after our dispatch" is measured against.diff --git a/beacon-sandbox/dispatch/dispatch_test.go b/beacon-sandbox/dispatch/dispatch_test.goindex d1932a12..fd78dca4 100644--- a/beacon-sandbox/dispatch/dispatch_test.go+++ b/beacon-sandbox/dispatch/dispatch_test.go@@ -3,6 +3,7 @@ package dispatchimport ("os""path/filepath"+ "regexp""sort""strings""testing"@@ -68,38 +69,110 @@ func TestFindRunDirsToleratesNothingCollected(t *testing.T) {}}-// The bug this pins is one that produced a confident wrong answer rather than an error: the run-// listing is newest-first, and the freshly dispatched run usually does not exist on the first poll,-// so accepting "the first id that differs from the pre-dispatch id" returned the second-newest-// *historical* run. The dispatch then judged a previous run's artifacts as this run's result.-//-// Selection is pure given a listing, so it is tested directly rather than through gh.-func TestNewRunSelectionRequiresAnIDGreaterThanTheFloor(t *testing.T) {+// Selection is pure given a listing, so it is tested directly rather than through gh. The rules+// it pins each fixed a confident wrong answer: judging a *historical* run (the listing is+// newest-first and the fresh run usually does not exist on the first poll), and judging another+// concurrent invocation's run (#681).+func TestOwnRunSelectionRequiresTheCorrelationID(t *testing.T) {+ const ours = "0123456789abcdef01234567"+ const theirs = "fedcba9876543210fedcba98"+ titled := func(id int64, correlation string) ghRun {+ title := "Windows sandbox"+ if correlation != "" {+ title += " " + correlationMarker(correlation)+ }+ return ghRun{DatabaseID: id, DisplayTitle: title}+ }+ pick := func(t *testing.T, runs []ghRun, before int64) int64 {+ t.Helper()+ got, err := pickOwnRun(runs, before, ours)+ if err != nil {+ t.Fatalf("unexpected error: %v", err)+ }+ return got+ }+// A realistic listing at the moment of dispatch: history only, newest first, none of it ours.- history := []ghRun{{DatabaseID: 500}, {DatabaseID: 499}, {DatabaseID: 498}}- if got := pickNewRun(history, 500); got != 0 {+ history := []ghRun{titled(500, ""), titled(499, ""), titled(498, "")}+ if got := pick(t, history, 500); got != 0 {t.Errorf("with only pre-existing runs the selection must find nothing, got %d", got)}- // The specific regression: 499 differs from 500 and is listed, but predates the dispatch.- if got := pickNewRun(history, 500); got == 499 {- t.Error("an older run must never be selected; that is the stale-verdict bug")- }// Once ours appears it is selected.- withOurs := append([]ghRun{{DatabaseID: 501}}, history...)- if got := pickNewRun(withOurs, 500); got != 501 {- t.Errorf("the run created after the floor must be selected, got %d", got)+ withOurs := append([]ghRun{titled(501, ours)}, history...)+ if got := pick(t, withOurs, 500); got != 501 {+ t.Errorf("the run carrying our id must be selected, got %d", got)+ }++ // #681: another dispatch's run landed first. The old rule took the earliest new run, 501,+ // which is theirs.+ withRace := append([]ghRun{titled(502, ours), titled(501, theirs)}, history...)+ if got := pick(t, withRace, 500); got != 502 {+ t.Errorf("with two new runs ours is the one carrying our id, not the earlier one; got %d", got)+ }++ // Only another invocation's run, or a manual dispatch, has appeared: not ours, so keep waiting.+ foreign := append([]ghRun{titled(502, ""), titled(501, theirs)}, history...)+ if got := pick(t, foreign, 500); got != 0 {+ t.Errorf("a run without our id must never be selected, got %d", got)+ }++ // The marker matches the whole id, never a fragment of a longer one.+ fragment := []ghRun{{DatabaseID: 501, DisplayTitle: "Windows sandbox [" + ours + "ff]"},+ {DatabaseID: 502, DisplayTitle: "Windows sandbox [ff" + ours + "]"}}+ if got := pick(t, fragment, 500); got != 0 {+ t.Errorf("an id embedded in a longer one must not match, got %d", got)}- // Two dispatches racing: follow the earlier new run, which is ours.- withRace := append([]ghRun{{DatabaseID: 502}, {DatabaseID: 501}}, history...)- if got := pickNewRun(withRace, 500); got != 501 {- t.Errorf("with two new runs the earlier one is ours, got %d", got)+ // Carrying our id is not enough on its own: a run at or below the floor predates the dispatch.+ if got := pick(t, []ghRun{titled(500, ours)}, 500); got != 0 {+ t.Errorf("a run at the floor predates the dispatch and must not be selected, got %d", got)}- // A first-ever dispatch has no history, so the floor is zero and any real run qualifies.- if got := pickNewRun([]ghRun{{DatabaseID: 7}}, 0); got != 7 {- t.Errorf("a zero floor must accept the first run, got %d", got)+ // A first-ever dispatch has no history, so the floor is zero and our run qualifies.+ if got := pick(t, []ghRun{titled(7, ours)}, 0); got != 7 {+ t.Errorf("a zero floor must accept our first run, got %d", got)+ }+}++// Two runs carrying one dispatch's random id means something reused it, and either could be the+// run whose verdict would be reported. That is an explicit error, not a choice.+func TestOwnRunSelectionRefusesTwoRunsCarryingTheID(t *testing.T) {+ const ours = "0123456789abcdef01234567"+ runs := []ghRun{+ {DatabaseID: 503, DisplayTitle: "Windows sandbox " + correlationMarker(ours)},+ {DatabaseID: 501, DisplayTitle: "Windows sandbox " + correlationMarker(ours)},+ {DatabaseID: 500, DisplayTitle: "Windows sandbox"},+ }+ got, err := pickOwnRun(runs, 500, ours)+ if err == nil {+ t.Fatalf("two runs carrying one correlation id must be refused, got run %d", got)+ }+ if got != 0 {+ t.Errorf("an ambiguous selection must not also return a run, got %d", got)+ }+ for _, want := range []string{"501", "503", ours, "refusing to guess"} {+ if !strings.Contains(err.Error(), want) {+ t.Errorf("the error should mention %q: %v", want, err)+ }+ }+}++// Each dispatch needs an id no concurrent one shares, in a form that survives a run title.+func TestCorrelationIDsAreUniqueHex(t *testing.T) {+ seen := map[string]bool{}+ for range 1000 {+ id, err := newCorrelationID()+ if err != nil {+ t.Fatal(err)+ }+ if !regexp.MustCompile(`^[0-9a-f]{24}$`).MatchString(id) {+ t.Fatalf("correlation id %q is not 24 lowercase hex characters", id)+ }+ if seen[id] {+ t.Fatalf("correlation id %q repeated", id)+ }+ seen[id] = true}}diff --git a/beacon-sandbox/dispatch/fakegh_test.go b/beacon-sandbox/dispatch/fakegh_test.gonew file mode 100644index 00000000..e48217c8--- /dev/null+++ b/beacon-sandbox/dispatch/fakegh_test.go@@ -0,0 +1,447 @@+package dispatch++// A fake GitHub, reached through a fake `gh`, so Run can be exercised end to end without an+// account, a network, or a real workflow.+//+// Run shells out to `gh` rather than calling the REST API, so the fake has to sit at that seam.+// The test binary doubles as the `gh` executable: it is copied into a temporary directory as `gh`,+// that directory is put first on PATH, and when it is invoked with fakeGHEnv set, TestMain+// forwards its arguments to an httptest server and replays the answer. The server holds the state+// a real repository would -- the run history, which inputs the workflow declares, the display title+// each run gets -- so concurrent callers observe one shared, consistent GitHub, exactly as two+// maintainers dispatching at once would.+//+// Nothing in the production code knows the fake exists. That is deliberate: the race this pins+// (#681) lives in the order of real gh calls, so the test drives those calls rather than a seam+// substituted for them.++import (+ "bytes"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "net/http/httptest"+ "os"+ "path/filepath"+ "runtime"+ "sort"+ "strconv"+ "strings"+ "sync"+ "testing"+)++const fakeGHEnv = "BEACON_SANDBOX_FAKE_GH_URL"++// fakeGHDir holds the one copy of this test binary installed as `gh`, made once per package run+// rather than once per test.+var fakeGHDir string++func TestMain(m *testing.M) {+ if url := os.Getenv(fakeGHEnv); url != "" {+ os.Exit(runFakeGH(url, os.Args[1:]))+ }+ dir, err := installFakeGH()+ if err != nil {+ fmt.Fprintln(os.Stderr, "install fake gh:", err)+ os.Exit(2)+ }+ fakeGHDir = dir+ code := m.Run()+ os.RemoveAll(dir)+ os.Exit(code)+}++// installFakeGH copies this test binary into a fresh directory under the name gh.+func installFakeGH() (string, error) {+ self, err := os.Executable()+ if err != nil {+ return "", err+ }+ dir, err := os.MkdirTemp("", "beacon-sandbox-fake-gh-")+ if err != nil {+ return "", err+ }+ name := "gh"+ if runtime.GOOS == "windows" {+ name += ".exe"+ }+ src, err := os.Open(self)+ if err != nil {+ return "", err+ }+ defer src.Close()+ dst, err := os.OpenFile(filepath.Join(dir, name), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o700)+ if err != nil {+ return "", err+ }+ if _, err := io.Copy(dst, src); err != nil {+ dst.Close()+ return "", err+ }+ return dir, dst.Close()+}++type fakeGHRequest struct {+ Args []string `json:"args"`+}++type fakeGHResponse struct {+ Stdout string `json:"stdout"`+ Exit int `json:"exit"`+ // Files are written under the --dir argument, which is how `gh run download` delivers.+ Files map[string]string `json:"files,omitempty"`+}++// runFakeGH is the whole of the fake executable: forward, replay, exit.+func runFakeGH(url string, args []string) int {+ body, _ := json.Marshal(fakeGHRequest{Args: args})+ resp, err := http.Post(url, "application/json", bytes.NewReader(body))+ if err != nil {+ fmt.Fprintln(os.Stderr, "fake gh:", err)+ return 2+ }+ defer resp.Body.Close()+ var out fakeGHResponse+ if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {+ fmt.Fprintln(os.Stderr, "fake gh: decode:", err)+ return 2+ }+ if len(out.Files) > 0 {+ dir := flagValue(args, "--dir")+ for rel, content := range out.Files {+ p := filepath.Join(dir, filepath.FromSlash(rel))+ if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {+ fmt.Fprintln(os.Stderr, "fake gh:", err)+ return 2+ }+ if err := os.WriteFile(p, []byte(content), 0o600); err != nil {+ fmt.Fprintln(os.Stderr, "fake gh:", err)+ return 2+ }+ }+ }+ fmt.Fprint(os.Stdout, out.Stdout)+ return out.Exit+}++func flagValue(args []string, name string) string {+ for i := 0; i+1 < len(args); i++ {+ if args[i] == name {+ return args[i+1]+ }+ }+ return ""+}++func flagValues(args []string, name string) []string {+ var vals []string+ for i := 0; i+1 < len(args); i++ {+ if args[i] == name {+ vals = append(vals, args[i+1])+ }+ }+ return vals+}++// fakeRun is one workflow run as the fake repository records it.+type fakeRun struct {+ ID int64+ Status string+ Conclusion string+ DisplayTitle string+ Inputs map[string]string+ // hiddenFor counts the listings the run is still invisible to, standing in for the delay+ // between a dispatch being accepted and its run appearing in the API.+ hiddenFor int+ // listsUntilDone counts the listings that report it in progress before it completes.+ listsUntilDone int+}++// fakeGitHub is the shared state behind every fake gh invocation.+type fakeGitHub struct {+ mu sync.Mutex+ nextID int64+ runs []*fakeRun+ // declared is the set of workflow_dispatch inputs the dispatched workflow accepts. GitHub+ // rejects any other input with a 422, which is what dispatching a ref whose workflow predates+ // an input looks like.+ declared map[string]bool+ // dispatched records each accepted dispatch, in the order GitHub assigned ids.+ dispatched []*fakeRun+ foreign []int64++ // Hooks run outside the lock, so a test can hold one caller at a point in its sequence until+ // another caller catches up -- which is how an interleaving is forced rather than hoped for.+ beforeList func()+ beforeDispatch func()+ afterDispatch func(run *fakeRun)++ // hideNewRunsFor and runListsUntilDone shape every run created by a dispatch.+ hideNewRunsFor int+ runListsUntilDone int+ // noRunName models a dispatched ref whose workflow accepts the correlation input but does not+ // render it into run-name, so every run is titled with the bare workflow name.+ noRunName bool+ // failLists makes that many listings fail the way a network blip does.+ failLists int+ // oldGH models a gh release that predates the displayTitle JSON field.+ oldGH bool+}++// newFakeGitHub starts the fake and puts a `gh` that talks to it first on PATH.+//+// configure, if not nil, shapes the fake before its server starts: the hooks and knobs are read+// by the server's goroutines, and setting them afterwards would be a data race.+//+// The history is a few completed runs so that the floor Run snapshots is a real, non-zero id --+// the situation the race needs, since both callers read the same floor. By default the fake+// accepts exactly the inputs this checkout's workflow declares.+func newFakeGitHub(t *testing.T, configure func(*fakeGitHub)) *fakeGitHub {+ t.Helper()+ f := &fakeGitHub{nextID: 501, declared: map[string]bool{}}+ for _, in := range workflowInputs(t) {+ f.declared[in] = true+ }+ for id := int64(498); id <= 500; id++ {+ f.runs = append(f.runs, &fakeRun{ID: id, Status: "completed", Conclusion: "success",+ DisplayTitle: "Windows sandbox"})+ }+ if configure != nil {+ configure(f)+ }+ srv := httptest.NewServer(http.HandlerFunc(f.serve))+ t.Cleanup(srv.Close)++ t.Setenv("PATH", fakeGHDir+string(os.PathListSeparator)+os.Getenv("PATH"))+ t.Setenv(fakeGHEnv, srv.URL)+ // Under -race every gh invocation is a race-enabled copy of this binary, and those sleep a+ // second on exit by default. Harmless, but it turns a sub-second test into a slow one.+ t.Setenv("GORACE", strings.TrimSpace(os.Getenv("GORACE")+" atexit_sleep_ms=0"))+ return f+}++// workflowInputs reads the inputs the real workflow declares, so the fake accepts exactly what+// GitHub would for a ref carrying this checkout's workflow.+func workflowInputs(t *testing.T) []string {+ t.Helper()+ wf := loadWorkflow(t)+ var names []string+ for name := range wf.On.WorkflowDispatch.Inputs {+ names = append(names, name)+ }+ sort.Strings(names)+ return names+}++func (f *fakeGitHub) serve(w http.ResponseWriter, r *http.Request) {+ var req fakeGHRequest+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {+ http.Error(w, err.Error(), http.StatusBadRequest)+ return+ }+ _ = json.NewEncoder(w).Encode(f.handle(req.Args))+}++func (f *fakeGitHub) handle(args []string) fakeGHResponse {+ switch {+ case len(args) >= 2 && args[0] == "auth" && args[1] == "status":+ return fakeGHResponse{Stdout: "Logged in to github.com as fake\n"}+ case len(args) >= 3 && args[0] == "workflow" && args[1] == "run":+ return f.dispatch(args)+ case len(args) >= 2 && args[0] == "run" && args[1] == "list":+ return f.list(args)+ case len(args) >= 3 && args[0] == "run" && args[1] == "download":+ return f.download(args)+ }+ return fakeGHResponse{Stdout: fmt.Sprintf("fake gh: unsupported command %q\n", args), Exit: 1}+}++func (f *fakeGitHub) dispatch(args []string) fakeGHResponse {+ if args[2] != Workflow {+ return fakeGHResponse{Stdout: "could not find workflow " + args[2] + "\n", Exit: 1}+ }+ inputs := map[string]string{}+ var unexpected []string+ for _, kv := range flagValues(args, "-f") {+ k, v, _ := strings.Cut(kv, "=")+ inputs[k] = v+ if !f.declared[k] {+ unexpected = append(unexpected, k)+ }+ }+ if len(unexpected) > 0 {+ // The shape of GitHub's real refusal, as gh prints it.+ quoted, _ := json.Marshal(unexpected)+ return fakeGHResponse{Exit: 1, Stdout: "could not create workflow dispatch event: HTTP 422: " ++ "Unexpected inputs provided: " + string(quoted) + "\n"}+ }+ if f.beforeDispatch != nil {+ f.beforeDispatch()+ }+ f.mu.Lock()+ run := &fakeRun{+ ID: f.nextID,+ Status: "queued",+ DisplayTitle: renderRunName(f.declared, inputs, f.noRunName),+ Inputs: inputs,+ hiddenFor: f.hideNewRunsFor,+ listsUntilDone: f.runListsUntilDone,+ }+ f.nextID+++ f.runs = append(f.runs, run)+ f.dispatched = append(f.dispatched, run)+ f.mu.Unlock()+ if f.afterDispatch != nil {+ f.afterDispatch(run)+ }+ return fakeGHResponse{Stdout: "Created workflow_dispatch event for " + Workflow + "\n"}+}++// addForeignRun records a run this process did not dispatch: another maintainer's, or a manual+// dispatch from the Actions tab.+func (f *fakeGitHub) addForeignRun(title string) int64 {+ f.mu.Lock()+ defer f.mu.Unlock()+ run := &fakeRun{ID: f.nextID, Status: "in_progress", DisplayTitle: title}+ f.nextID+++ f.runs = append(f.runs, run)+ f.foreign = append(f.foreign, run.ID)+ return run.ID+}++// foreignRuns reports the ids addForeignRun created. Read under the lock because the hooks that+// add them run on the server's goroutine.+func (f *fakeGitHub) foreignRuns() []int64 {+ f.mu.Lock()+ defer f.mu.Unlock()+ return append([]int64(nil), f.foreign...)+}++// renderRunName evaluates the workflow's run-name the way Actions would for these inputs. It only+// applies when the workflow declares the correlation input; a ref whose workflow does not has no+// run-name, and GitHub then titles a dispatched run with the workflow's name.+func renderRunName(declared map[string]bool, inputs map[string]string, noRunName bool) string {+ if !noRunName && declared[correlationInput] && inputs[correlationInput] != "" {+ return "Windows sandbox [" + inputs[correlationInput] + "]"+ }+ return "Windows sandbox"+}++func (f *fakeGitHub) list(args []string) fakeGHResponse {+ if flagValue(args, "--workflow") != Workflow {+ return fakeGHResponse{Stdout: "fake gh: wrong workflow\n", Exit: 1}+ }+ limit, err := strconv.Atoi(flagValue(args, "--limit"))+ if err != nil {+ return fakeGHResponse{Stdout: "fake gh: bad --limit\n", Exit: 1}+ }+ fields := strings.Split(flagValue(args, "--json"), ",")+ if f.beforeList != nil {+ f.beforeList()+ }+ f.mu.Lock()+ defer f.mu.Unlock()+ if f.failLists > 0 {+ f.failLists--+ return fakeGHResponse{Exit: 1, Stdout: "error connecting to api.github.com\ncheck your internet connection\n"}+ }+ var visible []*fakeRun+ for _, r := range f.runs {+ if r.hiddenFor > 0 {+ r.hiddenFor--+ continue+ }+ visible = append(visible, r)+ }+ sort.Slice(visible, func(i, j int) bool { return visible[i].ID > visible[j].ID })+ if len(visible) > limit {+ visible = visible[:limit]+ }+ out := make([]map[string]any, 0, len(visible))+ for _, r := range visible {+ // Advanced after being reported, so a caller sees each state at least once.+ row := map[string]any{}+ for _, field := range fields {+ switch field {+ case "databaseId":+ row[field] = r.ID+ case "status":+ row[field] = r.Status+ case "conclusion":+ row[field] = r.Conclusion+ case "url":+ row[field] = fmt.Sprintf("https://github.com/fake/fake/actions/runs/%d", r.ID)+ case "displayTitle":+ if f.oldGH {+ return fakeGHResponse{Stdout: "Unknown JSON field: \"displayTitle\"\n", Exit: 1}+ }+ row[field] = r.DisplayTitle+ default:+ // gh rejects an unknown --json field outright; so does the fake, so a typo in the+ // production field list cannot pass here and fail against the real gh.+ return fakeGHResponse{Stdout: "Unknown JSON field: " + field + "\n", Exit: 1}+ }+ }+ out = append(out, row)+ if r.Status != "completed" && r.Inputs != nil {+ if r.listsUntilDone > 0 {+ r.listsUntilDone--+ r.Status = "in_progress"+ } else {+ r.Status, r.Conclusion = "completed", "success"+ }+ }+ }+ body, _ := json.Marshal(out)+ return fakeGHResponse{Stdout: string(body)}+}++func (f *fakeGitHub) download(args []string) fakeGHResponse {+ id, err := strconv.ParseInt(args[2], 10, 64)+ if err != nil {+ return fakeGHResponse{Stdout: "fake gh: bad run id\n", Exit: 1}+ }+ f.mu.Lock()+ defer f.mu.Unlock()+ for _, r := range f.runs {+ if r.ID != id || r.Inputs == nil {+ continue+ }+ // Named the way the harness names a run directory, and carrying which dispatch produced+ // it, so a test can tell whose artifacts a caller brought home.+ dir := fmt.Sprintf("beacon-sandbox-runs/%s-%d", r.Inputs["scenario"], r.ID)+ return fakeGHResponse{Stdout: "", Files: map[string]string{+ dir + "/runtime.jsonl": fmt.Sprintf(`{"run":%d,"scenario":%q}`+"\n", r.ID, r.Inputs["scenario"]),+ }}+ }+ return fakeGHResponse{Stdout: "no artifacts found for run\n", Exit: 1}+}++// runFor reports the run GitHub created for the dispatch of this scenario.+func (f *fakeGitHub) runFor(scenario string) int64 {+ f.mu.Lock()+ defer f.mu.Unlock()+ for _, r := range f.dispatched {+ if r.Inputs["scenario"] == scenario {+ return r.ID+ }+ }+ return 0+}++// dispatches returns a copy of each accepted dispatch's inputs, in id order.+func (f *fakeGitHub) dispatches() []map[string]string {+ f.mu.Lock()+ defer f.mu.Unlock()+ out := make([]map[string]string, len(f.dispatched))+ for i, r := range f.dispatched {+ cp := map[string]string{}+ for k, v := range r.Inputs {+ cp[k] = v+ }+ out[i] = cp+ }+ return out+}diff --git a/beacon-sandbox/dispatch/run_test.go b/beacon-sandbox/dispatch/run_test.gonew file mode 100644index 00000000..4c9a0e7a--- /dev/null+++ b/beacon-sandbox/dispatch/run_test.go@@ -0,0 +1,337 @@+package dispatch++import (+ "os"+ "path/filepath"+ "regexp"+ "strconv"+ "strings"+ "sync"+ "sync/atomic"+ "testing"+ "time"+)++// waitOrFail blocks on ch, failing loudly rather than hanging the suite if the interleaving a test+// forces never happens.+func waitOrFail(t *testing.T, ch <-chan struct{}, what string) {+ t.Helper()+ select {+ case <-ch:+ case <-time.After(20 * time.Second):+ t.Errorf("timed out waiting for %s", what)+ }+}++// #681. Two callers dispatch at nearly the same time. Both snapshot the same floor before either+// dispatch lands, then both runs become visible before either caller looks for its own. Selecting+// "the smallest id above the floor" hands both callers the same run: one judges the other's+// artifacts, and the other's own run is never matched.+//+// The interleaving is forced, not hoped for. The fake GitHub holds each dispatch until both+// snapshots are taken, and holds every post-dispatch listing until both runs exist -- the exact+// window the issue traces. Each caller must come back with the run its own dispatch created and+// the artifacts that run produced, and never the other's.+func TestConcurrentDispatchesEachFollowTheirOwnRun(t *testing.T) {+ var snapshots, dispatches atomic.Int32+ snapshotsTaken := make(chan struct{})+ bothDispatched := make(chan struct{})+ var closeSnap, closeDisp sync.Once+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.beforeList = func() {+ if dispatches.Load() == 0 {+ // A listing before any dispatch has completed is a caller taking its floor.+ if snapshots.Add(1) == 2 {+ closeSnap.Do(func() { close(snapshotsTaken) })+ }+ return+ }+ waitOrFail(t, bothDispatched, "both dispatches to land")+ }+ gh.beforeDispatch = func() { waitOrFail(t, snapshotsTaken, "both callers to snapshot the floor") }+ gh.afterDispatch = func(*fakeRun) {+ if dispatches.Add(1) == 2 {+ closeDisp.Do(func() { close(bothDispatched) })+ }+ }+ })++ scenarios := []string{"w00-probe", "w03-hook-capture"}+ results := make([]Result, len(scenarios))+ errs := make([]error, len(scenarios))+ var wg sync.WaitGroup+ for i, sc := range scenarios {+ wg.Add(1)+ go func() {+ defer wg.Done()+ results[i], errs[i] = Run(fastOptions(t, sc))+ }()+ }+ wg.Wait()++ for i, sc := range scenarios {+ if errs[i] != nil {+ t.Fatalf("caller %s: %v", sc, errs[i])+ }+ want := gh.runFor(sc)+ if want == 0 {+ t.Fatalf("the fake never saw a dispatch for %s", sc)+ }+ if results[i].RunID != strconv.FormatInt(want, 10) {+ t.Errorf("caller %s followed run %s, but its own dispatch created run %d", sc,+ results[i].RunID, want)+ }+ assertOwnArtifacts(t, results[i], sc)+ }+ if results[0].RunID == results[1].RunID {+ t.Errorf("both callers attached to run %s; one of them judged the other's artifacts",+ results[0].RunID)+ }+}++// assertOwnArtifacts checks the caller brought home exactly the run directory its own dispatch+// produced. This is the consequence that matters: a wrong run id alone is a bookkeeping error,+// but judging another invocation's artifacts is a wrong verdict.+func assertOwnArtifacts(t *testing.T, res Result, scenario string) {+ t.Helper()+ if len(res.RunDirs) != 1 {+ t.Errorf("caller %s collected %d run dirs, want 1: %v", scenario, len(res.RunDirs), res.RunDirs)+ return+ }+ if got, want := filepath.Base(res.RunDirs[0]), scenario+"-"+res.RunID; got != want {+ t.Errorf("caller %s collected %s, want its own run's %s", scenario, got, want)+ }+ data, err := os.ReadFile(filepath.Join(res.RunDirs[0], "runtime.jsonl"))+ if err != nil {+ t.Errorf("caller %s: %v", scenario, err)+ return+ }+ if want := `"scenario":"` + scenario + `"`; !strings.Contains(string(data), want) {+ t.Errorf("caller %s judged another dispatch's log: %s", scenario, data)+ }+}++func fastOptions(t *testing.T, scenario string) Options {+ return Options{+ Repo: "fake/fake",+ Ref: "work",+ Scenario: scenario,+ ClaudeVersion: "2.1.220",+ OutDir: t.TempDir(),+ PollInterval: 10 * time.Millisecond,+ Timeout: 10 * time.Second,+ newRunPoll: 10 * time.Millisecond,+ newRunTimeout: 10 * time.Second,+ }+}++// The single-caller path, which every dispatch took before #681 and must still take: the run is+// not listed for the first few polls, then runs through its states, and the caller comes back+// with it and its artifacts. Also pins what the dispatch sends: the correlation id rides as a+// workflow input alongside the existing ones.+func TestSingleDispatchFollowsItsRunToCompletion(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.hideNewRunsFor = 3+ gh.runListsUntilDone = 3+ })++ res, err := Run(fastOptions(t, "w00-probe"))+ if err != nil {+ t.Fatalf("Run: %v", err)+ }+ if want := strconv.FormatInt(gh.runFor("w00-probe"), 10); res.RunID != want {+ t.Errorf("followed run %s, want %s", res.RunID, want)+ }+ if res.Conclusion != "success" {+ t.Errorf("conclusion = %q, want success", res.Conclusion)+ }+ if res.URL != "https://github.com/fake/fake/actions/runs/"+res.RunID {+ t.Errorf("URL = %q", res.URL)+ }+ assertOwnArtifacts(t, res, "w00-probe")++ sent := gh.dispatches()+ if len(sent) != 1 {+ t.Fatalf("expected one dispatch, got %d", len(sent))+ }+ if sent[0]["scenario"] != "w00-probe" || sent[0]["claude_version"] != "2.1.220" {+ t.Errorf("the existing inputs must still be sent unchanged, got %v", sent[0])+ }+ if !regexp.MustCompile(`^[0-9a-f]{24}$`).MatchString(sent[0][correlationInput]) {+ t.Errorf("the dispatch must carry a correlation id, got %q", sent[0][correlationInput])+ }+}++// A run that appears first but is not ours -- another invocation's, or a manual dispatch from the+// Actions tab -- is waited past, not taken. Sequential counterpart of the race: the foreign runs+// are the earliest above the floor, which is exactly what the old rule selected.+func TestDispatchWaitsPastRunsThatAreNotItsOwn(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.hideNewRunsFor = 2+ gh.beforeDispatch = func() {+ gh.addForeignRun("Windows sandbox " + correlationMarker("fedcba9876543210fedcba98"))+ gh.addForeignRun("Windows sandbox")+ }+ })++ res, err := Run(fastOptions(t, "w03-hook-capture"))+ if err != nil {+ t.Fatalf("Run: %v", err)+ }+ foreign := gh.foreignRuns()+ if len(foreign) != 2 {+ t.Fatalf("the fake should have added two foreign runs, got %v", foreign)+ }+ for _, id := range foreign {+ if res.RunID == strconv.FormatInt(id, 10) {+ t.Fatalf("attached to foreign run %d instead of our own", id)+ }+ }+ if want := strconv.FormatInt(gh.runFor("w03-hook-capture"), 10); res.RunID != want {+ t.Errorf("followed run %s, want our own %s", res.RunID, want)+ }+ assertOwnArtifacts(t, res, "w03-hook-capture")+}++// If our id never shows up but other runs did, the dispatch cannot know which, if any, is ours --+// for instance the dispatched ref's workflow accepts the input but does not render it into its+// title. That must be an explicit ambiguity error naming the runs, and nothing may be downloaded+// or judged.+func TestDispatchReportsAmbiguityWhenNoRunCarriesItsID(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.noRunName = true+ gh.beforeDispatch = func() { gh.addForeignRun("Windows sandbox") }+ })++ opts := fastOptions(t, "w00-probe")+ opts.newRunTimeout = 300 * time.Millisecond+ res, err := Run(opts)+ if err == nil {+ t.Fatalf("an uncorrelatable run must be an error, got run %s", res.RunID)+ }+ ours := strconv.FormatInt(gh.runFor("w00-probe"), 10)+ foreign := gh.foreignRuns()+ if len(foreign) != 1 {+ t.Fatalf("the fake should have added one foreign run, got %v", foreign)+ }+ for _, want := range []string{"refusing to guess", ours, strconv.FormatInt(foreign[0], 10),+ correlationInput, "run-name"} {+ if !strings.Contains(err.Error(), want) {+ t.Errorf("the ambiguity error should mention %q: %v", want, err)+ }+ }+ if res.RunID != "" || len(res.RunDirs) != 0 {+ t.Errorf("nothing may be selected or collected on ambiguity, got %+v", res)+ }+ assertNothingCollected(t, opts.OutDir)+}++// No run at all within the wait is the plain timeout it always was, not an ambiguity.+func TestDispatchTimesOutWhenNoRunAppears(t *testing.T) {+ newFakeGitHub(t, func(gh *fakeGitHub) { gh.hideNewRunsFor = 1 << 30 })++ opts := fastOptions(t, "w00-probe")+ opts.newRunTimeout = 200 * time.Millisecond+ _, err := Run(opts)+ if err == nil {+ t.Fatal("a run that never appears must be an error")+ }+ if !strings.Contains(err.Error(), "no new "+Workflow+" run appeared") {+ t.Errorf("want the no-run timeout, got: %v", err)+ }+ if strings.Contains(err.Error(), "refusing to guess") {+ t.Errorf("nothing appeared, so there is nothing ambiguous: %v", err)+ }+}++// A listing that fails transiently while waiting is retried, as before, rather than abandoning a+// run that is proceeding fine.+func TestDispatchRetriesAFailedListing(t *testing.T) {+ var failed atomic.Bool+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.hideNewRunsFor = 1+ gh.afterDispatch = func(*fakeRun) {+ gh.mu.Lock()+ gh.failLists = 2+ gh.mu.Unlock()+ failed.Store(true)+ }+ })+ res, err := Run(fastOptions(t, "w00-probe"))+ if err != nil {+ t.Fatalf("Run: %v", err)+ }+ if !failed.Load() {+ t.Fatal("the fake never injected the failure")+ }+ gh.mu.Lock()+ left := gh.failLists+ gh.mu.Unlock()+ if left != 0 {+ t.Errorf("%d injected listing failures were never hit, so the retry was not exercised", left)+ }+ assertOwnArtifacts(t, res, "w00-probe")+}++// The completion wait's own timeout still reports the run it was following, so the reader can+// go and look at it.+func TestDispatchReportsARunThatDoesNotComplete(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) { gh.runListsUntilDone = 1 << 30 })++ opts := fastOptions(t, "w00-probe")+ opts.Timeout = 200 * time.Millisecond+ res, err := Run(opts)+ if err == nil || !strings.Contains(err.Error(), "did not complete within") {+ t.Fatalf("want the completion timeout, got: %v", err)+ }+ if want := strconv.FormatInt(gh.runFor("w00-probe"), 10); res.RunID != want {+ t.Errorf("the timeout must still identify our run: got %q, want %s", res.RunID, want)+ }+}++// GitHub reads the workflow from the dispatched ref, so a branch cut before the correlation input+// existed rejects it with a 422. That is refused with the fix, not retried uncorrelated: an+// uncorrelated dispatch is precisely the one that can attach to someone else's run.+func TestDispatchRefusesARefWhoseWorkflowLacksTheCorrelationInput(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {+ gh.declared = map[string]bool{"scenario": true, "claude_version": true}+ })++ _, err := Run(fastOptions(t, "w00-probe"))+ if err == nil {+ t.Fatal("dispatching a ref without the correlation input must fail")+ }+ for _, want := range []string{correlationInput, "work", "rebase"} {+ if !strings.Contains(err.Error(), want) {+ t.Errorf("the error should mention %q: %v", want, err)+ }+ }+ if n := len(gh.dispatches()); n != 0 {+ t.Errorf("no uncorrelated fallback dispatch may be made, got %d", n)+ }+}++// displayTitle is what carries the correlation id, so a gh too old to report it cannot identify a+// run at all. That must fail before anything is dispatched, and say what to do.+func TestDispatchExplainsAGHTooOldForDisplayTitle(t *testing.T) {+ gh := newFakeGitHub(t, func(gh *fakeGitHub) { gh.oldGH = true })++ _, err := Run(fastOptions(t, "w00-probe"))+ if err == nil || !strings.Contains(err.Error(), "upgrade gh") {+ t.Fatalf("want an upgrade-gh error, got: %v", err)+ }+ if n := len(gh.dispatches()); n != 0 {+ t.Errorf("nothing may be dispatched when the run could never be identified, got %d", n)+ }+}++func assertNothingCollected(t *testing.T, outDir string) {+ t.Helper()+ entries, err := os.ReadDir(outDir)+ if err != nil {+ t.Fatal(err)+ }+ for _, e := range entries {+ t.Errorf("nothing should have been collected, found %s", e.Name())+ }+}diff --git a/beacon-sandbox/dispatch/workflow_test.go b/beacon-sandbox/dispatch/workflow_test.gonew file mode 100644index 00000000..1da28c6e--- /dev/null+++ b/beacon-sandbox/dispatch/workflow_test.go@@ -0,0 +1,132 @@+package dispatch++import (+ "os"+ "path/filepath"+ "regexp"+ "strings"+ "testing"++ "gopkg.in/yaml.v3"+)++type workflowInput struct {+ Description string `yaml:"description"`+ Required bool `yaml:"required"`+ Default *string `yaml:"default"`+ Type string `yaml:"type"`+}++type workflowStep struct {+ Name string `yaml:"name"`+ Run string `yaml:"run"`+ Env map[string]string `yaml:"env"`+ With map[string]string `yaml:"with"`+}++type workflowFile struct {+ Name string `yaml:"name"`+ RunName string `yaml:"run-name"`+ On struct {+ WorkflowDispatch struct {+ Inputs map[string]workflowInput `yaml:"inputs"`+ } `yaml:"workflow_dispatch"`+ } `yaml:"on"`+ Jobs map[string]struct {+ Env map[string]string `yaml:"env"`+ Steps []workflowStep `yaml:"steps"`+ } `yaml:"jobs"`+}++// loadWorkflow parses the workflow this package dispatches, from this checkout.+func loadWorkflow(t *testing.T) workflowFile {+ t.Helper()+ data, err := os.ReadFile(filepath.Join("..", "..", ".github", "workflows", Workflow))+ if err != nil {+ t.Fatal(err)+ }+ var wf workflowFile+ if err := yaml.Unmarshal(data, &wf); err != nil {+ t.Fatalf("parse %s: %v", Workflow, err)+ }+ return wf+}++// The dispatch only works if the workflow declares the correlation input and renders it into the+// run's title in the form pickOwnRun matches. Both sides live in different files, so this pins+// them together: change one without the other and this fails here rather than as a dispatch that+// never finds its run.+func TestWorkflowStampsTheCorrelationIDIntoTheRunName(t *testing.T) {+ wf := loadWorkflow(t)++ in, ok := wf.On.WorkflowDispatch.Inputs[correlationInput]+ if !ok {+ t.Fatalf("%s must declare the %s workflow_dispatch input", Workflow, correlationInput)+ }+ // Optional with an empty default, so a manual dispatch from the Actions tab still works and+ // keeps the plain title.+ if in.Required {+ t.Errorf("%s must be optional so manual dispatches keep working", correlationInput)+ }+ if in.Type != "string" {+ t.Errorf("%s type = %q, want string", correlationInput, in.Type)+ }+ if in.Default == nil || *in.Default != "" {+ t.Errorf("%s must default to empty", correlationInput)+ }++ // The run-name's formatted branch, rendered for an id, must produce exactly the marker the+ // selection looks for; its fallback must be the workflow name, which is what a run without a+ // run-name is titled and what a manual dispatch therefore still shows.+ m := regexp.MustCompile(`^\$\{\{\s*inputs\.` + correlationInput ++ `\s*&&\s*format\('([^']*)',\s*inputs\.` + correlationInput + `\)\s*\|\|\s*'([^']*)'\s*\}\}$`).+ FindStringSubmatch(strings.TrimSpace(wf.RunName))+ if m == nil {+ t.Fatalf("run-name must render inputs.%s with format(...) and fall back to a plain "++ "title; got %q", correlationInput, wf.RunName)+ }+ const id = "0123456789abcdef01234567"+ rendered := strings.ReplaceAll(m[1], "{0}", id)+ if got, err := pickOwnRun([]ghRun{{DatabaseID: 2, DisplayTitle: rendered}}, 1, id); err != nil || got != 2 {+ t.Errorf("a run titled %q must be selected for id %s (got %d, %v)", rendered, id, got, err)+ }+ if want := renderRunName(map[string]bool{correlationInput: true},+ map[string]string{correlationInput: id}, false); rendered != want {+ t.Errorf("the fake GitHub titles runs %q but the workflow renders %q; keep them in step",+ want, rendered)+ }+ if m[2] != wf.Name {+ t.Errorf("run-name fallback = %q, want the workflow name %q", m[2], wf.Name)+ }+}++// The id is only ever rendered into the title. Keeping it out of every script, env block and+// action input means an arbitrary string typed into the dispatch form cannot reach a shell, so the+// input needs none of the validation scenario and claude_version get.+func TestWorkflowNeverPassesTheCorrelationIDToAStep(t *testing.T) {+ wf := loadWorkflow(t)+ if len(wf.Jobs) == 0 {+ t.Fatal("parsed no jobs; the workflow shape changed and this test is no longer checking anything")+ }+ for name, job := range wf.Jobs {+ for k, v := range job.Env {+ if strings.Contains(v, correlationInput) {+ t.Errorf("job %s env %s references %s", name, k, correlationInput)+ }+ }+ for _, step := range job.Steps {+ fields := []string{step.Run}+ for _, v := range step.Env {+ fields = append(fields, v)+ }+ for _, v := range step.With {+ fields = append(fields, v)+ }+ for _, f := range fields {+ if strings.Contains(f, correlationInput) {+ t.Errorf("job %s step %q references %s", name, step.Name, correlationInput)+ }+ }+ }+ }+}diff --git a/docs/contributing/beacon-sandbox.mdx b/docs/contributing/beacon-sandbox.mdxindex 6eb2b15f..87c8e397 100644--- a/docs/contributing/beacon-sandbox.mdx+++ b/docs/contributing/beacon-sandbox.mdx@@ -242,6 +242,8 @@ gh run watchLeave `scenario` empty to run every Windows test. Each takes a few minutes, and the whole set takes about fifteen.+`beacon-sandbox run --provider github --scenario <id>` dispatches the same workflow and brings the artifacts home to judge. It also sends a random `correlation_id` input that the workflow puts in the run's title, so it follows only the run its own dispatch created, even when someone else dispatches at the same moment. A run dispatched by hand, as above, leaves that input empty and keeps the plain title.+| Test | Covers ||------|--------|| `w00-probe` | The `beacon ci exec` collection path. Known to be unreliable on Windows. See [#320](https://github.com/asymptote-labs/agent-beacon/issues/320) |