Skip to content

The cross-harness, self-improving memory layer for AI agents. Join our community: https://discord.com/invite/zdNChS2fBu

fix(sandbox): match each Windows dispatch to its own workflow run (#681)

Claude4271846e287e
diff --git a/.github/workflows/windows-sandbox.yml b/.github/workflows/windows-sandbox.yml
index 1388a1f2..3755defb 100644
--- a/.github/workflows/windows-sandbox.yml
+++ b/.github/workflows/windows-sandbox.yml
@@ -12,6 +12,12 @@
# reachable from untrusted code, so there is no pull_request trigger here and there must never be
# a pull_request_target one.
name: Windows sandbox
+# The correlation id is what lets `beacon-sandbox run --provider github` recognize the run its own
+# dispatch created. `gh workflow run` returns no run id, and two dispatches made moments apart are
+# otherwise indistinguishable: each caller would attach to whichever run appeared first (#681).
+# Manual dispatches leave it empty and keep the plain workflow name as their title. It is only ever
+# rendered into this title, never into a script, so it needs no validation of its own.
+run-name: ${{ inputs.correlation_id && format('Windows sandbox [{0}]', inputs.correlation_id) || 'Windows sandbox' }}
on:
workflow_dispatch:
@@ -25,6 +31,11 @@ on:
required: false
default: "2.1.220"
type: string
+ correlation_id:
+ description: "Set by beacon-sandbox to find this dispatch's run; leave empty when dispatching by hand"
+ required: false
+ default: ""
+ type: string
permissions:
contents: read
diff --git a/beacon-sandbox/AGENTS.md b/beacon-sandbox/AGENTS.md
index bbd26066..0df8eac3 100644
--- a/beacon-sandbox/AGENTS.md
+++ b/beacon-sandbox/AGENTS.md
@@ -218,6 +218,15 @@ it collected; this side downloads it and judges it with the same `check` package
go run ./cmd/beacon-sandbox run --provider github --scenario w00-probe
```
+Concurrent dispatches never mix up their results. Each one sends a random `correlation_id`
+input, the workflow stamps it into the run's title, and the dispatch follows only the run carrying
+it; runs without it are waited past. If none carrying it appears, the dispatch fails with an error
+naming the runs it saw rather than judging one of them. A branch cut before the workflow declared
+that input is refused with a request to rebase onto main. The runs still execute one at a time
+(the workflow's concurrency group), and GitHub keeps only one pending run per group, so a third
+concurrent dispatch cancels the one queued before it; that caller reports its own run as
+`cancelled` with nothing collected, which is a reason to re-run, not a Beacon failure.
+
Four things to tell the user about a Windows run, because none of them are visible in the verdict's
outcome line:
diff --git a/beacon-sandbox/README.md b/beacon-sandbox/README.md
index 72e91e16..bd08f081 100644
--- a/beacon-sandbox/README.md
+++ b/beacon-sandbox/README.md
@@ -87,6 +87,12 @@ environment secret rather than from your shell, so the artifact leak check has n
for and reports *unverified*, exactly as `--modal-secret` does. The in-runner argv scan still runs,
because the key genuinely is present there.
+Several dispatches can run at once without mixing up their results. Each sends a random
+`correlation_id` that the workflow puts in the run's title, and each follows only the run that
+carries its own id, so one maintainer's verdict is never built from another's artifacts. The
+workflow reads that input from the dispatched branch, so a branch cut before it existed needs a
+rebase onto main first.
+
Scenarios say which platform they are written for (`platform: windows`), defaulting to Linux, and
`run` reports how many it set aside rather than silently narrowing the suite.
diff --git a/beacon-sandbox/dispatch/dispatch.go b/beacon-sandbox/dispatch/dispatch.go
index 5376ccb9..673896b6 100644
--- a/beacon-sandbox/dispatch/dispatch.go
+++ b/beacon-sandbox/dispatch/dispatch.go
@@ -13,6 +13,8 @@
package dispatch
import (
+ "crypto/rand"
+ "encoding/hex"
"encoding/json"
"fmt"
"os"
@@ -27,6 +29,11 @@ import (
// Workflow is the workflow file this package drives.
const Workflow = "windows-sandbox.yml"
+// correlationInput is the workflow_dispatch input that ties a run to the dispatch that created it.
+// The workflow puts it in its run-name, so the run's display title carries it from the moment the
+// run exists.
+const correlationInput = "correlation_id"
+
// Options configures a dispatch.
type Options struct {
// Repo is owner/name. Empty lets gh infer it from the checkout.
@@ -51,6 +58,11 @@ type Options struct {
Timeout time.Duration
// Log receives progress lines.
Log func(string, ...any)
+
+ // newRunPoll and newRunTimeout bound the wait for the dispatched run to appear. Zero uses the
+ // defaults; they are unexported because only tests have a reason to shorten them.
+ newRunPoll time.Duration
+ newRunTimeout time.Duration
}
// Result describes a finished dispatch.
@@ -76,6 +88,12 @@ func Run(opts Options) (Result, error) {
// The workflow's own timeout-minutes is 45; allow for queueing on top of it.
opts.Timeout = 60 * time.Minute
}
+ if opts.newRunPoll <= 0 {
+ opts.newRunPoll = 5 * time.Second
+ }
+ if opts.newRunTimeout <= 0 {
+ opts.newRunTimeout = 5 * time.Minute
+ }
if err := requireGH(); err != nil {
return Result{}, err
}
@@ -92,10 +110,22 @@ func Run(opts Options) (Result, error) {
}
logf("dispatching against ref %s", opts.Ref)
- // The newest run id *before* dispatching, so the new one can be identified without guessing.
// `gh workflow run` prints nothing machine-readable and returns no id, which is the whole
- // difficulty here: polling for "the newest run" can otherwise latch onto a previous one that
- // is still in progress and report its result as this dispatch's.
+ // difficulty here: the run this dispatch creates has to be recognized among runs other people
+ // and other invocations create at the same time. So every dispatch carries an id of its own,
+ // the workflow puts it in the run's title, and only a run bearing it is accepted as ours.
+ //
+ // Timing alone cannot do this. An earlier version took the earliest run created after a
+ // snapshot of the newest id, and two callers that snapshot before either dispatch lands then
+ // both pick the first run to appear: one judges the other's artifacts as its own verdict while
+ // its own run goes unobserved (#681).
+ correlationID, err := newCorrelationID()
+ if err != nil {
+ return Result{}, err
+ }
+ // The newest run id before dispatching is kept as a floor. It is no longer what identifies
+ // our run, but a run older than the dispatch can never be ours, and the floor is what lets a
+ // timeout report which runs appeared meanwhile without carrying our id.
before, err := latestRunID(opts)
if err != nil {
return Result{}, err
@@ -114,12 +144,22 @@ func Run(opts Options) (Result, error) {
if opts.ClaudeVersion != "" {
args = append(args, "-f", "claude_version="+opts.ClaudeVersion)
}
+ args = append(args, "-f", correlationInput+"="+correlationID)
if out, err := runGH(args...); err != nil {
+ // GitHub reads the workflow from the dispatched ref, so a branch cut before the workflow
+ // declared the correlation input rejects it. Refused rather than retried without it: an
+ // uncorrelated dispatch is exactly the one that can attach to someone else's run.
+ if strings.Contains(out, "Unexpected inputs") && strings.Contains(out, correlationInput) {
+ return Result{}, fmt.Errorf("dispatch %s: the workflow on ref %s does not declare the "+
+ "%s input, so its run could not be told apart from a concurrent dispatch's; "+
+ "merge or rebase onto main so the ref carries the current workflow: %w\n%s",
+ Workflow, opts.Ref, correlationInput, err, out)
+ }
return Result{}, fmt.Errorf("dispatch %s: %w\n%s", Workflow, err, out)
}
- logf("dispatched %s; waiting for the run to appear", Workflow)
+ logf("dispatched %s (correlation id %s); waiting for the run to appear", Workflow, correlationID)
- runID, err := awaitNewRun(opts, before, logf)
+ runID, err := awaitNewRun(opts, before, correlationID, logf)
if err != nil {
return Result{}, err
}
@@ -200,17 +240,43 @@ type ghRun struct {
Status string `json:"status"`
Conclusion string `json:"conclusion"`
URL string `json:"url"`
+ // DisplayTitle is the run's rendered run-name, which carries the correlation id.
+ DisplayTitle string `json:"displayTitle"`
+}
+
+// newCorrelationID returns an id no other dispatch will carry.
+//
+// Random rather than derived from the time or the process: two callers on different machines can
+// start in the same instant, and the id has to differ precisely then.
+func newCorrelationID() (string, error) {
+ var b [12]byte
+ if _, err := rand.Read(b[:]); err != nil {
+ return "", fmt.Errorf("generate a dispatch correlation id: %w", err)
+ }
+ return hex.EncodeToString(b[:]), nil
+}
+
+// correlationMarker is how the workflow's run-name renders a correlation id in the run's title.
+// Bracketed so the match is on the whole id, never on a fragment of a longer word.
+func correlationMarker(id string) string {
+ return "[" + id + "]"
}
// listRuns returns the most recent runs of this workflow, newest first.
func listRuns(opts Options, limit int) ([]ghRun, error) {
args := []string{"run", "list", "--workflow", Workflow,
- "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url"}
+ "--limit", fmt.Sprint(limit), "--json", "databaseId,status,conclusion,url,displayTitle"}
if opts.Repo != "" {
args = append(args, "--repo", opts.Repo)
}
out, err := runGH(args...)
if err != nil {
+ // displayTitle is what carries the correlation id, and a gh too old to know the field
+ // rejects the whole listing. Said plainly, because gh's own message does not suggest it.
+ if strings.Contains(out, "Unknown JSON field") && strings.Contains(out, "displayTitle") {
+ return nil, fmt.Errorf("list runs of %s: this gh does not support the displayTitle "+
+ "field that identifies a dispatch's run; upgrade gh: %w\n%s", Workflow, err, out)
+ }
return nil, fmt.Errorf("list runs of %s: %w\n%s", Workflow, err, out)
}
var runs []ghRun
@@ -235,26 +301,27 @@ func latestRunID(opts Options) (int64, error) {
return maxRunID(runs), nil
}
-// awaitNewRun waits for a run created after the dispatch.
+// awaitNewRun waits for the run this dispatch created, identified by its correlation id.
//
-// Compared numerically, and that is the whole correctness of this function. GitHub run ids
-// increase monotonically, so "created after ours" is exactly "id greater than the newest id that
-// existed before we dispatched".
+// A run is ours only if its title carries our id; being new is necessary but not sufficient. Two
+// earlier rules each produced a confident wrong answer rather than an error:
//
-// An earlier version accepted the first listed run whose id merely *differed* from that one, which
-// is wrong in a way that produces a confident wrong answer rather than an error: the listing is
-// newest-first, the freshly dispatched run usually does not exist yet on the first poll, so the
-// second-newest *historical* run satisfied "differs" and was returned. The dispatch then waited on,
-// downloaded and judged a previous run's artifacts and reported that stale verdict as this run's
-// result -- the precise confusion this function exists to prevent. Reported by Cursor Bugbot.
+// - Accepting the first listed run whose id merely differed from the pre-dispatch newest. The
+// listing is newest-first and the fresh run usually does not exist on the first poll, so a
+// *historical* run was returned and its stale verdict reported as this run's. Reported by
+// Cursor Bugbot. The floor below still rules that out.
+// - Accepting the earliest run above that floor. Two callers that snapshot the same floor before
+// either dispatch lands both select the first new run, so one of them judges the other's
+// artifacts while its own run is never matched (#681).
//
-// The smallest qualifying id is taken rather than the newest, so that if another dispatch lands
-// while we poll, we follow the earlier one -- ours.
-func awaitNewRun(opts Options, before int64, logf func(string, ...any)) (string, error) {
- deadline := time.Now().Add(5 * time.Minute)
+// Runs that appear without our id are someone else's and are waited past, not taken. If our id
+// never shows up, the error names them rather than guessing which, if any, was ours.
+func awaitNewRun(opts Options, before int64, correlationID string, logf func(string, ...any)) (string, error) {
+ deadline := time.Now().Add(opts.newRunTimeout)
var lastErr error
+ var others []int64
for time.Now().Before(deadline) {
- runs, err := listRuns(opts, 10)
+ runs, err := listRuns(opts, 20)
if err != nil {
// Transient rather than fatal. A single failed API call used to abandon a run that was
// proceeding perfectly well, and the dispatch reported failure while the workflow it had
@@ -262,38 +329,85 @@ func awaitNewRun(opts Options, before int64, logf func(string, ...any)) (string,
// error kept so a persistent outage still reports the real cause rather than a timeout.
lastErr = err
logf("could not list runs (%v); retrying", firstLine(err.Error()))
- time.Sleep(5 * time.Second)
+ time.Sleep(opts.newRunPoll)
continue
}
lastErr = nil
- if found := pickNewRun(runs, before); found > 0 {
+ found, err := pickOwnRun(runs, before, correlationID)
+ if err != nil {
+ return "", err
+ }
+ if found > 0 {
return fmt.Sprint(found), nil
}
- time.Sleep(5 * time.Second)
+ others = newerRuns(runs, before)
+ time.Sleep(opts.newRunPoll)
}
if lastErr != nil {
- return "", fmt.Errorf("could not reach the GitHub API while waiting for a new %s run: %w",
- Workflow, lastErr)
- }
- return "", fmt.Errorf("no new %s run appeared within 5 minutes of dispatching; check "+
- "`gh run list --workflow %s`", Workflow, Workflow)
+ return "", fmt.Errorf("could not reach the GitHub API while waiting for the %s run with "+
+ "correlation id %s: %w", Workflow, correlationID, lastErr)
+ }
+ if len(others) > 0 {
+ // Explicitly ambiguous rather than resolved by timing. One of these may be ours with a
+ // title the workflow failed to stamp, or all of them may be other dispatches while ours is
+ // still not listed; nothing here can tell which, and picking one is how a caller ends up
+ // judging another invocation's artifacts.
+ return "", fmt.Errorf("no %s run carrying correlation id %s appeared within %s of "+
+ "dispatching, but %d other run(s) did (%s); refusing to guess which, if any, is this "+
+ "dispatch's. Check `gh run list --workflow %s` and that the dispatched ref's workflow "+
+ "puts inputs.%s in its run-name", Workflow, correlationID, opts.newRunTimeout,
+ len(others), joinIDs(others), Workflow, correlationInput)
+ }
+ return "", fmt.Errorf("no new %s run appeared within %s of dispatching; check "+
+ "`gh run list --workflow %s`", Workflow, opts.newRunTimeout, Workflow)
}
-// pickNewRun returns the earliest run created after the floor, or 0 if none has appeared.
+// pickOwnRun returns the run created after the floor whose title carries the correlation id, or
+// 0 if it has not appeared yet.
//
-// Separated from the polling loop so the selection rule -- the part that was wrong -- is testable
-// without a GitHub account or a live workflow.
-func pickNewRun(runs []ghRun, before int64) int64 {
- var found int64
+// Separated from the polling loop so the selection rule -- the part that was wrong, twice -- is
+// testable without a GitHub account or a live workflow. More than one run carrying the id is
+// reported rather than resolved: the id is random per dispatch, so that means something other
+// than this process used it, and either run could be the one whose verdict we would report.
+func pickOwnRun(runs []ghRun, before int64, correlationID string) (int64, error) {
+ marker := correlationMarker(correlationID)
+ var matches []int64
for _, r := range runs {
- if r.DatabaseID <= before {
+ if r.DatabaseID <= before || !strings.Contains(r.DisplayTitle, marker) {
continue
}
- if found == 0 || r.DatabaseID < found {
- found = r.DatabaseID
+ matches = append(matches, r.DatabaseID)
+ }
+ switch len(matches) {
+ case 0:
+ return 0, nil
+ case 1:
+ return matches[0], nil
+ }
+ sort.Slice(matches, func(i, j int) bool { return matches[i] < matches[j] })
+ return 0, fmt.Errorf("%d %s runs (%s) carry correlation id %s, which should belong to exactly "+
+ "one dispatch; refusing to guess which one to judge", len(matches), Workflow,
+ joinIDs(matches), correlationID)
+}
+
+// newerRuns lists the runs created after the floor, oldest first, for an ambiguity report.
+func newerRuns(runs []ghRun, before int64) []int64 {
+ var ids []int64
+ for _, r := range runs {
+ if r.DatabaseID > before {
+ ids = append(ids, r.DatabaseID)
}
}
- return found
+ sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] })
+ return ids
+}
+
+func joinIDs(ids []int64) string {
+ parts := make([]string, len(ids))
+ for i, id := range ids {
+ parts[i] = fmt.Sprint(id)
+ }
+ return strings.Join(parts, ", ")
}
// maxRunID is the floor "created after our dispatch" is measured against.
diff --git a/beacon-sandbox/dispatch/dispatch_test.go b/beacon-sandbox/dispatch/dispatch_test.go
index d1932a12..fd78dca4 100644
--- a/beacon-sandbox/dispatch/dispatch_test.go
+++ b/beacon-sandbox/dispatch/dispatch_test.go
@@ -3,6 +3,7 @@ package dispatch
import (
"os"
"path/filepath"
+ "regexp"
"sort"
"strings"
"testing"
@@ -68,38 +69,110 @@ func TestFindRunDirsToleratesNothingCollected(t *testing.T) {
}
}
-// The bug this pins is one that produced a confident wrong answer rather than an error: the run
-// listing is newest-first, and the freshly dispatched run usually does not exist on the first poll,
-// so accepting "the first id that differs from the pre-dispatch id" returned the second-newest
-// *historical* run. The dispatch then judged a previous run's artifacts as this run's result.
-//
-// Selection is pure given a listing, so it is tested directly rather than through gh.
-func TestNewRunSelectionRequiresAnIDGreaterThanTheFloor(t *testing.T) {
+// Selection is pure given a listing, so it is tested directly rather than through gh. The rules
+// it pins each fixed a confident wrong answer: judging a *historical* run (the listing is
+// newest-first and the fresh run usually does not exist on the first poll), and judging another
+// concurrent invocation's run (#681).
+func TestOwnRunSelectionRequiresTheCorrelationID(t *testing.T) {
+ const ours = "0123456789abcdef01234567"
+ const theirs = "fedcba9876543210fedcba98"
+ titled := func(id int64, correlation string) ghRun {
+ title := "Windows sandbox"
+ if correlation != "" {
+ title += " " + correlationMarker(correlation)
+ }
+ return ghRun{DatabaseID: id, DisplayTitle: title}
+ }
+ pick := func(t *testing.T, runs []ghRun, before int64) int64 {
+ t.Helper()
+ got, err := pickOwnRun(runs, before, ours)
+ if err != nil {
+ t.Fatalf("unexpected error: %v", err)
+ }
+ return got
+ }
+
// A realistic listing at the moment of dispatch: history only, newest first, none of it ours.
- history := []ghRun{{DatabaseID: 500}, {DatabaseID: 499}, {DatabaseID: 498}}
- if got := pickNewRun(history, 500); got != 0 {
+ history := []ghRun{titled(500, ""), titled(499, ""), titled(498, "")}
+ if got := pick(t, history, 500); got != 0 {
t.Errorf("with only pre-existing runs the selection must find nothing, got %d", got)
}
- // The specific regression: 499 differs from 500 and is listed, but predates the dispatch.
- if got := pickNewRun(history, 500); got == 499 {
- t.Error("an older run must never be selected; that is the stale-verdict bug")
- }
// Once ours appears it is selected.
- withOurs := append([]ghRun{{DatabaseID: 501}}, history...)
- if got := pickNewRun(withOurs, 500); got != 501 {
- t.Errorf("the run created after the floor must be selected, got %d", got)
+ withOurs := append([]ghRun{titled(501, ours)}, history...)
+ if got := pick(t, withOurs, 500); got != 501 {
+ t.Errorf("the run carrying our id must be selected, got %d", got)
+ }
+
+ // #681: another dispatch's run landed first. The old rule took the earliest new run, 501,
+ // which is theirs.
+ withRace := append([]ghRun{titled(502, ours), titled(501, theirs)}, history...)
+ if got := pick(t, withRace, 500); got != 502 {
+ t.Errorf("with two new runs ours is the one carrying our id, not the earlier one; got %d", got)
+ }
+
+ // Only another invocation's run, or a manual dispatch, has appeared: not ours, so keep waiting.
+ foreign := append([]ghRun{titled(502, ""), titled(501, theirs)}, history...)
+ if got := pick(t, foreign, 500); got != 0 {
+ t.Errorf("a run without our id must never be selected, got %d", got)
+ }
+
+ // The marker matches the whole id, never a fragment of a longer one.
+ fragment := []ghRun{{DatabaseID: 501, DisplayTitle: "Windows sandbox [" + ours + "ff]"},
+ {DatabaseID: 502, DisplayTitle: "Windows sandbox [ff" + ours + "]"}}
+ if got := pick(t, fragment, 500); got != 0 {
+ t.Errorf("an id embedded in a longer one must not match, got %d", got)
}
- // Two dispatches racing: follow the earlier new run, which is ours.
- withRace := append([]ghRun{{DatabaseID: 502}, {DatabaseID: 501}}, history...)
- if got := pickNewRun(withRace, 500); got != 501 {
- t.Errorf("with two new runs the earlier one is ours, got %d", got)
+ // Carrying our id is not enough on its own: a run at or below the floor predates the dispatch.
+ if got := pick(t, []ghRun{titled(500, ours)}, 500); got != 0 {
+ t.Errorf("a run at the floor predates the dispatch and must not be selected, got %d", got)
}
- // A first-ever dispatch has no history, so the floor is zero and any real run qualifies.
- if got := pickNewRun([]ghRun{{DatabaseID: 7}}, 0); got != 7 {
- t.Errorf("a zero floor must accept the first run, got %d", got)
+ // A first-ever dispatch has no history, so the floor is zero and our run qualifies.
+ if got := pick(t, []ghRun{titled(7, ours)}, 0); got != 7 {
+ t.Errorf("a zero floor must accept our first run, got %d", got)
+ }
+}
+
+// Two runs carrying one dispatch's random id means something reused it, and either could be the
+// run whose verdict would be reported. That is an explicit error, not a choice.
+func TestOwnRunSelectionRefusesTwoRunsCarryingTheID(t *testing.T) {
+ const ours = "0123456789abcdef01234567"
+ runs := []ghRun{
+ {DatabaseID: 503, DisplayTitle: "Windows sandbox " + correlationMarker(ours)},
+ {DatabaseID: 501, DisplayTitle: "Windows sandbox " + correlationMarker(ours)},
+ {DatabaseID: 500, DisplayTitle: "Windows sandbox"},
+ }
+ got, err := pickOwnRun(runs, 500, ours)
+ if err == nil {
+ t.Fatalf("two runs carrying one correlation id must be refused, got run %d", got)
+ }
+ if got != 0 {
+ t.Errorf("an ambiguous selection must not also return a run, got %d", got)
+ }
+ for _, want := range []string{"501", "503", ours, "refusing to guess"} {
+ if !strings.Contains(err.Error(), want) {
+ t.Errorf("the error should mention %q: %v", want, err)
+ }
+ }
+}
+
+// Each dispatch needs an id no concurrent one shares, in a form that survives a run title.
+func TestCorrelationIDsAreUniqueHex(t *testing.T) {
+ seen := map[string]bool{}
+ for range 1000 {
+ id, err := newCorrelationID()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !regexp.MustCompile(`^[0-9a-f]{24}$`).MatchString(id) {
+ t.Fatalf("correlation id %q is not 24 lowercase hex characters", id)
+ }
+ if seen[id] {
+ t.Fatalf("correlation id %q repeated", id)
+ }
+ seen[id] = true
}
}
diff --git a/beacon-sandbox/dispatch/fakegh_test.go b/beacon-sandbox/dispatch/fakegh_test.go
new file mode 100644
index 00000000..e48217c8
--- /dev/null
+++ b/beacon-sandbox/dispatch/fakegh_test.go
@@ -0,0 +1,447 @@
+package dispatch
+
+// A fake GitHub, reached through a fake `gh`, so Run can be exercised end to end without an
+// account, a network, or a real workflow.
+//
+// Run shells out to `gh` rather than calling the REST API, so the fake has to sit at that seam.
+// The test binary doubles as the `gh` executable: it is copied into a temporary directory as `gh`,
+// that directory is put first on PATH, and when it is invoked with fakeGHEnv set, TestMain
+// forwards its arguments to an httptest server and replays the answer. The server holds the state
+// a real repository would -- the run history, which inputs the workflow declares, the display title
+// each run gets -- so concurrent callers observe one shared, consistent GitHub, exactly as two
+// maintainers dispatching at once would.
+//
+// Nothing in the production code knows the fake exists. That is deliberate: the race this pins
+// (#681) lives in the order of real gh calls, so the test drives those calls rather than a seam
+// substituted for them.
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "runtime"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+ "testing"
+)
+
+const fakeGHEnv = "BEACON_SANDBOX_FAKE_GH_URL"
+
+// fakeGHDir holds the one copy of this test binary installed as `gh`, made once per package run
+// rather than once per test.
+var fakeGHDir string
+
+func TestMain(m *testing.M) {
+ if url := os.Getenv(fakeGHEnv); url != "" {
+ os.Exit(runFakeGH(url, os.Args[1:]))
+ }
+ dir, err := installFakeGH()
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "install fake gh:", err)
+ os.Exit(2)
+ }
+ fakeGHDir = dir
+ code := m.Run()
+ os.RemoveAll(dir)
+ os.Exit(code)
+}
+
+// installFakeGH copies this test binary into a fresh directory under the name gh.
+func installFakeGH() (string, error) {
+ self, err := os.Executable()
+ if err != nil {
+ return "", err
+ }
+ dir, err := os.MkdirTemp("", "beacon-sandbox-fake-gh-")
+ if err != nil {
+ return "", err
+ }
+ name := "gh"
+ if runtime.GOOS == "windows" {
+ name += ".exe"
+ }
+ src, err := os.Open(self)
+ if err != nil {
+ return "", err
+ }
+ defer src.Close()
+ dst, err := os.OpenFile(filepath.Join(dir, name), os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o700)
+ if err != nil {
+ return "", err
+ }
+ if _, err := io.Copy(dst, src); err != nil {
+ dst.Close()
+ return "", err
+ }
+ return dir, dst.Close()
+}
+
+type fakeGHRequest struct {
+ Args []string `json:"args"`
+}
+
+type fakeGHResponse struct {
+ Stdout string `json:"stdout"`
+ Exit int `json:"exit"`
+ // Files are written under the --dir argument, which is how `gh run download` delivers.
+ Files map[string]string `json:"files,omitempty"`
+}
+
+// runFakeGH is the whole of the fake executable: forward, replay, exit.
+func runFakeGH(url string, args []string) int {
+ body, _ := json.Marshal(fakeGHRequest{Args: args})
+ resp, err := http.Post(url, "application/json", bytes.NewReader(body))
+ if err != nil {
+ fmt.Fprintln(os.Stderr, "fake gh:", err)
+ return 2
+ }
+ defer resp.Body.Close()
+ var out fakeGHResponse
+ if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
+ fmt.Fprintln(os.Stderr, "fake gh: decode:", err)
+ return 2
+ }
+ if len(out.Files) > 0 {
+ dir := flagValue(args, "--dir")
+ for rel, content := range out.Files {
+ p := filepath.Join(dir, filepath.FromSlash(rel))
+ if err := os.MkdirAll(filepath.Dir(p), 0o700); err != nil {
+ fmt.Fprintln(os.Stderr, "fake gh:", err)
+ return 2
+ }
+ if err := os.WriteFile(p, []byte(content), 0o600); err != nil {
+ fmt.Fprintln(os.Stderr, "fake gh:", err)
+ return 2
+ }
+ }
+ }
+ fmt.Fprint(os.Stdout, out.Stdout)
+ return out.Exit
+}
+
+func flagValue(args []string, name string) string {
+ for i := 0; i+1 < len(args); i++ {
+ if args[i] == name {
+ return args[i+1]
+ }
+ }
+ return ""
+}
+
+func flagValues(args []string, name string) []string {
+ var vals []string
+ for i := 0; i+1 < len(args); i++ {
+ if args[i] == name {
+ vals = append(vals, args[i+1])
+ }
+ }
+ return vals
+}
+
+// fakeRun is one workflow run as the fake repository records it.
+type fakeRun struct {
+ ID int64
+ Status string
+ Conclusion string
+ DisplayTitle string
+ Inputs map[string]string
+ // hiddenFor counts the listings the run is still invisible to, standing in for the delay
+ // between a dispatch being accepted and its run appearing in the API.
+ hiddenFor int
+ // listsUntilDone counts the listings that report it in progress before it completes.
+ listsUntilDone int
+}
+
+// fakeGitHub is the shared state behind every fake gh invocation.
+type fakeGitHub struct {
+ mu sync.Mutex
+ nextID int64
+ runs []*fakeRun
+ // declared is the set of workflow_dispatch inputs the dispatched workflow accepts. GitHub
+ // rejects any other input with a 422, which is what dispatching a ref whose workflow predates
+ // an input looks like.
+ declared map[string]bool
+ // dispatched records each accepted dispatch, in the order GitHub assigned ids.
+ dispatched []*fakeRun
+ foreign []int64
+
+ // Hooks run outside the lock, so a test can hold one caller at a point in its sequence until
+ // another caller catches up -- which is how an interleaving is forced rather than hoped for.
+ beforeList func()
+ beforeDispatch func()
+ afterDispatch func(run *fakeRun)
+
+ // hideNewRunsFor and runListsUntilDone shape every run created by a dispatch.
+ hideNewRunsFor int
+ runListsUntilDone int
+ // noRunName models a dispatched ref whose workflow accepts the correlation input but does not
+ // render it into run-name, so every run is titled with the bare workflow name.
+ noRunName bool
+ // failLists makes that many listings fail the way a network blip does.
+ failLists int
+ // oldGH models a gh release that predates the displayTitle JSON field.
+ oldGH bool
+}
+
+// newFakeGitHub starts the fake and puts a `gh` that talks to it first on PATH.
+//
+// configure, if not nil, shapes the fake before its server starts: the hooks and knobs are read
+// by the server's goroutines, and setting them afterwards would be a data race.
+//
+// The history is a few completed runs so that the floor Run snapshots is a real, non-zero id --
+// the situation the race needs, since both callers read the same floor. By default the fake
+// accepts exactly the inputs this checkout's workflow declares.
+func newFakeGitHub(t *testing.T, configure func(*fakeGitHub)) *fakeGitHub {
+ t.Helper()
+ f := &fakeGitHub{nextID: 501, declared: map[string]bool{}}
+ for _, in := range workflowInputs(t) {
+ f.declared[in] = true
+ }
+ for id := int64(498); id <= 500; id++ {
+ f.runs = append(f.runs, &fakeRun{ID: id, Status: "completed", Conclusion: "success",
+ DisplayTitle: "Windows sandbox"})
+ }
+ if configure != nil {
+ configure(f)
+ }
+ srv := httptest.NewServer(http.HandlerFunc(f.serve))
+ t.Cleanup(srv.Close)
+
+ t.Setenv("PATH", fakeGHDir+string(os.PathListSeparator)+os.Getenv("PATH"))
+ t.Setenv(fakeGHEnv, srv.URL)
+ // Under -race every gh invocation is a race-enabled copy of this binary, and those sleep a
+ // second on exit by default. Harmless, but it turns a sub-second test into a slow one.
+ t.Setenv("GORACE", strings.TrimSpace(os.Getenv("GORACE")+" atexit_sleep_ms=0"))
+ return f
+}
+
+// workflowInputs reads the inputs the real workflow declares, so the fake accepts exactly what
+// GitHub would for a ref carrying this checkout's workflow.
+func workflowInputs(t *testing.T) []string {
+ t.Helper()
+ wf := loadWorkflow(t)
+ var names []string
+ for name := range wf.On.WorkflowDispatch.Inputs {
+ names = append(names, name)
+ }
+ sort.Strings(names)
+ return names
+}
+
+func (f *fakeGitHub) serve(w http.ResponseWriter, r *http.Request) {
+ var req fakeGHRequest
+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+ http.Error(w, err.Error(), http.StatusBadRequest)
+ return
+ }
+ _ = json.NewEncoder(w).Encode(f.handle(req.Args))
+}
+
+func (f *fakeGitHub) handle(args []string) fakeGHResponse {
+ switch {
+ case len(args) >= 2 && args[0] == "auth" && args[1] == "status":
+ return fakeGHResponse{Stdout: "Logged in to github.com as fake\n"}
+ case len(args) >= 3 && args[0] == "workflow" && args[1] == "run":
+ return f.dispatch(args)
+ case len(args) >= 2 && args[0] == "run" && args[1] == "list":
+ return f.list(args)
+ case len(args) >= 3 && args[0] == "run" && args[1] == "download":
+ return f.download(args)
+ }
+ return fakeGHResponse{Stdout: fmt.Sprintf("fake gh: unsupported command %q\n", args), Exit: 1}
+}
+
+func (f *fakeGitHub) dispatch(args []string) fakeGHResponse {
+ if args[2] != Workflow {
+ return fakeGHResponse{Stdout: "could not find workflow " + args[2] + "\n", Exit: 1}
+ }
+ inputs := map[string]string{}
+ var unexpected []string
+ for _, kv := range flagValues(args, "-f") {
+ k, v, _ := strings.Cut(kv, "=")
+ inputs[k] = v
+ if !f.declared[k] {
+ unexpected = append(unexpected, k)
+ }
+ }
+ if len(unexpected) > 0 {
+ // The shape of GitHub's real refusal, as gh prints it.
+ quoted, _ := json.Marshal(unexpected)
+ return fakeGHResponse{Exit: 1, Stdout: "could not create workflow dispatch event: HTTP 422: " +
+ "Unexpected inputs provided: " + string(quoted) + "\n"}
+ }
+ if f.beforeDispatch != nil {
+ f.beforeDispatch()
+ }
+ f.mu.Lock()
+ run := &fakeRun{
+ ID: f.nextID,
+ Status: "queued",
+ DisplayTitle: renderRunName(f.declared, inputs, f.noRunName),
+ Inputs: inputs,
+ hiddenFor: f.hideNewRunsFor,
+ listsUntilDone: f.runListsUntilDone,
+ }
+ f.nextID++
+ f.runs = append(f.runs, run)
+ f.dispatched = append(f.dispatched, run)
+ f.mu.Unlock()
+ if f.afterDispatch != nil {
+ f.afterDispatch(run)
+ }
+ return fakeGHResponse{Stdout: "Created workflow_dispatch event for " + Workflow + "\n"}
+}
+
+// addForeignRun records a run this process did not dispatch: another maintainer's, or a manual
+// dispatch from the Actions tab.
+func (f *fakeGitHub) addForeignRun(title string) int64 {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ run := &fakeRun{ID: f.nextID, Status: "in_progress", DisplayTitle: title}
+ f.nextID++
+ f.runs = append(f.runs, run)
+ f.foreign = append(f.foreign, run.ID)
+ return run.ID
+}
+
+// foreignRuns reports the ids addForeignRun created. Read under the lock because the hooks that
+// add them run on the server's goroutine.
+func (f *fakeGitHub) foreignRuns() []int64 {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]int64(nil), f.foreign...)
+}
+
+// renderRunName evaluates the workflow's run-name the way Actions would for these inputs. It only
+// applies when the workflow declares the correlation input; a ref whose workflow does not has no
+// run-name, and GitHub then titles a dispatched run with the workflow's name.
+func renderRunName(declared map[string]bool, inputs map[string]string, noRunName bool) string {
+ if !noRunName && declared[correlationInput] && inputs[correlationInput] != "" {
+ return "Windows sandbox [" + inputs[correlationInput] + "]"
+ }
+ return "Windows sandbox"
+}
+
+func (f *fakeGitHub) list(args []string) fakeGHResponse {
+ if flagValue(args, "--workflow") != Workflow {
+ return fakeGHResponse{Stdout: "fake gh: wrong workflow\n", Exit: 1}
+ }
+ limit, err := strconv.Atoi(flagValue(args, "--limit"))
+ if err != nil {
+ return fakeGHResponse{Stdout: "fake gh: bad --limit\n", Exit: 1}
+ }
+ fields := strings.Split(flagValue(args, "--json"), ",")
+ if f.beforeList != nil {
+ f.beforeList()
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ if f.failLists > 0 {
+ f.failLists--
+ return fakeGHResponse{Exit: 1, Stdout: "error connecting to api.github.com\ncheck your internet connection\n"}
+ }
+ var visible []*fakeRun
+ for _, r := range f.runs {
+ if r.hiddenFor > 0 {
+ r.hiddenFor--
+ continue
+ }
+ visible = append(visible, r)
+ }
+ sort.Slice(visible, func(i, j int) bool { return visible[i].ID > visible[j].ID })
+ if len(visible) > limit {
+ visible = visible[:limit]
+ }
+ out := make([]map[string]any, 0, len(visible))
+ for _, r := range visible {
+ // Advanced after being reported, so a caller sees each state at least once.
+ row := map[string]any{}
+ for _, field := range fields {
+ switch field {
+ case "databaseId":
+ row[field] = r.ID
+ case "status":
+ row[field] = r.Status
+ case "conclusion":
+ row[field] = r.Conclusion
+ case "url":
+ row[field] = fmt.Sprintf("https://github.com/fake/fake/actions/runs/%d", r.ID)
+ case "displayTitle":
+ if f.oldGH {
+ return fakeGHResponse{Stdout: "Unknown JSON field: \"displayTitle\"\n", Exit: 1}
+ }
+ row[field] = r.DisplayTitle
+ default:
+ // gh rejects an unknown --json field outright; so does the fake, so a typo in the
+ // production field list cannot pass here and fail against the real gh.
+ return fakeGHResponse{Stdout: "Unknown JSON field: " + field + "\n", Exit: 1}
+ }
+ }
+ out = append(out, row)
+ if r.Status != "completed" && r.Inputs != nil {
+ if r.listsUntilDone > 0 {
+ r.listsUntilDone--
+ r.Status = "in_progress"
+ } else {
+ r.Status, r.Conclusion = "completed", "success"
+ }
+ }
+ }
+ body, _ := json.Marshal(out)
+ return fakeGHResponse{Stdout: string(body)}
+}
+
+func (f *fakeGitHub) download(args []string) fakeGHResponse {
+ id, err := strconv.ParseInt(args[2], 10, 64)
+ if err != nil {
+ return fakeGHResponse{Stdout: "fake gh: bad run id\n", Exit: 1}
+ }
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ for _, r := range f.runs {
+ if r.ID != id || r.Inputs == nil {
+ continue
+ }
+ // Named the way the harness names a run directory, and carrying which dispatch produced
+ // it, so a test can tell whose artifacts a caller brought home.
+ dir := fmt.Sprintf("beacon-sandbox-runs/%s-%d", r.Inputs["scenario"], r.ID)
+ return fakeGHResponse{Stdout: "", Files: map[string]string{
+ dir + "/runtime.jsonl": fmt.Sprintf(`{"run":%d,"scenario":%q}`+"\n", r.ID, r.Inputs["scenario"]),
+ }}
+ }
+ return fakeGHResponse{Stdout: "no artifacts found for run\n", Exit: 1}
+}
+
+// runFor reports the run GitHub created for the dispatch of this scenario.
+func (f *fakeGitHub) runFor(scenario string) int64 {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ for _, r := range f.dispatched {
+ if r.Inputs["scenario"] == scenario {
+ return r.ID
+ }
+ }
+ return 0
+}
+
+// dispatches returns a copy of each accepted dispatch's inputs, in id order.
+func (f *fakeGitHub) dispatches() []map[string]string {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ out := make([]map[string]string, len(f.dispatched))
+ for i, r := range f.dispatched {
+ cp := map[string]string{}
+ for k, v := range r.Inputs {
+ cp[k] = v
+ }
+ out[i] = cp
+ }
+ return out
+}
diff --git a/beacon-sandbox/dispatch/run_test.go b/beacon-sandbox/dispatch/run_test.go
new file mode 100644
index 00000000..4c9a0e7a
--- /dev/null
+++ b/beacon-sandbox/dispatch/run_test.go
@@ -0,0 +1,337 @@
+package dispatch
+
+import (
+ "os"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "time"
+)
+
+// waitOrFail blocks on ch, failing loudly rather than hanging the suite if the interleaving a test
+// forces never happens.
+func waitOrFail(t *testing.T, ch <-chan struct{}, what string) {
+ t.Helper()
+ select {
+ case <-ch:
+ case <-time.After(20 * time.Second):
+ t.Errorf("timed out waiting for %s", what)
+ }
+}
+
+// #681. Two callers dispatch at nearly the same time. Both snapshot the same floor before either
+// dispatch lands, then both runs become visible before either caller looks for its own. Selecting
+// "the smallest id above the floor" hands both callers the same run: one judges the other's
+// artifacts, and the other's own run is never matched.
+//
+// The interleaving is forced, not hoped for. The fake GitHub holds each dispatch until both
+// snapshots are taken, and holds every post-dispatch listing until both runs exist -- the exact
+// window the issue traces. Each caller must come back with the run its own dispatch created and
+// the artifacts that run produced, and never the other's.
+func TestConcurrentDispatchesEachFollowTheirOwnRun(t *testing.T) {
+ var snapshots, dispatches atomic.Int32
+ snapshotsTaken := make(chan struct{})
+ bothDispatched := make(chan struct{})
+ var closeSnap, closeDisp sync.Once
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.beforeList = func() {
+ if dispatches.Load() == 0 {
+ // A listing before any dispatch has completed is a caller taking its floor.
+ if snapshots.Add(1) == 2 {
+ closeSnap.Do(func() { close(snapshotsTaken) })
+ }
+ return
+ }
+ waitOrFail(t, bothDispatched, "both dispatches to land")
+ }
+ gh.beforeDispatch = func() { waitOrFail(t, snapshotsTaken, "both callers to snapshot the floor") }
+ gh.afterDispatch = func(*fakeRun) {
+ if dispatches.Add(1) == 2 {
+ closeDisp.Do(func() { close(bothDispatched) })
+ }
+ }
+ })
+
+ scenarios := []string{"w00-probe", "w03-hook-capture"}
+ results := make([]Result, len(scenarios))
+ errs := make([]error, len(scenarios))
+ var wg sync.WaitGroup
+ for i, sc := range scenarios {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ results[i], errs[i] = Run(fastOptions(t, sc))
+ }()
+ }
+ wg.Wait()
+
+ for i, sc := range scenarios {
+ if errs[i] != nil {
+ t.Fatalf("caller %s: %v", sc, errs[i])
+ }
+ want := gh.runFor(sc)
+ if want == 0 {
+ t.Fatalf("the fake never saw a dispatch for %s", sc)
+ }
+ if results[i].RunID != strconv.FormatInt(want, 10) {
+ t.Errorf("caller %s followed run %s, but its own dispatch created run %d", sc,
+ results[i].RunID, want)
+ }
+ assertOwnArtifacts(t, results[i], sc)
+ }
+ if results[0].RunID == results[1].RunID {
+ t.Errorf("both callers attached to run %s; one of them judged the other's artifacts",
+ results[0].RunID)
+ }
+}
+
+// assertOwnArtifacts checks the caller brought home exactly the run directory its own dispatch
+// produced. This is the consequence that matters: a wrong run id alone is a bookkeeping error,
+// but judging another invocation's artifacts is a wrong verdict.
+func assertOwnArtifacts(t *testing.T, res Result, scenario string) {
+ t.Helper()
+ if len(res.RunDirs) != 1 {
+ t.Errorf("caller %s collected %d run dirs, want 1: %v", scenario, len(res.RunDirs), res.RunDirs)
+ return
+ }
+ if got, want := filepath.Base(res.RunDirs[0]), scenario+"-"+res.RunID; got != want {
+ t.Errorf("caller %s collected %s, want its own run's %s", scenario, got, want)
+ }
+ data, err := os.ReadFile(filepath.Join(res.RunDirs[0], "runtime.jsonl"))
+ if err != nil {
+ t.Errorf("caller %s: %v", scenario, err)
+ return
+ }
+ if want := `"scenario":"` + scenario + `"`; !strings.Contains(string(data), want) {
+ t.Errorf("caller %s judged another dispatch's log: %s", scenario, data)
+ }
+}
+
+func fastOptions(t *testing.T, scenario string) Options {
+ return Options{
+ Repo: "fake/fake",
+ Ref: "work",
+ Scenario: scenario,
+ ClaudeVersion: "2.1.220",
+ OutDir: t.TempDir(),
+ PollInterval: 10 * time.Millisecond,
+ Timeout: 10 * time.Second,
+ newRunPoll: 10 * time.Millisecond,
+ newRunTimeout: 10 * time.Second,
+ }
+}
+
+// The single-caller path, which every dispatch took before #681 and must still take: the run is
+// not listed for the first few polls, then runs through its states, and the caller comes back
+// with it and its artifacts. Also pins what the dispatch sends: the correlation id rides as a
+// workflow input alongside the existing ones.
+func TestSingleDispatchFollowsItsRunToCompletion(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.hideNewRunsFor = 3
+ gh.runListsUntilDone = 3
+ })
+
+ res, err := Run(fastOptions(t, "w00-probe"))
+ if err != nil {
+ t.Fatalf("Run: %v", err)
+ }
+ if want := strconv.FormatInt(gh.runFor("w00-probe"), 10); res.RunID != want {
+ t.Errorf("followed run %s, want %s", res.RunID, want)
+ }
+ if res.Conclusion != "success" {
+ t.Errorf("conclusion = %q, want success", res.Conclusion)
+ }
+ if res.URL != "https://github.com/fake/fake/actions/runs/"+res.RunID {
+ t.Errorf("URL = %q", res.URL)
+ }
+ assertOwnArtifacts(t, res, "w00-probe")
+
+ sent := gh.dispatches()
+ if len(sent) != 1 {
+ t.Fatalf("expected one dispatch, got %d", len(sent))
+ }
+ if sent[0]["scenario"] != "w00-probe" || sent[0]["claude_version"] != "2.1.220" {
+ t.Errorf("the existing inputs must still be sent unchanged, got %v", sent[0])
+ }
+ if !regexp.MustCompile(`^[0-9a-f]{24}$`).MatchString(sent[0][correlationInput]) {
+ t.Errorf("the dispatch must carry a correlation id, got %q", sent[0][correlationInput])
+ }
+}
+
+// A run that appears first but is not ours -- another invocation's, or a manual dispatch from the
+// Actions tab -- is waited past, not taken. Sequential counterpart of the race: the foreign runs
+// are the earliest above the floor, which is exactly what the old rule selected.
+func TestDispatchWaitsPastRunsThatAreNotItsOwn(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.hideNewRunsFor = 2
+ gh.beforeDispatch = func() {
+ gh.addForeignRun("Windows sandbox " + correlationMarker("fedcba9876543210fedcba98"))
+ gh.addForeignRun("Windows sandbox")
+ }
+ })
+
+ res, err := Run(fastOptions(t, "w03-hook-capture"))
+ if err != nil {
+ t.Fatalf("Run: %v", err)
+ }
+ foreign := gh.foreignRuns()
+ if len(foreign) != 2 {
+ t.Fatalf("the fake should have added two foreign runs, got %v", foreign)
+ }
+ for _, id := range foreign {
+ if res.RunID == strconv.FormatInt(id, 10) {
+ t.Fatalf("attached to foreign run %d instead of our own", id)
+ }
+ }
+ if want := strconv.FormatInt(gh.runFor("w03-hook-capture"), 10); res.RunID != want {
+ t.Errorf("followed run %s, want our own %s", res.RunID, want)
+ }
+ assertOwnArtifacts(t, res, "w03-hook-capture")
+}
+
+// If our id never shows up but other runs did, the dispatch cannot know which, if any, is ours --
+// for instance the dispatched ref's workflow accepts the input but does not render it into its
+// title. That must be an explicit ambiguity error naming the runs, and nothing may be downloaded
+// or judged.
+func TestDispatchReportsAmbiguityWhenNoRunCarriesItsID(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.noRunName = true
+ gh.beforeDispatch = func() { gh.addForeignRun("Windows sandbox") }
+ })
+
+ opts := fastOptions(t, "w00-probe")
+ opts.newRunTimeout = 300 * time.Millisecond
+ res, err := Run(opts)
+ if err == nil {
+ t.Fatalf("an uncorrelatable run must be an error, got run %s", res.RunID)
+ }
+ ours := strconv.FormatInt(gh.runFor("w00-probe"), 10)
+ foreign := gh.foreignRuns()
+ if len(foreign) != 1 {
+ t.Fatalf("the fake should have added one foreign run, got %v", foreign)
+ }
+ for _, want := range []string{"refusing to guess", ours, strconv.FormatInt(foreign[0], 10),
+ correlationInput, "run-name"} {
+ if !strings.Contains(err.Error(), want) {
+ t.Errorf("the ambiguity error should mention %q: %v", want, err)
+ }
+ }
+ if res.RunID != "" || len(res.RunDirs) != 0 {
+ t.Errorf("nothing may be selected or collected on ambiguity, got %+v", res)
+ }
+ assertNothingCollected(t, opts.OutDir)
+}
+
+// No run at all within the wait is the plain timeout it always was, not an ambiguity.
+func TestDispatchTimesOutWhenNoRunAppears(t *testing.T) {
+ newFakeGitHub(t, func(gh *fakeGitHub) { gh.hideNewRunsFor = 1 << 30 })
+
+ opts := fastOptions(t, "w00-probe")
+ opts.newRunTimeout = 200 * time.Millisecond
+ _, err := Run(opts)
+ if err == nil {
+ t.Fatal("a run that never appears must be an error")
+ }
+ if !strings.Contains(err.Error(), "no new "+Workflow+" run appeared") {
+ t.Errorf("want the no-run timeout, got: %v", err)
+ }
+ if strings.Contains(err.Error(), "refusing to guess") {
+ t.Errorf("nothing appeared, so there is nothing ambiguous: %v", err)
+ }
+}
+
+// A listing that fails transiently while waiting is retried, as before, rather than abandoning a
+// run that is proceeding fine.
+func TestDispatchRetriesAFailedListing(t *testing.T) {
+ var failed atomic.Bool
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.hideNewRunsFor = 1
+ gh.afterDispatch = func(*fakeRun) {
+ gh.mu.Lock()
+ gh.failLists = 2
+ gh.mu.Unlock()
+ failed.Store(true)
+ }
+ })
+ res, err := Run(fastOptions(t, "w00-probe"))
+ if err != nil {
+ t.Fatalf("Run: %v", err)
+ }
+ if !failed.Load() {
+ t.Fatal("the fake never injected the failure")
+ }
+ gh.mu.Lock()
+ left := gh.failLists
+ gh.mu.Unlock()
+ if left != 0 {
+ t.Errorf("%d injected listing failures were never hit, so the retry was not exercised", left)
+ }
+ assertOwnArtifacts(t, res, "w00-probe")
+}
+
+// The completion wait's own timeout still reports the run it was following, so the reader can
+// go and look at it.
+func TestDispatchReportsARunThatDoesNotComplete(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) { gh.runListsUntilDone = 1 << 30 })
+
+ opts := fastOptions(t, "w00-probe")
+ opts.Timeout = 200 * time.Millisecond
+ res, err := Run(opts)
+ if err == nil || !strings.Contains(err.Error(), "did not complete within") {
+ t.Fatalf("want the completion timeout, got: %v", err)
+ }
+ if want := strconv.FormatInt(gh.runFor("w00-probe"), 10); res.RunID != want {
+ t.Errorf("the timeout must still identify our run: got %q, want %s", res.RunID, want)
+ }
+}
+
+// GitHub reads the workflow from the dispatched ref, so a branch cut before the correlation input
+// existed rejects it with a 422. That is refused with the fix, not retried uncorrelated: an
+// uncorrelated dispatch is precisely the one that can attach to someone else's run.
+func TestDispatchRefusesARefWhoseWorkflowLacksTheCorrelationInput(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) {
+ gh.declared = map[string]bool{"scenario": true, "claude_version": true}
+ })
+
+ _, err := Run(fastOptions(t, "w00-probe"))
+ if err == nil {
+ t.Fatal("dispatching a ref without the correlation input must fail")
+ }
+ for _, want := range []string{correlationInput, "work", "rebase"} {
+ if !strings.Contains(err.Error(), want) {
+ t.Errorf("the error should mention %q: %v", want, err)
+ }
+ }
+ if n := len(gh.dispatches()); n != 0 {
+ t.Errorf("no uncorrelated fallback dispatch may be made, got %d", n)
+ }
+}
+
+// displayTitle is what carries the correlation id, so a gh too old to report it cannot identify a
+// run at all. That must fail before anything is dispatched, and say what to do.
+func TestDispatchExplainsAGHTooOldForDisplayTitle(t *testing.T) {
+ gh := newFakeGitHub(t, func(gh *fakeGitHub) { gh.oldGH = true })
+
+ _, err := Run(fastOptions(t, "w00-probe"))
+ if err == nil || !strings.Contains(err.Error(), "upgrade gh") {
+ t.Fatalf("want an upgrade-gh error, got: %v", err)
+ }
+ if n := len(gh.dispatches()); n != 0 {
+ t.Errorf("nothing may be dispatched when the run could never be identified, got %d", n)
+ }
+}
+
+func assertNothingCollected(t *testing.T, outDir string) {
+ t.Helper()
+ entries, err := os.ReadDir(outDir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, e := range entries {
+ t.Errorf("nothing should have been collected, found %s", e.Name())
+ }
+}
diff --git a/beacon-sandbox/dispatch/workflow_test.go b/beacon-sandbox/dispatch/workflow_test.go
new file mode 100644
index 00000000..1da28c6e
--- /dev/null
+++ b/beacon-sandbox/dispatch/workflow_test.go
@@ -0,0 +1,132 @@
+package dispatch
+
+import (
+ "os"
+ "path/filepath"
+ "regexp"
+ "strings"
+ "testing"
+
+ "gopkg.in/yaml.v3"
+)
+
+type workflowInput struct {
+ Description string `yaml:"description"`
+ Required bool `yaml:"required"`
+ Default *string `yaml:"default"`
+ Type string `yaml:"type"`
+}
+
+type workflowStep struct {
+ Name string `yaml:"name"`
+ Run string `yaml:"run"`
+ Env map[string]string `yaml:"env"`
+ With map[string]string `yaml:"with"`
+}
+
+type workflowFile struct {
+ Name string `yaml:"name"`
+ RunName string `yaml:"run-name"`
+ On struct {
+ WorkflowDispatch struct {
+ Inputs map[string]workflowInput `yaml:"inputs"`
+ } `yaml:"workflow_dispatch"`
+ } `yaml:"on"`
+ Jobs map[string]struct {
+ Env map[string]string `yaml:"env"`
+ Steps []workflowStep `yaml:"steps"`
+ } `yaml:"jobs"`
+}
+
+// loadWorkflow parses the workflow this package dispatches, from this checkout.
+func loadWorkflow(t *testing.T) workflowFile {
+ t.Helper()
+ data, err := os.ReadFile(filepath.Join("..", "..", ".github", "workflows", Workflow))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var wf workflowFile
+ if err := yaml.Unmarshal(data, &wf); err != nil {
+ t.Fatalf("parse %s: %v", Workflow, err)
+ }
+ return wf
+}
+
+// The dispatch only works if the workflow declares the correlation input and renders it into the
+// run's title in the form pickOwnRun matches. Both sides live in different files, so this pins
+// them together: change one without the other and this fails here rather than as a dispatch that
+// never finds its run.
+func TestWorkflowStampsTheCorrelationIDIntoTheRunName(t *testing.T) {
+ wf := loadWorkflow(t)
+
+ in, ok := wf.On.WorkflowDispatch.Inputs[correlationInput]
+ if !ok {
+ t.Fatalf("%s must declare the %s workflow_dispatch input", Workflow, correlationInput)
+ }
+ // Optional with an empty default, so a manual dispatch from the Actions tab still works and
+ // keeps the plain title.
+ if in.Required {
+ t.Errorf("%s must be optional so manual dispatches keep working", correlationInput)
+ }
+ if in.Type != "string" {
+ t.Errorf("%s type = %q, want string", correlationInput, in.Type)
+ }
+ if in.Default == nil || *in.Default != "" {
+ t.Errorf("%s must default to empty", correlationInput)
+ }
+
+ // The run-name's formatted branch, rendered for an id, must produce exactly the marker the
+ // selection looks for; its fallback must be the workflow name, which is what a run without a
+ // run-name is titled and what a manual dispatch therefore still shows.
+ m := regexp.MustCompile(`^\$\{\{\s*inputs\.` + correlationInput +
+ `\s*&&\s*format\('([^']*)',\s*inputs\.` + correlationInput + `\)\s*\|\|\s*'([^']*)'\s*\}\}$`).
+ FindStringSubmatch(strings.TrimSpace(wf.RunName))
+ if m == nil {
+ t.Fatalf("run-name must render inputs.%s with format(...) and fall back to a plain "+
+ "title; got %q", correlationInput, wf.RunName)
+ }
+ const id = "0123456789abcdef01234567"
+ rendered := strings.ReplaceAll(m[1], "{0}", id)
+ if got, err := pickOwnRun([]ghRun{{DatabaseID: 2, DisplayTitle: rendered}}, 1, id); err != nil || got != 2 {
+ t.Errorf("a run titled %q must be selected for id %s (got %d, %v)", rendered, id, got, err)
+ }
+ if want := renderRunName(map[string]bool{correlationInput: true},
+ map[string]string{correlationInput: id}, false); rendered != want {
+ t.Errorf("the fake GitHub titles runs %q but the workflow renders %q; keep them in step",
+ want, rendered)
+ }
+ if m[2] != wf.Name {
+ t.Errorf("run-name fallback = %q, want the workflow name %q", m[2], wf.Name)
+ }
+}
+
+// The id is only ever rendered into the title. Keeping it out of every script, env block and
+// action input means an arbitrary string typed into the dispatch form cannot reach a shell, so the
+// input needs none of the validation scenario and claude_version get.
+func TestWorkflowNeverPassesTheCorrelationIDToAStep(t *testing.T) {
+ wf := loadWorkflow(t)
+ if len(wf.Jobs) == 0 {
+ t.Fatal("parsed no jobs; the workflow shape changed and this test is no longer checking anything")
+ }
+ for name, job := range wf.Jobs {
+ for k, v := range job.Env {
+ if strings.Contains(v, correlationInput) {
+ t.Errorf("job %s env %s references %s", name, k, correlationInput)
+ }
+ }
+ for _, step := range job.Steps {
+ fields := []string{step.Run}
+ for _, v := range step.Env {
+ fields = append(fields, v)
+ }
+ for _, v := range step.With {
+ fields = append(fields, v)
+ }
+ for _, f := range fields {
+ if strings.Contains(f, correlationInput) {
+ t.Errorf("job %s step %q references %s", name, step.Name, correlationInput)
+ }
+ }
+ }
+ }
+}
diff --git a/docs/contributing/beacon-sandbox.mdx b/docs/contributing/beacon-sandbox.mdx
index 6eb2b15f..87c8e397 100644
--- a/docs/contributing/beacon-sandbox.mdx
+++ b/docs/contributing/beacon-sandbox.mdx
@@ -242,6 +242,8 @@ gh run watch
Leave `scenario` empty to run every Windows test. Each takes a few minutes, and the whole set takes about fifteen.
+`beacon-sandbox run --provider github --scenario <id>` dispatches the same workflow and brings the artifacts home to judge. It also sends a random `correlation_id` input that the workflow puts in the run's title, so it follows only the run its own dispatch created, even when someone else dispatches at the same moment. A run dispatched by hand, as above, leaves that input empty and keeps the plain title.
+
| Test | Covers |
|------|--------|
| `w00-probe` | The `beacon ci exec` collection path. Known to be unreliable on Windows. See [#320](https://github.com/asymptote-labs/agent-beacon/issues/320) |