Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

ci: the remote gate reuses passed GitHub full runs of the identical tree (GBRA-135 audit 2, L1)

garrytan32960d6ab0a1
diff --git a/docs/TESTING.md b/docs/TESTING.md
index a02cce0ea2..cb75989416 100644
--- a/docs/TESTING.md
+++ b/docs/TESTING.md
@@ -19,7 +19,7 @@ bun install --frozen-lockfile && bun run test && bun run verify
is CI's guard battery (about a minute). For E2E, follow [E2E test DB lifecycle](#e2e-test-db-lifecycle-always-follow-this) and
run `DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run test:e2e`.
`bun run ci:local` runs the whole CI gate in Docker; `bun run ci:ubicloud` runs
-it on ephemeral VMs in about five minutes.
+it on ephemeral VMs.
A failing `bun run test` writes every failure block to
`.context/test-failures.log`, one line per shard to `.context/test-summary.txt`
@@ -415,7 +415,7 @@ Test command tiers, each with a clear scope:
| `bun run verify` | CI's authoritative pre-test gate set, fanned out by `scripts/run-verify-parallel.sh` through a bounded worker pool (default `detect_cpus`; override `GBRAIN_VERIFY_MAX_PARALLEL`) with the heavy checks ordered first (typecheck, the two compile-embed checks, admin build, fuzz bundles, whole-tree greps), then the self-timed `SOLO_CHECKS` alone ([why](operations/verify-and-nightly-e2e.md#verify-solo-checks)). Two evals ride the unit lane instead ([why](operations/verify-and-nightly-e2e.md#evals-in-the-unit-lane)). The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~65-85s | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. Smart e2e runs only when `DATABASE_URL` is set and propagates its failure; otherwise it prints a skip notice to stderr. Use `ci:local` to provision the databases and require PgBouncer execution. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run ci:local` | Independent host gitleaks scans, then frozen dependencies, guards/typecheck, the complete serial and slow lanes, and four unit/E2E shards inside Docker. Each E2E shard has its own pgvector database; selected PgBouncer tests must execute against the transaction-mode pooler. Unit, serial, and slow lanes have database URL overrides unset. Any failed stage fails the command. Complete shard logs survive container teardown under `.context/ci-local-shards/`. `ci:local:diff` runs only gitleaks and the doc checks on a doc-only diff and the full gate otherwise; `--no-shard` runs unit/E2E sequentially. Doc-only diffs still require successful gitleaks scans. | Depends on the full corpus | Full local gate before shipping. |
-| `bun run ci:ubicloud` | The `ci:local` lanes (gitleaks, guards/typecheck, serial, slow, unit, all E2E with required PgBouncer execution) fanned out across ephemeral Ubicloud VMs from one heaviest-first work queue; `ci:ubicloud:diff` takes the same doc-only fast path as `ci:local:diff`. Needs `UBICLOUD_API_KEY` or `UBICLOUD_API_TOKEN`, no local Docker. See "Ubicloud fan-out" below. | ~5 min | Full gate before shipping when a Ubicloud token is available. |
+| `bun run ci:ubicloud` | The `ci:local` lanes (gitleaks, guards/typecheck, serial, slow, unit, all E2E with required PgBouncer execution) fanned out across ephemeral Ubicloud VMs; `ci:ubicloud:diff` takes the same doc-only fast path as `ci:local:diff`. Needs `UBICLOUD_API_KEY` or `UBICLOUD_API_TOKEN`, no local Docker. See "Ubicloud fan-out" below. | ~5 min | Full gate before shipping when a Ubicloud token is available. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | The `*.serial.test.ts` set listed in `scripts/serial-files.tsv`, one Bun process per file, run through a pool of concurrent per-file processes (`min(detect_cpus, 4)`, memory-adapted) heaviest-first from `scripts/serial-weights.json` (absent weights fall back to the corpus p75; scheduling only). The `EXCLUSIVE_FILES` in `scripts/run-serial-tests.sh` (machine-global state or timing, at most 4) run sequentially after the pool, on shard 1 only. Per-test timeout 120s; each pooled file is killed at 300s. `SHARD=N/M` partitions pooled files by duration. Externally killed files get one sequential rescue run. Knobs: `GBRAIN_SERIAL_POOL=N` (`1` is fully sequential), `GBRAIN_SERIAL_FILE_TIMEOUT`. | a few minutes at pool=4 | Debugging quarantined files; CI's serial-tests job. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential within a shard; `SHARD=N/M` fans out against separate databases (ci-local runs 4 containers). Activates the PGLite snapshot like every other runner (per-file cold-path opt-outs where the test asserts the path TO post-initSchema state), exporting it as an ABSOLUTE path so CLI children spawned with varying cwd still find it. | ~5-10min | Pre-ship; nightly. |
@@ -459,9 +459,7 @@ from one heaviest-first work queue; every VM is destroyed on exit. VM setup, sch
and flags are in [scripts/ubicloud/README.md](../scripts/ubicloud/README.md).
VMs are named `ubirun-<owner>-<epoch>-<suffix>`; set `UBI_OWNER` to your thread
-code. Teardown destroys every VM the run asked for, including creates still in
-flight, and stale-VM sweeps run only with `UBI_GC_HOURS` set and only on your
-own VMs. `scripts/ubicloud/ubi-runner.sh usage` shows vCPUs by owner.
+code. `scripts/ubicloud/ubi-runner.sh usage` shows vCPUs by owner.
A sleeping machine kills the run with no signal, so teardown never runs: on Capy, run
`ci:ubicloud` as a watched background operation. After any interrupted run, check
`ubi-runner.sh list --mine` and reap leftovers with `down NAME` (or `gc HOURS`, own VMs only).
@@ -469,6 +467,9 @@ A sleeping machine kills the run with no signal, so teardown never runs: on Capy
The project quota is 512 vCPUs, shared with PR CI; a full gate takes 64, or 160 when it is free.
In a multi-lane wave, lanes run `ci:ubicloud:diff` or targeted suites (`bun
test <files>`, `--lanes`); only the integrator runs the full gate.
+GitHub full-ci is the source of truth: `scripts/pregate-remote-gate.sh` takes its receipt
+from the passed Test and E2E full runs of the identical clean tree; any other
+tree runs the gate ([remote-gate reuse](ci-event-parity.md#remote-gate-reuse)).
### E2E backend matrix
@@ -739,7 +740,7 @@ Selected E2E direct-only run would duplicate it), persistence-validation.yml and
owner. Locally, `ci:local:diff` and `ci:ubicloud:diff` run gitleaks and
`scripts/ci-doc-checks.sh` (llms freshness, KEY_FILES byte caps, documented
paths, skill references, privacy guards) on a doc-only diff and the full gate
-otherwise, printing "E2E narrowing is retired".
+otherwise.
### Refactor wave 1 goldens
@@ -1268,7 +1269,7 @@ there even though they pass on Linux and macOS.
### CI vs local: intentionally divergent file sets
-- **CI matrix** (`.github/workflows/test.yml`) runs 22 `scripts/test-shard.sh` shards, two per runner, split by weight-aware LPT bin-packing (`scripts/sharding.ts`; unweighted files get the p75 weight) and includes `*.slow.test.ts` files without a dedicated job, plus keyless-allowlisted `evals/**/*.test.ts` (`test/scripts/evals-collection.test.ts`). Shards keep planned file order (`test-shard.sh --dry-run-list <i> 18`) and print their predicted weight (`sharding.ts --sum`). Every Bun job activates the PGLite schema snapshot (built in-runner, cached across jobs; the runner's hash check is authoritative). Serial files run across four `serial-tests` workers via `bun run test:serial`; `verify`, `typecheck` and the BrainBench memory-conformance gate (`scripts/ci-brainbench-gate.sh`, compared against `evals/brainbench/baselines/main.json`) have their own jobs, all aggregated by `test-status`. `e2e.yml` aggregates through `e2e-status`, with jsonb-parity gating Tier 2's token spend; scheduled runs also require the full-corpus lanes. Both aggregates reject failures, cancellations and unexpected skips. Only a master push reuses results: those of a passed full-suite run of its exact tree (`scripts/ci-reuse.ts`). CI is the ground truth for "did everything pass."
+- **CI matrix** (`.github/workflows/test.yml`) runs 22 `scripts/test-shard.sh` shards, two per runner, split by weight-aware LPT bin-packing (`scripts/sharding.ts`; unweighted files get the p75 weight) and includes `*.slow.test.ts` files without a dedicated job, plus keyless-allowlisted `evals/**/*.test.ts` (`test/scripts/evals-collection.test.ts`). Shards keep planned file order (`test-shard.sh --dry-run-list <i> 18`) and print their predicted weight (`sharding.ts --sum`). Every Bun job activates the PGLite schema snapshot (built in-runner, cached across jobs; the runner's hash check is authoritative). Serial files run across four `serial-tests` workers via `bun run test:serial`; `verify`, `typecheck` and the BrainBench memory-conformance gate (`scripts/ci-brainbench-gate.sh`, compared against `evals/brainbench/baselines/main.json`) have their own jobs, all aggregated by `test-status`. `e2e.yml` aggregates through `e2e-status`, with jsonb-parity gating Tier 2's token spend; scheduled runs also require the full-corpus lanes. Both aggregates reject failures, cancellations and unexpected skips. Only a master push and the remote gate reuse results: those of a passed full run of the exact tree (`scripts/ci-reuse.ts`). CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses the same weighted partitioner as CI and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Each shard runs its complete ordered selection with a fresh Bun process per file, without adding workers. Later groups still run after failures; missing summaries or file-completion evidence fail the shard. Local trades coverage for inner-loop speed; CI catches what local skips.
`test/scripts/run-unit-shard.test.ts` pins what the local loop includes, and
diff --git a/docs/ci-event-parity.md b/docs/ci-event-parity.md
index 3c02c4abf7..5af2d49d5c 100644
--- a/docs/ci-event-parity.md
+++ b/docs/ci-event-parity.md
@@ -99,6 +99,26 @@ reusing push only with a readable `reuse.json` naming the same workflow, the
push head and a valid source run. It prints the reused run per workflow and
writes `test_reused_run_id` / `e2e_reused_run_id`. Schedules never reuse.
+## Remote-gate reuse
+
+GitHub's full profile is the source of truth for the Ubicloud gate too.
+`scripts/pregate-remote-gate.sh`, the run that clears gstack's `remote_gate`
+obligation, first asks `bun scripts/ci-reuse.ts --gate` whether the clean
+working tree's commit, or another head with the same tree, already passed the
+complete full runs of both Test and E2E under the master-push rules above. If
+both did, the receipt comes from them: the same `PREGATE_REMOTE: tree= tests=
+failed=` line, with the runs' executed test identities as the counts, and no VM
+starts. A full run on the tree still in progress is waited for (90 minutes at
+most, a progress line every minute). A failed newest run on the head refuses
+with no receipt. Every other case runs the gate: a dirty tree, a commit GitHub
+doesn't have, a missing workflow, an API error, or `GBRAIN_PREGATE_REUSE=0`.
+
+Only the identical tree reuses. A head that /ship changed after full-ci has a
+new tree, so it needs its own passed full run or gate. A stamp-only difference
+reaches the head through gstack's stamp rule (`gstack-pregate clear --receipt`
+with the stamp files in `.gstack/ship-policy.json`), never through this check.
+Merge still requires `full-suite` and `e2e-full-suite` on the exact head.
+
## Dependency-audit rule
`dependency-audit` blocks on pushes, schedules, manual runs and pull requests
diff --git a/scripts/ci-reuse.ts b/scripts/ci-reuse.ts
index 0788ec76b0..493bc84bd5 100644
--- a/scripts/ci-reuse.ts
+++ b/scripts/ci-reuse.ts
@@ -32,6 +32,11 @@
* to --summary, and the reuse record (REUSE_FILE, read by
* scripts/release-gate.ts) to --record. Exit 0 whenever a decision was
* written (a fallback is not an error), 2 on usage.
+ *
+ * bun scripts/ci-reuse.ts --gate --sha <HEAD> --tree <tree> --repo <owner/name> [--wait-minutes N] [--poll-seconds S]
+ *
+ * Remote-gate reuse (decideGate, runGate): scripts/pregate-remote-gate.sh asks whether Test and E2E already
+ * passed their complete full runs on this exact tree; exit 0 reuse, 3 a full run on the tree failed, 4 no reuse.
*/
import { appendFileSync, writeFileSync } from 'node:fs';
import { fetchArtifactJson, inert, restClient, type GitHubClient } from './lib/gh-issue.ts';
@@ -109,19 +114,8 @@ export async function decide(deps: ReuseDeps, p: Profile, sha: string, repo: str
if (!why) return accept(run, tree, rejected, 'the merged head');
rejected.push({ run_id: run.id, ...why });
}
- const tried = new Set(primary.map(r => r.id));
- const heads = new Map<string, string>();
- for (const run of (await deps.recentSuccessfulRuns(p.file)).sort((a, b) => b.id - a.id)) {
- if (tried.has(run.id) || !isCandidate(run) || run.head_sha === head || run.head_sha === sha || run.head_repository?.full_name !== repo) continue;
- if (!heads.has(run.head_sha)) {
- if (heads.size >= MAX_FALLBACK_HEADS) continue;
- heads.set(run.head_sha, (await deps.commit(run.head_sha)).tree);
- }
- if (heads.get(run.head_sha) !== tree) continue;
- const why = await reject(deps, p, run, repo, tree, tree);
- if (!why) return accept(run, tree, rejected, `head ${run.head_sha.slice(0, 12)} (same tree as the merged head)`);
- rejected.push({ run_id: run.id, ...why });
- }
+ const other = await sameTreeRun(deps, p, repo, tree, new Set(primary.map(r => r.id)), [head, sha], rejected);
+ if (other) return accept(other, tree, rejected, `head ${other.head_sha.slice(0, 12)} (same tree as the merged head)`);
} catch (e) {
return { reuse: false, reason: 'api_error', detail: `reading the push commit, runs, jobs or artifacts failed: ${e instanceof Error ? e.message : String(e)}`, tree, rejected };
}
@@ -129,10 +123,103 @@ export async function decide(deps: ReuseDeps, p: Profile, sha: string, repo: str
return { reuse: false, reason: rejected[0]!.reason, detail: rejected.map(r => r.detail).join('; '), tree, rejected };
}
+/** The newest passed full run of `p` on another head whose commit tree is `tree` (at most MAX_FALLBACK_HEADS heads read). */
+async function sameTreeRun(deps: ReuseDeps, p: Profile, repo: string, tree: string, tried: Set<number>, skipShas: string[], rejected: Rejection[]): Promise<FullRun | undefined> {
+ const heads = new Map<string, string>();
+ for (const run of (await deps.recentSuccessfulRuns(p.file)).sort((a, b) => b.id - a.id)) {
+ if (tried.has(run.id) || !isCandidate(run) || skipShas.includes(run.head_sha) || run.head_repository?.full_name !== repo) continue;
+ if (!heads.has(run.head_sha)) {
+ if (heads.size >= MAX_FALLBACK_HEADS) continue;
+ heads.set(run.head_sha, (await deps.commit(run.head_sha)).tree);
+ }
+ if (heads.get(run.head_sha) !== tree) continue;
+ const why = await reject(deps, p, run, repo, tree, tree);
+ if (!why) return run;
+ rejected.push({ run_id: run.id, ...why });
+ }
+ return undefined;
+}
+
function accept(run: FullRun, tree: string, rejected: Rejection[], where: string): Decision {
return { reuse: true, reason: 'identical_tree_full_run', detail: `run ${run.id} attempt ${run.run_attempt} on ${where} ran the complete full suite on tree ${tree}`, tree, source: { run_id: run.id, run_attempt: run.run_attempt, sha: run.head_sha }, rejected };
}
+/**
+ * Remote-gate reuse (scripts/pregate-remote-gate.sh, docs/ci-event-parity.md#remote-gate-reuse): GitHub's full
+ * profile is the source of truth, so a clean working tree whose commit `head` (or another head with the same tree)
+ * already passed the complete full run of a workflow needs no Ubicloud gate. Same fail-closed checks as a push:
+ * the run's latest attempt and its full-suite job succeeded and its tested-tree artifact records `tree`. Only
+ * the identical tree reuses; a stamp-only difference goes through gstack-pregate's stamp rule, never through here.
+ * The newest run on `head` decides: in progress waits, a failure refuses (no receipt, no VMs), cancelled or
+ * not a full run looks further back; anything else is `none` and the caller runs the VM gate.
+ */
+export type GateState = 'reuse' | 'pending' | 'failed' | 'none';
+export interface GateDecision { state: GateState; detail: string; tests?: number; failed?: number; url?: string }
+export interface GateDeps extends ReuseDeps {
+ /** Executed (pass, fail) test identities in the run's executed-identities artifact, or why they cannot be read. */
+ executed(runId: number): Promise<{ tests: number; failed: number } | { reason: string }>;
+}
+
+export async function decideGate(deps: GateDeps, p: Profile, head: string, tree: string, repo: string): Promise<GateDecision> {
+ const rejected: Rejection[] = [];
+ try {
+ const headTree = (await deps.commit(head)).tree;
+ if (headTree !== tree) return { state: 'none', detail: `${p.workflow}: commit ${head} has tree ${headTree} on GitHub, not the working tree ${tree}` };
+ const primary = (await deps.runsForHead(p.file, head)).filter(r => r.head_sha === head && isCandidate(r)).sort((a, b) => b.id - a.id);
+ for (const run of primary) {
+ const why = await reject(deps, p, run, repo, tree, headTree);
+ if (!why) return gateAccept(deps, p, run, tree, repo);
+ if (why.reason === 'run_in_progress') return { state: 'pending', detail: `${p.workflow} ${why.detail}` };
+ if (why.reason === 'run_failed' || why.reason === 'full_suite_not_success') return { state: 'failed', detail: `${p.workflow} ${why.detail}`, url: runUrl(repo, run) };
+ rejected.push({ run_id: run.id, ...why });
+ }
+ const other = await sameTreeRun(deps, p, repo, tree, new Set(primary.map(r => r.id)), [head], rejected);
+ if (other) return gateAccept(deps, p, other, tree, repo);
+ } catch (e) {
+ return { state: 'none', detail: `${p.workflow}: reading commits, runs, jobs or artifacts failed: ${e instanceof Error ? e.message : String(e)}` };
+ }
+ return { state: 'none', detail: rejected.length ? `${p.workflow}: ${rejected.map(r => r.detail).join('; ')}` : `${p.workflow}: no workflow_dispatch or push run on ${head} or on another head with tree ${tree}` };
+}
+
+const runUrl = (repo: string, run: FullRun) => `https://github.com/${repo}/actions/runs/${run.id}/attempts/${run.run_attempt}`;
+
+async function gateAccept(deps: GateDeps, p: Profile, run: FullRun, tree: string, repo: string): Promise<GateDecision> {
+ const counts = await deps.executed(run.id);
+ if ('reason' in counts) return { state: 'none', detail: `${p.workflow}: ${counts.reason}` };
+ if (counts.tests <= 0) return { state: 'none', detail: `${p.workflow}: run ${run.id}'s executed-identities artifact lists no executed test` };
+ return { state: 'reuse', detail: `${p.workflow} run ${runUrl(repo, run)} passed the complete full suite on tree ${tree}`, url: runUrl(repo, run), ...counts };
+}
+
+/**
+ * The gate's decision over Test and E2E, waiting while a full run on the tree is still in progress. Prints progress
+ * at least every poll (gstack's stall timeout kills a silent gate) and, on reuse, a last line
+ * `GBRAIN_REUSE_RESULT tests=<n> failed=<n>`. Exit codes: 0 reuse, 3 a full run on this tree failed, 4 no reuse.
+ */
+export async function runGate(deps: GateDeps, o: { head: string; tree: string; repo: string; waitMs: number; pollMs: number; log: (line: string) => void; sleep?: (ms: number) => Promise<unknown>; now?: () => number }): Promise<number> {
+ const now = o.now ?? Date.now;
+ const deadline = now() + o.waitMs;
+ for (;;) {
+ const ds = await Promise.all([PROFILES.test, PROFILES.e2e].map(p => decideGate(deps, p, o.head, o.tree, o.repo)));
+ const failed = ds.find(d => d.state === 'failed');
+ if (failed) {
+ o.log(`[ci-reuse] gate: ${failed.detail} (${failed.url}); the full run on this tree failed, so no receipt and no Ubicloud gate`);
+ return 3;
+ }
+ if (ds.every(d => d.state === 'reuse')) {
+ for (const d of ds) o.log(`[ci-reuse] gate: ${d.detail} (${d.tests} tests, ${d.failed} failed)`);
+ o.log(`GBRAIN_REUSE_RESULT tests=${ds.reduce((n, d) => n + d.tests!, 0)} failed=${ds.reduce((n, d) => n + d.failed!, 0)}`);
+ return 0;
+ }
+ const pending = ds.filter(d => d.state === 'pending');
+ if (!pending.length || now() >= deadline) {
+ for (const d of ds.filter(x => x.state !== 'reuse')) o.log(`[ci-reuse] gate: no reuse: ${inert(d.detail, 600)}`);
+ return 4;
+ }
+ o.log(`[ci-reuse] gate: waiting for ${pending.map(d => d.detail).join('; ')} (${Math.ceil((deadline - now()) / 60_000)} min left)`);
+ await (o.sleep ?? Bun.sleep)(o.pollMs);
+ }
+}
+
/**
* Every job of one attempt. The jobs API pages at up to 100, but GitHub has answered a 100-job page of a rerun attempt
* with a 502 on every retry while 30-job pages of the same attempt answered (Release run 39712021085, v0.60.361.0), so a
@@ -174,6 +261,19 @@ export function apiDeps(client: GitHubClient): ReuseDeps {
};
}
+export function gateDeps(client: GitHubClient): GateDeps {
+ return {
+ ...apiDeps(client),
+ async executed(runId) {
+ const got = await fetchArtifactJson(client, runId, 'executed-identities', 'executed-identities.json');
+ if (got.reason) return { reason: got.reason };
+ const ids = (got.value as { head?: { identities?: Array<{ status?: string }> } } | null)?.head?.identities;
+ if (!Array.isArray(ids)) return { reason: `run ${runId}'s executed-identities.json has no head.identities list` };
+ return { tests: ids.filter(i => i.status === 'pass' || i.status === 'fail').length, failed: ids.filter(i => i.status === 'fail').length };
+ },
+ };
+}
+
/** The validated source run of a reuse record, or why the record cannot prove one. */
export function checkRecord(value: unknown, workflow: string, pushSha: string): { run_id: number; run_attempt: number; sha: string } | string {
const rec = value as Record<string, unknown> | null | undefined;
@@ -205,7 +305,16 @@ export function writeDecision(d: Decision, p: Profile, opts: { repo: string; sha
return line;
}
-if (import.meta.main) {
+if (import.meta.main && process.argv.includes('--gate')) {
+ const flag = (name: string) => { const i = process.argv.indexOf(name); return i >= 0 ? process.argv[i + 1] : undefined; };
+ const head = flag('--sha') ?? '', tree = flag('--tree') ?? '', repo = flag('--repo') ?? '';
+ const minutes = Number(flag('--wait-minutes') ?? '90'), poll = Number(flag('--poll-seconds') ?? '60');
+ if (!SHA.test(head) || !SHA.test(tree) || !/^[\w.-]+\/[\w.-]+$/.test(repo) || !(minutes >= 0) || !(poll > 0)) {
+ console.log('Usage: bun scripts/ci-reuse.ts --gate --sha <40-hex HEAD> --tree <40-hex tree> --repo <owner/name> [--wait-minutes 90] [--poll-seconds 60]');
+ process.exit(2);
+ }
+ process.exit(await runGate(gateDeps(restClient(repo)), { head, tree, repo, waitMs: minutes * 60_000, pollMs: poll * 1000, log: line => console.log(line) }));
+} else if (import.meta.main) {
const flag = (name: string) => { const i = process.argv.indexOf(name); return i >= 0 ? process.argv[i + 1] : undefined; };
const key = flag('--workflow') ?? '';
const sha = flag('--sha') ?? '';
diff --git a/scripts/pregate-remote-gate.sh b/scripts/pregate-remote-gate.sh
index e64650f333..91ba8af69a 100755
--- a/scripts/pregate-remote-gate.sh
+++ b/scripts/pregate-remote-gate.sh
@@ -9,7 +9,18 @@
#
# PREGATE_REMOTE: tree=<tree> tests=<n> failed=<n>
#
-# It runs the development form of the Ubicloud gate, `bun run ci:ubicloud:diff`:
+# GitHub's full profile is the source of truth (docs/ci-event-parity.md#remote-gate-reuse):
+# when the working tree is clean and Test and E2E already passed their complete
+# full runs on this exact tree (`bun scripts/ci-reuse.ts --gate`: full-suite and
+# e2e-full-suite green in the latest attempt, tested-tree artifacts naming the
+# tree), the receipt comes from those runs and no VM starts. A full run on the
+# tree still in progress is waited for; one that failed refuses (exit 1, no
+# receipt, no VMs). <tests> and <failed> are then the runs' executed identities.
+# Any other tree, a stamp-only one included, runs the gate below; gstack's
+# stamp rule (`gstack-pregate clear --receipt`) handles a stamp-only head.
+# GBRAIN_PREGATE_REUSE=0 always runs the gate.
+#
+# Otherwise it runs the development form of the Ubicloud gate, `bun run ci:ubicloud:diff`:
# a doc-only tree runs the doc checks and gitleaks, any other tree every lane,
# a superset of whatever pregate selected, so PREGATE_FILES_FILE is not read.
# The full gate (`bun run ci:ubicloud`) runs once per PR, at handoff, outside
@@ -27,7 +38,9 @@
# environment. Extra arguments go to ci:ubicloud (for example --vms 8).
# Spend is not reported (no spent=), so gstack counts it at the cap.
#
-# Seam: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`
+# Seams: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`,
+# GBRAIN_PREGATE_REUSE_CMD replaces `bun scripts/ci-reuse.ts --gate`, and
+# GBRAIN_PREGATE_REPO names owner/repo instead of the origin remote
# (test/scripts/pregate-remote-gate.test.ts).
set -uo pipefail
@@ -57,20 +70,37 @@ fi
log=$(mktemp "${TMPDIR:-/tmp}/pregate-remote-gate-log-XXXXXX") || fail "cannot create a log file"
trap 'rm -f "$log"' EXIT
-gate_cmd=${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff}
-# shellcheck disable=SC2086
-env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL \
- bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"
-rc=${PIPESTATUS[0]}
+reused=""
+repo=${GBRAIN_PREGATE_REPO:-$(git remote get-url origin 2>/dev/null | sed -E 's#\.git$##' | sed -nE 's#^.*[/:]([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)$#\1#p')}
+if [ "${GBRAIN_PREGATE_REUSE:-1}" != 0 ] && [ -n "$repo" ] && [ "$before" = "$(git rev-parse 'HEAD^{tree}' 2>/dev/null)" ]; then
+ reuse_cmd=${GBRAIN_PREGATE_REUSE_CMD:-bun scripts/ci-reuse.ts --gate}
+ bash -c "$reuse_cmd \"\$@\"" pregate-remote-gate --sha "$(git rev-parse HEAD)" --tree "$before" --repo "$repo" 2>&1 | tee "$log"
+ case ${PIPESTATUS[0]} in
+ 0) reused=$(sed -nE 's/^GBRAIN_REUSE_RESULT tests=([0-9]+) failed=([0-9]+)$/\1 \2/p' "$log" | tail -1) ;;
+ 3) fail "a full run on tree $before failed; fix it or rerun it, no receipt" 1 ;;
+ esac
+fi
+
+if [ -n "$reused" ]; then
+ totals=$reused
+ rc=0
+ [ "${reused#* }" = 0 ] || rc=1
+else
+ gate_cmd=${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff}
+ # shellcheck disable=SC2086
+ env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL \
+ bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"
+ rc=${PIPESTATUS[0]}
+ totals=$(awk '
+ /^=== ci-ubicloud summary ===$/ { seen = 1; next }
+ seen && /^ +[a-z0-9-]+ +[0-9]+ passed, [0-9]+ failed/ { passed += $2; failed += $4; rows++ }
+ END { if (rows) print passed + failed, failed }
+ ' "$log")
+fi
after=$(content_tree) || fail "cannot fingerprint the working tree after the gate"
[ "$after" = "$before" ] || fail "the working tree changed during the gate ($before -> $after); no receipt"
-totals=$(awk '
- /^=== ci-ubicloud summary ===$/ { seen = 1; next }
- seen && /^ +[a-z0-9-]+ +[0-9]+ passed, [0-9]+ failed/ { passed += $2; failed += $4; rows++ }
- END { if (rows) print passed + failed, failed }
-' "$log")
[ -n "$totals" ] || fail "the gate printed no ci-ubicloud summary (exit $rc); no receipt" "$([ "$rc" -ne 0 ] && echo "$rc" || echo 1)"
read -r tests failed <<<"$totals"
diff --git a/test/scripts/ci-reuse.test.ts b/test/scripts/ci-reuse.test.ts
index 8f87205216..74b7d441a5 100644
--- a/test/scripts/ci-reuse.test.ts
+++ b/test/scripts/ci-reuse.test.ts
@@ -12,7 +12,7 @@ import { afterAll, describe, expect, test } from 'bun:test';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
-import { apiDeps, attemptJobs, checkRecord, decide, PROFILES, writeDecision, type Commit, type JobRow, type FullRun, type ReuseDeps } from '../../scripts/ci-reuse.ts';
+import { apiDeps, attemptJobs, checkRecord, decide, decideGate, PROFILES, runGate, writeDecision, type Commit, type GateDeps, type JobRow, type FullRun, type ReuseDeps } from '../../scripts/ci-reuse.ts';
import { GET_RETRY_DELAYS_MS, restClient, type GitHubClient } from '../../scripts/lib/gh-issue.ts';
const REPO = 'owner-example/repo-example';
@@ -304,3 +304,96 @@ describe('ci-reuse outputs', () => {
expect(typeof checkRecord(undefined, 'Test', M)).toBe('string');
});
});
+
+// Remote-gate reuse (scripts/pregate-remote-gate.sh): the Ubicloud gate's receipt comes from GitHub's complete
+// full runs of Test and E2E on the identical tree, or the gate runs. A code change after full-ci never reuses.
+describe('ci-reuse remote-gate decision', () => {
+ const A = '5'.repeat(40);
+ const B = '6'.repeat(40);
+ const T2 = 'c'.repeat(40);
+ const files = { 'test.yml': PROFILES.test, 'e2e.yml': PROFILES.e2e } as const;
+ type File = keyof typeof files;
+ const run = (file: File, over: Partial<FullRun> = {}): FullRun & { file: File } => ({ ...prRun({ id: file === 'test.yml' ? 600 : 700, head_sha: A, ...over }), file });
+ const tree = (r: FullRun & { file: File }, over: Record<string, unknown> = {}) => ({ ...treeJson(r, { workflow: files[r.file].workflow }), ...over });
+
+ function gate(o: { commits?: Record<string, Commit>; runs: Array<FullRun & { file: File }>; artifacts?: Record<number, unknown>; jobs?: Record<string, JobRow[]>; executed?: Record<number, { tests: number; failed: number } | { reason: string }> }): GateDeps {
+ const commits = o.commits ?? { [A]: { tree: T, parents: [BASE] } };
+ const art = o.artifacts ?? Object.fromEntries(o.runs.map(r => [r.id, tree(r)]));
+ return {
+ async commit(sha) { if (!commits[sha]) throw new Error(`GitHub API GET git/commits/${sha} -> 404`); return commits[sha]!; },
+ async runsForHead(file, sha) { return o.runs.filter(r => r.file === file && r.head_sha === sha); },
+ async recentSuccessfulRuns(file) { return o.runs.filter(r => r.file === file && r.conclusion === 'success'); },
+ async jobs(runId, attempt) {
+ const r = o.runs.find(x => x.id === runId)!;
+ return o.jobs?.[`${runId}/${attempt}`] ?? [{ name: files[r.file].job, status: 'completed', conclusion: 'success' }];
+ },
+ async artifact(runId, name) {
+ const r = o.runs.find(x => x.id === runId)!;
+ expect(name).toBe(files[r.file].artifact);
+ return art[runId] === undefined ? { reason: `run ${runId} has no '${name}' artifact` } : { value: art[runId] };
+ },
+ async executed(runId) { return o.executed?.[runId] ?? { tests: runId, failed: 0 }; },
+ };
+ }
+ const go = async (deps: GateDeps, head = A, t0 = T, waitMs = 0) => {
+ const lines: string[] = [];
+ let clock = 0;
+ const code = await runGate(deps, { head, tree: t0, repo: REPO, waitMs, pollMs: 60_000, log: l => lines.push(l), sleep: async ms => { clock += ms; }, now: () => clock });
+ return { code, lines, last: lines.at(-1) };
+ };
+
+ test('Test and E2E full runs that passed on this exact tree are the receipt: summed executed tests, no gate', async () => {
+ const r = await go(gate({ runs: [run('test.yml'), run('e2e.yml')] }));
+ expect(r.code).toBe(0);
+ expect(r.last).toBe('GBRAIN_REUSE_RESULT tests=1300 failed=0');
+ expect(r.lines.join('\n')).toContain(`https://github.com/${REPO}/actions/runs/600/attempts/1`);
+ });
+
+ test('a code change after full-ci is refused: the new head has its own tree and no full run of it, so the gate runs', async () => {
+ const deps = gate({ commits: { [A]: { tree: T, parents: [BASE] }, [B]: { tree: T2, parents: [A] } }, runs: [run('test.yml'), run('e2e.yml')] });
+ expect(await decideGate(deps, PROFILES.test, B, T2, REPO)).toMatchObject({ state: 'none' });
+ const r = await go(deps, B, T2);
+ expect(r.code).toBe(4);
+ expect(r.lines.join('\n')).not.toContain('GBRAIN_REUSE_RESULT');
+ // An unpushed code change: GitHub has no such commit, so nothing reuses either.
+ expect((await go(gate({ runs: [run('test.yml'), run('e2e.yml')] }), B, T2)).code).toBe(4);
+ });
+
+ test('a working tree that is not the head commit tree on GitHub never reuses', async () => {
+ expect(await decideGate(gate({ runs: [run('test.yml')] }), PROFILES.test, A, T2, REPO)).toMatchObject({ state: 'none' });
+ });
+
+ test('a failed newest full run on the head refuses (exit 3) even when an older run on it passed', async () => {
+ const deps = gate({ runs: [run('test.yml', { id: 610, conclusion: 'failure' }), run('test.yml'), run('e2e.yml')] });
+ const r = await go(deps);
+ expect(r.code).toBe(3);
+ expect(r.last).toContain('runs/610/attempts/1');
+ });
+
+ test('a full run in progress on the head is waited for, and a timeout falls back to the gate', async () => {
+ let polls = 0;
+ const live = run('test.yml', { status: 'in_progress', conclusion: null });
+ const deps = gate({ runs: [live, run('e2e.yml')] });
+ const slow = { ...deps, async runsForHead(file: string, sha: string) { if (file === 'test.yml' && ++polls >= 3) Object.assign(live, { status: 'completed', conclusion: 'success' }); return deps.runsForHead(file, sha); } };
+ const r = await go(slow, A, T, 10 * 60_000);
+ expect(r.code).toBe(0);
+ expect(r.lines.filter(l => l.includes('waiting for')).length).toBe(2);
+ const stuck = await go(gate({ runs: [run('test.yml', { status: 'queued', conclusion: null }), run('e2e.yml')] }), A, T, 2 * 60_000);
+ expect(stuck.code).toBe(4);
+ });
+
+ test('both workflows are required: a missing E2E full run, a cheap artifact or an unreadable executed list falls back', async () => {
+ expect((await go(gate({ runs: [run('test.yml')] }))).code).toBe(4);
+ const t1 = run('test.yml'), e1 = run('e2e.yml');
+ expect((await go(gate({ runs: [t1, e1], artifacts: { 600: tree(t1), 700: tree(e1, { profile: 'cheap' }) } }))).code).toBe(4);
+ expect((await go(gate({ runs: [t1, e1], executed: { 700: { reason: 'no executed-identities artifact' } } }))).code).toBe(4);
+ expect((await go(gate({ runs: [t1, e1], executed: { 700: { tests: 0, failed: 0 } } }))).code).toBe(4);
+ });
+
+ test('a cancelled newest run looks further back; a passed run of the same tree on another head (a merge commit) reuses', async () => {
+ expect((await go(gate({ runs: [run('test.yml', { id: 620, conclusion: 'cancelled' }), run('test.yml'), run('e2e.yml')] }))).code).toBe(0);
+ const M2 = '7'.repeat(40);
+ const deps = gate({ commits: { [A]: { tree: T, parents: [BASE] }, [M2]: { tree: T, parents: [BASE, A] } }, runs: [run('test.yml'), run('e2e.yml')] });
+ expect(await decideGate(deps, PROFILES.test, M2, T, REPO)).toMatchObject({ state: 'reuse', tests: 600 });
+ });
+});
diff --git a/test/scripts/pregate-remote-gate.test.ts b/test/scripts/pregate-remote-gate.test.ts
index 2176b26a31..646e0cdf4a 100644
--- a/test/scripts/pregate-remote-gate.test.ts
+++ b/test/scripts/pregate-remote-gate.test.ts
@@ -11,7 +11,13 @@
* PREGATE_TREE mismatch reaches the gate, or the default gate is anything
* but the development form `bun run ci:ubicloud:diff` (the full gate runs
* once per PR, at handoff).
- * Seam: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`.
+ * Remote-gate reuse: a clean tree whose Test and E2E full runs passed on GitHub
+ * takes its receipt from them (same line, byte for byte) and starts no gate; a
+ * failed full run refuses; a dirty tree, no repository, GBRAIN_PREGATE_REUSE=0
+ * or no reuse runs the gate.
+ * Seams: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`,
+ * GBRAIN_PREGATE_REUSE_CMD replaces `bun scripts/ci-reuse.ts --gate`,
+ * GBRAIN_PREGATE_REPO names the repository.
*/
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
import { spawnSync } from 'node:child_process';
@@ -46,7 +52,7 @@ afterEach(() => rmSync(repo, { recursive: true, force: true }));
function run(gate: string, env: Record<string, string | undefined> = {}) {
const r = spawnSync('bash', [SCRIPT], {
cwd: repo, encoding: 'utf8',
- env: { ...process.env, GIT_CONFIG_NOSYSTEM: '1', UBI_OWNER: 'gbra0test', PREGATE_TREE: undefined, GBRAIN_PREGATE_GATE_CMD: gate, ...env },
+ env: { ...process.env, GIT_CONFIG_NOSYSTEM: '1', UBI_OWNER: 'gbra0test', PREGATE_TREE: undefined, GBRAIN_PREGATE_REPO: undefined, GBRAIN_PREGATE_REUSE: undefined, GBRAIN_PREGATE_REUSE_CMD: undefined, GBRAIN_PREGATE_GATE_CMD: gate, ...env },
});
return { code: r.status, out: r.stdout, err: r.stderr };
}
@@ -125,4 +131,56 @@ describe('pregate-remote-gate.sh', () => {
expect(stale.err).toContain('not the tree pregate selected');
expect(spawnSync('test', ['-e', marker]).status).not.toBe(0);
});
+
+ describe('reuse of GitHub full runs on the identical tree', () => {
+ const GREEN_GATE = `printf '%s\\n' '=== ci-ubicloud summary ===' ' unit 4 passed, 0 failed, 1s compute'`;
+ const reuser = (body: string) => {
+ const file = join(repo, '.git', 'reuse.sh');
+ writeFileSync(file, `echo "reuse-args: $*" > .git/reuse-args\n${body}\n`);
+ return `bash ${file}`;
+ };
+ const ran = (name: string) => spawnSync('test', ['-e', join(repo, '.git', name)]).status === 0;
+ const gateMarking = `touch .git/gate-ran; ${GREEN_GATE}`;
+
+ it('takes the receipt from the full runs, with the gate\'s exact line, and starts no gate', () => {
+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser("echo '[ci-reuse] gate: Test run passed'; echo 'GBRAIN_REUSE_RESULT tests=58853 failed=0'") });
+ const tree = git('rev-parse', 'HEAD^{tree}');
+ expect(r.out.endsWith(`PREGATE_REMOTE: tree=${tree} tests=58853 failed=0\n`)).toBe(true);
+ expect(r.code).toBe(0);
+ expect(ran('gate-ran')).toBe(false);
+ expect(spawnSync('cat', [join(repo, '.git', 'reuse-args')], { encoding: 'utf8' }).stdout.trim())
+ .toBe(`reuse-args: --sha ${git('rev-parse', 'HEAD')} --tree ${tree} --repo owner-example/repo-example`);
+ });
+
+ it('a full run on the tree that failed refuses: exit 1, no receipt, no gate', () => {
+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser('exit 3') });
+ expect(r.out).not.toContain('PREGATE_REMOTE:');
+ expect(r.err).toContain('failed');
+ expect(r.code).toBe(1);
+ expect(ran('gate-ran')).toBe(false);
+ });
+
+ it('no reuse runs the gate, whose summary is the receipt', () => {
+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser('exit 4') });
+ expect(ran('gate-ran')).toBe(true);
+ expect(r.out.trim().split('\n').at(-1)).toBe(`PREGATE_REMOTE: tree=${contentTree()} tests=4 failed=0`);
+ expect(r.code).toBe(0);
+ });
+
+ it('a dirty tree, GBRAIN_PREGATE_REUSE=0 or no repository never asks for reuse', () => {
+ const env = { GBRAIN_PREGATE_REUSE_CMD: reuser("echo 'GBRAIN_REUSE_RESULT tests=9 failed=0'") };
+ run(gateMarking, { ...env, GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE: '0' });
+ run(gateMarking, env);
+ writeFileSync(join(repo, 'edited.txt'), 'not committed\n');
+ const dirty = run(gateMarking, { ...env, GBRAIN_PREGATE_REPO: 'owner-example/repo-example' });
+ expect(ran('reuse-args')).toBe(false);
+ expect(dirty.out.trim().split('\n').at(-1)).toBe(`PREGATE_REMOTE: tree=${contentTree()} tests=4 failed=0`);
+ });
+
+ it('reads the repository from the origin remote', () => {
+ git('remote', 'add', 'origin', 'https://git.example.invalid/owner-example/repo-example.git');
+ run(gateMarking, { GBRAIN_PREGATE_REUSE_CMD: reuser('exit 4') });
+ expect(spawnSync('cat', [join(repo, '.git', 'reuse-args')], { encoding: 'utf8' }).stdout).toContain('--repo owner-example/repo-example');
+ });
+ });
});