Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
ci: the remote gate reuses passed GitHub full runs of the identical tree (GBRA-135 audit 2, L1)
garrytan32960d6ab0a1
diff --git a/docs/TESTING.md b/docs/TESTING.mdindex a02cce0ea2..cb75989416 100644--- a/docs/TESTING.md+++ b/docs/TESTING.md@@ -19,7 +19,7 @@ bun install --frozen-lockfile && bun run test && bun run verifyis CI's guard battery (about a minute). For E2E, follow [E2E test DB lifecycle](#e2e-test-db-lifecycle-always-follow-this) andrun `DATABASE_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test bun run test:e2e`.`bun run ci:local` runs the whole CI gate in Docker; `bun run ci:ubicloud` runs-it on ephemeral VMs in about five minutes.+it on ephemeral VMs.A failing `bun run test` writes every failure block to`.context/test-failures.log`, one line per shard to `.context/test-summary.txt`@@ -415,7 +415,7 @@ Test command tiers, each with a clear scope:| `bun run verify` | CI's authoritative pre-test gate set, fanned out by `scripts/run-verify-parallel.sh` through a bounded worker pool (default `detect_cpus`; override `GBRAIN_VERIFY_MAX_PARALLEL`) with the heavy checks ordered first (typecheck, the two compile-embed checks, admin build, fuzz bundles, whole-tree greps), then the self-timed `SOLO_CHECKS` alone ([why](operations/verify-and-nightly-e2e.md#verify-solo-checks)). Two evals ride the unit lane instead ([why](operations/verify-and-nightly-e2e.md#evals-in-the-unit-lane)). The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~65-85s | Before pushing; before `/ship`. || `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. Smart e2e runs only when `DATABASE_URL` is set and propagates its failure; otherwise it prints a skip notice to stderr. Use `ci:local` to provision the databases and require PgBouncer execution. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. || `bun run ci:local` | Independent host gitleaks scans, then frozen dependencies, guards/typecheck, the complete serial and slow lanes, and four unit/E2E shards inside Docker. Each E2E shard has its own pgvector database; selected PgBouncer tests must execute against the transaction-mode pooler. Unit, serial, and slow lanes have database URL overrides unset. Any failed stage fails the command. Complete shard logs survive container teardown under `.context/ci-local-shards/`. `ci:local:diff` runs only gitleaks and the doc checks on a doc-only diff and the full gate otherwise; `--no-shard` runs unit/E2E sequentially. Doc-only diffs still require successful gitleaks scans. | Depends on the full corpus | Full local gate before shipping. |-| `bun run ci:ubicloud` | The `ci:local` lanes (gitleaks, guards/typecheck, serial, slow, unit, all E2E with required PgBouncer execution) fanned out across ephemeral Ubicloud VMs from one heaviest-first work queue; `ci:ubicloud:diff` takes the same doc-only fast path as `ci:local:diff`. Needs `UBICLOUD_API_KEY` or `UBICLOUD_API_TOKEN`, no local Docker. See "Ubicloud fan-out" below. | ~5 min | Full gate before shipping when a Ubicloud token is available. |+| `bun run ci:ubicloud` | The `ci:local` lanes (gitleaks, guards/typecheck, serial, slow, unit, all E2E with required PgBouncer execution) fanned out across ephemeral Ubicloud VMs; `ci:ubicloud:diff` takes the same doc-only fast path as `ci:local:diff`. Needs `UBICLOUD_API_KEY` or `UBICLOUD_API_TOKEN`, no local Docker. See "Ubicloud fan-out" below. | ~5 min | Full gate before shipping when a Ubicloud token is available. || `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. || `bun run test:serial` | The `*.serial.test.ts` set listed in `scripts/serial-files.tsv`, one Bun process per file, run through a pool of concurrent per-file processes (`min(detect_cpus, 4)`, memory-adapted) heaviest-first from `scripts/serial-weights.json` (absent weights fall back to the corpus p75; scheduling only). The `EXCLUSIVE_FILES` in `scripts/run-serial-tests.sh` (machine-global state or timing, at most 4) run sequentially after the pool, on shard 1 only. Per-test timeout 120s; each pooled file is killed at 300s. `SHARD=N/M` partitions pooled files by duration. Externally killed files get one sequential rescue run. Knobs: `GBRAIN_SERIAL_POOL=N` (`1` is fully sequential), `GBRAIN_SERIAL_FILE_TIMEOUT`. | a few minutes at pool=4 | Debugging quarantined files; CI's serial-tests job. || `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential within a shard; `SHARD=N/M` fans out against separate databases (ci-local runs 4 containers). Activates the PGLite snapshot like every other runner (per-file cold-path opt-outs where the test asserts the path TO post-initSchema state), exporting it as an ABSOLUTE path so CLI children spawned with varying cwd still find it. | ~5-10min | Pre-ship; nightly. |@@ -459,9 +459,7 @@ from one heaviest-first work queue; every VM is destroyed on exit. VM setup, schand flags are in [scripts/ubicloud/README.md](../scripts/ubicloud/README.md).VMs are named `ubirun-<owner>-<epoch>-<suffix>`; set `UBI_OWNER` to your thread-code. Teardown destroys every VM the run asked for, including creates still in-flight, and stale-VM sweeps run only with `UBI_GC_HOURS` set and only on your-own VMs. `scripts/ubicloud/ubi-runner.sh usage` shows vCPUs by owner.+code. `scripts/ubicloud/ubi-runner.sh usage` shows vCPUs by owner.A sleeping machine kills the run with no signal, so teardown never runs: on Capy, run`ci:ubicloud` as a watched background operation. After any interrupted run, check`ubi-runner.sh list --mine` and reap leftovers with `down NAME` (or `gc HOURS`, own VMs only).@@ -469,6 +467,9 @@ A sleeping machine kills the run with no signal, so teardown never runs: on CapyThe project quota is 512 vCPUs, shared with PR CI; a full gate takes 64, or 160 when it is free.In a multi-lane wave, lanes run `ci:ubicloud:diff` or targeted suites (`buntest <files>`, `--lanes`); only the integrator runs the full gate.+GitHub full-ci is the source of truth: `scripts/pregate-remote-gate.sh` takes its receipt+from the passed Test and E2E full runs of the identical clean tree; any other+tree runs the gate ([remote-gate reuse](ci-event-parity.md#remote-gate-reuse)).### E2E backend matrix@@ -739,7 +740,7 @@ Selected E2E direct-only run would duplicate it), persistence-validation.yml andowner. Locally, `ci:local:diff` and `ci:ubicloud:diff` run gitleaks and`scripts/ci-doc-checks.sh` (llms freshness, KEY_FILES byte caps, documentedpaths, skill references, privacy guards) on a doc-only diff and the full gate-otherwise, printing "E2E narrowing is retired".+otherwise.### Refactor wave 1 goldens@@ -1268,7 +1269,7 @@ there even though they pass on Linux and macOS.### CI vs local: intentionally divergent file sets-- **CI matrix** (`.github/workflows/test.yml`) runs 22 `scripts/test-shard.sh` shards, two per runner, split by weight-aware LPT bin-packing (`scripts/sharding.ts`; unweighted files get the p75 weight) and includes `*.slow.test.ts` files without a dedicated job, plus keyless-allowlisted `evals/**/*.test.ts` (`test/scripts/evals-collection.test.ts`). Shards keep planned file order (`test-shard.sh --dry-run-list <i> 18`) and print their predicted weight (`sharding.ts --sum`). Every Bun job activates the PGLite schema snapshot (built in-runner, cached across jobs; the runner's hash check is authoritative). Serial files run across four `serial-tests` workers via `bun run test:serial`; `verify`, `typecheck` and the BrainBench memory-conformance gate (`scripts/ci-brainbench-gate.sh`, compared against `evals/brainbench/baselines/main.json`) have their own jobs, all aggregated by `test-status`. `e2e.yml` aggregates through `e2e-status`, with jsonb-parity gating Tier 2's token spend; scheduled runs also require the full-corpus lanes. Both aggregates reject failures, cancellations and unexpected skips. Only a master push reuses results: those of a passed full-suite run of its exact tree (`scripts/ci-reuse.ts`). CI is the ground truth for "did everything pass."+- **CI matrix** (`.github/workflows/test.yml`) runs 22 `scripts/test-shard.sh` shards, two per runner, split by weight-aware LPT bin-packing (`scripts/sharding.ts`; unweighted files get the p75 weight) and includes `*.slow.test.ts` files without a dedicated job, plus keyless-allowlisted `evals/**/*.test.ts` (`test/scripts/evals-collection.test.ts`). Shards keep planned file order (`test-shard.sh --dry-run-list <i> 18`) and print their predicted weight (`sharding.ts --sum`). Every Bun job activates the PGLite schema snapshot (built in-runner, cached across jobs; the runner's hash check is authoritative). Serial files run across four `serial-tests` workers via `bun run test:serial`; `verify`, `typecheck` and the BrainBench memory-conformance gate (`scripts/ci-brainbench-gate.sh`, compared against `evals/brainbench/baselines/main.json`) have their own jobs, all aggregated by `test-status`. `e2e.yml` aggregates through `e2e-status`, with jsonb-parity gating Tier 2's token spend; scheduled runs also require the full-corpus lanes. Both aggregates reject failures, cancellations and unexpected skips. Only a master push and the remote gate reuse results: those of a passed full run of the exact tree (`scripts/ci-reuse.ts`). CI is the ground truth for "did everything pass."- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses the same weighted partitioner as CI and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Each shard runs its complete ordered selection with a fresh Bun process per file, without adding workers. Later groups still run after failures; missing summaries or file-completion evidence fail the shard. Local trades coverage for inner-loop speed; CI catches what local skips.`test/scripts/run-unit-shard.test.ts` pins what the local loop includes, anddiff --git a/docs/ci-event-parity.md b/docs/ci-event-parity.mdindex 3c02c4abf7..5af2d49d5c 100644--- a/docs/ci-event-parity.md+++ b/docs/ci-event-parity.md@@ -99,6 +99,26 @@ reusing push only with a readable `reuse.json` naming the same workflow, thepush head and a valid source run. It prints the reused run per workflow andwrites `test_reused_run_id` / `e2e_reused_run_id`. Schedules never reuse.+## Remote-gate reuse++GitHub's full profile is the source of truth for the Ubicloud gate too.+`scripts/pregate-remote-gate.sh`, the run that clears gstack's `remote_gate`+obligation, first asks `bun scripts/ci-reuse.ts --gate` whether the clean+working tree's commit, or another head with the same tree, already passed the+complete full runs of both Test and E2E under the master-push rules above. If+both did, the receipt comes from them: the same `PREGATE_REMOTE: tree= tests=+failed=` line, with the runs' executed test identities as the counts, and no VM+starts. A full run on the tree still in progress is waited for (90 minutes at+most, a progress line every minute). A failed newest run on the head refuses+with no receipt. Every other case runs the gate: a dirty tree, a commit GitHub+doesn't have, a missing workflow, an API error, or `GBRAIN_PREGATE_REUSE=0`.++Only the identical tree reuses. A head that /ship changed after full-ci has a+new tree, so it needs its own passed full run or gate. A stamp-only difference+reaches the head through gstack's stamp rule (`gstack-pregate clear --receipt`+with the stamp files in `.gstack/ship-policy.json`), never through this check.+Merge still requires `full-suite` and `e2e-full-suite` on the exact head.+## Dependency-audit rule`dependency-audit` blocks on pushes, schedules, manual runs and pull requestsdiff --git a/scripts/ci-reuse.ts b/scripts/ci-reuse.tsindex 0788ec76b0..493bc84bd5 100644--- a/scripts/ci-reuse.ts+++ b/scripts/ci-reuse.ts@@ -32,6 +32,11 @@* to --summary, and the reuse record (REUSE_FILE, read by* scripts/release-gate.ts) to --record. Exit 0 whenever a decision was* written (a fallback is not an error), 2 on usage.+ *+ * bun scripts/ci-reuse.ts --gate --sha <HEAD> --tree <tree> --repo <owner/name> [--wait-minutes N] [--poll-seconds S]+ *+ * Remote-gate reuse (decideGate, runGate): scripts/pregate-remote-gate.sh asks whether Test and E2E already+ * passed their complete full runs on this exact tree; exit 0 reuse, 3 a full run on the tree failed, 4 no reuse.*/import { appendFileSync, writeFileSync } from 'node:fs';import { fetchArtifactJson, inert, restClient, type GitHubClient } from './lib/gh-issue.ts';@@ -109,19 +114,8 @@ export async function decide(deps: ReuseDeps, p: Profile, sha: string, repo: strif (!why) return accept(run, tree, rejected, 'the merged head');rejected.push({ run_id: run.id, ...why });}- const tried = new Set(primary.map(r => r.id));- const heads = new Map<string, string>();- for (const run of (await deps.recentSuccessfulRuns(p.file)).sort((a, b) => b.id - a.id)) {- if (tried.has(run.id) || !isCandidate(run) || run.head_sha === head || run.head_sha === sha || run.head_repository?.full_name !== repo) continue;- if (!heads.has(run.head_sha)) {- if (heads.size >= MAX_FALLBACK_HEADS) continue;- heads.set(run.head_sha, (await deps.commit(run.head_sha)).tree);- }- if (heads.get(run.head_sha) !== tree) continue;- const why = await reject(deps, p, run, repo, tree, tree);- if (!why) return accept(run, tree, rejected, `head ${run.head_sha.slice(0, 12)} (same tree as the merged head)`);- rejected.push({ run_id: run.id, ...why });- }+ const other = await sameTreeRun(deps, p, repo, tree, new Set(primary.map(r => r.id)), [head, sha], rejected);+ if (other) return accept(other, tree, rejected, `head ${other.head_sha.slice(0, 12)} (same tree as the merged head)`);} catch (e) {return { reuse: false, reason: 'api_error', detail: `reading the push commit, runs, jobs or artifacts failed: ${e instanceof Error ? e.message : String(e)}`, tree, rejected };}@@ -129,10 +123,103 @@ export async function decide(deps: ReuseDeps, p: Profile, sha: string, repo: strreturn { reuse: false, reason: rejected[0]!.reason, detail: rejected.map(r => r.detail).join('; '), tree, rejected };}+/** The newest passed full run of `p` on another head whose commit tree is `tree` (at most MAX_FALLBACK_HEADS heads read). */+async function sameTreeRun(deps: ReuseDeps, p: Profile, repo: string, tree: string, tried: Set<number>, skipShas: string[], rejected: Rejection[]): Promise<FullRun | undefined> {+ const heads = new Map<string, string>();+ for (const run of (await deps.recentSuccessfulRuns(p.file)).sort((a, b) => b.id - a.id)) {+ if (tried.has(run.id) || !isCandidate(run) || skipShas.includes(run.head_sha) || run.head_repository?.full_name !== repo) continue;+ if (!heads.has(run.head_sha)) {+ if (heads.size >= MAX_FALLBACK_HEADS) continue;+ heads.set(run.head_sha, (await deps.commit(run.head_sha)).tree);+ }+ if (heads.get(run.head_sha) !== tree) continue;+ const why = await reject(deps, p, run, repo, tree, tree);+ if (!why) return run;+ rejected.push({ run_id: run.id, ...why });+ }+ return undefined;+}+function accept(run: FullRun, tree: string, rejected: Rejection[], where: string): Decision {return { reuse: true, reason: 'identical_tree_full_run', detail: `run ${run.id} attempt ${run.run_attempt} on ${where} ran the complete full suite on tree ${tree}`, tree, source: { run_id: run.id, run_attempt: run.run_attempt, sha: run.head_sha }, rejected };}+/**+ * Remote-gate reuse (scripts/pregate-remote-gate.sh, docs/ci-event-parity.md#remote-gate-reuse): GitHub's full+ * profile is the source of truth, so a clean working tree whose commit `head` (or another head with the same tree)+ * already passed the complete full run of a workflow needs no Ubicloud gate. Same fail-closed checks as a push:+ * the run's latest attempt and its full-suite job succeeded and its tested-tree artifact records `tree`. Only+ * the identical tree reuses; a stamp-only difference goes through gstack-pregate's stamp rule, never through here.+ * The newest run on `head` decides: in progress waits, a failure refuses (no receipt, no VMs), cancelled or+ * not a full run looks further back; anything else is `none` and the caller runs the VM gate.+ */+export type GateState = 'reuse' | 'pending' | 'failed' | 'none';+export interface GateDecision { state: GateState; detail: string; tests?: number; failed?: number; url?: string }+export interface GateDeps extends ReuseDeps {+ /** Executed (pass, fail) test identities in the run's executed-identities artifact, or why they cannot be read. */+ executed(runId: number): Promise<{ tests: number; failed: number } | { reason: string }>;+}++export async function decideGate(deps: GateDeps, p: Profile, head: string, tree: string, repo: string): Promise<GateDecision> {+ const rejected: Rejection[] = [];+ try {+ const headTree = (await deps.commit(head)).tree;+ if (headTree !== tree) return { state: 'none', detail: `${p.workflow}: commit ${head} has tree ${headTree} on GitHub, not the working tree ${tree}` };+ const primary = (await deps.runsForHead(p.file, head)).filter(r => r.head_sha === head && isCandidate(r)).sort((a, b) => b.id - a.id);+ for (const run of primary) {+ const why = await reject(deps, p, run, repo, tree, headTree);+ if (!why) return gateAccept(deps, p, run, tree, repo);+ if (why.reason === 'run_in_progress') return { state: 'pending', detail: `${p.workflow} ${why.detail}` };+ if (why.reason === 'run_failed' || why.reason === 'full_suite_not_success') return { state: 'failed', detail: `${p.workflow} ${why.detail}`, url: runUrl(repo, run) };+ rejected.push({ run_id: run.id, ...why });+ }+ const other = await sameTreeRun(deps, p, repo, tree, new Set(primary.map(r => r.id)), [head], rejected);+ if (other) return gateAccept(deps, p, other, tree, repo);+ } catch (e) {+ return { state: 'none', detail: `${p.workflow}: reading commits, runs, jobs or artifacts failed: ${e instanceof Error ? e.message : String(e)}` };+ }+ return { state: 'none', detail: rejected.length ? `${p.workflow}: ${rejected.map(r => r.detail).join('; ')}` : `${p.workflow}: no workflow_dispatch or push run on ${head} or on another head with tree ${tree}` };+}++const runUrl = (repo: string, run: FullRun) => `https://github.com/${repo}/actions/runs/${run.id}/attempts/${run.run_attempt}`;++async function gateAccept(deps: GateDeps, p: Profile, run: FullRun, tree: string, repo: string): Promise<GateDecision> {+ const counts = await deps.executed(run.id);+ if ('reason' in counts) return { state: 'none', detail: `${p.workflow}: ${counts.reason}` };+ if (counts.tests <= 0) return { state: 'none', detail: `${p.workflow}: run ${run.id}'s executed-identities artifact lists no executed test` };+ return { state: 'reuse', detail: `${p.workflow} run ${runUrl(repo, run)} passed the complete full suite on tree ${tree}`, url: runUrl(repo, run), ...counts };+}++/**+ * The gate's decision over Test and E2E, waiting while a full run on the tree is still in progress. Prints progress+ * at least every poll (gstack's stall timeout kills a silent gate) and, on reuse, a last line+ * `GBRAIN_REUSE_RESULT tests=<n> failed=<n>`. Exit codes: 0 reuse, 3 a full run on this tree failed, 4 no reuse.+ */+export async function runGate(deps: GateDeps, o: { head: string; tree: string; repo: string; waitMs: number; pollMs: number; log: (line: string) => void; sleep?: (ms: number) => Promise<unknown>; now?: () => number }): Promise<number> {+ const now = o.now ?? Date.now;+ const deadline = now() + o.waitMs;+ for (;;) {+ const ds = await Promise.all([PROFILES.test, PROFILES.e2e].map(p => decideGate(deps, p, o.head, o.tree, o.repo)));+ const failed = ds.find(d => d.state === 'failed');+ if (failed) {+ o.log(`[ci-reuse] gate: ${failed.detail} (${failed.url}); the full run on this tree failed, so no receipt and no Ubicloud gate`);+ return 3;+ }+ if (ds.every(d => d.state === 'reuse')) {+ for (const d of ds) o.log(`[ci-reuse] gate: ${d.detail} (${d.tests} tests, ${d.failed} failed)`);+ o.log(`GBRAIN_REUSE_RESULT tests=${ds.reduce((n, d) => n + d.tests!, 0)} failed=${ds.reduce((n, d) => n + d.failed!, 0)}`);+ return 0;+ }+ const pending = ds.filter(d => d.state === 'pending');+ if (!pending.length || now() >= deadline) {+ for (const d of ds.filter(x => x.state !== 'reuse')) o.log(`[ci-reuse] gate: no reuse: ${inert(d.detail, 600)}`);+ return 4;+ }+ o.log(`[ci-reuse] gate: waiting for ${pending.map(d => d.detail).join('; ')} (${Math.ceil((deadline - now()) / 60_000)} min left)`);+ await (o.sleep ?? Bun.sleep)(o.pollMs);+ }+}+/*** Every job of one attempt. The jobs API pages at up to 100, but GitHub has answered a 100-job page of a rerun attempt* with a 502 on every retry while 30-job pages of the same attempt answered (Release run 39712021085, v0.60.361.0), so a@@ -174,6 +261,19 @@ export function apiDeps(client: GitHubClient): ReuseDeps {};}+export function gateDeps(client: GitHubClient): GateDeps {+ return {+ ...apiDeps(client),+ async executed(runId) {+ const got = await fetchArtifactJson(client, runId, 'executed-identities', 'executed-identities.json');+ if (got.reason) return { reason: got.reason };+ const ids = (got.value as { head?: { identities?: Array<{ status?: string }> } } | null)?.head?.identities;+ if (!Array.isArray(ids)) return { reason: `run ${runId}'s executed-identities.json has no head.identities list` };+ return { tests: ids.filter(i => i.status === 'pass' || i.status === 'fail').length, failed: ids.filter(i => i.status === 'fail').length };+ },+ };+}+/** The validated source run of a reuse record, or why the record cannot prove one. */export function checkRecord(value: unknown, workflow: string, pushSha: string): { run_id: number; run_attempt: number; sha: string } | string {const rec = value as Record<string, unknown> | null | undefined;@@ -205,7 +305,16 @@ export function writeDecision(d: Decision, p: Profile, opts: { repo: string; shareturn line;}-if (import.meta.main) {+if (import.meta.main && process.argv.includes('--gate')) {+ const flag = (name: string) => { const i = process.argv.indexOf(name); return i >= 0 ? process.argv[i + 1] : undefined; };+ const head = flag('--sha') ?? '', tree = flag('--tree') ?? '', repo = flag('--repo') ?? '';+ const minutes = Number(flag('--wait-minutes') ?? '90'), poll = Number(flag('--poll-seconds') ?? '60');+ if (!SHA.test(head) || !SHA.test(tree) || !/^[\w.-]+\/[\w.-]+$/.test(repo) || !(minutes >= 0) || !(poll > 0)) {+ console.log('Usage: bun scripts/ci-reuse.ts --gate --sha <40-hex HEAD> --tree <40-hex tree> --repo <owner/name> [--wait-minutes 90] [--poll-seconds 60]');+ process.exit(2);+ }+ process.exit(await runGate(gateDeps(restClient(repo)), { head, tree, repo, waitMs: minutes * 60_000, pollMs: poll * 1000, log: line => console.log(line) }));+} else if (import.meta.main) {const flag = (name: string) => { const i = process.argv.indexOf(name); return i >= 0 ? process.argv[i + 1] : undefined; };const key = flag('--workflow') ?? '';const sha = flag('--sha') ?? '';diff --git a/scripts/pregate-remote-gate.sh b/scripts/pregate-remote-gate.shindex e64650f333..91ba8af69a 100755--- a/scripts/pregate-remote-gate.sh+++ b/scripts/pregate-remote-gate.sh@@ -9,7 +9,18 @@## PREGATE_REMOTE: tree=<tree> tests=<n> failed=<n>#-# It runs the development form of the Ubicloud gate, `bun run ci:ubicloud:diff`:+# GitHub's full profile is the source of truth (docs/ci-event-parity.md#remote-gate-reuse):+# when the working tree is clean and Test and E2E already passed their complete+# full runs on this exact tree (`bun scripts/ci-reuse.ts --gate`: full-suite and+# e2e-full-suite green in the latest attempt, tested-tree artifacts naming the+# tree), the receipt comes from those runs and no VM starts. A full run on the+# tree still in progress is waited for; one that failed refuses (exit 1, no+# receipt, no VMs). <tests> and <failed> are then the runs' executed identities.+# Any other tree, a stamp-only one included, runs the gate below; gstack's+# stamp rule (`gstack-pregate clear --receipt`) handles a stamp-only head.+# GBRAIN_PREGATE_REUSE=0 always runs the gate.+#+# Otherwise it runs the development form of the Ubicloud gate, `bun run ci:ubicloud:diff`:# a doc-only tree runs the doc checks and gitleaks, any other tree every lane,# a superset of whatever pregate selected, so PREGATE_FILES_FILE is not read.# The full gate (`bun run ci:ubicloud`) runs once per PR, at handoff, outside@@ -27,7 +38,9 @@# environment. Extra arguments go to ci:ubicloud (for example --vms 8).# Spend is not reported (no spent=), so gstack counts it at the cap.#-# Seam: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`+# Seams: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`,+# GBRAIN_PREGATE_REUSE_CMD replaces `bun scripts/ci-reuse.ts --gate`, and+# GBRAIN_PREGATE_REPO names owner/repo instead of the origin remote# (test/scripts/pregate-remote-gate.test.ts).set -uo pipefail@@ -57,20 +70,37 @@ filog=$(mktemp "${TMPDIR:-/tmp}/pregate-remote-gate-log-XXXXXX") || fail "cannot create a log file"trap 'rm -f "$log"' EXIT-gate_cmd=${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff}-# shellcheck disable=SC2086-env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL \- bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"-rc=${PIPESTATUS[0]}+reused=""+repo=${GBRAIN_PREGATE_REPO:-$(git remote get-url origin 2>/dev/null | sed -E 's#\.git$##' | sed -nE 's#^.*[/:]([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)$#\1#p')}+if [ "${GBRAIN_PREGATE_REUSE:-1}" != 0 ] && [ -n "$repo" ] && [ "$before" = "$(git rev-parse 'HEAD^{tree}' 2>/dev/null)" ]; then+ reuse_cmd=${GBRAIN_PREGATE_REUSE_CMD:-bun scripts/ci-reuse.ts --gate}+ bash -c "$reuse_cmd \"\$@\"" pregate-remote-gate --sha "$(git rev-parse HEAD)" --tree "$before" --repo "$repo" 2>&1 | tee "$log"+ case ${PIPESTATUS[0]} in+ 0) reused=$(sed -nE 's/^GBRAIN_REUSE_RESULT tests=([0-9]+) failed=([0-9]+)$/\1 \2/p' "$log" | tail -1) ;;+ 3) fail "a full run on tree $before failed; fix it or rerun it, no receipt" 1 ;;+ esac+fi++if [ -n "$reused" ]; then+ totals=$reused+ rc=0+ [ "${reused#* }" = 0 ] || rc=1+else+ gate_cmd=${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff}+ # shellcheck disable=SC2086+ env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL \+ bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"+ rc=${PIPESTATUS[0]}+ totals=$(awk '+ /^=== ci-ubicloud summary ===$/ { seen = 1; next }+ seen && /^ +[a-z0-9-]+ +[0-9]+ passed, [0-9]+ failed/ { passed += $2; failed += $4; rows++ }+ END { if (rows) print passed + failed, failed }+ ' "$log")+fiafter=$(content_tree) || fail "cannot fingerprint the working tree after the gate"[ "$after" = "$before" ] || fail "the working tree changed during the gate ($before -> $after); no receipt"-totals=$(awk '- /^=== ci-ubicloud summary ===$/ { seen = 1; next }- seen && /^ +[a-z0-9-]+ +[0-9]+ passed, [0-9]+ failed/ { passed += $2; failed += $4; rows++ }- END { if (rows) print passed + failed, failed }-' "$log")[ -n "$totals" ] || fail "the gate printed no ci-ubicloud summary (exit $rc); no receipt" "$([ "$rc" -ne 0 ] && echo "$rc" || echo 1)"read -r tests failed <<<"$totals"diff --git a/test/scripts/ci-reuse.test.ts b/test/scripts/ci-reuse.test.tsindex 8f87205216..74b7d441a5 100644--- a/test/scripts/ci-reuse.test.ts+++ b/test/scripts/ci-reuse.test.ts@@ -12,7 +12,7 @@ import { afterAll, describe, expect, test } from 'bun:test';import { mkdtempSync, readFileSync, rmSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';-import { apiDeps, attemptJobs, checkRecord, decide, PROFILES, writeDecision, type Commit, type JobRow, type FullRun, type ReuseDeps } from '../../scripts/ci-reuse.ts';+import { apiDeps, attemptJobs, checkRecord, decide, decideGate, PROFILES, runGate, writeDecision, type Commit, type GateDeps, type JobRow, type FullRun, type ReuseDeps } from '../../scripts/ci-reuse.ts';import { GET_RETRY_DELAYS_MS, restClient, type GitHubClient } from '../../scripts/lib/gh-issue.ts';const REPO = 'owner-example/repo-example';@@ -304,3 +304,96 @@ describe('ci-reuse outputs', () => {expect(typeof checkRecord(undefined, 'Test', M)).toBe('string');});});++// Remote-gate reuse (scripts/pregate-remote-gate.sh): the Ubicloud gate's receipt comes from GitHub's complete+// full runs of Test and E2E on the identical tree, or the gate runs. A code change after full-ci never reuses.+describe('ci-reuse remote-gate decision', () => {+ const A = '5'.repeat(40);+ const B = '6'.repeat(40);+ const T2 = 'c'.repeat(40);+ const files = { 'test.yml': PROFILES.test, 'e2e.yml': PROFILES.e2e } as const;+ type File = keyof typeof files;+ const run = (file: File, over: Partial<FullRun> = {}): FullRun & { file: File } => ({ ...prRun({ id: file === 'test.yml' ? 600 : 700, head_sha: A, ...over }), file });+ const tree = (r: FullRun & { file: File }, over: Record<string, unknown> = {}) => ({ ...treeJson(r, { workflow: files[r.file].workflow }), ...over });++ function gate(o: { commits?: Record<string, Commit>; runs: Array<FullRun & { file: File }>; artifacts?: Record<number, unknown>; jobs?: Record<string, JobRow[]>; executed?: Record<number, { tests: number; failed: number } | { reason: string }> }): GateDeps {+ const commits = o.commits ?? { [A]: { tree: T, parents: [BASE] } };+ const art = o.artifacts ?? Object.fromEntries(o.runs.map(r => [r.id, tree(r)]));+ return {+ async commit(sha) { if (!commits[sha]) throw new Error(`GitHub API GET git/commits/${sha} -> 404`); return commits[sha]!; },+ async runsForHead(file, sha) { return o.runs.filter(r => r.file === file && r.head_sha === sha); },+ async recentSuccessfulRuns(file) { return o.runs.filter(r => r.file === file && r.conclusion === 'success'); },+ async jobs(runId, attempt) {+ const r = o.runs.find(x => x.id === runId)!;+ return o.jobs?.[`${runId}/${attempt}`] ?? [{ name: files[r.file].job, status: 'completed', conclusion: 'success' }];+ },+ async artifact(runId, name) {+ const r = o.runs.find(x => x.id === runId)!;+ expect(name).toBe(files[r.file].artifact);+ return art[runId] === undefined ? { reason: `run ${runId} has no '${name}' artifact` } : { value: art[runId] };+ },+ async executed(runId) { return o.executed?.[runId] ?? { tests: runId, failed: 0 }; },+ };+ }+ const go = async (deps: GateDeps, head = A, t0 = T, waitMs = 0) => {+ const lines: string[] = [];+ let clock = 0;+ const code = await runGate(deps, { head, tree: t0, repo: REPO, waitMs, pollMs: 60_000, log: l => lines.push(l), sleep: async ms => { clock += ms; }, now: () => clock });+ return { code, lines, last: lines.at(-1) };+ };++ test('Test and E2E full runs that passed on this exact tree are the receipt: summed executed tests, no gate', async () => {+ const r = await go(gate({ runs: [run('test.yml'), run('e2e.yml')] }));+ expect(r.code).toBe(0);+ expect(r.last).toBe('GBRAIN_REUSE_RESULT tests=1300 failed=0');+ expect(r.lines.join('\n')).toContain(`https://github.com/${REPO}/actions/runs/600/attempts/1`);+ });++ test('a code change after full-ci is refused: the new head has its own tree and no full run of it, so the gate runs', async () => {+ const deps = gate({ commits: { [A]: { tree: T, parents: [BASE] }, [B]: { tree: T2, parents: [A] } }, runs: [run('test.yml'), run('e2e.yml')] });+ expect(await decideGate(deps, PROFILES.test, B, T2, REPO)).toMatchObject({ state: 'none' });+ const r = await go(deps, B, T2);+ expect(r.code).toBe(4);+ expect(r.lines.join('\n')).not.toContain('GBRAIN_REUSE_RESULT');+ // An unpushed code change: GitHub has no such commit, so nothing reuses either.+ expect((await go(gate({ runs: [run('test.yml'), run('e2e.yml')] }), B, T2)).code).toBe(4);+ });++ test('a working tree that is not the head commit tree on GitHub never reuses', async () => {+ expect(await decideGate(gate({ runs: [run('test.yml')] }), PROFILES.test, A, T2, REPO)).toMatchObject({ state: 'none' });+ });++ test('a failed newest full run on the head refuses (exit 3) even when an older run on it passed', async () => {+ const deps = gate({ runs: [run('test.yml', { id: 610, conclusion: 'failure' }), run('test.yml'), run('e2e.yml')] });+ const r = await go(deps);+ expect(r.code).toBe(3);+ expect(r.last).toContain('runs/610/attempts/1');+ });++ test('a full run in progress on the head is waited for, and a timeout falls back to the gate', async () => {+ let polls = 0;+ const live = run('test.yml', { status: 'in_progress', conclusion: null });+ const deps = gate({ runs: [live, run('e2e.yml')] });+ const slow = { ...deps, async runsForHead(file: string, sha: string) { if (file === 'test.yml' && ++polls >= 3) Object.assign(live, { status: 'completed', conclusion: 'success' }); return deps.runsForHead(file, sha); } };+ const r = await go(slow, A, T, 10 * 60_000);+ expect(r.code).toBe(0);+ expect(r.lines.filter(l => l.includes('waiting for')).length).toBe(2);+ const stuck = await go(gate({ runs: [run('test.yml', { status: 'queued', conclusion: null }), run('e2e.yml')] }), A, T, 2 * 60_000);+ expect(stuck.code).toBe(4);+ });++ test('both workflows are required: a missing E2E full run, a cheap artifact or an unreadable executed list falls back', async () => {+ expect((await go(gate({ runs: [run('test.yml')] }))).code).toBe(4);+ const t1 = run('test.yml'), e1 = run('e2e.yml');+ expect((await go(gate({ runs: [t1, e1], artifacts: { 600: tree(t1), 700: tree(e1, { profile: 'cheap' }) } }))).code).toBe(4);+ expect((await go(gate({ runs: [t1, e1], executed: { 700: { reason: 'no executed-identities artifact' } } }))).code).toBe(4);+ expect((await go(gate({ runs: [t1, e1], executed: { 700: { tests: 0, failed: 0 } } }))).code).toBe(4);+ });++ test('a cancelled newest run looks further back; a passed run of the same tree on another head (a merge commit) reuses', async () => {+ expect((await go(gate({ runs: [run('test.yml', { id: 620, conclusion: 'cancelled' }), run('test.yml'), run('e2e.yml')] }))).code).toBe(0);+ const M2 = '7'.repeat(40);+ const deps = gate({ commits: { [A]: { tree: T, parents: [BASE] }, [M2]: { tree: T, parents: [BASE, A] } }, runs: [run('test.yml'), run('e2e.yml')] });+ expect(await decideGate(deps, PROFILES.test, M2, T, REPO)).toMatchObject({ state: 'reuse', tests: 600 });+ });+});diff --git a/test/scripts/pregate-remote-gate.test.ts b/test/scripts/pregate-remote-gate.test.tsindex 2176b26a31..646e0cdf4a 100644--- a/test/scripts/pregate-remote-gate.test.ts+++ b/test/scripts/pregate-remote-gate.test.ts@@ -11,7 +11,13 @@* PREGATE_TREE mismatch reaches the gate, or the default gate is anything* but the development form `bun run ci:ubicloud:diff` (the full gate runs* once per PR, at handoff).- * Seam: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`.+ * Remote-gate reuse: a clean tree whose Test and E2E full runs passed on GitHub+ * takes its receipt from them (same line, byte for byte) and starts no gate; a+ * failed full run refuses; a dirty tree, no repository, GBRAIN_PREGATE_REUSE=0+ * or no reuse runs the gate.+ * Seams: GBRAIN_PREGATE_GATE_CMD replaces `bun run ci:ubicloud:diff`,+ * GBRAIN_PREGATE_REUSE_CMD replaces `bun scripts/ci-reuse.ts --gate`,+ * GBRAIN_PREGATE_REPO names the repository.*/import { afterEach, beforeEach, describe, expect, it } from 'bun:test';import { spawnSync } from 'node:child_process';@@ -46,7 +52,7 @@ afterEach(() => rmSync(repo, { recursive: true, force: true }));function run(gate: string, env: Record<string, string | undefined> = {}) {const r = spawnSync('bash', [SCRIPT], {cwd: repo, encoding: 'utf8',- env: { ...process.env, GIT_CONFIG_NOSYSTEM: '1', UBI_OWNER: 'gbra0test', PREGATE_TREE: undefined, GBRAIN_PREGATE_GATE_CMD: gate, ...env },+ env: { ...process.env, GIT_CONFIG_NOSYSTEM: '1', UBI_OWNER: 'gbra0test', PREGATE_TREE: undefined, GBRAIN_PREGATE_REPO: undefined, GBRAIN_PREGATE_REUSE: undefined, GBRAIN_PREGATE_REUSE_CMD: undefined, GBRAIN_PREGATE_GATE_CMD: gate, ...env },});return { code: r.status, out: r.stdout, err: r.stderr };}@@ -125,4 +131,56 @@ describe('pregate-remote-gate.sh', () => {expect(stale.err).toContain('not the tree pregate selected');expect(spawnSync('test', ['-e', marker]).status).not.toBe(0);});++ describe('reuse of GitHub full runs on the identical tree', () => {+ const GREEN_GATE = `printf '%s\\n' '=== ci-ubicloud summary ===' ' unit 4 passed, 0 failed, 1s compute'`;+ const reuser = (body: string) => {+ const file = join(repo, '.git', 'reuse.sh');+ writeFileSync(file, `echo "reuse-args: $*" > .git/reuse-args\n${body}\n`);+ return `bash ${file}`;+ };+ const ran = (name: string) => spawnSync('test', ['-e', join(repo, '.git', name)]).status === 0;+ const gateMarking = `touch .git/gate-ran; ${GREEN_GATE}`;++ it('takes the receipt from the full runs, with the gate\'s exact line, and starts no gate', () => {+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser("echo '[ci-reuse] gate: Test run passed'; echo 'GBRAIN_REUSE_RESULT tests=58853 failed=0'") });+ const tree = git('rev-parse', 'HEAD^{tree}');+ expect(r.out.endsWith(`PREGATE_REMOTE: tree=${tree} tests=58853 failed=0\n`)).toBe(true);+ expect(r.code).toBe(0);+ expect(ran('gate-ran')).toBe(false);+ expect(spawnSync('cat', [join(repo, '.git', 'reuse-args')], { encoding: 'utf8' }).stdout.trim())+ .toBe(`reuse-args: --sha ${git('rev-parse', 'HEAD')} --tree ${tree} --repo owner-example/repo-example`);+ });++ it('a full run on the tree that failed refuses: exit 1, no receipt, no gate', () => {+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser('exit 3') });+ expect(r.out).not.toContain('PREGATE_REMOTE:');+ expect(r.err).toContain('failed');+ expect(r.code).toBe(1);+ expect(ran('gate-ran')).toBe(false);+ });++ it('no reuse runs the gate, whose summary is the receipt', () => {+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser('exit 4') });+ expect(ran('gate-ran')).toBe(true);+ expect(r.out.trim().split('\n').at(-1)).toBe(`PREGATE_REMOTE: tree=${contentTree()} tests=4 failed=0`);+ expect(r.code).toBe(0);+ });++ it('a dirty tree, GBRAIN_PREGATE_REUSE=0 or no repository never asks for reuse', () => {+ const env = { GBRAIN_PREGATE_REUSE_CMD: reuser("echo 'GBRAIN_REUSE_RESULT tests=9 failed=0'") };+ run(gateMarking, { ...env, GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE: '0' });+ run(gateMarking, env);+ writeFileSync(join(repo, 'edited.txt'), 'not committed\n');+ const dirty = run(gateMarking, { ...env, GBRAIN_PREGATE_REPO: 'owner-example/repo-example' });+ expect(ran('reuse-args')).toBe(false);+ expect(dirty.out.trim().split('\n').at(-1)).toBe(`PREGATE_REMOTE: tree=${contentTree()} tests=4 failed=0`);+ });++ it('reads the repository from the origin remote', () => {+ git('remote', 'add', 'origin', 'https://git.example.invalid/owner-example/repo-example.git');+ run(gateMarking, { GBRAIN_PREGATE_REUSE_CMD: reuser('exit 4') });+ expect(spawnSync('cat', [join(repo, '.git', 'reuse-args')], { encoding: 'utf8' }).stdout).toContain('--repo owner-example/repo-example');+ });+ });});