Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix: remote-gate reuse lets the newest full run on the tree decide, never reuses a tree that changes its own full-ci, and strips keys

garrytane9ab90cc3170
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 34be326ed6..79a64f4a87 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,7 +14,7 @@ identifiers and attribution are available in the pre-removal Git revision
**The remote gate takes its receipt from GitHub's full-ci when the exact tree already passed there, so a merge slot no longer pays for a second full test pass on Ubicloud.**
-`scripts/pregate-remote-gate.sh` prints the `PREGATE_REMOTE:` line that clears /ship's remote-gate obligation, and it used to start Ubicloud VMs every time, even when GitHub had just run the same tests on the same tree. Now, when the working tree is clean and both the Test and E2E full runs passed on GitHub for that exact tree (`full-suite` and `e2e-full-suite` green in the latest attempt, their tree artifacts naming the tree), it prints the same line from those runs and starts no VM. A full run still in progress on the tree is waited for, up to 90 minutes. A full run on the tree that failed refuses with the run's URL instead of spending on VMs. Any other tree runs the VM gate as before, including a tree that differs only in release stamps (gstack's stamp rule covers that case), a dirty working tree, or `GBRAIN_PREGATE_REUSE=0`. On reuse, `tests=` counts the executed test identities of both runs.
+`scripts/pregate-remote-gate.sh` prints the `PREGATE_REMOTE:` line that clears /ship's remote-gate obligation, and it used to start Ubicloud VMs every time, even when GitHub had just run the same tests on the same tree. Now, when the working tree is clean and both the Test and E2E full runs passed on GitHub for that exact tree (`full-suite` and `e2e-full-suite` green in the latest attempt, their tree artifacts naming the tree), it prints the same line from those runs and starts no VM. A full run still in progress on the tree is waited for, up to 90 minutes. A full run on the tree that failed refuses with the run's URL instead of spending on VMs. The newest full run on the tree decides, across every recent commit with that tree, so an older pass cannot outrank a newer failure. Any other tree runs the VM gate as before, including a tree that differs only in release stamps (gstack's stamp rule covers that case), a tree that changes `.github/workflows/test.yml`, `.github/workflows/e2e.yml` or `scripts/ci-reuse.ts` relative to master (its runs would attest to themselves), a dirty working tree, or `GBRAIN_PREGATE_REUSE=0`. On reuse, `tests=` counts the executed test identities of both runs.
### Itemized changes
diff --git a/docs/ci-event-parity.md b/docs/ci-event-parity.md
index 4ba97f7a55..4646d6ca77 100644
--- a/docs/ci-event-parity.md
+++ b/docs/ci-event-parity.md
@@ -109,11 +109,14 @@ complete full runs of both Test and E2E under the master-push rules above. If
both did, the receipt comes from them: the same `PREGATE_REMOTE: tree= tests=
failed=` line, with the runs' executed test identities as the counts, and no VM
starts. A full run on the tree still in progress is waited for (90 minutes at
-most, a progress line every minute). A failed newest full run refuses with no
-receipt. The newest full run on the head decides, then the newest full run on
-each other head with the same tree, so an empty commit cannot skip a newer
-failure; a run that did not run the full job (a diagnostic dispatch, a push that
-reused) is skipped. Every other case runs the gate: a dirty tree, a commit GitHub
+most, a progress line every minute). The newest full run on the tree decides,
+across the head and every recent head with the same tree, so neither an empty
+commit nor an older pass elsewhere skips a newer failure; a failed one refuses
+with no receipt. A run that did not run the full job (a diagnostic dispatch, a
+push that reused) is skipped. A tree that changes `.github/workflows/test.yml`,
+`.github/workflows/e2e.yml` or `scripts/ci-reuse.ts` relative to
+`origin/master` never reuses, since its runs would attest to themselves, and the
+check runs without the model and database keys, like the gate. Every other case runs the gate: a dirty tree, a commit GitHub
doesn't have, a missing workflow, an API error, or `GBRAIN_PREGATE_REUSE=0`.
Only the identical tree reuses. A head that /ship changed after full-ci has a
diff --git a/scripts/ci-reuse.ts b/scripts/ci-reuse.ts
index 4ff05b82b3..878f06d27d 100644
--- a/scripts/ci-reuse.ts
+++ b/scripts/ci-reuse.ts
@@ -144,9 +144,9 @@ function accept(run: FullRun, tree: string, rejected: Rejection[], where: string
* already passed the complete full run of a workflow needs no Ubicloud gate. Same fail-closed checks as a push:
* the run's latest attempt and its full-suite job succeeded and its tested-tree artifact records `tree`. Only
* the identical tree reuses; a stamp-only difference goes through gstack-pregate's stamp rule, never through here.
- * The newest full run on `head` decides, then the same for each other head with the same tree (newest first): in
- * progress waits, a failure refuses (no receipt, no VMs), cancelled or not a full run looks further back; anything else
- * is `none` and the caller runs the VM gate.
+ * The newest full run on the tree decides, across `head` and every other recent head with the same tree: in progress
+ * waits, a failure refuses (no receipt, no VMs), cancelled or not a full run (its full job skipped or absent: a
+ * diagnostic dispatch, a push that reused) looks further back; anything else is `none` and the caller runs the VM gate.
*/
export type GateState = 'reuse' | 'pending' | 'failed' | 'none';
export interface GateDecision { state: GateState; detail: string; tests?: number; failed?: number; url?: string }
@@ -162,16 +162,26 @@ export async function decideGate(deps: GateDeps, p: Profile, head: string, tree:
try {
const headTree = (await deps.commit(head)).tree;
if (headTree !== tree) return { state: 'none', detail: `${p.workflow}: commit ${head} has tree ${headTree} on GitHub, not the working tree ${tree}` };
- const own = await headGate(deps, p, head, tree, repo, rejected);
- if (own) return own;
- const heads = new Map<string, string>();
+ const shas = [head];
+ const trees = new Map<string, string>();
for (const run of (await deps.recentRuns(p.file)).sort((a, b) => b.id - a.id)) {
- if (!isCandidate(run) || run.head_sha === head || run.head_repository?.full_name !== repo || heads.has(run.head_sha)) continue;
- if (heads.size >= MAX_FALLBACK_HEADS) break;
- heads.set(run.head_sha, (await deps.commit(run.head_sha)).tree);
- if (heads.get(run.head_sha) !== tree) continue;
- const other = await headGate(deps, p, run.head_sha, tree, repo, rejected);
- if (other) return other;
+ if (!isCandidate(run) || run.head_sha === head || run.head_repository?.full_name !== repo || trees.has(run.head_sha)) continue;
+ if (trees.size >= MAX_FALLBACK_HEADS) break;
+ trees.set(run.head_sha, (await deps.commit(run.head_sha)).tree);
+ if (trees.get(run.head_sha) === tree) shas.push(run.head_sha);
+ }
+ const runs = (await Promise.all(shas.map(async sha => (await deps.runsForHead(p.file, sha)).filter(r => r.head_sha === sha && isCandidate(r))))).flat().sort((a, b) => b.id - a.id);
+ for (const run of runs) {
+ const full = (await deps.jobs(run.id, run.run_attempt)).filter(j => j.name === p.job);
+ if (full.some(j => j.conclusion === 'skipped') || (run.status === 'completed' && full.length === 0)) {
+ rejected.push({ run_id: run.id, reason: 'full_suite_missing', detail: `run ${run.id} attempt ${run.run_attempt} did not run ${p.job} (not a full-profile run)` });
+ continue;
+ }
+ const why = await reject(deps, p, run, repo, tree, tree);
+ if (!why) return gateAccept(deps, p, run, tree, repo);
+ if (why.reason === 'run_in_progress') return { state: 'pending', detail: `${p.workflow} ${why.detail}` };
+ if (why.reason === 'run_failed' || why.reason === 'full_suite_not_success') return { state: 'failed', detail: `${p.workflow} ${why.detail}`, url: runUrl(repo, run) };
+ rejected.push({ run_id: run.id, ...why });
}
} catch (e) {
return { state: 'none', detail: `${p.workflow}: reading commits, runs, jobs or artifacts failed: ${e instanceof Error ? e.message : String(e)}` };
@@ -179,27 +189,6 @@ export async function decideGate(deps: GateDeps, p: Profile, head: string, tree:
return { state: 'none', detail: rejected.length ? `${p.workflow}: ${rejected.map(r => r.detail).join('; ')}` : `${p.workflow}: no workflow_dispatch or push run on ${head} or on another head with tree ${tree}` };
}
-/**
- * The newest full run of `p` on commit `sha` (whose tree is `tree`) decides; a run that did not run the full job
- * (a diagnostic dispatch, a push that reused) is skipped. Undefined when `sha` has no deciding run.
- */
-async function headGate(deps: GateDeps, p: Profile, sha: string, tree: string, repo: string, rejected: Rejection[]): Promise<GateDecision | undefined> {
- const runs = (await deps.runsForHead(p.file, sha)).filter(r => r.head_sha === sha && isCandidate(r)).sort((a, b) => b.id - a.id);
- for (const run of runs) {
- const full = (await deps.jobs(run.id, run.run_attempt)).filter(j => j.name === p.job);
- if (full.some(j => j.conclusion === 'skipped') || (run.status === 'completed' && full.length === 0)) {
- rejected.push({ run_id: run.id, reason: 'full_suite_missing', detail: `run ${run.id} attempt ${run.run_attempt} did not run ${p.job} (not a full-profile run)` });
- continue;
- }
- const why = await reject(deps, p, run, repo, tree, tree);
- if (!why) return gateAccept(deps, p, run, tree, repo);
- if (why.reason === 'run_in_progress') return { state: 'pending', detail: `${p.workflow} ${why.detail}` };
- if (why.reason === 'run_failed' || why.reason === 'full_suite_not_success') return { state: 'failed', detail: `${p.workflow} ${why.detail}`, url: runUrl(repo, run) };
- rejected.push({ run_id: run.id, ...why });
- }
- return undefined;
-}
-
const runUrl = (repo: string, run: FullRun) => `https://github.com/${repo}/actions/runs/${run.id}/attempts/${run.run_attempt}`;
async function gateAccept(deps: GateDeps, p: Profile, run: FullRun, tree: string, repo: string): Promise<GateDecision> {
diff --git a/scripts/pregate-remote-gate.sh b/scripts/pregate-remote-gate.sh
index 91ba8af69a..181fe8ed92 100755
--- a/scripts/pregate-remote-gate.sh
+++ b/scripts/pregate-remote-gate.sh
@@ -18,6 +18,10 @@
# receipt, no VMs). <tests> and <failed> are then the runs' executed identities.
# Any other tree, a stamp-only one included, runs the gate below; gstack's
# stamp rule (`gstack-pregate clear --receipt`) handles a stamp-only head.
+# A tree that changes .github/workflows/test.yml, .github/workflows/e2e.yml or
+# scripts/ci-reuse.ts relative to origin/master (or has no origin/master)
+# never reuses: its runs would attest to themselves. The reuse check runs
+# without the model and database keys, like the gate.
# GBRAIN_PREGATE_REUSE=0 always runs the gate.
#
# Otherwise it runs the development form of the Ubicloud gate, `bun run ci:ubicloud:diff`:
@@ -70,11 +74,28 @@ fi
log=$(mktemp "${TMPDIR:-/tmp}/pregate-remote-gate-log-XXXXXX") || fail "cannot create a log file"
trap 'rm -f "$log"' EXIT
+strip_keys() { env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL "$@"; }
+
reused=""
repo=${GBRAIN_PREGATE_REPO:-$(git remote get-url origin 2>/dev/null | sed -E 's#\.git$##' | sed -nE 's#^.*[/:]([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)$#\1#p')}
+reuse_ok=1
if [ "${GBRAIN_PREGATE_REUSE:-1}" != 0 ] && [ -n "$repo" ] && [ "$before" = "$(git rev-parse 'HEAD^{tree}' 2>/dev/null)" ]; then
+ if ! git rev-parse -q --verify 'origin/master^{commit}' >/dev/null; then
+ echo "pregate-remote-gate: no origin/master to compare the full-ci workflows against, so it runs the gate instead of reusing" >&2
+ reuse_ok=0
+ fi
+ for path in .github/workflows/test.yml .github/workflows/e2e.yml scripts/ci-reuse.ts; do
+ [ "$reuse_ok" = 1 ] || break
+ git diff --quiet origin/master HEAD -- "$path" && continue
+ echo "pregate-remote-gate: the tree changes $path relative to origin/master, so it runs the gate instead of reusing its own runs" >&2
+ reuse_ok=0
+ done
+else
+ reuse_ok=0
+fi
+if [ "$reuse_ok" = 1 ]; then
reuse_cmd=${GBRAIN_PREGATE_REUSE_CMD:-bun scripts/ci-reuse.ts --gate}
- bash -c "$reuse_cmd \"\$@\"" pregate-remote-gate --sha "$(git rev-parse HEAD)" --tree "$before" --repo "$repo" 2>&1 | tee "$log"
+ strip_keys bash -c "$reuse_cmd \"\$@\"" pregate-remote-gate --sha "$(git rev-parse HEAD)" --tree "$before" --repo "$repo" 2>&1 | tee "$log"
case ${PIPESTATUS[0]} in
0) reused=$(sed -nE 's/^GBRAIN_REUSE_RESULT tests=([0-9]+) failed=([0-9]+)$/\1 \2/p' "$log" | tail -1) ;;
3) fail "a full run on tree $before failed; fix it or rerun it, no receipt" 1 ;;
@@ -88,8 +109,7 @@ if [ -n "$reused" ]; then
else
gate_cmd=${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff}
# shellcheck disable=SC2086
- env -u OPENAI_API_KEY -u ANTHROPIC_API_KEY -u VOYAGE_API_KEY -u GEMINI_API_KEY -u JEV_TYPESAFE_API_KEY -u DATABASE_URL \
- bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"
+ strip_keys bash -c "$gate_cmd \"\$@\"" pregate-remote-gate "$@" 2>&1 | tee "$log"
rc=${PIPESTATUS[0]}
totals=$(awk '
/^=== ci-ubicloud summary ===$/ { seen = 1; next }
diff --git a/test/scripts/ci-reuse.test.ts b/test/scripts/ci-reuse.test.ts
index f81435a3e4..c62e9831df 100644
--- a/test/scripts/ci-reuse.test.ts
+++ b/test/scripts/ci-reuse.test.ts
@@ -462,6 +462,16 @@ describe('ci-reuse remote-gate decision', () => {
const C3 = '9'.repeat(40);
const failedElsewhere = gate({ commits: { ...commits, [C3]: { tree: T, parents: [C2] } }, runs: [run('test.yml', { id: 640, head_sha: C2, conclusion: 'failure' }), run('test.yml'), run('e2e.yml')] });
expect(await decideGate(failedElsewhere, PROFILES.test, C3, T, REPO)).toMatchObject({ state: 'failed' });
+ // The newest full run across every head with the tree decides, whichever head has the newest run of any kind:
+ // a diagnostic dispatch on A (full job skipped) does not let A's old pass outrank B's newer failure, and the
+ // requested head's own older pass does not either.
+ const ordered = gate({
+ commits: { ...commits, [C3]: { tree: T, parents: [C2] } },
+ runs: [run('test.yml', { id: 650 }), run('test.yml', { id: 640, head_sha: C2, conclusion: 'failure' }), run('test.yml'), run('e2e.yml')],
+ jobs: { '650/1': [{ name: PROFILES.test.job, status: 'completed', conclusion: 'skipped' }] },
+ });
+ expect(await decideGate(ordered, PROFILES.test, C3, T, REPO)).toMatchObject({ state: 'failed' });
+ expect(await decideGate(ordered, PROFILES.test, A, T, REPO)).toMatchObject({ state: 'failed' });
});
test('a cancelled newest run looks further back; a passed run of the same tree on another head (a merge commit) reuses', async () => {
diff --git a/test/scripts/pregate-remote-gate.test.ts b/test/scripts/pregate-remote-gate.test.ts
index b5b2e57e8f..f54c128434 100644
--- a/test/scripts/pregate-remote-gate.test.ts
+++ b/test/scripts/pregate-remote-gate.test.ts
@@ -141,6 +141,36 @@ describe('pregate-remote-gate.sh', () => {
};
const ran = (name: string) => spawnSync('test', ['-e', join(repo, '.git', name)]).status === 0;
const gateMarking = `touch .git/gate-ran; ${GREEN_GATE}`;
+ beforeEach(() => git('update-ref', 'refs/remotes/origin/master', 'HEAD'));
+
+ it('runs the reuse check without the model and database keys the gate strips', () => {
+ const keys = ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'VOYAGE_API_KEY', 'GEMINI_API_KEY', 'JEV_TYPESAFE_API_KEY', 'DATABASE_URL'];
+ const seen = keys.map(k => `echo "${k}=\${${k}:-unset}" >> .git/reuse-env`).join('; ');
+ run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser(`${seen}; exit 4`), ...Object.fromEntries(keys.map(k => [k, 'secret-example'])) });
+ expect(spawnSync('cat', [join(repo, '.git', 'reuse-env')], { encoding: 'utf8' }).stdout.trim().split('\n')).toEqual(keys.map(k => `${k}=unset`));
+ });
+
+ it('a tree that changes the full-ci workflows or ci-reuse.ts relative to origin/master never reuses its own runs', () => {
+ for (const path of ['.github/workflows/test.yml', '.github/workflows/e2e.yml', 'scripts/ci-reuse.ts']) {
+ rmSync(join(repo, '.git', 'reuse-args'), { force: true });
+ rmSync(join(repo, '.git', 'gate-ran'), { force: true });
+ mkdirSync(join(repo, path, '..'), { recursive: true });
+ writeFileSync(join(repo, path), `changed ${path}\n`);
+ git('add', '.');
+ git('commit', '-q', '-m', `change ${path}`);
+ const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser("echo 'GBRAIN_REUSE_RESULT tests=9 failed=0'") });
+ expect(ran('reuse-args')).toBe(false);
+ expect(ran('gate-ran')).toBe(true);
+ expect(r.err).toContain(`pregate-remote-gate: the tree changes ${path} relative to origin/master`);
+ expect(r.out.trim().split('\n').at(-1)).toBe(`PREGATE_REMOTE: tree=${contentTree()} tests=4 failed=0`);
+ git('update-ref', 'refs/remotes/origin/master', 'HEAD');
+ }
+ // Without origin/master to compare against, reuse is not trusted either.
+ git('update-ref', '-d', 'refs/remotes/origin/master');
+ rmSync(join(repo, '.git', 'reuse-args'), { force: true });
+ run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser("echo 'GBRAIN_REUSE_RESULT tests=9 failed=0'") });
+ expect(ran('reuse-args')).toBe(false);
+ });
it('takes the receipt from the full runs, with the gate\'s exact line, and starts no gate', () => {
const r = run(gateMarking, { GBRAIN_PREGATE_REPO: 'owner-example/repo-example', GBRAIN_PREGATE_REUSE_CMD: reuser("echo '[ci-reuse] gate: Test run passed'; echo 'GBRAIN_REUSE_RESULT tests=58853 failed=0'") });