Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(review): opencode rate limits no longer end a turn, a killed turn drains for 8 s, T5 requires a gbrain MCP tool call, --list refuses a changed inventory builder
garrytan5ebcb01f98f5
diff --git a/.github/workflows/heavy-tests.yml b/.github/workflows/heavy-tests.ymlindex 70255d5dd7..a3e5aab751 100644--- a/.github/workflows/heavy-tests.yml+++ b/.github/workflows/heavy-tests.yml@@ -959,9 +959,16 @@ jobs:- name: Report credentialed coverage not run in keyless laneif: matrix.lane == 'keyless'+ env:+ GBRAIN_PAID_LANES: ${{ vars.GBRAIN_PAID_LANES }}run: |echo "## OpenCode: keyless coverage only" >> "$GITHUB_STEP_SUMMARY"- echo "No paid turn ran in this lane. The nightly schedule runs the separate credentialed lane, whose paid nonce SMOKE gates; on a dispatch, request run_opencode_paid or set GBRAIN_OPENCODE_PAID_E2E=1 (GBRAIN_PAID_LANES=0 keeps it off either way)." >> "$GITHUB_STEP_SUMMARY"+ if [ "${GBRAIN_PAID_LANES:-}" = "0" ]; then+ echo "::warning::opencode door: no nonce SMOKE ran. The repository variable GBRAIN_PAID_LANES='0' switches the credentialed lane off, so this run proves no gbrain MCP tool call." >&2+ echo "No paid turn ran and no nonce SMOKE gates this run: the repository variable \`GBRAIN_PAID_LANES\` is \`0\`, which switches the credentialed lane off. Unset it (or set it to \`1\`) for the gating paid SMOKE." >> "$GITHUB_STEP_SUMMARY"+ else+ echo "No paid turn ran in this lane. The nightly schedule runs the separate credentialed lane, whose paid nonce SMOKE gates; on a dispatch, request run_opencode_paid or set GBRAIN_OPENCODE_PAID_E2E=1." >> "$GITHUB_STEP_SUMMARY"+ fi- name: Preconditions (credentialed lane requires secret)if: matrix.lane == 'credentialed'diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f5f38ebbd9..b7e072b0ac 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -25,10 +25,10 @@ identifiers and attribution are available in the pre-removal Git revision### Itemized changes-- `test/helpers/agent-harness.ts`: `opencodeOneShotTurn` reads main-turn `stream error` lines from `<XDG_DATA_HOME>/opencode/log` written during the turn, returns them as `streamErrors`, and ends the turn on a quota or rate-limit refusal (`providerRefusal`). `runOneShotSpawn` takes an `abortWhen` poll.-- `test/helpers/agent-harness.unit.test.ts`: stub-binary probes pin the quota fast-fail (it hangs to the timeout without the fix), a transient error that opencode recovers from, and per-turn log offsets.-- `test/e2e/install-real-opencode.serial.test.ts`: T4 runs only with `GBRAIN_REAL_OPENCODE_FREE_TIER_E2E=1`, adds logged stream errors to its provider-error message, and skips its second attempt after a quota refusal. T5 pins `anthropic/claude-haiku-5-5` and refreshes the model list before its spend gate.-- `.github/workflows/heavy-tests.yml`: the schedule requests the credentialed lane (`GBRAIN_PAID_LANES=0` still turns it off), the keyless lane expects 4 passes, the credentialed step keeps provider keys, and the canary sets `GBRAIN_REAL_OPENCODE_FREE_TIER_E2E=1`. `test/scripts/heavy-door-workflow.test.ts` pins all four.+- `test/helpers/agent-harness.ts`: `opencodeOneShotTurn` reads main-turn `stream error` lines from `<XDG_DATA_HOME>/opencode/log` written during the turn, returns them as `streamErrors`, and ends the turn on a quota refusal (`providerRefusal`). A provider rate limit, which opencode retries through, does not end it. `runOneShotSpawn` takes an `abortWhen` poll, and once it kills the child, a descendant that holds the output pipes open gets 8 s instead of the rest of the timeout.+- `test/helpers/agent-harness.unit.test.ts`: stub-binary probes pin the quota fast-fail (it hangs to the timeout without the fix), the fast-fail with a descendant holding the pipes, a transient error and a rate limit that opencode recovers from, and per-turn log offsets.+- `test/e2e/install-real-opencode.serial.test.ts`: T4 runs only with `GBRAIN_REAL_OPENCODE_FREE_TIER_E2E=1`, adds logged stream errors to its provider-error message, and skips its second attempt after a quota refusal. T5 pins `anthropic/claude-haiku-5-5`, refreshes the model list before its spend gate, and, like T4, reads the turn's JSON events and requires a `gbrain_*` MCP tool call, since opencode's file tools could find the nonce without one.+- `.github/workflows/heavy-tests.yml`: the schedule requests the credentialed lane (`GBRAIN_PAID_LANES=0` still turns it off), the keyless lane expects 4 passes and its summary warns that no nonce check gates the run when `GBRAIN_PAID_LANES=0`, the credentialed step keeps provider keys, and the canary sets `GBRAIN_REAL_OPENCODE_FREE_TIER_E2E=1`. `test/scripts/heavy-door-workflow.test.ts` pins all four.- `docs/mcp/OPENCODE-CLI-PIN.md`: the per-IP free cap, where its trace is logged, the new gating split and two triage rows.**A merge slot's full-ci runs can now clear gstack's remote gate on their own, so each tree runs the full suite once.** `gstack-pregate clear --remote pregate.json/remote_gate --run <Test run> --run <E2E run> --allow-repo-commands` verifies both runs on GitHub (same tree, both full jobs green in the newest attempt, no newer red run, under 24 hours old) and checks that the files they executed cover the gate's inventory. Checked against Test run 41342917113 and E2E run 41342918866 on one tree: `REUSE: accepted … tests=68132` and `PREGATE_VERIFY: ok`, with all 5,701 inventory files covered. A tree that changes the full-ci workflows or the scripts that attest their runs still can't vouch for itself. It gets no inventory, so gstack refuses those runs and the gate runs, the same rule remote-gate reuse already follows.@@ -37,7 +37,7 @@ identifiers and attribution are available in the pre-removal Git revision- `.gstack/pregate.json`: `run_profiles` `test` (`test.yml`, `full-suite`, artifact `full-suite-tree`) and `e2e` (`e2e.yml`, `e2e-full-suite`, artifact `e2e-full-suite-tree`), both accepting `workflow_dispatch` and `push`, plus `remote_gate_runs` with `bash scripts/pregate-remote-gate.sh --list` as the inventory.- `.gstack/ship-policy.json`: `min_gstack_version` 1.91.377.0, the first gstack that reads these keys. An older gstack now refuses with an upgrade fix instead of an invalid-config error.-- `scripts/pregate-remote-gate.sh`: `--list` refuses (no inventory) for a tree that changes `test.yml`, `e2e.yml`, `ci-reuse.ts`, `ci-full-suite-tree.sh`, `ci-executed-counts.ts` or `lib/gh-issue.ts` relative to `origin/master`, when there is no `origin/master`, and under `GBRAIN_PREGATE_REUSE=0`. The reuse path and `--list` share one check. `test/scripts/pregate-remote-gate.test.ts` pins this.+- `scripts/pregate-remote-gate.sh`: `--list` refuses (no inventory) for a tree that changes `test.yml`, `e2e.yml`, `ci-reuse.ts`, `ci-full-suite-tree.sh`, `ci-executed-counts.ts` or `lib/gh-issue.ts`, or the inventory's own builders `pregate-remote-gate.sh` and `ci-ubicloud.ts`, relative to `origin/master`, when there is no `origin/master`, and under `GBRAIN_PREGATE_REUSE=0`. The reuse path and `--list` share one check. `test/scripts/pregate-remote-gate.test.ts` pins this.- `docs/ci-event-parity.md`: the profiles and the refusal.**`ubi-runner.sh reap` now destroys only the caller's own VMs, and pack, usage and reap work on macOS and on an empty project (gstack #3756, #3746).** Reap used to destroy any owner's VM whose TTL had ended. That is how one project's `up` destroyed another lane's VMs (GST-573). Now it destroys only VMs whose owner tag equals `UBI_OWNER` and logs `reap: left N expired VMs of other owners` for the rest. The operator reaper, `reap-expired.ts`, still covers every owner. gbrain keeps `UBI_REAP=0` by default, so `up` still reaps nothing unless asked. On macOS, `pack` exited 1 under bsdtar after the VM was already created. On a project with no VMs, Ubicloud's bare-newline list read as a VM named `-`, and reap died on it. A failed size read made `usage` print a short total, which the quota preflight took as room.@@ -45,7 +45,7 @@ identifiers and attribution are available in the pre-removal Git revision### Itemized changes- `scripts/ubicloud/ubi-runner.sh`, ported from gstack 74836e41 and 693ee713:- - `cmd_reap` checks the owner tag first and counts other owners' expired VMs.+ - `cmd_reap` checks the owner tag first and counts other owners' expired VMs. The quota refusal says reap frees only this owner's VMs and names `usage` and `reap-expired.ts` for the rest.- `vm_rows` skips a blank row.- `cmd_usage` exits 1 with no total when the list or a size read fails, and skips a VM whose `show` answers 404.- `quota_preflight` retries an unreadable usage while `UBI_QUOTA_WAIT` lasts, then logs it and leaves the decision to the API.diff --git a/scripts/pregate-remote-gate.sh b/scripts/pregate-remote-gate.shindex 4f5cb86c29..9cdd48d26e 100755--- a/scripts/pregate-remote-gate.sh+++ b/scripts/pregate-remote-gate.sh@@ -62,12 +62,13 @@ fail() { echo "pregate-remote-gate: $1" >&2; exit "${2:-1}"; }# Whether the checked-out commit's own full-ci runs may vouch for it: never when# it changes the full-ci workflows or the scripts that attest their runs relative-# to origin/master (or there is no origin/master to compare against). Prints the-# reason on stderr and returns 1 when they may not.+# to origin/master (or there is no origin/master to compare against). Extra+# arguments name more paths that may not change. Prints the reason on stderr and+# returns 1 when they may not.runs_may_attest() {git rev-parse -q --verify 'origin/master^{commit}' >/dev/null || { echo "pregate-remote-gate: no origin/master to compare the full-ci workflows against" >&2; return 1; }local path- for path in .github/workflows/test.yml .github/workflows/e2e.yml scripts/ci-reuse.ts scripts/ci-full-suite-tree.sh scripts/ci-executed-counts.ts scripts/lib/gh-issue.ts; do+ for path in .github/workflows/test.yml .github/workflows/e2e.yml scripts/ci-reuse.ts scripts/ci-full-suite-tree.sh scripts/ci-executed-counts.ts scripts/lib/gh-issue.ts "$@"; dogit diff --quiet origin/master HEAD -- "$path" && continueecho "pregate-remote-gate: the tree changes $path relative to origin/master, so its own runs cannot attest to it" >&2return 1@@ -79,15 +80,17 @@ runs_may_attest() {# selection and inventory), so gstack-pregate clear --run can check that the# full-ci runs it is given executed every one (remote_gate_runs.inventory,# docs/ci-event-parity.md#remote-gate-reuse). A failing or empty inventory exits-# non-zero, and so does a tree whose runs may not attest to it (runs_may_attest)-# or GBRAIN_PREGATE_REUSE=0, so gstack refuses the runs and the gate runs. It-# starts no VM and needs no UBI_OWNER.+# non-zero, and so does a tree whose runs may not attest to it (runs_may_attest,+# here also refusing a change to this script or to scripts/ci-ubicloud.ts, the+# code that builds the inventory, since gstack runs the branch's own copy of+# them) or GBRAIN_PREGATE_REUSE=0, so gstack refuses the runs and the gate runs.+# It starts no VM and needs no UBI_OWNER.if [ "${1:-}" = "--list" ]; thenshiftTOP=$(git rev-parse --show-toplevel 2>/dev/null) || fail "not inside a git checkout" 2cd "$TOP" || exit 2[ "${GBRAIN_PREGATE_REUSE:-1}" != 0 ] || fail "GBRAIN_PREGATE_REUSE=0: no inventory, so no full-ci run clears the gate; run the gate" 1- runs_may_attest || fail "no inventory, so no full-ci run clears the gate; run the gate (bash scripts/pregate-remote-gate.sh)" 1+ runs_may_attest scripts/pregate-remote-gate.sh scripts/ci-ubicloud.ts || fail "no inventory, so no full-ci run clears the gate; run the gate (bash scripts/pregate-remote-gate.sh)" 1listed=$(bash -c "${GBRAIN_PREGATE_GATE_CMD:-bun run ci:ubicloud:diff} --list \"\$@\"" pregate-remote-gate "$@") || fail "the gate's inventory failed (exit $?); no list" 1[ -n "$listed" ] || fail "the gate's inventory is empty; no list" 1printf '%s\n' "$listed"diff --git a/scripts/ubicloud/ubi-runner.sh b/scripts/ubicloud/ubi-runner.shindex b0033ff355..40d7d40f36 100755--- a/scripts/ubicloud/ubi-runner.sh+++ b/scripts/ubicloud/ubi-runner.sh@@ -579,7 +579,7 @@ quota_preflight() {if [ "$remaining" -le 0 ]; thenlog "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)"cmd_usage >&2 || true- die "create refused before any request: vCPU quota exhausted (quota full); wait for running VMs to finish (UBI_QUOTA_WAIT=<seconds> waits here), run $0 reap, use a smaller size, or set UBI_QUOTA_VCPUS to the project's real ceiling"+ die "create refused before any request: vCPU quota exhausted (quota full); wait for running VMs to finish (UBI_QUOTA_WAIT=<seconds> waits here), run $0 reap (it frees only this owner's expired VMs; $0 usage shows who holds the rest, and scripts/ubicloud/reap-expired.ts destroys other owners' expired ones), use a smaller size, or set UBI_QUOTA_VCPUS to the project's real ceiling"filog "quota: $used of $max vCPUs in use and $need needed; waiting up to ${remaining}s more for room"step=$(( remaining < 30 ? remaining : 30 ))diff --git a/test/e2e/install-real-opencode.serial.test.ts b/test/e2e/install-real-opencode.serial.test.tsindex a415bbd91a..a600d438ac 100644--- a/test/e2e/install-real-opencode.serial.test.ts+++ b/test/e2e/install-real-opencode.serial.test.ts@@ -514,11 +514,19 @@ describe.skipIf(!CAN_RUN_PAID)('install real-opencode door — paid tier (anthrohome,timeoutMs: 240_000,model: PAID_MODEL,+ format: 'json',binDir,paid: true,});+ try { writeFileSync(join(home, 'door-turn.jsonl'), turn.finalText); } catch { /* evidence */ }+ const parsed = parseOpencodeJsonl(turn.finalText.split('\n'));+ const errors = [...parsed.errors, ...turn.streamErrors];+ if (!parsed.finalText.includes(nonce) && errors.length) throw new Error(`opencode turn ended with provider error(s): ${errors.join('; ')}`);expect(turn.exitCode).toBe(0);- expect(turn.finalText).toContain(nonce);- expect(turn.finalText).not.toContain('NO-GBRAIN-TOOL');+ expect(parsed.finalText).toContain(nonce);+ expect(parsed.finalText).not.toContain('NO-GBRAIN-TOOL');+ // opencode also has fs/shell tools, so the nonce alone does not prove the+ // MCP door: at least one gbrain MCP tool must have fired (as in T4).+ expect(parsed.toolCalls.some((t) => t.startsWith('gbrain_'))).toBe(true);}, 600_000);});diff --git a/test/helpers/agent-harness.ts b/test/helpers/agent-harness.tsindex 3a77f207fa..2ec84ab237 100644--- a/test/helpers/agent-harness.ts+++ b/test/helpers/agent-harness.ts@@ -728,12 +728,15 @@ export interface OneShotSpawnResult {abortReason?: string;}+/** How long a killed one-shot child's output may keep draining. */+const KILL_DRAIN_GRACE_MS = 8_000;+/*** Shared one-shot spawn core: timeout → kill, kill(9) escalation after 5s* (agents may leave a daemon/MCP-server child holding the pipes open past- * the parent's death), and a BOUNDED stream drain (timeout + 30s cap) so a- * grandchild holding the pipe fds can never hang a retry loop. Exit 124 on- * timeout.+ * the parent's death), and a BOUNDED stream drain (timeout + 30s cap, or+ * KILL_DRAIN_GRACE_MS once the child is killed) so a grandchild holding the+ * pipe fds can never hang a retry loop. Exit 124 on timeout.*/export async function runOneShotSpawn(opts: {argv: string[];@@ -754,7 +757,10 @@ export async function runOneShotSpawn(opts: {let timedOut = false;let abortReason: string | undefined;+ let killed: () => void = () => {};+ const killedAt = new Promise<void>((r) => { killed = r; });const kill = () => {+ killed();try { proc.kill(); } catch { /* already dead */ }setTimeout(() => { try { proc.kill(9); } catch { /* already dead */ } }, 5_000);};@@ -772,13 +778,24 @@ export async function runOneShotSpawn(opts: {const drainCap = opts.timeoutMs + 30_000;// The cap timer is CLEARED when the real promise wins — an uncancelled// drainCap timer (timeoutMs + 30s) would keep Bun's event loop alive for- // minutes after every successful door run.+ // minutes after every successful door run. Once the child is killed, a+ // descendant that kept stdout/stderr open (an MCP server, a provider retry+ // helper) gets KILL_DRAIN_GRACE_MS, not the rest of drainCap.const bounded = <T>(p: Promise<T>, fallback: T): Promise<T> => {let capTimer: ReturnType<typeof setTimeout> | undefined;+ let graceTimer: ReturnType<typeof setTimeout> | undefined;+ let settled = false;const cap = new Promise<T>((r) => {capTimer = setTimeout(() => r(fallback), drainCap);});- return Promise.race([p, cap]).finally(() => clearTimeout(capTimer)) as Promise<T>;+ const grace = killedAt.then(() => new Promise<T>((r) => {+ if (!settled) graceTimer = setTimeout(() => r(fallback), KILL_DRAIN_GRACE_MS);+ }));+ return Promise.race([p, cap, grace]).finally(() => {+ settled = true;+ clearTimeout(capTimer);+ clearTimeout(graceTimer);+ }) as Promise<T>;};const [stdout, stderrText] = await Promise.all([bounded(new Response(proc.stdout).text(), ''),@@ -1170,8 +1187,10 @@ export interface OpencodeTurnResult extends OneShotSpawnResult {/** #6049: opencode answers a quota refusal (free tier: "Free usage limit* reached", HTTP 429) by retrying after the server's retry-after, hours- * later, with nothing on stdout. Waiting can never produce an answer. */-const OPENCODE_QUOTA_REFUSAL = /usage limit|rate limit|quota/i;+ * later, with nothing on stdout. Waiting can never produce an answer. A+ * provider rate limit ("rate limit exceeded", Anthropic's 429) clears in+ * seconds and opencode's retry recovers from it, so it never matches. */+const OPENCODE_QUOTA_REFUSAL = /usage limit|quota/i;/** Main-turn (`small=false`) `stream error` lines appended to opencode's* file log (<XDG_DATA_HOME>/opencode/log) since `offsets` was taken. */diff --git a/test/helpers/agent-harness.unit.test.ts b/test/helpers/agent-harness.unit.test.tsindex 9925e2bfba..2ff186e518 100644--- a/test/helpers/agent-harness.unit.test.ts+++ b/test/helpers/agent-harness.unit.test.ts@@ -703,6 +703,9 @@ describe('opencodeOneShotTurn — provider refusals logged by opencode surface iconst TITLE_LINE ='timestamp=2026-11-01T16:05:37.341Z level=ERROR run=a6ecc561 message="stream error" providerID=opencode modelID=big-pickle ' +'session.id=ses_x small=true agent=title mode=primary error.error="AI_RetryError: Failed after 3 attempts. Last error: upstream 502"';+ const RATE_LIMIT_LINE =+ 'timestamp=2026-11-01T16:05:31.565Z level=ERROR run=c1 message="stream error" providerID=anthropic modelID=claude-haiku-5-5 ' ++ 'session.id=ses_z small=false agent=build mode=primary error.error="AI_APICallError: Rate limit exceeded; retry after 1 second"';const TRANSIENT_LINE ='timestamp=2026-11-01T16:05:31.565Z level=ERROR run=b1 message="stream error" providerID=opencode modelID=big-pickle ' +'session.id=ses_y small=false agent=build mode=primary error.error="AI_APICallError: Bad Gateway"';@@ -750,6 +753,39 @@ describe('opencodeOneShotTurn — provider refusals logged by opencode surface i}}, 60_000);+ test('a provider rate limit is a transient error: the turn runs on and returns what opencode answers after its retry', async () => {+ const dir = mkdtempSync(join(tmpdir(), 'gb-oc-ratelimit-'));+ try {+ const home = join(dir, 'home');+ const text = JSON.stringify({ type: 'text', part: { text: 'after-retry' } });+ const bin = stubOpencode(dir, `printf '%s\\n' '${RATE_LIMIT_LINE}' >> "$log/opencode.log"\nsleep 3\nprintf '%s\\n' '${text}'`);+ const turn = await withEnv({ OPENCODE_BIN: bin }, () =>+ opencodeOneShotTurn({ prompt: 'p', cwd: dir, home, timeoutMs: 30_000, format: 'json' }));+ expect(turn.exitCode).toBe(0);+ expect(turn.providerRefusal).toBeUndefined();+ expect(parseOpencodeJsonl(turn.finalText.split('\n')).finalText).toBe('after-retry');+ expect(turn.streamErrors).toEqual(['AI_APICallError: Rate limit exceeded; retry after 1 second']);+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 60_000);++ test('a quota refusal still ends the turn within seconds when a descendant keeps the output pipes open', async () => {+ const dir = mkdtempSync(join(tmpdir(), 'gb-oc-quota-desc-'));+ try {+ const home = join(dir, 'home');+ const bin = stubOpencode(dir, `printf '%s\\n' '${QUOTA_LINE}' >> "$log/opencode.log"\nsleep 25 &\nexec sleep 600`);+ const t0 = Date.now();+ const turn = await withEnv({ OPENCODE_BIN: bin }, () =>+ opencodeOneShotTurn({ prompt: 'p', cwd: dir, home, timeoutMs: 30_000, format: 'json' }));+ expect(Date.now() - t0).toBeLessThan(15_000);+ expect(turn.timedOut).toBe(false);+ expect(turn.providerRefusal).toContain('Free usage limit reached');+ } finally {+ rmSync(dir, { recursive: true, force: true });+ }+ }, 90_000);+test('stream errors already in the log from an earlier turn in the same home are not attributed to this turn', async () => {const dir = mkdtempSync(join(tmpdir(), 'gb-oc-offset-'));try {diff --git a/test/scripts/pregate-remote-gate.test.ts b/test/scripts/pregate-remote-gate.test.tsindex a736a5a321..307c0f883a 100644--- a/test/scripts/pregate-remote-gate.test.ts+++ b/test/scripts/pregate-remote-gate.test.ts@@ -323,7 +323,8 @@ describe('pregate-remote-gate.sh --list', () => {it('a tree whose full-ci runs would attest to themselves gets no inventory, so gstack clear --run refuses it and the gate runs', () => {const gate = 'touch .git/list-ran; echo test/a.test.ts';const refused = (r: { code: number | null; out: string }) => { expect(r.code).toBe(1); expect(r.out).toBe(''); };- for (const path of ['.github/workflows/test.yml', '.github/workflows/e2e.yml', 'scripts/ci-reuse.ts', 'scripts/ci-full-suite-tree.sh', 'scripts/ci-executed-counts.ts', 'scripts/lib/gh-issue.ts']) {+ // pregate-remote-gate.sh and ci-ubicloud.ts build the inventory itself, and gstack runs the branch's own copy.+ for (const path of ['.github/workflows/test.yml', '.github/workflows/e2e.yml', 'scripts/ci-reuse.ts', 'scripts/ci-full-suite-tree.sh', 'scripts/ci-executed-counts.ts', 'scripts/lib/gh-issue.ts', 'scripts/pregate-remote-gate.sh', 'scripts/ci-ubicloud.ts']) {mkdirSync(join(repo, path, '..'), { recursive: true });writeFileSync(join(repo, path), `changed ${path}\n`);git('add', '.');