Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(autopilot): a clean global pass refreshes the verdict autopilot --status reads (#7854)

garrytan11bcfa459886
diff --git a/docs/architecture/key-files/core-minions-1.md b/docs/architecture/key-files/core-minions-1.md
index faee7a1cf1..7bf2e3b07f 100644
--- a/docs/architecture/key-files/core-minions-1.md
+++ b/docs/architecture/key-files/core-minions-1.md
@@ -32,7 +32,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/minions/errors.ts` — dependency-free `UnrecoverableError` and `JobDeferredError(reason, message, retryInMs)`, re-exported through `types.ts` for compatibility. Delegated policy imports this leaf directly, avoiding initialization cycles through durable authority. Fresh-process import tests preserve the shared error identity and isolated-worker reconstruction. A `JobDeferredError` returns the job to `delayed` for `retryInMs` through `MinionQueue.deferJob` (no attempt counted, no stacktrace entry); the worker, the facts drain and the isolation codec (`errorKind: 'deferred'`) all route it (`test/facts-absorb-keyless-deferral.test.ts`).
-- `src/core/minions/global-maintenance-stall.ts` — brain-wide maintenance stall verdict (GBRA-130 C-6): `assessGlobalMaintenanceStall` (stale `last_global_at` plus failed jobs or a phase streak, or passes stamping past a failing phase or a triage deferral that stopped shrinking, #7307; a deferral streak's fix is `gbrain dream retriage`) feeds doctor `global_maintenance_stalled`; the daemon writes it to `autopilot-global-maintenance.json` each full-cycle tick (`recordGlobalMaintenanceStallStatus`) and `autopilot --status` reads it without the database (`readGlobalMaintenanceVerdict`).
+- `src/core/minions/global-maintenance-stall.ts` — brain-wide maintenance stall verdict (GBRA-130 C-6): `assessGlobalMaintenanceStall` (stale `last_global_at` plus failed jobs or a phase streak, or passes stamping past a failing phase or a triage deferral that stopped shrinking, #7307; a deferral streak's fix is `gbrain dream retriage`) feeds doctor `global_maintenance_stalled`; the daemon writes it to `autopilot-global-maintenance.json` each full-cycle tick (`recordGlobalMaintenanceStallStatus`), and so does a global-maintenance pass that stamps `last_global_at` (#7854, `test/autopilot-status-verdict-7854.test.ts`), and `autopilot --status` reads it without the database (`readGlobalMaintenanceVerdict`).
- `src/core/minions/handler-timeouts.ts` — per-handler-type defaults for BOTH per-job time knobs, co-located so they can't drift apart unseen (they are different quantities: the budget bounds total runtime, the lease bounds dead-worker reclaim — never derive one from the other). `HANDLER_DEFAULT_TIMEOUT_MS`: 30 min for `subagent`, `subagent_aggregator`, `embed-backfill`, `autopilot-cycle`, `autopilot-global-maintenance`, `extract` (a brain-wide `extract --stale` pass); 10 min for `chronicle_extract` + `facts-absorb`; 60 min for `contextual_reindex_per_chunk`. `HANDLER_DEFAULT_LOCK_DURATION_MS`: 300 s for the long LLM/loop handlers and for `sync` (its blocking git helpers can each outlast a 30 s lease), 120 s for the single-LLM-call handlers, `shell` deliberately absent (fast dead-worker reclaim; verify-before-evict protects it anyway). `defaultTimeoutMsFor` / `defaultLockDurationMsFor` return the mapped default or `null` (short handlers keep the tight null-default wall-clock / the 30 s worker lease). `clampLockDurationMs` + `LOCK_DURATION_MS_MIN/MAX` ([5s, 1h]) is the ONE clamp shared by `queue.add`, the CLI `--lock-duration-ms` flag (`--dry-run` echoes the clamped value), and the MCP `submit_job` param; the same bound is mirrored in SQL at claim time and by the `minion_jobs.lock_duration_ms` range CHECK, so every layer agrees. Layers (explicit value always wins): `MinionQueue.add()` stamps at submit; `MinionQueue.claim()` COALESCEs NULL columns from the maps (durable invariant); migration v128 one-shot backfilled `timeout_ms` with authoring-time snapshot values — do NOT sync v128 when editing the maps (`lock_duration_ms` has no backfill: NULL = worker default). `DEFAULT_CHAT_PER_TURN_TIMEOUT_MS` (30 min) and `resolveChatPerTurnTimeoutMs` resolve `ai.chat.per_turn_timeout_ms`, the per-turn chat timeout the subagent handler hands every gateway-loop turn (`ToolLoopOpts.turnTimeoutMs` → `ChatOpts.timeoutMs`, replacing the 300 s gateway backstop) and the oneshot sub-budget ceiling; the job's own signal still ends a turn at the job deadline (#4921). Pinned by `test/minions.test.ts` + `test/migrations-v128.test.ts` + `test/migrations-v130.test.ts` + `test/minions/subagent-turn-timeout.test.ts`.
diff --git a/src/core/minions/handlers/autopilot-global-maintenance.ts b/src/core/minions/handlers/autopilot-global-maintenance.ts
index 3106b2229b..030421f60f 100644
--- a/src/core/minions/handlers/autopilot-global-maintenance.ts
+++ b/src/core/minions/handlers/autopilot-global-maintenance.ts
@@ -424,6 +424,15 @@ export function makeAutopilotGlobalMaintenanceHandler(engine: BrainEngine, deps:
console.warn(`[autopilot-global-maintenance] failed to stamp last_global_at: ${e instanceof Error ? e.message : String(e)}`);
}
}
+ // #7854: the verdict `autopilot --status` reads is otherwise rewritten only on the daemon's next full-cycle tick.
+ if (stamped) {
+ try {
+ const { recordGlobalMaintenanceStallStatus } = await import('../global-maintenance-stall.ts');
+ await recordGlobalMaintenanceStallStatus(engine);
+ } catch (e) {
+ console.warn(`[autopilot-global-maintenance] failed to refresh the status verdict: ${e instanceof Error ? e.message : String(e)}`);
+ }
+ }
const streaks = Object.entries(progress.failures).map(([phase, s]) => ({ phase, ...s, fix: phaseStreakFix(phase, s) }));
const stampedPast = stamped ? streaks.filter(s => progress.pass_bypassed!.includes(s.phase)) : [];
diff --git a/test/autopilot-status-verdict-7854.test.ts b/test/autopilot-status-verdict-7854.test.ts
new file mode 100644
index 0000000000..95d177ca21
--- /dev/null
+++ b/test/autopilot-status-verdict-7854.test.ts
@@ -0,0 +1,59 @@
+/**
+ * #7854: `gbrain autopilot --status` reads the global-maintenance verdict file the daemon writes on a full-cycle
+ * tick. A clean pass that stamps `autopilot.last_global_at` must refresh that file, or --status keeps reporting
+ * `stalled` (exit 1) until the next full-cycle tick, which on a healthy brain can be hours away.
+ */
+import { afterAll, beforeAll, expect, test } from 'bun:test';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { LAST_GLOBAL_AT_KEY, type CycleReport, type PhaseResult } from '../src/core/cycle.ts';
+import { makeAutopilotGlobalMaintenanceHandler } from '../src/core/minions/handlers/autopilot-global-maintenance.ts';
+import { recordGlobalMaintenanceStallStatus } from '../src/core/minions/global-maintenance-stall.ts';
+import { runCli } from './helpers/cli-spawn.ts';
+import { withEnv } from './helpers/with-env.ts';
+
+let engine: PGLiteEngine;
+const home = mkdtempSync(join(tmpdir(), 'gbrain-7854-'));
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+});
+
+afterAll(async () => {
+ await engine.disconnect();
+ rmSync(home, { recursive: true, force: true });
+});
+
+test('a clean pass after a stalled verdict clears what autopilot --status reports', async () => {
+ let purge: 'ok' | 'fail' = 'fail';
+ const runCycle = async (_engine: unknown, opts: { phases?: string[] }): Promise<CycleReport> => {
+ const phase = opts.phases![0]!;
+ const result = (phase === 'purge' && purge === 'fail'
+ ? { phase, status: 'fail', duration_ms: 1, summary: 'invalid_data', details: {}, error: { class: 'InternalError', code: 'invalid_data', message: 'invalid_data' } }
+ : { phase, status: 'ok', duration_ms: 1, summary: 'ok', details: {} }) as PhaseResult;
+ return { schema_version: '1', timestamp: new Date().toISOString(), duration_ms: 1, status: result.status === 'fail' ? 'failed' : 'ok', phases: [result], totals: {} } as unknown as CycleReport;
+ };
+ let jobId = 7854;
+ const pass = () => withEnv({ GBRAIN_HOME: home }, () => makeAutopilotGlobalMaintenanceHandler(engine, { runCycle: runCycle as never })(
+ { id: ++jobId, attempts_made: 0, signal: undefined, deadlineAtMs: null, data: { phases: ['orphans', 'purge'], repoPath: null } } as never));
+
+ await engine.setConfig('autopilot.global_floor_min', '60');
+ await engine.setConfig(LAST_GLOBAL_AT_KEY, new Date(Date.now() - 5 * 60 * 60_000).toISOString());
+ await pass();
+ await withEnv({ GBRAIN_HOME: home }, () => recordGlobalMaintenanceStallStatus(engine));
+ const stalled = await runCli(['autopilot', '--status', '--json'], { home });
+ expect({ exit: stalled.exitCode, reason: JSON.parse(stalled.stdout).global_maintenance?.reason }).toEqual({ exit: 1, reason: 'stalled' });
+
+ purge = 'ok';
+ const before = await engine.getConfig(LAST_GLOBAL_AT_KEY);
+ await pass();
+ expect(await engine.getConfig(LAST_GLOBAL_AT_KEY)).not.toBe(before);
+ const cleared = await runCli(['autopilot', '--status', '--json'], { home });
+ const verdict = JSON.parse(cleared.stdout).global_maintenance;
+ expect({ exit: cleared.exitCode, status: verdict?.status, last_global_at: verdict?.last_global_at })
+ .toEqual({ exit: 0, status: 'ok', last_global_at: await engine.getConfig(LAST_GLOBAL_AT_KEY) });
+}, 120_000);