Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(autopilot): a clean global pass refreshes the verdict autopilot --status reads (#7854)
garrytan11bcfa459886
diff --git a/docs/architecture/key-files/core-minions-1.md b/docs/architecture/key-files/core-minions-1.mdindex faee7a1cf1..7bf2e3b07f 100644--- a/docs/architecture/key-files/core-minions-1.md+++ b/docs/architecture/key-files/core-minions-1.md@@ -32,7 +32,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/minions/errors.ts` — dependency-free `UnrecoverableError` and `JobDeferredError(reason, message, retryInMs)`, re-exported through `types.ts` for compatibility. Delegated policy imports this leaf directly, avoiding initialization cycles through durable authority. Fresh-process import tests preserve the shared error identity and isolated-worker reconstruction. A `JobDeferredError` returns the job to `delayed` for `retryInMs` through `MinionQueue.deferJob` (no attempt counted, no stacktrace entry); the worker, the facts drain and the isolation codec (`errorKind: 'deferred'`) all route it (`test/facts-absorb-keyless-deferral.test.ts`).-- `src/core/minions/global-maintenance-stall.ts` — brain-wide maintenance stall verdict (GBRA-130 C-6): `assessGlobalMaintenanceStall` (stale `last_global_at` plus failed jobs or a phase streak, or passes stamping past a failing phase or a triage deferral that stopped shrinking, #7307; a deferral streak's fix is `gbrain dream retriage`) feeds doctor `global_maintenance_stalled`; the daemon writes it to `autopilot-global-maintenance.json` each full-cycle tick (`recordGlobalMaintenanceStallStatus`) and `autopilot --status` reads it without the database (`readGlobalMaintenanceVerdict`).+- `src/core/minions/global-maintenance-stall.ts` — brain-wide maintenance stall verdict (GBRA-130 C-6): `assessGlobalMaintenanceStall` (stale `last_global_at` plus failed jobs or a phase streak, or passes stamping past a failing phase or a triage deferral that stopped shrinking, #7307; a deferral streak's fix is `gbrain dream retriage`) feeds doctor `global_maintenance_stalled`; the daemon writes it to `autopilot-global-maintenance.json` each full-cycle tick (`recordGlobalMaintenanceStallStatus`), and so does a global-maintenance pass that stamps `last_global_at` (#7854, `test/autopilot-status-verdict-7854.test.ts`), and `autopilot --status` reads it without the database (`readGlobalMaintenanceVerdict`).- `src/core/minions/handler-timeouts.ts` — per-handler-type defaults for BOTH per-job time knobs, co-located so they can't drift apart unseen (they are different quantities: the budget bounds total runtime, the lease bounds dead-worker reclaim — never derive one from the other). `HANDLER_DEFAULT_TIMEOUT_MS`: 30 min for `subagent`, `subagent_aggregator`, `embed-backfill`, `autopilot-cycle`, `autopilot-global-maintenance`, `extract` (a brain-wide `extract --stale` pass); 10 min for `chronicle_extract` + `facts-absorb`; 60 min for `contextual_reindex_per_chunk`. `HANDLER_DEFAULT_LOCK_DURATION_MS`: 300 s for the long LLM/loop handlers and for `sync` (its blocking git helpers can each outlast a 30 s lease), 120 s for the single-LLM-call handlers, `shell` deliberately absent (fast dead-worker reclaim; verify-before-evict protects it anyway). `defaultTimeoutMsFor` / `defaultLockDurationMsFor` return the mapped default or `null` (short handlers keep the tight null-default wall-clock / the 30 s worker lease). `clampLockDurationMs` + `LOCK_DURATION_MS_MIN/MAX` ([5s, 1h]) is the ONE clamp shared by `queue.add`, the CLI `--lock-duration-ms` flag (`--dry-run` echoes the clamped value), and the MCP `submit_job` param; the same bound is mirrored in SQL at claim time and by the `minion_jobs.lock_duration_ms` range CHECK, so every layer agrees. Layers (explicit value always wins): `MinionQueue.add()` stamps at submit; `MinionQueue.claim()` COALESCEs NULL columns from the maps (durable invariant); migration v128 one-shot backfilled `timeout_ms` with authoring-time snapshot values — do NOT sync v128 when editing the maps (`lock_duration_ms` has no backfill: NULL = worker default). `DEFAULT_CHAT_PER_TURN_TIMEOUT_MS` (30 min) and `resolveChatPerTurnTimeoutMs` resolve `ai.chat.per_turn_timeout_ms`, the per-turn chat timeout the subagent handler hands every gateway-loop turn (`ToolLoopOpts.turnTimeoutMs` → `ChatOpts.timeoutMs`, replacing the 300 s gateway backstop) and the oneshot sub-budget ceiling; the job's own signal still ends a turn at the job deadline (#4921). Pinned by `test/minions.test.ts` + `test/migrations-v128.test.ts` + `test/migrations-v130.test.ts` + `test/minions/subagent-turn-timeout.test.ts`.diff --git a/src/core/minions/handlers/autopilot-global-maintenance.ts b/src/core/minions/handlers/autopilot-global-maintenance.tsindex 3106b2229b..030421f60f 100644--- a/src/core/minions/handlers/autopilot-global-maintenance.ts+++ b/src/core/minions/handlers/autopilot-global-maintenance.ts@@ -424,6 +424,15 @@ export function makeAutopilotGlobalMaintenanceHandler(engine: BrainEngine, deps:console.warn(`[autopilot-global-maintenance] failed to stamp last_global_at: ${e instanceof Error ? e.message : String(e)}`);}}+ // #7854: the verdict `autopilot --status` reads is otherwise rewritten only on the daemon's next full-cycle tick.+ if (stamped) {+ try {+ const { recordGlobalMaintenanceStallStatus } = await import('../global-maintenance-stall.ts');+ await recordGlobalMaintenanceStallStatus(engine);+ } catch (e) {+ console.warn(`[autopilot-global-maintenance] failed to refresh the status verdict: ${e instanceof Error ? e.message : String(e)}`);+ }+ }const streaks = Object.entries(progress.failures).map(([phase, s]) => ({ phase, ...s, fix: phaseStreakFix(phase, s) }));const stampedPast = stamped ? streaks.filter(s => progress.pass_bypassed!.includes(s.phase)) : [];diff --git a/test/autopilot-status-verdict-7854.test.ts b/test/autopilot-status-verdict-7854.test.tsnew file mode 100644index 0000000000..95d177ca21--- /dev/null+++ b/test/autopilot-status-verdict-7854.test.ts@@ -0,0 +1,59 @@+/**+ * #7854: `gbrain autopilot --status` reads the global-maintenance verdict file the daemon writes on a full-cycle+ * tick. A clean pass that stamps `autopilot.last_global_at` must refresh that file, or --status keeps reporting+ * `stalled` (exit 1) until the next full-cycle tick, which on a healthy brain can be hours away.+ */+import { afterAll, beforeAll, expect, test } from 'bun:test';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { LAST_GLOBAL_AT_KEY, type CycleReport, type PhaseResult } from '../src/core/cycle.ts';+import { makeAutopilotGlobalMaintenanceHandler } from '../src/core/minions/handlers/autopilot-global-maintenance.ts';+import { recordGlobalMaintenanceStallStatus } from '../src/core/minions/global-maintenance-stall.ts';+import { runCli } from './helpers/cli-spawn.ts';+import { withEnv } from './helpers/with-env.ts';++let engine: PGLiteEngine;+const home = mkdtempSync(join(tmpdir(), 'gbrain-7854-'));++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+});++afterAll(async () => {+ await engine.disconnect();+ rmSync(home, { recursive: true, force: true });+});++test('a clean pass after a stalled verdict clears what autopilot --status reports', async () => {+ let purge: 'ok' | 'fail' = 'fail';+ const runCycle = async (_engine: unknown, opts: { phases?: string[] }): Promise<CycleReport> => {+ const phase = opts.phases![0]!;+ const result = (phase === 'purge' && purge === 'fail'+ ? { phase, status: 'fail', duration_ms: 1, summary: 'invalid_data', details: {}, error: { class: 'InternalError', code: 'invalid_data', message: 'invalid_data' } }+ : { phase, status: 'ok', duration_ms: 1, summary: 'ok', details: {} }) as PhaseResult;+ return { schema_version: '1', timestamp: new Date().toISOString(), duration_ms: 1, status: result.status === 'fail' ? 'failed' : 'ok', phases: [result], totals: {} } as unknown as CycleReport;+ };+ let jobId = 7854;+ const pass = () => withEnv({ GBRAIN_HOME: home }, () => makeAutopilotGlobalMaintenanceHandler(engine, { runCycle: runCycle as never })(+ { id: ++jobId, attempts_made: 0, signal: undefined, deadlineAtMs: null, data: { phases: ['orphans', 'purge'], repoPath: null } } as never));++ await engine.setConfig('autopilot.global_floor_min', '60');+ await engine.setConfig(LAST_GLOBAL_AT_KEY, new Date(Date.now() - 5 * 60 * 60_000).toISOString());+ await pass();+ await withEnv({ GBRAIN_HOME: home }, () => recordGlobalMaintenanceStallStatus(engine));+ const stalled = await runCli(['autopilot', '--status', '--json'], { home });+ expect({ exit: stalled.exitCode, reason: JSON.parse(stalled.stdout).global_maintenance?.reason }).toEqual({ exit: 1, reason: 'stalled' });++ purge = 'ok';+ const before = await engine.getConfig(LAST_GLOBAL_AT_KEY);+ await pass();+ expect(await engine.getConfig(LAST_GLOBAL_AT_KEY)).not.toBe(before);+ const cleared = await runCli(['autopilot', '--status', '--json'], { home });+ const verdict = JSON.parse(cleared.stdout).global_maintenance;+ expect({ exit: cleared.exitCode, status: verdict?.status, last_global_at: verdict?.last_global_at })+ .toEqual({ exit: 0, status: 'ok', last_global_at: await engine.getConfig(LAST_GLOBAL_AT_KEY) });+}, 120_000);