Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Park an upkeep item inside serve after 30 unfinished slices so a large brain stops retrying forever; the scheduled run or gbrain maintain --unattended finishes it, and status and doctor say so

garrytan0de6e3675654
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e07116b95e..299dd8b20f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,7 +22,7 @@ identifiers and attribution are available in the pre-removal Git revision
### Upkeep runs while your agent is connected, and `--schedule` covers the rest
-**An open `gbrain serve` now does the free upkeep for you, so a default install needs no service, cron job or second command.** When the agent connected to `gbrain serve` (stdio or HTTP) goes quiet, serve runs `gbrain maintain --unattended` in slices of about 3 seconds, at most one full round a day per brain, and stops a slice the moment a tool call arrives. It writes the same record that `gbrain maintain --status` and doctor read, with runner `serve`. It never deletes a page, and on a keyless brain it never spends anything. When two serves share one brain, only one runs it and the other skips. Turn it off with `gbrain config set maintenance.auto false` (or `GBRAIN_SWEEP=0`, which also stops the idle sweep).
+**An open `gbrain serve` now does the free upkeep for you, so a default install needs no service, cron job or second command.** When the agent connected to `gbrain serve` (stdio or HTTP) goes quiet, serve runs `gbrain maintain --unattended` in slices of about 3 seconds, at most one full round a day per brain, and stops a slice the moment a tool call arrives. It writes the same record that `gbrain maintain --status` and doctor read, with runner `serve`. It never deletes a page, and on a keyless brain it never spends anything. When two serves share one brain, only one runs it and the other skips. A source too big to finish within 30 slices is parked: the rest of the round finishes, serve stops doing upkeep until the next round (it tries the parked source once more after each restart), and `gbrain maintain --status` and doctor warn and name it, because `gbrain maintain --unattended` or the schedule below has to finish it. Turn it off with `gbrain config set maintenance.auto false` (or `GBRAIN_SWEEP=0`, which also stops the idle sweep).
- `gbrain maintain --schedule` is for a brain with no agent session open most days. It installs a job that runs `gbrain maintain --unattended` every 6 hours (launchd on macOS, a systemd user timer on Linux, cron otherwise). It asks before installing, `--dry-run` shows exactly what it would write, and `--unschedule` removes it. Nothing installs it by default. A scheduled run that finds a live `gbrain serve` on the brain exits 0 with `maintenance_owned_by_serve` and leaves the work to serve.
- `gbrain init` and `gbrain setup <harness>` now say how upkeep works in their first-run settings block, and how to turn it off.
@@ -45,11 +45,11 @@ identifiers and attribution are available in the pre-removal Git revision
- `src/commands/autopilot-refusals.ts` (new) and `src/core/error-registry.ts`: new codes `autopilot_no_repo`, `autopilot_pglite_daemon`, `cycle_already_running`, `engine_degraded`, `maintenance_disabled` and `maintenance_owned_by_autopilot`.
- Tests: `test/maintenance-unattended.test.ts` (PGLite and Postgres) and `test/e2e/maintain-unattended-keyless-pglite.test.ts`.
- `src/core/maintenance/idle.ts` (new): `runIdleUpkeep` (the stdio serve calls it from its idle tick, after the sweep) and `armIdleUpkeep` (the HTTP serve's new idle timer), with a 3-second budget per run. `src/core/foreground-activity.ts` gains `onForegroundRequest`, which aborts a run in flight, and `src/commands/serve-http.ts` marks each `/mcp` POST as foreground work.
-- `src/core/maintenance/run.ts`: a PGLite brain runs `CHECKPOINT` after a finished round, recorded as the `checkpoint` step.
+- `src/core/maintenance/run.ts`: a PGLite brain runs `CHECKPOINT` after a finished round, recorded as the `checkpoint` step. A budgeted run counts each item's unfinished slices and parks the item at `UPKEEP_PARK_AFTER_SLICES` (30; a fresh 10,000-page brain needs 24 quiet slices for its first round), recorded under `parked`. Without this, a source whose facts walk alone outlasts a 3-second slice (past about 12,000 pages) restarted forever and the round never finished.
- `src/commands/extract.ts`, `src/core/cycle.ts` and `src/core/cycle/connector-extract.ts`: the cycle's stale link drain now stops between batches of 25 pages when the cycle is aborted. Before, a first upkeep run on a 10,000-page brain with no repository spent 64 seconds in this one step, past its 3-second budget, and tool calls made meanwhile took about twice as long.
- `src/core/cycle/extract-facts.ts`: the facts walk now gives the event loop a turn every 50 pages. On PGLite, queries finish without ever letting timers or stdin run, so on a 10,000-page brain the walk froze the serve for about 2 seconds and a tool call sent during it waited until the walk ended (p99 1.7 seconds). With the fix, the freeze is under 150 ms, a call stops the walk, and tool calls during upkeep stay at p99 241 ms against 139 ms with upkeep off.
- `src/commands/maintain-schedule.ts` (new): `--schedule`, `--unschedule` and the live-serve check, routed before the engine opens. New codes `maintenance_owned_by_serve`, `schedule_unsupported` (Windows and containers) and `schedule_install_failed`.
-- Tests: `test/serve-idle-upkeep-gbra438.test.ts`, `test/maintenance-idle-gbra438.test.ts` (PGLite and Postgres, including two serves on one brain), `test/maintain-schedule-gbra438.test.ts` and `test/e2e/serve-idle-upkeep-keyless-pglite.test.ts`.
+- Tests: `test/serve-idle-upkeep-gbra438.test.ts`, `test/maintenance-idle-gbra438.test.ts` (PGLite and Postgres, including two serves on one brain), `test/maintenance-park-gbra438.test.ts`, `test/extract-stale-managed-signal-gbra438.test.ts` (PGLite and Postgres), `test/maintain-schedule-gbra438.test.ts` and `test/e2e/serve-idle-upkeep-keyless-pglite.test.ts`.
## [0.60.699.0] - 2026-11-12
diff --git a/docs/architecture/key-files/commands-4-continued.md b/docs/architecture/key-files/commands-4-continued.md
index 9f45135175..6d2fd2a206 100644
--- a/docs/architecture/key-files/commands-4-continued.md
+++ b/docs/architecture/key-files/commands-4-continued.md
@@ -35,5 +35,5 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/embedding-migration-cli.ts` — pure, positive migration control contract, shared by the pre-engine CLI boundary and the command parser (re-exported by `commands/migrate-embeddings.ts`). Validates original argv as well as command arguments, so consumed global timeout/progress flags cannot evade validation. Unknown controls, unsupported equals forms, duplicate flags, missing values, invalid numeric ranges and unknown pace modes refuse rather than falling back, truncating, clamping or widening work. Only implemented globals are accepted; no new fact-only or deadline behavior. The flag-registry generator includes this peeled surface; the registry's embed/import unions do not define migration legality.
- `test/migrate-embeddings-recovery-cli.serial.test.ts` — real migration CLI subprocesses over isolated synthetic PGLite brains. Pins recovery envelopes and private-safe help/stderr, then exercises unsupported scope/work flags, consumed timeout, malformed numeric controls, supported pacing/globals/status/help, zero dispatch and unchanged DB/config snapshots on both routes. Direct parser cases use the same adversarial matrix. Help directs deferred work back through the migration with the existing total authorization, never standalone/background embedding. The static `EMBEDDING_MIGRATION_RECOVERY` is shared with the local-only admin operation; its unchanged discriminators and restrictions remain covered by `test/migrate-embeddings-op-contract.serial.test.ts`. Archived pending projections still block destructive transitions; no silent restore or source exclusion is introduced.
-- `src/core/maintenance/run.ts` + `src/core/maintenance/status.ts` — GBRA-438 free unattended upkeep. `unattendedPlan` is the `--safe` list plus `bookkeeping_gc`, split into one global item (brain-wide phases, once per window under `gbrain-cycle`) and one item per source; cycles run with `stamp: 'none'` and `deferFactEmbedding` on a keyed brain. `runUnattendedMaintenance` never throws: preflight skips (`maintenance_disabled`, `engine_degraded`, `migrations_pending`, `maintenance_owned_by_autopilot` when `autopilot.last_global_at` is under 48h old, `write_capacity_backpressure`, `cycle_already_running`) exit 0; a failed step records its code (exit 1 for `maintain`) and three same-code failures back the step off to once a week. The embedding step runs under `maintenance.embed_daily_usd` (default $1 per UTC day, 0 = off; spend in `maintenance.embed_spend`). The record is `maintenance.last_run` (database config, so repo-less brains have one). `readUpkeepStatus` is the one view behind `--status` and doctor. Tests: `test/maintenance-unattended.test.ts` (both engines).
+- `src/core/maintenance/run.ts` + `src/core/maintenance/status.ts` — GBRA-438 free unattended upkeep. `unattendedPlan` is the `--safe` list plus `bookkeeping_gc`, split into one global item (brain-wide phases, once per window under `gbrain-cycle`) and one item per source; cycles run with `stamp: 'none'` and `deferFactEmbedding` on a keyed brain. `runUnattendedMaintenance` never throws: preflight skips (`maintenance_disabled`, `engine_degraded`, `migrations_pending`, `maintenance_owned_by_autopilot` when `autopilot.last_global_at` is under 48h old, `write_capacity_backpressure`, `cycle_already_running`) exit 0; a failed step records its code (exit 1 for `maintain`) and three same-code failures back the step off to once a week. A budgeted run parks an item unfinished after `UPKEEP_PARK_AFTER_SLICES` (30) slices in `parked` (skipped by later budgeted runs, rerun by an unbudgeted run or `retryParked`, a serve's first idle run; `readUpkeepStatus` state `parked` warns); test `test/maintenance-park-gbra438.test.ts`. The embedding step runs under `maintenance.embed_daily_usd` (default $1 per UTC day, 0 = off; spend in `maintenance.embed_spend`). The record is `maintenance.last_run` (database config, so repo-less brains have one). `readUpkeepStatus` is the one view behind `--status` and doctor. Tests: `test/maintenance-unattended.test.ts` (both engines).
- `src/core/maintenance/idle.ts` + `src/commands/maintain-schedule.ts` — GBRA-438 PR 2, the upkeep runners. `runIdleUpkeep` runs `runUnattendedMaintenance` as runner `serve` with `IDLE_UPKEEP_BUDGET_MS` (3 s) when upkeep is on and due (`upkeepDue`); the stdio serve calls it from its idle tick after the sweep, in the same in-flight slot, and stdin data, `onForegroundRequest` or shutdown aborts it. `armIdleUpkeep` is the HTTP serve's idle timer: `mcpForegroundMiddleware` (in `src/core/foreground-activity.ts`, mounted on `/mcp` by `serve-http.ts`) marks each `/mcp` POST foreground, the first tick only opens the window, one run at a time. A finished round on PGLite ends with a `CHECKPOINT` step. `maintain-schedule.ts` is routed engine-free from `src/cli/main.ts`: `--schedule` (consent `persistent_install`, `--dry-run` preview) writes a wrapper and a launchd plist, systemd user service and timer, or a `# gbrain-maintain` crontab line (pure renderers, every 6 hours); Windows and containers refuse `schedule_unsupported`; `--unschedule` removes it. A scheduled `--unattended --runner schedule` run exits 0 `maintenance_owned_by_serve` while a live serve holds the brain (PGLite lock holder, or the CLI socket on Postgres). Tests: `test/serve-idle-upkeep-gbra438.test.ts`, `test/maintenance-idle-gbra438.test.ts` (both engines, two serves on one brain), `test/maintain-schedule-gbra438.test.ts`, `test/e2e/serve-idle-upkeep-keyless-pglite.test.ts`. The cycle hands its signal to the repo-less stale link drain (`extractStaleFromDB` `signal`), so a slice stops between 25-page batches, and `runExtractFacts` yields a macrotask every 50 pages of its walk, so a timer or stdin can trip the abort mid-walk on PGLite. Bench: `scripts/bench/serve-upkeep-latency.ts` (tool-call latency during idle upkeep on a keyless repo-less PGLite brain, against a sweep-only baseline).
diff --git a/docs/guides/cron-schedule.md b/docs/guides/cron-schedule.md
index f70c195f7c..9a22f87af0 100644
--- a/docs/guides/cron-schedule.md
+++ b/docs/guides/cron-schedule.md
@@ -38,7 +38,9 @@ scheduling surfaces — reach for these first:
backoff, and an audit trail. See the `minion-orchestrator` skill.
- **`gbrain serve`** — while the agent connected to it is idle, it runs the
free upkeep (`gbrain maintain --unattended`) once a day, with nothing to
- install. `gbrain maintain --status` shows the last run.
+ install. A source too big to finish in its short idle slices is parked after
+ 30 tries and finished by the next full run. `gbrain maintain --status` shows
+ the last run and anything parked.
- **`gbrain maintain --schedule`** — for a brain no agent opens most days:
installs a job that runs the same upkeep every 6 hours (launchd, a systemd
user timer or cron). Opt-in; preview with `--dry-run`.
diff --git a/llms-full.txt b/llms-full.txt
index b8cf7b9a20..9bd6452fb9 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -7094,7 +7094,9 @@ scheduling surfaces — reach for these first:
backoff, and an audit trail. See the `minion-orchestrator` skill.
- **`gbrain serve`** — while the agent connected to it is idle, it runs the
free upkeep (`gbrain maintain --unattended`) once a day, with nothing to
- install. `gbrain maintain --status` shows the last run.
+ install. A source too big to finish in its short idle slices is parked after
+ 30 tries and finished by the next full run. `gbrain maintain --status` shows
+ the last run and anything parked.
- **`gbrain maintain --schedule`** — for a brain no agent opens most days:
installs a job that runs the same upkeep every 6 hours (launchd, a systemd
user timer or cron). Opt-in; preview with `--dry-run`.
diff --git a/plugin-variants/gbrain-coding/skills/maintain/SKILL.md b/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
index 59bc1c0665..a8db41fdcd 100644
--- a/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
+++ b/plugin-variants/gbrain-coding/skills/maintain/SKILL.md
@@ -305,8 +305,9 @@ Note: extracted entries improve structured queries (`gbrain timeline`), not vect
### Upkeep and autopilot check
`gbrain maintain --status` shows the last free-upkeep run, which an open
-`gbrain serve` does while idle. For a brain no agent opens most days, offer
-`gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
+`gbrain serve` does while idle; an item serve could not finish in 30 slices is
+listed as parked, and `gbrain maintain --unattended` finishes it. For a brain
+no agent opens most days, offer `gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
verify autopilot:
```bash
gbrain autopilot --status
diff --git a/plugin/skills/maintain/SKILL.md b/plugin/skills/maintain/SKILL.md
index 59bc1c0665..a8db41fdcd 100644
--- a/plugin/skills/maintain/SKILL.md
+++ b/plugin/skills/maintain/SKILL.md
@@ -305,8 +305,9 @@ Note: extracted entries improve structured queries (`gbrain timeline`), not vect
### Upkeep and autopilot check
`gbrain maintain --status` shows the last free-upkeep run, which an open
-`gbrain serve` does while idle. For a brain no agent opens most days, offer
-`gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
+`gbrain serve` does while idle; an item serve could not finish in 30 slices is
+listed as parked, and `gbrain maintain --unattended` finishes it. For a brain
+no agent opens most days, offer `gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
verify autopilot:
```bash
gbrain autopilot --status
diff --git a/scripts/bench/serve-upkeep-latency.ts b/scripts/bench/serve-upkeep-latency.ts
index f3c1b9edbc..6d0f88ba87 100644
--- a/scripts/bench/serve-upkeep-latency.ts
+++ b/scripts/bench/serve-upkeep-latency.ts
@@ -13,6 +13,12 @@
*
* bun scripts/bench/serve-upkeep-latency.ts --pages 1401 --out ~/upkeep-latency [--calls 150] [--links 3]
* [--interval-ms 1000] [--gap-ms 1200-3200] [--baseline-calls 150] [--max-copies 8]
+ * [--small-source-pages 0] [--quiet-ms 0]
+ *
+ * --small-source-pages adds a second source (zz-small, after default in item order) so a default source too big for
+ * the slice budget shows the give-up rule: it parks and zz-small still finishes. --quiet-ms adds a quiet arm: a fresh
+ * copy served with no tool calls for that long, then served again after a restart (its first idle run retries parked
+ * items), reporting the upkeep slices per serve process, when the last one ended, and the parked record.
*
* Writes calls.jsonl and summary.json under --out and prints p50/p95/p99/max per arm and tag.
*/
@@ -26,6 +32,7 @@ const CALLS = Number(flag('calls', '150')), BASELINE_CALLS = Number(flag('baseli
const INTERVAL_MS = Number(flag('interval-ms', '1000')), MAX_COPIES = Number(flag('max-copies', '8'));
const [GAP_MIN, GAP_MAX] = flag('gap-ms', '1200-3200').split('-').map(Number) as [number, number];
const OUT = resolve(flag('out', join(process.cwd(), 'upkeep-latency')));
+const SMALL_PAGES = Number(flag('small-source-pages', '0')), QUIET_MS = Number(flag('quiet-ms', '0'));
const CLI = join(import.meta.dir, '..', '..', 'src', 'cli.ts');
const WORDS = ['alpha', 'harbor', 'ledger', 'orbit', 'meadow', 'signal', 'copper', 'lantern', 'summit', 'river', 'quartz', 'falcon'];
@@ -36,10 +43,11 @@ function keylessEnv(home: string, extra: Record<string, string> = {}): Record<st
return { ...env, HOME: home, GBRAIN_HOME: home, GBRAIN_SKIP_STARTUP_HOOKS: '1', ...extra };
}
-async function gbrain(home: string, args: string[]): Promise<void> {
+async function gbrain(home: string, args: string[]): Promise<string> {
const p = Bun.spawn([process.execPath, '--no-env-file', CLI, ...args], { cwd: home, env: keylessEnv(home), stdout: 'pipe', stderr: 'pipe' });
- const [code, err] = await Promise.all([p.exited, new Response(p.stderr).text()]);
+ const [code, out, err] = await Promise.all([p.exited, new Response(p.stdout).text(), new Response(p.stderr).text()]);
if (code !== 0) throw new Error(`gbrain ${args.join(' ')} exited ${code}: ${err.slice(-2000)}`);
+ return out;
}
const quantile = (xs: number[], q: number) => { const s = [...xs].sort((a, b) => a - b); return s.length ? s[Math.min(s.length - 1, Math.ceil(q * s.length) - 1)]! : NaN; };
@@ -61,6 +69,19 @@ async function buildPristine(): Promise<string> {
writeFileSync(join(home, 'notes', `${name(i)}.md`), `---\ntitle: Page ${i}\ntype: concept\n---\n# Page ${i}\n\nPage ${i} is about ${words}. It relates to ${links}.\n\n## Timeline\n\n- **2026-${month}-${day}** | Notes — Page ${i} was reviewed.\n`);
}
await gbrain(home, ['import', join(home, 'notes'), '--no-embed']);
+ if (SMALL_PAGES > 0) {
+ mkdirSync(join(home, 'small'), { recursive: true });
+ for (let i = 0; i < SMALL_PAGES; i++) writeFileSync(join(home, 'small', `s${i}.md`), `---
+title: Small ${i}
+type: concept
+---
+# Small ${i}
+
+Small ${i} relates to [[s${(i + 1) % SMALL_PAGES}]].
+`);
+ await gbrain(home, ['sources', 'add', 'zz-small', '--path', join(home, 'small'), '--force']);
+ await gbrain(home, ['import', join(home, 'small'), '--source', 'zz-small', '--no-embed']);
+ }
cpSync(join(home, '.gbrain'), join(OUT, 'snapshot', '.gbrain'), { recursive: true });
writeFileSync(join(OUT, 'snapshot', 'built'), `${Date.now() - started}\n`);
console.log(`built ${PAGES} pages in ${Math.round((Date.now() - started) / 1000)} s`);
@@ -69,14 +90,19 @@ async function buildPristine(): Promise<string> {
interface Call { arm: string; copy: number; tag: 'upkeep' | 'sweep' | 'idle'; ms: number; ok: boolean }
-async function runCopy(arm: 'upkeep' | 'baseline', copy: number, home: string, want: () => boolean, record: (c: Call) => void): Promise<{ upkeepFinished: number[]; roundDone: boolean }> {
- rmSync(join(home, '.gbrain'), { recursive: true, force: true });
- cpSync(join(OUT, 'snapshot', '.gbrain'), join(home, '.gbrain'), { recursive: true });
+async function runCopy(arm: 'upkeep' | 'baseline' | 'quiet', copy: number, home: string, want: () => boolean, record: (c: Call) => void,
+ opts: { restore?: boolean; quietMs?: number } = {}): Promise<{ upkeepFinished: number[]; roundDone: boolean; lastUpkeepAt: number | null; servedMs: number; parked: unknown }> {
+ if (opts.restore !== false) {
+ rmSync(join(home, '.gbrain'), { recursive: true, force: true });
+ cpSync(join(OUT, 'snapshot', '.gbrain'), join(home, '.gbrain'), { recursive: true });
+ }
+ const servedAt = Date.now();
if (arm === 'baseline') await gbrain(home, ['config', 'set', 'maintenance.auto', 'false']);
const serve = Bun.spawn([process.execPath, '--no-env-file', CLI, 'serve'], { cwd: home, env: keylessEnv(home, { GBRAIN_SWEEP_IDLE_INTERVAL_MS: String(INTERVAL_MS) }), stdin: 'pipe', stdout: 'pipe', stderr: 'pipe' });
let log = '';
const pending = new Map<number, (ok: boolean) => void>();
- const errReader = (async () => { for await (const chunk of serve.stderr) { const text = new TextDecoder().decode(chunk); log += text; appendFileSync(join(OUT, `${arm}-${copy}.serve.log`), text); } })();
+ const upkeepEnds: number[] = [];
+ const errReader = (async () => { for await (const chunk of serve.stderr) { const text = new TextDecoder().decode(chunk); log += text; for (const _ of text.matchAll(/idle upkeep: finished/g)) upkeepEnds.push(Date.now() - servedAt); appendFileSync(join(OUT, `${arm}-${copy}.serve.log`), text); } })();
const outReader = (async () => {
let buf = '';
for await (const chunk of serve.stdout) {
@@ -102,6 +128,7 @@ async function runCopy(arm: 'upkeep' | 'baseline', copy: number, home: string, w
await send('initialize', { protocolVersion: '2025-06-18', capabilities: {}, clientInfo: { name: 'upkeep-latency', version: '1' } });
serve.stdin.write(JSON.stringify({ jsonrpc: '2.0', method: 'notifications/initialized' }) + '\n');
for (let w = 0; w < 5; w++) await send('tools/call', { name: 'search', arguments: { query: WORDS[w]! } });
+ if (opts.quietMs) await sleep(opts.quietMs);
let quietSince = Date.now(), lastStarts = count(/idle upkeep: started/g), i = 0;
while (want()) {
await sleep(GAP_MIN + Math.random() * (GAP_MAX - GAP_MIN));
@@ -121,7 +148,8 @@ async function runCopy(arm: 'upkeep' | 'baseline', copy: number, home: string, w
await Promise.allSettled([errReader, outReader]);
}
const upkeepFinished = [...log.matchAll(/idle upkeep: finished in (\d+) ms/g)].map((m) => Number(m[1]));
- return { upkeepFinished, roundDone };
+ const status = JSON.parse(await gbrain(home, ['maintain', '--status', '--json'])) as { state: string; parked: unknown };
+ return { upkeepFinished, roundDone, lastUpkeepAt: upkeepEnds.length ? upkeepEnds[upkeepEnds.length - 1]! : null, servedMs: Date.now() - servedAt, parked: { state: status.state, parked: status.parked } };
}
mkdirSync(OUT, { recursive: true });
@@ -136,7 +164,15 @@ let baselineCalls = 0;
await runCopy('baseline', 0, home, () => baselineCalls++ < BASELINE_CALLS, record);
for (let copy = 0; copy < MAX_COPIES && duringUpkeep() < CALLS; copy++) {
const r = await runCopy('upkeep', copy, home, () => duringUpkeep() < CALLS, record);
- runs.push({ copy, round_done: r.roundDone, upkeep_slices: r.upkeepFinished.length, slice_ms: stats(r.upkeepFinished) });
+ runs.push({ copy, round_done: r.roundDone, upkeep_slices: r.upkeepFinished.length, slice_ms: stats(r.upkeepFinished), status: r.parked });
+}
+const quiet: Record<string, unknown>[] = [];
+if (QUIET_MS > 0) {
+ for (const restart of [false, true]) {
+ const r = await runCopy('quiet', 0, home, () => false, record, { restore: !restart, quietMs: QUIET_MS });
+ quiet.push({ restart, served_ms: r.servedMs, upkeep_slices: r.upkeepFinished.length, slice_ms: stats(r.upkeepFinished),
+ last_upkeep_ended_ms: r.lastUpkeepAt, quiet_tail_ms: r.servedMs - (r.lastUpkeepAt ?? 0), status: r.parked });
+ }
}
const pick = (arm: string, tag?: string) => calls.filter((c) => c.arm === arm && (!tag || c.tag === tag)).map((c) => c.ms);
@@ -149,6 +185,7 @@ const summary = {
added_vs_baseline_ms: { p95: during.p95 - base.p95, p99: during.p99 - base.p99, max: during.max - base.max },
failed_calls: calls.filter((c) => !c.ok).length,
copies: runs,
+ ...(quiet.length ? { quiet } : {}),
};
writeFileSync(join(OUT, 'summary.json'), JSON.stringify(summary, null, 2) + '\n');
console.log(JSON.stringify(summary, null, 2));
diff --git a/skills/maintain/SKILL.md b/skills/maintain/SKILL.md
index 4280452a6e..c9aaad2378 100644
--- a/skills/maintain/SKILL.md
+++ b/skills/maintain/SKILL.md
@@ -304,8 +304,9 @@ Note: extracted entries improve structured queries (`gbrain timeline`), not vect
### Upkeep and autopilot check
`gbrain maintain --status` shows the last free-upkeep run, which an open
-`gbrain serve` does while idle. For a brain no agent opens most days, offer
-`gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
+`gbrain serve` does while idle; an item serve could not finish in 30 slices is
+listed as parked, and `gbrain maintain --unattended` finishes it. For a brain
+no agent opens most days, offer `gbrain maintain --schedule` (asks consent). On a Postgres brain with a repo,
verify autopilot:
```bash
gbrain autopilot --status
diff --git a/skills/skills.lock.json b/skills/skills.lock.json
index 1569784d97..3aef22b0c6 100644
--- a/skills/skills.lock.json
+++ b/skills/skills.lock.json
@@ -95,7 +95,7 @@
"idea-lineage/SKILL.md": "8cb36179dbf94abc17065613b8e773b1bb2b22670742aa77347b91ac885a05c5",
"idea-lineage/routing-eval.jsonl": "ee2e00704b9accb7dd58bb8f126a3bc04a2c40be499180fa505dbf6d5061cd41",
"ingest/SKILL.md": "1acc220ce6e9e65876fa2ccede6ce11f9c75457feb6695ef904955f8debecaae",
- "maintain/SKILL.md": "f2ae80775596f41309efdb90858a471d2229f06ff4a783b87f841cd084588276",
+ "maintain/SKILL.md": "b54e1a3d46f22c16cfd8bed1d7088d693a1ee96214ebeba7c43d24645300564d",
"manifest.json": "357aa185d380f0a357cb15cd513317e290cbc5fcf5ca4e3860b0e829fdcbf1e4",
"mcp-access/SKILL.md": "1bc16338b89f476aa8c24ffec79c989fa60716793998af264d357f7886d2bc96",
"measure-before-you-fix/SKILL.md": "11906da29a1a31976b13d4796b686d9a9981825737846a85f32d88bf431b9b36",
diff --git a/src/commands/doctor/checks/unattended-maintenance.ts b/src/commands/doctor/checks/unattended-maintenance.ts
index 43710573f5..51e698be6c 100644
--- a/src/commands/doctor/checks/unattended-maintenance.ts
+++ b/src/commands/doctor/checks/unattended-maintenance.ts
@@ -3,7 +3,7 @@
* (or autopilot's freshness when an autopilot owns upkeep) through the same
* view as `gbrain maintain --status`. Off and autopilot-owned read as ok; a
* brain younger than 48 hours with no run is information; a run older than
- * 48 hours, or a step failing three runs in a row, warns with the command
+ * 48 hours, a step failing three runs in a row, or an item the serve parked, warns with the command
* that runs free upkeep now.
*/
import type { Check } from '../../doctor.ts';
@@ -20,6 +20,7 @@ function details(s: UpkeepStatus): Record<string, unknown> {
pending_purge_count: s.pending_purge_count, paid_phases_opt_in: s.paid_phases_opt_in,
embed_spend_today_usd: s.embed_spend_today_usd, embed_daily_cap_usd: s.embed_daily_cap_usd,
...(s.failing ? { failing: s.failing } : {}),
+ ...(s.parked.length ? { parked: s.parked } : {}),
...(s.alternative_fix ? { alternative_fix: renderAction(s.alternative_fix, cliRenderContext()) } : {}),
};
}
@@ -28,7 +29,7 @@ export async function checkUnattendedMaintenance(engine: Parameters<typeof readU
try {
const s = await readUpkeepStatus(engine, opts.nowMs);
if (s.state === 'never') return infoCheck(name, s.message, 'not_applicable', undefined, details(s));
- if (s.state === 'overdue' || s.state === 'failing') return { name, status: 'warn', message: s.message, details: details(s), ...(s.fix ? { fix: s.fix } : {}) };
+ if (s.state === 'overdue' || s.state === 'failing' || s.state === 'parked') return { name, status: 'warn', message: s.message, details: details(s), ...(s.fix ? { fix: s.fix } : {}) };
return { name, status: 'ok', message: s.message, details: details(s) };
} catch (error) {
return checkError(name, 'read the unattended upkeep record', error);
diff --git a/src/commands/maintain.ts b/src/commands/maintain.ts
index 10e6e76168..c8772fd355 100644
--- a/src/commands/maintain.ts
+++ b/src/commands/maintain.ts
@@ -341,6 +341,7 @@ async function runStatusCli(engine: BrainEngine, opts: MaintainOptions): Promise
}
console.log(status.message);
if (status.last_run) console.log(`Last run: ${status.last_run} (${status.runner}), next due: ${status.next_due_at}`);
+ for (const p of status.parked) console.log(`Parked: ${p.key} since ${p.since} (${p.slices} slices)`);
if (status.held.length) console.log(`Held back on this brain (they would call a provider): ${status.held.join(', ')}`);
console.log(`Left for you (paid, opt-in; gbrain dream or an autopilot runs them): ${status.paid_phases_opt_in.join(', ')}`);
if (fix) console.log(`Fix: ${fix.argv!.join(' ')}: ${fix.why}`);
diff --git a/src/commands/serve.ts b/src/commands/serve.ts
index c25d973a3b..c361222e8f 100644
--- a/src/commands/serve.ts
+++ b/src/commands/serve.ts
@@ -3,6 +3,7 @@ import { PoolStallWatch } from '../core/postgres-engine/pool-stall-watch.ts';
import { installRestartRequiredExit, recordServeTransport, resolveRestartRequiredExitSeconds, SERVE_RESTART_REQUIRED_EXIT_ENV } from './serve-health.ts';
import { SERVE_UNHEALTHY_EXIT_CODE } from '../core/exit-codes.ts';
import type { BrainEngine } from '../core/engine.ts';
+import type { IdleUpkeep } from '../core/maintenance/idle.ts';
// C10: `--fail-fast` / GBRAIN_SERVE_FAIL_FAST=1 — exit with the classified envelope instead of status-only/degraded mode (decided in src/cli.ts's connect path; F4's status mode consults it too).
export { serveFailFastRequested, writeServeFailFastEnvelope } from '../core/serve-fail-fast.ts';
import { isEngineDegraded as isEngineDegradedForServe } from '../core/degraded-marker.ts';
@@ -80,11 +81,11 @@ function idleSweepIntervalMs(): number {
}
/** GBRA-438: idle upkeep, skipped while a delegated sync owns the event loop (the sweep's rule); loaded lazily, off the boot path. */
-function defaultIdleUpkeep(log: (line: string) => void): (engine: BrainEngine, signal: AbortSignal) => Promise<unknown> {
- return async (engine, signal) => {
+function defaultIdleUpkeep(log: (line: string) => void): IdleUpkeep {
+ return async (engine, signal, retryParked) => {
if ((await loadSyncRunner()).isDelegatedSyncRunning()) return;
const { runIdleUpkeep } = await import('../core/maintenance/idle.ts');
- await runIdleUpkeep(engine, { runner: 'serve', signal, log });
+ await runIdleUpkeep(engine, { runner: 'serve', signal, retryParked, log });
};
}
@@ -92,8 +93,9 @@ function defaultIdleUpkeep(log: (line: string) => void): (engine: BrainEngine, s
* GBRA-438: the stdio serve's idle upkeep, one run at a time. abort() (stdin data), a foreground request or stop()
* (shutdown) aborts the run in flight. run() never throws.
*/
-function idleUpkeepSlot(upkeep: (engine: BrainEngine, signal: AbortSignal) => Promise<unknown>): { run(engine: BrainEngine): Promise<void>; abort(): void; stop(): void } {
+function idleUpkeepSlot(upkeep: IdleUpkeep): { run(engine: BrainEngine): Promise<void>; abort(): void; stop(): void } {
let running: AbortController | null = null;
+ let first = true;
const abort = (): void => { running?.abort(); };
const unsubscribe = onForegroundRequest(abort);
return {
@@ -101,7 +103,9 @@ function idleUpkeepSlot(upkeep: (engine: BrainEngine, signal: AbortSignal) => Pr
if (running) return;
const controller = new AbortController();
running = controller;
- try { await upkeep(engine, controller.signal); } catch { /* idle upkeep is best-effort; never kill serve */ } finally { running = null; }
+ const retryParked = first;
+ first = false;
+ try { await upkeep(engine, controller.signal, retryParked); } catch { /* idle upkeep is best-effort; never kill serve */ } finally { running = null; }
},
abort,
stop() { unsubscribe(); abort(); },
@@ -183,8 +187,8 @@ export interface ServeOptions {
sweep?: (engine: BrainEngine) => Promise<unknown>;
// Test seam for idle upkeep (GBRA-438): one run with the abort signal stdin
// data or a foreground request trips. Defaults to runIdleUpkeep as runner
- // 'serve' (on and due only, small budget, never at boot).
- upkeep?: (engine: BrainEngine, signal: AbortSignal) => Promise<unknown>;
+ // 'serve' (on and due only, small budget, never at boot); retryParked is true on the process's first run.
+ upkeep?: IdleUpkeep;
// Kill switch seam for the idle sweep. Defaults to
// `process.env.GBRAIN_SWEEP !== '0'` when omitted.
sweepEnabled?: boolean;
diff --git a/src/core/maintenance/idle.ts b/src/core/maintenance/idle.ts
index 0b579a1b7f..966098d0bc 100644
--- a/src/core/maintenance/idle.ts
+++ b/src/core/maintenance/idle.ts
@@ -21,6 +21,8 @@ export interface IdleUpkeepOpts {
runner: string;
signal: AbortSignal;
budgetMs?: number;
+ /** The process's first idle run: parked items get one more set of slices (UpkeepOpts.retryParked). */
+ retryParked?: boolean;
log?: (line: string) => void;
/** Test seams, passed to the runner. */
now?: () => Date;
@@ -32,26 +34,30 @@ export interface IdleUpkeepOpts {
export async function runIdleUpkeep(engine: BrainEngine, opts: IdleUpkeepOpts): Promise<UpkeepOutcome | null> {
try {
if (opts.signal.aborted || !await upkeepEnabled(engine)) return null;
- if (!upkeepDue(await readUpkeepRecord(engine), (opts.now ?? (() => new Date()))())) return null;
+ const record = await readUpkeepRecord(engine);
+ if (!upkeepDue(record, (opts.now ?? (() => new Date()))()) && !(opts.retryParked && record?.parked?.length)) return null;
} catch {
return null;
}
const startedAt = Date.now();
opts.log?.(`[gbrain serve] idle upkeep: started`);
const outcome = await runUnattendedMaintenance(engine, {
- runner: opts.runner, budgetMs: opts.budgetMs ?? IDLE_UPKEEP_BUDGET_MS, signal: opts.signal,
+ runner: opts.runner, budgetMs: opts.budgetMs ?? IDLE_UPKEEP_BUDGET_MS, signal: opts.signal, ...(opts.retryParked ? { retryParked: true } : {}),
...(opts.now ? { now: opts.now } : {}), ...(opts.providers ? { providers: opts.providers } : {}), ...(opts.cycle ? { cycle: opts.cycle } : {}),
});
opts.log?.(`[gbrain serve] idle upkeep: finished in ${Date.now() - startedAt} ms (${outcome.status}${outcome.code ? ` ${outcome.code}` : ''})`);
return outcome;
}
+/** One idle upkeep run; `retryParked` is true on the process's first run. */
+export type IdleUpkeep = (engine: BrainEngine, signal: AbortSignal, retryParked: boolean) => Promise<unknown>;
+
export interface IdleUpkeepTimerDeps {
intervalMs: number;
setInterval: (fn: () => void, ms: number) => unknown;
clearInterval: (handle: unknown) => void;
/** One run with the abort signal a request trips; defaults to `runIdleUpkeep` as runner `serve`. */
- upkeep?: (engine: BrainEngine, signal: AbortSignal) => Promise<unknown>;
+ upkeep?: IdleUpkeep;
log?: (line: string) => void;
}
@@ -61,9 +67,10 @@ export interface IdleUpkeepTimerDeps {
* only opens the window. A request that starts mid-run aborts it.
*/
export function armIdleUpkeep(engine: BrainEngine, deps: IdleUpkeepTimerDeps): { stop(): void } {
- const upkeep = deps.upkeep ?? ((e: BrainEngine, signal: AbortSignal) => runIdleUpkeep(e, { runner: 'serve', signal, log: deps.log }));
+ const upkeep: IdleUpkeep = deps.upkeep ?? ((e, signal, retryParked) => runIdleUpkeep(e, { runner: 'serve', signal, retryParked, log: deps.log }));
const sawForeground = foregroundWindow();
let opened = false;
+ let first = true;
let running: AbortController | null = null;
const unsubscribe = onForegroundRequest(() => running?.abort());
const handle = deps.setInterval(() => {
@@ -72,8 +79,10 @@ export function armIdleUpkeep(engine: BrainEngine, deps: IdleUpkeepTimerDeps): {
if (active || running) return;
const controller = new AbortController();
running = controller;
+ const retryParked = first;
+ first = false;
Promise.resolve()
- .then(() => upkeep(engine, controller.signal))
+ .then(() => upkeep(engine, controller.signal, retryParked))
.catch(() => { /* best-effort; never kill serve */ })
.finally(() => { if (running === controller) running = null; });
}, deps.intervalMs);
diff --git a/src/core/maintenance/run.ts b/src/core/maintenance/run.ts
index 50fe3d1b8e..a8dc2ebc71 100644
--- a/src/core/maintenance/run.ts
+++ b/src/core/maintenance/run.ts
@@ -45,6 +45,12 @@ export const UPKEEP_INTERVAL_MS = 24 * 60 * 60 * 1000;
/** A step failing with the same code this many runs in a row backs off to once a week. */
export const UPKEEP_BACKOFF_FAILURES = 3;
export const UPKEEP_BACKOFF_MS = 7 * 24 * 60 * 60 * 1000;
+/**
+ * A budgeted (serve) run parks an item that started and did not finish in this many slices of one round: the round
+ * settles without it, budgeted runs skip it, and an unbudgeted run (`gbrain maintain --unattended`, the schedule) or a
+ * serve's first idle run (`retryParked`) runs it again.
+ */
+export const UPKEEP_PARK_AFTER_SLICES = 30;
/** An autopilot whose global pass is younger than this owns upkeep. */
export const AUTOPILOT_OWNS_WITHIN_MS = 48 * 60 * 60 * 1000;
@@ -131,8 +137,14 @@ export interface UpkeepRecord {
status: 'ok' | 'partial' | 'failed';
steps: UpkeepStep[];
held: CyclePhase[];
- /** Items already done in an unfinished round; null once the round finished. */
- cursor: { round_started_at: string; done: string[] } | null;
+ /**
+ * Items already settled in an unfinished round; null once the round finished. `attempts` counts a budgeted run's
+ * unfinished slices per item, `retry` names parked items this round runs again, and `due_at` is the due time a
+ * round that only reruns parked items keeps.
+ */
+ cursor: { round_started_at: string; done: string[]; attempts?: Record<string, number>; retry?: string[]; due_at?: string } | null;
+ /** Items budgeted runs gave up on (UPKEEP_PARK_AFTER_SLICES), with their total unfinished slices. */
+ parked?: Array<{ key: string; since: string; slices: number }>;
next_due_at: string;
consecutive_failures: UpkeepFailure[];
embed?: { spent_today_usd: number; cap_usd: number };
@@ -155,6 +167,8 @@ export interface UpkeepOpts {
/** Wall-clock budget for the whole run; an unfinished round resumes on the next run. */
budgetMs?: number;
signal?: AbortSignal;
+ /** A serve's first idle run: parked items get one more set of UPKEEP_PARK_AFTER_SLICES slices. */
+ retryParked?: boolean;
/** Test seams. */
providers?: SafePhaseProviders;
capacityPaused?: (engine: BrainEngine) => Promise<{ command: string; config_key: string } | null>;
@@ -356,24 +370,54 @@ export async function runUnattendedMaintenance(engine: BrainEngine, opts: Upkeep
if (prior && refused.code) await engine.setConfig(MAINTENANCE_LAST_RUN_KEY, JSON.stringify({ ...prior, last_skip: { at: now.toISOString(), code: refused.code } }));
return refused;
}
- if (!upkeepDue(prior, now)) {
+ const budgeted = opts.budgetMs !== undefined;
+ const sources = (await engine.listAllSources()).sort((a, b) => a.id.localeCompare(b.id));
+ const keys = ['global', ...sources.map((source) => `source:${source.id}`)];
+ const parked = (prior?.parked ?? []).filter((p) => keys.includes(p.key)).map((p) => ({ ...p }));
+ const retrying = !budgeted || opts.retryParked ? parked.map((p) => p.key) : [];
+ const retryOnly = !upkeepDue(prior, now);
+ if (retryOnly && retrying.length === 0) {
return { status: 'not_due', exit: 0, message: `Upkeep already ran at ${prior!.at}; next due at ${prior!.next_due_at}.`, record: prior! };
}
const plan = unattendedPlan(opts.providers ?? await readSafePhaseProviders(engine));
- const sources = (await engine.listAllSources()).sort((a, b) => a.id.localeCompare(b.id));
- const round = prior?.cursor ?? { round_started_at: now.toISOString(), done: [] as string[] };
+ const round = prior?.cursor ?? (retryOnly
+ ? { round_started_at: now.toISOString(), done: keys.filter((key) => !retrying.includes(key)), due_at: prior!.next_due_at }
+ : { round_started_at: now.toISOString(), done: [] as string[] });
+ const attempts: Record<string, number> = { ...round.attempts };
+ const retry = new Set([...(round.retry ?? []), ...retrying]);
+ for (const key of retrying) delete attempts[key];
+ const done = round.done.filter((key) => !retrying.includes(key));
const failures = prior?.consecutive_failures ?? [];
const items: WorkItem[] = [
{ key: 'global', phases: plan.global },
...sources.map((source) => ({ key: `source:${source.id}`, sourceId: source.id, phases: plan.source })),
- ].filter((item) => !round.done.includes(item.key));
- const deadline = opts.budgetMs !== undefined ? Date.now() + opts.budgetMs : Infinity;
+ ].filter((item) => !done.includes(item.key));
+ const deadline = budgeted ? Date.now() + opts.budgetMs! : Infinity;
const steps: UpkeepStep[] = [];
- const done = [...round.done];
let lockBusy = false;
for (const item of items) {
if (Date.now() >= deadline || opts.signal?.aborted) break;
const scope = item.sourceId ?? 'global';
+ if (budgeted && !retry.has(item.key) && parked.some((p) => p.key === item.key)) {
+ steps.push({ name: 'cycle', scope, status: 'skipped', reason: 'parked' });
+ done.push(item.key);
+ continue;
+ }
+ const settle = (): void => {
+ done.push(item.key);
+ delete attempts[item.key];
+ retry.delete(item.key);
+ };
+ const unfinished = (): void => {
+ if (!budgeted) return;
+ const slices = (attempts[item.key] ?? 0) + 1;
+ if (slices < UPKEEP_PARK_AFTER_SLICES) { attempts[item.key] = slices; return; }
+ const entry = parked.find((p) => p.key === item.key);
+ if (entry) entry.slices += slices;
+ else parked.push({ key: item.key, since: now.toISOString(), slices });
+ steps.push({ name: 'cycle', scope, status: 'skipped', reason: 'parked' });
+ settle();
+ };
const phases = item.phases.filter((phase) => !backedOff(failures, phase, scope, now));
for (const phase of item.phases) if (!phases.includes(phase)) steps.push({ name: phase, scope, status: 'skipped', reason: 'backed_off' });
const source = item.sourceId ? sources.find((s) => s.id === item.sourceId) : undefined;
@@ -391,13 +435,16 @@ export async function runUnattendedMaintenance(engine: BrainEngine, opts: Upkeep
const report = await (opts.cycle ?? runCycle)(engine, cycleOpts);
if (report.status === 'skipped' && report.reason === 'cycle_already_running') { lockBusy = true; break; }
steps.push(...stepsFrom(report, scope));
- if (report.reason === 'aborted' || report.reason === 'lock_stolen') break;
+ if (report.reason === 'aborted') { unfinished(); break; }
+ if (report.reason === 'lock_stolen') break;
}
- done.push(item.key);
+ settle();
+ const at = parked.findIndex((p) => p.key === item.key);
+ if (at >= 0) parked.splice(at, 1);
} catch (error) {
- if (combined?.signal.aborted) break;
+ if (combined?.signal.aborted) { unfinished(); break; }
steps.push({ name: 'cycle', scope, status: 'fail', code: (error as { code?: string } | null)?.code ?? 'cycle_failed' });
- done.push(item.key);
+ settle();
} finally {
combined?.dispose();
}
@@ -418,7 +465,7 @@ export async function runUnattendedMaintenance(engine: BrainEngine, opts: Upkeep
if (finished && engine.kind === 'pglite' && !opts.signal?.aborted) steps.push(await checkpointStep(engine));
const failed = steps.filter((step) => step.status === 'fail');
const nextFailures: UpkeepFailure[] = failures
- .filter((f) => !steps.some((s) => s.name === f.step && s.scope === f.scope && s.status !== 'fail' && s.reason !== 'backed_off'));
+ .filter((f) => !steps.some((s) => s.name === f.step && s.scope === f.scope && s.status !== 'fail' && s.reason !== 'backed_off' && s.reason !== 'parked'));
for (const step of failed) {
const entry = nextFailures.find((f) => f.step === step.name && f.scope === step.scope);
if (entry && entry.code === step.code) { entry.count += 1; entry.last_at = now.toISOString(); }
@@ -431,9 +478,13 @@ export async function runUnattendedMaintenance(engine: BrainEngine, opts: Upkeep
status: failed.length === 0 ? 'ok' : failed.length === steps.filter((s) => s.status !== 'skipped').length ? 'failed' : 'partial',
steps,
held: plan.held,
- cursor: finished ? null : { round_started_at: round.round_started_at, done },
- next_due_at: new Date(finished ? now.getTime() + UPKEEP_INTERVAL_MS : now.getTime()).toISOString(),
+ cursor: finished ? null : {
+ round_started_at: round.round_started_at, done,
+ ...(Object.keys(attempts).length ? { attempts } : {}), ...(retry.size ? { retry: [...retry] } : {}), ...(round.due_at ? { due_at: round.due_at } : {}),
+ },
+ next_due_at: finished ? round.due_at ?? new Date(now.getTime() + UPKEEP_INTERVAL_MS).toISOString() : now.toISOString(),
consecutive_failures: nextFailures,
+ ...(parked.length ? { parked } : {}),
...(embed ? { embed } : {}),
};
await engine.setConfig(MAINTENANCE_LAST_RUN_KEY, JSON.stringify(record));
diff --git a/src/core/maintenance/status.ts b/src/core/maintenance/status.ts
index 271311de93..ab49b0dedb 100644
--- a/src/core/maintenance/status.ts
+++ b/src/core/maintenance/status.ts
@@ -14,7 +14,7 @@ import {
/** Past this age, an upkeep run (or a brain with none) reads as overdue. */
export const UPKEEP_OVERDUE_MS = 48 * 60 * 60 * 1000;
-export type UpkeepState = 'ok' | 'never' | 'off' | 'autopilot' | 'overdue' | 'failing';
+export type UpkeepState = 'ok' | 'never' | 'off' | 'autopilot' | 'overdue' | 'failing' | 'parked';
export interface UpkeepStatus {
state: UpkeepState;
@@ -25,6 +25,8 @@ export interface UpkeepStatus {
held: string[];
next_due_at: string | null;
failing: UpkeepFailure | null;
+ /** Items the serve's budgeted runs gave up on; an unbudgeted run finishes them. */
+ parked: NonNullable<UpkeepRecord['parked']>;
pending_purge_count: number | null;
paid_phases_opt_in: string[];
embed_spend_today_usd: number | null;
@@ -89,7 +91,7 @@ export async function readUpkeepStatus(engine: BrainEngine, nowMs: number = Date
const keyed = isAvailable('embedding');
const base = {
runner: record?.runner ?? null, last_run: record?.at ?? null, steps: record?.steps ?? [], held: record?.held ?? [],
- next_due_at: record?.next_due_at ?? null, failing: null as UpkeepFailure | null, pending_purge_count: pending,
+ next_due_at: record?.next_due_at ?? null, failing: null as UpkeepFailure | null, parked: record?.parked ?? [], pending_purge_count: pending,
paid_phases_opt_in: [...PAID_PHASES], embed_spend_today_usd: keyed ? spent : null, embed_daily_cap_usd: keyed ? cap : null,
};
const purgeNote = pending
@@ -115,6 +117,11 @@ export async function readUpkeepStatus(engine: BrainEngine, nowMs: number = Date
if (age > UPKEEP_OVERDUE_MS) {
return { ...base, fix, alternative_fix: scheduleFix(), state: 'overdue', message: `Free upkeep last ran ${ago(age)} ago (${record.runner}), past its 48h window.${SCHEDULE_NOTE}${purgeNote}${spendNote}` };
}
+ if (base.parked.length) {
+ const named = base.parked.map((p) => `${p.key} after ${p.slices} slices`).join(', ');
+ return { ...base, fix, alternative_fix: scheduleFix(), state: 'parked',
+ message: `Upkeep inside serve parked ${named}; the scheduled run or gbrain maintain --unattended finishes it.${purgeNote}${spendNote}` };
+ }
const ran = record.steps.filter((s) => s.status === 'ok' || s.status === 'warn').length;
const due = Date.parse(record.next_due_at) - nowMs;
return { ...base, state: 'ok',
diff --git a/test/maintenance-park-gbra438.test.ts b/test/maintenance-park-gbra438.test.ts
new file mode 100644
index 0000000000..32bc1dacc3
--- /dev/null
+++ b/test/maintenance-park-gbra438.test.ts
@@ -0,0 +1,128 @@
+/**
+ * GBRA-438 PR 2: the give-up rule for budgeted (serve) upkeep runs
+ * (src/core/maintenance/run.ts UPKEEP_PARK_AFTER_SLICES).
+ *
+ * 1. Protects: a work item that starts and does not finish in
+ * UPKEEP_PARK_AFTER_SLICES budgeted slices is parked, the items behind it
+ * run and the round finishes, so the idle serve goes quiet until the next
+ * round instead of restarting the same item forever. --status and doctor
+ * name the parked item; an unbudgeted run (gbrain maintain --unattended)
+ * finishes it and clears it; retryParked (a serve's first idle run) gives
+ * it exactly one more set of slices.
+ * 2. Fails when: a slow item is retried without bound, blocks the items
+ * behind it, keeps the serve doing upkeep work after the round, is
+ * reported as ok, or stays parked after a full run finished it.
+ * 3. Real seams: a real PGLite brain whose default source is too large for
+ * the budget (the extract_facts walk alone outlasts it); runCycle runs for
+ * real and is only counted.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { runCycle, type CycleOpts } from '../src/core/cycle.ts';
+import { UPKEEP_PARK_AFTER_SLICES, readUpkeepRecord, runUnattendedMaintenance, type SafePhaseProviders } from '../src/core/maintenance/run.ts';
+import { runIdleUpkeep } from '../src/core/maintenance/idle.ts';
+import { readUpkeepStatus } from '../src/core/maintenance/status.ts';
+import { checkUnattendedMaintenance } from '../src/commands/doctor/checks/unattended-maintenance.ts';
+import { runMaintain } from '../src/commands/maintain.ts';
+import { _resetCliExitVerdictForTests } from '../src/core/cli-force-exit.ts';
+
+const KEYLESS: SafePhaseProviders = { embedding: false, decideConflict: false, fenceRepairModel: false, contentRepairModel: false };
+const BUDGET_MS = 250;
+const PAGES = 2000;
+
+describe('budgeted upkeep parks an item that cannot finish inside its slices (PGLite, repo-less)', () => {
+ let engine: BrainEngine;
+ const calls: string[] = [];
+ const counted = (e: BrainEngine | null, o: CycleOpts) => { calls.push(o.sourceId ?? 'global'); return runCycle(e, o); };
+ const slice = (extra: { retryParked?: boolean } = {}) =>
+ runIdleUpkeep(engine, { runner: 'serve', signal: new AbortController().signal, providers: KEYLESS, budgetMs: BUDGET_MS, cycle: counted, ...extra });
+
+ beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({ engine: 'pglite' } as never);
+ await engine.initSchema();
+ for (let i = 0; i < PAGES; i++) {
+ await engine.putPage(`bulk/p${i}`, { type: 'concept', title: `P${i}`, compiled_truth: `P${i} relies on [[bulk/p${(i + 1) % PAGES}]].` });
+ }
+ await engine.executeRaw("INSERT INTO sources (id, name) VALUES ('zz-small', 'zz-small')");
+ for (let i = 0; i < 3; i++) await engine.putPage(`small/s${i}`, { type: 'concept', title: `S${i}`, compiled_truth: `S${i} is a plain page.` }, { sourceId: 'zz-small' });
+ }, 120_000);
+ afterAll(async () => {
+ await engine.disconnect();
+ _resetCliExitVerdictForTests();
+ process.exitCode = 0;
+ });
+
+ test('the slow source parks, the source behind it runs, the round finishes and the serve goes quiet; status, doctor, retry and a full run follow', async () => {
+ // (i) + (ii): slices until the round finishes.
+ let finished = null as Awaited<ReturnType<typeof slice>>;
+ const sliceSteps: NonNullable<NonNullable<typeof finished>['record']>['steps'] = [];
+ for (let n = 0; n < UPKEEP_PARK_AFTER_SLICES + 10 && !finished; n++) {
+ const outcome = await slice();
+ sliceSteps.push(...outcome?.record?.steps ?? []);
+ if (outcome?.record && outcome.record.cursor === null) finished = outcome;
+ }
+ expect(finished).not.toBeNull();
+ const record = (await readUpkeepRecord(engine))!;
+ expect(record.cursor).toBeNull();
+ expect(record.parked).toEqual([{ key: 'source:default', since: expect.any(String), slices: UPKEEP_PARK_AFTER_SLICES }]);
+ expect(calls.filter((c) => c === 'default')).toHaveLength(UPKEEP_PARK_AFTER_SLICES);
+ expect(calls).toContain('zz-small');
+ expect(sliceSteps.filter((s) => s.reason === 'parked')).toEqual([{ name: 'cycle', scope: 'default', status: 'skipped', reason: 'parked' }]);
+ expect(sliceSteps.some((s) => s.scope === 'zz-small' && s.name === 'extract_facts' && s.status === 'ok')).toBe(true);
+ expect(Date.parse(record.next_due_at)).toBeGreaterThan(Date.now());
+
+ // (iii): the next idle runs do nothing: no cycle call, no record write.
+ const before = JSON.stringify(await readUpkeepRecord(engine));
+ calls.length = 0;
+ for (let n = 0; n < 3; n++) expect(await slice()).toBeNull();
+ expect(await runUnattendedMaintenance(engine, { runner: 'serve', budgetMs: BUDGET_MS, providers: KEYLESS, cycle: counted })).toMatchObject({ status: 'not_due' });
+ expect(calls).toEqual([]);
+ expect(JSON.stringify(await readUpkeepRecord(engine))).toBe(before);
+
+ // (iv): --status (text and JSON) and doctor name the parked item and warn.
+ const status = await readUpkeepStatus(engine);
+ expect(status.state).toBe('parked');
+ expect(status.parked).toEqual(record.parked!);
+ expect(status.message).toContain(`parked source:default after ${UPKEEP_PARK_AFTER_SLICES} slices`);
+ expect(status.message).toContain('gbrain maintain --unattended');
+ const check = await checkUnattendedMaintenance(engine);
+ expect(check).toMatchObject({ status: 'warn', fix: { argv: ['gbrain', 'maintain', '--unattended'] },
+ details: { state: 'parked', parked: record.parked, alternative_fix: { argv: ['gbrain', 'maintain', '--schedule'] } } });
+ const lines: string[] = [];
+ const log = console.log;
+ console.log = (...a: unknown[]) => { lines.push(a.map(String).join(' ')); };
+ try {
+ await runMaintain(engine, ['--status']);
+ await runMaintain(engine, ['--status', '--json']);
+ } finally {
+ console.log = log;
+ }
+ expect(lines.join('\n')).toContain(`Parked: source:default since ${record.parked![0]!.since} (${UPKEEP_PARK_AFTER_SLICES} slices)`);
+ expect(JSON.parse(lines[lines.length - 1]!)).toMatchObject({ state: 'parked', parked: record.parked });
+
+ // (vi): a serve's first idle run (retryParked) gives it exactly one more set of slices, then it parks again.
+ const dueAt = record.next_due_at;
+ calls.length = 0;
+ expect(await slice({ retryParked: true })).not.toBeNull();
+ for (let n = 0; n < UPKEEP_PARK_AFTER_SLICES + 10; n++) if (!await slice()) break;
+ expect(calls).toEqual(Array(UPKEEP_PARK_AFTER_SLICES).fill('default'));
+ const retried = (await readUpkeepRecord(engine))!;
+ expect(retried.cursor).toBeNull();
+ expect(retried.parked).toEqual([{ key: 'source:default', since: record.parked![0]!.since, slices: 2 * UPKEEP_PARK_AFTER_SLICES }]);
+ expect(retried.next_due_at).toBe(dueAt);
+ expect(await slice()).toBeNull();
+
+ // (v): an unbudgeted run (gbrain maintain --unattended) finishes the parked item and clears it.
+ calls.length = 0;
+ const full = await runUnattendedMaintenance(engine, { runner: 'cli', providers: KEYLESS, cycle: counted });
+ expect(full).toMatchObject({ status: 'ran', exit: 0 });
+ expect(calls).toEqual(['default']);
+ const cleared = (await readUpkeepRecord(engine))!;
+ expect(cleared.cursor).toBeNull();
+ expect(cleared.parked ?? []).toEqual([]);
+ expect(cleared.steps.some((s) => s.scope === 'default' && s.name === 'extract_facts' && s.status === 'ok')).toBe(true);
+ expect((await readUpkeepStatus(engine)).state).toBe('ok');
+ }, 180_000);
+});
diff --git a/test/maintenance-unattended.test.ts b/test/maintenance-unattended.test.ts
index e5914bc7b6..d11a243959 100644
--- a/test/maintenance-unattended.test.ts
+++ b/test/maintenance-unattended.test.ts
@@ -344,8 +344,8 @@ for (const kind of testBackends()) {
await record(Date.now() - HOUR);
const out = await maintainJson(['--status', '--json']);
expect(Object.keys(out).sort()).toEqual(['embed_daily_cap_usd', 'embed_spend_today_usd', 'failing', 'held', 'last_run', 'message', 'next_due_at',
- 'paid_phases_opt_in', 'pending_purge_count', 'runner', 'state', 'steps']);
- expect(out).toMatchObject({ state: 'ok', runner: 'cli', embed_spend_today_usd: null, embed_daily_cap_usd: null, pending_purge_count: 0 });
+ 'paid_phases_opt_in', 'parked', 'pending_purge_count', 'runner', 'state', 'steps']);
+ expect(out).toMatchObject({ state: 'ok', runner: 'cli', embed_spend_today_usd: null, embed_daily_cap_usd: null, pending_purge_count: 0, parked: [] });
expect(out.paid_phases_opt_in).toContain('embed');
});
diff --git a/test/serve-idle-upkeep-gbra438.test.ts b/test/serve-idle-upkeep-gbra438.test.ts
index ad9b6fa61c..00a2916a6f 100644
--- a/test/serve-idle-upkeep-gbra438.test.ts
+++ b/test/serve-idle-upkeep-gbra438.test.ts
@@ -54,6 +54,7 @@ function harness() {
const cleared: unknown[] = [];
const events: string[] = [];
const runs: AbortSignal[] = [];
+ const retries: boolean[] = [];
let sweepGate = gate();
let upkeepGate = gate();
const opts: ServeOptions = {
@@ -62,15 +63,16 @@ function harness() {
setInterval: (fn) => { timers.push(fn); return timers.length; }, clearInterval: (h) => { cleared.push(h); },
mcpStdio: false, bootTimeoutMs: 0, sweepEnabled: true, factsDrain: false,
sweep: async () => { events.push('sweep:start'); await sweepGate.promise; events.push('sweep:end'); },
- upkeep: async (_engine, signal) => {
+ upkeep: async (_engine, signal, retryParked) => {
events.push('upkeep:start');
runs.push(signal);
+ retries.push(retryParked);
await Promise.race([upkeepGate.promise, new Promise<void>((r) => signal.addEventListener('abort', () => r()))]);
events.push(signal.aborted ? 'upkeep:aborted' : 'upkeep:end');
},
};
return {
- stdin, signals, timers, cleared, events, runs, opts,
+ stdin, signals, timers, cleared, events, runs, retries, opts,
tick: () => timers[0]!(),
openSweep: () => { sweepGate.open(); sweepGate = gate(); },
openUpkeep: () => { upkeepGate.open(); upkeepGate = gate(); },
@@ -115,6 +117,21 @@ describe('stdio serve: idle upkeep on the idle tick', () => {
expect(h.events.filter((e) => e === 'upkeep:start')).toHaveLength(1);
});
+ test("the process's first idle run retries parked items (retryParked); later runs do not", async () => {
+ const h = harness();
+ await runServe(engineOf('pglite'), [], h.opts);
+ h.tick(); h.tick();
+ for (const n of [1, 2]) {
+ await until(() => h.events.filter((e) => e === 'sweep:start').length === n);
+ h.openSweep();
+ await until(() => h.runs.length === n);
+ h.openUpkeep();
+ await until(() => h.events.filter((e) => e === 'upkeep:end').length === n);
+ if (n === 1) h.tick();
+ }
+ expect(h.retries).toEqual([true, false]);
+ });
+
test('stdin data aborts the run in flight', async () => {
const h = harness();
await runServe(engineOf('pglite'), [], h.opts);
@@ -203,6 +220,7 @@ describe('HTTP serve: the same idle tick, keyed on request activity', () => {
expect(h.runs).toHaveLength(1);
h.tick();
await until(() => h.runs.length === 2);
+ expect(h.retries).toEqual([true, false]);
h.openUpkeep();
h.stopServer();
await served;