Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(doctor): runtime_build_skew skips heartbeat rows of processes proven dead (GST-2688)
garrytan1a1061ba9050
diff --git a/changelog.d/capy-gst-2688-old-updater-leaf.md b/changelog.d/capy-gst-2688-old-updater-leaf.mdnew file mode 100644index 0000000000..8d42fa8637--- /dev/null+++ b/changelog.d/capy-gst-2688-old-updater-leaf.md@@ -0,0 +1,8 @@+---+important: regression+---+**Upgrading from an older release (about v0.60.150 to v0.60.248) no longer ends in exit 1, and doctor no longer reports a stopped serve as a second running build.**++### Fixed+- `gbrain upgrade` run from an older release finishes with exit 0 again. Those releases load one more file from the new install into the old process after the swap, and that file pulled in code needing functions the old process didn't have, so the upgrade printed an `internal_error` and exited 1 even though the new binary and its migrations were in place. That file now loads nothing but the line it prints (GST-2688).+- `gbrain doctor`'s `runtime_build_skew` check no longer counts a process that has already exited as a running build. An older serve stopped with SIGTERM can leave its heartbeat row looking fresh for up to 30 seconds; a row whose process the check can prove is gone (same machine boot, pid gone or reused) is now skipped, while a row it can't prove dead still counts (GST-2688).diff --git a/docs/architecture/key-files/commands-2-continued.md b/docs/architecture/key-files/commands-2-continued.mdindex 3a7d492c13..920b01f3d3 100644--- a/docs/architecture/key-files/commands-2-continued.md+++ b/docs/architecture/key-files/commands-2-continued.md@@ -12,7 +12,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/doctor/checks/knowledge-health.ts` — `facts_extraction_health` reads the last 24 h of `facts:absorb` rows in `ingest_log` (written by `src/core/facts/absorb-log.ts`) per source and reason (`split_part(summary, ':', 1)`) and warns when a failure reason reaches `facts.absorb_warn_threshold` (default 10) in a source. Two reasons are not failures and never count: `write_pending_accepted` (the absorb log read the facts write's receipt back from the journal and it is still queued, running or recovering after the job's publish wait) and `write_deferred_unadmitted` (`writer_lock_unavailable` before admission, no receipt; the job retries). Deferrals warn on their own when, in a source, more than 10 or one over an hour old have no later completed `facts-absorb` job for the same page; that warning's fix is the read-only `gbrain sources writer status --source <id> --json`. The failure count is history: the message gives the incidents in the rolling 24 h window and when the newest ages out, and `details.facts_absorb` (`incidents_24h`, `ages_out_at`, `jobs_still_dead`, `dead_jobs`) adds the current state, the `facts-absorb` jobs dead or failed in the last 24 h with no later completed job for the same page (source and slug); they never change the status, and when no other fix applies the newest one gets the paid `gbrain jobs retry <id>` fix (`retry_job` over MCP, asks first). Tests: `test/doctor-facts-absorb-contention-6778.test.ts`, `test/facts-absorb-publish-wait-6778.test.ts`, `test/doctor-facts-absorb-history-6761.test.ts`.- `src/commands/doctor/checks/persistence-consumers.ts` — #6317 (B3, T1c): doctor `two_consumers_on_host` (warn only when two live rows of resident kinds on this host have both been alive longer than 30 s; every other row, a `put` or a row younger than 30 s, is listed in `details`) and `consumers_without_heartbeat` (a `claim_phase.owner` pid on a running claim of this host with no heartbeat row: an older process that never defers; named with its kind and version from the stamp). Both carry a structured `agentFix` (`gbrain sources writer status --json`) and `doctorVerify`; with `persistence.single_consumer` off the overlap is reported as the configured behavior. `serve_build_drift` (#6696 C6, its own `serveBuildDriftEntry`, one `listHostConsumers` read, ok on PGLite with no read) compares the live resident rows (not this process) with this binary's `VERSION` through `compareWriterVersions` (an unparsable version counts as older): an older row warns "serve is running an older build than this checkout; restart serve and workers" with `details { count, this_version, owner, older, newer }` and a `fix` (actor `user`, `doctorVerify`) whose argv is the installed serve service's restart (`systemctl --user restart gbrain-serve.service` when `systemdUnitPath` exists on Linux, `launchctl kickstart -k gui/<uid>/com.gbrain.serve` when `launchdPlistPath` exists on macOS; `serveBuildDriftCheck`'s `ServiceHost` argument injects platform, home and uid) or else `kill <pid>`; a newer row is reported in an ok message; no rows names `consumers_without_heartbeat` for a pre-heartbeat serve. Registered in `DOCTOR_CHECK_REGISTRY` (the consumer entry, then `serveBuildDriftEntry`) before `searchModeEntry` and categorised `ops` in `src/core/doctor-categories.ts`. Tests: `test/doctor-serve-build-drift.test.ts`.-- `src/commands/doctor/checks/runtime-build-skew.ts` — #7245: doctor `runtime_build_skew` (standard tier, both engines, read-only). Collects serve and the other live resident consumers from their heartbeat rows (Postgres), worker parents from the worker registry (`version`, `install_root`), the supervisor from its PID file (`readSupervisorBuild`), and job-child builds from `job_build_skew:` history on the newest 500 jobs of the last 24 h while their worker pid lives. Fails on 2+ builds, warns when only an install differs; fix: ask, then `gbrain upgrade --restart-only`; the message names the manual restart: serve (`restartArgv`, exported from persistence-consumers.ts), else the supervisor (`gbrain jobs supervisor stop`), else a worker pid. Output: pids and versions only. Pinned by `test/doctor-runtime-build-skew.test.ts`.+- `src/commands/doctor/checks/runtime-build-skew.ts` — #7245: doctor `runtime_build_skew` (standard tier, both engines, read-only). Collects serve and the other live resident consumers from their heartbeat rows (Postgres), skipping a row this process can prove dead (`consumerRowProvablyDead`: an old serve stopped on SIGTERM can leave its row renewed for 30 s, GST-2688), worker parents from the worker registry (`version`, `install_root`), the supervisor from its PID file (`readSupervisorBuild`), and job-child builds from `job_build_skew:` history on the newest 500 jobs of the last 24 h while their worker pid lives. Fails on 2+ builds, warns when only an install differs; fix: ask, then `gbrain upgrade --restart-only`; the message names the manual restart: serve (`restartArgv`, exported from persistence-consumers.ts), else the supervisor (`gbrain jobs supervisor stop`), else a worker pid. Output: pids and versions only. Pinned by `test/doctor-runtime-build-skew.test.ts`.- `src/commands/dream-retriage.ts` — `gbrain dream retriage`: re-scores synthesize's list (`withConversationPages`, #7307) via the shared `runTriagePass` (with `maxMs: 0` — operator sweeps run to completion; `--limit` slices the discovered list caller-side) and reconciles the queued private-queue backlog: synth-v2 rows verdict-gated, plus any row stranded in a provably-dead `dream-inline-*` queue regardless of key family (patterns children, legacy grammars) — doctor's orphaned_private_queue check points here, so the repair selects everything the check can flag. Liveness uses ownership correlation (mirrors the doctor check): a live `gbrain_cycle_locks` row suppresses conversion only for queues born at/after its `acquired_at`; an older crashed cycle's queue stays repairable while a new cycle runs (unknown birth/acquisition stays fail-safe possibly-live). Spend-gated: upfront estimate via `canonicalLookup` on the resolved triage model, confirmation above `SPEND_CONFIRM_USD` ($5, in `dream-retriage-constants.ts`; unpriced models gate on `UNPRICED_CONFIRM_FILES`=500) through `requireConsent` (effect `paid`; `--yes`, `--max-usd`, tokenmax or a per-run preapproval authorize it; otherwise a TTY prompt or exit 3 with the consent payload, `--json` never implying it); the budget (`--max-usd`, else the derived consent cap) soft-stops via the pass's `shouldStop` seam (estimate-based) and bounds the `--audit-rejects` half. `--reconcile-queue` (opt-in — cancels queued work): selects waiting/delayed/paused `dream:synth-v2:%` jobs across ALL queues, parses keys with `parseSynthV2Key`, then per row, reading THE shared gate (`passesTriageGate` from `triage-rescue.ts` — threshold pass OR the verified-segment rescue; a hand-rolled `score >= threshold` here would cancel exactly the jobs the rescue admitted): matched below the gate → cancel; matched above the gate but `waiting` in a stale `dream-inline-*` queue → cancel as `converted_for_resubmit` (the backlog conversion — cancelled rows release their idempotency slot so the next cycle re-adds into a live drain); matched above the gate in a live queue → keep; matched-but-unscored → keep (never cancel on missing data); unmatched → keep unless `--cancel-unmatched`; key-source vs `data.source_id` disagreement → skip + `source_mismatch`; status re-checked immediately before each cancel (rows turned `active` are skipped; residual race matches cancelJob's best-effort contract). Legacy `dream:synth:` keys are excluded at the SQL LIKE filter — never candidates. `--source` scopes cancels (`other_source` counted); `--threshold`/`--since`/`--force`/`--dry-run` (zero judge calls, zero cancels — cached scores only, uncached files report `needs_triage`). `--audit-rejects <n>` re-judges N stride-sampled files that the gate actually REJECTED (below threshold AND not rescued — auditing a rescued file as a rejection would misreport the disagreement rate) with the SYNTHESIS model and reports that rate — the operator calibration loop. Reports carry `rescued`/`verified_segments` so a rescued file reads as accepted everywhere the command renders. Exit 0 success (even when nothing cancelled), 1 no engine/corpus, 2 usage or declined spend gate. Pinned by `test/dream-retriage.test.ts`. With System One S7 on, cached decide verdicts (cache identity from `resolveTriageDecide`) count as valid, the slot threshold joins every gate read, `runTriagePass` gets `decide`, and the spend estimate prices Jev windows (`estimateTriageDecideUsd`).diff --git a/src/commands/doctor/checks/runtime-build-skew.ts b/src/commands/doctor/checks/runtime-build-skew.tsindex 3263e99be9..e6ee1e5ad4 100644--- a/src/commands/doctor/checks/runtime-build-skew.ts+++ b/src/commands/doctor/checks/runtime-build-skew.ts@@ -9,7 +9,9 @@* and with the build each one's install holds on disk now:** - serve and the other resident consumers: their `persistence_consumers`- * heartbeat `version` (Postgres; PGLite keeps none);+ * heartbeat `version` (Postgres; PGLite keeps none), minus rows this process+ * can prove dead (GST-2688: an old serve stopped on SIGTERM can leave its row+ * renewed for 30 s);* - the jobs supervisor: lines 4-5 of its PID file (`readSupervisorBuild`);* - worker parents (`gbrain jobs work`, autopilot-spawned ones included): the* `version` and `install_root` of their worker-registry entries;@@ -28,7 +30,7 @@import { homedir } from 'node:os';import type { Check } from '../../doctor.ts';import type { BrainEngine } from '../../../core/engine.ts';-import { listHostConsumers, RESIDENT_CONSUMER_KINDS } from '../../../core/persistence/consumer-heartbeat.ts';+import { consumerRowProvablyDead, listHostConsumers, RESIDENT_CONSUMER_KINDS } from '../../../core/persistence/consumer-heartbeat.ts';import { localHostIdentity } from '../../../core/persistence/identity.ts';import { compareWriterVersions } from '../../../core/persistence/writer-versions.ts';import { installedVersion } from '../../../core/runtime-build.ts';@@ -94,7 +96,7 @@ export async function runtimeBuildSkewCheck(engine: BrainEngine, hostId: string,try {const byPid = new Map<number, RunningBuild>();for (const row of await listHostConsumers(engine, hostId)) {- if (row.liveness !== 'live' || row.self || !RESIDENT_CONSUMER_KINDS.includes(row.kind)) continue;+ if (row.liveness !== 'live' || row.self || !RESIDENT_CONSUMER_KINDS.includes(row.kind) || consumerRowProvablyDead(row)) continue;byPid.set(row.pid, { role: row.kind, pid: row.pid, version: row.version || null, installed_version: null });}for (const entry of sources.workers()) {diff --git a/test/doctor-runtime-build-skew.test.ts b/test/doctor-runtime-build-skew.test.tsindex 9ee31f34ef..3887fc1b29 100644--- a/test/doctor-runtime-build-skew.test.ts+++ b/test/doctor-runtime-build-skew.test.ts@@ -17,6 +17,7 @@ import { PGLiteEngine } from '../src/core/pglite-engine.ts';import { MinionQueue } from '../src/core/minions/queue.ts';import type { BrainEngine } from '../src/core/engine.ts';import { VERSION } from '../src/version.ts';+import { consumerIdentity } from '../src/core/persistence/consumer-heartbeat.ts';const OLD = '0.60.1.0';const OLDER = '0.60.0.9';@@ -26,10 +27,10 @@ const home = mkdtempSync(join(tmpdir(), 'gbrain-runtime-skew-'));const service = { platform: 'linux', home, uid: 501 };afterAll(() => rmSync(home, { recursive: true, force: true }));-const heartbeat = (over: { kind?: string; pid?: number; version?: string } = {}) => ({- host_id: HOST, pid: over.pid ?? 4242, nonce: `n-${over.pid ?? 4242}`, pid_ns: null, kind: over.kind ?? 'serve', mode: 'full',+const heartbeat = (over: { kind?: string; pid?: number; version?: string; pid_ns?: string | null; boot_id?: string | null; process_start?: string | null } = {}) => ({+ host_id: HOST, pid: over.pid ?? 4242, nonce: `n-${over.pid ?? 4242}`, pid_ns: over.pid_ns ?? null, kind: over.kind ?? 'serve', mode: 'full',started_at: '2026-10-14 05:00:00+00', renewed_at: '2026-10-14 12:00:00+00', restart_required: false, root_barrier_age_ms: null, pool: null,- host_json_path: `${INSTALL}/host.json`, persistence_home: INSTALL, minted_under: null, version: over.version ?? VERSION, boot_id: null, process_start: null,+ host_json_path: `${INSTALL}/host.json`, persistence_home: INSTALL, minted_under: null, version: over.version ?? VERSION, boot_id: over.boot_id ?? null, process_start: over.process_start ?? null,renewed_age_ms: '2000', age_ms: '25200000' });const childLine = (workerPid: number, workerVersion: string) => jobChildBuildVerdict(JSON.stringify({ version: workerVersion, protocol_version: CHILD_READINESS_PROTOCOL_VERSION,features: ['local-configuration-outcome-v1', 'postgres-cancellation-v1'] }), workerPid)!.note;@@ -101,6 +102,22 @@ describe('doctor runtime_build_skew', () => {expect(check.message).toContain('gbrain jobs supervisor start --detach');});+ // GST-2688: a serve stopped on SIGTERM whose old build never deletes its heartbeat row still reads as renewed for+ // up to 30 s. A row this process can prove dead (same boot and pid namespace, pid gone) is not a running build;+ // a row it cannot prove dead still counts, so the check never goes quiet on a live second build.+ const self = consumerIdentity();+ test.skipIf(!self.boot_id || !self.pid_ns || !self.start)('a freshly renewed row whose process has exited is not a running build', async () => {+ const exited = Bun.spawnSync(['true']).pid;+ const markers = { pid_ns: self.pid_ns, boot_id: self.boot_id, process_start: '1' };+ const check = await runtimeBuildSkewCheck(stub([heartbeat({ pid: exited, version: OLD, ...markers }), heartbeat()]).engine, HOST, sources(), service);+ expect(check.status).toBe('ok');+ expect(check.details).toMatchObject({ count: 0, builds: [VERSION], processes: [{ role: 'serve', pid: 4242, version: VERSION }] });+ const alive = await runtimeBuildSkewCheck(stub([heartbeat({ pid: process.pid, version: OLD, pid_ns: self.pid_ns, boot_id: self.boot_id, process_start: self.start }), heartbeat()]).engine,+ HOST, sources(), service);+ expect(alive.status).toBe('fail');+ expect(alive.message).toContain(`serve pid ${process.pid} v${OLD}`);+ });+test('job children of a worker that is gone are history, not a warning; an unrecorded build is named', async () => {const check = await runtimeBuildSkewCheck(stub([heartbeat()], [{ id: 9, line: childLine(9999, OLD) }]).engine, HOST, sources({supervisor: { pid: 6161, version: null, install_root: null } }), service);