Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(autopilot): restart the serve only on evidence it is older; keep doctor's held warning past restart_serve rows (#7475 review)
garrytanfc7bbc4407eb
diff --git a/docs/architecture/key-files/commands-6.md b/docs/architecture/key-files/commands-6.mdindex a1475dc70b..1ddd7a002c 100644--- a/docs/architecture/key-files/commands-6.md+++ b/docs/architecture/key-files/commands-6.md@@ -13,7 +13,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/commands/transcripts.ts` — the transcripts command family, all local-only (`ctx.remote=false` by construction). `recent`: raw `.txt` corpus reads via `listRecentTranscripts` (same library as the gated `get_recent_transcripts` op). `ingest <path-or-glob>`: the cross-harness session importer — resolves ONE source id (6-tier chain), threads activePack once, streams progress (phase `transcripts.ingest`), and calls `runTranscriptsIngest` with the CLI's `makeContext` context, so on a writer-claimed brain each rendered part is submitted as a `put_page` write request naming the page's current revision (a part whose publication would change nothing is screened request-free first and takes no admission; unmanaged brains and dry runs keep the direct importer; `recover codex --apply` passes the same context; pinned by `test/transcripts-managed-ingest.test.ts`); embedding is OFF by default (embed backfill is the catch-up lane; the embed flag opts in); the max-bytes flag (validated size string, e.g. `4gb`) overrides the per-format file/store byte caps for oversized stores while omission preserves each adapter's native default; the since-last watermark is an op-checkpoint (op `transcripts-ingest`, fingerprint = source + pathspec + format + adapter version + any explicit byte cap via `ingestCheckpointFingerprintInput` — a checkpoint written under one cap is never silently reused under another, so a capped run's skipped tail can't read as scanned; the pathspec binds the user-stated paths resolved + sorted, and the no-arg/all lane substitutes hostname + sorted harness roots so DB-backed checkpoints shared across a brain's machines never let one machine inherit another's watermark and skip sessions it never scanned) advanced ONLY after a clean, untruncated, non-dry scan (a Codex sub-agent rollout, `session_meta` payload.source.subagent, has no user turn by design: the adapter reports `subagentRollout`, the scan counts it in `subagentRollouts` and prints `N sub-agent rollouts (no user turns by design)`, and it keeps the scan clean; an ordinary assistant-only rollout is still `userTurnsMissing` drift); no-arg = confined discovery table, the all flag imports the discovered set. `status`: found-vs-imported gap table (disk scan vs ONE conversation-pages frontmatter query — `executeRaw`, both engines — in `src/core/transcripts/discover.ts`) — the correctness surface that catches late-arriving sessions no watermark can. The facts flag hands EVERY touched slug (including hash-skipped) to `src/core/transcripts/ingest-facts.ts`; a page that fails extraction counts in `facts.pagesFailed`, stays unfinished, and makes the command exit 1. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + SELF_HELP_WITHOUT_ENGINE (engine-free help). `expandTilde(raw)` (bare `~`, leading `~/` or `~\\` → home dir; `~user` forms and a mid-string `~` stay literal) is the one tilde expansion `expandPaths` applies to every path spec. `recover codex` (`src/commands/transcripts-recover.ts`, logic `src/core/transcripts/recover.ts`; D15 for #5163): finds codex conversation pages in one source with no `**User**` anchor, re-parses the retained rollouts (discovery root + archived store, or given paths) and classifies each session recoverable / still userless / unrecoverable (rollout gone); preview by default, `--apply` re-imports only the recoverable rollouts through `runTranscriptsIngest` (content-hash dedup, no facts, no watermark write). Pinned by `test/transcripts-recover-codex.test.ts`. `audit-secrets [--source-id S] [--json]` (`src/commands/transcripts-audit.ts`, logic `src/core/transcripts/secret-audit.ts`): the read-only installed-base audit that streams imported conversation pages (`type = 'conversation'` with `transcript_import` frontmatter) in id-keyset batches of 25, scans title and body with the transcript-lane detectors, lists slug, source, hit counts per pattern and up to 20 line numbers (never values or previews), and caches the summary without slugs in config row `transcripts.secret_audit` for the doctor check `transcript_secret_exposure` (`src/commands/doctor/checks/transcript-secrets.ts`, brain category), which counts imported conversation pages and reads that summary (never scans): no audit, an older `TRANSCRIPT_SECRET_AUDIT_VERSION` or a source-scoped audit warns with the audit as fix; hits warn with the audit's age; a clean full audit is ok. Pinned by `test/transcripts-secret-audit.test.ts`.- `src/commands/upgrade.ts` — self-update CLI. It sets `GBRAIN_UPGRADE_OWNS_MIGRATIONS=1` for the install step, so package postinstall (`scripts/postinstall.ts`) skips `apply-migrations`; post-upgrade owns them. An install step that fails or times out after `gbrain --version` already reports a newer version counts as installed. The run ends with a `Binary:` line (installed version, or upgrade failed) and a `Migrations:` line: complete; running, with the lock holder's host and pid when post-upgrade exits 75; or failed, with the recovery command (`--force-retry <v>` for wedged versions). `runPostUpgrade()` enumerates migrations from the TS registry (`src/commands/migrations/index.ts`) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. After schema apply it prints the one-shot auto_chronicle default-on `[AGENT]` notice (`core/chronicle/upgrade-notice.ts`). Before the swap of a bun-link, package or ClawHub install it refuses (exit 78, nothing changed) a target whose Bun floor is above the host's Bun or unreadable (`--no-bun-floor-check` overrides, manual only; `src/core/bun-floor.ts`); after the swap it runs `gbrain --help` as a smoke that passes the startup Bun check and, on failure, exits 1 with per-method recovery (#5855). Everything the post-swap tail needs is imported statically, so the old process never loads new-release modules into its cached graph (#6604; the supervisor step constant lives in the import-free leaf `src/commands/movement-step.ts`; guard: `test/upgrade-post-swap-imports.test.ts`). A tag-pinned reinstall hint (`bunReinstallCommand`, `reinstallHintLines`) probes `git ls-remote --exit-code --tags … refs/tags/v<x>` before `bun remove -g gbrain`, prints a restore line, and on Windows prints one step per line; `planPinnedReinstall` probes the tag itself (published / not published / unknown) and falls back to the newest published `v<x>` tag (`latestPublishedVersion`) when the requested one is missing (#6662, D24; `test/upgrade-release-tag-hint.test.ts`). A verified version equal to the running one after the install step is no swap (#6828): it prints `No newer version was installed`, skips `last_upgrade`, `JUST_UPGRADED`, the cache reset and the smoke, and on a tag pin with no newer published tag the hint moves the install off the pin with `to_version: latest`, never a cached unpublished version (#6718). With `GBRAIN_SWAP_RECEIPT` set it writes a `SwapReceipt` (`installed` / `not_installed` / `installed_broken`) for the autopilot channel (`test/autopilot-unpublished-target-6828.test.ts`). Tests: `upgrade-migration-outcome.test.ts`, `upgrade-no-autopilot.serial.test.ts`, `upgrade-bun-link-arc.serial.test.ts`. `post-upgrade --json` (D2, `json: 'document'`): human lines go to stderr and stdout is one report `{status, apply_migrations:{exit_code}, schema, warnings[]}`; it calls `applyMigrations()` and still ends on any status apply-migrations returns (a failure is one envelope leading with the report). GBRA-175: `--channel latest|stable` (default `self_upgrade.channel`): `resolveStableRelease` maps the commit `refs/tags/stable` peels to onto its `v<version>` tag through `git ls-remote`, and `installStableRelease` installs exactly that release (bun: remove then add `#v<version>`, restoring the previous release on failure; bun-link: fast-forward only; binary: `fetchReleaseByTag`), never downgrading. Before the swap `preflightHostRestart` refuses (`restart_refused`, exit 3) when a running process could not be restarted; after post-upgrade the new build runs `gbrain upgrade --restart-only` (`--no-restart` skips it). Tests: `upgrade-stable-channel.test.ts`. A global install pinned to a tag or commit (`bunGlobalExactTagPin(root, true)`) is replaced with the newest published release the same way instead of `bun update`, which keeps the pin. `parseChannelArg` and `resolveStableRelease` live in `src/core/stable-channel.ts` (re-exported).-- `src/commands/upgrade-restart.ts` — GBRA-175: the host side of the upgrade restart. Real process sources (consumer heartbeat rows on Postgres, the supervisor PID file, the worker registry, `ps -U <uid> -ww -o pid=,ppid=,etime=,command=` with `/proc/<pid>/environ` GBRAIN_HOME matching on Linux; the install version above each entry point), the installed serve and autopilot services with the pid each manager runs (`hostServices`: `systemctl --user show -p MainPID`, `launchctl list`), `terminateIfUnchanged` (SIGTERM an MCP pid only while `ps` still shows the same server and start time), `restartRefusal` (ask_user + `--stop-mcp` when only MCP servers block, else the first manual step for the user), `preflightHostRestart` (pre-swap, every process stale) and `runHostRestart` (`gbrain upgrade --restart-only [--stop-mcp] [--dry-run] [--json]`: plan, run, verify for up to 180 s with the warm-socket probe `probeCliSocket`). #7475: `restartServeAfterSelfUpgrade`, called each autopilot-daemon tick, restarts the serve the serve service runs once it is on an older build than the relaunched autopilot. It runs once per autopilot process, only in auto mode, in quiet hours and when the brain is idle (`computeAutopilotIdle`), and audits as `restart_serve`. `planServeRestart` reports a serve the service does not run, or one on another install, as `manual` and never restarts it. Tests: `upgrade-restart.test.ts`, `autopilot-serve-restart-7475.test.ts`.+- `src/commands/upgrade-restart.ts` — GBRA-175: the host side of the upgrade restart. Real process sources (consumer heartbeat rows on Postgres, the supervisor PID file, the worker registry, `ps -U <uid> -ww -o pid=,ppid=,etime=,command=` with `/proc/<pid>/environ` GBRAIN_HOME matching on Linux; the install version above each entry point), the installed serve and autopilot services with the pid each manager runs (`hostServices`: `systemctl --user show -p MainPID`, `launchctl list`), `terminateIfUnchanged` (SIGTERM an MCP pid only while `ps` still shows the same server and start time), `restartRefusal` (ask_user + `--stop-mcp` when only MCP servers block, else the first manual step for the user), `preflightHostRestart` (pre-swap, every process stale) and `runHostRestart` (`gbrain upgrade --restart-only [--stop-mcp] [--dry-run] [--json]`: plan, run, verify for up to 180 s with the warm-socket probe `probeCliSocket`). #7475: `restartServeAfterSelfUpgrade`, called each autopilot-daemon tick, restarts the serve the serve service runs once it is on an older build than the relaunched autopilot. It runs once per autopilot process, only in auto mode, in quiet hours and when the brain is idle (`computeAutopilotIdle`), and audits as `restart_serve`. `planServeRestart` reports a serve the service does not run, or one on another install, as `manual` and never restarts it. A serve counts as older only on evidence (a recorded build below this one, or no recorded build and a start before the install's `package.json` was written); a newer or unknown serve, or an install on disk that is not this build, is left alone. Tests: `upgrade-restart.test.ts`, `autopilot-serve-restart-7475.test.ts`.- `src/core/upgrade-restart.ts` — GBRA-175: the pure half of the upgrade restart. `collectHostProcesses` (the process table is the authority: a heartbeat, PID-file or registry record whose pid `ps` does not show as gbrain is dropped; `gbrainEntry` skips wrappers such as tini, and a process's parent is its nearest gbrain ancestor), `planRestart` (a service restarts only processes under its manager's pid; `supervisorRestartArgv` keeps the launch options and PID file, read from `--pid-file <path>`, and drops a `--flag=value` token that is neither a global flag nor `--job-isolation=`, because the supervisor that holds it ignored it and this build's `=` gate refuses it (#7619); anything else is a `ManualStep`), `runRestartSteps`, `restartProblems` (old pids gone, one build, roles and the supervised worker back, warm socket accepts) and `unverifiedBuilds`. A stale process whose `install_root` is another install than the one being upgraded gets a `replace_copy` step first when it is a Bun global copy (`bunGlobalCopyOf`: probe the tag, `bun remove -g` then `bun add -g` under its `BUN_INSTALL`, restore on failure), else a manual step from `copyFix`; after the restart `mismatchedBuilds` names any process back on another build, and the host's `mismatchFix` chains that replacement with `gbrain upgrade --restart-only`.- `src/commands/watch.ts` — `gbrain watch`: the push transport. Reads turns from stdin as they arrive (`user:`/`assistant:` prefixes; unprefixed = user turn), keeps a rolling in-process window (`--window-turns`, default 4), calls `volunteerContext` per turn, streams pointers to stdout (`--json` for JSONL with turn attribution), logs `channel: 'watch'` events with a per-session id. Session dedupe feeds already-pushed slugs back as priorContext so the core's slug-only suppression dedupes. Blocks in the stdin iteration (interactive alive until Ctrl-C/Ctrl-D; piped ends at EOF) — deliberately NOT in DAEMON_COMMANDS; SIGINT closes the stream so teardown flows through finishCliTeardown. Per-turn resolution failures are fail-open. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS (thin clients use the `volunteer_context` MCP op). Pinned by `test/watch-command.test.ts`.diff --git a/src/commands/doctor/checks/upgrade-health.ts b/src/commands/doctor/checks/upgrade-health.tsindex ecc04477e7..84e657bbd0 100644--- a/src/commands/doctor/checks/upgrade-health.ts+++ b/src/commands/doctor/checks/upgrade-health.ts@@ -62,7 +62,7 @@ export function upgradeErrorResolved(* in ts order.*/function heldAutoUpgrade(recent: SelfUpgradeAuditEvent[], runningVersion: string): SelfUpgradeAuditEvent | null {- const last = recent.filter((e) => e.channel === 'autopilot').at(-1);+ const last = recent.filter((e) => e.channel === 'autopilot' && e.action !== 'restart_serve').at(-1);if (last?.action !== ('unsupported_runtime' satisfies SelfUpgradeAction) || !last.latest) return null;return isNewerVersion(runningVersion, last.latest) ? last : null;}diff --git a/src/commands/upgrade-restart.ts b/src/commands/upgrade-restart.tsindex 3f734683b8..30f8734111 100644--- a/src/commands/upgrade-restart.ts+++ b/src/commands/upgrade-restart.ts@@ -8,7 +8,7 @@* onto the build already installed (doctor's `runtime_build_skew` fix);* `--dry-run` prints the plan and changes nothing.*/-import { existsSync, readFileSync, realpathSync } from 'node:fs';+import { existsSync, readFileSync, realpathSync, statSync } from 'node:fs';import { homedir } from 'node:os';import { delimiter, dirname, join } from 'node:path';import { spawnSync } from '../core/spawn.ts';@@ -24,6 +24,7 @@ import { readSupervisorBuild, readSupervisorPid } from '../core/minions/supervisimport { parseEtimeToMs, readWorkers } from '../core/minions/worker-registry.ts';import { installedVersion, runtimeInstallRoot } from '../core/runtime-build.ts';import { VERSION } from '../version.ts';+import { compareWriterVersions } from '../core/persistence/writer-versions.ts';import { resolveQuietHoursWindow, resolveSelfUpgradeMode } from '../core/self-upgrade.ts';import { evaluateQuietHours } from '../core/minions/quiet-hours.ts';import { logSelfUpgrade } from '../core/audit/self-upgrade-audit.ts';@@ -298,9 +299,16 @@ export async function runHostRestart(opts: HostRestartOptions & { dryRun: booleaexport type ServeRestartOutcome = { outcome: 'no_service' | 'current' } | { outcome: 'manual' | 'restarted' | 'unverified'; detail: string };-/** #7475: the serve-only restart plan when the serve service can move every stale serve onto `expected` by itself; otherwise the outcome. */-export function planServeRestart(processes: readonly HostProcess[], serveService: ServiceHandle, expected: string, ownRoot: string | null): RestartPlan | ServeRestartOutcome {- const plan = planRestart(processes.filter(p => p.role === 'serve'), { expected, serveService, autopilotService: null, stopMcp: false, ownInstallRoot: ownRoot });+/**+ * #7475: the serve-only restart plan when the serve service can move every older serve onto `expected` by itself;+ * otherwise the outcome. A serve counts as older only on evidence: a recorded build below `expected`, or no recorded+ * build and a start before the install on disk was written (`installedMs`). A newer or unknown serve is left alone.+ */+export function planServeRestart(processes: readonly HostProcess[], serveService: ServiceHandle, expected: string, ownRoot: string | null, installedMs: number | null = null): RestartPlan | ServeRestartOutcome {+ const older = processes.filter(p => p.role === 'serve' && (p.version !== null+ ? (compareWriterVersions(p.version, expected) ?? 0) < 0+ : installedMs !== null && p.started_ms !== null && p.started_ms < installedMs));+ const plan = planRestart(older, { expected, serveService, autopilotService: null, stopMcp: false, ownInstallRoot: ownRoot });if (!plan.stale.length) return { outcome: 'current' };if (plan.manual.length || plan.steps.some(step => step.kind !== 'service')) {return { outcome: 'manual', detail: `run \`gbrain upgrade --restart-only\`: ${[...describeSteps(plan), ...plan.manual.map(m => m.instruction)].join(' ')}` };@@ -320,7 +328,12 @@ export async function restartStaleServeService(): Promise<ServeRestartOutcome> {const { sources, close, pidFile } = await hostSources();try {const collectServe = async () => (await collectHostProcesses(sources)).filter(p => p.role === 'serve');- const plan = planServeRestart(await collectServe(), services.serve, VERSION, ownInstallRoot());+ const root = ownInstallRoot();+ const onDisk = installedVersion(root);+ if (onDisk !== VERSION) return { outcome: 'manual', detail: `the install on disk holds ${onDisk ? `v${onDisk}` : 'an unknown build'}, not this autopilot's v${VERSION}; run \`gbrain upgrade --restart-only\`` };+ let installedMs: number | null = null;+ try { installedMs = statSync(join(root!, 'package.json')).mtimeMs; } catch { /* no install time: a serve with no recorded build stays as it is */ }+ const plan = planServeRestart(await collectServe(), services.serve, VERSION, root, installedMs);if (!('steps' in plan)) return plan;const { probeCliSocket } = await import('../core/cli-socket/probe.ts');const warmBefore = await probeCliSocket();diff --git a/test/autopilot-serve-restart-7475.test.ts b/test/autopilot-serve-restart-7475.test.tsindex a62b4ec2d0..00bcbd0401 100644--- a/test/autopilot-serve-restart-7475.test.ts+++ b/test/autopilot-serve-restart-7475.test.ts@@ -9,7 +9,8 @@ import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';import { planServeRestart, restartServeAfterSelfUpgrade, type ServeRestartOutcome } from '../src/commands/upgrade-restart.ts';-import { readRecentSelfUpgrades } from '../src/core/audit/self-upgrade-audit.ts';+import { logSelfUpgrade, readRecentSelfUpgrades } from '../src/core/audit/self-upgrade-audit.ts';+import { checkSelfUpgradeHealth } from '../src/commands/doctor/checks/upgrade-health.ts';import type { HostProcess } from '../src/core/upgrade-restart.ts';import { withEnv } from './helpers/with-env.ts';@@ -37,6 +38,17 @@ describe('planServeRestart (#7475)', () => {expect('detail' in plan && plan.detail).toContain('kill 77');});+ test('a serve with no recorded build restarts only when it started before the install was written', () => {+ expect(planServeRestart([serve(9, null, { started_ms: 1_000 })], SERVE_SERVICE, NEW, null, 5_000)).toMatchObject({ steps: [{ kind: 'service', pids: [9] }] });+ expect(planServeRestart([serve(9, null, { started_ms: 9_000 })], SERVE_SERVICE, NEW, null, 5_000)).toEqual({ outcome: 'current' });+ expect(planServeRestart([serve(9, null)], SERVE_SERVICE, NEW, null, null)).toEqual({ outcome: 'current' });+ expect(planServeRestart([serve(9, null, { started_ms: null })], SERVE_SERVICE, NEW, null, 5_000)).toEqual({ outcome: 'current' });+ });++ test('a serve on a newer build than the autopilot is left alone', () => {+ expect(planServeRestart([serve(9, '0.60.999.0')], SERVE_SERVICE, NEW, null, 5_000)).toEqual({ outcome: 'current' });+ });+test('a serve the service starts from another install is reported, not restarted onto the old build', () => {const other = '/opt/example/install/global/node_modules/gbrain';const plan = planServeRestart([serve(9, OLD, { install_root: other, install_version: OLD })], SERVE_SERVICE, NEW, '/srv/gbrain-example');@@ -111,4 +123,13 @@ describe('restartServeAfterSelfUpgrade (#7475)', () => {expect(r.state.tried).toBe(false);}});++ test('a restart_serve row does not hide doctor\'s held auto-upgrade warning', async () => {+ const { home, audit } = brainHome('auto', true);+ await withEnv({ GBRAIN_HOME: home, GBRAIN_AUDIT_DIR: audit }, async () => {+ logSelfUpgrade({ channel: 'autopilot', action: 'unsupported_runtime', current: '0.42.0', latest: '0.99.0', outcome: 'skipped', reason: 'gbrain 0.99.0 requires Bun >=9.0.0' });+ await restartServeAfterSelfUpgrade(idle, { state: { tried: false }, restart: async () => ({ outcome: 'unverified', detail: 'serve did not come back' }) });+ expect(checkSelfUpgradeHealth().message).toContain('Auto-upgrade to 0.99.0 held');+ });+ });});