Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Report a shared-skills migration stuck on a source conflict as owed work, with a read-only first step (#9480)

Garry Taneef74759b59e
diff --git a/behavior-changes.d/migration_host_work_owed.md b/behavior-changes.d/migration_host_work_owed.md
new file mode 100644
index 0000000000..c1dc033193
--- /dev/null
+++ b/behavior-changes.d/migration_host_work_owed.md
@@ -0,0 +1,4 @@
+---
+kind: fix
+---
+A migration that stopped three or more runs in a row on a conflict in one or more sources, and recorded that host work is still owed (today only v0.53.0, the shared-skills content migration), is no longer called history once a newer migration completes. gbrain apply-migrations --list, the notice other commands and gbrain serve print at startup, and gbrain doctor (as a warning) now all say the work is still owed and that it does not clear without a person, and they all name the same first step, gbrain apply-migrations --migration 0.53.0 --dry-run --json, which lists each source's conflict and changes nothing. They give the order that works: after the user reviews each conflict, gbrain apply-migrations --force-retry 0.53.0 clears the wedge, then the command the inspect output names for that source (or gbrain apply-migrations --yes) settles it. That inspect command used to refuse a wedged version as not registered; it now works. Running a wedged migration by name, for example the accept command a conflict message names, still exits 2 but now says the version is wedged and names those steps, instead of calling it not registered. Nothing runs the migration again by itself, and the exit status of apply-migrations, post-upgrade and upgrade does not change.
diff --git a/changelog.d/capy-migration-host-work-owed.md b/changelog.d/capy-migration-host-work-owed.md
new file mode 100644
index 0000000000..49670de561
--- /dev/null
+++ b/changelog.d/capy-migration-host-work-owed.md
@@ -0,0 +1,6 @@
+**A stuck shared-skills migration is reported as owed work everywhere, with a read-only first step, instead of being called history.**
+
+### Fixed
+- A migration that stopped three or more runs in a row on a source conflict and still owes host work (v0.53.0 today) is no longer called "history, not outstanding work" because a newer migration completed. `gbrain apply-migrations --list`, the notice on other commands and `gbrain serve` startup, and `gbrain doctor` (a warning) now say the same thing: the work is still owed, it does not clear without a person, and the first step is `gbrain apply-migrations --migration 0.53.0 --dry-run --json`, which lists each source's conflict and changes nothing. They give the order that works: review each conflict, clear the wedge with `gbrain apply-migrations --force-retry 0.53.0`, then run the command the inspect output names for that source, or `gbrain apply-migrations --yes`. That inspect command no longer refuses a wedged version as "not registered", and running a wedged migration by name (for example the accept command a conflict message names) still exits 2 but says it is wedged and names those steps. Nothing runs the migration again by itself, and no exit status changes.
+
+<!-- behavior-change: migration_host_work_owed -->
diff --git a/docs/architecture/key-files/commands-1.md b/docs/architecture/key-files/commands-1.md
index 54c0407488..2b6bea7a29 100644
--- a/docs/architecture/key-files/commands-1.md
+++ b/docs/architecture/key-files/commands-1.md
@@ -3,7 +3,7 @@
[Subsystem index](../KEY_FILES.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-- `src/commands/apply-migrations.ts` and `src/commands/migrations/v0_11_0.ts` — forced dry-runs return a no-write action preview before opening a database or loading/copying the migration ledger. Any failed phase makes the migration partial, stops the chain and exits nonzero; ordinary pending-host-work partial results retain their resume behavior. Explicit `--force-retry` recovers a historical false-complete record. No-autopilot skips remain successful and do not install services. Phase F (`v0_11_0-install.ts`) reinstalls only an existing autopilot. Every run except `--list` and `--dry-run` holds the orchestration lock (`src/core/migration-orchestration-lock.ts`) from before its first ledger write through its final status: a Postgres TTL lease row `gbrain-apply-migrations` in `gbrain_cycle_locks` (survives transaction-mode poolers; an unreachable database or a schema without the table runs unlocked until the preflight creates it), or on PGLite a native lock on `<datastore>.gbrain-migrations.lock`, separate from the datastore lock so orchestrators still open the datastore. A second runner prints `another apply-migrations is running (host …, pid …)` and exits 75; dead holders are taken over by the existing db-lock and kernel rules. The locked body returns its exit status so the lock is released before `process.exit`. Tests: `apply-migrations-safety.serial.test.ts`, `apply-migrations-orchestration-lock.test.ts`, `e2e/apply-migrations-orchestration-lock.test.ts` and `e2e/migration-preview-safety.test.ts`. D2 (`json: 'document'`): every `--json` path is one single-line document (`status` unconfigured/dry_run/listed/ok/partial/up_to_date, `plan`, `previews`, `results[]`, `retry_markers`, `schema`); a non-zero exit is one envelope leading with those keys (`invalid_params` exit 2, `migration_failed` with reasons, `migrations_pending` with `schema_pending`, `database_error` for `--require-db`, `migrations_running` exit 75, `migration_lease_lost` exit 1 with `lease`). #6089: the run never exits 0 with the schema behind (`src/commands/apply-migrations-schema.ts`); an applied reconcile re-check is not pending work, so with nothing else pending a schema the pre-flight left behind fails before the re-check, and after orchestrators the version is read again. #6028 diagnostic: `assertHeld` raises `MigrationLeaseLostError` (`migration_lease_lost`, details say whether the acquisition token and time still match, never the token) when the conflicting lease row names this pid and host, instead of `migrations_running`; tests `e2e/migration-lease-lost.test.ts`. A4: `Migration.effects` (v0.11.0: `persistent_install`, its Phase F autopilot install) is asked through `consentGate` before the orchestrator runs (skipped under `--no-autopilot-install` or on PGLite); `--yes`, `--non-interactive` (mapped to `persistent_install` only), a preapproval or a TTY yes authorize, otherwise exit 3. `applyMigrations()` returns `{exitCode, failure}` for in-process callers (post-upgrade); `runApplyMigrations()` exits. Child CLIs use `cliChildStdio()`. Tests: `test/cli-json-commands.test.ts`. the plan adds `pending_fresh_install` (unrun setup work on a brain `gbrain init` created; runs like `pending`, reported as expected work, never as a failed upgrade); `--list --json` prints one row per migration (`fresh_install: true` on init stamps) plus `next.argv` when work remains and a `wedged[]` entry (version, `fix.argv` `--force-retry <v>`) per wedged migration; a run that leaves a wedged migration (3+ consecutive partials) that no newer migration ran past (doctor's forward-progress rule, `forwardProgressPast` in `src/core/migration-ledger.ts`; a superseded wedge is listed as `wedged_superseded` in the plan and is history) never reports up to date: it fails `migrations_pending` reason `wedged` (exit 1, fix `next: ask_user`, verify `--list --json`), so `post-upgrade` exits non-zero and `gbrain upgrade` prints `Migrations: wedged`; tests `apply-migrations-wedged-6696.test.ts`; every wedged-migration message (run envelope, `--list`, stderr, `gbrain upgrade`) points at the migration's partial rows in this brain's `migrations/completed.jsonl` (`phases[].detail`, if present) through `wedgedLedgerHint` (`src/core/migration-ledger.ts`); a phase's `argv` prints as its next step. v0.11.0 Phase F (`src/commands/migrations/v0_11_0-install.ts`) installs the autopilot only for the brain's `sync.repo_path`; without one it installs nothing, is `skipped` with the exact install command (the default source's `local_path`, else `<REPO_PATH>`), records an `autopilot-install-needs-repo` pending host-work item, and the migration completes; doctor's `minions_migration` line (`src/commands/doctor/checks/autopilot-install-pending.ts`) carries that argv (`persistent_install`) until an autopilot is installed for this brain. Phase F records a `reason` code on those outcomes (`no_repo_path`, `autopilot_already_installed`, `install_failed`). `clearRepoPathWedge` writes a `retry` marker (reason `phase_f_no_repo_path`) before the plan when every partial behind a v0.11.0 wedge failed only the install phase (whatever its detail; the old child's stderr was never captured) and this Postgres brain's `sync.repo_path` is unset, so Phase F would now skip; with it set, or any other wedge, the `--force-retry` advice stands. Forced dry-runs return a no-write action preview before opening a database or loading/copying the migration ledger. Any failed phase makes the migration partial, stops the chain and exits nonzero; ordinary pending-host-work partial results retain their resume behavior. Explicit `--force-retry` recovers a historical false-complete record. Every run except `--list` and `--dry-run` holds the orchestration lock (`src/core/migration-orchestration-lock.ts`) from before its first ledger write through its final status: a Postgres TTL lease row `gbrain-apply-migrations` in `gbrain_cycle_locks` (survives transaction-mode poolers; an unreachable database or a schema without the table runs unlocked until the preflight creates it), or on PGLite a native lock on `<datastore>.gbrain-migrations.lock`, separate from the datastore lock so orchestrators still open the datastore. A second runner prints `another apply-migrations is running (host …, pid …)` and exits 75; dead holders are taken over by the existing db-lock and kernel rules. The locked body returns its exit status so the lock is released before `process.exit`. Tests: `apply-migrations-safety.serial.test.ts`, `apply-migrations-orchestration-lock.test.ts`, `e2e/apply-migrations-orchestration-lock.test.ts` and `e2e/migration-preview-safety.test.ts`.
+- `src/commands/apply-migrations.ts` and `src/commands/migrations/v0_11_0.ts` — forced dry-runs return a no-write action preview before opening a database or loading/copying the migration ledger. Any failed phase makes the migration partial, stops the chain and exits nonzero; ordinary pending-host-work partial results retain their resume behavior. Explicit `--force-retry` recovers a historical false-complete record. No-autopilot skips remain successful and do not install services. Phase F (`v0_11_0-install.ts`) reinstalls only an existing autopilot. Every run except `--list` and `--dry-run` holds the orchestration lock (`src/core/migration-orchestration-lock.ts`) from before its first ledger write through its final status: a Postgres TTL lease row `gbrain-apply-migrations` in `gbrain_cycle_locks` (survives transaction-mode poolers; an unreachable database or a schema without the table runs unlocked until the preflight creates it), or on PGLite a native lock on `<datastore>.gbrain-migrations.lock`, separate from the datastore lock so orchestrators still open the datastore. A second runner prints `another apply-migrations is running (host …, pid …)` and exits 75; dead holders are taken over by the existing db-lock and kernel rules. The locked body returns its exit status so the lock is released before `process.exit`. Tests: `apply-migrations-safety.serial.test.ts`, `apply-migrations-orchestration-lock.test.ts`, `e2e/apply-migrations-orchestration-lock.test.ts` and `e2e/migration-preview-safety.test.ts`. D2 (`json: 'document'`): every `--json` path is one single-line document (`status` unconfigured/dry_run/listed/ok/partial/up_to_date, `plan`, `previews`, `results[]`, `retry_markers`, `schema`); a non-zero exit is one envelope leading with those keys (`invalid_params` exit 2, `migration_failed` with reasons, `migrations_pending` with `schema_pending`, `database_error` for `--require-db`, `migrations_running` exit 75, `migration_lease_lost` exit 1 with `lease`). #6089: the run never exits 0 with the schema behind (`src/commands/apply-migrations-schema.ts`); an applied reconcile re-check is not pending work, so with nothing else pending a schema the pre-flight left behind fails before the re-check, and after orchestrators the version is read again. #6028 diagnostic: `assertHeld` raises `MigrationLeaseLostError` (`migration_lease_lost`, details say whether the acquisition token and time still match, never the token) when the conflicting lease row names this pid and host, instead of `migrations_running`; tests `e2e/migration-lease-lost.test.ts`. A4: `Migration.effects` (v0.11.0: `persistent_install`, its Phase F autopilot install) is asked through `consentGate` before the orchestrator runs (skipped under `--no-autopilot-install` or on PGLite); `--yes`, `--non-interactive` (mapped to `persistent_install` only), a preapproval or a TTY yes authorize, otherwise exit 3. `applyMigrations()` returns `{exitCode, failure}` for in-process callers (post-upgrade); `runApplyMigrations()` exits. Child CLIs use `cliChildStdio()`. Tests: `test/cli-json-commands.test.ts`. the plan adds `pending_fresh_install` (unrun setup work on a brain `gbrain init` created; runs like `pending`, reported as expected work, never as a failed upgrade); `--list --json` prints one row per migration (`fresh_install: true` on init stamps) plus `next.argv` when work remains and a `wedged[]` entry (version, `fix.argv` `--force-retry <v>`) per wedged migration; a run that leaves a wedged migration (3+ consecutive partials) that no newer migration ran past (doctor's forward-progress rule, `forwardProgressPast` in `src/core/migration-ledger.ts`; a superseded wedge is listed as `wedged_superseded` in the plan and is history; a v0.53.0 wedge whose latest row records `apply_migrations_pending` still owes host work, `hostWorkOwedForVersion` (other migrations set that flag on ordinary partials and keep the forward-progress rule), is never superseded: it is listed as `wedged_host_work` with the read-only `--migration <v> --dry-run --json` inspect as its fix and `ask_user` `--force-retry` after it, nothing reruns it, and it fails a run only when no newer migration ran past it; tests `migration-host-work-owed.test.ts`) never reports up to date: it fails `migrations_pending` reason `wedged` (exit 1, fix `next: ask_user`, verify `--list --json`), so `post-upgrade` exits non-zero and `gbrain upgrade` prints `Migrations: wedged`; tests `apply-migrations-wedged-6696.test.ts`; every wedged-migration message (run envelope, `--list`, stderr, `gbrain upgrade`) points at the migration's partial rows in this brain's `migrations/completed.jsonl` (`phases[].detail`, if present) through `wedgedLedgerHint` (`src/core/migration-ledger.ts`); a phase's `argv` prints as its next step. v0.11.0 Phase F (`src/commands/migrations/v0_11_0-install.ts`) installs the autopilot only for the brain's `sync.repo_path`; without one it installs nothing, is `skipped` with the exact install command (the default source's `local_path`, else `<REPO_PATH>`), records an `autopilot-install-needs-repo` pending host-work item, and the migration completes; doctor's `minions_migration` line (`src/commands/doctor/checks/autopilot-install-pending.ts`) carries that argv (`persistent_install`) until an autopilot is installed for this brain. Phase F records a `reason` code on those outcomes (`no_repo_path`, `autopilot_already_installed`, `install_failed`). `clearRepoPathWedge` writes a `retry` marker (reason `phase_f_no_repo_path`) before the plan when every partial behind a v0.11.0 wedge failed only the install phase (whatever its detail; the old child's stderr was never captured) and this Postgres brain's `sync.repo_path` is unset, so Phase F would now skip; with it set, or any other wedge, the `--force-retry` advice stands. Forced dry-runs return a no-write action preview before opening a database or loading/copying the migration ledger. Any failed phase makes the migration partial, stops the chain and exits nonzero; ordinary pending-host-work partial results retain their resume behavior. Explicit `--force-retry` recovers a historical false-complete record. Every run except `--list` and `--dry-run` holds the orchestration lock (`src/core/migration-orchestration-lock.ts`) from before its first ledger write through its final status: a Postgres TTL lease row `gbrain-apply-migrations` in `gbrain_cycle_locks` (survives transaction-mode poolers; an unreachable database or a schema without the table runs unlocked until the preflight creates it), or on PGLite a native lock on `<datastore>.gbrain-migrations.lock`, separate from the datastore lock so orchestrators still open the datastore. A second runner prints `another apply-migrations is running (host …, pid …)` and exits 75; dead holders are taken over by the existing db-lock and kernel rules. The locked body returns its exit status so the lock is released before `process.exit`. Tests: `apply-migrations-safety.serial.test.ts`, `apply-migrations-orchestration-lock.test.ts`, `e2e/apply-migrations-orchestration-lock.test.ts` and `e2e/migration-preview-safety.test.ts`.
- `src/commands/migrations/run-engine.ts` — one PGLite engine for consecutive migrations of an apply-migrations run (each open starts an embedded Postgres, about 130 ms). `withRunEngineSession` wraps the locked run; `setRunEngineSharing` enables sharing for the opt-out read, each migration flagged `sharesRunEngine` (v0.11.0, v0.14.0, v0.22.4, v0.28.0, v0.53.0) and the post-orchestrator schema check, and disabling it (before an unflagged migration, `--list`, `--dry-run` or a consent prompt) closes the engine. `connectMigrationEngine` returns the shared engine with a no-op close while sharing, otherwise a new engine its caller closes (Postgres always). `releaseRunEngine` closes it before a `gbrain` child that needs the brain (v0.11.0's `jobs smoke`). A fresh setup's migration child opens the brain three times instead of eight (counting the `jobs smoke` child); the ledger it writes is pinned by `test/fixtures/goldens/setup/fresh-setup-migration-ledger.json`. Tests: `test/migrations-run-engine.test.ts`.
- `src/commands/apply-migrations-schema.ts` — #6089 schema half of the apply-migrations exit contract. `connectAtSchemaVersion` connects the configured brain and reads `config.version` (pre-flight, `--force-schema`); the post-orchestrator check opens it through `connectMigrationEngine`. `checkSchemaAfterOrchestrators` re-reads the version after the orchestrators ran: on Postgres only when the pre-flight connected; on PGLite in-process once the orchestrators released the datastore (the pre-flight stays skipped, #1100; never a child process), applying the schema migrations under `--yes`/`--non-interactive`; a busy PGLite datastore is rethrown. `schemaPendingError` builds `migrations_pending` with reason `not_applied` (fix reruns with `--yes`), `still_behind` (after an authorized run) or `schema_unreadable` (both fix `next: report` with `gbrain doctor --json` as verify, never another `--yes`), each naming the pending schema versions. `reportDatabaseUnreachable` keeps an unreachable Postgres non-fatal but prints the `GBRAIN_DB_ACCESS <reason>` marker. Tests: `test/apply-migrations-schema-exit.test.ts` (PGLite, unreachable Postgres, post-upgrade, postinstall) and `test/e2e/apply-migrations-schema-exit.test.ts` (Postgres).
diff --git a/docs/guides/shared-brain-skills.md b/docs/guides/shared-brain-skills.md
index 137bb7cf15..bffcae3e06 100644
--- a/docs/guides/shared-brain-skills.md
+++ b/docs/guides/shared-brain-skills.md
@@ -538,6 +538,13 @@ before parking existed) resumes with
`gbrain apply-migrations --migration 0.53.0 --yes`; the oversized source parks
and the migration completes.
+A 0.53.0 migration wedged on a source conflict still owes that work, even after
+a newer migration completes, and it does not clear without a person. Run
+`gbrain apply-migrations --migration 0.53.0 --dry-run --json` (it changes
+nothing) to see each source's conflict, resolve it with the user, and only then
+run `gbrain apply-migrations --force-retry 0.53.0` and the command the output
+names for that source, or `gbrain apply-migrations --yes`.
+
## Troubleshoot, leave, and recover
Inspect this installation's local receipt without credentials or a live host:
diff --git a/src/cli/migrations-notice.ts b/src/cli/migrations-notice.ts
index 28e2999e3f..55cb6a781c 100644
--- a/src/cli/migrations-notice.ts
+++ b/src/cli/migrations-notice.ts
@@ -19,13 +19,17 @@ export function migrationsNotice(why: string): Notice {
export async function pendingUpgradeMigrationsNotice(engine: BrainEngine, command: string, version: string): Promise<Notice | null> {
if (ORCHESTRATOR_MIGRATION_OWNERS.has(command)) return null;
try {
- const { declinedMigrations, migrationLedgerSummary, upgradeLedgerApplies } = await import('../core/migration-ledger.ts');
- if (!upgradeLedgerApplies(engine)) return null;
- const summary = migrationLedgerSummary(version, await declinedMigrations(engine));
- if (summary.wedged.length > 0) {
+ const ledger = await import('../core/migration-ledger.ts');
+ if (!ledger.upgradeLedgerApplies(engine)) return null;
+ const summary = ledger.migrationLedgerSummary(version, await ledger.declinedMigrations(engine));
+ const history = ledger.indexCompletedEntries((await import('../core/preferences.ts')).loadCompletedMigrations());
+ const owed = summary.wedged.filter(v => ledger.hostWorkOwedForVersion(v, history));
+ const wedged = summary.wedged.filter(v => !owed.includes(v));
+ if (wedged.length > 0) {
const { wedgedMigrationFix } = await import('../commands/apply-migrations.ts');
- return { ...migrationsNotice(`Upgrade migration(s) ${summary.wedged.join(', ')} failed several runs in a row and are wedged; the command ran, and apply-migrations --yes will not clear them until the cause is fixed.`), fix: wedgedMigrationFix(summary.wedged) };
+ return { ...migrationsNotice(`Upgrade migration(s) ${wedged.join(', ')} failed several runs in a row and are wedged; the command ran, and apply-migrations --yes will not clear them until the cause is fixed.`), fix: wedgedMigrationFix(wedged) };
}
+ if (owed.length > 0) return { ...migrationsNotice(`${ledger.hostWorkOwedMessage(owed)} The command ran.`), fix: ledger.hostWorkOwedFix(owed) };
const pending = [...summary.pending, ...summary.partial];
if (pending.length === 0) return null;
const notice = migrationsNotice(`${pending.length} upgrade migration(s) have not finished on this brain yet (${pending.join(', ')}); the command ran, and gbrain apply-migrations --yes --no-autopilot-install finishes the upgrade.`);
diff --git a/src/commands/apply-migrations.ts b/src/commands/apply-migrations.ts
index 6e54aa873d..218c327368 100644
--- a/src/commands/apply-migrations.ts
+++ b/src/commands/apply-migrations.ts
@@ -36,6 +36,9 @@ import {
isPendingFreshInstall,
hostWorkInspectArgv,
hostWorkPendingForVersion,
+ hostWorkOwedFix,
+ hostWorkOwedForVersion,
+ hostWorkOwedMessage,
MAX_CONSECUTIVE_PARTIALS,
wedgedLedgerHint,
} from '../core/migration-ledger.ts';
@@ -261,6 +264,8 @@ interface Plan {
wedged: Migration[];
/** #6696 C5: wedged in the ledger, but a newer migration ran past it (doctor's forward-progress rule): history, not outstanding work. */
wedged_superseded: Migration[];
+ /** Wedged with host work still owed (its latest row records apply_migrations_pending): never superseded and never run by itself; a person resolves the conflict first. */
+ wedged_host_work: Migration[];
/** #6466: declined by the brain's config (`Migration.optOut`); never run, listed as `declined`. */
declined: Migration[];
}
@@ -283,7 +288,7 @@ interface Plan {
* skip v0.11.0 when running v0.11.1. Compare against completed.jsonl.
*/
function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: string, declined: ReadonlySet<string> = new Set(), hostWork: ReadonlySet<string> = new Set()): Plan {
- const plan: Plan = { applied: [], applied_pending_host_work: [], partial: [], pending: [], pending_fresh_install: [], skippedFuture: [], wedged: [], wedged_superseded: [], declined: [] };
+ const plan: Plan = { applied: [], applied_pending_host_work: [], partial: [], pending: [], pending_fresh_install: [], skippedFuture: [], wedged: [], wedged_superseded: [], wedged_host_work: [], declined: [] };
const ledger = [...idx.byVersion.values()].flat();
for (const m of migrations) {
if (filterVersion && m.version !== filterVersion) continue;
@@ -297,6 +302,7 @@ function buildPlan(idx: CompletedIndex, installed: string, filterVersion?: strin
plan.applied.push(m);
if (hostWork.has(m.version) && hostWorkPendingForVersion(m.version, idx.byVersion)) plan.applied_pending_host_work.push(m);
} else if (status === 'partial') plan.partial.push(m);
+ else if (status === 'wedged' && hostWorkOwedForVersion(m.version, idx.byVersion)) plan.wedged_host_work.push(m);
else if (status === 'wedged') (forwardProgressPast(m.version, ledger) === null ? plan.wedged : plan.wedged_superseded).push(m);
else if (isPendingFreshInstall(m.version, idx.freshVersion)) plan.pending_fresh_install.push(m);
else plan.pending.push(m);
@@ -330,7 +336,7 @@ function listRows(plan: Plan): Array<{ status: string; m: Migration }> {
return [
...plan.applied.map(m => ({ status: plan.applied_pending_host_work.includes(m) ? 'applied_pending_host_work' : 'applied', m })),
...plan.partial.map(m => ({ status: 'partial', m })),
- ...[...plan.wedged, ...plan.wedged_superseded].map(m => ({ status: 'wedged', m })),
+ ...[...plan.wedged, ...plan.wedged_host_work, ...plan.wedged_superseded].map(m => ({ status: 'wedged', m })),
...plan.pending.map(m => ({ status: 'pending', m })),
...plan.pending_fresh_install.map(m => ({ status: 'pending_fresh_install', m })),
...plan.declined.map(m => ({ status: 'declined', m })),
@@ -355,11 +361,12 @@ function printList(plan: Plan, installed: string, dbProbe: DbProbeOutcome): void
const setup = plan.pending_fresh_install.length;
for (const m of plan.declined) console.log(`v${m.version} is declined by config (${m.optOut?.key} is false); turn it back on: gbrain config set ${m.optOut?.key} true`);
for (const m of plan.wedged) console.log(`v${m.version} is wedged (${MAX_CONSECUTIVE_PARTIALS}+ consecutive partial runs). Read ${wedgedLedgerHint()}, fix the cause, then: ${shellQuote(forceRetryArgv(m.version))}`);
+ if (plan.wedged_host_work.length) console.log(hostWorkOwedMessage(plan.wedged_host_work.map(m => m.version)));
for (const m of plan.applied_pending_host_work) {
console.log(`v${m.version} is applied_pending_host_work: its mechanical checkpoint is recorded, but host publication or client activation is not done. `
+ `Inspect each source's next step: ${shellQuote(hostWorkInspectArgv(m.version))}; once that work is done, \`gbrain apply-migrations --yes\` records it in the migration's report.`);
}
- if (needsWork === 0 && setup === 0 && plan.wedged.length === 0) {
+ if (needsWork === 0 && setup === 0 && plan.wedged.length === 0 && plan.wedged_host_work.length === 0) {
if (plan.applied_pending_host_work.length === 0) console.log('All migrations up to date.');
return;
}
@@ -386,6 +393,8 @@ function listJson(plan: Plan, idx: CompletedIndex, installed: string, dbProbe: D
})),
needs_action: plan.pending.length + plan.partial.length,
...(plan.wedged.length ? { wedged: plan.wedged.map(m => ({ version: m.version, fix: { argv: forceRetryArgv(m.version), command: shellQuote(forceRetryArgv(m.version)) } })) } : {}),
+ ...(plan.wedged_host_work.length ? { wedged_host_work: plan.wedged_host_work.map(m => ({ version: m.version,
+ fix: { argv: hostWorkInspectArgv(m.version), command: shellQuote(hostWorkInspectArgv(m.version)) } })) } : {}),
pending_fresh_install: plan.pending_fresh_install.length,
...(plan.applied_pending_host_work.length ? { applied_pending_host_work: plan.applied_pending_host_work.map(m => ({ version: m.version,
fix: { argv: hostWorkInspectArgv(m.version), command: shellQuote(hostWorkInspectArgv(m.version)) } })) } : {}),
@@ -422,6 +431,11 @@ function printDryRun(plan: Plan, installed: string, dbProbe: DbProbeOutcome): vo
for (const m of plan.declined) console.log(` ⊘ v${m.version} — ${m.optOut?.key} is false`);
console.log('');
}
+ if (plan.wedged_host_work.length) {
+ console.log('Wedged with host work owed (not run; a person resolves each source\'s conflict first):');
+ for (const m of plan.wedged_host_work) console.log(` ✗ v${m.version} — inspect each source: ${shellQuote(hostWorkInspectArgv(m.version))}`);
+ console.log('');
+ }
if (plan.skippedFuture.length) {
console.log('Skipped (newer than installed binary):');
for (const m of plan.skippedFuture) console.log(` ⧗ v${m.version}`);
@@ -494,6 +508,10 @@ interface RunReport {
failure?: OperationError;
/** #6696 C5: wedged migrations the run cannot clear itself; a run that otherwise succeeds fails `migrations_pending` reason `wedged`. */
wedged?: readonly Migration[];
+ /** Wedged with host work owed and no newer migration past it: fails like `wedged`, with the read-only inspect as its fix. */
+ wedgedHostWork?: readonly Migration[];
+ /** Every wedged migration with host work owed, past a newer migration or not: the run never says all migrations are up to date while one is listed. */
+ hostWorkOwed?: readonly Migration[];
}
function unknownMigrationError(version: string): OperationError {
@@ -502,6 +520,14 @@ function unknownMigrationError(version: string): OperationError {
{ fix: { argv: ['gbrain', COMMAND, '--list'], consent: [], actor: 'agent', requires_exclusive: false, why: 'Lists every registered migration and its status.' } });
}
+/** A named run of a wedged migration: it runs by name again only after its wedge is cleared; --list and --dry-run still inspect it. */
+function wedgedByNameError(version: string, hostWorkOwed: boolean): OperationError {
+ return usageError(`Migration v${version} is wedged (${MAX_CONSECUTIVE_PARTIALS}+ consecutive partial runs), so it does not run by name until the wedge is cleared.`,
+ `${hostWorkOwed ? `See each source's conflict first (read-only): ${shellQuote(hostWorkInspectArgv(version))}. ` : `Read ${wedgedLedgerHint()} for the last failure and fix its cause. `}`
+ + `Then clear the wedge with ${shellQuote(forceRetryArgv(version))} and run this command again.`,
+ { fix: hostWorkOwed ? hostWorkOwedFix([version]) : wedgedMigrationFix([version]) });
+}
+
/** The same invocation again once the underlying failure is fixed; any approval it needs is asked by the run itself. */
function rerunFix(args: readonly string[], why: string, yes = false) {
// Approval flags are dropped: the rerun asks for any consent effect itself.
@@ -729,11 +755,16 @@ async function runLockedMigrations(
doc.database = dbProbe;
doc.plan = Object.fromEntries(Object.entries(plan).map(([state, entries]) => [state, entries.map((migration: Migration) => migration.version)]));
report.wedged = plan.wedged;
- reportWedgedMigrations(plan.wedged, plan.wedged_superseded, completed);
+ report.hostWorkOwed = plan.wedged_host_work;
+ report.wedgedHostWork = plan.wedged_host_work.filter(m => forwardProgressPast(m.version, completed) === null);
+ reportWedgedMigrations(plan.wedged, plan.wedged_superseded, plan.wedged_host_work, completed);
- if (cli.specificMigration && plan.applied.length + plan.partial.length + plan.pending.length + plan.pending_fresh_install.length + plan.skippedFuture.length + plan.declined.length === 0) {
- console.error(`No migration registered with version "${cli.specificMigration}". Run \`gbrain apply-migrations --list\` to see registered versions.`);
- fail('invalid', unknownMigrationError(cli.specificMigration));
+ const namedWedged = plan.wedged.length + plan.wedged_superseded.length + plan.wedged_host_work.length;
+ const readOnlyWedged = cli.list || cli.dryRun ? namedWedged : 0;
+ if (cli.specificMigration && plan.applied.length + plan.partial.length + plan.pending.length + plan.pending_fresh_install.length + plan.skippedFuture.length + plan.declined.length + readOnlyWedged === 0) {
+ const error = namedWedged ? wedgedByNameError(cli.specificMigration, plan.wedged_host_work.length > 0) : unknownMigrationError(cli.specificMigration);
+ console.error(`${error.message} ${error.suggestion}`);
+ fail('invalid', error);
return 2;
}
// #6466: an explicit run of a declined migration refuses (T1) instead of overriding brain-level config every machine shares.
@@ -751,7 +782,7 @@ async function runLockedMigrations(
if (cli.list) { printList(plan, installed, dbProbe); return listExit; }
if (cli.dryRun) {
const previews: Array<{ version: string; preview?: unknown; error?: string }> = [];
- for (const migration of [...plan.applied, ...plan.partial, ...plan.pending, ...plan.pending_fresh_install, ...plan.wedged, ...plan.wedged_superseded]) {
+ for (const migration of [...plan.applied, ...plan.partial, ...plan.pending, ...plan.pending_fresh_install, ...plan.wedged, ...plan.wedged_host_work, ...plan.wedged_superseded]) {
if (!migration.preview) continue;
try { previews.push({ version: migration.version, preview: await migration.preview(orchestratorOptsFrom(cli)) }); }
catch (error) { previews.push({ version: migration.version, error: error instanceof Error ? error.message : 'Inventory unavailable.' }); }
@@ -790,7 +821,7 @@ async function runLockedMigrations(
.sort((left, right) => compareVersions(left.version, right.version));
if (toRun.length === 0) return upToDateOrWedged(report);
if (!schemaBehind && pendingWork.length === 0) {
- console.log('All migrations up to date. This covers orchestrator checkpoints only; host publication and client activation are being rechecked.');
+ console.log(`${report.hostWorkOwed?.length ? othersUpToDate(report.hostWorkOwed) : 'All migrations up to date.'} This covers orchestrator checkpoints only; host publication and client activation are being rechecked.`);
}
// A4: a migration whose orchestrator performs a consent effect (v0.11.0's
@@ -1031,14 +1062,30 @@ function forceRetryArgv(version: string): string[] { return ['gbrain', COMMAND,
function upToDateOrWedged(report: RunReport): number {
const wedged = failWedged(report);
if (wedged !== undefined) return wedged;
- console.log('All migrations up to date.');
report.doc.status = 'up_to_date';
+ const owed = report.hostWorkOwed ?? [];
+ if (owed.length) report.doc.wedged_host_work = owed.map(m => m.version);
+ console.log(owed.length ? othersUpToDate(owed) : 'All migrations up to date.');
return 0;
}
+/** Said instead of "All migrations up to date" while a wedged migration still owes host work (reported above; the exit status is unchanged). */
+function othersUpToDate(owed: readonly Migration[]): string {
+ return `Every other migration is up to date; ${owed.map(m => `v${m.version}`).join(', ')} still owes host work a person has to resolve (see above).`;
+}
+
/** #6696 C5: a run that leaves a wedged migration fails `migrations_pending` (reason `wedged`, exit 1); undefined when none is wedged. */
function failWedged(report: RunReport): number | undefined {
const wedged = report.wedged ?? [];
+ const owed = report.wedgedHostWork ?? [];
+ if (!wedged.length && owed.length) {
+ report.doc.status = 'wedged';
+ report.doc.wedged = owed.map(m => m.version);
+ report.failure = opError('migrations_pending', hostWorkOwedMessage(owed.map(m => m.version)),
+ 'Run the read-only inspect in the fix, show the user each source\'s conflict, and clear the wedge only after they resolve it.',
+ { reason: 'wedged', fix: hostWorkOwedFix(owed.map(m => m.version)) });
+ return 1;
+ }
if (!wedged.length) return undefined;
const versions = wedged.map(m => `v${m.version}`).join(', ');
report.doc.status = 'wedged';
@@ -1060,7 +1107,8 @@ export function wedgedMigrationFix(wedged: readonly string[]): Action {
}
/** Bug 3 — surface wedged migrations as a loud, actionable error; the run continues (applied/partial/pending are still reported and run). */
-function reportWedgedMigrations(wedged: readonly Migration[], superseded: readonly Migration[], completed: readonly CompletedMigrationEntry[]): void {
+function reportWedgedMigrations(wedged: readonly Migration[], superseded: readonly Migration[], owed: readonly Migration[], completed: readonly CompletedMigrationEntry[]): void {
+ if (owed.length) console.error(`\n${hostWorkOwedMessage(owed.map(m => m.version))}`);
for (const m of superseded) {
console.error(`\nMigration v${m.version} has ${MAX_CONSECUTIVE_PARTIALS}+ consecutive partial runs, but v${forwardProgressPast(m.version, completed)} completed after it, `
+ 'so it is history, not outstanding work. Nothing to do.');
diff --git a/src/commands/doctor/checks/local-runtime.ts b/src/commands/doctor/checks/local-runtime.ts
index 8e3120c93a..799b598387 100644
--- a/src/commands/doctor/checks/local-runtime.ts
+++ b/src/commands/doctor/checks/local-runtime.ts
@@ -8,7 +8,7 @@
*/
import { loadCompletedMigrations } from '../../../core/preferences.ts';
-import { MIGRATION_OPT_OUT_KEYS, declinedMigrations, indexCompletedEntries, isFreshInstallStamp, migrationLedgerSummary, trailingPartialCount, upgradeLedgerApplies } from '../../../core/migration-ledger.ts';
+import { MIGRATION_OPT_OUT_KEYS, declinedMigrations, hostWorkOwedFix, hostWorkOwedForVersion, hostWorkOwedMessage, indexCompletedEntries, isFreshInstallStamp, migrationLedgerSummary, trailingPartialCount, upgradeLedgerApplies } from '../../../core/migration-ledger.ts';
import { VERSION } from '../../../version.ts';
import { pendingFreshInstallCheck } from './pending-fresh-install.ts';
import { autopilotInstallPendingCheck } from './autopilot-install-pending.ts';
@@ -153,10 +153,13 @@ async function runMinionsMigration(ctx: DoctorContext): Promise<Check[]> {
// version pending again, as statusForVersion and get_health read it: it
// is reported below as not run yet, never as still wedged.
const history = indexCompletedEntries(completed);
+ // A wedge whose latest row records apply_migrations_pending still owes host work: never superseded, reported as a warning.
+ const owed = [...history.keys()].filter(v => !declined.has(v) && compareVersions(v, VERSION) <= 0 && hostWorkOwedForVersion(v, history)).sort(compareVersions);
const stuck = Array.from(byVersion.entries())
.filter(([v, s]) => {
if (!s.partial || s.complete) return false;
if (history.get(v)?.at(-1)?.status === 'retry') return false;
+ if (owed.includes(v)) return false;
// Forward-progress override: if any version >= v has completed, the
// partial is stale. compareVersions returns 1 when first arg is newer.
const supersededBy = completedVersions.find(cv => compareVersions(cv, v) >= 0);
@@ -238,6 +241,8 @@ async function runMinionsMigration(ctx: DoctorContext): Promise<Check[]> {
}
const hostWork = migrationHostWorkCheck(history, declined, engine ? await hostWorkMigrations(engine).catch(() => new Set<string>()) : new Set<string>());
if (hostWork) checks.push(hostWork);
+ if (owed.length > 0) checks.push({ name: 'minions_migration', status: 'warn', message: hostWorkOwedMessage(owed), fix: hostWorkOwedFix(owed),
+ details: { code: 'migration_host_work_owed', wedged_host_work: owed } });
const skippedInstall = autopilotInstallPendingCheck();
if (skippedInstall) checks.push(skippedInstall);
// Note: the "no preferences.json but schema is v7+" case is detected
diff --git a/src/commands/doctor/report-remote.ts b/src/commands/doctor/report-remote.ts
index 4b244398a8..e601002024 100644
--- a/src/commands/doctor/report-remote.ts
+++ b/src/commands/doctor/report-remote.ts
@@ -13,7 +13,7 @@ import type { BrainEngine } from '../../core/engine.ts';
import { LATEST_VERSION } from '../../core/migrate.ts';
import { loadConfig } from '../../core/config.ts';
import { loadCompletedMigrations } from '../../core/preferences.ts';
-import { indexCompletedEntries, isFreshInstallStamp, migrationLedgerSummary, trailingPartialCount } from '../../core/migration-ledger.ts';
+import { hostWorkOwedForVersion, hostWorkOwedMessage, indexCompletedEntries, isFreshInstallStamp, migrationLedgerSummary, trailingPartialCount } from '../../core/migration-ledger.ts';
import { VERSION } from '../../version.ts';
import { pendingFreshInstallCheck } from './checks/pending-fresh-install.ts';
import { compareVersions } from '../migrations/index.ts';
@@ -234,10 +234,12 @@ export async function doctorReportRemote(
const completedVersions = Array.from(byVersion.entries()).filter(([, s]) => s.ran).map(([v]) => v);
// A trailing retry marker makes the version pending (same rule as the local doctor and get_health).
const history = indexCompletedEntries(completed);
+ const owed = [...history.keys()].filter(v => compareVersions(v, VERSION) <= 0 && hostWorkOwedForVersion(v, history)).sort(compareVersions);
const stuck = Array.from(byVersion.entries())
.filter(([v, s]) => {
if (!s.partial || s.complete) return false;
if (history.get(v)?.at(-1)?.status === 'retry') return false;
+ if (owed.includes(v)) return false;
const supersededBy = completedVersions.find(cv => compareVersions(cv, v) >= 0);
return supersededBy === undefined;
})
@@ -274,6 +276,8 @@ export async function doctorReportRemote(
const setup = pendingFreshInstallCheck();
if (setup) checks.push(setup);
}
+ if (owed.length > 0) checks.push({ name: 'minions_migration', status: 'warn', message: `On the brain host: ${hostWorkOwedMessage(owed)}`,
+ details: { code: 'migration_host_work_owed', wedged_host_work: owed } });
} catch {
// Best-effort. A broken JSONL on the brain server should not stop the
// remote doctor.
diff --git a/src/commands/upgrade.ts b/src/commands/upgrade.ts
index 8b8ca703df..b57ba5336e 100644
--- a/src/commands/upgrade.ts
+++ b/src/commands/upgrade.ts
@@ -8,7 +8,7 @@ import { opError, type OperationError } from '../core/ops/contract.ts';
import { writeCliError } from '../cli/cli-error.ts';
import { VERSION } from '../version.ts';
import type { InteractiveProbe } from '../core/interaction.ts';
-import { forwardProgressPast, migrationLedgerSummary, wedgedLedgerHint } from '../core/migration-ledger.ts';
+import { forwardProgressPast, hostWorkOwedForVersion, hostWorkOwedMessage, indexCompletedEntries, migrationLedgerSummary, wedgedLedgerHint } from '../core/migration-ledger.ts';
import { loadCompletedMigrations } from '../core/preferences.ts';
import { MIGRATIONS_RUNNING_EXIT_CODE, readMigrationLockHolder } from '../core/migration-orchestration-lock.ts';
import { loadConfig, gbrainPath } from '../core/config.ts';
@@ -523,7 +523,11 @@ export async function describeMigrationFailure(error: unknown, newVersion: strin
+ 'Wait for it to finish; `gbrain doctor` shows migration progress.';
}
const ledger = newVersion ? loadCompletedMigrations() : [];
- const wedged = newVersion ? migrationLedgerSummary(newVersion).wedged.filter(v => forwardProgressPast(v, ledger) === null) : [];
+ const history = indexCompletedEntries(ledger);
+ const summary = newVersion ? migrationLedgerSummary(newVersion).wedged : [];
+ const owed = summary.filter(v => hostWorkOwedForVersion(v, history));
+ const wedged = summary.filter(v => !owed.includes(v) && forwardProgressPast(v, ledger) === null);
+ if (owed.length > 0 && wedged.length === 0) return `Migrations: wedged. ${hostWorkOwedMessage(owed)}`;
if (wedged.length > 0) {
return `Migrations: wedged (${wedged.map(v => `v${v}`).join(', ')} ended partial 3+ times in a row). Fix the cause recorded in ${wedgedLedgerHint()}, then run: `
+ `${wedged.map(v => `gbrain apply-migrations --force-retry ${v}`).join(' && ')} && gbrain apply-migrations --yes`;
diff --git a/src/core/migration-ledger.ts b/src/core/migration-ledger.ts
index 6d337f07c2..cb65b3bb16 100644
--- a/src/core/migration-ledger.ts
+++ b/src/core/migration-ledger.ts
@@ -13,6 +13,7 @@
* test/migration-ledger.test.ts so it cannot drift silently.
*/
+import type { Action } from './agent-output.ts';
import type { BrainEngine } from './engine.ts';
import { loadCompletedMigrations, preferencesPaths, type CompletedMigrationEntry } from './preferences.ts';
import { schemaVersionAtConnect } from './migrate-pending.ts';
@@ -135,6 +136,41 @@ export function hostWorkInspectArgv(version: string): string[] {
return ['gbrain', 'apply-migrations', '--migration', version, '--dry-run', '--json'];
}
+/** The one migration whose wedge can owe host work: v0.53.0, `SHARED_SKILLS_HOST_WORK_VERSION` (not imported: its module pulls in the shared-skills graph). */
+const HOST_WORK_OWED_VERSION = '0.53.0';
+
+/**
+ * v0.53.0 wedged with its latest row recording `apply_migrations_pending`: it stopped on a conflict in one or more sources
+ * that a person has to settle, and that work is still owed. A newer migration never supersedes it, and apply-migrations
+ * never runs it again by itself, so it stays until a person resolves the conflict and clears the wedge. Other migrations
+ * (v0.11.0, v0.14.0) set the same flag on ordinary partial runs, so their wedges keep the forward-progress rule.
+ */
+export function hostWorkOwedForVersion(version: string, byVersion: Map<string, CompletedMigrationEntry[]>): boolean {
+ return version === HOST_WORK_OWED_VERSION && statusForVersion(version, byVersion) === 'wedged' && byVersion.get(version)?.at(-1)?.apply_migrations_pending === true;
+}
+
+function forceRetryArgv(version: string): string[] { return ['gbrain', 'apply-migrations', '--force-retry', version]; }
+
+/** The one description of a wedged version with host work owed, shared by every surface that reports it. */
+export function hostWorkOwedMessage(versions: readonly string[]): string {
+ const first = versions[0]!;
+ return `Migration ${versions.map(v => `v${v}`).join(', ')} stopped ${MAX_CONSECUTIVE_PARTIALS}+ runs in a row on a conflict in one or more sources that a person has to resolve, and that work is still owed; `
+ + `a newer migration does not replace it, and it does not clear without a person. See each source's conflict (read-only): ${hostWorkInspectArgv(first).join(' ')}. `
+ + `Once the user has reviewed each one, clear the wedge with ${forceRetryArgv(first).join(' ')}, then run the command that output names for each source, or gbrain apply-migrations --yes.`;
+}
+
+/** The fix for a wedged version with host work owed: the read-only per-source inspect first, then ask the user to clear it once the conflict is resolved. */
+export function hostWorkOwedFix(versions: readonly string[]): Action {
+ const first = versions[0]!;
+ const verify = { argv: ['gbrain', 'apply-migrations', '--list', '--json'] };
+ return { argv: hostWorkInspectArgv(first), consent: [], actor: 'agent', requires_exclusive: false, verify,
+ why: `Lists each source's remaining step and its conflict for v${first}, with the command that settles it; changes nothing.`,
+ then: { ask_user: true, argv: forceRetryArgv(first), consent: [], actor: 'agent', requires_exclusive: true, verify,
+ why: `Writes a 'retry' marker so v${first} can run again, including the per-source command the inspect step names; a run that does not settle a conflict stops on it again.`,
+ user_message: `gbrain migration v${first} is stuck on a conflict in one or more sources that needs your decision; the inspect step lists each source and what conflicts there. `
+ + `It will not clear by itself. Once you have reviewed each conflict, clear it with ${forceRetryArgv(first).join(' ')}, then run the command the inspect step names for each source, or gbrain apply-migrations --yes. Clear it now?` } };
+}
+
export function trailingPartialCount(entries: readonly CompletedMigrationEntry[]): number {
let consecutive = 0;
for (let i = entries.length - 1; i >= 0 && entries[i].status === 'partial'; i--) consecutive++;
diff --git a/test/migration-host-work-owed.test.ts b/test/migration-host-work-owed.test.ts
new file mode 100644
index 0000000000..dcac1e9510
--- /dev/null
+++ b/test/migration-host-work-owed.test.ts
@@ -0,0 +1,248 @@
+/**
+ * A wedged migration whose latest row records apply_migrations_pending still owes host work (for v0.53.0, a conflict
+ * in one or more sources). A newer migration completing after it does not make it history, and nothing clears it
+ * without a person. Every surface says so with the same read-only next step (the per-source inspect), offers the
+ * forced retry only after the conflict is resolved, and leaves the migration and every exit status as they were.
+ *
+ * One throwaway PGLite brain drives the CLI (HOME / GBRAIN_HOME per subprocess; process.env is never mutated); the
+ * notice, both doctors and the upgrade message run in-process against a ledger-only home.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { tmpdir } from 'node:os';
+import { fileURLToPath } from 'node:url';
+import { withEnv } from './helpers/with-env.ts';
+import { pendingUpgradeMigrationsNotice } from '../src/cli/migrations-notice.ts';
+import { buildChecks } from '../src/commands/doctor.ts';
+import { doctorReportRemote } from '../src/commands/doctor/report-remote.ts';
+import { describeMigrationFailure } from '../src/commands/upgrade.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { getDbUrlSource } from '../src/core/config.ts';
+import { VERSION } from '../src/version.ts';
+import { SHARED_SKILLS_HOST_WORK_VERSION } from '../src/core/shared-skills/migration-host-work.ts';
+
+const REPO = fileURLToPath(new URL('..', import.meta.url)).replace(/\/$/, '');
+const OWED = '0.53.0';
+const INSPECT = ['gbrain', 'apply-migrations', '--migration', OWED, '--dry-run', '--json'];
+const RETRY = ['gbrain', 'apply-migrations', '--force-retry', OWED];
+const owedRow = (n: number) => ({ ts: `2026-10-0${n}T00:00:00.000Z`, version: OWED, status: 'partial', apply_migrations_pending: true,
+ phases: [{ name: 'content-checkpoints', status: 'complete', detail: 'Mechanical inventory is durable; publication and enrollment are NOT complete.' }] });
+/** v0.11.0 records apply_migrations_pending on every partial run that left host TODOs: an ordinary wedge, never a source conflict. */
+const LEGACY = '0.11.0';
+const legacyRow = (n: number) => ({ ts: `2026-10-0${n}T00:00:00.000Z`, version: LEGACY, status: 'partial', apply_migrations_pending: true,
+ phases: [{ name: 'record', status: 'complete', detail: 'status=partial (ledger write in runner)' }] });
+let home: string;
+let base: string[];
+let engine: PGLiteEngine;
+
+async function cli(args: string[]): Promise<{ exitCode: number; stdout: string; stderr: string }> {
+ const proc = Bun.spawn(['bun', 'run', `${REPO}/src/cli.ts`, ...args], {
+ cwd: REPO, env: { ...process.env, HOME: home, GBRAIN_HOME: home, GBRAIN_NO_AUTOPILOT_INSTALL: '1' }, stdout: 'pipe', stderr: 'pipe',
+ });
+ const [stdout, stderr, exitCode] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text(), proc.exited]);
+ return { exitCode, stdout, stderr };
+}
+
+const ledgerPath = () => join(home, '.gbrain', 'migrations', 'completed.jsonl');
+const ledger = () => readFileSync(ledgerPath(), 'utf8');
+/**
+ * The fresh brain's ledger with v0.53.0 replaced by three partial runs that recorded host work. With
+ * `superseded`, a real v0.60.31 run is forward progress past it; without, every v0.60.31 row is an init fresh-install
+ * stamp, which is not forward progress, so nothing has run past the wedge.
+ */
+function owe(superseded: boolean): void {
+ const kept = base.filter(line => JSON.parse(line).version !== OWED).map(line => {
+ const row = JSON.parse(line);
+ return !superseded && row.version === '0.60.31' ? JSON.stringify({ ...row, fresh_install: true }) : line;
+ });
+ const ran = superseded ? [JSON.stringify({ version: '0.60.31', status: 'complete' })] : [];
+ writeFileSync(ledgerPath(), [...kept, ...ran, ...[1, 2, 3].map(n => JSON.stringify(owedRow(n)))].join('\n') + '\n');
+}
+
+beforeAll(async () => {
+ home = realpathSync(mkdtempSync(join(tmpdir(), 'gbrain-host-work-owed-')));
+ expect((await cli(['init', '--pglite', '--no-embedding', '--json'])).exitCode).toBe(0);
+ expect((await cli(['apply-migrations', '--yes', '--no-autopilot-install'])).exitCode).toBe(0);
+ base = ledger().split('\n').filter(line => line.trim());
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ expect(base.some(line => JSON.parse(line).version === '0.60.31' && JSON.parse(line).status === 'complete')).toBe(true);
+}, 240_000);
+
+afterAll(async () => {
+ await engine?.disconnect();
+ try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ }
+});
+
+/** The read-only inspect first, then the ask: rendered (`next`) on the wire, stored (`ask_user`) on an in-process doctor check. */
+function expectOwedFix(fix: any): void {
+ if ('next' in fix) expect(fix).toMatchObject({ next: 'run', argv: INSPECT, then: { next: 'ask_user', argv: RETRY } });
+ else {
+ expect(fix).toMatchObject({ argv: INSPECT, then: { ask_user: true, argv: RETRY } });
+ expect(fix.ask_user).toBeUndefined();
+ }
+ expect(fix.then.user_message).toContain('conflict');
+ expect(fix.then.user_message).toContain('will not clear by itself');
+}
+
+describe('apply-migrations with a wedged migration that still owes host work', () => {
+ test('--list never calls it history, names the inspect step first and does not say up to date', async () => {
+ owe(true);
+ const list = await cli(['apply-migrations', '--list']);
+ expect(list.exitCode).toBe(0);
+ const out = list.stdout + list.stderr;
+ expect(out).not.toContain('history, not outstanding work');
+ expect(out).not.toContain('All migrations up to date');
+ expect(out).toMatch(new RegExp(`wedged\\s+${OWED.replace(/\./g, '\\.')}`));
+ expect(out).toContain(INSPECT.join(' '));
+ expect(out.indexOf(INSPECT.join(' '))).toBeLessThan(out.indexOf(RETRY.join(' ')));
+ expect(out).toContain('does not clear without a person');
+ }, 120_000);
+
+ test('--list --json lists it with the read-only inspect as its fix, never as superseded', async () => {
+ owe(true);
+ const list = await cli(['apply-migrations', '--list', '--json']);
+ expect(list.exitCode).toBe(0);
+ const doc = JSON.parse(list.stdout);
+ expect(doc.plan.wedged_superseded).not.toContain(OWED);
+ expect(doc.plan.wedged_host_work).toEqual([OWED]);
+ expect(doc.wedged_host_work).toEqual([{ version: OWED, fix: { argv: INSPECT, command: INSPECT.join(' ') } }]);
+ expect(doc.migrations.find((m: any) => m.version === OWED).status).toBe('wedged');
+ }, 120_000);
+
+ test('the inspect step it names runs on the wedged version and changes nothing', async () => {
+ owe(true);
+ const before = ledger();
+ const inspect = await cli(INSPECT.slice(1));
+ expect(inspect.exitCode).toBe(0);
+ expect(JSON.parse(inspect.stdout).previews.map((p: any) => p.version)).toEqual([OWED]);
+ expect(ledger()).toBe(before);
+ }, 120_000);
+
+ test('--yes leaves it untouched and still exits 0 when a newer migration ran past it', async () => {
+ owe(true);
+ const before = ledger();
+ const run = await cli(['apply-migrations', '--yes', '--no-autopilot-install', '--json']);
+ expect(run.exitCode).toBe(0);
+ const doc = JSON.parse(run.stdout);
+ expect(doc.status).toBe('up_to_date');
+ expect(doc.wedged_host_work).toEqual([OWED]);
+ expect(run.stderr).toContain(INSPECT.join(' '));
+ expect(ledger()).toBe(before);
+ }, 120_000);
+
+ test('--yes still fails migrations_pending (exit 1) when nothing ran past it, with the inspect step as its fix', async () => {
+ owe(false);
+ const before = ledger();
+ const run = await cli(['apply-migrations', '--yes', '--no-autopilot-install', '--json']);
+ expect(run.exitCode).toBe(1);
+ const envelope = JSON.parse(run.stdout);
+ expect(envelope).toMatchObject({ code: 'migrations_pending', reason: 'wedged' });
+ expectOwedFix(envelope.fix);
+ expect(ledger()).toBe(before);
+ }, 120_000);
+
+ test('a superseded wedge of another migration that recorded host TODOs stays history, never owed host work', async () => {
+ writeFileSync(ledgerPath(), [...base.filter(line => JSON.parse(line).version !== LEGACY), JSON.stringify({ version: '0.60.31', status: 'complete' }),
+ ...[1, 2, 3].map(n => JSON.stringify(legacyRow(n)))].join('\n') + '\n');
+ const list = await cli(['apply-migrations', '--list', '--json']);
+ expect(list.exitCode).toBe(0);
+ const doc = JSON.parse(list.stdout);
+ expect(doc.plan.wedged_superseded).toEqual([LEGACY]);
+ expect(doc.plan.wedged_host_work).toEqual([]);
+ expect(doc.wedged_host_work).toBeUndefined();
+ }, 120_000);
+
+ test('post-upgrade leaves it untouched and keeps its exit status', async () => {
+ owe(true);
+ const before = ledger();
+ expect((await cli(['post-upgrade', '--no-autopilot-install'])).exitCode).toBe(0);
+ expect(ledger()).toBe(before);
+ owe(false);
+ const stuck = ledger();
+ expect((await cli(['post-upgrade', '--no-autopilot-install'])).exitCode).toBe(1);
+ expect(ledger()).toBe(stuck);
+ }, 240_000);
+
+ test('a named run of it refuses with the steps that clear it, and runs by name once the wedge is cleared', async () => {
+ owe(true);
+ const before = ledger();
+ const named = ['apply-migrations', '--migration', OWED, '--yes', '--no-autopilot-install', '--json'];
+ const refused = await cli(named);
+ expect(refused.exitCode).toBe(2);
+ const envelope = JSON.parse(refused.stdout);
+ expect(envelope.code).toBe('invalid_params');
+ expect(envelope.message).toContain(`v${OWED} is wedged`);
+ expect(envelope.suggestion).toContain(RETRY.join(' '));
+ expectOwedFix(envelope.fix);
+ expect(ledger()).toBe(before);
+ expect((await cli(RETRY.slice(1))).exitCode).toBe(0);
+ const run = await cli(named);
+ expect(run.exitCode).toBe(0);
+ expect(JSON.parse(run.stdout).code).toBeUndefined();
+ }, 240_000);
+});
+
+describe('the other surfaces say the same thing', () => {
+ function ledgerHome(superseded: boolean, wedge: object[] = [1, 2, 3].map(owedRow)): string {
+ const dir = mkdtempSync(join(tmpdir(), 'gbrain-host-work-owed-ledger-'));
+ mkdirSync(join(dir, '.gbrain', 'migrations'), { recursive: true });
+ writeFileSync(join(dir, '.gbrain', 'config.json'), JSON.stringify({ engine: 'pglite' }));
+ const rows = [...wedge, { version: '0.60.31', status: 'complete', ...(superseded ? {} : { fresh_install: true }) }];
+ writeFileSync(join(dir, '.gbrain', 'migrations', 'completed.jsonl'), rows.map(r => JSON.stringify(r)).join('\n') + '\n');
+ return dir;
+ }
+ const scoped = <T>(superseded: boolean, fn: () => Promise<T>, wedge?: object[]) => {
+ const dir = ledgerHome(superseded, wedge);
+ return withEnv({ HOME: dir, GBRAIN_HOME: dir, DATABASE_URL: undefined, GBRAIN_DATABASE_URL: undefined }, fn)
+ .finally(() => rmSync(dir, { recursive: true, force: true }));
+ };
+
+ test('the connect and serve notice leads with the inspect step, not the forced retry', async () => {
+ const notice = await scoped(true, () => pendingUpgradeMigrationsNotice({ getConfig: async () => null } as never, 'serve', VERSION));
+ expect(notice?.code).toBe('migrations_pending');
+ expect(notice!.fix).toMatchObject({ argv: INSPECT, then: { ask_user: true, argv: RETRY } });
+ expect(notice!.fix!.ask_user).toBeUndefined();
+ expect(notice!.why).toContain('does not clear without a person');
+ });
+
+ for (const superseded of [true, false]) {
+ test(`doctor warns with the inspect step (${superseded ? 'a newer migration ran past it' : 'nothing ran past it'})`, async () => {
+ const rows = await scoped(superseded, async () => (await buildChecks(null, ['--fast', '--json'], getDbUrlSource())).filter(c => c.name === 'minions_migration'));
+ expect(rows.some(c => c.status === 'fail')).toBe(false);
+ const row = rows.find(c => (c.details as any)?.code === 'migration_host_work_owed');
+ expect(row?.status).toBe('warn');
+ expect(row!.message).toContain(INSPECT.join(' '));
+ expectOwedFix(row!.fix);
+ });
+ }
+
+ test('the remote doctor reports it as a warning on the brain host', async () => {
+ const rows = await scoped(true, async () => (await doctorReportRemote(engine, { remote: true })).checks.filter(c => c.name === 'minions_migration'));
+ expect(rows.some(c => c.status === 'fail')).toBe(false);
+ const row = rows.find(c => (c.details as any)?.code === 'migration_host_work_owed');
+ expect(row?.status).toBe('warn');
+ expect(row!.message).toContain(INSPECT.join(' '));
+ }, 120_000);
+
+ test('a superseded wedge of another migration that recorded host TODOs is not reported as owed host work', async () => {
+ expect(SHARED_SKILLS_HOST_WORK_VERSION).toBe(OWED);
+ const legacy = [1, 2, 3].map(legacyRow);
+ const rows = await scoped(true, async () => (await buildChecks(null, ['--fast', '--json'], getDbUrlSource())).filter(c => c.name === 'minions_migration'), legacy);
+ expect(rows.some(c => (c.details as any)?.code === 'migration_host_work_owed')).toBe(false);
+ const remote = await scoped(true, async () => (await doctorReportRemote(engine, { remote: true })).checks.filter(c => c.name === 'minions_migration'), legacy);
+ expect(remote.some(c => (c.details as any)?.code === 'migration_host_work_owed')).toBe(false);
+ const notice = await scoped(true, () => pendingUpgradeMigrationsNotice({ getConfig: async () => null } as never, 'serve', VERSION), legacy);
+ expect(notice?.why ?? '').not.toContain('does not clear without a person');
+ const line = await scoped(true, () => describeMigrationFailure(new Error('exit 1'), VERSION), legacy);
+ expect(line).not.toContain(`--migration ${LEGACY}`);
+ }, 120_000);
+
+ test('upgrade\'s failure line names the inspect step, never a forced retry first', async () => {
+ const line = await scoped(true, () => describeMigrationFailure(new Error('exit 1'), VERSION));
+ expect(line).toContain(INSPECT.join(' '));
+ expect(line.indexOf(INSPECT.join(' '))).toBeLessThan(line.indexOf(RETRY.join(' ')));
+ });
+});