Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Merge master into capy/gbra251-ci-wave-2
garrytane9869f184456
diff --cc docs/architecture/key-files/core-persistence-continued.mdindex 18c8dea2b5,f3322aa9a1..8f25bdc923--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@@ -4,7 -4,7 +4,7 @@@Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.The #6188 fence check and fence repair live in [fence repair](core-fence-repair.md). Managed connector ingestion, connector item holds and the no-op kernel (checkpoint validation and the fault seams: [request indexes and fault seams](core-persistence-request-indexes.md)). Also the persistence verb types and errors, database-write and manual links, write attribution and loop-fact retirement; the managed purge of deleted pages is under [memory purge](memory-purge.md).- - `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` (every consumer topology scan) also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host, from an instant probe of the refresh lock) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured fix as the v1 `fix` RenderedAction when it has one, else the legacy command string; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), Postgres via `test/e2e/worktree-refresh-*postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.-- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it a member is `deferred`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` / `sync_deferred` when a member blocks or is deferred. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.++- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it a member is `deferred`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` / `sync_deferred` when a member blocks or is deferred. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-*postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The guard reads its state as the writing role and fails closed: a hidden or missing `persistence_brain` or `git_canonical_brain` row, or a source or page row hidden by row-level security, raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`), and the counter refuses a recount under RLS on `sources` (`test/writer-fence-fail-closed-7636.test.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).diff --cc test/worktree-refresh-settle-matrix-7679.test.tsindex 28408622f3,0000000000..6a680ed8e1mode 100644,000000..100644--- a/test/worktree-refresh-settle-matrix-7679.test.ts+++ b/test/worktree-refresh-settle-matrix-7679.test.ts@@@ -1,130 -1,0 +1,144 @@@+/**+ * #7679 state matrix 7c of `gbrain sources refresh`: an interrupted refresh row of each kind, settled by the owner scan, a+ * plain refresh and --resume. Split from persistence-worktree-refresh.test.ts with 7d+ * (worktree-refresh-unmoved-matrix-7679.test.ts) so each Postgres file fits run-e2e.sh's per-file cap (#8025). Postgres runs+ * through test/e2e/worktree-refresh-settle-matrix-postgres.test.ts.+ */+import { afterAll, beforeAll, expect, test } from 'bun:test';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';+import { refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';+import { withCoordinatedWrite } from '../src/core/persistence/context.ts';+import { TEST_WRITE_ATTRIBUTION } from './helpers/write-attribution.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { testBackends } from './helpers/test-backends.ts';+import { commitFile, git, page, withRefreshFixture } from './helpers/worktree-refresh-fixture.ts';++const backends = testBackends();+const engines: BrainEngine[] = [];+let closePostgres: (() => Promise<void>) | undefined;++beforeAll(async () => {+ if (backends.includes('pglite')) {+ const engine = new PGLiteEngine();+ await engine.connect({}); await engine.initSchema(); engines.push(engine);+ }+ if (backends.includes('postgres')) {+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);+ engines.push(pg.engine); closePostgres = pg.close;+ }+}, 120_000);+afterAll(async () => {+ for (const engine of engines) { await disposePersistenceConsumer(engine); if (engine.kind === 'pglite') await engine.disconnect(); }+ await closePostgres?.();+});++const refusedWith = async (promise: Promise<unknown>, code: string) => {+ const error = await promise.then(() => null, e => e as { code?: string; suggestion?: string; message?: string; docs?: string });+ expect(error, `expected refusal ${code}`).not.toBeNull();+ expect(error!.code).toBe(code);+ return error!;+};++test('7c. #7679 state matrix: an interrupted refresh of each kind, settled by the owner scan, a plain refresh and --resume', async () => {+ // Value: protects=each kind of interrupted refresh row leaves the row section 1.5 names when the owner scan, a plain refresh (adoption) or+ // --resume (resumeHeld) settles it; fails_when=a kind falls through to a sibling's path (an unverified HEAD adopted as the target, a syncing+ // row with an unrelated HEAD completed or refreshed over, a dead drain left standing); why_new=the #7679 tests cover syncing orphans with HEAD at+ // the target only; seam=none; no_claim=a real process death (the crash is a hook throw; the refresh lock is released as on exit).+ type Kind = 'draining' | 'fenced_old' | 'fenced_target' | 'fenced_unrelated' | 'merged_target' | 'merged_unrelated' | 'syncing_at_target' | 'syncing_descendant'+ | 'syncing_unrelated' | 'syncing_unrelated_synced';+ type Action = 'owner_scan' | 'plain' | 'resume';+ const kinds: Kind[] = ['draining', 'fenced_old', 'fenced_target', 'fenced_unrelated', 'merged_target', 'merged_unrelated', 'syncing_at_target', 'syncing_descendant',+ 'syncing_unrelated', 'syncing_unrelated_synced'];+ const stateOf = (kind: Kind) => kind.split('_')[0] as 'draining' | 'fenced' | 'merged' | 'syncing';+ const crashAt = { draining: 'drained', fenced: 'fenced', merged: 'fenced', syncing: 'syncing' } as const;+ // The owner scan's row and moved count; a merged row's bookkeeping leaves it syncing, and a syncing row with an unrelated HEAD waits for --resume.+ const scanned: Record<Kind, [string, number]> = { draining: ['aborted', 1], fenced_old: ['aborted', 1], fenced_target: ['syncing', 1], fenced_unrelated: ['recovery_required', 1],+ merged_target: ['syncing', 1], merged_unrelated: ['recovery_required', 1], syncing_at_target: ['completed', 1], syncing_descendant: ['syncing', 0], syncing_unrelated: ['syncing', 0], syncing_unrelated_synced: ['syncing', 0] };+ const cells = kinds.flatMap(kind => (['owner_scan', 'plain', 'resume'] as Action[]).map(action => [kind, action] as [Kind, Action]));- for (const engine of engines) for (const [kind, action] of cells) await withRefreshFixture(engine, async f => {++ // Test cost (#7782): a cell that leaves the checkout settled (no unfinished refresh, a HEAD the upstream contains, every member synced to it) hands++ // its fixture to the next cell instead of building a brain per cell, so each chain is at most one settled cell and then one unsettled cell.++ const endsSettled = ([kind, action]: [Kind, Action]) => !kind.includes('_unrelated') && kind !== 'syncing_descendant'++ && !(action === 'owner_scan' && (kind === 'fenced_target' || kind === 'merged_target'));++ const settledCells = cells.filter(endsSettled), unsettledCells = cells.filter(cell => !endsSettled(cell));++ const chains = [...unsettledCells.map((cell, i) => settledCells[i] ? [settledCells[i], cell] : [cell]), ...settledCells.slice(unsettledCells.length).map(cell => [cell])];++ expect(chains.flat()).toHaveLength(cells.length);++ for (const engine of engines) for (const chain of chains) await withRefreshFixture(engine, async f => { for (const [kind, action] of chain) {+ const label = `${kind} x ${action}`;- const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));++ const earlier = await f.refreshRows();++ if (earlier.length) {++ expect(earlier.map(r => r.state).filter(state => state !== 'aborted' && state !== 'completed'), `${label}: handoff`).toEqual([]);++ expect(git(f.root, 'merge-base', 'HEAD', 'origin/main'), `${label}: handoff`).toBe(f.head());++ expect([await f.lastCommit(f.alpha), await f.lastCommit(f.beta)], `${label}: handoff`).toEqual([f.head(), f.head()]);++ }++ const cellRows = async () => (await f.refreshRows()).filter(r => !earlier.some(e => e.id === r.id));++ const target = f.push('alpha/two.md', page('Alpha two', `Upstream for ${label}.`));+ const old = f.head();+ await expect(refreshWorktree(f.engine, f.alpha, { hooks: { boundary: point => { if (point === crashAt[stateOf(kind)]) throw new Error(`simulated crash at ${point}`); } } }))+ .rejects.toThrow('simulated crash');+ await disposePersistenceConsumer(f.engine);- const [orphan] = await f.refreshRows();++ const [orphan] = await cellRows();+ // The resident consumer's owner scan may already have settled it; restore what the crash left.+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state=$2,completed_at=NULL WHERE id=$1::uuid`, [orphan.id, stateOf(kind)]);+ if (kind === 'fenced_target' || kind === 'merged_target') git(f.root, 'merge', '-q', '--ff-only', target);+ if (kind === 'syncing_descendant') commitFile(f.root, 'beta/local.md', page('Local', 'Committed on top of the target.'));+ if (kind.includes('_unrelated')) { git(f.root, 'reset', '-q', '--hard', old); commitFile(f.root, 'beta/unrelated.md', page('Unrelated', 'Not the upstream commit.')); }+ if (kind === 'syncing_at_target' || kind === 'syncing_unrelated_synced') for (const member of [f.alpha, f.beta]) await f.engine.transaction(tx => withCoordinatedWrite(tx, [member],+ () => tx.executeRaw('UPDATE sources SET last_commit=$2 WHERE id=$1', [member, target]), TEST_WRITE_ATTRIBUTION));+ const head = f.head();- const rows = async () => (await f.refreshRows()).map(r => r.state);++ const rows = async () => (await cellRows()).map(r => r.state);+ if (kind === 'draining') {+ // Ship audit (Step 7). Value: protects=writer status tells an orphaned draining refresh's writers to --resume it, never to retry (only a live+ // drain finishes on its own); fails_when=the draining branch of next stops checking liveness (master's wording); why_new=test 10 reads a live+ // drain only; seam=none; no_claim=a real process death (the crash is a hook throw; the refresh lock is released as on exit).+ const [status] = (await runPersistenceAdministration(f.engine, 'writer_status', { source_id: f.alpha }) as { worktree_refreshes: Array<Record<string, any>> }).worktree_refreshes;+ expect(status, label).toMatchObject({ refresh_id: orphan.id, state: 'draining', liveness: 'orphaned', auto_recovery: 'next_refresh',+ fix: { argv: ['gbrain', 'sources', 'refresh', f.alpha, '--resume'] }, next: `gbrain sources refresh ${f.alpha} --resume` });+ }+ if (action === 'owner_scan') {+ const [state, moved] = scanned[kind];+ expect(await resumeWorktreeRefreshes(f.engine), label).toBe(moved);+ expect(await rows(), label).toEqual([state]);+ expect(f.head(), label).toBe(head);- return;++ continue;+ }+ const run = refreshWorktree(f.engine, f.alpha, action === 'resume' ? { resume: true } : {});+ if (kind.includes('_unrelated')) {+ await refusedWith(run, 'refresh_recovery_required');+ expect(await rows(), label).toEqual(['recovery_required']);+ expect(f.head(), label).toBe(head);- return;++ continue;+ }+ if (kind === 'syncing_descendant' && action === 'plain') {+ // The adoption completes the orphan at the descendant HEAD; the refresh itself then refuses the unpushed local commit.+ await refusedWith(run, 'refresh_diverged');+ expect(await rows(), label).toEqual(['completed']);+ expect(f.head(), label).toBe(head);+ expect(await f.lastCommit(f.alpha), label).toBe(head);+ expect(await f.lastCommit(f.beta), label).toBe(head);- return;++ continue;+ }+ const result = await run;+ const rolledBack = kind === 'draining' || kind === 'fenced_old';+ if (action === 'resume') {+ expect(result, label).toMatchObject({ status: rolledBack ? 'abandoned' : 'completed', refresh_id: orphan.id });+ expect(await rows(), label).toEqual([rolledBack ? 'aborted' : 'completed']);+ expect(f.head(), label).toBe(head);- return;++ continue;+ }+ if (rolledBack) {+ expect(result, label).toMatchObject({ status: 'completed', target_head: target, recovered: { refresh_id: orphan.id, state: 'aborted', status: 'abandoned' } });+ expect(await rows(), label).toEqual(['aborted', 'completed']);+ } else {+ expect(result, label).toMatchObject({ status: 'already_current', recovered: { refresh_id: orphan.id, state: 'completed', status: 'completed', previous_holder: { pid: process.pid } } });+ expect(await rows(), label).toEqual(['completed']);+ }+ const synced = kind === 'syncing_descendant' ? head : target;+ expect(f.head(), label).toBe(synced);+ expect(await f.lastCommit(f.alpha), label).toBe(synced);+ expect(await f.lastCommit(f.beta), label).toBe(synced);- });++ } });+}, 900_000);+diff --cc test/worktree-refresh-unmoved-matrix-7679.test.tsindex 5d33474aad,0000000000..f0138433fbmode 100644,000000..100644--- a/test/worktree-refresh-unmoved-matrix-7679.test.ts+++ b/test/worktree-refresh-unmoved-matrix-7679.test.ts@@@ -1,123 -1,0 +1,138 @@@+/**+ * #7679 state matrix 7d of `gbrain sources refresh`: an interrupted refresh that the unattended paths cannot verify, or+ * that a live or remote holder owns, is never moved. Split from persistence-worktree-refresh.test.ts with 7c+ * (worktree-refresh-settle-matrix-7679.test.ts) so each Postgres file fits run-e2e.sh's per-file cap (#8025). Postgres runs+ * through test/e2e/worktree-refresh-unmoved-matrix-postgres.test.ts.+ */+import { afterAll, beforeAll, expect, test } from 'bun:test';+import { randomUUID } from 'node:crypto';+import { readFileSync, writeFileSync } from 'node:fs';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { tryAcquireNativeLock } from '../src/core/persistence/native-lock.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';+import { refreshLockPath, refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';+import { withCoordinatedWrite } from '../src/core/persistence/context.ts';+import { TEST_WRITE_ATTRIBUTION } from './helpers/write-attribution.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { testBackends } from './helpers/test-backends.ts';+import { git, page, withRefreshFixture } from './helpers/worktree-refresh-fixture.ts';++const backends = testBackends();+const engines: BrainEngine[] = [];+let closePostgres: (() => Promise<void>) | undefined;++beforeAll(async () => {+ if (backends.includes('pglite')) {+ const engine = new PGLiteEngine();+ await engine.connect({}); await engine.initSchema(); engines.push(engine);+ }+ if (backends.includes('postgres')) {+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);+ engines.push(pg.engine); closePostgres = pg.close;+ }+}, 120_000);+afterAll(async () => {+ for (const engine of engines) { await disposePersistenceConsumer(engine); if (engine.kind === 'pglite') await engine.disconnect(); }+ await closePostgres?.();+});++const refusedWith = async (promise: Promise<unknown>, code: string) => {+ const error = await promise.then(() => null, e => e as { code?: string; suggestion?: string; message?: string; docs?: string });+ expect(error, `expected refusal ${code}`).not.toBeNull();+ expect(error!.code).toBe(code);+ return error!;+};++test('7d. #7679 state matrix: an interrupted refresh that the unattended paths cannot verify, or that a live or remote holder owns, is never moved', async () => {+ // Value: protects=a plain refresh leaves an orphan untouched (state, holder, previous_holder, HEAD) when git cannot read HEAD (refusing+ // refresh_in_progress), when publication/topology recovery is pending or the worktree is not active (refusing refresh_recovery_required, the+ // blocker that must clear first), when a live process holds the refresh lock, or when its holder is on another host; the owner scan aborts a dead+ // drain whatever HEAD reads and leaves every other kind on an unreadable HEAD; fails_when=adoption runs recovery on a null HEAD, ahead of the+ // recovery gate, against a live lock or another host's holder, or the scan escalates an unreadable HEAD; why_new=7c covers readable HEADs with+ // free locks and no pending recovery only; seam=.git/HEAD overwritten with garbage, a recovering topology change row, the worktree state column,+ // the refresh lock held in this process, a foreign holder host id; no_claim=the race where HEAD becomes unreadable between adoption's two reads.+ type Kind = 'draining' | 'fenced_old' | 'fenced_target' | 'merged_target' | 'syncing_lagging' | 'syncing_at_target' | 'recovery_required';+ type Action = 'plain_head_unreadable' | 'scan_head_unreadable' | 'plain_recovery_pending' | 'plain_worktree_inactive' | 'live_holder' | 'other_host_holder';+ const kinds: Kind[] = ['draining', 'fenced_old', 'fenced_target', 'merged_target', 'syncing_lagging', 'syncing_at_target', 'recovery_required'];- const actions: Action[] = ['plain_head_unreadable', 'scan_head_unreadable', 'plain_recovery_pending', 'plain_worktree_inactive', 'live_holder', 'other_host_holder'];++ // The scan aborts a dead drain, so scan_head_unreadable runs last.++ const actions: Action[] = ['plain_head_unreadable', 'plain_recovery_pending', 'plain_worktree_inactive', 'live_holder', 'other_host_holder', 'scan_head_unreadable'];+ const stateOf = (kind: Kind) => kind === 'recovery_required' ? 'recovery_required' : kind.split('_')[0];+ const crashAt = (kind: Kind) => kind === 'draining' ? 'drained' : kind.startsWith('syncing') ? 'syncing' : 'fenced';- const cells = kinds.flatMap(kind => actions.map(action => [kind, action] as [Kind, Action]));- for (const engine of engines) for (const [kind, action] of cells) await withRefreshFixture(engine, async f => {- const label = `${kind} x ${action}`;++ // Test cost (#7782): each kind builds one brain and one interrupted refresh, and every action runs on it in turn. No action but the last may move++ // the refresh, so each one undoes its seam and hands the next action the journal, HEAD and worktree it started with.++ for (const engine of engines) for (const kind of kinds) await withRefreshFixture(engine, async f => {+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));+ await expect(refreshWorktree(f.engine, f.alpha, { hooks: { boundary: point => { if (point === crashAt(kind)) throw new Error(`simulated crash at ${point}`); } } }))+ .rejects.toThrow('simulated crash');+ await disposePersistenceConsumer(f.engine);+ const [orphan] = await f.refreshRows();+ const state = stateOf(kind);+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state=$2,completed_at=NULL WHERE id=$1::uuid`, [orphan.id, state]);+ if (kind === 'fenced_target' || kind === 'merged_target') git(f.root, 'merge', '-q', '--ff-only', target);+ if (kind === 'syncing_at_target') for (const member of [f.alpha, f.beta]) await f.engine.transaction(tx => withCoordinatedWrite(tx, [member],+ () => tx.executeRaw('UPDATE sources SET last_commit=$2 WHERE id=$1', [member, target]), TEST_WRITE_ATTRIBUTION));+ const head = f.head();+ const settled = async () => (await f.refreshRows()).map(r => [r.state, (r.outcome.holder as { pid?: number } | undefined)?.pid, r.outcome.previous_holder]);+ const before = await settled();+ const plainCode = kind === 'recovery_required' ? 'refresh_recovery_required' : 'refresh_in_progress';+ const writerStatus = async () => (await runPersistenceAdministration(f.engine, 'writer_status', { source_id: f.alpha }) as { worktree_refreshes: Array<Record<string, any>> }).worktree_refreshes[0];- if (action === 'plain_recovery_pending' || action === 'plain_worktree_inactive') {- if (action === 'plain_recovery_pending') await f.engine.executeRaw(`INSERT INTO persistence_topology_changes(principal_id,request_id,digest,operation,source_id,worktree_ids,state,recovery)- VALUES($1::uuid,$2::uuid,'example','reclone',$3,ARRAY[$4::uuid],'recovering',$5::text::jsonb)`,- [randomUUID(), randomUUID(), f.alpha, f.worktreeId, JSON.stringify({ worktreeId: f.worktreeId })]);- else await f.engine.executeRaw(`UPDATE persistence_worktrees SET state='recovering' WHERE id=$1::uuid`, [f.worktreeId]);- await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_recovery_required');- // Status does not promise that the next refresh finishes an orphan the recovery gate holds back.- if (kind !== 'recovery_required') expect(await writerStatus(), label).toMatchObject({ liveness: 'orphaned', auto_recovery: 'none',- fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', f.alpha, '--json'] } });- } else if (action === 'live_holder') {- const lock = await tryAcquireNativeLock(refreshLockPath(f.worktreeId));- expect(lock, label).not.toBeNull();- try {- expect(await resumeWorktreeRefreshes(f.engine), label).toBe(0);++ const journal = async () => JSON.stringify(await f.refreshRows());++ const worktree = async () => (await f.engine.executeRaw<{ state: string; topology: number }>(`SELECT state,(SELECT count(*)::int FROM persistence_topology_changes++ WHERE $1::uuid = ANY(worktree_ids)) AS topology FROM persistence_worktrees WHERE id=$1::uuid`, [f.worktreeId]))[0];++ const pristine = { journal: await journal(), worktree: await worktree() };++ for (const action of actions) {++ const label = `${kind} x ${action}`;++ if (action === 'plain_recovery_pending' || action === 'plain_worktree_inactive') {++ const topologyRequest = randomUUID();++ if (action === 'plain_recovery_pending') await f.engine.executeRaw(`INSERT INTO persistence_topology_changes(principal_id,request_id,digest,operation,source_id,worktree_ids,state,recovery)++ VALUES($1::uuid,$2::uuid,'example','reclone',$3,ARRAY[$4::uuid],'recovering',$5::text::jsonb)`,++ [randomUUID(), topologyRequest, f.alpha, f.worktreeId, JSON.stringify({ worktreeId: f.worktreeId })]);++ else await f.engine.executeRaw(`UPDATE persistence_worktrees SET state='recovering' WHERE id=$1::uuid`, [f.worktreeId]);++ await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_recovery_required');++ // Status does not promise that the next refresh finishes an orphan the recovery gate holds back.++ if (kind !== 'recovery_required') expect(await writerStatus(), label).toMatchObject({ liveness: 'orphaned', auto_recovery: 'none',++ fix: { argv: ['gbrain', 'sources', 'writer', 'status', '--source', f.alpha, '--json'] } });++ if (action === 'plain_recovery_pending') await f.engine.executeRaw('DELETE FROM persistence_topology_changes WHERE request_id=$1::uuid', [topologyRequest]);++ else await f.engine.executeRaw('UPDATE persistence_worktrees SET state=$2 WHERE id=$1::uuid', [f.worktreeId, pristine.worktree.state]);++ } else if (action === 'live_holder') {++ const lock = await tryAcquireNativeLock(refreshLockPath(f.worktreeId));++ expect(lock, label).not.toBeNull();++ try {++ expect(await resumeWorktreeRefreshes(f.engine), label).toBe(0);++ await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);++ await refusedWith(refreshWorktree(f.engine, f.alpha, { resume: true }), 'refresh_in_progress');++ expect(await writerStatus(), label).toMatchObject({ liveness: 'live', auto_recovery: 'none' });++ } finally { await lock!.release(); }++ } else if (action === 'other_host_holder') {++ const [{ outcome }] = await f.refreshRows();++ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,host_id}',to_jsonb($2::text)) WHERE id=$1::uuid`, [orphan.id, randomUUID()]);+ await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);- await refusedWith(refreshWorktree(f.engine, f.alpha, { resume: true }), 'refresh_in_progress');- expect(await writerStatus(), label).toMatchObject({ liveness: 'live', auto_recovery: 'none' });- } finally { await lock!.release(); }- } else if (action === 'other_host_holder') {- await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,host_id}',to_jsonb($2::text)) WHERE id=$1::uuid`, [orphan.id, randomUUID()]);- await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);- if (kind !== 'recovery_required') expect(await writerStatus(), label).toMatchObject({ liveness: 'unknown', auto_recovery: 'unknown', fix: { actor: 'host_admin' },- next: `gbrain sources writer status --source ${f.alpha} --json` });- } else {- const headFile = join(f.root, '.git', 'HEAD');- const saved = readFileSync(headFile, 'utf8');- writeFileSync(headFile, 'not a ref\n');- try {- if (action === 'plain_head_unreadable') {- const refused = await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);- if (kind !== 'recovery_required') expect(refused.suggestion, label).toContain(`gbrain sources writer status --source ${f.alpha} --json`);- } else {- // The scan aborts a dead drain (it does not depend on HEAD) and leaves every other kind for a scan that can read HEAD.- expect(await resumeWorktreeRefreshes(f.engine), label).toBe(kind === 'draining' ? 1 : 0);- expect((await f.refreshRows()).map(r => r.state), label).toEqual([kind === 'draining' ? 'aborted' : state]);- }- } finally { writeFileSync(headFile, saved); }- if (action === 'scan_head_unreadable') { expect(f.head(), label).toBe(head); return; }++ if (kind !== 'recovery_required') expect(await writerStatus(), label).toMatchObject({ liveness: 'unknown', auto_recovery: 'unknown', fix: { actor: 'host_admin' },++ next: `gbrain sources writer status --source ${f.alpha} --json` });++ await f.engine.executeRaw('UPDATE persistence_worktree_refreshes SET outcome=$2::text::jsonb WHERE id=$1::uuid', [orphan.id, JSON.stringify(outcome)]);++ } else {++ const headFile = join(f.root, '.git', 'HEAD');++ const saved = readFileSync(headFile, 'utf8');++ writeFileSync(headFile, 'not a ref\n');++ try {++ if (action === 'plain_head_unreadable') {++ const refused = await refusedWith(refreshWorktree(f.engine, f.alpha), plainCode);++ if (kind !== 'recovery_required') expect(refused.suggestion, label).toContain(`gbrain sources writer status --source ${f.alpha} --json`);++ } else {++ // The scan aborts a dead drain (it does not depend on HEAD) and leaves every other kind for a scan that can read HEAD.++ expect(await resumeWorktreeRefreshes(f.engine), label).toBe(kind === 'draining' ? 1 : 0);++ expect((await f.refreshRows()).map(r => r.state), label).toEqual([kind === 'draining' ? 'aborted' : state]);++ }++ } finally { writeFileSync(headFile, saved); }++ if (action === 'scan_head_unreadable') { expect(f.head(), label).toBe(head); continue; }++ }++ expect(await settled(), label).toEqual(before);++ expect(f.head(), label).toBe(head);++ await disposePersistenceConsumer(f.engine);++ expect({ journal: await journal(), worktree: await worktree() }, `${label}: handoff`).toEqual(pristine);+ }- expect(await settled(), label).toEqual(before);- expect(f.head(), label).toBe(head);+ });+}, 900_000);