Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(refresh): an orphaned worktree refresh recovers on its own; status names its holder (#7679)
garrytan57571c3071ce
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.mdindex c689681b49..22a3a275ce 100644--- a/docs/architecture/key-files/core-persistence-continued.md+++ b/docs/architecture/key-files/core-persistence-continued.md@@ -4,7 +4,7 @@Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.The #6188 fence check and fence repair live in [fence repair](core-fence-repair.md). Managed connector ingestion, connector item holds and the no-op kernel (checkpoint validation and the fault seams: [request indexes and fault seams](core-persistence-request-indexes.md)). Also the persistence verb types and errors, database-write and manual links, write attribution and loop-fact retirement; the managed purge of deleted pages is under [memory purge](memory-purge.md).-- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies`, so on owner start) touches only interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts`; while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.+- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies`, so on owner start) touches only interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` (every consumer topology scan) also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host, from an instant probe of the refresh lock) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts`; while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).diff --git a/docs/architecture/key-files/managed-sync-bookkeeping.md b/docs/architecture/key-files/managed-sync-bookkeeping.mdindex f630c4760a..eb26985400 100644--- a/docs/architecture/key-files/managed-sync-bookkeeping.md+++ b/docs/architecture/key-files/managed-sync-bookkeeping.md@@ -48,7 +48,7 @@ run, and the supersession of a checkpoint whose cursor another run replaced. The- `src/commands/doctor/checks/managed-sync-movement.ts` + `src/core/persistence/sync-movement.ts` — #6317 (B4, I2): the movement reader and its doctor check. `readManagedSyncMovement` snapshots, per active managed source, the newest `completed_at` of committed `managed_sync_*` receipts in the current source incarnation (the watermark: one probe of the partial index `persistence_requests_sync_watermark`, v223; a count would shrink under compaction), the head claim's `step_since`, the `managed-sync` cursor (`index/total`, `held`) and `sources.last_sync_at`/`last_commit`, and whether a drain or full consumer is live on the owner host. `data_moving` is false when an unfinished cursor has had neither a committed receipt nor a head step advance for longer than `persistence.preparation_ceiling_ms` while a consumer is live; with none live the state is `parked` (informational). `sources status --json` and `writer status --json` print `data_moving`, `not_moving_since` and `movement_state` beside `sync_running`, and `sources status` prints the writer-status command beside `data_moving: false`; doctor `managed_sync_not_moving` (warn, scored for `not_moving` only) uses the same rule and command as its fix, and fails on a parked cursor whose admitted queue has no consumer past the ceiling (#6552; fix: the resume sync); `serve` prints one `[gbrain notice]` when a source flips. A failed backlog read is `unknown`, never `nothing_pending` (doctor warns). `/health` stays liveness-only.- `src/commands/sources-writer-movement.ts` — #6317 (I2): `gbrain sources writer movement [<source>] [--wait <dur>] [--warn-only] [--json]`, the deploy gate `gbrain upgrade` and `post-upgrade` end by printing (bare, so it inherits the default window; neither runs it, because neither restarts the resident processes). The printed step is `MOVEMENT_SUPERVISOR_STEP` from the import-free leaf `src/commands/movement-step.ts` (re-exported here), so upgrade loads it before the swap (#6604). Same source selection as `writer status` (`persistence-admin.ts`). It snapshots through `sync-movement.ts`, waits `max(300 s, preparation budget + 60 s)` (printed as the effective budget; `--wait <dur>` takes sync's duration syntax) and judges each source: `moved` (a new committed publication; a cursor advanced only by holds or a `last_sync_at` touched by an up-to-date check is not movement) and `current` exit 0; `held` exits 0 with a warning and the hold kind's route (`gbrain repair fences --source <id>`; `preparation_stalled` → writer status, then `gbrain sources retry-held <id>`); `within_allowance` (the head's step advanced inside the window) exits 0 with `retry_after_ms`; `not_moving` exits 1 with `managed_sync_not_moving` (`reason: movement_check`, `fix.argv` the writer-status command, `fix.verify` this command); `unknown` (backlog unreadable) exits 1 with `movement_unknown`; other read failures keep their code. `--warn-only` prints the same envelopes and exits 0. Flags are regenerated into `src/core/cli-flag-registry.generated.ts` with `bun run build:flag-registry`.- `src/core/serve-head-watcher.ts` + `src/core/persistence/sync-freshness.ts` + `src/commands/doctor/checks/managed-sync-latency.ts` — #6858 ask 6: on-commit sync and its latency metric. `startServeHeadWatcher` (two call sites: `installResidentTickers` in `src/commands/serve.ts`, stopped before `shutdownDelegatedSync`, and `runServeHttp` in `src/commands/serve-http.ts`, beside `startMovementWatch` and stopped on the same shutdown path) re-reads `sync.on_commit` every 30 s; while on it polls `git rev-parse HEAD` in each managed source checkout this host owns (`GBRAIN_SERVE_HEAD_POLL_MS`, default 3 s; an `fs.watch` on the git dir only nudges the poll), debounces 3 s (`GBRAIN_SERVE_HEAD_DEBOUNCE_MS`) and starts one managed sync through the unchanged `startDelegatedSync` with this host's `cli` registration. It never starts beside a running delegated sync or a held drain lease (the move stays pending and fires once after), skips a HEAD `sources.last_commit` already has, resumes a resumable stop up to `MAX_RESUMES`, and logs each distinct error once. `readSyncFreshness` samples committed `managed_sync_import` receipts of the last 24 h (not first syncs or working-tree entries): commit time is the git committer time of `intent.target`, searchable is `completed_at`, embedded is the page's newest `content_chunks.embedded_at` once every chunk has one, sampled only from a receipt no later committed write to the same page supersedes and only at or after its `completed_at`; p50/p95 in one aggregate over the `persistence_requests_sync_watermark` predicate. #7604: the supersede check reads the newest committed sequence per (source, slug) once, over the rows after the oldest sample (a materialized CTE), so the read stays linear in the samples after a killed pass leaves tens of thousands of receipts (a per-receipt subquery is quadratic and reaches the 300 s statement timeout). `gbrain sync status` (the `freshness` block) and `gbrain status` print it; doctor `managed_sync_latency` grades p95 against `sync.freshness_target_seconds` (default 300; fail over max(1 h, target)) only with `sync.on_commit` on or a target set. Tests: `test/serve-head-watcher.test.ts`, `test/serve-head-watcher-managed.serial.test.ts`, `test/serve-http-head-watcher.serial.test.ts`, `test/sync-freshness-6858.test.ts`. #7189: each `SourceFreshness` carries `behind` (`readGitBehind`, injectable `BehindReader`: `rev-list --count <last_commit>..HEAD` and the oldest of those commits' committer age; null without a checkout or a synced commit), printed by `formatSyncFreshness`.-- `src/core/persistence/sync-status.ts` (status coherence) — GBRA-150: `state: blocked` means the source cannot advance (the drain outcome's meaning); a file hold that waits on a person while the source keeps syncing reads `synced` with reason `held_needs_human`, `needs_human` and `next` unchanged (`test/sync-status-blocked-definition.test.ts`); `detail` adds `landing` / `landing_read` (`landing-gaps.ts`). #7189: a `last_error` whose cursor failure row (`op_checkpoints` `managed-sync-failure`) is older than the source's newest committed managed-sync receipt carries `superseded` and sets neither `needs_human`, `state` nor `next` (`liveError`); `needs_human_from` names the hold or the error. #7243 (GBRA-133 PR 3a): every status carries `last_commit` (`sources.last_commit`) and `live` (the drain-lease holder still renewing it, from `readDrainLease`, read only when the source read's `gbrain_cycle_locks` probe finds a row, with `job_id` for a Minions sync job and `stalled` from `holderStalled`; #7596: a holder that stopped renewing the lease (`readDrainLease`, `drainLeaseLive`) is not `live`, and an open pass with no live holder reads `pass_interrupted` (`passStatusState`) with an `interrupted` block (pass, last row write, the dead holder), which `sync-blocked.ts` also lists for doctor `managed_sync_blocked`; test `test/sync-pass-interrupted-7596.test.ts`; `live_job_id` also takes a job holder of a pass); `supersedingAttempt` marks a `last_error` `superseded` (`by`, `age_seconds`) when evidence newer than the failure row passed its phase (`resume` < `discovery` < `freeze`/`admission` < `receipt`/`checkpoint`; evidence: the pass row's `started_at` (pin), a level read (`timings_ms` set or an `up_to_date` idle tick), an uncommitted managed import or delete admitted later, a later commit). The CLI (`sync/operator.ts` `holdsListing`) bounds the holds: `holds_summary` (`topCounts` by code and by state: `needs_human`, `actionable`, `held`) over every hold, then `PLAIN_STATUS_HOLDS` (20) rows needs-human first, or with `--detail` a page of the shared listing contract (`bounded-listing.ts`, kind `sync_status_holds:<source>:<state>`, `--limit`/`--cursor`/`--all`, `--hold-state`), `holds_page` only when a page left rows out or took a cursor, the rows last. Test: `test/sync-status-truthful-7243.test.ts` (both engines).+- `src/core/persistence/sync-status.ts` (status coherence) — GBRA-150: `state: blocked` means the source cannot advance (the drain outcome's meaning); a file hold that waits on a person while the source keeps syncing reads `synced` with reason `held_needs_human`, `needs_human` and `next` unchanged (`test/sync-status-blocked-definition.test.ts`); `detail` adds `landing` / `landing_read` (`landing-gaps.ts`). #7189: a `last_error` whose cursor failure row (`op_checkpoints` `managed-sync-failure`) is older than the source's newest committed managed-sync receipt carries `superseded` and sets neither `needs_human`, `state` nor `next` (`liveError`); `needs_human_from` names the hold or the error. #7243 (GBRA-133 PR 3a): every status carries `last_commit` (`sources.last_commit`) and `live` (the drain-lease holder still renewing it, from `readDrainLease`, read only when the source read's `gbrain_cycle_locks` probe finds a row, with `job_id` for a Minions sync job and `stalled` from `holderStalled`; #7596: a holder that stopped renewing the lease (`readDrainLease`, `drainLeaseLive`) is not `live`, and an open pass with no live holder reads `pass_interrupted` (`passStatusState`) with an `interrupted` block (pass, last row write, the dead holder), which `sync-blocked.ts` also lists for doctor `managed_sync_blocked`; test `test/sync-pass-interrupted-7596.test.ts`; `live_job_id` also takes a job holder of a pass); `supersedingAttempt` marks a `last_error` `superseded` (`by`, `age_seconds`) when evidence newer than the failure row passed its phase (`resume` < `discovery` < `freeze`/`admission` < `receipt`/`checkpoint`; evidence: the pass row's `started_at` (pin), a level read (`timings_ms` set or an `up_to_date` idle tick), an uncommitted managed import or delete admitted later, a later commit). The CLI (`sync/operator.ts` `holdsListing`) bounds the holds: `holds_summary` (`topCounts` by code and by state: `needs_human`, `actionable`, `held`) over every hold, then `PLAIN_STATUS_HOLDS` (20) rows needs-human first, or with `--detail` a page of the shared listing contract (`bounded-listing.ts`, kind `sync_status_holds:<source>:<state>`, `--limit`/`--cursor`/`--all`, `--hold-state`), `holds_page` only when a page left rows out or took a cursor, the rows last. Test: `test/sync-status-truthful-7243.test.ts` (both engines). #7679: `worktree_refresh` is the active refresh of the source's worktree (read in the source statement, so no extra statement) with its holder, `liveness`/`orphaned` and fix; an orphaned one's `--resume` fix becomes `next` unless a person's decision comes first.- `src/core/persistence/resolve-paths.ts` — GBRA-150 `resolveSourcePaths`: what gbrain did with each file path of a source, by its own rules, so a landing check stops copying them. Per path: the sync slug (`resolveSlugForPath`), presence in the last synced commit and at HEAD (Git object reads), the skip reason (`unsyncableReason`, plus `typed_frontmatter` for a skipped metafile), the open hold, the live page by origin (`source_path`) and by slug (`slug_owner` when another file owns it, so a case-folded twin never reads as this file's page), a differing frontmatter `slug:`, chunk and embedding counts (`listEmbeddingStateRows` with its `slugs` filter), and one of `RESOLVE_PATH_STATES` with `searchable` (true / 'keyword' / false), `why` and an existing-command `fix`. Bounded: at most 200 paths, indexed lookups only, nothing written. Tests: `test/sources-resolve-path.test.ts`.- `src/core/page-searchability.ts` — #7190 `readPageSearchability`: each live page's state, `searchable: true | 'keyword' | false`, chunks, `unembedded` (visible chunks with a NULL active-column vector) and `embedding_hold`, folded from `listEmbeddingStateRows` and the budget-hold read. Shared by resolve-path and `get_write_request`'s `applied`; `gbrain pages status` (#7181) can compose it with its Git view.- `src/core/persistence/applied-watermark.ts` — #7190 `readAppliedWatermark` (sources status, MCP `sources_status`, `sync --json`) and the cheap `readWriteWatermark` (`get_write_request`): `indexed_through` is `sources.last_commit` (only a finished pass moves it), `embedded_through` reads the oldest page with a NULL vector (held pages count), `committed_through_request` is one below the oldest unsettled request of the incarnation, `pending` adds requests, files behind HEAD, held files and withheld deletes, `writer_heartbeat` is the drain-lease holder and its last progress (redacted for remote callers).diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 0b17ccaedf..445466d277 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -519,7 +519,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.| <a id="refresh_diverged"></a>`refresh_diverged` | `refresh_diverged` | The checkout has commits its upstream lacks, so a fast-forward is impossible. Nothing changed. | Ask the user, then publish the local commits merged with the upstream: `gbrain git commit-push --push-only --keep-checkout --path <checkout>` merges in memory, pushes the merge, refuses on a conflicting file and never moves the checkout (the refusal's fix fills in the real path and counts the local commits). Then retry `gbrain sources refresh <source>`, which fast-forwards to the merge. `gbrain sources reclone <source>` replaces the checkout instead and drops the local commits. || <a id="refresh_dirty"></a>`refresh_dirty` | `refresh_dirty` | Uncommitted or untracked files overlap the incoming upstream changes (the first 20 are named), or `git merge --ff-only` refused for the same reason. HEAD did not move. Uncommitted files outside the incoming changes do not refuse; they are kept and listed as `preserved_uncommitted`. | Commit or discard the named paths, then retry. Check `gbrain sources writer status <source> --json` first: a pending Git effect may be about to commit a gbrain-published file. || <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |-| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active, or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. | `gbrain sources refresh <source> --resume`, then retry the source command with the same request ID. |+| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its pid and host), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. || <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. || <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. || <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |@@ -529,7 +529,14 @@ A refresh is recorded in `persistence_worktree_refreshes` and survives a crash.When the owner restarts it finishes the deterministic steps on its own: aninterrupted drain or a fence with HEAD unchanged is abandoned, and a mergethat already reached the verified upstream commit moves on to syncing.-`gbrain sources refresh <source> --resume` runs the remaining syncs. Doctor+`gbrain sources refresh <source> --resume` runs the remaining syncs. A+refresh whose process died (for example a container replaced mid-refresh)+does not wedge the checkout: the owner completes a syncing one once every+source has synced to its target, and the next plain `gbrain sources refresh`+on the owner host finishes any other one the same way `--resume` does, then+refreshes. A refresh whose process is still running is never taken over.+`gbrain sources writer status` and `gbrain sync status` show who holds each+refresh and whether it is orphaned. Doctorreports `worktree_refresh_stuck` for a refresh active longer than 15 minutes,with the resume command.diff --git a/src/core/persistence/sync-status.ts b/src/core/persistence/sync-status.tsindex 3800be9766..38cf5f07dd 100644--- a/src/core/persistence/sync-status.ts+++ b/src/core/persistence/sync-status.ts@@ -51,6 +51,7 @@ import { listIdentityDowngrades, withIdentityRemedies } from '../sync-identity-dimport { GIT_PUSH_CODES, readGitPushHolds, type GitPushHold } from './git-push-holds.ts';import { holderStalled, readDrainLease, type DrainHolder } from './sync-lease.ts';import { syncLockId } from '../db-lock.ts';+import { ACTIVE_REFRESH_STATES_SQL } from './worktree-refresh-schema.ts';export interface SyncStatusHold extends SyncFaultVerdict {code: string;@@ -185,6 +186,12 @@ export interface SyncStatus {* falls back to the first one's fix.*/fence_holds: import('../generations/fence.ts').FenceHoldEntry[];+ /**+ * #7679: the active `gbrain sources refresh` of the source's worktree, or null: its state, the process recorded as its+ * holder, whether that process still holds the refresh lock (`orphaned`; null when the lock is on another host), and the+ * fix. An orphaned refresh's fix names `--resume`, and `next` takes it unless a person's decision comes first.+ */+ worktree_refresh: ({ refresh_id: string; state: string; updated_at: string } & Awaited<ReturnType<typeof import('./worktree-refresh.ts').refreshHolderReport>>) | null;next: Action | null;/*** #6712 PR 2 (L5): the source's state in the sync outcome vocabulary (`synced` | `resumable` | `blocked`) and why@@ -223,6 +230,7 @@ export interface SyncStatus {landing_read: SyncStatusRead;}export type SyncStatusState = 'synced' | 'resumable' | 'blocked';+type RefreshProbeRow = { id: string; worktree_id: string; state: import('./worktree-refresh-schema.ts').WorktreeRefreshState; source_ids: string[]; outcome: Record<string, unknown>; updated_at: string; owner_host_id: string | null };/** Why the source is in its state; one closed list. */export const SYNC_STATUS_REASONS = ['nothing_pending', 'held_by_design', 'held_needs_human', 'cursor_unfinished', 'live_job', 'last_error', 'holds_actionable', 'recovery_pending', 'request_pending','status_unknown', 'needs_human', 'head_blocked', 'recovery_blocked',@@ -327,12 +335,14 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, optsconst admissionSql = syncAdmissionSettingSql(4);const [source] = await engine.executeRaw<{ incarnation: string; last_commit: string | null; last_sync_at: Date | string | null; drain_lock: boolean; pass_value: PassRow['value'] | null; pass_updated_at: Date | string | null; withheld: DeletesWithheld | null;admission_stored: string | null; admission_row_value: string | null; admission_set_at: string | null; admission_pinned: boolean | null; admission_destructive: boolean | null;- writeback_failed: string | null }>(+ writeback_failed: string | null; refresh: RefreshProbeRow | string | null }>(`SELECT s.incarnation::text AS incarnation, s.last_commit, s.last_sync_at, EXISTS (SELECT 1 FROM gbrain_cycle_locks l WHERE l.id=$7) AS drain_lock,p.completed_keys->0 AS pass_value, p.updated_at AS pass_updated_at, w.completed_keys->0 AS withheld, ${admissionSql.columns}, ${syncAdmissionEffectSql(7, 's')},(SELECT r.request_id::text FROM persistence_effects e JOIN persistence_requests r ON r.id=e.request_id WHERE e.kind='git' AND e.state='failed'AND e.request_id IN (SELECT id::uuid FROM jsonb_array_elements_text(CASE WHEN jsonb_typeof(p.completed_keys->0->'obligations'->'converged'->'writeback')='array'- THEN p.completed_keys->0->'obligations'->'converged'->'writeback' ELSE '[]'::jsonb END) AS w(id)) ORDER BY e.id LIMIT 1) AS writeback_failed FROM sources s+ THEN p.completed_keys->0->'obligations'->'converged'->'writeback' ELSE '[]'::jsonb END) AS w(id)) ORDER BY e.id LIMIT 1) AS writeback_failed,+ (SELECT row_to_json(f) FROM (SELECT r.id,r.worktree_id,r.state,r.source_ids,r.outcome,r.updated_at,w.owner_host_id FROM persistence_worktree_refreshes r+ LEFT JOIN persistence_worktrees w ON w.id=r.worktree_id WHERE s.id=ANY(r.source_ids) AND r.state IN ${ACTIVE_REFRESH_STATES_SQL} LIMIT 1) f) AS refresh FROM sources sLEFT JOIN op_checkpoints p ON p.op=$2 AND p.fingerprint=s.id||':'||s.incarnation::textLEFT JOIN op_checkpoints w ON w.op=$3 AND w.fingerprint=s.id||':'||s.incarnation::text${admissionSql.joins}@@ -407,6 +417,10 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, optsconst passState = passRow ? passStatusState(passRow, withheld, source.writeback_failed, Date.now(), interrupted) : null;const resume = !header && passRow ? passResumeArgs(sourceId, passRow) : managedSyncResumeArgs({ sourceId, processingOptions: header?.processingOptions, syncOptions: header?.syncOptions });const human = holds.find(hold => hold.needs_human) ?? (liveError?.needs_human ? liveError : undefined);+ const refreshRow = typeof source.refresh === 'string' ? JSON.parse(source.refresh) as RefreshProbeRow : source.refresh;+ const worktreeRefresh = refreshRow ? { refresh_id: refreshRow.id, state: refreshRow.state, updated_at: new Date(refreshRow.updated_at).toISOString(),+ ...await (await import('./worktree-refresh.ts')).refreshHolderReport(refreshRow) } : null;+ const refreshFix = worktreeRefresh?.liveness !== 'live' ? worktreeRefresh?.fix ?? null : null;// #6643: a knock-on or a superseded run is not a failure of its page: the resume re-freezes the entry.const selfHealing = liveError?.detail === 'knock_on' || liveError?.detail === 'run_replaced';const actionable = holds.filter(hold => !hold.needs_human && hold.safe_actions[0] !== 'none');@@ -420,6 +434,7 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, opts: human? { ...('fix' in human ? human.fix : { argv: ['gbrain', 'sources', 'status', sourceId, '--json'], consent: [], actor: 'host_admin' as const, requires_exclusive: false, why: human.human_reason ?? '' }),user_message: `Sync of source ${sourceId} needs a decision: ${'slug' in human && human.slug ? `page ${human.slug} (${('path' in human ? human.path : undefined) ?? ''})` : `code ${human.code}`}: ${human.human_reason ?? 'see the hold'}` }+ : refreshFix ? refreshFix: actionable.length? { argv: ['gbrain', 'sync', 'unblock', '--source', sourceId, '--apply', '--json'], consent: [], actor: 'agent', requires_exclusive: false, verify,why: `${actionable.length} held file(s) have a safe action: unblock schedules each one whose condition holds (a committed edit, a stalled preparation) for a re-screen`@@ -452,7 +467,7 @@ export async function readSyncStatus(engine: BrainEngine, sourceId: string, optscommitted_last_10m: Number(committed?.n ?? 0), ...estimate, live_job_id: liveJob ?? holderJob, holds, identity_downgrades: await withIdentityRemedies(engine, await listIdentityDowngrades(engine, [sourceId])), last_error: lastError, needs_human: !!human, ...(human ? { human_reason: human.human_reason } : {}),needs_human_from: !human ? null : 'path' in human && 'held_since' in human ? { kind: 'hold', path: human.path, slug: human.slug, code: human.code } : { kind: 'last_error', code: human.code },resume_argv: resume, ...blockers, cursors: detail ? await liveCursors(engine, sourceId, source.incarnation, selected?.key ?? null, failures) : [],- terminal_failures: terminal, connector_last_error: connectorLastError, git_push_holds: gitPushHolds, fence_holds: fenceHolds, next: next ?? recovery_next ?? passNext ?? (stateReason === 'request_pending' && oldest.status === 'ok' && oldest.request ? { argv: ['gbrain', 'sync', ...resume], consent: [], actor: 'agent', requires_exclusive: false, verify,+ terminal_failures: terminal, connector_last_error: connectorLastError, git_push_holds: gitPushHolds, fence_holds: fenceHolds, worktree_refresh: worktreeRefresh, next: next ?? recovery_next ?? passNext ?? (stateReason === 'request_pending' && oldest.status === 'ok' && oldest.request ? { argv: ['gbrain', 'sync', ...resume], consent: [], actor: 'agent', requires_exclusive: false, verify,why: `Write request ${oldest.request.request_id} of this source is still ${oldest.request.state}${oldest.request.step ? ` (step ${oldest.request.step})` : ''}; a running owner finishes it on its own, and the next sync of this source drains it.` } satisfies Action: stateReason === 'status_unknown' ? { argv: ['gbrain', 'sync', 'status', '--source', sourceId, '--json'], consent: [], actor: 'agent', requires_exclusive: false, verify,why: `The oldest-blocker or recovery probe did not answer within ${SYNC_STATUS_PROBE_MS / 1000} s, so this status does not know the state (it is not reported synced). Rerun it after a short wait; gbrain doctor --json checks the database if it keeps timing out.` } satisfies Actiondiff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.tsindex 1b851f4e4f..bf37648beb 100644--- a/src/core/persistence/worktree-refresh.ts+++ b/src/core/persistence/worktree-refresh.ts@@ -16,8 +16,12 @@* A per-worktree refresh lock (`locks/refresh-<worktree>.lock`) is held by the* refreshing process for its whole run, so recovery never touches the row of a* live refresh: holding that native lock proves the previous process exited.+ * #7679: the row records its holder (`outcome.holder`: host, pid, process start),+ * so status can say who holds it, and a row whose lock is free is orphaned: the+ * next plain refresh finishes it through the `--resume` path before refreshing.*/import { mkdirSync, mkdtempSync, rmSync } from 'node:fs';+import { hostname } from 'node:os';import { join } from 'node:path';import type { BrainEngine } from '../engine.ts';import type { SyncResult } from '../../commands/sync.ts';@@ -45,7 +49,7 @@ export interface WorktreeRefreshRow {topology_generation: string | number; state: WorktreeRefreshState; old_head: string; target_head: string; upstream_ref: string;preserved_uncommitted: string[]; outcome: Record<string, unknown>; created_at: string | Date; updated_at: string | Date; completed_at: string | Date | null;}-export type RefreshBoundary = 'drained' | 'fenced' | 'merged';+export type RefreshBoundary = 'drained' | 'fenced' | 'merged' | 'syncing';export interface RefreshHooks { boundary?: (point: RefreshBoundary, row: WorktreeRefreshRow) => Promise<void> | void }export interface RefreshOptions {dryRun?: boolean; resume?: boolean; abandon?: boolean;@@ -62,7 +66,17 @@ export interface WorktreeRefreshResult {refresh_id: string | null; worktree_id: string; source_ids: string[]; old_head: string | null; target_head: string | null;upstream_ref: string | null; preserved_uncommitted: string[]; incoming_files?: number;synced: RefreshMemberSync[]; sync_blocked?: RefreshSyncBlocked[]; next?: string;-}+ /** #7679: an orphaned refresh this run finished (or rolled back) through the `--resume` path before its own refresh. */+ recovered?: { refresh_id: string; state: WorktreeRefreshState; status: WorktreeRefreshResult['status']; previous_holder: RefreshHolder | null };+}+/** #7679: the process that holds a refresh's lock, recorded on the row so status can name it. */+export interface RefreshHolder { host_id: string | null; hostname: string; pid: number; started_at: string }+/** `live`: a process holds the refresh lock; `orphaned`: none does; `unknown`: the lock is on another host, or unreadable. */+export type RefreshLiveness = 'live' | 'orphaned' | 'unknown';+const refreshHolder = (): RefreshHolder => ({ host_id: existingLocalHostId(), hostname: hostname(), pid: process.pid,+ started_at: new Date(Date.now() - process.uptime() * 1000).toISOString() });+const holderOf = (row: Pick<WorktreeRefreshRow, 'outcome'>): RefreshHolder | null => (row.outcome?.holder as RefreshHolder | undefined) ?? null;+const describeHolder = (holder: RefreshHolder | null) => holder ? `pid ${holder.pid} on ${holder.hostname} (host ${holder.host_id ?? 'unknown'}), started ${holder.started_at}` : 'a process that recorded no holder';const DRAIN_POLL_MS = 250;const MAX_MEMBER_SYNC_PASSES = 20;@@ -209,7 +223,7 @@ async function precheck(engine: BrainEngine, git: Git, sourceId: string, fetchTiassertRefreshOwner(binding, sourceId, 'move its checkout');const root = binding.local_path;const active = await readActiveRefresh(engine, binding.worktree_id);- if (active) throw refusal('refresh_in_progress', `Refresh ${active.id} of this worktree is ${active.state}; only one refresh per worktree runs at a time.`,+ if (active) throw refusal('refresh_in_progress', `Refresh ${active.id} of this worktree is ${active.state}, held by ${describeHolder(holderOf(active))}; only one refresh per worktree runs at a time.`,`gbrain sources refresh ${sourceId} --resume`, active.id);if (binding.state !== 'active' || await worktreeRecoveryPending(engine, binding)) throw refusal('refresh_recovery_required','Publication or topology recovery is pending on this worktree; the checkout must not move until it is recovered.',@@ -405,6 +419,13 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, optif (opts.resume || opts.abandon) return resumeOrAbandon(engine, sourceId, opts);const fetchTimeoutMs = await configuredMs(engine, opts.fetchTimeoutMs, 'GBRAIN_REFRESH_FETCH_TIMEOUT_MS', 'sources.refresh_fetch_timeout_ms', 120_000, 2 ** 31 - 1);const waitDrainMs = await configuredMs(engine, opts.waitDrainMs, 'GBRAIN_REFRESH_DRAIN_WAIT_MS', 'sources.refresh_drain_wait_ms', 60_000);+ const recovered = opts.dryRun ? null : await adoptOrphanedRefresh(engine, sourceId);+ if (recovered && recovered.status !== 'completed' && recovered.status !== 'abandoned') return recovered;+ const result = await refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs);+ return recovered?.recovered ? { ...result, recovered: recovered.recovered } : result;+}++async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshOptions, fetchTimeoutMs: number, waitDrainMs: number): Promise<WorktreeRefreshResult> {return withGit(async git => {const plan = await precheck(engine, git, sourceId, fetchTimeoutMs);const base = { refresh_id: null, worktree_id: plan.binding.worktree_id, source_ids: plan.members, old_head: plan.oldHead,@@ -428,7 +449,7 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt}return { ...base, status: blocked.length ? 'sync_blocked' : 'already_current', synced, ...(blocked.length ? { sync_blocked: blocked, next: blocked.map(b => b.resume).join(' ; ') } : {}) };}- const refreshLock = await tryAcquireNativeLock(refreshLockPath(plan.binding.worktree_id));+ const refreshLock = await acquireNativeLock(refreshLockPath(plan.binding.worktree_id), { timeoutMs: 1000 });if (!refreshLock) throw refusal('refresh_in_progress', 'Another process is refreshing this worktree right now.', `gbrain sources refresh ${sourceId} --resume`);try {const principal = await topologyPrincipal(engine);@@ -449,6 +470,7 @@ export async function refreshWorktree(engine: BrainEngine, sourceId: string, opt}await opts.hooks?.boundary?.('drained', row);const merged = await fenceAndMerge(engine, git, sourceId, plan, row, opts.hooks);+ await opts.hooks?.boundary?.('syncing', merged);const sync = await syncMembers(engine, git, merged, plan.root);return finished(await readRefresh(engine, row.id), sync, plan.preserved);} finally { await refreshLock.release(); }@@ -467,10 +489,10 @@ async function fenceDraining(engine: BrainEngine, sourceId: string, plan: Plan,throw refusal('refresh_source_changed', 'The worktree ownership or source membership changed after the refresh precheck; nothing moved.', `Retry gbrain sources refresh ${sourceId}`);}const [row] = await tx.executeRaw<WorktreeRefreshRow>(`INSERT INTO persistence_worktree_refreshes- (worktree_id,source_ids,principal_id,owner_epoch,topology_generation,state,old_head,target_head,upstream_ref,preserved_uncommitted)- VALUES($1::uuid,$2::text[],$3::uuid,$4,$5,'draining',$6,$7,$8,$9::text[]) RETURNING *`,+ (worktree_id,source_ids,principal_id,owner_epoch,topology_generation,state,old_head,target_head,upstream_ref,preserved_uncommitted,outcome)+ VALUES($1::uuid,$2::text[],$3::uuid,$4,$5,'draining',$6,$7,$8,$9::text[],$10::text::jsonb) RETURNING *`,[plan.binding.worktree_id, plan.members, principal, String(current.owner_epoch), String(current.topology_generation),- plan.oldHead, plan.targetHead, plan.upstreamRef, plan.preserved]);+ plan.oldHead, plan.targetHead, plan.upstreamRef, plan.preserved, JSON.stringify({ holder: refreshHolder() })]);return row;})).catch(error => {if ((error as { code?: string }).code === '23505' && !classifyOriginUniqueViolation(error)) throw refusal('refresh_in_progress', 'Another refresh of this worktree started first.', `gbrain sources refresh ${sourceId} --resume`);@@ -557,16 +579,18 @@ async function recoverOne(engine: BrainEngine, git: Git, row: WorktreeRefreshRow}/**- * Owner-startup recovery: every interrupted refresh (draining, fenced or- * merged) on this host's worktrees whose refreshing process has exited (its- * refresh lock is free) is converged one step, so the scan finds nothing on- * the next tick. Member syncs are not run here: a `syncing` row is finished by- * `--resume`, or by the next cycle sync plus admission's lazy completion.+ * Owner recovery (the consumer's topology scan): every interrupted refresh on+ * this host's worktrees whose refreshing process has exited (its refresh lock+ * is free) is converged one step, so the scan finds nothing on the next tick.+ * Member syncs are not run here. #7679: an orphaned `syncing` row whose members+ * all synced to the target or a descendant of it (a later cycle sync caught+ * them up) is completed here; one whose members lag is finished by the next+ * plain `gbrain sources refresh`, which runs the `--resume` path, or by `--resume`.*/export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostId?: string } = {}): Promise<number> {const host = opts.hostId ?? localHostId();const rows = await engine.executeRaw<WorktreeRefreshRow>(`SELECT f.* FROM persistence_worktree_refreshes f- JOIN persistence_worktrees w ON w.id=f.worktree_id WHERE w.owner_host_id=$1::uuid AND f.state IN ('draining','fenced','merged')+ JOIN persistence_worktrees w ON w.id=f.worktree_id WHERE w.owner_host_id=$1::uuid AND f.state IN ('draining','fenced','merged','syncing')ORDER BY f.created_at LIMIT 16`, [host]);let moved = 0;for (const row of rows) {@@ -576,7 +600,10 @@ export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostIif (!lock) continue;try {await withGit(async git => {- const after = await withTopologyLocks(engine, row.source_ids[0], async () => recoverOne(engine, git, await readRefresh(engine, row.id), root), undefined, 0);+ const after = await withTopologyLocks(engine, row.source_ids[0], async () => {+ const current = await readRefresh(engine, row.id);+ return current.state === 'syncing' ? completeSyncedOrphan(engine, git, current, root) : recoverOne(engine, git, current, root);+ }, undefined, 0);if (after.state !== row.state) moved++;});} catch (error) {@@ -589,6 +616,18 @@ export async function resumeWorktreeRefreshes(engine: BrainEngine, opts: { hostIreturn moved;}+/** #7679: an orphaned `syncing` row completes once every member's `last_commit` reached the target (or a descendant); otherwise it is left as it is. */+async function completeSyncedOrphan(engine: BrainEngine, git: Git, row: WorktreeRefreshRow, root: string): Promise<WorktreeRefreshRow> {+ const synced: RefreshMemberSync[] = [];+ for (const sourceId of row.source_ids) {+ const commit = await lastCommit(engine, sourceId);+ if (!await reachedTarget(git, root, row, commit)) return row;+ synced.push({ source_id: sourceId, status: 'up_to_date', last_commit: commit });+ }+ await engine.transaction(tx => transition(tx, row, ['syncing'], 'completed', { synced, completed_by: 'owner_recovery' }));+ return readRefresh(engine, row.id);+}+async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: RefreshOptions): Promise<WorktreeRefreshResult> {const binding = await getWorktreeBinding(engine, sourceId, existingLocalHostId());if (!await managedPersistenceEnabled(engine) || !binding) throw refusal('refresh_not_managed',@@ -618,24 +657,65 @@ async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: Refrreturn { ...empty, status: 'abandoned', refresh_id: row.id, source_ids: row.source_ids, old_head: row.old_head, target_head: row.target_head,upstream_ref: row.upstream_ref, preserved_uncommitted: row.preserved_uncommitted, next: `gbrain sources refresh ${sourceId}` };}- let row = await withTopologyLocks(engine, sourceId, async () => recoverOne(engine, git, await readRefresh(engine, active.id), root), undefined, 30_000);- if (row.state === 'recovery_required' && head === row.old_head) throw refusal('refresh_recovery_required',- `The checkout is back at the pre-refresh commit ${row.old_head}; nothing remains to resume.`, `gbrain sources refresh ${sourceId} --abandon`, row.id);- if (row.state === 'recovery_required') {- const stored = (row.outcome.refusal ?? {}) as { cause?: string; fix?: string };- throw refusal('refresh_recovery_required', stored.cause ?? 'The refresh could not verify the checkout HEAD.',- stored.fix ?? `gbrain sources writer status ${sourceId}, then gbrain sources refresh ${sourceId} --resume`, row.id);- }- if (row.state !== 'syncing') return { ...empty, status: 'abandoned', refresh_id: row.id, source_ids: row.source_ids, old_head: row.old_head,- target_head: row.target_head, upstream_ref: row.upstream_ref, next: `The interrupted refresh was rolled back without moving HEAD; start again with gbrain sources refresh ${sourceId}` };- startPersistenceConsumer(engine, loadConfig() ?? { engine: engine.kind });- const sync = await syncMembers(engine, git, row, root);- row = await readRefresh(engine, row.id);- return finished(row, sync);+ return resumeHeld(engine, git, sourceId, active.id, root, head, empty);});} finally { await lock.release(); }}+/**+ * `--resume` under the refresh lock the caller holds: one recovery step, then the member syncs of a `syncing` row.+ * #7679: the caller becomes the row's holder, and the holder it replaced is kept as `previous_holder`.+ */+async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refreshId: string, root: string, head: string | null, empty: WorktreeRefreshResult): Promise<WorktreeRefreshResult> {+ await engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=outcome||jsonb_build_object('holder',$2::text::jsonb,'previous_holder',outcome->'holder')+ WHERE id=$1::uuid AND state IN ${ACTIVE_REFRESH_STATES_SQL}`, [refreshId, JSON.stringify(refreshHolder())]);+ let row = await withTopologyLocks(engine, sourceId, async () => recoverOne(engine, git, await readRefresh(engine, refreshId), root), undefined, 30_000);+ if (row.state === 'recovery_required' && head === row.old_head) throw refusal('refresh_recovery_required',+ `The checkout is back at the pre-refresh commit ${row.old_head}; nothing remains to resume.`, `gbrain sources refresh ${sourceId} --abandon`, row.id);+ if (row.state === 'recovery_required') {+ const stored = (row.outcome.refusal ?? {}) as { cause?: string; fix?: string };+ throw refusal('refresh_recovery_required', stored.cause ?? 'The refresh could not verify the checkout HEAD.',+ stored.fix ?? `gbrain sources writer status ${sourceId}, then gbrain sources refresh ${sourceId} --resume`, row.id);+ }+ const recovered = { refresh_id: row.id, state: row.state, previous_holder: (row.outcome.previous_holder as RefreshHolder | null | undefined) ?? null };+ if (row.state !== 'syncing') return { ...empty, status: 'abandoned', refresh_id: row.id, source_ids: row.source_ids, old_head: row.old_head,+ target_head: row.target_head, upstream_ref: row.upstream_ref, next: `The interrupted refresh was rolled back without moving HEAD; start again with gbrain sources refresh ${sourceId}`,+ recovered: { ...recovered, status: 'abandoned' } };+ startPersistenceConsumer(engine, loadConfig() ?? { engine: engine.kind });+ const sync = await syncMembers(engine, git, row, root);+ row = await readRefresh(engine, row.id);+ const result = finished(row, sync);+ return { ...result, recovered: { ...recovered, state: row.state, status: result.status } };+}++/**+ * #7679: a plain `gbrain sources refresh` that finds an active refresh whose lock no process holds (its refresher died,+ * for example a container replaced mid-refresh) finishes it through the `--resume` path instead of refusing+ * `refresh_in_progress` on every scheduled run. Returns null when there is nothing to adopt: no active row, a+ * `recovery_required` row (the user decides how to reset), a live holder (the lock stays busy for a second), or a+ * caller that is not the owner (the precheck refuses each case by name).+ */+async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string): Promise<WorktreeRefreshResult | null> {+ if (!await managedPersistenceEnabled(engine)) return null;+ const hostId = existingLocalHostId();+ const binding = await getWorktreeBinding(engine, sourceId, hostId);+ if (!binding || !hostId || binding.owner_host_id !== hostId || !binding.local_path) return null;+ const active = await readActiveRefresh(engine, binding.worktree_id);+ if (!active || active.state === 'recovery_required') return null;+ // A holder recorded on another host is never adopted: this host's lock cannot say whether it is alive.+ if ((holderOf(active)?.host_id ?? hostId) !== hostId) return null;+ const lock = await acquireNativeLock(refreshLockPath(binding.worktree_id), { timeoutMs: 1000 });+ if (!lock) return null;+ try {+ const orphan = await readActiveRefresh(engine, binding.worktree_id);+ if (!orphan || orphan.state === 'recovery_required' || (holderOf(orphan)?.host_id ?? hostId) !== hostId) return null;+ const root = binding.local_path;+ const empty: WorktreeRefreshResult = { status: 'nothing_to_resume', refresh_id: null, worktree_id: binding.worktree_id, source_ids: orphan.source_ids,+ old_head: null, target_head: null, upstream_ref: null, preserved_uncommitted: [], synced: [] };+ return await withGit(async git => resumeHeld(engine, git, sourceId, orphan.id, root, await headOf(git, root), empty));+ } finally { await lock.release(); }+}+/*** A managed sync started elsewhere (cycle, autopilot, `gbrain sync`) while a* refresh drains or holds the fence is refused before discovery, so it never@@ -654,15 +734,55 @@ export async function assertManagedSyncAllowed(engine: BrainEngine, worktreeId:throw error;}-/** `gbrain sources writer status`: active refreshes (optionally of one source's worktree) with the command that moves each on. */+/**+ * #7679: whether a process still holds an active refresh's lock. The lock lives on the host that ran the refresh (the+ * recorded holder's, else the worktree owner's), so only that host can tell; a free lock means the refresher exited.+ * The probe takes the lock for an instant and releases it; a refresh starting at that instant waits a second for it.+ */+export async function refreshLiveness(worktreeId: string, holderHostId: string | null): Promise<RefreshLiveness> {+ const local = existingLocalHostId();+ if (!local || holderHostId !== local) return 'unknown';+ const lock = await tryAcquireNativeLock(refreshLockPath(worktreeId)).catch(() => undefined);+ if (lock === undefined) return 'unknown';+ if (!lock) return 'live';+ await lock.release();+ return 'orphaned';+}++/** #7679: the holder of an active refresh, whether it is orphaned, and the step that moves it on (`fix` names `--resume` whenever no process will). */+export async function refreshHolderReport(row: Pick<WorktreeRefreshRow, 'id' | 'worktree_id' | 'state' | 'source_ids' | 'outcome'> & { owner_host_id: string | null }):+ Promise<{ holder: RefreshHolder | null; liveness: RefreshLiveness; orphaned: boolean | null; auto_recovery: 'next_refresh' | 'none' | 'unknown'; fix: Action | null }> {+ const holder = holderOf(row);+ const holderHost = holder?.host_id ?? row.owner_host_id;+ const liveness = await refreshLiveness(row.worktree_id, holderHost);+ const source = row.source_ids[0];+ const resume = ['gbrain', 'sources', 'refresh', source, '--resume'];+ const verify = { argv: ['gbrain', 'sources', 'writer', 'status', '--source', source, '--json'] };+ const base = { holder, liveness, orphaned: liveness === 'unknown' ? null : liveness === 'orphaned' };+ if (row.state === 'recovery_required') return { ...base, auto_recovery: 'none', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,+ why: `Refresh ${row.id} could not verify the checkout HEAD, so nothing recovers it on its own. --resume changes nothing here: it prints the two ways out (reset to the start commit and --abandon, or to the upstream commit and --resume); resetting is destructive, so ask the user first.` } };+ if (liveness === 'live') return { ...base, auto_recovery: 'none', fix: null };+ if (liveness === 'orphaned') return { ...base, auto_recovery: 'next_refresh', fix: { argv: resume, consent: [], actor: 'agent', requires_exclusive: false, verify,+ why: `No process holds refresh ${row.id}'s lock (its holder, ${describeHolder(holder)}, exited), so it is orphaned. The next gbrain sources refresh ${source} finishes it on its own through the --resume path`+ + `${row.state === 'syncing' ? ', and the owner completes it once every member has synced to its target' : ''}; --resume finishes it now.` } };+ return { ...base, auto_recovery: 'unknown', fix: { argv: verify.argv, consent: [], actor: 'host_admin', requires_exclusive: false,+ why: `Refresh ${row.id}'s lock lives on host ${holderHost ?? 'unknown'}, not this one, so only that host can tell whether ${describeHolder(holder)} is alive. Run this there; if it reports the refresh orphaned, or that host is gone for good, gbrain sources refresh ${source} --resume on the owner host finishes it (a plain refresh does not adopt another host's refresh).` } };+}++/** `gbrain sources writer status`: active refreshes (optionally of one source's worktree) with their holder and the command that moves each on. */export async function activeWorktreeRefreshes(engine: BrainEngine, sourceId?: string): Promise<Array<Record<string, unknown>>> {- const rows = await engine.executeRaw<WorktreeRefreshRow>(`SELECT * FROM persistence_worktree_refreshes WHERE state IN ${ACTIVE_REFRESH_STATES_SQL}- AND ($1::text IS NULL OR $1=ANY(source_ids)) ORDER BY created_at`, [sourceId ?? null]);- return rows.map(row => ({ refresh_id: row.id, worktree_id: row.worktree_id, state: row.state, source_ids: row.source_ids,- old_head: row.old_head, target_head: row.target_head, upstream_ref: row.upstream_ref, updated_at: new Date(row.updated_at).toISOString(),- ...(row.outcome.refusal ? { refusal: row.outcome.refusal } : {}),- next: row.state === 'draining' ? 'Writes to this worktree are refused while queued work drains; retry them after about a second.'- : `gbrain sources refresh ${row.source_ids[0]} --resume` }));+ const rows = await engine.executeRaw<WorktreeRefreshRow & { owner_host_id: string | null }>(`SELECT f.*,w.owner_host_id::text AS owner_host_id FROM persistence_worktree_refreshes f+ LEFT JOIN persistence_worktrees w ON w.id=f.worktree_id WHERE f.state IN ${ACTIVE_REFRESH_STATES_SQL}+ AND ($1::text IS NULL OR $1=ANY(f.source_ids)) ORDER BY f.created_at`, [sourceId ?? null]);+ return Promise.all(rows.map(async row => {+ const report = await refreshHolderReport(row);+ return { refresh_id: row.id, worktree_id: row.worktree_id, state: row.state, source_ids: row.source_ids,+ old_head: row.old_head, target_head: row.target_head, upstream_ref: row.upstream_ref, updated_at: new Date(row.updated_at).toISOString(),+ ...(row.outcome.refusal ? { refusal: row.outcome.refusal } : {}), ...report,+ next: report.liveness === 'live' && row.state === 'draining' ? 'Writes to this worktree are refused while queued work drains; retry them after about a second.'+ : report.liveness === 'live' ? `Wait for it: ${describeHolder(report.holder)} holds the refresh lock and finishes it.`+ : `gbrain sources refresh ${row.source_ids[0]} --resume` };+ }));}/** Doctor `worktree_refresh_stuck`: active refreshes older than `minutes`. */diff --git a/test/e2e/worktree-refresh-postgres.test.ts b/test/e2e/worktree-refresh-postgres.test.tsindex 22d1c98c2c..4d70b235bc 100644--- a/test/e2e/worktree-refresh-postgres.test.ts+++ b/test/e2e/worktree-refresh-postgres.test.ts@@ -5,4 +5,5 @@ import { registerPostgresTests } from '../helpers/test-backends.ts';await registerPostgresTests(() => import('../persistence-worktree-refresh.test.ts'),() => import('../persistence-worktree-refresh-restart.test.ts'),+ () => import('../worktree-refresh-orphan-7679.test.ts'),);diff --git a/test/helpers/worktree-refresh-child.ts b/test/helpers/worktree-refresh-child.tsindex ebc9bdde0f..1cf5aceef7 100644--- a/test/helpers/worktree-refresh-child.ts+++ b/test/helpers/worktree-refresh-child.ts@@ -4,8 +4,11 @@* and before the merged bookkeeping commits. `restart`: open the brain as a* fresh owner, let the resident consumer's startup recovery move the refresh,* then `--resume` it and report the final state as one JSON line.+ * `kill-syncing` (#7679): start a refresh and SIGKILL this process once the row+ * is `syncing` (merged, member syncs not yet run), printing its pid first.*/import { randomUUID } from 'node:crypto';+import { writeSync } from 'node:fs';import type { GBrainConfig } from '../../src/core/config.ts';import type { BrainEngine } from '../../src/core/engine.ts';import { PGLiteEngine } from '../../src/core/pglite-engine.ts';@@ -15,14 +18,21 @@ import { disposePersistenceConsumer, startPersistenceConsumer } from '../../src/import { refreshWorktree } from '../../src/core/persistence/worktree-refresh.ts';import { assertSafeE2eDatabaseUrl } from './db-guard.ts';-const input = JSON.parse(process.env.GBRAIN_TEST_REFRESH_CHILD!) as { database: GBrainConfig; sourceId: string; worktreeId: string; mode: 'crash' | 'restart' };+const input = JSON.parse(process.env.GBRAIN_TEST_REFRESH_CHILD!) as { database: GBrainConfig; sourceId: string; worktreeId: string; mode: 'crash' | 'restart' | 'kill-syncing' };if (input.database.database_url) assertSafeE2eDatabaseUrl(input.database.database_url);const engine: BrainEngine = input.database.engine === 'pglite' ? new PGLiteEngine() : new PostgresEngine();await engine.connect(input.database);const state = async () => (await engine.executeRaw<{ state: string }>('SELECT state FROM persistence_worktree_refreshes WHERE worktree_id=$1::uuid ORDER BY created_at DESC LIMIT 1', [input.worktreeId]))[0]?.state ?? null;-if (input.mode === 'crash') {+if (input.mode === 'kill-syncing') {+ await refreshWorktree(engine, input.sourceId, { hooks: { boundary: point => {+ if (point !== 'syncing') return;+ writeSync(1, `REFRESH_KILLED_IN_SYNCING ${process.pid}\n`);+ process.kill(process.pid, 'SIGKILL');+ } } });+ process.stdout.write('REFRESH_UNEXPECTED_FINISH\n');+} else if (input.mode === 'crash') {const requestId = randomUUID();await submitPageMutation({ engine, sourceId: input.sourceId, remote: false, dryRun: false, config: { engine: engine.kind, embedding_disabled: true } as never,logger: { info() {}, warn() {}, error() {} } }, { operation: 'put_page', waitMs: 0,diff --git a/test/persistence-worktree-refresh.test.ts b/test/persistence-worktree-refresh.test.tsindex 6f0d676654..dbf2770916 100644--- a/test/persistence-worktree-refresh.test.ts+++ b/test/persistence-worktree-refresh.test.ts@@ -556,6 +556,8 @@ test('#6712 wave: a refresh completes once every member synced to a descendant oexpect((await f.refreshRows()).map(r => r.state)).toEqual(['completed']);expect((await f.put(f.alpha, 'notes/after-descendant', page('After', 'admitted'))).state ?? 'committed').toBe('committed');// A refresh left in syncing (its process died) whose members already synced past the target completes on --resume.+ // #7679: the owner scan would complete it on its own, so the consumer is stopped to leave it to --resume.+ await disposePersistenceConsumer(f.engine);const [row] = await f.refreshRows();await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET state='syncing',completed_at=NULL WHERE id=$1::uuid`, [row.id]);expect(await refreshWorktree(f.engine, f.alpha, { resume: true })).toMatchObject({ status: 'completed', refresh_id: row.id });diff --git a/test/worktree-refresh-orphan-7679.test.ts b/test/worktree-refresh-orphan-7679.test.tsnew file mode 100644index 0000000000..760218b34b--- /dev/null+++ b/test/worktree-refresh-orphan-7679.test.ts@@ -0,0 +1,199 @@+/**+ * #7679: an orphaned `gbrain sources refresh` (its process killed while the row+ * is `syncing`, for example a container replaced mid-refresh) recovers without+ * an operator, and a live refresh is never stolen.+ *+ * Protects: the next plain refresh finishes an orphaned refresh through the+ * `--resume` path, then refreshes; owner recovery completes an orphaned+ * `syncing` row once its members synced to the target or a descendant of it;+ * writer status and sync status name the recorded holder, whether it is+ * orphaned, and a fix naming `--resume`.+ * Regression it catches: every scheduled refresh refusing `refresh_in_progress`+ * forever after a crash (and every page write to the worktree refusing+ * `worktree_refreshing`), owner recovery skipping `syncing` rows, status that+ * cannot tell a live refresh from an orphan, and recovery adopting a refresh+ * whose process still holds the refresh lock or whose holder is on another host.+ * Seams: a real child process SIGKILLs itself at the refresh's `syncing`+ * boundary; the live holder is a refresh paused at that boundary in this+ * process (the native lock is per open file, so this process cannot take it).+ */+import { afterAll, beforeAll, expect, test } from 'bun:test';+import { spawnSync } from 'node:child_process';+import { randomUUID } from 'node:crypto';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import type { GBrainConfig } from '../src/core/config.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';+import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { readSyncStatus } from '../src/core/persistence/sync-status.ts';+import { performManagedSync } from '../src/core/persistence/sync-run.ts';+import { refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { testBackends } from './helpers/test-backends.ts';+import { withEnv } from './helpers/with-env.ts';+import { commitFile, git, page, setupRefreshFixture, type RefreshFixture } from './helpers/worktree-refresh-fixture.ts';++const backends = testBackends();+const dir = mkdtempSync(join(tmpdir(), 'gbrain-refresh-orphan-'));+const cases: Array<{ engine: BrainEngine; database: GBrainConfig; close: () => Promise<void> }> = [];++beforeAll(async () => {+ if (backends.includes('pglite')) {+ const database = { engine: 'pglite', database_path: join(dir, 'brain.pglite') } as GBrainConfig;+ const engine = new PGLiteEngine();+ await engine.connect(database); await engine.initSchema();+ cases.push({ engine, database, close: () => engine.disconnect() });+ }+ if (backends.includes('postgres')) {+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!);+ cases.push({ engine: pg.engine, database: { engine: 'postgres', database_url: pg.databaseUrl } as GBrainConfig, close: pg.close });+ }+}, 120_000);+afterAll(async () => {+ for (const item of cases) { await disposePersistenceConsumer(item.engine); await item.close(); }+ rmSync(dir, { recursive: true, force: true });+});++type Case = typeof cases[number];+type Row = { id: string; state: string; outcome: Record<string, any> };+const refusedWith = async (promise: Promise<unknown>, code: string) => {+ const error = await promise.then(() => null, e => e as { code?: string; message?: string; suggestion?: string });+ expect(error, `expected refusal ${code}`).not.toBeNull();+ expect(error!.code).toBe(code);+ return error!;+};+const writerRefreshes = async (f: RefreshFixture, sourceId: string) =>+ (await runPersistenceAdministration(f.engine, 'writer_status', { source_id: sourceId }) as { worktree_refreshes: Array<Record<string, any>> }).worktree_refreshes;++/** A fixture whose refresh process was SIGKILLed at `syncing`: HEAD is at `target`, the members still at `old`. Runs `fn` with GBRAIN_HOME set. */+async function withKilledRefresh(item: Case, fn: (k: { f: RefreshFixture; target: string; old: string; pid: number; orphan: Row }) => Promise<void>): Promise<void> {+ const caseDir = mkdtempSync(join(dir, 'case-'));+ const home = join(caseDir, 'home');+ await withEnv({ GBRAIN_HOME: home }, async () => {+ const f = await setupRefreshFixture(item.engine, caseDir);+ await disposePersistenceConsumer(item.engine);+ const old = f.head();+ const target = f.push('alpha/two.md', page('Alpha two', 'An observation whose refresh was killed.'));+ if (item.engine.kind === 'pglite') await item.engine.disconnect();+ const run = spawnSync(process.execPath, [join(import.meta.dir, 'helpers/worktree-refresh-child.ts')], { encoding: 'utf8', timeout: 90_000,+ env: { ...process.env, GBRAIN_HOME: home, GBRAIN_TEST_REFRESH_CHILD: JSON.stringify({ database: item.database, sourceId: f.alpha, worktreeId: f.worktreeId, mode: 'kill-syncing' }) } });+ if (item.engine.kind === 'pglite') await item.engine.connect(item.database);+ const line = run.stdout.split('\n').find(row => row.startsWith('REFRESH_KILLED_IN_SYNCING '));+ expect(line, `${run.stdout}\n${run.stderr}`).toBeDefined();+ expect(run.signal).toBe('SIGKILL');+ const pid = Number(line!.split(' ')[1]);+ const rows = await f.refreshRows() as Row[];+ expect(rows.map(row => row.state)).toEqual(['syncing']);+ expect(rows[0].outcome.holder).toMatchObject({ pid });+ expect(f.head()).toBe(target);+ expect(await f.lastCommit(f.alpha)).toBe(old);+ try { await fn({ f, target, old, pid, orphan: rows[0] }); }+ finally { await disposePersistenceConsumer(item.engine); rmSync(caseDir, { recursive: true, force: true }); }+ });+}++test('#7679: a refresh killed mid-syncing is reported orphaned with its holder, and the next scheduled refresh finishes it, then refreshes', async () => {+ for (const item of cases) await withKilledRefresh(item, async ({ f, target, pid, orphan }) => {+ const resume = ['gbrain', 'sources', 'refresh', f.alpha, '--resume'];+ const [status] = await writerRefreshes(f, f.beta);+ expect(status).toMatchObject({ refresh_id: orphan.id, state: 'syncing', liveness: 'orphaned', orphaned: true, auto_recovery: 'next_refresh',+ holder: { pid }, fix: { argv: resume, actor: 'agent' }, next: resume.join(' ') });+ const sync = await readSyncStatus(f.engine, f.beta, { detail: false });+ expect(sync.worktree_refresh).toMatchObject({ refresh_id: orphan.id, state: 'syncing', orphaned: true, holder: { pid } });+ expect(sync.next?.argv).toEqual(resume);+ // The wedge: every page write to the worktree is refused while the orphan stands.+ await refusedWith(f.put(f.beta, 'notes/while-orphaned', page('While orphaned', 'refused')), 'worktree_refreshing');++ const next = f.push('alpha/three.md', page('Alpha three', 'Landed upstream while the refresh was orphaned.'));+ const result = await refreshWorktree(f.engine, f.alpha);+ expect(result).toMatchObject({ status: 'completed', old_head: target, target_head: next,+ recovered: { refresh_id: orphan.id, state: 'completed', status: 'completed', previous_holder: { pid } } });+ const rows = await f.refreshRows() as Row[];+ expect(rows.map(row => row.state)).toEqual(['completed', 'completed']);+ expect(rows[0].outcome.previous_holder).toMatchObject({ pid });+ expect(rows[0].outcome.holder.pid).toBe(process.pid);+ expect(f.head()).toBe(next);+ expect(await f.lastCommit(f.alpha)).toBe(next);+ expect(await f.lastCommit(f.beta)).toBe(next);+ expect(await writerRefreshes(f, f.beta)).toEqual([]);+ expect((await readSyncStatus(f.engine, f.beta, { detail: false })).worktree_refresh).toBeNull();+ expect((await f.put(f.beta, 'notes/after-recovery', page('After', 'admitted'))).state ?? 'committed').toBe('committed');+ });+}, 300_000);++test('#7679: owner recovery completes an orphaned syncing refresh once every member synced past its target, and leaves one whose members lag', async () => {+ for (const item of cases) await withKilledRefresh(item, async ({ f, target, orphan }) => {+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['syncing']);+ // An agent commits on top of the target and cycle syncs catch both members up to that descendant.+ const descendant = commitFile(f.root, 'beta/local.md', page('Local', 'Committed on top of the refresh target.'));+ for (const id of [f.alpha, f.beta]) await performManagedSync(f.engine, { sourceId: id, noPull: true, noEmbed: true });+ expect(git(f.root, 'merge-base', '--is-ancestor', target, descendant)).toBe('');+ expect(await f.lastCommit(f.alpha)).toBe(descendant);+ expect(await f.lastCommit(f.beta)).toBe(descendant);+ // The owner scan (the consumer the syncs started runs it too) completes the orphan; nothing else does.+ const deadline = Date.now() + 20_000;+ while ((await f.refreshRows())[0].state === 'syncing' && Date.now() < deadline) await resumeWorktreeRefreshes(f.engine);+ const [row] = await f.refreshRows() as Row[];+ expect(row).toMatchObject({ id: orphan.id, state: 'completed', outcome: { completed_by: 'owner_recovery' } });+ expect((await f.put(f.alpha, 'notes/after-recovery', page('After', 'admitted'))).state ?? 'committed').toBe('committed');+ });+}, 300_000);++test('#7679: a live refresh is never stolen: owner recovery, a plain refresh and --resume leave it to its holder, and status reports it live', async () => {+ for (const item of cases) {+ const caseDir = mkdtempSync(join(dir, 'live-'));+ await withEnv({ GBRAIN_HOME: join(caseDir, 'home') }, async () => {+ const f = await setupRefreshFixture(item.engine, caseDir);+ try {+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream while a refresh is live.'));+ let reached!: () => void, release!: () => void;+ const atSyncing = new Promise<void>(resolve => { reached = resolve; });+ const gate = new Promise<void>(resolve => { release = resolve; });+ const live = refreshWorktree(f.engine, f.alpha, { hooks: { boundary: async point => { if (point === 'syncing') { reached(); await gate; } } } });+ await atSyncing;+ try {+ const [held] = await f.refreshRows() as Row[];+ expect(held.state).toBe('syncing');+ // Both members caught up while the holder is paused: still not the owner scan's to complete.+ for (const id of [f.alpha, f.beta]) await performManagedSync(f.engine, { sourceId: id, noPull: true, noEmbed: true });+ expect(await f.lastCommit(f.alpha)).toBe(target);+ expect(await resumeWorktreeRefreshes(f.engine)).toBe(0);+ const refused = await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_in_progress');+ expect(refused.message).toContain(`pid ${process.pid}`);+ await refusedWith(refreshWorktree(f.engine, f.alpha, { resume: true }), 'refresh_in_progress');+ const [row] = await f.refreshRows() as Row[];+ expect(row).toMatchObject({ id: held.id, state: 'syncing' });+ expect(row.outcome.holder).toEqual(held.outcome.holder);+ expect(row.outcome.previous_holder).toBeUndefined();+ const [status] = await writerRefreshes(f, f.alpha);+ expect(status).toMatchObject({ refresh_id: held.id, liveness: 'live', orphaned: false, auto_recovery: 'none', fix: null, holder: { pid: process.pid } });+ expect(status.next).toContain('Wait for it');+ const sync = await readSyncStatus(f.engine, f.alpha, { detail: false });+ expect(sync.worktree_refresh).toMatchObject({ liveness: 'live', orphaned: false });+ expect(sync.next?.argv ?? []).not.toContain('--resume');+ } finally { release(); }+ expect(await live).toMatchObject({ status: 'completed', target_head: target });+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['completed']);+ } finally { await disposePersistenceConsumer(item.engine); }+ });+ rmSync(caseDir, { recursive: true, force: true });+ }+}, 300_000);++test('#7679: an orphan whose recorded holder is on another host is reported unknown and is not adopted by a plain refresh; --resume on the owner finishes it', async () => {+ for (const item of cases) await withKilledRefresh(item, async ({ f, target, orphan }) => {+ const elsewhere = randomUUID();+ await f.engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=jsonb_set(outcome,'{holder,host_id}',to_jsonb($2::text)) WHERE id=$1::uuid`, [orphan.id, elsewhere]);+ const [status] = await writerRefreshes(f, f.alpha);+ expect(status).toMatchObject({ liveness: 'unknown', orphaned: null, auto_recovery: 'unknown', fix: { actor: 'host_admin' } });+ expect(status.fix.why).toContain(elsewhere);+ expect(status.fix.why).toContain('--resume');+ await refusedWith(refreshWorktree(f.engine, f.alpha), 'refresh_in_progress');+ expect((await f.refreshRows()).map(row => row.state)).toEqual(['syncing']);+ expect(await refreshWorktree(f.engine, f.alpha, { resume: true })).toMatchObject({ status: 'completed', refresh_id: orphan.id, target_head: target });+ });+}, 300_000);