Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(refresh): recover a refused-merge crash by aborting, using the reflog to tell it from a merge that ran (GST-1674)
garrytan43a84a5ead52
diff --git a/.github/workflows/tla-spec.yml b/.github/workflows/tla-spec.ymlindex 0286c21939..2fc173c475 100644--- a/.github/workflows/tla-spec.yml+++ b/.github/workflows/tla-spec.yml@@ -49,6 +49,7 @@ on:- 'src/core/persistence/import-deferral.ts'- 'src/core/persistence/local-edit-adoption.ts'- 'test/persistence-worktree-refresh.test.ts'+ - 'test/persistence-worktree-refresh-restart.test.ts'- 'test/pending-import-async-git-7436.test.ts'push:branches: [master]@@ -93,6 +94,7 @@ on:- 'src/core/persistence/import-deferral.ts'- 'src/core/persistence/local-edit-adoption.ts'- 'test/persistence-worktree-refresh.test.ts'+ - 'test/persistence-worktree-refresh-restart.test.ts'- 'test/pending-import-async-git-7436.test.ts'schedule:- cron: '40 9 * * *' # nightly, 09:40 UTCdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 0011813b94..ded2ab982d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,16 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**A refresh killed after git refused its fast-forward no longer leaves the checkout fenced until a person acts.** When another agent commits on the start commit with plain git right before a refresh merges, git refuses the fast-forward and the refresh aborts, keeping that commit. If the refresh process died between the refused merge and that abort, recovery used to mark the refresh `recovery_required`, so writes to the checkout stayed fenced (or deferred, with the refresh write barrier on) until someone reset the checkout. Recovery (the owner's scan or `gbrain sources refresh --resume`) now aborts it the same way, with the fence lifted and the commit kept, and the next refresh completes. It tells this case from a merge that ran and was then reset by reading the reflog of HEAD and its branch since the merge started; without a reflog, or for a refresh an earlier version started, it still marks `recovery_required`.++### Itemized changes++- `src/core/persistence/worktree-refresh.ts`: the update that marks `outcome.merge_started` also records `merge_started_at` (epoch ms, the refresher's clock). `recoverOne` aborts a `fenced` row `refresh_interrupted` when its merge started, HEAD descends from the start commit without the target, no member synced off HEAD, and the reflogs of HEAD and its branch (`reachedTargetSince`) show no move to the target since `merge_started_at` less 5 s. A missing, empty or unreadable reflog keeps `recovery_required`. `--abandon` lifts such a row the same way instead of marking it `recovery_required`.+- `spec/tla/refresh-barrier`: `RefusedAbortable` models the new recovery rule, `FfMerge` aborts a refused merge as the code has since #8765, and `RecoveryOnlyByPolicy` now allows `recovery_required` only for a merge that ran. The guard mutant `resume-marks-refused-merge-recovery` reverts the rule.+- Tests: `test/persistence-worktree-refresh-restart.test.ts` SIGKILLs a real refresh process right after git refused its merge, on PGLite and Postgres: recovery aborts it, a write deferred behind it commits and the next refresh completes; with the reflog deleted, or with no `merge_started_at`, it stays `recovery_required`.+## [0.60.611.0] - 2026-11-08**Doctor `foreign_writers` no longer reads an unreadable ledger or a skipped repository as clean, the owner-path commit guard fails closed, and the opt-in local-edit adoption (`persistence.adopt_local_edits`, off by default) gets the hardening it needed before any source turns it on.** `foreign_writers` now marks a scan partial when a root's Git probe fails, when roots pass the 50-root cap, or when the commit ledger exists but cannot be read. It names a foreign commit that carries a `gbrain-` trailer as possibly an older gbrain without the ledger, and still fails it. `gbrain git commit-push` refuses a repository that only encloses a registered checkout, even when an older gbrain left a stale marker in it. Adoption keeps its refusal for a conflicted path or a merge in progress, a stale editor save, a segmented or oversized page, and a caller whose claim is gone, and checks the claim, the conflict and the merge again in the transaction that publishes its commit. Its quiet wait is capped at the foreground budget, and an adoption whose publication failed after its commit is retried instead of left orphaned.diff --git a/docs/architecture/key-files/core-persistence-git.md b/docs/architecture/key-files/core-persistence-git.mdindex 96d460d74a..39b5721774 100644--- a/docs/architecture/key-files/core-persistence-git.md+++ b/docs/architecture/key-files/core-persistence-git.md@@ -4,7 +4,7 @@Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.The managed Git writer (`gbrain sources refresh`), git-canonical memory (the git gap census, declared database-only classes, the door) and the checkout reads beside them.-- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified (#8617: a `refresh_source_changed` for a moved HEAD or work that arrived after the drain check is retried once in the same run from a fresh precheck; a second refusal returns the first with the retry's cause appended), dirt rechecked, `git merge --ff-only`, HEAD verified again (a HEAD that descends from the target after a successful merge, with no member synced off it, counts as merged: a plain-git commit landed in between; a refused fast-forward whose HEAD only gained commits on the start commit aborts `refresh_source_changed`, GST-1674), then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it a member is `deferred`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` / `sync_deferred` when a member blocks or is deferred. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit, and never on a `recovery_required` row once a member's `last_commit` is past it (not it or an ancestor; GST-1457: the next sync would soft-delete what it synced), and `--resume` finishes a `recovery_required` row only at a HEAD that contains every member commit on that refresh's line. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-*postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.+- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified (#8617: a `refresh_source_changed` for a moved HEAD or work that arrived after the drain check is retried once in the same run from a fresh precheck; a second refusal returns the first with the retry's cause appended), dirt rechecked, `git merge --ff-only`, HEAD verified again (a HEAD that descends from the target after a successful merge, with no member synced off it, counts as merged: a plain-git commit landed in between; a refused fast-forward whose HEAD only gained commits on the start commit aborts `refresh_source_changed`, GST-1674), then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it a member is `deferred`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` / `sync_deferred` when a member blocks or is deferred. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; so does a fenced row whose merge started (`outcome.merge_started_at`, epoch ms) and whose HEAD descends from the old HEAD without the target, when the reflogs of HEAD and its branch show no move to the target since then (git refused the fast-forward, GST-1674; no readable reflog stays `recovery_required`); HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit, and never on a `recovery_required` row once a member's `last_commit` is past it (not it or an ancestor; GST-1457: the next sync would soft-delete what it synced), and `--resume` finishes a `recovery_required` row only at a HEAD that contains every member commit on that refresh's line. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge, and after a refused merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-*postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.- `src/core/persistence/refresh-barrier.ts` (+ deferred admission and claim in `journal.ts`, migration v292 `src/core/schema-migrations/v292-refresh-write-barrier.ts`) — the refresh write barrier (`docs/designs/REFRESH_WRITE_BARRIER.md`, `docs/guides/write-refusals.md#deferred-writes`). Off by default (`persistence.refresh_barrier`). With it on, a single page or memory write that reaches a worktree during a refresh is journaled in state `deferred` (tied to `deferred_refresh_id`) and answered pending instead of refused `worktree_refreshing`. Only a claim that declares the barrier moves a deferred row, and only after its refresh is released. `deferredReplayRefusal` re-checks the write's authority and reads the refresh's `old_head..target_head` diff with git, so a file the refresh changed ends `revision_conflict` with nothing applied. A write still waiting past `persistence.refresh_barrier_max_wait_ms` (15 minutes) ends `refresh_barrier_expired`. A write during a refresh in `recovery_required` defers too, and since GST-1582 there is no per-refresh cap (`persistence.refresh_barrier_max_deferred` is no longer read). A write whose page file holds a committed change sync has not imported yet is held with `deferred_import_from` (migration v294) instead of refused `pending_import`, and ends `pending_import_expired` past the wait limit (or four wait limits, at least four hours, after admission when it is held again and again). GST-1623: either kind of deferred write ends `revision_conflict` when a later write to its page committed first, and a refresh-deferred write is checked against its refresh before an import hold, which then clears `deferred_refresh_id`. `refresh-retry-hint.ts` derives `worktree_refreshing`'s `retry_after_ms` from the worktree's refresh history. `gbrain upgrade --check-downgrade` exits 3 `downgrade_blocked_deferred_writes` while any deferred write is unfinished. Tests: `test/refresh-write-barrier.test.ts`, `test/refresh-barrier-state-audit.test.ts`.- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, split by content evidence in `direction_evidence`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only with each source's `live_pages` (counted in SQL before the scan), a source the budget stopped or never reached as partial (`PARTIAL, <read> of <live> live page(s) read`). Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60, connector imports #7971). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The guard reads its state as the writing role and fails closed: a hidden or missing `persistence_brain` or `git_canonical_brain` row, or a source or page row hidden by row-level security, raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`), and the counter refuses a recount under RLS on `sources` (`test/writer-fence-fail-closed-7636.test.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).diff --git a/docs/designs/REFRESH_WRITE_BARRIER.md b/docs/designs/REFRESH_WRITE_BARRIER.mdindex 5b665ca949..5b964ffbd2 100644--- a/docs/designs/REFRESH_WRITE_BARRIER.md+++ b/docs/designs/REFRESH_WRITE_BARRIER.md@@ -276,9 +276,14 @@ Deferred rows are durable rows; nothing about them lives in process memory.- Killed while `fenced`, before the merge started, with HEAD moved past `old_head` on its own line(writes that landed while it drained, or a commit gbrain did not make): recovery aborts the row,as the live tick would, and its deferred writes commit (GBEV-44, GST-1674).+ - Killed after the merge started but before HEAD reached the target (a commit landed on `old_head`+ right before the merge, so git refused the fast-forward): the row also records+ `outcome.merge_started_at`, and when the reflogs of HEAD and its branch show no move to the target+ since then, recovery aborts the row the same way and keeps that commit (GST-1674). Without a+ readable reflog, or on a row an older version wrote, it stays `recovery_required`.- Killed after the merge, with HEAD at the target or a commit on top of it and no member synced offit: recovery records the merge and the next refresh finishes the member syncs, with no `--resume`.- - A merge that started and a HEAD that no longer holds the target (a reset and an unrelated commit)+ - A merge that ran and a HEAD that no longer holds the target (a reset and an unrelated commit)stays `recovery_required` for the operator.- The merge runs only when the update that records `merge_started` matched the `fenced` row; a rowrecovery moved meanwhile ends the refresh `refresh_source_changed` with nothing merged (GST-1623).diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 1a0f1d073d..c89e2d57e9 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -572,7 +572,7 @@ this anchor (`--json` puts them in one object on stdout); the command exits 1.| <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree, source and refresh rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the five oldest open transactions in this database holding locks on those tables (pid, application, state, how long the transaction has been open). They are candidates: the blocker may not be among them, because Postgres does not show which row each one holds. It never repeats their SQL, and says so when that lookup failed or took longer than 3 s. HEAD did not move: no fence was set, or the draining refresh was aborted. When the same holder also blocked that abort, the refusal says the row stays `draining`, and the next `gbrain sources refresh <source>` resumes it. | Retry `gbrain sources refresh <source>`. If it repeats, a transaction holding one of those rows is stuck: `why` lists the oldest candidates, so confirm the blocker on the brain host before ending any transaction. || <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. || <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. When HEAD moved or work reached the checkout after the drain check, the run already retried once from a fresh precheck, and the message names why that retry was refused too. | Retry `gbrain sources refresh <source>`. |-| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. (A commit that lands on the upstream commit right after the refresh's own merge is not this case: the refresh records the merge and syncs it. Nor is one that lands on the start commit right before the merge: git refuses the fast-forward and the refresh ends `refresh_source_changed`.) Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. Writer status reads HEAD and names the one step left when HEAD already sits at one of those commits: `--resume` at the upstream commit, `--abandon` at the start commit when no member source has synced past it. When one has (its `last_commit` is set and is neither the start commit nor an ancestor of it; a commit git cannot find counts too), `--abandon` refuses with this code and changes nothing, because the next sync would roll that member's pages back; `--resume` refuses too, and both name the one way out that keeps those pages: the fix (`next: ask_user`) previews with `git -C <checkout> status --short`, fast-forwards the checkout with `git merge --ff-only` to the commit the members synced (the upstream commit, or a later one a member already synced; it keeps uncommitted edits and refuses if one overlaps an incoming file), then runs `--resume`. When git refuses, the fix names a hard reset to that commit instead (destructive: it discards local commits and uncommitted edits not in that commit). `--resume` finishes a `recovery_required` refresh whose HEAD is the upstream commit or descends from it, unless a member synced a commit on this refresh's line (or one git cannot place) that HEAD does not contain; then it refuses with the same fast-forward fix. |+| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. (A commit that lands on the upstream commit right after the refresh's own merge is not this case: the refresh records the merge and syncs it. Nor is one that lands on the start commit right before the merge: git refuses the fast-forward and the refresh ends `refresh_source_changed`; if its process dies before that, recovery aborts the refresh the same way when the checkout's reflog shows HEAD never reached the upstream commit.) Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. Writer status reads HEAD and names the one step left when HEAD already sits at one of those commits: `--resume` at the upstream commit, `--abandon` at the start commit when no member source has synced past it. When one has (its `last_commit` is set and is neither the start commit nor an ancestor of it; a commit git cannot find counts too), `--abandon` refuses with this code and changes nothing, because the next sync would roll that member's pages back; `--resume` refuses too, and both name the one way out that keeps those pages: the fix (`next: ask_user`) previews with `git -C <checkout> status --short`, fast-forwards the checkout with `git merge --ff-only` to the commit the members synced (the upstream commit, or a later one a member already synced; it keeps uncommitted edits and refuses if one overlaps an incoming file), then runs `--resume`. When git refuses, the fix names a hard reset to that commit instead (destructive: it discards local commits and uncommitted edits not in that commit). `--resume` finishes a `recovery_required` refresh whose HEAD is the upstream commit or descends from it, unless a member synced a commit on this refresh's line (or one git cannot place) that HEAD does not contain; then it refuses with the same fast-forward fix. || <a id="worktree_refreshing"></a>`worktree_refreshing` | `worktree_refreshing` | A write, or a managed sync started by a cycle or another command, reached a checkout while a refresh is draining or merging it. It was not saved. `detail` is `retry_after_ms=<n>`, estimated from the refresh itself: the median duration of the worktree's last completed refreshes minus how long this one has run, never below 15 s while it syncs or 2 s in its other phases (so a refresh of several minutes is never answered with a 1 s retry). The message names the phase, how long the refresh has run and typically takes, and the CLI's human output ends with one line, `Nothing was saved; retry the same command in about <n> s (retry_after_ms=<n>).`, which survives `| tail -1`. Writes to database-only memory are not affected. An interactive write (CLI, MCP) waits out the fence for up to 10 s, or its explicit `wait_ms` if lower, before this refusal; a background write is refused at once. The refusal is `retryable`, its `fix.next` is `wait`, and the CLI exits 1; the memory verbs keep their frozen `unavailable` code with `canonical: worktree_refreshing`. With `persistence.refresh_barrier` on, a single page or memory write waits behind the refresh instead ([deferred writes](#deferred-writes)). | Retry the same request ID after `retry_after_ms`; nothing was recorded, so the retry cannot write twice. `gbrain sources writer status --source <source> --json` shows the refresh. A cycle's sync is retried by the next cycle. |A refresh is recorded in `persistence_worktree_refreshes` and survives a crash.diff --git a/spec/tla/README.md b/spec/tla/README.mdindex c31cbe348f..97dc9b421e 100644--- a/spec/tla/README.md+++ b/spec/tla/README.md@@ -167,7 +167,7 @@ invariant. Behind constants it also models the stacked follow-ups: writes held fEvery config kills the refreshing process once (`MaxCrashes = 1`), at any step, after which `recoverOne` resumes therefresh row it left. A refresh that finds HEAD moved at the merge aborts and the tick retries, as `fenceAndMerge` does,-so `recovery_required` is reached only through recovery or a refused merge. `RefreshBarrier.cfg` also lets+so `recovery_required` is reached only through recovery; a refused merge aborts too (GST-1674). `RefreshBarrier.cfg` also letssomeone reset the checkout to old_head after the merge (`MayReset`).`test/spec-refresh-barrier.test.ts` keeps the default config on #8682 alone, every safety config on every invariant,@@ -184,7 +184,7 @@ every config killing the refresh at least once, and the workflow's path filter c| `FenceNeverMidPrepare` | the fence never comes up, and the checkout never moves, while a write is claimed || `DeferredWaitsForRelease`, `RefreshChangeNeverOverwritten` | a deferred write commits only after its refresh is released, and never over a file that refresh changed || `NoDeferToReleased`, `ReleasedWhenEnded` | admission never defers to a released refresh; a completed or aborted refresh is released |-| `RecoveryOnlyByPolicy` | a refresh goes `recovery_required` (a person must act while every write deferred to it waits) only in the policy case recorded in `rwhy`: its merge started and HEAD no longer holds the target (`merge_started_off_target`: a reset plus an unrelated commit, or a plain-git commit that made the ff-only merge refuse); never for a merge that never started, nor with HEAD at or past the target |+| `RecoveryOnlyByPolicy` | a refresh goes `recovery_required` (a person must act while every write deferred to it waits) only in the policy case recorded in `rwhy`: its merge ran and HEAD no longer holds the target (`merge_started_off_target`: a reset plus an unrelated commit); never for a merge that never started or that git refused, nor with HEAD at or past the target || `ExpiryOnlyWhileWaiting` | an expiry ends only a write that still waits (its refresh unreleased, or its import not landed) || `RestoreLeavesNoOpenWrite` | a restore leaves no write admitted before it open (finding C) || `NoApplyAfterRevoke` | no write commits after its credential was revoked |@@ -203,7 +203,7 @@ every config killing the refresh at least once, and the workflow's path filter c| `ConsAdopt`, `ConsRelease` | #8708 `adoptLocalEdit`, then the claim's release || `ConsDone` | `completeWrite` of a refusal || `RefBegin` … `CompleteCommit` | worktree-refresh.ts: `fenceDraining`, the drain poll, `fenceAndMerge` (re-counts under `lockTopologyRows`), the ff-only merge under the commit lock, `bookkeeping`, `syncMembers`, `resumeOrAbandon`, and the v292 release trigger on completed or aborted |-| `Crash`, `RecoverBegin`, `RecoverCommit` | the refreshing process killed (its open transaction rolls back), then `recoverOne` from the owner's topology scan or `--resume`: draining aborts; fenced with HEAD at old_head aborts, and so does a fenced row whose merge never started (`outcome.merge_started` false: `rmstart`), whoever moved HEAD; fenced with HEAD holding the target (the target or a strict descendant) runs `bookkeeping`; syncing with HEAD at or past the target stays syncing; a `recovery_required` row whose HEAD holds the target runs `bookkeeping`, and otherwise stays; anything else is `markRecoveryRequired` |+| `Crash`, `RecoverBegin`, `RecoverCommit` | the refreshing process killed (its open transaction rolls back), then `recoverOne` from the owner's topology scan or `--resume`: draining aborts; fenced with HEAD at old_head aborts, and so does a fenced row whose merge never started (`outcome.merge_started` false: `rmstart`), whoever moved HEAD, and one whose merge started but never reached the target (`rmstart` with `rtgt` 0; the code reads the reflogs of HEAD and its branch since `outcome.merge_started_at`, and keeps `recovery_required` when it cannot); fenced with HEAD holding the target (the target or a strict descendant) runs `bookkeeping`; syncing with HEAD at or past the target stays syncing; a `recovery_required` row whose HEAD holds the target runs `bookkeeping`, and otherwise stays; anything else is `markRecoveryRequired` || `ExpRefBegin`, `ExpRefCommit` | `expireDeferredWrites`: the protocol, the refresh row re-read `released_at IS NULL FOR NO KEY UPDATE`, `completeWrite` || `ExpImpBegin`, `ExpImpRecheck`, `ExpImpCommit` | #8705's import expiry in `expireDeferredWrites` || `SyncBegin` … `SyncStamp` | a managed sync pass, whose rows go through the claim like writes: no write passes the queued row importing its page or a running sync row, and a sync row waits behind earlier writes and is never claimed while a write runs; the page row, then its checkpoint row's `UPDATE sources SET last_commit` (sync-prepare.ts) |@@ -233,6 +233,7 @@ every config killing the refresh at least once, and the workflow's path filter c| `import-expiry-locks-source-first` | guard | `NoDeadlock` (Import) | the import expiry's lock order; a Postgres expiry overlapping a fence (import-write-barrier.test.ts) || `import-expiry-skips-source-lock` | guard | `ExpiryOnlyWhileWaiting` (Import) | the import expiry's `FOR SHARE OF s`; a Postgres import landing while the expiry waits (import-write-barrier.test.ts) || `resume-marks-unmerged-recovery` | guard | `RecoveryOnlyByPolicy` | `movedBeforeMerge` (the finding below): the code edit aborts only a HEAD moved by gbrain's own commits; the refresh killed at fenced after a foreign commit, aborted by the owner scan |+| `resume-marks-refused-merge-recovery` | guard | `RecoveryOnlyByPolicy` | `refusedBeforeAbort` (GST-1674): a refresh killed after git refused its fast-forward, before the abort, is marked `recovery_required`; the refresh SIGKILLed right after the refused merge (persistence-worktree-refresh-restart.test.ts), aborted by the owner scan || `resume-needs-exact-target` | guard | `RecoveryOnlyByPolicy` | `recoverOne`'s bookkeeping branch: with HEAD only exactly at the target, a commit between the merge and bookkeeping strands the row; the refresh killed after its merge |#8708's local-edit adoption has no guard mutant of its own: `RefreshBarrierAdopt.cfg` checks it against every safety property.@@ -250,10 +251,15 @@ strands the row the same way (`resume-needs-exact-target`), since recovery wanteThe fix (#8705) records `outcome.merge_started`, false at the fence and set in its own transaction just before`git merge --ff-only`. Recovery aborts a fenced row whose merge never started, and runs `bookkeeping` for one whose HEAD-is the target or a strict descendant of it. `recovery_required` remains only for a merge that started on a HEAD that no+is the target or a strict descendant of it. `recovery_required` remains only for a merge that ran on a HEAD that nolonger holds the target (`merge_started_off_target`): a reset plus an unrelated commit (test 7 in-test/persistence-worktree-refresh.test.ts), or a plain-git commit between the HEAD check and the merge, which makes the-ff-only merge refuse. Every config passes with the fix, and that policy case stays reachable.+test/persistence-worktree-refresh.test.ts). Every config passes with the fix, and that policy case stays reachable.++GST-1674 closed the last other case: a plain-git commit between the HEAD check and the merge makes the ff-only merge+refuse. The live tick aborts it (#8765), and a process killed between the refused merge and that abort is aborted by+recovery when the reflogs of HEAD and its branch show HEAD never reached the target since `outcome.merge_started_at`+(`RefusedAbortable`, `resume-marks-refused-merge-recovery`). The model reads that fact from `rtgt` and assumes the+reflog is kept; without one the code keeps `recovery_required`, which leaves the row for a person, as before.### Races found in #8705diff --git a/spec/tla/refresh-barrier/RefreshBarrier.tla b/spec/tla/refresh-barrier/RefreshBarrier.tlaindex 44d7bf9a76..0184ff8ce1 100644--- a/spec/tla/refresh-barrier/RefreshBarrier.tla+++ b/spec/tla/refresh-barrier/RefreshBarrier.tla@@ -152,8 +152,9 @@ RefIs(S) == cur # None /\ rstate[cur] \in SWorktree == IF local # None THEN local ELSE fileV\* HEAD holds refresh i's target: its merge ran and HEAD is the target or a descendant (history is one line until a reset).HoldsTarget(i) == rtgt[i] # 0 /\ head >= rtgt[i] /\ ~lost[i]-\* markRecoveryRequired's case: a merge that started on a HEAD that no longer holds the target is policy; anything else is not.-Why(i) == IF rmstart[i] /\ ~HoldsTarget(i) THEN "merge_started_off_target" ELSE "unexplained"+\* markRecoveryRequired's case: a merge that ran (HEAD reached the target) on a HEAD that no longer holds it is policy;+\* anything else is not. GST-1674: a merge that started but never reached the target (git refused it) is not policy.+Why(i) == IF rmstart[i] /\ rtgt[i] # 0 /\ ~HoldsTarget(i) THEN "merge_started_off_target" ELSE "unexplained"Free(p, r, m) == \A q \in Procs \ {p} : ~Conflict(m, held[q][r])L(r, m) == <<r, m>>@@ -210,6 +211,10 @@ AdoptOneSided == fileV = dbV\* HEAD descends from old_head and the target is not in it, whoever made the commits. memberOutsideHead is FALSE here: no\* member syncs past HEAD.FencedAbortable(i) == ~rmstart[i]+\* GST-1674: recoverOne also aborts one whose merge started but whose HEAD never reached the target since (the reflogs of HEAD+\* and its branch show no move to target_head after outcome.merge_started_at): git refused the fast-forward, the process died+\* before the abort. Without a readable reflog the code keeps recovery_required; the model assumes the reflog is kept.+RefusedAbortable(i) == rmstart[i] /\ rtgt[i] = 0\* recoverOne runs bookkeeping for an orphaned fenced refresh whose HEAD holds the target (the target or a strict descendant).ResumeAtTarget(i) == HoldsTarget(i)@@ -549,7 +554,7 @@ MergeStartCommit ==cw, cprep, cfin, ew, aref, aread, passHead, sseq>>\* Plain git can still commit between the HEAD check and the merge (only gbrain's commits take the commit lock): the ff-only-\* merge then refuses with HEAD neither old_head nor the target, and the live tick marks recovery_required.+\* merge then refuses with HEAD on that commit, and the live tick aborts refresh_source_changed and retries (GST-1674, #8765).FfMerge ==/\ pc[RFR] = "r_ff"/\ IF head = rhead[nref]@@ -558,7 +563,7 @@ FfMerge ==ELSE NewCommit(fileV) /\ UNCHANGED <<nver, rmoved>>/\ rtgt' = [rtgt EXCEPT ![nref] = head + 1]/\ Need(RFR, <<L(REF(nref), "NKU"), L(SRC, "NKU")>>) /\ Go(RFR, "r_bk")- ELSE /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_mark") /\ UNCHANGED <<head, verAt, fileV, nver, rmoved, rtgt>>+ ELSE /\ Need(RFR, <<L(REF(nref), "NKU")>>) /\ Go(RFR, "r_abort") /\ UNCHANGED <<head, verAt, fileV, nver, rmoved, rtgt>>/\ UNCHANGED <<reqv, rstate, rel, rchg, rhead, rmstart, lost, rwhy, cur, nref, dbV, hist, lastCommit, local, nforeign, nlocal,envv, ghostv, held, cw, cprep, cfin, ew, aref, aread, passHead, sseq>>@@ -632,7 +637,7 @@ RecoverCommit ==/\ LET s == rstate[nref]IN CASE s \notin RefActive -> \* another process finished it (the admission that completes a synced refresh)/\ UNCHANGED <<rstate, rel, rwhy, cur>> /\ Go(RFR, "idle") /\ Need(RFR, <<>>)- [] s = "draining" \/ (s = "fenced" /\ (head = rhead[nref] \/ FencedAbortable(nref))) ->+ [] s = "draining" \/ (s = "fenced" /\ (head = rhead[nref] \/ FencedAbortable(nref) \/ RefusedAbortable(nref))) ->/\ rstate' = [rstate EXCEPT ![nref] = "aborted"] /\ rel' = [rel EXCEPT ![nref] = ReleaseOn("aborted")]/\ cur' = None /\ Go(RFR, "idle") /\ Need(RFR, <<>>) /\ UNCHANGED rwhy[] s = "fenced" /\ ResumeAtTarget(nref) -> \* bookkeepingdiff --git a/spec/tla/refresh-barrier/mutants.json b/spec/tla/refresh-barrier/mutants.jsonindex 466f8c487e..a05b1c8db2 100644--- a/spec/tla/refresh-barrier/mutants.json+++ b/spec/tla/refresh-barrier/mutants.json@@ -442,6 +442,32 @@]}},+ {+ "id": "resume-marks-refused-merge-recovery",+ "kind": "guard",+ "title": "recovery marks an orphaned fenced refresh recovery_required when git refused its fast-forward (a plain-git commit on old_head) and the process died before the abort (GST-1674 reverted)",+ "expect": "RecoveryOnlyByPolicy",+ "spec": {+ "file": "RefreshBarrier.tla",+ "before": "RefusedAbortable(i) == rmstart[i] /\\ rtgt[i] = 0",+ "after": "RefusedAbortable(i) == FALSE"+ },+ "code": {+ "edits": [+ {+ "file": "src/core/persistence/worktree-refresh.ts",+ "before": " const startedAt = row.outcome.merge_started_at;\n return row.state === 'fenced'",+ "after": " const startedAt = row.outcome.merge_started_at;\n return false && row.state === 'fenced'"+ }+ ],+ "tests": [+ {+ "test": "test/persistence-worktree-refresh-restart.test.ts",+ "filter": "a refresh killed after git refused its fast-forward recovers by aborting"+ }+ ]+ }+ },{"id": "resume-needs-exact-target","kind": "guard",diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex d4fc1e147c..3c8c87616a 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1478,6 +1478,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;// GST-1623 items 4 and 6-9, #8755. release:restamp stamps since at the merge slot.{ since: '0.60.611.0', id: 'doctor_foreign_writers_partial_gst1623', text: 'Doctor `foreign_writers` no longer reads what it could not scan as clean. A source root inside a Git checkout whose `git rev-parse` fails or times out, roots past the 50-root scan cap, a commit ledger that exists but cannot be read, and a `git status` that failed or timed out now mark the scan partial, each listed in the new `partial_reasons` and named in the message. A scan that read no repository but skipped one warns instead of reporting not_applicable. `git status` now gets up to 30 s within doctor\'s budget instead of the fixed git timeout. A foreign commit whose trailers include a `gbrain-` trailer still fails the check, and the message names it as possibly an older gbrain without the commit ledger (`gbrain_trailer: true`).' },{ since: '0.60.611.0', id: 'commit_push_enclosing_repo_gst1623', text: 'On a brain home that records its source checkouts, `gbrain git commit-push` without `--path` now also refuses `checkout_not_a_source` in a repository that only encloses a registered checkout (a clone above it): a registered checkout must contain the resolved one, or be a directory of it that its HEAD tracks. A checkout registry that cannot be listed now refuses `writer_coordinator_required` instead of an internal error.' },+ { since: 'TBD', id: 'refresh_refused_merge_recovery_gst1674', text: 'A `gbrain sources refresh` whose process died after git refused its fast-forward (a plain-git commit landed on the start commit right before the merge) and before the refresh aborted is now aborted by recovery (the owner\'s scan or `--resume`, and `--abandon` abandons it) with the fence lifted and that commit kept, instead of being marked `recovery_required` and holding the fence, and every write deferred behind it, until a person acted. Recovery reads the reflog of HEAD and its branch to tell this from a merge that ran and was then reset; without a reflog (`core.logAllRefUpdates` off) or for a refresh an earlier version started, it still marks `recovery_required`.' },{ since: '0.60.611.0', id: 'owner_guard_fail_closed_gst1623', text: 'The owner-path commit guard of `gbrain sources guard` now refuses the commit (exit 1, with a message) when `git rev-parse --absolute-git-dir` fails, instead of letting it through; a hook that belongs to another git dir still passes. A guard installed by an earlier version keeps its old script until `gbrain sources guard <source> --install` runs again.' },{ since: '0.60.611.0', id: 'adopt_local_edits_hardening_8755', text: 'With `persistence.adopt_local_edits` on for a source (off by default), a local edit is no longer adopted, and the write keeps its `file_database_drift` refusal, when its path has merge conflicts, while a merge, cherry-pick, revert or rebase is in progress, when the file equals one of the page\'s last 50 revisions or drops rows the current revision added (a stale editor save), when the page is segmented, or when the write\'s claim ended meanwhile. The quiet wait is capped at 30 s (it was up to three `persistence.adopt_quiet_ms` windows, 180 s at that key\'s cap), and an empty `persistence.adopt_quiet_ms` reads as the 5000 default instead of no wait. Doctor `local_edit_adoptions` still counts an adoption after its receipt is compacted.' },{ since: '0.60.611.0', id: 'managed_sync_kind_reserved_8765', text: 'A page write that passes a `kind` starting with `managed_sync_` (for example `capture` with `kind: managed_sync_import`) is now refused `invalid_params` and writes nothing: that prefix is reserved for managed sync, and with `persistence.refresh_barrier` on a waiting write now tells sync\'s own imports apart from later writes by the operation that wrote them, not by the kind. Use another kind, or drop `kind`.' },diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.tsindex d262d221b6..b5ecd37202 100644--- a/src/core/persistence/worktree-refresh.ts+++ b/src/core/persistence/worktree-refresh.ts@@ -179,6 +179,37 @@ async function movedBeforeMerge(engine: BrainEngine, git: Git, root: string, row&& await ancestry(git, root, row.old_head, head) === true && await ancestry(git, root, row.target_head, head) === false&& !await memberOutsideHead(engine, git, root, row, head);}+/** Reflog timestamps have one-second resolution, so an entry written just after merge_started_at can read up to a second before it. */+const REFLOG_SLACK_MS = 5_000;+/**+ * GST-1674: whether the reflogs of HEAD and the branch it names record a move to the target at or after `sinceMs` (less REFLOG_SLACK_MS):+ * true or false, or null when git fails or either reflog is missing, empty or does not end at `head` (core.logAllRefUpdates off, a deleted log).+ */+async function reachedTargetSince(git: Git, root: string, row: WorktreeRefreshRow, head: string, sinceMs: number): Promise<boolean | null> {+ const branch = await git(['-C', root, 'symbolic-ref', '-q', 'HEAD']);+ if (branch.code !== 0 && branch.code !== 1) return null;+ for (const ref of ['HEAD', ...(branch.code === 0 ? [branch.stdout.trim()] : [])]) {+ const out = await git(['-C', root, 'log', '-g', '--format=%H %gd', '--date=unix', ref, '--']);+ if (out.code !== 0) return null;+ const entries = out.stdout.split('\n').filter(Boolean).map(line => /^([0-9a-f]{40,64}) \S*@\{(\d+)\}$/.exec(line));+ if (!entries.length || entries.some(entry => !entry) || entries[0]![1] !== head) return null;+ if (entries.some(entry => entry![1] === row.target_head && Number(entry![2]) * 1000 >= sinceMs - REFLOG_SLACK_MS)) return true;+ }+ return false;+}+/**+ * GST-1674: a `fenced` row whose merge started (`outcome.merge_started_at`) but never moved HEAD to the target: a plain-git commit landed on+ * old_head right before the merge, git refused the fast-forward, and the process was killed before the live refresh aborted. HEAD descends+ * from old_head without the target, no member synced off it, and the reflogs show HEAD never reached the target since the merge started.+ * Any doubt (no merge_started_at on an older row, no readable reflog) leaves the row to markRecoveryRequired, as does a merge that ran and+ * was then reset (test 7).+ */+async function refusedBeforeAbort(engine: BrainEngine, git: Git, root: string, row: WorktreeRefreshRow, head: string | null): Promise<boolean> {+ const startedAt = row.outcome.merge_started_at;+ return row.state === 'fenced' && row.outcome.merge_started === true && typeof startedAt === 'number' && head !== null && head !== row.old_head+ && await ancestry(git, root, row.old_head, head) === true && await ancestry(git, root, row.target_head, head) === false+ && !await memberOutsideHead(engine, git, root, row, head) && await reachedTargetSince(git, root, row, head, startedAt) === false;+}const keepSyncedWayOut = (row: Pick<WorktreeRefreshRow, 'source_ids'>, root: string, head: string) =>`Ask the user first: git -C ${q(root)} merge --ff-only ${head} moves the checkout to the commit its members already synced (it keeps uncommitted edits git can carry; git -C ${q(root)} status --short lists them), then gbrain sources refresh ${row.source_ids[0]} --resume finishes the refresh and keeps the pages its members synced. If git refuses (an uncommitted edit overlaps an incoming file, or HEAD is not an ancestor of ${head}), ask the user before git -C ${q(root)} reset --hard ${head}, which discards local commits and uncommitted edits not in it, then --resume.`;/** GST-1457: the refusal when a member synced past old_head; its fix fast-forwards to the commit the members synced (ask_user, preview first), then --resume. */@@ -901,7 +932,7 @@ async function fenceAndMerge(engine: BrainEngine, git: Git, sourceId: string, pl// GST-1623: the merge runs only when this marked the row; a row recovery moved off `fenced` meanwhile is not merged over.const marked = await engine.transaction(async tx => {await tx.executeRaw("SELECT set_config('synchronous_commit','on',true)");- return tx.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=outcome||'{"merge_started":true}'::jsonb,updated_at=now() WHERE id=$1::uuid AND state='fenced' RETURNING id`, [row.id]);+ return tx.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=outcome||jsonb_build_object('merge_started',true,'merge_started_at',$2::bigint),updated_at=now() WHERE id=$1::uuid AND state='fenced' RETURNING id`, [row.id, Date.now()]);});if (!marked.length) throw refusal('refresh_source_changed', 'The refresh record was moved by recovery before the merge; nothing was merged.', `gbrain sources refresh ${sourceId} --resume`, row.id);let merge = await git(['-C', plan.root, 'merge', '--ff-only', '--no-edit', '--quiet', row.target_head], 120_000);@@ -975,6 +1006,10 @@ async function recoverOne(engine: BrainEngine, git: Git, row: WorktreeRefreshRowconst by = await ownCommits(git, root, row.old_head, head!) ? OWN_COMMITS : 'a commit or checkout gbrain did not make';const cause = `The refreshing process stopped before merging; HEAD moved from ${row.old_head} to ${head} (${by}) and nothing was merged.`;await engine.transaction(tx => transition(tx, row, ['fenced'], 'aborted', { refusal: { code: 'refresh_interrupted', cause }, observed_head: head }));+ } else if (await refusedBeforeAbort(engine, git, root, row, head)) {+ const by = await ownCommits(git, root, row.old_head, head!) ? OWN_COMMITS : 'a commit gbrain did not make';+ const cause = `The refreshing process stopped at its merge: HEAD moved from ${row.old_head} to ${head} (${by}) right before it, so git refused the fast-forward (or never ran it), and nothing was merged.`;+ await engine.transaction(tx => transition(tx, row, ['fenced'], 'aborted', { refusal: { code: 'refresh_interrupted', cause }, observed_head: head }));} else if (['fenced', 'merged', 'recovery_required'].includes(row.state) && head !== null && (head === row.target_head || await isAncestor(git, root, row.target_head, head))&& await isAncestor(git, root, row.old_head, row.target_head) && ((row.state !== 'recovery_required' && head === row.target_head) || !await memberOutsideHead(engine, git, root, row, head))) {await bookkeeping(engine, row);@@ -1103,7 +1138,7 @@ async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: Refrconst row = await readRefresh(engine, active.id);if (row.state === 'syncing') throw refusal('refresh_in_progress', 'The checkout already holds the upstream commit; abandoning now would leave files ahead of the database.',`gbrain sources refresh ${sourceId} --resume`, row.id);- if (row.state !== 'draining' && head !== row.old_head && !await movedBeforeMerge(engine, git, root, row, head)) {+ if (row.state !== 'draining' && head !== row.old_head && !await movedBeforeMerge(engine, git, root, row, head) && !await refusedBeforeAbort(engine, git, root, row, head)) {if (head === row.target_head) throw refusal('refresh_in_progress', `The checkout is at the refresh target ${row.target_head}; finish it instead of abandoning.`,`gbrain sources refresh ${sourceId} --resume`, row.id);throw await markRecoveryRequired(engine, row, head, root);diff --git a/test/helpers/worktree-refresh-child.ts b/test/helpers/worktree-refresh-child.tsindex 1cf5aceef7..583b94c9c9 100644--- a/test/helpers/worktree-refresh-child.ts+++ b/test/helpers/worktree-refresh-child.ts@@ -6,6 +6,8 @@* then `--resume` it and report the final state as one JSON line.* `kill-syncing` (#7679): start a refresh and SIGKILL this process once the row* is `syncing` (merged, member syncs not yet run), printing its pid first.+ * `kill-refused-merge` (GST-1674): start a refresh; the caller's `git` on PATH+ * SIGKILLs this process once git refused the fast-forward.*/import { randomUUID } from 'node:crypto';import { writeSync } from 'node:fs';@@ -18,7 +20,7 @@ import { disposePersistenceConsumer, startPersistenceConsumer } from '../../src/import { refreshWorktree } from '../../src/core/persistence/worktree-refresh.ts';import { assertSafeE2eDatabaseUrl } from './db-guard.ts';-const input = JSON.parse(process.env.GBRAIN_TEST_REFRESH_CHILD!) as { database: GBrainConfig; sourceId: string; worktreeId: string; mode: 'crash' | 'restart' | 'kill-syncing' };+const input = JSON.parse(process.env.GBRAIN_TEST_REFRESH_CHILD!) as { database: GBrainConfig; sourceId: string; worktreeId: string; mode: 'crash' | 'restart' | 'kill-syncing' | 'kill-refused-merge' };if (input.database.database_url) assertSafeE2eDatabaseUrl(input.database.database_url);const engine: BrainEngine = input.database.engine === 'pglite' ? new PGLiteEngine() : new PostgresEngine();await engine.connect(input.database);@@ -32,6 +34,9 @@ if (input.mode === 'kill-syncing') {process.kill(process.pid, 'SIGKILL');} } });process.stdout.write('REFRESH_UNEXPECTED_FINISH\n');+} else if (input.mode === 'kill-refused-merge') {+ await refreshWorktree(engine, input.sourceId).catch(error => process.stdout.write(`REFRESH_REFUSED ${(error as { code?: string }).code}\n`));+ process.stdout.write('REFRESH_UNEXPECTED_FINISH\n');} else if (input.mode === 'crash') {const requestId = randomUUID();await submitPageMutation({ engine, sourceId: input.sourceId, remote: false, dryRun: false, config: { engine: engine.kind, embedding_disabled: true } as never,diff --git a/test/persistence-worktree-refresh-restart.test.ts b/test/persistence-worktree-refresh-restart.test.tsindex bc8856e754..db8edf23a3 100644--- a/test/persistence-worktree-refresh-restart.test.ts+++ b/test/persistence-worktree-refresh-restart.test.ts@@ -6,23 +6,35 @@* the refresh to `syncing` (only because HEAD is the verified target and the* old HEAD is its ancestor), `--resume` completes it with every member at the* target commit, and the write accepted before the fence is committed.+ * GST-1674: a process killed after git refused the fast-forward (a plain-git+ * commit landed on the start commit right before the merge) and before the+ * abort recovers by aborting with that commit kept, so the fence lifts and no+ * deferred write waits on a person, and --abandon abandons it; without a reflog that shows HEAD never+ * reached the target after the merge started, it stays recovery_required.* Regression it catches: recovery adopting an unverified HEAD, recovery not* wired into owner startup, or accepted writes lost across the crash.- * Seams: a real child process exits inside the refresh's `merged` boundary.+ * Seams: a real child process exits inside the refresh's `merged` boundary,+ * or a `git` on PATH SIGKILLs it right after git refused its merge.*/import { afterAll, beforeAll, expect, test } from 'bun:test';import { spawnSync } from 'node:child_process';-import { mkdtempSync, rmSync } from 'node:fs';+import { randomUUID } from 'node:crypto';+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';import type { BrainEngine } from '../src/core/engine.ts';import type { GBrainConfig } from '../src/core/config.ts';import { PGLiteEngine } from '../src/core/pglite-engine.ts';-import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';+import { disposePersistenceConsumer, waitForWrite } from '../src/core/persistence/service.ts';+import { getWriteRequestById } from '../src/core/persistence/journal.ts';+import { resetWriteSwitches } from '../src/core/persistence/switches.ts';+import { refreshWorktree, resumeWorktreeRefreshes } from '../src/core/persistence/worktree-refresh.ts';+import { commitAndPush } from '../src/core/git/commit-push.ts';+import type { WriteRequest } from '../src/core/persistence/model.ts';import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';import { testBackends } from './helpers/test-backends.ts';import { withEnv } from './helpers/with-env.ts';-import { git, page, setupRefreshFixture } from './helpers/worktree-refresh-fixture.ts';+import { git, page, setupRefreshFixture, type RefreshFixture } from './helpers/worktree-refresh-fixture.ts';const backends = testBackends();const dir = mkdtempSync(join(tmpdir(), 'gbrain-refresh-restart-'));@@ -45,10 +57,10 @@ afterAll(async () => {rmSync(dir, { recursive: true, force: true });});-function child(env: Record<string, string>, payload: Record<string, unknown>): { status: number | null; stdout: string; stderr: string } {+function child(env: Record<string, string>, payload: Record<string, unknown>): { status: number | null; signal: NodeJS.Signals | null; stdout: string; stderr: string } {const run = spawnSync(process.execPath, [join(import.meta.dir, 'helpers/worktree-refresh-child.ts')],{ env: { ...process.env, ...env, GBRAIN_TEST_REFRESH_CHILD: JSON.stringify(payload) }, encoding: 'utf8', timeout: 90_000 });- return { status: run.status, stdout: run.stdout, stderr: run.stderr };+ return { status: run.status, signal: run.signal, stdout: run.stdout, stderr: run.stderr };}test('2. O-ENG-9: a crash after git merge and before bookkeeping converges after restart: syncing, then completed', async () => {@@ -86,3 +98,101 @@ test('2. O-ENG-9: a crash after git merge and before bookkeeping converges afterexpect(report.requests.every(row => row.state === 'committed')).toBe(true);}}, 300_000);++const FOREIGN_ON_TOP = "mkdir -p alpha && printf -- '---\\ntitle: Foreign\\ntype: note\\n---\\nPlain git.\\n' > alpha/foreign.md && git add alpha/foreign.md && git -c user.name=Example -c [email protected] -c commit.gpgsign=false commit -qm foreign";+type Row = { id: string; state: string; outcome: Record<string, any> };++/**+ * GST-1674: a refresh process SIGKILLed after git refused its fast-forward and before it aborted. The `git` on PATH commits on the start+ * commit right before the refresh's own `git merge --ff-only`, runs that merge (git refuses it), then kills the refresh process.+ * Runs `fn` with GBRAIN_HOME set and the barrier on, the row left `fenced` with merge_started true and HEAD on the plain-git commit.+ */+async function withRefusedMergeKill(index: number, fn: (k: { f: RefreshFixture; old: string; target: string; foreign: string; orphan: Row }) => Promise<void>): Promise<void> {+ const item = cases[index]!;+ const caseDir = mkdtempSync(join(dir, `refused-${index}-`));+ const home = join(caseDir, 'home');+ await withEnv({ GBRAIN_HOME: home, GBRAIN_REFRESH_BARRIER: '1' }, async () => {+ resetWriteSwitches();+ if (item.engine.kind === 'pglite') await item.engine.connect(item.database);+ const f = await setupRefreshFixture(item.engine, caseDir);+ await disposePersistenceConsumer(item.engine);+ const old = f.head();+ const target = f.push('alpha/two.md', page('Alpha two', 'Upstream.'));+ const bin = join(caseDir, 'kill-bin'), armed = join(caseDir, 'kill-armed'), script = join(caseDir, 'kill-race.sh');+ mkdirSync(bin, { recursive: true });+ writeFileSync(script, `cd '${f.root}' || exit 1\n${FOREIGN_ON_TOP}\n`);+ writeFileSync(join(bin, 'git'), `#!/bin/sh\ncase " $* " in *" merge --ff-only "*) if [ -f '${armed}' ]; then rm -f '${armed}'; PATH='${process.env.PATH}' sh '${script}' >/dev/null 2>&1; PATH='${process.env.PATH}' git "$@"; kill -9 $PPID; exit 1; fi;; esac\nPATH='${process.env.PATH}' exec git "$@"\n`, { mode: 0o755 });+ writeFileSync(armed, '');+ if (item.engine.kind === 'pglite') await item.engine.disconnect();+ const run = child({ GBRAIN_HOME: home, GBRAIN_REFRESH_BARRIER: '1', PATH: `${bin}:${process.env.PATH}` },+ { database: item.database, sourceId: f.alpha, worktreeId: f.worktreeId, mode: 'kill-refused-merge' });+ if (item.engine.kind === 'pglite') await item.engine.connect(item.database);+ try {+ expect(run.signal, `${run.stdout}\n${run.stderr}`).toBe('SIGKILL');+ const rows = await f.refreshRows() as Row[];+ expect(rows.map(row => [row.state, row.outcome.merge_started])).toEqual([['fenced', true]]);+ const foreign = f.head();+ expect(git(f.root, 'rev-parse', 'HEAD~1')).toBe(old);+ expect(git(f.root, 'log', '-1', '--format=%s')).toBe('foreign');+ expect(() => git(f.root, 'merge-base', '--is-ancestor', target, foreign)).toThrow();+ await fn({ f, old, target, foreign, orphan: rows[0]! });+ } finally { await disposePersistenceConsumer(item.engine); rmSync(caseDir, { recursive: true, force: true }); }+ });+}++test('GST-1674: a refresh killed after git refused its fast-forward recovers by aborting with the commit kept; its deferred write commits and the next refresh completes', async () => {+ for (const index of cases.keys()) await withRefusedMergeKill(index, async ({ f, target, foreign, orphan }) => {+ const requestId = randomUUID();+ const pending = await f.put(f.alpha, 'notes/held', page('Held', 'Written while the refresh was fenced.'), requestId).then(() => null, error => error as { code?: string });+ expect(pending?.code).toBe('write_pending');+ const [held] = await f.engine.executeRaw<WriteRequest>('SELECT * FROM persistence_requests WHERE request_id=$1::uuid', [requestId]);+ expect([held!.state, held!.deferred_refresh_id]).toEqual(['deferred', orphan.id]);++ await resumeWorktreeRefreshes(f.engine);+ const [row] = await f.refreshRows() as Row[];+ expect(row!.state).toBe('aborted');+ expect(row!.outcome.refusal).toMatchObject({ code: 'refresh_interrupted' });+ expect(row!.outcome.refusal.cause).toContain('refused');+ expect(row!.outcome.observed_head).toBe(foreign);+ expect(typeof row!.outcome.merge_started_at).toBe('number');+ expect(f.head()).toBe(foreign);++ git(f.root, 'config', 'user.name', 'Example'); git(f.root, 'config', 'user.email', '[email protected]'); git(f.root, 'config', 'commit.gpgsign', 'false');+ commitAndPush({ repo: f.root, cwd: f.root, pushOnly: true, keepCheckout: true });+ expect((await refreshWorktree(f.engine, f.alpha)).status).toBe('completed');+ expect((await f.refreshRows()).map(r => r.state)).toEqual(['aborted', 'completed']);+ const head = f.head();+ for (const commit of [foreign, target]) expect(git(f.root, 'merge-base', '--is-ancestor', commit, head)).toBe('');+ for (const source of [f.alpha, f.beta]) expect(await f.lastCommit(source)).toBe(head);+ const done = await waitForWrite(f.engine, (await getWriteRequestById(f.engine, held!.id))!, { engine: f.engine.kind } as never, 30_000);+ expect(done.state).toBe('committed');+ const [left] = await f.engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM persistence_requests WHERE worktree_id=$1::uuid AND state='deferred'", [f.worktreeId]);+ expect(left!.n).toBe(0);+ });+}, 300_000);++test('GST-1674: the same crash stays recovery_required when the reflog is gone or the row predates merge_started_at', async () => {+ for (const doubt of ['reflog', 'started_at'] as const) for (const index of cases.keys()) await withRefusedMergeKill(index, async ({ f, foreign }) => {+ if (doubt === 'reflog') {+ git(f.root, 'config', 'core.logAllRefUpdates', 'false');+ rmSync(join(f.root, '.git', 'logs'), { recursive: true, force: true });+ } else {+ await f.engine.executeRaw("UPDATE persistence_worktree_refreshes SET outcome=outcome-'merge_started_at' WHERE worktree_id=$1::uuid", [f.worktreeId]);+ }+ await resumeWorktreeRefreshes(f.engine);+ const [row] = await f.refreshRows() as Row[];+ expect([doubt, row!.state]).toEqual([doubt, 'recovery_required']);+ expect(row!.outcome.observed_head).toBe(foreign);+ expect(f.head()).toBe(foreign);+ });+}, 300_000);++test('GST-1674: --abandon on a refresh killed after git refused its fast-forward abandons it with the commit kept, not recovery_required', async () => {+ for (const index of cases.keys()) await withRefusedMergeKill(index, async ({ f, foreign }) => {+ const result = await refreshWorktree(f.engine, f.alpha, { abandon: true });+ expect(result.status).toBe('abandoned');+ const [row] = await f.refreshRows() as Row[];+ expect([row!.state, row!.outcome.refusal?.code]).toEqual(['aborted', 'refresh_abandoned']);+ expect(f.head()).toBe(foreign);+ });+}, 300_000);