Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Merge remote-tracking branch 'origin/master' into capy/gbra260-refresh-wave

garrytan564984bc7205
diff --cc CHANGELOG.md
index 9c72b849aa,7365190e50..45ac327b6d
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -10,18 -10,19 +10,31 @@@ credits are retained; no result has bee
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**`gbrain sources refresh` says what is actually holding an interrupted refresh, and its owner recovery stops holding locks it doesn't need (#8123).**
+
+Writer status for a `recovery_required` refresh now reads the checkout HEAD: at the upstream commit it names `--resume`, at the start commit `--abandon`, and only another HEAD asks the user to reset. An orphan whose HEAD git cannot read names `git -C <checkout> status` instead of promising that the next refresh finishes it, and the owner scan logs it once every 15 minutes instead of swallowing it. A refresh lock file this host cannot open reads `liveness: unknown` with the lock path, and a plain refresh refuses `refresh_in_progress` instead of failing with a raw lock error. A refresh held on another host names that host in `next`. A plain refresh that rolled back an orphan and then refused keeps that recovery in its refusal message. A delegated refresh (a resident PGLite owner) prints the same structured `fix` as a direct one.
+
+### Itemized changes
+
+- `src/core/persistence/worktree-refresh.ts`: the owner scan checks member progress before it takes the refresh lock, caches a syncing orphan it could not complete because of its HEAD, fails one row (not the scan) on a lock error, and prunes its caches each scan (#7712, #7728, #7730, #7731); `refreshHolderReport` reads HEAD (#7712, #7731); `next` is host-scoped and shell-quoted (#7731).
+- `src/core/persistence/ipc.ts`: a refusal's `fix`, `why`, `reason` and `contract_version` survive the owner socket (#7707, #8130).
+- Tests: the #7679 state matrix gains the `--abandon` action for every kind (#7718-#7727, #7729), plus red tests for the owner scan's syncing filter and test 7's adoption guard (#7708) and the unreadable-lock branch (#7728).
+
+ ## [0.60.530.0] - 2026-10-31
+
+ **`gbrain sources refresh` no longer exits with a bare `internal_error` when another database transaction holds a row its fence needs. A fence that waits past its 5 s lock timeout now refuses as `refresh_lock_timeout`, and `why` names the open transactions holding those tables. Nothing moves, and a later run completes once the holder is gone. Any other unexpected error now prints a typed `internal_error` refusal with its stack in `why` and the refresh row it left, which the next run resumes.**
+
+ Before, a stuck transaction holding a source, worktree or brain row (a long-running job, for example) made an upstream-moved refresh fail after about 5 s with `canceling statement due to lock timeout`. That surfaced as a top-level `internal_error` with no `status`, no stack and no holder. If it happened after the drain, it also left a `draining` row for the next run to clean up.
+
+ ### Itemized changes
+
+ - `src/core/persistence/worktree-refresh.ts`: Postgres `55P03` in either fence transaction (`fenceDraining`, `fenceAndMerge`) is the new `refresh_lock_timeout` refusal (registry, catalogue, `docs/guides/write-refusals.md`, `docs/guides/error-codes.md`). Its `why` lists the five oldest open transactions in the current database holding locks on `persistence_brain`, `persistence_worktrees`, `sources` or `persistence_worktree_refreshes` (pid, application, state, seconds open). These are candidates, and the blocker may not be among them, because `pg_locks` does not show which row each one holds; the fix says to confirm the blocker before ending any transaction. Plain readers, which cannot block the fence, are left out, the holders' SQL text is never copied into `why`, and a lookup that fails or takes longer than 3 s says so (the statement is cancelled at that bound, and after the drain the lookup runs only once the topology and worktree locks are released). A refusal after the drain aborts the draining row as other fence refusals do. That abort has its own 5 s lock timeout. When the same holder blocks it, the row stays `draining` for the next run instead of the refresh hanging, and the refusal says so.
+ - `src/core/persistence/worktree-refresh.ts`: `refreshWorktree` turns any other non-`OperationError` throw into `internal_error` with the error name, error code (a SQLSTATE or a Node code) and message (credentials, connection details and the home directory redacted), the stack in `why`, and a fix naming the active refresh row and its state, or saying the row could not be read (also when that lookup takes more than 3 s; its statements are cancelled at that bound). A deadlock (`40P01`), serialization failure (`40001`), cancelled statement (`57014`) or lock timeout outside the fence (`55P03`) is marked retryable. The fix repeats `--dry-run`, `--abandon`, `--resume`, `--wait-drain` and `--fetch-timeout-ms` when given. Home-directory redaction replaces only the home path itself, not a sibling path that shares its prefix. The row is not changed, and a host identity file that cannot be read still yields that refusal. A database access error the classifier recognizes (connection refused, a missing table) is rethrown unchanged, so it keeps its `gbrain db-repair` or `gbrain apply-migrations` fix. The already-current path no longer throws a `TypeError` of its own when a member sync rejects with a non-object.
+ - `src/commands/sources-refresh.ts`: refusals print `why` (a `why` key under `--json`, a `Why:` line otherwise).
+ - Test: `test/worktree-refresh-unexpected-throw.test.ts` (PGLite, and Postgres under `DATABASE_URL`; the lock-timeout cases hold a source row from a second connection, before the fence and after the drain, and check that a plain reader is not named and no SQL text is echoed; an unreadable host identity file still gives a typed refusal, an access error is not wrapped, credentials and the home directory are redacted without touching a sibling path, `--abandon` and `--dry-run` survive into the fix, a lookup that never answers still gives the refusal and is cancelled (row lookup and lock-holder lookup), transient SQLSTATEs are retryable, the holder list is worded as candidates, and an abort blocked by the same holder still returns and says the row stays draining).
+
## [0.60.529.0] - 2026-10-31
**A pull request that adds a file to the PGLite snapshot inputs no longer gets stopped at gstack's pregate secrets check by its own workflow cache keys (#8107).**
diff --cc docs/guides/write-refusals.md
index 96ad4aa2b8,de58eee842..5ac70addd8
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@@ -551,11 -551,12 +551,12 @@@ this anchor (`--json` puts them in one
| <a id="refresh_diverged"></a>`refresh_diverged` | `refresh_diverged` | The checkout has commits its upstream lacks, so a fast-forward is impossible. Nothing changed. | Ask the user, then publish the local commits merged with the upstream: `gbrain git commit-push --push-only --keep-checkout --path <checkout>` merges in memory, pushes the merge, refuses on a conflicting file and never moves the checkout (the refusal's fix fills in the real path and counts the local commits). Then retry `gbrain sources refresh <source>`, which fast-forwards to the merge. `gbrain sources reclone <source>` replaces the checkout instead and drops the local commits. |
| <a id="refresh_dirty"></a>`refresh_dirty` | `refresh_dirty` | Uncommitted or untracked files overlap the incoming upstream changes (the first 20 are named), or `git merge --ff-only` refused for the same reason. HEAD did not move. Uncommitted files outside the incoming changes do not refuse; they are kept and listed as `preserved_uncommitted`. | Commit or discard the named paths, then retry. Check `gbrain sources writer status <source> --json` first: a pending Git effect may be about to commit a gbrain-published file. |
| <a id="sync_in_progress"></a>`sync_in_progress` | `sync_in_progress` | A managed sync of one of the checkout's sources has an unfinished cursor. Moving the checkout now would strand it, and the refresh never waits on a sync. `gbrain repair frontmatter --apply` refuses the same way while an unfinished cursor still names a selected file. | Run the printed `gbrain sync --source <source> --no-pull ...` resume command (it keeps the cursor's options), then retry (for a repair, preview again and approve the new hash). |
-| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again). A refresh whose holder is recorded on another host is never taken over either: writer status shows its `liveness` as `unknown`, and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
+| <a id="refresh_in_progress"></a>`refresh_in_progress` | `refresh_in_progress` | Another refresh of the same checkout is active and its process is still running (the message names its holder's pid and host when the active refresh row is already readable), or a command that changes which sources share the checkout (`sources add`, `remove`, `archive`, `restore`, `reclone`) ran while one was active. A refresh whose process died is not a reason to refuse: the next refresh on the owner host finishes it first (`--dry-run` does not, so it still refuses while the orphan stands, and neither does a refresh that cannot read the checkout's HEAD: it leaves the orphan as it is and refuses until git can read HEAD again; writer status then names `git -C <checkout> status` instead of the next refresh). A refresh whose holder is recorded on another host, or whose lock file this host cannot open, is never taken over either: writer status shows its `liveness` as `unknown` (its `next` names the host to run on), and `gbrain sources refresh <source> --resume` on the owner host finishes it once that host's refresh is known to have stopped. Pending publication or topology recovery on the checkout refuses `refresh_recovery_required` first. An interrupted refresh that needs a reset decision refuses `refresh_recovery_required` instead. | Wait for the running refresh, or check it with `gbrain sources writer status <source>` (`liveness`, `holder`). `gbrain sources refresh <source> --resume` finishes an orphaned one now; then retry the source command with the same request ID. |
| <a id="refresh_drain_timeout"></a>`refresh_drain_timeout` | `refresh_drain_timeout` | Queued writes or Git effects on the checkout did not finish within `--wait-drain` (default 60 s; `sources.refresh_drain_wait_ms`; `GBRAIN_REFRESH_DRAIN_WAIT_MS`). The counts are in the message. The fence was lifted and HEAD did not move. | Retry, or wait longer with `--wait-drain <seconds>`. Inspect what is queued with `gbrain sources writer status <source> --json`. |
+ | <a id="refresh_lock_timeout"></a>`refresh_lock_timeout` | `refresh_lock_timeout` | The refresh fence locks the brain, worktree, source and refresh rows with a 5 s lock timeout, and another database transaction held one of them longer. `why` lists the five oldest open transactions in this database holding locks on those tables (pid, application, state, how long the transaction has been open). They are candidates: the blocker may not be among them, because Postgres does not show which row each one holds. It never repeats their SQL, and says so when that lookup failed or took longer than 3 s. HEAD did not move: no fence was set, or the draining refresh was aborted. When the same holder also blocked that abort, the refusal says the row stays `draining`, and the next `gbrain sources refresh <source>` resumes it. | Retry `gbrain sources refresh <source>`. If it repeats, a transaction holding one of those rows is stuck: `why` lists the oldest candidates, so confirm the blocker on the brain host before ending any transaction. |
| <a id="refresh_sync_timeout"></a>`refresh_sync_timeout` | `refresh_sync_timeout` | Reported per source in a refresh's `sync_blocked`, not as a refusal: the source's managed sync committed nothing within its share of the refresh sync budget (each member gets an equal share of what is left; `sources.refresh_sync_budget_ms`, `GBRAIN_REFRESH_SYNC_BUDGET_MS`, default 5 min), for example because no consumer claimed its write. The message names that write and its state. The checkout is at the upstream commit, the refresh completed and released its lock, and the source is behind until its next sync. A sync stopped by the drain's own rules is reported with that stop's code instead (`drain_stalled` and the others). | Run the `resume` command (`gbrain sync --source <source> --no-pull`). If it stalls the same way, check the write it names with `gbrain sources writer status <source> --json`. |
| <a id="refresh_source_changed"></a>`refresh_source_changed` | `refresh_source_changed` | HEAD moved (an outside commit or checkout), or the checkout's ownership or set of sources changed, between the precheck and the merge. Nothing was merged and the fence was lifted. | Retry `gbrain sources refresh <source>`. |
-| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. |
+| <a id="refresh_recovery_required"></a>`refresh_recovery_required` | `refresh_recovery_required` | Publication or source recovery is pending on the checkout, or a refresh was interrupted and HEAD is neither the commit it started from nor the verified upstream commit. Writes to the checkout stay refused until it is resolved; gbrain never adopts a HEAD it cannot verify. | `gbrain sources writer status <source>`, then `gbrain sources refresh <source> --resume`. When HEAD is unverified the fix names both ways out: reset to the start commit and run `--abandon`, or reset to the upstream commit and run `--resume`. Resetting is destructive, so ask the user first. Writer status reads HEAD and names the one step left when HEAD already sits at one of those commits: `--resume` at the upstream commit, `--abandon` at the start commit. |
| <a id="worktree_refreshing"></a>`worktree_refreshing` | `worktree_refreshing` | A write, or a managed sync started by a cycle or another command, reached a checkout while a refresh is draining or merging it. It was not journaled. `detail` is `retry_after_ms=1000`. Writes to database-only memory are not affected. | Retry the same request ID after about a second. `gbrain sources writer status <source>` shows the refresh. A cycle's sync is retried by the next cycle. |
A refresh is recorded in `persistence_worktree_refreshes` and survives a crash.