Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(sync): #8072 a held over-ceiling file is not re-read every tick; bound a refresh's member syncs

garrytanbf6cd31026b2
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index b303cb0d36..2297ddf2db 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -4,7 +4,7 @@
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
The #6188 fence check and fence repair live in [fence repair](core-fence-repair.md). Managed connector ingestion, connector item holds and the no-op kernel (checkpoint validation and the fault seams: [request indexes and fault seams](core-persistence-request-indexes.md)). Also the persistence verb types and errors, database-write and manual links, write attribution and loop-fact retirement; the managed purge of deleted pages is under [memory purge](memory-purge.md).
-- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` (every consumer topology scan) also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host, from an instant probe of the refresh lock) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command), which the refusal hints point at. Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured fix as the v1 `fix` RenderedAction when it has one, else the legacy command string; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.
+- `src/core/persistence/{worktree-refresh,worktree-refresh-schema}.ts` — `gbrain sources refresh` (F0), the one sanctioned managed Git writer. `persistence_worktree_refreshes` holds one row per refresh with a partial unique index (one active refresh per worktree; its 23505 maps to `refresh_in_progress`). `refreshWorktree` prechecks with no database state (managed, owner host, upstream, no active refresh, no recovery, no unexhausted managed-sync cursor, hardened `git fetch` bounded by `sources.refresh_fetch_timeout_ms`, ancestry, dirty paths intersected with the incoming diff), then: `draining` (inserted under `withTopologyLocks` + `lockTopologyRows`; admission in `journal.ts` refuses `worktree_refreshing` for every request with that `worktree_id`, and `assertManagedSyncAllowed` refuses a managed sync before discovery), poll until no queued/running/recovering request, no queued/running `git`/`withdrawal-mirror` effect and no recovery record remain (else `refresh_drain_timeout`, aborted), `fenced` (both native locks held, rechecked in one transaction; `refreshFenceClear` stops request and effect claims), HEAD verified, dirt rechecked, `git merge --ff-only`, HEAD verified again, then one transaction `merged` + upstream observation + `syncing`; native locks are released and the managed `--no-pull` sync runs per member in one budget (`sources.refresh_sync_budget_ms`; past it members report `deferred`, the row stays `syncing`) (only `managed_sync_*` `submit_job` admissions pass during `syncing`); `completed` when every member's `last_commit` is the target or descends from it, or with `outcome.sync_blocked` when a member sync blocks. A per-worktree `refresh-<worktree>.lock` held for the whole run proves liveness: `resumeWorktreeRefreshes` (called from `recoverSourceTopologies` on every consumer topology scan) touches interrupted `draining`/`fenced`/`merged` rows whose lock is free and applies the section 1.5 table (draining or fenced at the old HEAD abort; HEAD at the verified target moves to `syncing`; anything else is `recovery_required`, which admission refuses with `refresh_recovery_required`). Admission also completes a `syncing` row whose members all reached the target. #7679: each row records its holder (`outcome.holder`: host id, hostname, pid, process start; `previous_holder` after a recovery adopts it). A row whose lock is free is orphaned: `resumeWorktreeRefreshes` also completes an orphaned `syncing` row once every member's `last_commit` reached the target or a descendant, and a plain `gbrain sources refresh` on the owner host finishes any orphaned non-`recovery_required` row through the `--resume` path (`adoptOrphanedRefresh` + `resumeHeld`) before its own refresh, reporting it as `recovered`. A holder recorded on another host is never adopted. `refreshHolderReport` (writer status `worktree_refreshes`, sync status `worktree_refresh`) names the holder, `liveness` (`live` / `orphaned` / `unknown` off the lock's host) and a fix naming `--resume`. `--resume` converges and runs the member syncs; `--abandon` lifts a fence only with HEAD back at the start commit. `gbrain sources writer status` lists active refreshes as `worktree_refreshes` (state, heads, the next command). Doctor `worktree_refresh_stuck` lists rows active over 15 minutes. CLI body: `src/commands/sources-refresh.ts` (#7685: `--json` prints a refusal's structured `fix` when it has one; `refresh_in_progress` carries the writer-status read); while a resident PGLite owner (`gbrain serve`, stdio MCP) holds the database, `src/commands/sources-refresh-delegate.ts` runs the refresh inside it through the `writer_refresh` administration operation (trusted CLI registration only) and prints the same result or refusal. The consumer's idle probe applies `refreshFenceClear` to effects like the claims, so a fenced worktree's queued effects do not wake it. Tests: `test/persistence-worktree-refresh.test.ts`, `test/persistence-worktree-refresh-restart.test.ts` (child-process crash after merge), `test/worktree-refresh-orphan-7679.test.ts` (SIGKILL mid-syncing, live-holder no-steal), `test/oversize-hold-rescreen-8072.test.ts` (budget), Postgres via `test/e2e/worktree-refresh-postgres.test.ts`, owner delegation in `test/persistence-sync-stdio-owner.test.ts`.
- `src/core/persistence/git-gap.ts` + `src/commands/sources-git-gap.ts` + `src/commands/doctor/checks/db-ahead-of-git.ts` — git-canonical memory, wave G1 (`docs/designs/GIT_CANONICAL_MEMORY.md#the-git-gap-census`), read-only. `declaredDatabaseOnly` is the one predicate for pages that may not have a tracked file, tested in order: a read-only mirror source, a private page (`isPrivatePage`), an alias-private page (`isDerivationPrivate`), lineage-private (computed in SQL from `privatePagesFilterFragment`), `database_only_reason`, a `storage.db_only` prefix (`isSourceDbOnlySlug`, refusing an invalid gbrain.yml), a never-filed derive-phase page (page-prepare.ts `isNeverFiledDerivedPage`: no `source_path`/`source_uri` under `DERIVE_PHASE_DB_ONLY_DEFAULTS`), a path the checkout's Git ignores (`ignoredPaths`: filter-free `check-ignore -z --stdin`, tracked files never reported, exit 1 = none, null when Git cannot answer). The equality contract compares a committed blob with the page's export view (`readExportPage`, withdrawals overlaid; the census reads a batch's views with `readExportPages`, the same snapshots in two statements, #7617): an inline blob parsed as reconcile parses a canonical file, aligned with the stored type, subtype and #6429 stamp and compared by `reconcileCanonical` digest; a segmented page by the Git blob id of its streamed export. HEAD is read with hardened plumbing only (`readHeadTree`: one `ls-tree -r -l -z --full-tree` under the source's directory; `readBlobs`: `cat-file --batch` in calls of at most 64 MiB) and the working tree by hashing raw bytes (`worktreeBlobOid`), never `git status`/`git diff`, so no repository filter, textconv or fsmonitor runs. `gbrain sources git-gap <id> [--sample n] [--json]` (trusted local CLI, not an MCP op) puts every live page in one state (`equal`, `database_ahead`, `git_ahead` when the sync import record's `blob_oid`/`disk_blob_oid` differs from HEAD, else folded into `database_ahead` and counted in `database_ahead_without_import_record`, `declared_database_only` per class, `undeclared_database_only`, `worktree_only`, `unparseable`, `unsafe_path`), reports the working tree separately, and counts rows with no Git home (unfenced active facts, fenced facts on a missing page or a page without a committed file, links whose `link_source` is not in `DERIVED_LINK_SOURCES`, timeline rows on pages without a file), open `sync-hold` records by code and parked effect targets by kind and code. Each 256-page batch reads in its own read-only transaction. Doctor `db_ahead_of_git` runs the census per non-archived source whose directory is a Git checkout (others are only counted) under `GBRAIN_DOCTOR_GIT_GAP_TIMEOUT_MS` (default 10 s) and the entry deadline; it is always `ok` (informational) and lists totals only, a source the budget stopped or never reached as partial. Tests: `test/git-gap-census.test.ts` (PGLite and Postgres).
- `src/core/persistence/git-declared.ts` + `src/core/persistence/git-canonical-schema.ts` (migration `git_canonical_declared`) — git-canonical memory, wave G7a: the declared database-only classes of a strict source (`sources.config.canonical = "git"`, default off). `strictPageClass` judges a page by its slug and prior file (`declaredPathClass`: `db_only_prefix` for a `storage.db_only` dir, `derive_folder` for a never-filed derive-phase page; G3's indexer reads the same function) and then its rendered content (`privacyClass`: `private`, `alias_private`, or `lineage_private` from the same SQL remote reads use, `privatePagesFilterFragment`, over the page's own values). `page-prepare.ts` `prepareFileTarget` asks it before it builds a strict file target: a declared page gets no target (no file, no door commit, no Git effect), an untracked ignored target is `git_ignored`, a page turning private gets the removal of the file it still has in the worktree or at HEAD, and a path whose `filter` attribute is `lfs` passes `assertLfsTarget` only with `storage.git_lfs` and a checkout that configures the LFS filter (`git_lfs_not_enabled`). `preparePrivateTransition` (page-prepare.ts) is that removal for other writers (GBRA-60). In `publishMutation` and `publishGroup`, `assertNoPrivateCommit` refuses a file target carrying a private page's bytes before anything is written (`git_private_commit_refused`; a group rolls back whole), and `declareStrictPublication` (after apply, before the door commit) refuses a declared page's file or an undeclared file-less page (`git_home_required`), stamps `pages.database_only_reason` and, for a privacy class, clears `source_path`/`source_uri` in the transaction of the door commit that removed the file. A managed import builds its own file target, so it meets only these publication checks. `assertStrictGitHome` is the refusal for writers with no file form yet (G4, G5). The managed writer guard (`writer-guard-schema.ts`) refuses rows of a strict source unless the writer declares `gbrain.git_protocol` (`withCoordinatedWrite` does), behind the `git_canonical_brain.strict_sources` counter kept by `sources` triggers (`git_protocol_required`, mapped in `publication-failure.ts`). The guard reads its state as the writing role and fails closed: a hidden or missing `persistence_brain` or `git_canonical_brain` row, or a source or page row hidden by row-level security, raises `writer_coordinator_required` (`managed_writer_guard:unreadable_state`), and the counter refuses a recount under RLS on `sources` (`test/writer-fence-fail-closed-7636.test.ts`). The census (`git-gap.ts`) adds `database_only_source` (a source with no checkout) and maps the stamped reasons to their own classes. Tests: `test/git-canonical-declared-g7a.test.ts` (both engines).
diff --git a/docs/architecture/key-files/google-and-loops-continued.md b/docs/architecture/key-files/google-and-loops-continued.md
index 28ea569fdb..461d799b3a 100644
--- a/docs/architecture/key-files/google-and-loops-continued.md
+++ b/docs/architecture/key-files/google-and-loops-continued.md
@@ -15,7 +15,7 @@
- **`src/core/persistence/sync-window.ts`** — #5984 admit-ahead for draining managed Postgres syncs. While the cursor's `group` publishes, `admitAhead` (`sync-run.ts`) freezes and admits the next group and records it in the cursor's `window` (at most one group), so the consumer claims it as soon as the publishing group commits under the unchanged per-worktree FIFO claim. Window members name the previous group's last request in `intent.after`. #6712 (one page, one fate): a window group publishes whatever its predecessor became; nothing is cancelled for another page's failure, and dependency edges (`sync-group.ts` `dependencyKeys`, `freezeFollowers`' `taken`) keep two changes of one page out of the same group and out of the window while one is in flight. Publication validation (`sync-prepare.ts`) re-checks the predecessor once per transaction and reads the cursor `FOR KEY SHARE`, accepting `pending`, `group` and `window` members, so the drain's cursor saves never wait for the publishing group. After a failed page the members after it stay admitted at the head of the window, so every admitted request's identity stays in the cursor until its own receipt is read (`groupStep`, `promoteWindow`). Nothing is admitted ahead while foreground writes are recent (one was queued on the worktree in the last minute, `FOREGROUND_RECENT_MS`) or when the next entry is not groupable. Older binaries refuse a window member they did not admit (cursor membership), so a mixed-version drain fails closed instead of publishing twice; finish a drain before downgrading. A claimed head is ordered by `claimedPredecessorState`: a group member follows the member before it in its own group (its `after` names only the previous group), anything else follows `after`. `claimedHeadOrder` (the consumer, before claiming followers) only orders: under lanes a member whose predecessor member is unfinished releases its claim and waits; a predecessor that ended, committed or not, lets it publish (#6712). `WINDOW_CANCEL_MESSAGE` and `laneFaultCancelMessage` stay only so older receipts still read as legacy knock-ons. Test: `test/sync-group-member-order.test.ts`.
- **`src/core/persistence/sync-lanes.ts`** + **`worktree-lease.ts`** — #5984 lanes. The drain opens a lane policy for its worktree (`openLanes`, keyed by a per-drain run id that lane groups carry as `intent.lane`; `closeLaneRun` at drain end stops new lane claims and waits up to 90 s for the lane tasks still running, counted from the head's claim by `laneTask`). While it is open, this process's consumer claims the next admitted lane group head out of FIFO order (`claimNextLaneHead` in journal.ts: every earlier request claimed, any unfinished one a running request of the same run) up to the effective lane count (`laneRoots`), and runs it as a lane (`executeClaimedGroup` with `lane`). Lane publications share the worktree's native lock in this process (`acquireWorktreeShared` / `worktree-lease.ts`; one lane acquires the lock while lanes arriving meanwhile wait and join, and a lease stays registered until its lock is released); every exclusive `acquireWorktree` first yields the lease (marks it draining and wounded, waits up to its budget), and a lease drains for new claims after `LEASE_TURN_MS` of continuous holding. Lane transactions begin in manifest order (`awaitLaneBegin`) and commit in manifest order (`awaitLaneTurn`, inside the transaction before the counters: the predecessor `intent.after` must have ended, committed or not (#6712); a requeued or unadmitted one releases it, a wounded lease or 60 s rolls it back). Publication validation accepts a lane group whose predecessor is still queued or running (`windowPredecessorAllows`). `laneFallback` in group-publish.ts releases or falls back to the single path (#6712: never cancels), and a lock or statement timeout steps the lane count down (`stepDownLanes`). Lane settings (`--lanes`/`--no-lanes` > `GBRAIN_SYNC_LANES` > `sync.lanes` > 6, clamped to the pool's long-hold capacity minus 3) resolve in `resolveBulkSettings` (sync-group.ts); with lanes the drain keeps twice the lane count of groups admitted ahead (`admitAhead`).
- **`src/core/persistence/sync-run.ts`** + **`sync-authority.ts`** — one immutable request at a time, bounded fairness and pinned-target resumption. A completed discovery that finds nothing to do (`up_to_date`) stamps only `sources.last_sync_at`, inside `withCoordinatedWrite` and guarded by source id and incarnation, so a quiet managed source stays fresh in doctor and a run that outlived a re-added source stamps nothing; dry runs and partial runs do not stamp. Cursors and requests freeze `noEmbed`, `noExtract` and `noSchemaPack`. #6712 PR 1: a resume inherits the cursor's processing opt-outs OR-wise (`adoptCursor`, one CAS that also stamps the resumer's validated authority) and reports a flag it overrode in `inherited_options`; nothing refuses over processing options, and `explicitProcessing` is gone. Admitted requests keep their own intent principal: they are read under it (`admittedBy`), and a writer change drops only unadmitted frozen entries. Every managed pass runs under the source's drain lease (`withDrainLease`, `src/core/persistence/sync-lease.ts`), whose token fences admission, the settled/held CAS, fan-in and the cursor CAS (`assertDrainLease`). Explicit idle ordinary-source retry creates fresh discovery/admission and retains terminal receipts, also when the run has a recorded failure and its pending write already committed before the cursor advanced; company consent retry retains its approved manifest/run. Active work blocks replacement. Claimed/inactive sync, including connector/company dispatch, remains refused: existing leases and native locks do not fence already-running engine-only legacy writers. No automatic activation or new transition protocol. `freezeEntry` checks a rename's source page (id, revision, origin) as well as its target slug, and a committed rename counts in `renamed`. #5522: a new-file entry (`pageId: null`) whose origin another cursor of the source has since imported is re-frozen against that page when it is live, holds exactly that origin at the entry's slug in the cursor's incarnation, and the sync preparer finds its content unchanged; otherwise the origin refusal stands, and the manifest is never rewritten. #5751: the working-tree screen passes the kernel waivers for working-tree entries and counts them into `legacySkips` (printed by `sync-diagnostics.ts` with `gbrain repair contextual-mode`). A terminal `checkpoint_validation_timeout` receipt gets its hint from `checkpoint-validation.ts`. Content holds (#5988, `sync-screen.ts`): `freezeEntry` screens each import (company-profile cursors and `sync.holds=fail` excepted); a refused file becomes a hold written in the same transaction as the cursor step (`saveCursor`'s `inTx`), with no request admitted, and an over-bound pinned blob is held by size before it is read. A cursor whose pending request failed with a content refusal (`isContentRefusal`, old message shapes included) converts in place with its stored options: held, or re-frozen once per distinct bytes (`converted`), so a refusal the screen misses stays blocked without minting receipts. #6188: a failed fence refusal (`fenceReceiptLocation`: the durable `error_detail.fence`, the message, or an older release's message) holds the same bytes even when the screen admits them (`prepareTimeFenceHold`, reason `prepare_time`); a compacted `invalid_params` / `take_row_collision` receipt converts only when the re-screen holds. A page request of the run that fails with a fence refusal is held in the same invocation at the terminal-failure branch (single path and a bulk group's failed member) after the source's other requests settle within the run's wait budget, else the run returns `partial`; it is converted once, gets no failure-ledger row and is logged as a conversion. `sync.holds=fail` and company-profile cursors keep blocking with the typed refusal. #6278: a `preparation_stalled` receipt (the owner gave up preparing a member) is converted the same way by both converters: an import holds by its bytes, a delete as a delete hold (`meta.deleted`, `upstream_version` null), the checkpoint never holds; `preparationStalledHold` (sync-screen.ts) records the receipt's step and the resume arguments. Both converters write their hold through `holdUnderGate`, the narrowed gate: the failed entry's admitted suffix (window, later group members, this run's queued lane rows) is cancelled first, then the hold waits only for active suffix descendants and unfinished requests naming the entry's page (`UNFINISHED_PAGE_REQUEST_SQL`, now matching path and origin too), evaluated inside the transaction that writes the hold under the page-key → cursor lock order; its doc comment states the invariant the old source-wide wait protected. The cursor's `breaker` counts the run's stalled receipts (identities and streak, saved with each CAS): `holdsEscalated`'s rule over the run's screened imports or `BREAKER_STREAK` (5) in a row trips it, the run stops `blocked_by_failures` with `SyncResult.breaker` (drain `preparation_systemic`), and the next run's `convertBlockedCursor` re-freezes the entry and schedules the run's stalled holds for a re-screen instead of holding; a committed page resets the streak. `sameContentAtOrigin` and the waiver screens race the sync preparer against `persistence.sync_preparation_ms` (`raceSyncBudget`, sync-waivers.ts) and fail open. Tests: `test/managed-sync-preparation-stalled-holds.test.ts`, `test/sync-waiver-budget.test.ts`. Results carry this run's holds, the outstanding total, escalation, `converted_from_failed` and `recovered_frontmatter` (`buildHoldReport`); dry runs list `would_hold` / `screen_skipped`. After the checkpoint commits the pass stamps drain step `finish` and emits `managed_sync.finish`; then (unless `noExtract`) the owner derives links for the source's stale pages through `links-maintenance.ts` under `FINISH_BUDGET_MS` (pages left over stay stale for `gbrain extract --stale`), so a PGLite-delegated sync and a Postgres sync take the same path; the result carries `links`, and an extraction failure leaves pages stale without failing the committed sync. #6278 (B7): when the checkpoint commits while `sources retry-held` re-screens are scheduled for the source (scheduled against a backlog, or while this cursor ran), the pass returns `partial` / `writer_yield` instead of `synced`, so the drain's next pass retires the finished cursor and discovers again, taking them; a `queue_capacity` refused for outstanding requests (`isWriteCapacityWait`) is left out of the failure ledger because the drain waits it out. #6696: the cursor key comes from `selectManagedCursor` (`sync-cursor-key.ts`), which every managed caller crosses: a run without `--strategy` and without a company plan keys a fresh cursor with the source's configured strategy, while an unfinished cursor under the raw key keeps resuming under it, so its admitted requests keep matching. `SyncOpts.embedDeferred` (the cost gate's auto-defer) is frozen on the intent and treated as `noEmbed` by preparation, never as a processing option a resume adopts or compares (`test/persistence-sync-cursor-strategy-6696.test.ts`).
-- **`src/core/persistence/sync-holds.ts`** + **`sync-screen.ts`** + **`sync-blobs.ts`** — Git holds (#5988). One `op_checkpoints` row per held file (`sync-hold`, fingerprint `source:incarnation:path`) plus a per-source `sync-hold-summary` row (`count`, and `stale`: holds whose page exists); `writeGitHold` / `clearGitHold` lock the summary row and are conditional on the writing run's discovery time (`observed_at`), so an older run never replaces or clears a newer hold. Managed and legacy sync write the same rows; `readGitSourceHolds` (current incarnation only) feeds `retry-held` and legacy sync, and `readGitHoldListing` (count from the summary row, rows limited per source in SQL) feeds `sources status`, never `readAllSourceHolds`. `recordSyncConversion` logs each blocked request a managed run converted in place (`sync-conversions`, newest 20 per source, in the transaction that saves the converted cursor); `sources status` shows the newest 5. `gitHoldItem` gives every surface code, reason, key, line, location-only message, `fix` Action and `docs` anchor. #6188 `invalid_fence` holds carry `meta.fence` (reason, fence, section, rows, columns, section line), the file line of the first bad row (`meta.line`, from `fenceFileLine` when the screen holds it), `fence_version` and the last repair attempt (`fence_repair`); the summary row counts them (`fences`), and `holdRepairSteps` is the one router (D6) every surface prints: fence and slug-conflict holds (`HoldRepairRoute.fences` + `slug_conflicts`) preview the content-repair lane, `gbrain repair content --source <id>` (`contentRepairArgv`, #6377; the text says the next maintenance run repairs the fence and slug-conflict holds it can by itself when `FenceAutoRepair.active`), other holds preview `gbrain repair frontmatter`, a mixed source names both; one fence hold's own fix stays that file's `gbrain repair fences --only <path>` preview (`fenceHoldFix`). One fence hold's fix (`gitHoldFix` → `fence-repair/hold-fix.ts` `fenceHoldFix`) is that file's `--only <path>` preview with the step its state calls for, and `gitHoldItem` carries the D16 `fence` object (classes, tier, `auto_retry`, `next_attempt_after`); callers that can read the maintenance state pass it (`fenceAutoRepairFor`). `holdRescreenDue` is the one re-screen predicate managed discovery and legacy sync share (retry request, database-dependent codes, an older frontmatter reader or fence screen). Policy keys: `sync.holds` (`hold`/`fail`), `sync.hold_cap` (detail shown, default 500), `sync.hold_escalate_count` (50), `sync.hold_escalate_pct` (5, from 40 screened imports), `sync.parser_regression` (`stop`/`hold`). #6278: `preparation_stalled` holds carry `meta.stall` (request, step, `waiting_on`, attempts, the writing gbrain version, the sync's resume arguments) and count under `stalled` in the summary row and `HoldRepairRoute`; `gitHoldFix` routes them to `gbrain sources writer status --source <id> --json`, then `sources retry-held`, then the option-preserving sync (`stalledHoldSteps`), never to frontmatter repair, and `holdRescreenDue` re-screens them on a gbrain version change; `buildHoldReport` routes the escalation text by code. `sync-import-provenance` records (`recordSyncImportProvenance`, batched `recordSyncImportProvenances`) also carry `disk_sha256` / `disk_blob_oid` (the bytes the write left on disk), `revision` and `parser` (`syncImportParser`: `SYNC_IMPORT_PARSER_VERSION`, Markdown chunker, schema pack; never the release); records from before them, or from a repair, lack them and readers take them as unknown. `screenFrozenImport` builds its input with `frozenImportScreenInput` (rename base, the run's pack, sanity and purge tombstones) and screens it with `screenFrozenContent`, both shared with the slug-conflict repair so it judges a held file exactly as the next sync would; it admits a fence Tier 1 fixes (`fences.normalize`, read once per run only when needed; the dry run lists it in `would_normalize`), turns a refusal into a hold, or throws `sync_parser_regression` only when `sync-import-provenance` proves a validating gbrain imported the same bytes. `sync-blobs.ts` reads blob ids/sizes with chunked `ls-tree -l` and contents with `cat-file --batch`; `freezeEntry` reads each entry's pinned blob and content through `readPinnedBlob` / `readPinnedContent` (via `sync-screen.ts` `pinnedBlob` / `pinnedContent`): a miss lists that entry and up to 255 of the next committed imports in one `ls-tree` and loads their contents (8 MiB slices cut by window position, not read order; over-bound blobs excluded) in one `cat-file --batch`, kept per (Git root, full commit id) until the next window, so a managed first sync spawns a few Git processes per window instead of an `ls-tree` and a `git show` per page; a non-commit target, a window git refuses to list, or a blob the window cannot answer falls back to the per-entry read (`__setPinnedWindowForTests(0)` restores per-entry reads; `test/managed-sync-pinned-reads.test.ts` proves identical rows on PGLite and Postgres). `held-reads.ts` is the read side: `get_page.file_held`, `stale` on search/query hits and the `held_files` retrieval meta and notice, from one summary read per call, cached per request, plus, when the scope holds files and the call returned hits, one read for the hits' holds (by page id, `op_checkpoints_sync_hold_page_idx`, migration v200) and the count of holds that could match the query (`stampHeldHits`; the notice names it, also names embedding-held pages, and escalates); `stalled` coverage routes a `preparation_stalled` hold to the host operator's writer-status fix instead of a file repair; remote callers get codes and counts, never paths, and a fix addressed to the brain host operator (a held fence page gets `fenceHoldRelay`: the owner-host command and whether it clears by itself). Tests: `test/sync-holds-store.test.ts`, `test/persistence-sync-holds.test.ts`, `test/sync-all-hold-notes.test.ts`, `test/held-read-signals.test.ts`. #5493/#6529: managed discovery admits image imports whenever multimodal embedding is on (company plans refuse). Off, `freezeEntry` holds each image import as `managed_image_sync_unsupported` (`managedImageHold`, dry-run screen too) before any content read, whatever `sync.holds` says: the rest of the source imports, the hold is keyed by the pinned blob (or working bytes), and `holdRescreenDue` re-screens it on change, removal, multimodal on or `sources retry-held`. On, `freezeEntry` freezes the bytes as base64 (`pinnedImageBytes`, `frozenImageIntent`), an oversize image takes `oversizeHold`, identical bytes are a no-op by hash, and `sync-prepare.ts` routes the entry to `sync-image.ts`. The summary row counts `images`; those holds never count toward `holds_escalated` or `git_held_files`.
+- **`src/core/persistence/sync-holds.ts`** + **`sync-screen.ts`** + **`sync-blobs.ts`** — Git holds (#5988). One `op_checkpoints` row per held file (`sync-hold`, fingerprint `source:incarnation:path`) plus a per-source `sync-hold-summary` row (`count`, and `stale`: holds whose page exists); `writeGitHold` / `clearGitHold` lock the summary row and are conditional on the writing run's discovery time (`observed_at`), so an older run never replaces or clears a newer hold. Managed and legacy sync write the same rows; `readGitSourceHolds` (current incarnation only) feeds `retry-held` and legacy sync, and `readGitHoldListing` (count from the summary row, rows limited per source in SQL) feeds `sources status`, never `readAllSourceHolds`. `recordSyncConversion` logs each blocked request a managed run converted in place (`sync-conversions`, newest 20 per source, in the transaction that saves the converted cursor); `sources status` shows the newest 5. `gitHoldItem` gives every surface code, reason, key, line, location-only message, `fix` Action and `docs` anchor. #6188 `invalid_fence` holds carry `meta.fence` (reason, fence, section, rows, columns, section line), the file line of the first bad row (`meta.line`, from `fenceFileLine` when the screen holds it), `fence_version` and the last repair attempt (`fence_repair`); the summary row counts them (`fences`), and `holdRepairSteps` is the one router (D6) every surface prints: fence and slug-conflict holds (`HoldRepairRoute.fences` + `slug_conflicts`) preview the content-repair lane, `gbrain repair content --source <id>` (`contentRepairArgv`, #6377; the text says the next maintenance run repairs the fence and slug-conflict holds it can by itself when `FenceAutoRepair.active`), other holds preview `gbrain repair frontmatter`, a mixed source names both; one fence hold's own fix stays that file's `gbrain repair fences --only <path>` preview (`fenceHoldFix`). One fence hold's fix (`gitHoldFix` → `fence-repair/hold-fix.ts` `fenceHoldFix`) is that file's `--only <path>` preview with the step its state calls for, and `gitHoldItem` carries the D16 `fence` object (classes, tier, `auto_retry`, `next_attempt_after`); callers that can read the maintenance state pass it (`fenceAutoRepairFor`). `holdRescreenDue` is the one re-screen predicate managed discovery and legacy sync share (retry request, database-dependent codes, an older frontmatter reader or fence screen); a streamed `file_too_large` hold (`meta.streamed`: read whole) re-screens on change, retry or a new gbrain (#8072). Policy keys: `sync.holds` (`hold`/`fail`), `sync.hold_cap` (detail shown, default 500), `sync.hold_escalate_count` (50), `sync.hold_escalate_pct` (5, from 40 screened imports), `sync.parser_regression` (`stop`/`hold`). #6278: `preparation_stalled` holds carry `meta.stall` (request, step, `waiting_on`, attempts, the writing gbrain version, the sync's resume arguments) and count under `stalled` in the summary row and `HoldRepairRoute`; `gitHoldFix` routes them to `gbrain sources writer status --source <id> --json`, then `sources retry-held`, then the option-preserving sync (`stalledHoldSteps`), never to frontmatter repair, and `holdRescreenDue` re-screens them on a gbrain version change; `buildHoldReport` routes the escalation text by code. `sync-import-provenance` records (`recordSyncImportProvenance`, batched `recordSyncImportProvenances`) also carry `disk_sha256` / `disk_blob_oid` (the bytes the write left on disk), `revision` and `parser` (`syncImportParser`: `SYNC_IMPORT_PARSER_VERSION`, Markdown chunker, schema pack; never the release); records from before them, or from a repair, lack them and readers take them as unknown. `screenFrozenImport` builds its input with `frozenImportScreenInput` (rename base, the run's pack, sanity and purge tombstones) and screens it with `screenFrozenContent`, both shared with the slug-conflict repair so it judges a held file exactly as the next sync would; it admits a fence Tier 1 fixes (`fences.normalize`, read once per run only when needed; the dry run lists it in `would_normalize`), turns a refusal into a hold, or throws `sync_parser_regression` only when `sync-import-provenance` proves a validating gbrain imported the same bytes. `sync-blobs.ts` reads blob ids/sizes with chunked `ls-tree -l` and contents with `cat-file --batch`; `freezeEntry` reads each entry's pinned blob and content through `readPinnedBlob` / `readPinnedContent` (via `sync-screen.ts` `pinnedBlob` / `pinnedContent`): a miss lists that entry and up to 255 of the next committed imports in one `ls-tree` and loads their contents (8 MiB slices cut by window position, not read order; over-bound blobs excluded) in one `cat-file --batch`, kept per (Git root, full commit id) until the next window, so a managed first sync spawns a few Git processes per window, not two per page; a non-commit target, a window git refuses to list, or a blob the window cannot answer falls back to the per-entry read (`test/managed-sync-pinned-reads.test.ts` proves identical rows on PGLite and Postgres). `held-reads.ts` is the read side: `get_page.file_held`, `stale` on search/query hits and the `held_files` retrieval meta and notice, from one summary read per call, cached per request, plus, when the scope holds files and the call returned hits, one read for the hits' holds (by page id, `op_checkpoints_sync_hold_page_idx`) and the count of holds that could match the query (`stampHeldHits`; the notice names it, also names embedding-held pages, and escalates); `stalled` coverage routes a `preparation_stalled` hold to the host operator's writer-status fix instead of a file repair; remote callers get codes and counts, never paths, and a fix addressed to the brain host operator (a held fence page gets `fenceHoldRelay`: the owner-host command and whether it clears by itself). Tests: `test/sync-holds-store.test.ts`, `test/persistence-sync-holds.test.ts`, `test/sync-all-hold-notes.test.ts`, `test/held-read-signals.test.ts`. #5493/#6529: managed discovery admits image imports whenever multimodal embedding is on (company plans refuse). Off, `freezeEntry` holds each image import as `managed_image_sync_unsupported` (`managedImageHold`, dry-run screen too) before any content read, whatever `sync.holds` says: the rest of the source imports, the hold is keyed by the pinned blob (or working bytes), and `holdRescreenDue` re-screens it on change, removal, multimodal on or `sources retry-held`. On, `freezeEntry` freezes the bytes as base64 (`pinnedImageBytes`, `frozenImageIntent`), an oversize image takes `oversizeHold`, identical bytes are a no-op by hash, and `sync-prepare.ts` routes the entry to `sync-image.ts`. The summary row counts `images`; those holds never count toward `holds_escalated` or `git_held_files`.
- **`src/core/persistence/file-repair.ts`** — `managed_file_repair` (#5988), the owner-internal `put_page` kind `gbrain repair frontmatter --apply` submits (accepted only from the trusted local CLI with `managedFileImport`; MCP is refused by the reserved-field gate, remote authority by the preparer). `managedRepairRoot` resolves the owner through `checkOwner` (`owner-refusal.ts`) and refuses `owner_unavailable` with the condition as its reason; `submitManagedFileRepair` re-reads the source's `RepairBusySet` (`repair-busy.ts`) right before admission and refuses `sync_in_progress` when a write in flight or an unfinished sync's frozen manifest names the path, origin, slug or a rename endpoint (#6278: a sync can start while a fence repair waits on its model call). The preparer confines the path (`isWriteTargetContained` plus a component-wise `lstat` walk, no symlinks), requires the file's sha256 to equal the previewed `beforeHash` (`changed_since_preview` otherwise), recomputes `prepareRepairPublication` (the `screenSyncImport` + `importFromContent` preparation a managed sync import gets) and refuses a different result digest or a canonical overlay. It publishes the exact bytes as `file` (`expectedBeforeHash` = previewed hash; no `file_database_drift` check), applies the import (a held new file creates its page, a `renameFrom` moves the page with `updateSlug`), clears the Git hold (only while it is the `holdObservedAt` observation when the intent carries one, so a hold a newer sync wrote stays) and records `sync-import-provenance` in the same transaction; the file write queues the Git target effect. An import of a segmented body seals itself, so the inline text seal is skipped and its `afterCommit` is forwarded. A fence repair (`gbrain repair fences`) passes its `FenceRepairReceipt` as `fenceRepair`: submission and the preparer refuse (`invalid_params`, nothing written) a receipt `parseFenceRepairReceipt` rejects or whose `before_sha256`/`after_sha256` are not the previewed and written bytes; the outcome carries it as `fence_repair`, the file target carries the commit note from `fenceRepairCommit` (`effect-model.ts`), and every refusal and fix names `gbrain repair fences --source <id>` instead of `gbrain repair frontmatter` (`FileRepairKind`, also taken by `confinedRepairTarget` and `managedRepairRoot`). Tests: `test/repair-frontmatter.test.ts`, `test/fence-repair-receipts.test.ts`.
- **`src/core/persistence/sync-prepare.ts`** — guarded raw fingerprint/revision, source root and active coordinator checks before preparation and at locked publication. A Markdown import's preparation carries `importProvenance` (`SyncPreparedMutation`); its publication writes it in the publish transaction with the page id, `revision` and the disk half (`syncDiskProvenance` over the canonical writeback, else `syncDiskBytes`), and a no-op waiver writes the same record instead. The checkpoint (`applyManagedSyncCheckpoint`) keeps a repository-root source's pinned slug mode (`sync-slug-mode.ts`). Old imports/checkpoints without durable processing consent refuse under their original IDs; old deletions require their immutable origin manifest and a still-matching page origin. `noEmbed` suppresses the effect itself; `noSchemaPack` bypasses pack loading; optional extraction stays disabled, but mandatory provider-free canonical projections remain. A filename-derived slug mismatch is accepted on every platform only when the page's stored canonical path is this file and the parsed slug is that page's slug; a different declared slug still refuses. CRLF equivalence never weakens raw CAS. No uncoordinated filesystem writes. The checkpoint validates receipts through `checkpoint-validation.ts` and, for a run whose entries another cursor imported (`overtaken`, #5522), also commits when that cursor already checkpointed the same target. Markdown imports pass `screenSyncImport` (the already-published working-tree exemption, then `screenImportContent` with the expected slug and recorded-origin exemption), the same screen the freeze-time hold screen runs, so a file is held exactly when publication would refuse it. Each import, deletion and no-op apply clears the path's Git hold (`holdObservedAt`), the checkpoint clears `releasedHolds` and ignores `supersededRequests` (failed content refusals the run converted in place), and every Markdown import records `sync-import-provenance` (raw sha256, blob, gbrain version, quoting recoveries) and reports `recovered_frontmatter` / `comment_value` in its outcome. Tests: `persistence-sync-origin-native.serial.test.ts`, `persistence-sync-options.serial.test.ts`, `persistence-sync-company.serial.test.ts`. A same-slug `renameFrom` accepts the page's recorded old origin. A different-slug rename prepares against the page where it stands, locks the old slug as an additional page key, and at publication moves the page with `updateSlug` (same id, inbound links and history, `old -> new` in `slug_aliases`, withdrawn claims follow), re-infers an implicit type from the new path, then re-prepares and applies the file content at the new slug, so the move never writes back to the file. Tests: `persistence-managed-lifecycle.test.ts`. `syncGitTopLevel` answers the checkout's `git rev-parse --show-toplevel` once per root and sync run, reading it again when the top level's `.git` entry is replaced or removed or a `.git` appears between it and the root. #6278: `prepareManagedSyncMutation` calls `enterClaimStep` at each await boundary (`managed_sync_active`, `authority`, `worktree_binding`, `configured_root`, `knowledge_publication`, `manifest_entry`, `git_rev_parse`, `origin_check`, `page_snapshot`, `import_screen`, `import_content`, `canonical_projections` and the rest, each naming `git`, `fs` or `db` where it knows), so a stall reports its step; the consumer's budget and the cancellation reaching those boundaries are in persistence-stalls.md. An intent with `embedDeferred` (#6696) prepares as `noEmbed` whatever the cursor's stored options say.
- **`src/core/persistence/links-maintenance.ts`** — `extractManagedStaleLinks` (then the mention pass unless `mentions: false`, which sync passes) derives links for pages whose extraction watermark is stale with `prepareAutomaticLinks` (put_page's replace contract; other producers' edges stay) and adds the coordinator's unrecorded canonical timeline tuples (insert-only) and each page's watermark in the same `withCoordinatedWrite` transaction, bound to the revision it read; an edited page or unresolved attendance stays stale. A page skipped for unresolved attendance is marked attendance-blocked at that revision (`pages.links_attendance_blocked_revision` / `_at`, written by `markPagesAttendanceBlocked` in `engine-sql/pages.ts` only while the revision still matches, cleared by `markPagesExtractedBatch` with the watermark); it is still reconsidered every run, and the `links_extraction_lag` doctor check counts it apart from lag while the marker equals the page's current revision. GBRA-130 A-1 (site `links.extract_stale`, one `ItemRun` per source): a failing page is reported (`failed`, `failures`) and skipped; a run-scoped error stops the run after in-flight pages settle, recording nothing; a tripped breaker stops a scoped run (`item_failures_systemic`) or skips that source in a brain-wide run (`systemic`). Content failures are counted and held by `links-extraction-holds.ts`; a page of a deleted source fails `unknown_source` (`PageSourceMissingError`). The sync tail inherits this. It is the one managed extraction path: managed sync, `extractStaleFromDB` on a managed brain, and the owner delegation all run it. `runManagedStaleExtraction` backs `gbrain extract --stale` on managed brains, locally or through the `writer_extract_stale` administration operation inside a PGLite owner (`src/commands/extract-stale-delegate.ts`, trusted CLI registration only). `links-preparation.ts` retargets a link written against a renamed page's old slug to the live canonical page through `slug_aliases`, as `get_page` does. Tests: `persistence-managed-lifecycle.test.ts`, `persistence-sync-stdio-owner.serial.test.ts`, `test/pathological/a1-links-extract.test.ts` (both engines).
diff --git a/docs/guides/live-sync.md b/docs/guides/live-sync.md
index f532f585b0..0f2bdc09c3 100644
--- a/docs/guides/live-sync.md
+++ b/docs/guides/live-sync.md
@@ -653,6 +653,11 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
+ The member syncs of one run stop starting work after
+ `sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
+ so a scheduled run never holds the refresh lock much longer than that:
+ a run past it reports `syncing` (members `deferred`) and the next
+ `gbrain sources refresh <source-id>` continues where it stopped.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.md
index a918b2cda4..ef4f658c03 100644
--- a/docs/guides/write-refusals.md
+++ b/docs/guides/write-refusals.md
@@ -245,7 +245,7 @@ with each file and fix the ones that need no guessing."*
| <a id="invalid_frontmatter-ambiguous_identity_key"></a>`invalid_frontmatter` (`ambiguous_identity_key`) | `slug`, `type`, `id` or `source_id` appears twice, or its line was swallowed into another value, so gbrain cannot tell which page the file is. | Edit the file by hand: keep exactly one line for the key. The user decides which value is right. |
| <a id="invalid_frontmatter-ambiguous_protected_key"></a>`invalid_frontmatter` (`ambiguous_protected_key`) | A key that decides access or provenance (`visibility`, `derived_from`, …) is malformed, duplicated, swallowed into another value, rewritten by quoting, or inside an unclosed fence. gbrain never reads such a key as a broader value. | Edit the file by hand: write the key on one line with one quoted value. Ask the user which visibility is intended; never guess it. |
| <a id="frontmatter_slug_conflict"></a>`frontmatter_slug_conflict` | The file's frontmatter `slug:` names a different page than its path does (earlier releases reported this as `invalid_params`). The path decides the slug. When the named slug has no page and no file (#6865), the managed sync itself removes the line after it releases its drain lease, imports the file under its path slug, clears the hold, commits the edit and keeps the declared slug as an alias; a file edited since the hold, an unfinished sync or a remote job waits for the next pass (`sync.slug_conflicts.auto_remove false` turns it off; `slug_lines_removed` counts it). The hold has three outcomes once the content-repair lane's `slug-conflicts` kind runs it (the maintenance run's `content_repair` phase, `gbrain sync unblock --apply`, or `gbrain repair content --source <source>`): **removed** (the named slug resolves to no page and names no other file, or to a page of a different type sharing no title token: the stray `slug:` line is removed, the file imported and committed, the hold cleared, with a receipt; a model under the `fences.repair` caps makes the same call when the two pages are not obviously different), **merge recommended** ([`merge_recommended`](#merge_recommended): the two files describe the same thing; gbrain recommends the merge and names the canonical page but never merges pages by itself), or **undecided** ([`content_repair_needs_human`](#content_repair_needs_human)). The lane judges the file with the screen managed sync froze it with: a hold the sync screen now admits as the file is (its slug line names the file's own page as written, or it no longer declares a conflicting slug) is **imported** unchanged in one guarded write that clears only the hold observation the lane read (a hold a newer sync wrote stays, with a retry), and a file sync would hold whatever the lane writes is [`slug_page_mismatch`](#slug_page_mismatch). Until that kind ships, the lane leaves the hold to the frontmatter repair preview below. | Nothing to run when the lane removes the line: the hold clears on the sync it prints. For a recommendation or an undecided hold, `gbrain sync status --source <source> --json` carries the paragraph (`human_reason`): merge the unique sections into the canonical page by hand or decide which page keeps the slug, delete or edit the file, commit, then `gbrain sync --source <source> --no-pull`. `human_reason` asks to "decide which page keeps the slug" only when the line names a live page (a database-only page is labelled) and names any other frontmatter fixes; a hold with no verdict never asks a person, and an older gbrain's verdict is rechecked by the next sync or maintenance run. By hand at any time: remove the `slug:` line or make it match the path, or move the file to the path its slug names (`gbrain repair frontmatter --source <source> --include-ambiguous` proposes removing the line, for approval). |
-| <a id="file_too_large"></a>`file_too_large` | Over the page size limit: 24 MiB for a Markdown page, 5 MB for a code file ([limits per entry point](memory-boundaries.md#large-page-limits)). Every page under the limit is stored, chunked and searchable whatever its size, so this hold only fires above it, and the same bytes are held on every retry. A smaller page refused by an MCP or HTTP transport limit is a different refusal: send it by file. | Leave the file out of the source: read `gbrain config get sync.exclude`, then `gbrain config set sync.exclude '<current list>,<path>'`; or, if it must be in the brain, split it into files under the limit and commit. The next `gbrain sync --source <source> --no-pull` clears the hold, also when nothing was committed since (#6712 PR 2). |
+| <a id="file_too_large"></a>`file_too_large` | Over the page size limit: 24 MiB for a Markdown page, 5 MB for a code file ([limits per entry point](memory-boundaries.md#large-page-limits)). Every page under the limit is stored, chunked and searchable whatever its size, so this hold only fires above it, and the same bytes are held on every retry. A file sync had to read whole to refuse (over 24 MiB with segmented storage on) is not read again until it changes, gbrain is upgraded, or `gbrain sources retry-held <source>` schedules it; run that after changing a setting its message names. A smaller page refused by an MCP or HTTP transport limit is a different refusal: send it by file. | Leave the file out of the source: read `gbrain config get sync.exclude`, then `gbrain config set sync.exclude '<current list>,<path>'`; or, if it must be in the brain, split it into files under the limit and commit. The next `gbrain sync --source <source> --no-pull` clears the hold, also when nothing was committed since (#6712 PR 2). |
| <a id="managed_image_sync_unsupported"></a>`managed_image_sync_unsupported` | Managed sync imports image files only while multimodal embedding is on (`GBRAIN_EMBEDDING_MULTIMODAL=true`): the image page, its OCR text and its image vector are built by the coordinated image importer at preparation, and `source_path` is recorded. While multimodal embedding is off, each image import is held with this code instead of refusing the whole sync (#5493); deleting or renaming an image still reconciles its page. These holds count as missing coverage (the image is not searchable) but never escalate, because they are not a content fault. | Nothing is broken. To import the images, set `GBRAIN_EMBEDDING_MULTIMODAL=true` on the owner and run `gbrain sync --source <source> --no-pull`: the next discovery re-screens every image hold and imports the files (#6529). To stop holding them instead, leave them out of the source (`gbrain config get sync.exclude`, then `gbrain config set sync.exclude '<current list>,**/*.png'`); that clears the holds on the next sync. A held image is also re-screened when it changes and on `gbrain sources retry-held <source>`. |
| <a id="content_rejected"></a>`content_rejected` | The content-sanity gate matched junk and the operator set `content_sanity.junk_disposition` to `reject`, so the file is never imported and always visible. | Remove the matched junk from the file, or ask the user whether `junk_disposition` should go back to `quarantine`; that setting is the user's decision. A managed-sync hold re-screens on every sync, so either clears it on the next one. |
| <a id="purged_content"></a>`purged_content` | The file (or written content) has the content hash of a page the owner purged with `gbrain delete <slug> --purge`, under any slug, or carries a purged claim the database refuses. Purge leaves a text-free tombstone so a stale file, re-sync or revert cannot bring the content back. | Delete or edit the file and commit; an edited file imports normally. Only if the purge was a mistake, the owner lists tombstones with `gbrain pages purges list` and clears one with `gbrain pages unpurge <slug>` on the brain host; a managed-sync hold re-screens on every sync, so the next one imports it. A Google or GitHub connector item refused this way is skipped and the sync continues; `gbrain sources status` lists it ("skipped as purged content"). See [expire versus purge](memory-boundaries.md#purge). |
diff --git a/llms-full.txt b/llms-full.txt
index 707718d6d4..a007f2d123 100644
--- a/llms-full.txt
+++ b/llms-full.txt
@@ -7389,6 +7389,11 @@ vars — incident-time escape hatches, not everyday knobs.
`--wait-drain <seconds>` (default 60; `sources.refresh_drain_wait_ms`,
`GBRAIN_REFRESH_DRAIN_WAIT_MS`) and `--fetch-timeout-ms` (default 120000;
`sources.refresh_fetch_timeout_ms`, `GBRAIN_REFRESH_FETCH_TIMEOUT_MS`).
+ The member syncs of one run stop starting work after
+ `sources.refresh_sync_budget_ms` (default 300000; `GBRAIN_REFRESH_SYNC_BUDGET_MS`),
+ so a scheduled run never holds the refresh lock much longer than that:
+ a run past it reports `syncing` (members `deferred`) and the next
+ `gbrain sources refresh <source-id>` continues where it stopped.
A refresh whose process died (a crash, a replaced container) does not
wedge the checkout: the owner finishes what it can on its own, and the next
`gbrain sources refresh <source-id>` on the owner host finishes the rest
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index d80fca694b..7f311b70ec 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -47,4 +47,4 @@ src/commands/doctor/checks/extraction-sync.ts 1500 ratchet +7 GBRA-130 A-1: link
src/core/search/evidence-delivery.ts 1755 ratchet +5 W5: EvidencePlan.rankCut resolved with the plan (logic in search/page-rank-cut.ts); +20 W5 whole_pages: wholeCost prices a block as delivery would emit it whole (needs the private Block/enrichBlock internals) and the cut is wired before allocate (cutWholePages lives in search/page-rank-cut.ts); new row: crossed the 1500 unlisted cap at the #6367 merge (master 1479, its explicit-budget cap) by the C1 date header threaded through every block path, cap included; pagePlanHits (search/page-plan-hits.ts) and the header loader (evidence-date.ts blockDateHeaders) were peeled first; peel candidate: allocateCapped; +8 at the #6396 merge: page evidence text cut through the fence-chunk trust overlay and the caller minTrust floor (re-summed: our 1504 + master +8); +119 Cat 40 next wave F2: the search.long_page_unit policy (withLongPagePolicy, selectLongPages size preflight with page/byte/deadline limits) and the long-page fetch and fallback in deliverEvidence, kept in the shared seam the plan names (gbrain-evals docs/plans/2026-10-11-cat40-next-wave, F2); +17 F2 rework to the frozen-comparison policy: search.long_page_min_chars (rendered-body characters, default 20,000) read in withLongPagePolicy, a character count in the preflight, the rendered-body confirmation in deliverEvidence, and the long-page row exemption in capDeliveredSnippets (re-summed at the 0.60.225.0 merge: master 1537 + F2 136); +39 Wave B PR 2 lane 2b: planSegmentedBlock joins the windows search/segmented-evidence.ts reads around a segmented page's hits into one run-marked document (needs the private Doc/Piece/buildDoc/locateAnchor internals), planBlock's unit cut split into blockFromDoc for both, sections bounded by their run, selectionCost counts a gap between runs, and the fetch skips the whole-body fence overlay for a segmented page; +4 segmented pages read at once before planning (segmentedReads; the read is search/segmented-evidence.ts, the statements page-body/read-range.ts); +17 GBRA-156: tail trim plan flag, DeliverOptions.query, DeliveryMeta.tail_trim, the trim call and the frozen-hit score/query library fields (logic in search/evidence-tail-trim.ts; interface fields cannot move); +7 GBRA-205: EvidencePlan.titleHeader and DeliveryMeta.title_header (interface fields cannot move) and the titles option on the header loader (logic in search/evidence-date.ts; the query budget key lives in search/evidence-packing.ts); +11 #7600: assemble_evidence quotes conversation rows and capEvidenceToBudget counts the quote fence (logic in search/transcript-quote.ts); +2 GBRA-156 /ship review: the cut-path admission and title room count the quote fence; +2 GBRA-156 /ship review pass 2: capEvidenceToBudget counts the fence only for the paths that fence (quote flag; recall counts the text as returned)
src/commands/config.ts 1524 ratchet new row: crossed the 1500 unlisted cap in wave 15 P7.4 #5461: set-time validation of think.pages_budget_chars / think.excerpt_ceiling_chars (range + parser live in core/think/pages-budget.ts), mirroring the cycle.consolidate.cluster_threshold hook; +1 wave 16 G5.1 (#6495): set-time refusal hook for search.keyword_statement_timeout_ms (check lives in commands/config/keyword-timeout-value.ts); +1 wave 18b lane 5 (#5012 item 6): search.contextual_retrieval prints the existing-pages hint after the write (text lives in commands/config/contextual-mode-hint.ts); +1 #6712 PR 2: retired/rollback sync key dispatch line (logic in core/sync-config-keys.ts handleSyncKeyConfig); +1 #6865: sync.slug_conflicts.auto_remove boolean validation at set time (refusal text in repair/slug-autoheal.ts); +1 #6948: sync.origin_twins.auto_resolve boolean validation at set time (refusal text in repair/origin-twins-heal.ts); +2 #6795: config set chat_model/expansion_model help line and the model-pin import (shadow report and file-pin unset live in commands/config/model-pin-shadow.ts); +1 #6712 PR 4b: sync.admission set-time hook (value logic in commands/config/sync-admission-value.ts); +4 #7130: set-time refusal hook for sync.max_deletes / sync.max_delete_ratio (and .source.<id>) / purge.retention_hours (check lives in core/sync-reconcile.ts deleteSafetyConfigRefusal); +1 GBRA-150: sync.metafile_pages set-time route (body in config/sync-metafile-pages-value.ts); re-summed at the v0.60.330.0 merge (1520 measured); +2 #7242 minions.terminal_hook file-plane key and its set branch (logic in minions/terminal-hook.ts); +1 GBRA-161: generations.* set-time route (refusal in commands/config/generations-values.ts); +1 GBRA-207 #7630: sync.repo_path set-time refusal hook (refusal in core/sync-config-keys.ts syncRepoPathRefusal)
src/commands/mcp-expose.ts 1519 ratchet hosted_option: Session carries the gbrain.io coaching notice on the first-receipt run and Tailscale/service friction exits (copy, refs and gating live in core/onboard/hosted-option.ts); +7 GBRA-199: --surface help lines, surfaceChosen option field and the advertise pass-through (registrationAdvertise in core/mcp-registration.ts)
-src/core/behavior-change-notice.ts 1570 ratchet the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +2 GBRA-205 saved_facts_stems row and its comment (re-summed at the #7901 0.60.492.0 merge: 1541 + 2); +2 GBRA-205 search.query_facts_block default-on row and its approval comment (re-summed at the #7610 0.60.490.0 merge: 1539 + 2); +2 the #7900 content checkpoint host-move row (since TBD until the merge slot), +2 the #7897 private-hold sweep row (since TBD until the merge slot), +2 the #7880 ingest_capture and company-brain private-hold row (since TBD until the merge slot), +5 the frozen-source wave comment and four rows (#7868, #7850, #7869/#7870, #7871), +2 the #7848 visibility_noncanonical hold row (since TBD until the merge slot), +2 the #7846 extract_atoms session-source row (re-summed at the 0.60.481.0 merge: master 1524 + 2), +2 the #7805 synthesize session-source and #7841 visibility_not_applied rows (re-summed at the 0.60.480.0 merge: master 1522 + 2), +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2), +2 the #7814 retired v0.13.1 grandfather row, +2 the #7618 conversion withhold/progress/push row (master 1520 + 2); the behavior-change table split (#7607) retires this row; +4 GBRA-188 #7489: the wave comment and three rows (brain-first instructions, starter surface default, per-turn rule); +1 GBRA-188 plugin_turn_hook_gbra188 row; +1 GBRA-242 provider_key_rules_7959 row (#7959), +2 the GBRA-199 registration_lists_verbs row (since TBD until the merge slot); +2 GBRA-133 #7652: the #7611 pass_abandoned reader row and the lean read-only serve socket row (re-summed at the 0.60.503.0 merge: master 1555 + 2); +1 GBRA-243: the per-turn hook disclosure row (rows are data and cannot move); +4 GBRA-156 #7598 harness wave comment and rows (remote page cap, sources_not_readable refusal, sources add next step); +4 GBRA-156 #7601 search_quotes_transcript_rows comment and row; +1 GBRA-175 #7475 autopilot_restarts_serve_7475 row (re-summed at the #8012 merge: 1566 + 1); +3 #7997/#8014/#8015 rows (redaction markers, company db_only keep, ignored supabase_only scope)
+src/core/behavior-change-notice.ts 1572 ratchet the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +2 GBRA-205 saved_facts_stems row and its comment (re-summed at the #7901 0.60.492.0 merge: 1541 + 2); +2 GBRA-205 search.query_facts_block default-on row and its approval comment (re-summed at the #7610 0.60.490.0 merge: 1539 + 2); +2 the #7900 content checkpoint host-move row (since TBD until the merge slot), +2 the #7897 private-hold sweep row (since TBD until the merge slot), +2 the #7880 ingest_capture and company-brain private-hold row (since TBD until the merge slot), +5 the frozen-source wave comment and four rows (#7868, #7850, #7869/#7870, #7871), +2 the #7848 visibility_noncanonical hold row (since TBD until the merge slot), +2 the #7846 extract_atoms session-source row (re-summed at the 0.60.481.0 merge: master 1524 + 2), +2 the #7805 synthesize session-source and #7841 visibility_not_applied rows (re-summed at the 0.60.480.0 merge: master 1522 + 2), +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2), +2 the #7814 retired v0.13.1 grandfather row, +2 the #7618 conversion withhold/progress/push row (master 1520 + 2); the behavior-change table split (#7607) retires this row; +4 GBRA-188 #7489: the wave comment and three rows (brain-first instructions, starter surface default, per-turn rule); +1 GBRA-188 plugin_turn_hook_gbra188 row; +1 GBRA-242 provider_key_rules_7959 row (#7959), +2 the GBRA-199 registration_lists_verbs row (since TBD until the merge slot); +2 GBRA-133 #7652: the #7611 pass_abandoned reader row and the lean read-only serve socket row (re-summed at the 0.60.503.0 merge: master 1555 + 2); +1 GBRA-243: the per-turn hook disclosure row (rows are data and cannot move); +4 GBRA-156 #7598 harness wave comment and rows (remote page cap, sources_not_readable refusal, sources add next step); +4 GBRA-156 #7601 search_quotes_transcript_rows comment and row; +1 GBRA-175 #7475 autopilot_restarts_serve_7475 row (re-summed at the #8012 merge: 1566 + 1); +3 #7997/#8014/#8015 rows (redaction markers, company db_only keep, ignored supabase_only scope); +2 GBRA-250 #8072 rows (oversize hold left unread, refresh sync budget; since TBD until the merge slot)
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index a834798000..49e1969327 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1372,6 +1372,8 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.514.0', id: 'storage_ignored_dirs_8015', text: 'When gbrain.yml lists a deprecated `supabase_only` next to canonical keys (which ignores it), a managed sync now holds only the missing-file pages under the ignored directories `db_only` does not already cover, and deletes other missing-file pages as usual; it used to hold every missing-file page of the source. `gbrain storage status` now names those directories, and the deprecation warning no longer points to a `gbrain doctor --fix` rename that does not exist. A `db_only` or `supabase_only` line whose inline value the parser refuses (a missing comma, a nested list, a bare scalar) now counts as unresolved even when another key resolves, so missing-file pages are held rather than deleted.' },
{ since: '0.60.514.0', id: 'company_db_only_keep_8014', text: 'A company-brain source now keeps a live page under a declared `storage.db_only` directory when its file is not in the approved manifest, and keeps every such page while its gbrain.yml storage section is invalid, as other managed sources do; it used to delete them. The decision reads gbrain.yml at the approved commit, never the working tree.' },
{ since: '0.60.514.0', id: 'redaction_marker_not_credential_7997', text: 'The transcript `labeled_credential` detector no longer reads an importer redaction marker (`login **<REDACTED:user-pattern>**`) as a typed user/password pair, so a correctly redacted page clears `gbrain transcripts audit-secrets`; a password written beside a marker is still found. The audit detector version moves to 3, so `gbrain doctor` asks for a fresh audit once.' },
+ { since: 'TBD', id: 'oversize_hold_unread_8072', text: 'A Markdown file over 24 MiB that sync had to read whole to refuse (segmented storage on, held `file_too_large`, for example for a facts fence) is no longer read again by every sync: it stays held, unread, until it changes, gbrain is upgraded, or `gbrain sources retry-held <source>` schedules it. After changing a setting the hold message names, run `gbrain sources retry-held <source>`. A file held from its size alone still re-checks on every sync.' },
+ { since: 'TBD', id: 'refresh_sync_budget_8072', text: 'The member syncs of one `gbrain sources refresh` run stop starting work after `sources.refresh_sync_budget_ms` (default 5 minutes; `GBRAIN_REFRESH_SYNC_BUDGET_MS`). A run past it reports `syncing` with the members it left `deferred`, and the next `gbrain sources refresh <source>` continues it, so a slow sync no longer holds the refresh lock across scheduled runs.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/persistence/sync-holds.ts b/src/core/persistence/sync-holds.ts
index 523f8a8ca4..2869d53aba 100644
--- a/src/core/persistence/sync-holds.ts
+++ b/src/core/persistence/sync-holds.ts
@@ -203,6 +203,8 @@ export interface GitHoldMeta {
identity_paths?: string[];
/** GBRA-130 `page_identity_changed` (`identity_differs`): the frontmatter ids of the page and of the file. */
identity_ids?: { page: string; file: string };
+ /** #8072 `file_too_large`: the streamed (segmented) preparation read the whole file to refuse it, under this gbrain; it re-screens when the file changes, on retry, or under another gbrain. */
+ streamed?: { gbrain_version: string };
}
/** #6188: why the last fence repair left a hold in place, and when the maintenance run tries it again (null: it does not). */
@@ -599,12 +601,14 @@ export async function holdBasisMoved(engine: Pick<BrainEngine, 'executeRaw'>, so
* the file changes. GBRA-133: a `git_object_missing` hold rests on the repository's objects (a fetch restores them
* under the same blob id), so it re-screens on every sync; a `page_write_failed` hold of a freeze fault (origin `freeze`,
* S-63) is re-tested by every sync until it has been held `HOLD_ATTEMPTS_NEEDS_HUMAN` times, so a transient fault clears.
+ * #8072: a `file_too_large` hold is re-screened by every sync only while that costs its size (an lstat): one the streamed
+ * preparation refused after reading the whole file (`meta.streamed`) waits for its bytes to change, a retry or another gbrain.
*/
/** GBRA-133: the `file_drift` screen version; 1 is #7340's rule (a page with no import record compares its file at the baseline commit). */
export const FILE_DRIFT_VERSION = 1;
export function holdRescreenDue(record: Pick<GitHoldRecord, 'code' | 'meta'>, retried: boolean): boolean {
- return retried || ['frontmatter_slug_conflict', 'file_too_large', 'rename_held', 'parser_regression', 'content_rejected', 'write_gate_rejected', 'purged_content', 'legacy_slug_conflict', 'origin_conflict', 'page_identity_changed', 'git_object_missing'].includes(record.code)
+ return retried || (record.code === 'file_too_large' && record.meta.streamed?.gbrain_version !== VERSION) || ['frontmatter_slug_conflict', 'rename_held', 'parser_regression', 'content_rejected', 'write_gate_rejected', 'purged_content', 'legacy_slug_conflict', 'origin_conflict', 'page_identity_changed', 'git_object_missing'].includes(record.code)
// #6712: a transient hold (the write owner stalled, not the file) is retried by every sync until it has been held HOLD_ATTEMPTS_NEEDS_HUMAN times.
|| (outOfAttemptsHold(record.code) && (record.meta.attempts ?? 1) < HOLD_ATTEMPTS_NEEDS_HUMAN)
|| (record.code === 'managed_image_sync_unsupported' && isMultimodalEnabled())
@@ -855,7 +859,8 @@ export function gitHoldFix(record: Pick<GitHoldRecord, 'source_id' | 'path' | 'c
}
case 'file_too_large':
return { argv: ['gbrain', 'config', 'get', 'sync.exclude'], consent: [], actor: 'agent', requires_exclusive: false,
- why: `${record.path} is over the page size limit. The size limit is fixed, and every page under it syncs and is searchable. Leave the file out of the source: read the current sync.exclude list, then run gbrain config set sync.exclude '<current list>,${record.path}'; or, if it must be in the brain, split it into files under the limit and commit. The next gbrain sync --source ${source} --no-pull clears the hold.`,
+ why: `${record.path} is over the page size limit. The size limit is fixed, and every page under it syncs and is searchable. Leave the file out of the source: read the current sync.exclude list, then run gbrain config set sync.exclude '<current list>,${record.path}'; or, if it must be in the brain, split it into files under the limit and commit. The next gbrain sync --source ${source} --no-pull clears the hold.`
+ + (record.meta.streamed ? ` Sync read the whole file to refuse it, so it does not read it again until it changes, gbrain is upgraded, or gbrain sources retry-held ${source} schedules it (after a setting its message names changes).` : ''),
verify: { argv: ['gbrain', 'sources', 'status', source, '--json'] } };
case 'purged_content':
return { argv: ['gbrain', 'pages', 'purges', 'list', '--source', source], consent: [], actor: 'user', requires_exclusive: false,
diff --git a/src/core/persistence/sync-screen.ts b/src/core/persistence/sync-screen.ts
index f2dcad868f..dcd9734092 100644
--- a/src/core/persistence/sync-screen.ts
+++ b/src/core/persistence/sync-screen.ts
@@ -229,8 +229,9 @@ export function originConflictHold(entry: Pick<SyncEntry, 'path' | 'sourcePath'
/** Wave B: the hold for a file its streamed preparation refused, by the receipt's content-refusal code and message. */
export function streamedRefusalHold(entry: Pick<SyncEntry, 'path' | 'sourcePath' | 'working' | 'renameFrom' | 'renameHeld'>, slug: string, pageId: number | null,
done: Pick<WriteRequest, 'error_code' | 'error_message'>, blobOid: string | null | undefined): HeldEntry {
- return heldEntry(entry, slug, pageId, { code: done.error_code as ContentRefusal['code'], message: done.error_message ?? 'The streamed import refused this file.' },
+ const held = heldEntry(entry, slug, pageId, { code: done.error_code as ContentRefusal['code'], message: done.error_message ?? 'The streamed import refused this file.' },
null, blobOid ? { oid: blobOid } : null);
+ return held.code === 'file_too_large' ? { ...held, meta: { ...held.meta, streamed: { gbrain_version: VERSION } } } : held;
}
export function oversizeHold(entry: Pick<SyncEntry, 'path' | 'sourcePath' | 'working' | 'renameFrom' | 'renameHeld'>, slug: string, pageId: number | null, size: number | null, blob: TreeBlob | null | undefined): HeldEntry {
diff --git a/src/core/persistence/worktree-refresh.ts b/src/core/persistence/worktree-refresh.ts
index 0a8a2c838d..34dac39b58 100644
--- a/src/core/persistence/worktree-refresh.ts
+++ b/src/core/persistence/worktree-refresh.ts
@@ -57,6 +57,8 @@ export interface RefreshOptions {
waitDrainMs?: number;
/** Flag value; falls back to GBRAIN_REFRESH_FETCH_TIMEOUT_MS, then `sources.refresh_fetch_timeout_ms`, then 120 s. */
fetchTimeoutMs?: number;
+ /** #8072: how long the member syncs of one run may take; falls back to GBRAIN_REFRESH_SYNC_BUDGET_MS, then `sources.refresh_sync_budget_ms`, then 5 min. */
+ syncBudgetMs?: number;
hooks?: RefreshHooks;
}
export interface RefreshMemberSync { source_id: string; status: string; last_commit: string | null }
@@ -79,6 +81,7 @@ const holderOf = (row: Pick<WorktreeRefreshRow, 'outcome'>): RefreshHolder | nul
const describeHolder = (holder: RefreshHolder | null) => holder ? `pid ${holder.pid} on ${holder.hostname} (host ${holder.host_id ?? 'unknown'}), started ${holder.started_at}` : 'a process that recorded no holder';
const DRAIN_POLL_MS = 250;
+const SYNC_BUDGET_MS = 300_000;
const MAX_MEMBER_SYNC_PASSES = 20;
const GIT_ENV = { GIT_TERMINAL_PROMPT: '0', GCM_INTERACTIVE: 'Never', LC_ALL: 'C', GIT_OPTIONAL_LOCKS: '0' };
@@ -376,15 +379,16 @@ async function reachedTarget(git: Git, root: string, row: WorktreeRefreshRow, co
}
/** Steps 6-7: the managed `--no-pull` sync per member, then completion when every member reached target_head (or a descendant). */
-async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRow, root: string): Promise<{ synced: RefreshMemberSync[]; blocked: RefreshSyncBlocked[]; atTarget: boolean }> {
- const { performManagedSync } = await import('./sync-run.ts');
+async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRow, root: string, budgetMs: number): Promise<{ synced: RefreshMemberSync[]; blocked: RefreshSyncBlocked[]; atTarget: boolean }> {
+ const sync = await boundedMemberSync(budgetMs);
const synced: RefreshMemberSync[] = [], blocked: RefreshSyncBlocked[] = [];
for (const sourceId of row.source_ids) {
let status = 'up_to_date';
for (let pass = 0; pass < MAX_MEMBER_SYNC_PASSES; pass++) {
if (await reachedTarget(git, root, row, await lastCommit(engine, sourceId))) break;
+ if (sync.spent()) { status = 'deferred'; break; }
let result: SyncResult;
- try { result = await performManagedSync(engine, { sourceId, noPull: true }); }
+ try { result = await sync.run(engine, sourceId); }
catch (error) {
const failure = error instanceof OperationError ? error : opError('storage_error', error instanceof Error ? error.message : String(error),
`The managed sync of source ${sourceId} failed during refresh ${row.id}. Check its writer status, fix the cause, then resume with gbrain sync --source ${sourceId} --no-pull.`);
@@ -409,6 +413,21 @@ async function syncMembers(engine: BrainEngine, git: Git, row: WorktreeRefreshRo
{ synced, ...(blocked.length ? { sync_blocked: blocked } : {}) }));
return { synced, blocked, atTarget };
}
+/**
+ * #8072: the member syncs of one refresh share one wall-clock budget, so a run never holds the refresh lock for longer than
+ * that plus one entry: each pass is a managed-sync slice that yields `partial` (`writer_yield`) at an entry boundary once the
+ * budget ran, and no pass starts after it. A member left behind reports `deferred`; the refresh stays `syncing` and the next
+ * plain refresh (or `--resume`) continues it.
+ */
+async function boundedMemberSync(budgetMs: number) {
+ const { performManagedSync } = await import('./sync-run.ts');
+ const deadline = performance.now() + budgetMs;
+ return {
+ spent: () => performance.now() >= deadline,
+ run: (engine: BrainEngine, sourceId: string) => performManagedSync(engine, { sourceId, noPull: true, drainHolder: { kind: 'cli', id: null } },
+ { maxPages: Number.MAX_SAFE_INTEGER, maxMs: Math.max(0, deadline - performance.now()) }),
+ };
+}
async function lastCommit(engine: BrainEngine, sourceId: string): Promise<string | null> {
const [row] = await engine.executeRaw<{ last_commit: string | null }>('SELECT last_commit FROM sources WHERE id=$1', [sourceId]);
return row?.last_commit ?? null;
@@ -427,16 +446,17 @@ function finished(row: WorktreeRefreshRow, sync: { synced: RefreshMemberSync[];
* host only; refusals throw catalogue errors (`code`, cause, literal fix, docs).
*/
export async function refreshWorktree(engine: BrainEngine, sourceId: string, opts: RefreshOptions = {}): Promise<WorktreeRefreshResult> {
- if (opts.resume || opts.abandon) return resumeOrAbandon(engine, sourceId, opts);
+ const syncBudgetMs = await configuredMs(engine, opts.syncBudgetMs, 'GBRAIN_REFRESH_SYNC_BUDGET_MS', 'sources.refresh_sync_budget_ms', SYNC_BUDGET_MS);
+ if (opts.resume || opts.abandon) return resumeOrAbandon(engine, sourceId, opts, syncBudgetMs);
const fetchTimeoutMs = await configuredMs(engine, opts.fetchTimeoutMs, 'GBRAIN_REFRESH_FETCH_TIMEOUT_MS', 'sources.refresh_fetch_timeout_ms', 120_000, 2 ** 31 - 1);
const waitDrainMs = await configuredMs(engine, opts.waitDrainMs, 'GBRAIN_REFRESH_DRAIN_WAIT_MS', 'sources.refresh_drain_wait_ms', 60_000);
- const recovered = opts.dryRun ? null : await adoptOrphanedRefresh(engine, sourceId);
+ const recovered = opts.dryRun ? null : await adoptOrphanedRefresh(engine, sourceId, syncBudgetMs);
if (recovered && recovered.status !== 'completed' && recovered.status !== 'abandoned') return recovered;
- const result = await refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs);
+ const result = await refreshOnce(engine, sourceId, opts, fetchTimeoutMs, waitDrainMs, syncBudgetMs);
return recovered?.recovered ? { ...result, recovered: recovered.recovered } : result;
}
-async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshOptions, fetchTimeoutMs: number, waitDrainMs: number): Promise<WorktreeRefreshResult> {
+async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshOptions, fetchTimeoutMs: number, waitDrainMs: number, syncBudgetMs: number): Promise<WorktreeRefreshResult> {
return withGit(async git => {
const plan = await precheck(engine, git, sourceId, fetchTimeoutMs);
const base = { refresh_id: null, worktree_id: plan.binding.worktree_id, source_ids: plan.members, old_head: plan.oldHead,
@@ -445,12 +465,14 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
next: plan.oldHead === plan.targetHead ? `Already at ${plan.upstreamRef}; nothing to fast-forward.` : `gbrain sources refresh ${sourceId}` };
if (plan.oldHead === plan.targetHead) {
const synced: RefreshMemberSync[] = [], blocked: RefreshSyncBlocked[] = [];
- const { performManagedSync } = await import('./sync-run.ts');
+ const sync = await boundedMemberSync(syncBudgetMs);
for (const member of plan.members) {
await recordUpstreamObservation(engine, member);
- if (await lastCommit(engine, member) === plan.oldHead) { synced.push({ source_id: member, status: 'up_to_date', last_commit: plan.oldHead }); continue; }
+ const commit = await lastCommit(engine, member);
+ if (commit === plan.oldHead) { synced.push({ source_id: member, status: 'up_to_date', last_commit: plan.oldHead }); continue; }
+ if (sync.spent()) { synced.push({ source_id: member, status: 'deferred', last_commit: commit }); continue; }
try {
- const result = await performManagedSync(engine, { sourceId: member, noPull: true });
+ const result = await sync.run(engine, member);
if (result.status === 'blocked_by_failures') blocked.push({ source_id: member, code: result.managedWrite?.write_error ?? 'sync_blocked',
message: result.managedWrite?.message ?? 'The managed sync is blocked by a failed write.', resume: `gbrain sync --source ${member} --no-pull` });
synced.push({ source_id: member, status: result.status, last_commit: await lastCommit(engine, member) });
@@ -458,7 +480,9 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
blocked.push({ source_id: member, code: (error as OperationError).code ?? 'storage_error', message: (error as Error).message, resume: `gbrain sync --source ${member} --no-pull` });
}
}
- return { ...base, status: blocked.length ? 'sync_blocked' : 'already_current', synced, ...(blocked.length ? { sync_blocked: blocked, next: blocked.map(b => b.resume).join(' ; ') } : {}) };
+ const deferred = synced.some(member => member.status === 'deferred');
+ return { ...base, status: blocked.length ? 'sync_blocked' : 'already_current', synced, ...(blocked.length ? { sync_blocked: blocked, next: blocked.map(b => b.resume).join(' ; ') }
+ : deferred ? { next: `gbrain sources refresh ${sourceId}` } : {}) };
}
const refreshLock = await acquireNativeLock(refreshLockPath(plan.binding.worktree_id), { timeoutMs: 1000 });
if (!refreshLock) throw inProgress('Another process is refreshing this worktree right now.', sourceId);
@@ -482,7 +506,7 @@ async function refreshOnce(engine: BrainEngine, sourceId: string, opts: RefreshO
await opts.hooks?.boundary?.('drained', row);
const merged = await fenceAndMerge(engine, git, sourceId, plan, row, opts.hooks);
await opts.hooks?.boundary?.('syncing', merged);
- const sync = await syncMembers(engine, git, merged, plan.root);
+ const sync = await syncMembers(engine, git, merged, plan.root, syncBudgetMs);
return finished(await readRefresh(engine, row.id), sync, plan.preserved);
} finally { await refreshLock.release(); }
});
@@ -666,7 +690,7 @@ async function membersAtTarget(engine: BrainEngine, git: Git, row: WorktreeRefre
return synced;
}
-async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: RefreshOptions): Promise<WorktreeRefreshResult> {
+async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: RefreshOptions, syncBudgetMs: number): Promise<WorktreeRefreshResult> {
const binding = await getWorktreeBinding(engine, sourceId, existingLocalHostId());
if (!await managedPersistenceEnabled(engine) || !binding) throw refusal('refresh_not_managed',
`Source ${sourceId} has no claimed canonical worktree on a managed brain, so there is no refresh to resume.`, `gbrain sync --source ${sourceId}`);
@@ -695,7 +719,7 @@ async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: Refr
return { ...empty, status: 'abandoned', refresh_id: row.id, source_ids: row.source_ids, old_head: row.old_head, target_head: row.target_head,
upstream_ref: row.upstream_ref, preserved_uncommitted: row.preserved_uncommitted, next: `gbrain sources refresh ${sourceId}` };
}
- return resumeHeld(engine, git, sourceId, active.id, root, head, empty);
+ return resumeHeld(engine, git, sourceId, active.id, root, head, empty, syncBudgetMs);
});
} finally { await lock.release(); }
}
@@ -704,7 +728,7 @@ async function resumeOrAbandon(engine: BrainEngine, sourceId: string, opts: Refr
* `--resume` under the refresh lock the caller holds: one recovery step, then the member syncs of a `syncing` row.
* #7679: the caller becomes the row's holder; `previous_holder` keeps the first holder replaced (the refresher that died), so a refused --resume never hides it.
*/
-async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refreshId: string, root: string, head: string | null, empty: WorktreeRefreshResult, unattended = false): Promise<WorktreeRefreshResult> {
+async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refreshId: string, root: string, head: string | null, empty: WorktreeRefreshResult, syncBudgetMs: number, unattended = false): Promise<WorktreeRefreshResult> {
await engine.executeRaw(`UPDATE persistence_worktree_refreshes SET outcome=outcome||jsonb_build_object('holder',$2::text::jsonb,'previous_holder',COALESCE(outcome->'previous_holder',outcome->'holder'))
WHERE id=$1::uuid AND state IN ${ACTIVE_REFRESH_STATES_SQL}`, [refreshId, JSON.stringify(refreshHolder())]);
let row = await withTopologyLocks(engine, sourceId, async () => recoverOne(engine, git, await readRefresh(engine, refreshId), root, unattended), undefined, 30_000);
@@ -720,7 +744,7 @@ async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refre
target_head: row.target_head, upstream_ref: row.upstream_ref, next: `The interrupted refresh was rolled back without moving HEAD; start again with gbrain sources refresh ${sourceId}`,
recovered: { ...recovered, status: 'abandoned' } };
startPersistenceConsumer(engine, loadConfig() ?? { engine: engine.kind });
- const sync = await syncMembers(engine, git, row, root);
+ const sync = await syncMembers(engine, git, row, root, syncBudgetMs);
row = await readRefresh(engine, row.id);
const result = finished(row, sync);
return { ...result, recovered: { ...recovered, state: row.state, status: result.status } };
@@ -734,7 +758,7 @@ async function resumeHeld(engine: BrainEngine, git: Git, sourceId: string, refre
* a live holder (the lock stays busy for a second), a caller that is not the owner, or a checkout HEAD git cannot read
* (an unattended run never escalates on it); the precheck then refuses each case by name.
*/
-async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string): Promise<WorktreeRefreshResult | null> {
+async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string, syncBudgetMs: number): Promise<WorktreeRefreshResult | null> {
if (!await managedPersistenceEnabled(engine)) return null;
const hostId = existingLocalHostId();
const binding = await getWorktreeBinding(engine, sourceId, hostId);
@@ -754,7 +778,7 @@ async function adoptOrphanedRefresh(engine: BrainEngine, sourceId: string): Prom
old_head: null, target_head: null, upstream_ref: null, preserved_uncommitted: [], synced: [] };
return await withGit(async git => {
const head = await headOf(git, root);
- return head === null ? null : resumeHeld(engine, git, sourceId, orphan.id, root, head, empty, true);
+ return head === null ? null : resumeHeld(engine, git, sourceId, orphan.id, root, head, empty, syncBudgetMs, true);
});
} finally { await lock.release(); }
}
diff --git a/test/cli-socket.test.ts b/test/cli-socket.test.ts
index 43285388f4..451ae6cff4 100644
--- a/test/cli-socket.test.ts
+++ b/test/cli-socket.test.ts
@@ -98,7 +98,7 @@ const NOT_READ_BEHAVIOR = new Set([
'GBRAIN_CORPUS_WINDOWS_PER_SWEEP', 'GBRAIN_DEFERRED_EMBED_RETRY_MS', 'GBRAIN_DRAIN_TIMEOUT_MS', 'GBRAIN_FLUSH_GRACE_MS', 'GBRAIN_GITHUB_PAT',
'GBRAIN_GITHUB_REPO', 'GBRAIN_GLOBAL_MAINTENANCE_TIMEOUT_MS', 'GBRAIN_LLM_HALT_COOLDOWN', 'GBRAIN_MAX_FENCES_PER_PAGE', 'GBRAIN_MAX_SKILL_MD_BYTES',
'GBRAIN_MAX_WALK_DEPTH', 'GBRAIN_ORPHANED_PRIVATE_QUEUE_MINUTES', 'GBRAIN_PHANTOM_REDIRECT_LIMIT', 'GBRAIN_PLANNER_AUTO_ANALYZE',
- 'GBRAIN_REFRESH_DRAIN_WAIT_MS', 'GBRAIN_REFRESH_FETCH_TIMEOUT_MS', 'GBRAIN_SKIP_REFERENCE_SWEEP', 'GBRAIN_SKIP_STARTUP_HOOKS', 'GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD',
+ 'GBRAIN_REFRESH_DRAIN_WAIT_MS', 'GBRAIN_REFRESH_FETCH_TIMEOUT_MS', 'GBRAIN_REFRESH_SYNC_BUDGET_MS', 'GBRAIN_SKIP_REFERENCE_SWEEP', 'GBRAIN_SKIP_STARTUP_HOOKS', 'GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD',
'GBRAIN_WEDGED_QUEUE_WARN_MINUTES', 'GBRAIN_ANTHROPIC_MAX_INFLIGHT',
'GBRAIN_SERVE_BOOT_TIMEOUT_SECONDS', 'GBRAIN_SERVE_EOF_DRAIN_MS', 'GBRAIN_SERVE_FAIL_FAST', 'GBRAIN_SERVE_HEAD_DEBOUNCE_MS', 'GBRAIN_SERVE_HEAD_POLL_MS',
'GBRAIN_SERVE_RESTART_REQUIRED_EXIT_SECONDS', 'GBRAIN_SERVE_STALL_WATCHDOG_MS', 'GBRAIN_SERVE_SYNC_IPC', 'GBRAIN_SERVE_SYNC_SETTLE_MS',