Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
docs: sync-drain key-files entry names claimed head reads in the stall-line age (ship docs audit)
garrytana8e1db4405ed
diff --git a/docs/architecture/key-files/google-and-loops-continued.md b/docs/architecture/key-files/google-and-loops-continued.mdindex b7dfdb7b04..dcfa3decf7 100644--- a/docs/architecture/key-files/google-and-loops-continued.md+++ b/docs/architecture/key-files/google-and-loops-continued.md@@ -11,7 +11,7 @@- `src/core/persistence/{maintenance,root-registry,filesystem-guard,writer-guard-schema}.ts` — fail-closed legacy writer coverage: durable private refusal records and shared refusal-only root markers work before PGLite connect, resolve symlink ancestors, and retain stale roots conservatively. An unchanged per-tick refresh writes nothing: the registry directory and records are chmodded only when their mode differs and the directory is fsynced only when a record changed (`root-registry-idle-chmod-5297.test.ts`). The selected PGLite path is registered at connect/init and source claim refresh, so managed reinit refuses even offline; unmanaged backup acquires the stable sibling native lock through rename. SQL guards include free-text aliases and canonical source sync checkpoints; an update that changes only embedding, telemetry or write-attribution columns of facts/takes is a physical projection and passes without the coordinator. `docs/architecture/canonical-writers.tsv` is the finite writer census; `canonical-writer-inventory.test.ts` rejects unreviewed additions, and `persistence-writer-fences.test.ts`/`pglite-maintenance-fence.test.ts` exercise real engines, competing processes and separate homes.-- **`src/core/persistence/sync-drain.ts`** — #5984: drains a managed sync in one invocation. `runDrain` re-enters the single-pass `performManagedSync` (which keeps returning `partial / writer_pending | writer_yield`) until the cursor is done, the caller's signal stops it, a strict registered run deadline is about 15 s away (`forward-progress.ts` `registerRunDeadline`, set by `cli.ts` when it arms the sync watchdog), a writer head needs intervention (`recovery_required`, `owner_unavailable`, unexpected bytes), or the no-progress detector sees the awaited request and its worktree head unchanged for 30 s and 3 passes with nothing claimable here. It retries only `worktree_refreshing` (its `retry_after_ms`), `database_contention`, connection errors and statement timeouts (3 attempts). Every page commit notes forward progress, so the progress-aware watchdog never stops a progressing drain. The result carries `drain: { outcome: synced | resumable | blocked, stop_reason, written, waived, remaining, rate_pages_per_min, eta_seconds, stall }`; `drainNext` builds `next` for the CLI from the shared `syncResumeCommand` (`sync-reconcile.ts`). The CLI (`sync/run.ts` single source, `--all`, `--watch`), serve's delegated runner and the `sync` job handler (#6706, bounded by `SyncOpts.deadlineAtMs`) set `SyncOpts.drain`; the PGLite owner delegate uses `runDrain` over IPC; the cycle, `sync-administration.ts` and `worktree-refresh.ts` keep their own loops. #6706: `runDrain` takes a caller `deadlineAtMs` (the earlier of it and the strict run deadline wins; a past deadline makes zero passes; a far one re-arms past the 2^31−1 ms timer limit), reports `resumable / deadline` when the deadline lands in a retry backoff, abandons a pass still running `DEADLINE_PASS_GRACE_MS` after it, and `deadlinePassed` stops the extra pass and the next stage; `jobDisposition` classifies a drained result for jobs (`synced`, `continue` when `drainNext` is safe to loop, `needs_operator`). `src/core/minions/handlers/sync-job-drain.ts` runs it in the `sync` job: `planJobDrain` (`drain`, `lanes: 1`, a deadline from `job.deadlineAtMs` or the row's wall-clock limit via `WALL_CLOCK_THRESHOLD_MS_SQL`, at most an hour, less 60 s; job-data controls only from trusted submitters; `sync.job_drain=false` turns it off), throttled progress, and `settleJobDrain` (the CLI's `outcome`/`drain`/`next`, `incomplete`, at most one continuation on a worker-backed brain, priority ≥ 5, whose `next` then observes it). A sync job that finds another live sync job on its source skips `sync_in_progress` (`sync-handler-drain-6706.test.ts`). `readManagedSyncBacklog` reads unfinished cursors' `progress` window (stamped by `sync-run.ts` on each advance, reset per drain) for `gbrain doctor` (`managed_sync_backlog`) and `sources status`, with `resume_args` built by `sync-reconcile.ts` `managedSyncResumeArgs`. #6278: the no-progress fingerprint keys on state, blocked reason, head id/state and the head claim's phase/step/since (`drainClaimOf`), never on lease columns, so a renewed-forever claim reads as no progress; a live preparation is allowed its budget plus `STALL_GRACE_MS` (budgets read from `persistence.*_preparation_ms` / `preparation_ceiling_ms`), a lapsed claim keeps the plain window (`stall.cause: owner_missing`), a preparation this process's pid owns past its allowance ends `resumable` / `preparation_abandoned`, and `SyncResult.breaker` ends `blocked` / `preparation_systemic`; `stall` carries `phase`, `step`, `waiting_on`, `cause`, and a timer prints `stalled <N>s on <step>` while nothing commits: between passes from the fingerprint's claim, inside a pass (a grouped publish parked in preparation never returns to the loop) from `StallProbe.head`, a 2 s-bounded read of the source's oldest running (else queued) request that prints nothing when no write is unfinished (B6). B7: a pass refused for write capacity (`isWriteCapacityWait`, `admission-retry.ts`: a `queue_capacity` whose `detail` is `outstanding=N limit=M`) is retried with backoff up to 5 s apart while the drain prints `waiting for write capacity (N outstanding of M)`; past the no-progress window it ends `blocked` / `write_capacity` with `drain.capacity` and writer status as `next`, keeping the last pass's result (or an empty one) for the report; a cumulative cap still throws. A pass that finishes one cursor and yields for a scheduled re-screen starts a new run on the next pass, and the report sums the page counts of every run it finished. Tests: `test/sync-drain.test.ts`, `test/sync-drain-claims.test.ts`, `test/managed-sync-drain.test.ts`, `test/sync-drain-write-capacity.test.ts`.+- **`src/core/persistence/sync-drain.ts`** — #5984: drains a managed sync in one invocation. `runDrain` re-enters the single-pass `performManagedSync` (which keeps returning `partial / writer_pending | writer_yield`) until the cursor is done, the caller's signal stops it, a strict registered run deadline is about 15 s away (`forward-progress.ts` `registerRunDeadline`, set by `cli.ts` when it arms the sync watchdog), a writer head needs intervention (`recovery_required`, `owner_unavailable`, unexpected bytes), or the no-progress detector sees the awaited request and its worktree head unchanged for 30 s and 3 passes with nothing claimable here. It retries only `worktree_refreshing` (its `retry_after_ms`), `database_contention`, connection errors and statement timeouts (3 attempts). Every page commit notes forward progress, so the progress-aware watchdog never stops a progressing drain. The result carries `drain: { outcome: synced | resumable | blocked, stop_reason, written, waived, remaining, rate_pages_per_min, eta_seconds, stall }`; `drainNext` builds `next` for the CLI from the shared `syncResumeCommand` (`sync-reconcile.ts`). The CLI (`sync/run.ts` single source, `--all`, `--watch`), serve's delegated runner and the `sync` job handler (#6706, bounded by `SyncOpts.deadlineAtMs`) set `SyncOpts.drain`; the PGLite owner delegate uses `runDrain` over IPC; the cycle, `sync-administration.ts` and `worktree-refresh.ts` keep their own loops. #6706: `runDrain` takes a caller `deadlineAtMs` (the earlier of it and the strict run deadline wins; a past deadline makes zero passes; a far one re-arms past the 2^31−1 ms timer limit), reports `resumable / deadline` when the deadline lands in a retry backoff, abandons a pass still running `DEADLINE_PASS_GRACE_MS` after it, and `deadlinePassed` stops the extra pass and the next stage; `jobDisposition` classifies a drained result for jobs (`synced`, `continue` when `drainNext` is safe to loop, `needs_operator`). `src/core/minions/handlers/sync-job-drain.ts` runs it in the `sync` job: `planJobDrain` (`drain`, `lanes: 1`, a deadline from `job.deadlineAtMs` or the row's wall-clock limit via `WALL_CLOCK_THRESHOLD_MS_SQL`, at most an hour, less 60 s; job-data controls only from trusted submitters; `sync.job_drain=false` turns it off), throttled progress, and `settleJobDrain` (the CLI's `outcome`/`drain`/`next`, `incomplete`, at most one continuation on a worker-backed brain, priority ≥ 5, whose `next` then observes it). A sync job that finds another live sync job on its source skips `sync_in_progress` (`sync-handler-drain-6706.test.ts`). `readManagedSyncBacklog` reads unfinished cursors' `progress` window (stamped by `sync-run.ts` on each advance, reset per drain) for `gbrain doctor` (`managed_sync_backlog`) and `sources status`, with `resume_args` built by `sync-reconcile.ts` `managedSyncResumeArgs`. #6278: the no-progress fingerprint keys on state, blocked reason, head id/state and the head claim's phase/step/since (`drainClaimOf`), never on lease columns, so a renewed-forever claim reads as no progress; a live preparation is allowed its budget plus `STALL_GRACE_MS` (budgets read from `persistence.*_preparation_ms` / `preparation_ceiling_ms`), a lapsed claim keeps the plain window (`stall.cause: owner_missing`), a preparation this process's pid owns past its allowance ends `resumable` / `preparation_abandoned`, and `SyncResult.breaker` ends `blocked` / `preparation_systemic`; `stall` carries `phase`, `step`, `waiting_on`, `cause`, and a timer prints `stalled <N>s on <step>` while nothing commits: between passes from the fingerprint's claim, inside a pass (a grouped publish parked in preparation never returns to the loop) from `StallProbe.head`, a 2 s-bounded read of the source's oldest running (else queued) request that prints nothing when no write is unfinished (B6); `inPassHeadAge` takes every head read, a claimed one included, so a queued head after a claimed one counts its age from the read before it (a claimed head prints its claim's step age). B7: a pass refused for write capacity (`isWriteCapacityWait`, `admission-retry.ts`: a `queue_capacity` whose `detail` is `outstanding=N limit=M`) is retried with backoff up to 5 s apart while the drain prints `waiting for write capacity (N outstanding of M)`; past the no-progress window it ends `blocked` / `write_capacity` with `drain.capacity` and writer status as `next`, keeping the last pass's result (or an empty one) for the report; a cumulative cap still throws. A pass that finishes one cursor and yields for a scheduled re-screen starts a new run on the next pass, and the report sums the page counts of every run it finished. Tests: `test/sync-drain.test.ts`, `test/sync-drain-claims.test.ts`, `test/managed-sync-drain.test.ts`, `test/sync-drain-write-capacity.test.ts`.- **`src/core/persistence/sync-window.ts`** — #5984 admit-ahead for draining managed Postgres syncs. While the cursor's `group` publishes, `admitAhead` (`sync-run.ts`) freezes and admits the next group and records it in the cursor's `window` (at most one group), so the consumer claims it as soon as the publishing group commits under the unchanged per-worktree FIFO claim. Window members name the previous group's last request in `intent.after`. #6712 (one page, one fate): a window group publishes whatever its predecessor became; nothing is cancelled for another page's failure, and dependency edges (`sync-group.ts` `dependencyKeys`, `freezeFollowers`' `taken`) keep two changes of one page out of the same group and out of the window while one is in flight. Publication validation (`sync-prepare.ts`) re-checks the predecessor once per transaction and reads the cursor `FOR KEY SHARE`, accepting `pending`, `group` and `window` members, so the drain's cursor saves never wait for the publishing group. After a failed page the members after it stay admitted at the head of the window, so every admitted request's identity stays in the cursor until its own receipt is read (`groupStep`, `promoteWindow`). Nothing is admitted ahead while foreground writes are recent (one was queued on the worktree in the last minute, `FOREGROUND_RECENT_MS`) or when the next entry is not groupable. Older binaries refuse a window member they did not admit (cursor membership), so a mixed-version drain fails closed instead of publishing twice; finish a drain before downgrading. A claimed head is ordered by `claimedPredecessorState`: a group member follows the member before it in its own group (its `after` names only the previous group), anything else follows `after`. `claimedHeadOrder` (the consumer, before claiming followers) only orders: under lanes a member whose predecessor member is unfinished releases its claim and waits; a predecessor that ended, committed or not, lets it publish (#6712). `WINDOW_CANCEL_MESSAGE` and `laneFaultCancelMessage` stay only so older receipts still read as legacy knock-ons. Test: `test/sync-group-member-order.test.ts`.- **`src/core/persistence/sync-lanes.ts`** + **`worktree-lease.ts`** — #5984 lanes. The drain opens a lane policy for its worktree (`openLanes`, keyed by a per-drain run id that lane groups carry as `intent.lane`; `closeLaneRun` at drain end stops new lane claims and waits up to 90 s for the lane tasks still running, counted from the head's claim by `laneTask`). While it is open, this process's consumer claims the next admitted lane group head out of FIFO order (`claimNextLaneHead` in journal.ts: every earlier request claimed, any unfinished one a running request of the same run) up to the effective lane count (`laneRoots`), and runs it as a lane (`executeClaimedGroup` with `lane`). Lane publications share the worktree's native lock in this process (`acquireWorktreeShared` / `worktree-lease.ts`; one lane acquires the lock while lanes arriving meanwhile wait and join, and a lease stays registered until its lock is released); every exclusive `acquireWorktree` first yields the lease (marks it draining and wounded, waits up to its budget), and a lease drains for new claims after `LEASE_TURN_MS` of continuous holding. Lane transactions begin in manifest order (`awaitLaneBegin`) and commit in manifest order (`awaitLaneTurn`, inside the transaction before the counters: the predecessor `intent.after` must have ended, committed or not (#6712); a requeued or unadmitted one releases it, a wounded lease or 60 s rolls it back). Publication validation accepts a lane group whose predecessor is still queued or running (`windowPredecessorAllows`). `laneFallback` in group-publish.ts releases or falls back to the single path (#6712: never cancels), and a lock or statement timeout steps the lane count down (`stepDownLanes`). Lane settings (`--lanes`/`--no-lanes` > `GBRAIN_SYNC_LANES` > `sync.lanes` > 6, clamped to the pool's long-hold capacity minus 3) resolve in `resolveBulkSettings` (sync-group.ts); with lanes the drain keeps twice the lane count of groups admitted ahead (`admitAhead`).- **`src/core/persistence/sync-run.ts`** + **`sync-authority.ts`** — one immutable request at a time, bounded fairness and pinned-target resumption. A completed discovery that finds nothing to do (`up_to_date`) stamps only `sources.last_sync_at`, inside `withCoordinatedWrite` and guarded by source id and incarnation, so a quiet managed source stays fresh in doctor and a run that outlived a re-added source stamps nothing; dry runs and partial runs do not stamp. Cursors and requests freeze `noEmbed`, `noExtract` and `noSchemaPack`. #6712 PR 1: a resume inherits the cursor's processing opt-outs OR-wise (`adoptCursor`, one CAS that also stamps the resumer's validated authority) and reports a flag it overrode in `inherited_options`; nothing refuses over processing options, and `explicitProcessing` is gone. Admitted requests keep their own intent principal: they are read under it (`admittedBy`), and a writer change drops only unadmitted frozen entries. Every managed pass runs under the source's drain lease (`withDrainLease`, `src/core/persistence/sync-lease.ts`), whose token fences admission, the settled/held CAS, fan-in and the cursor CAS (`assertDrainLease`). Explicit idle ordinary-source retry creates fresh discovery/admission and retains terminal receipts, also when the run has a recorded failure and its pending write already committed before the cursor advanced; company consent retry retains its approved manifest/run. Active work blocks replacement. Claimed/inactive sync, including connector/company dispatch, remains refused: existing leases and native locks do not fence already-running engine-only legacy writers. No automatic activation or new transition protocol. `freezeEntry` checks a rename's source page (id, revision, origin) as well as its target slug, and a committed rename counts in `renamed`. #5522: a new-file entry (`pageId: null`) whose origin another cursor of the source has since imported is re-frozen against that page when it is live, holds exactly that origin at the entry's slug in the cursor's incarnation, and the sync preparer finds its content unchanged; otherwise the origin refusal stands, and the manifest is never rewritten. #5751: the working-tree screen passes the kernel waivers for working-tree entries and counts them into `legacySkips` (printed by `sync-diagnostics.ts` with `gbrain repair contextual-mode`). A terminal `checkpoint_validation_timeout` receipt gets its hint from `checkpoint-validation.ts`. Content holds (#5988, `sync-screen.ts`): `freezeEntry` screens each import (company-profile cursors and `sync.holds=fail` excepted); a refused file becomes a hold written in the same transaction as the cursor step (`saveCursor`'s `inTx`), with no request admitted, and an over-bound pinned blob is held by size before it is read. A cursor whose pending request failed with a content refusal (`isContentRefusal`, old message shapes included) converts in place with its stored options: held, or re-frozen once per distinct bytes (`converted`), so a refusal the screen misses stays blocked without minting receipts. #6188: a failed fence refusal (`fenceReceiptLocation`: the durable `error_detail.fence`, the message, or an older release's message) holds the same bytes even when the screen admits them (`prepareTimeFenceHold`, reason `prepare_time`); a compacted `invalid_params` / `take_row_collision` receipt converts only when the re-screen holds. A page request of the run that fails with a fence refusal is held in the same invocation at the terminal-failure branch (single path and a bulk group's failed member) after the source's other requests settle within the run's wait budget, else the run returns `partial`; it is converted once, gets no failure-ledger row and is logged as a conversion. `sync.holds=fail` and company-profile cursors keep blocking with the typed refusal. #6278: a `preparation_stalled` receipt (the owner gave up preparing a member) is converted the same way by both converters: an import holds by its bytes, a delete as a delete hold (`meta.deleted`, `upstream_version` null), the checkpoint never holds; `preparationStalledHold` (sync-screen.ts) records the receipt's step and the resume arguments. Both converters write their hold through `holdUnderGate`, the narrowed gate: the failed entry's admitted suffix (window, later group members, this run's queued lane rows) is cancelled first, then the hold waits only for active suffix descendants and unfinished requests naming the entry's page (`UNFINISHED_PAGE_REQUEST_SQL`, now matching path and origin too), evaluated inside the transaction that writes the hold under the page-key → cursor lock order; its doc comment states the invariant the old source-wide wait protected. The cursor's `breaker` counts the run's stalled receipts (identities and streak, saved with each CAS): `holdsEscalated`'s rule over the run's screened imports or `BREAKER_STREAK` (5) in a row trips it, the run stops `blocked_by_failures` with `SyncResult.breaker` (drain `preparation_systemic`), and the next run's `convertBlockedCursor` re-freezes the entry and schedules the run's stalled holds for a re-screen instead of holding; a committed page resets the streak. `sameContentAtOrigin` and the waiver screens race the sync preparer against `persistence.sync_preparation_ms` (`raceSyncBudget`, sync-waivers.ts) and fail open. Tests: `test/managed-sync-preparation-stalled-holds.test.ts`, `test/sync-waiver-budget.test.ts`. Results carry this run's holds, the outstanding total, escalation, `converted_from_failed` and `recovered_frontmatter` (`buildHoldReport`); dry runs list `would_hold` / `screen_skipped`. After the checkpoint commits the pass stamps drain step `finish` and emits `managed_sync.finish`; then (unless `noExtract`) the owner derives links for the source's stale pages through `links-maintenance.ts` under `FINISH_BUDGET_MS` (pages left over stay stale for `gbrain extract --stale`), so a PGLite-delegated sync and a Postgres sync take the same path; the result carries `links`, and an extraction failure leaves pages stale without failing the committed sync. #6278 (B7): when the checkpoint commits while `sources retry-held` re-screens are scheduled for the source (scheduled against a backlog, or while this cursor ran), the pass returns `partial` / `writer_yield` instead of `synced`, so the drain's next pass retires the finished cursor and discovers again, taking them; a `queue_capacity` refused for outstanding requests (`isWriteCapacityWait`) is left out of the failure ledger because the drain waits it out. #6696: the cursor key comes from `selectManagedCursor` (`sync-cursor-key.ts`), which every managed caller crosses: a run without `--strategy` and without a company plan keys a fresh cursor with the source's configured strategy, while an unfinished cursor under the raw key keeps resuming under it, so its admitted requests keep matching. `SyncOpts.embedDeferred` (the cost gate's auto-defer) is frozen on the intent and treated as `noEmbed` by preparation, never as a processing option a resume adopts or compares (`test/persistence-sync-cursor-strategy-6696.test.ts`).