Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

A write killed mid-way no longer holds up its source, and receipts report the Git commit (GST-2572)

garrytane207ba0e4b81
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e620385123..0b77e2fee6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,23 @@ credits are retained; no result has been reassigned to another provider. Origina
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+### A write killed in the middle no longer holds up its source
+
+**When a gbrain process dies while it holds the write at the head of a source's queue, the next write clears it at once instead of waiting.** A crash or `kill -9` used to leave that write's claim running with nobody renewing it, and every later write to the source waited behind it (`write_pending`) for the rest of its lease: 30 s while the write was preparing, about 6 minutes while it was publishing. On PGLite that wait happened on every such kill.
+
+- On a PGLite brain, the next gbrain process that opens the brain releases every claim written before it started and runs those writes again. The brain admits one process at a time, so their owners have exited. A process that cannot open the brain releases nothing.
+- On Postgres, a process killed on the same machine was already released at once; that is unchanged. A process on another machine, in another container or from before a reboot still waits out its lease, because gbrain cannot prove it is gone.
+- `gbrain sources writer status` now says so: a claim whose owner has provably exited reads `owner_exited: true`, `resumes_on_its_own: true` and `owner_live: false`, and its `next` is `claim_owner_exited` (nothing to do) instead of `claim_running` ("the owner is still renewing").
+- [docs/guides/troubleshooting.md](docs/guides/troubleshooting.md#stuck-write-head) lists how long each case takes and what status prints meanwhile.
+
+### A write receipt says when its Git commit landed
+
+**A receipt read after the write (`gbrain write-request <id>`, `get_write_request`, `list_write_requests`) now reports where Git is.** For a page on an ordinary (not `canonical: git`) checkout, `persistence.git_state` used to stay `queued` forever, even after the background Git effect committed the file, and no commit was named anywhere. It now comes from the Git effect: `committed` with the commit in `persistence.git_commit`, or `superseded`, `skipped`, `failed` or `retrying` with the cause in `persistence.git_reason`, and `queued` only while the effect is queued. Nothing stored is rewritten: a Git effect from an earlier build reads `committed` with no commit. Receipts of `canonical: git` sources are unchanged.
+
+Reported by @ethanbeard (#9227).
+
## [0.60.700.0] - 2026-11-12
### A gbrain commit no longer reverts someone else's commit of the same file
diff --git a/docs/guides/git-canonical.md b/docs/guides/git-canonical.md
index fcbe6e9629..da791fbd73 100644
--- a/docs/guides/git-canonical.md
+++ b/docs/guides/git-canonical.md
@@ -16,7 +16,7 @@ operator's view.
| | A mixed source (the default) | A strict source (`canonical: git`) |
|---|---|---|
-| A page write | The file and the database row are published first; a background Git effect commits and pushes the file later, once Git durability is on. | The file is written and committed under the checkout's writer lock, then the row commits. The receipt names the commit (`persistence.git_commit`, `git_state: committed`). The Git effect only pushes. |
+| A page write | The file and the database row are published first; a background Git effect commits and pushes the file later, once Git durability is on. The write's reply says `git_state: queued`; a receipt read later (`gbrain write-request <id>`, `get_write_request`) reads the Git effect: `committed` with the commit in `persistence.git_commit`, or `superseded`, `skipped`, `failed` or `retrying` with `persistence.git_reason`. An effect recorded by a build before this one reads `committed` with no commit. | The file is written and committed under the checkout's writer lock, then the row commits. The receipt names the commit (`persistence.git_commit`, `git_state: committed`). The Git effect only pushes. |
| A crash between the two | Recovery restores the file's previous bytes. | The committed file is kept, and the write's replay finishes the row against it. |
| What sync reads | The working tree and the database; disagreements become holds such as `file_drift` or `worktree_dirty`. | Committed blobs at HEAD only. Each committed version is imported over what the database holds, and a committed removal deletes the page. |
| An uncommitted edit in the checkout | Held as `worktree_dirty`. | Left on disk and not indexed until someone commits it. |
diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md
index 52f553f07a..c83b1f88c0 100644
--- a/docs/guides/troubleshooting.md
+++ b/docs/guides/troubleshooting.md
@@ -49,6 +49,7 @@
| [Sync held a file for its facts or takes fence (`Held <path>: invalid_fence …`)](#held-fence-files) | nothing for most holds: the next maintenance run repairs and commits the fence. To see or do it now, preview `gbrain repair fences --source <id>` and run its printed `--apply --expect <hash>`; to pause automatic repair, `gbrain config set fences.repair.enabled false` (ask the user first). A `manual` reason needs the edit the hold names | agent; the apply needs no extra consent | none (model repair spends within the `fences.repair.*` caps; raising them is `paid`, the user's call) | `gbrain doctor --only fence_integrity --json` |
| A managed catch-up on a live checkout keeps dying (`revision_conflict`, `pinned_git_worktree_conflict`, `page_identity_changed`, `write ECONNABORTED`), or you want an agent to drive the recovery | `gbrain sync status --source <id> --json` (cursor, `committed_last_10m`, each hold and the last error with `class` / `safe_actions` / `needs_human`); when nothing moved and `needs_human` is false, `gbrain sync unblock --source <id> --apply`, then the sync it prints ([runbook](sync-unblock-runbook.md)) | agent; a person only when `needs_human` names a slug | none | `gbrain sync status --source <id> --json` |
| [Every write on a source stays pending; writer status shows a request `recovering` with `unexpected_file_bytes`](repair.md#stuck-recovery) | preview `gbrain repair stuck-recovery --source <id>` on the owner host, show the user the diff it prints, then run the choice they pick: `--keep prior` or `--keep disk` with `--apply --expect <hash> --yes`; never delete the file to unblock it | agent; the apply after the user picks a version | `destructive` (both versions are backed up first) | `gbrain sources writer status --source <id> --json` shows no recovering request |
+| [Every write on a source stays pending after a gbrain process was killed mid-write; writer status shows `claim_owner_exited`](#stuck-write-head) | nothing: the head clears by itself, at once on the next write on PGLite and for an owner on this machine on Postgres, and after its lease (30 s, or about 6 minutes while publishing) for a Postgres owner on another machine | none | none | `gbrain sources writer status --source <id> --json` no longer lists the request as `running` |
| [Sync held a file with `worktree_dirty`](#worktree-dirty-hold) | commit (or restore) the file, then `gbrain sync unblock --source <id> --apply` and the sync it prints | agent; whoever edits the file commits it | none | `gbrain sources status <id> --json` |
| A managed sync of a source far behind HEAD prints `staged catch-up: synced to <commit>, short of HEAD`, or `sliced catch-up: committed the <phase> slice of <commit> through <path>` | Staging is automatic: `gbrain sync --source <id> --no-pull` takes stage after stage (each the furthest first-parent commit whose delta fits 100,000 entries and 16 MiB) until it reaches HEAD, and a stop resumes at the stored cursor; a scheduled sync takes one stage per run. A first sync, a `--full` sync or one commit that alone exceeds the bound is cut into path slices instead (#6349): every import slice, then bounded deletion slices, pinned at one target; `last_commit` moves only when the last slice settles, and the same command resumes an unfinished target at its stored slice. Only one file whose manifest entry alone exceeds the byte bound still refuses `request_too_large`, naming it; exclude it (`--exclude`, or `gbrain config set sync.exclude`) and rerun | agent | none | `gbrain sync status --source <id> --json` (cursor target reaches HEAD) |
| A write, sync, connector run or activation refuses `storage_error` with `requires exactly one known boolean persistence state` (`detail: persistence_mode_unknown`) | The `persistence_brain` singleton is missing, duplicated or malformed, so gbrain cannot tell whether the brain is managed and refuses instead of writing around the owner (#6482). Run `gbrain sources writer status --probe --json` on the brain host (it reports `mode: unknown` and the problem instead of refusing), restore the one row (`gbrain doctor --json` names the repair), then rerun the command | host operator | none | `gbrain doctor --json` |
@@ -116,6 +117,17 @@
<a id="worktree-dirty-hold"></a>**Sync held a file with `worktree_dirty`?** The working tree holds uncommitted bytes for the file that match neither the pinned commit nor the page (an agent mid-edit on a live checkout), so sync held it instead of overwriting the edit and kept going; nothing is lost and the run finished. Commit the file (or restore it), then `gbrain sync unblock --source <id> --apply` re-screens every held file that is now committed and prints the sync to run; a later commit that changes the file re-screens it on its own. A file held three times is reported `needs_human` by `gbrain sync status`: find out what keeps editing it. Bytes committed at HEAD past the pinned target are imported as HEAD has them, never held. See [worktree_dirty](write-refusals.md#worktree_dirty) and the [sync unblock runbook](sync-unblock-runbook.md).
+<a id="stuck-write-head"></a>**Every write on a source stays pending (`write_pending`) after a gbrain process was killed in the middle of a write?** A process that dies while it holds the write at the head of a source's queue (a crash, `kill -9`, an out-of-memory kill) leaves that write's claim `running` with no one renewing it. Later writes to the source are accepted and wait behind it. The head clears by itself, and no command or cancel is needed:
+
+| Where the killed process ran | When the head clears |
+|---|---|
+| A PGLite brain | At once, when the next gbrain process opens the brain (the next write, sync or `serve` start). The brain admits one process at a time, so every claim written before that process started belongs to a process that has exited, and its first scan releases them. A second process that cannot open the brain releases nothing. |
+| A Postgres brain, killed on this machine (same boot, same pid namespace) | At once on the next write from this machine, or within 5 s when a `serve` or sync is running here. |
+| A Postgres brain, killed on another machine, in another container or before a reboot | When its lease runs out: 30 s while the write was preparing, about 6 minutes (365 s) while it was publishing. gbrain cannot prove that process is gone, so it waits. |
+| Either engine, killed after the page file was written | At once: the next write's recovery scan finishes or restores the file. |
+
+The released write runs again and commits; a write whose OAuth client was revoked meanwhile ends `failed` with `permission_denied`. Meanwhile `gbrain sources writer status --source <id> --json` names the request as a blocker. When it can prove the owner gone it prints `claim.owner_exited: true`, `resumes_on_its_own: true`, `owner_live: false` and `claim.next.code: claim_owner_exited` (nothing to do; rerun the same status after the next write). For an owner it cannot prove gone it prints `claim_running` until the lease runs out and `claim_lapsed` after. Extract's derived-link updates do not wait behind such a head: the claim is a row, and the page lock the killed transaction held ended with it.
+
<a id="catch-up-stuck"></a>**Managed catch-up stuck (one write `running` for minutes while the rest waits), held N files as `preparation_stalled`, or the drain stopped with `preparation_abandoned`, `preparation_systemic` or `write_capacity`?** Every managed write's *preparation* has a deadline (`persistence.sync_preparation_ms`, default 120 s, for sync members; `persistence.maintenance_preparation_ms`, default 120 s, for maintenance writes; a hard ceiling `persistence.preparation_ceiling_ms`, default 600 s; `persistence.max_preparation_attempts`, default 2; all behind the `preparation_deadlines` write switch), so one stuck write is cut off at its budget, but a cut-off preparation that ignores cancellation keeps its root blocked until the ceiling (600 s from the claim's start), when the owner frees the root and the next claim holds the entry; the rest of the source keeps publishing meanwhile, and a sync beside such an owner prints `stalled <N>s on <step>` with the owner's pid and kind from the allowance to the ceiling instead of stopping. One wedge therefore costs up to ten minutes and one held file, never the run. The runbook branches on what you see; every command below is read-only unless it says otherwise, and the files themselves are never the problem in this branch. Start every branch with `gbrain doctor --only serve_build_drift --json`: on a warning, restart serve and its workers on this build first (see [serve build drift](#serve-build-drift)) and stop there until it reads ok.
1. **While it is stuck.** The progress line prints `stalled <N>s on <step>` instead of an ETA. Run `gbrain sources writer status --source <id> --json`: the blocked request's `claim.stall` names the step (`origin_check`, `knowledge_publication`, …), `waiting_on` (`git`, `fs`, `db`, `pool`, or an honest `unknown`), `step_age_ms`, the budget, and the owner process (kind, pid, gbrain version). An owner whose version predates this release never gives up: upgrade it and restart it (`gbrain serve`, the jobs worker or autopilot; the drain's own process ends with the drain). Otherwise wait out the budget: the write is released, retried once, then held. A release does not always free the root: a preparation that ignores cancellation holds its root barrier until the ceiling, and `writer status --json` ends the claim in a `claim.next` that says so (`claim_running` with `retry_after_ms` to the ceiling, or `claim_overdue` as `tell_user_to_run` naming the owner pid once the ceiling has passed or the owner reads `restart_required`).
@@ -181,7 +193,7 @@ Turning the `preparation_deadlines` switch off (`gbrain config set persistence.p
<a id="managed-sync-not-moving"></a>**The catch-up is parked (`sources status --json` says `data_moving: false`, doctor warns `managed_sync_not_moving`, `gbrain sources writer movement` exited 1), or the drain stopped `drain_stalled` with `cause: owner_wedged_here`?** Liveness is not movement: `/health` can be ok, the pid alive and `sync_running: true` while no page has committed for weeks. A managed source is *not moving* when an unfinished `managed-sync` cursor has had neither a committed `managed_sync_*` receipt nor a head step advance for longer than `persistence.preparation_ceiling_ms` (600 s) while a drain or a full consumer is live on the owner host; with nothing live it is `movement_state: parked` (a cursor left between cron runs; informational, not scored). A parked cursor whose write queue still holds admitted requests while no consumer runs and nothing has moved for longer than that ceiling is stranded: nothing will ever drain it, so doctor `managed_sync_not_moving` fails (#6552, #6712 PR 2) and its fix is the resume sync with the cursor's stored options (`gbrain sync --source <id> …`), which consumes the queue itself; `details.stranded` counts such sources. When the sync backlog itself cannot be read, every surface says `movement_state: unknown` (`unknown.reason: backlog_read_failed`) instead of `nothing_pending`: doctor warns with `details.health: unknown`, `sources status` prints a `movement UNKNOWN` line, and `gbrain sources writer movement` exits 1 with code `movement_unknown` (`--warn-only` exits 0); rerun the movement command, and run `gbrain doctor --json` if the read keeps failing. Every surface routes to the same two read-only calls, so the agent reaches the owner, the step and the next action without SQL:
1. `gbrain sources status <id> --json`: `data_moving`, `not_moving_since`, `movement_state` and, beside `data_moving: false`, the exact `gbrain sources writer status --source <id> --json` to run next. Doctor's `managed_sync_not_moving` fix is that same command.
-2. `gbrain sources writer status --source <id> --json`: every running claim ends in one `claim.next` envelope (`code` `claim_running`, `claim_overdue` or `claim_lapsed`). `claim_running` carries `retry_after_ms` and a `run` of the same status command while the owner is inside its allowance (the owner's own budget will cut the write; nothing to do but rerun after the delay), `tell_user_to_run` naming the owner `kind` and pid past the ceiling or when the owner reads `restart_required` (restart that process), `run` the printed resume command for a lapsed claim. `host.consumers` lists the processes on this host holding a consumer (pid, kind, mode `full|waiter_only`, started_at, the pool gauge), so a second consumer is visible in the same call.
+2. `gbrain sources writer status --source <id> --json`: every running claim ends in one `claim.next` envelope (`code` `claim_running`, `claim_overdue`, `claim_lapsed` or `claim_owner_exited`, the last for an owner that has exited; see [stuck write head](#stuck-write-head)). `claim_running` carries `retry_after_ms` and a `run` of the same status command while the owner is inside its allowance (the owner's own budget will cut the write; nothing to do but rerun after the delay), `tell_user_to_run` naming the owner `kind` and pid past the ceiling or when the owner reads `restart_required` (restart that process), `run` the printed resume command for a lapsed claim. `host.consumers` lists the processes on this host holding a consumer (pid, kind, mode `full|waiter_only`, started_at, the pool gauge), so a second consumer is visible in the same call.
3. After the restart (or after waiting out the ceiling), `gbrain sources retry-held <id>` re-screens the entry the owner held and prints the sync to run; `gbrain sources writer movement <id>` then waits one window and exits 0 only when a page committed (`moved`) or nothing is pending (`current`). See the [transcript](#managed-sync-wedge-transcript) below for what each step prints.
A `drain_stalled` stop with `cause: owner_wedged_here` is the same situation seen from the sync CLI: a live owner on this host holds the head write past the ceiling or its heartbeat row reads wedged. Before the ceiling the stop is loop-safe (`next.safe_to_loop: true`, `retry_after_ms` set); past it the stop names the pid to restart. Reference: [drain stops](write-refusals.md#drain-stalled).
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 590ab065e6..6fd24e5896 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1563,6 +1563,8 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.693.0', id: 'member_save_withdrawn_scope_gst2346', text: 'A remote `remember` onto a member\'s own page (`memory/members/<id>`) is refused `fact_withdrawn` only when that member could read what was withdrawn: a fact on its own page, its own copy withdrawn by the brain host, or for the brain owner a fact it reads. A claim withdrawn by the brain host or the owner that the member never read no longer refuses its save: the fact saves as the member\'s own active row with the same reply as a new claim. A schema migration makes the database insert guards mark a row on a member page only by that page\'s own withdrawals; a withdrawal still expires matching rows on member pages when it is made, and strikes them in the page\'s file, so an import of a member page\'s file no longer marks a row withdrawn by a withdrawal of another subject. The same holds for each item of a `remember` batch (`items`), a replay with the same `request_id`, a `replaces` and the write receipt (`get_write_request`). Trusted local saves keep the check they had.' },
{ since: '0.60.694.0', id: 'owner_labels_guarded_gst2521', text: 'Labels the owner sets with `gbrain repair visibility` (or its `--readers` setter, or `gbrain repair private-labels`) now raise the page\'s label guard to operator_curated, so a later local agent write at the page\'s own tier that drops or widens them is held like a write over an owner page, with a `lower_page` item. A local write below operator_curated whose `readable_by` would leave the brain owner out of a page the owner reads now publishes with the owner kept in the readers (the rest of the narrowing applies), and its item records the requested readers. Accepting such an item (`gbrain trust confirm tp<id>`) on a page whose own tier is an agent\'s writes only the labels and keeps the agent\'s tier; on an owner page it still endorses the edit at the owner\'s tier. The guard is stored in the database (`page_label_guards`) and set only by label repairs run after the upgrade, so labels set earlier, and a managed Git brain rebuilt from a clone, start unguarded; re-running `gbrain repair visibility --slug` or `--readers` on a page that already has those labels raises the guard without changing them. The owner\'s own label writes, synced owner edits, agent narrowing that keeps the owner, and brains with `access.owner_reads_restricted` on are unchanged.' },
{ since: '0.60.694.0', id: 'page_label_widening_held_gst2518', text: 'A local agent write (`put_page`, `capture`, `edit_page`, `append_page`, `revert_version`, `restore_page`) less trusted than a page that drops or widens its `visibility`, `access`, `confidentiality` or `readable_by` now publishes with the page\'s earlier labels kept, in the database and in the committed file, and its receipt lists them under `privacy_labels_kept`; so does a maintenance job\'s derived page publication, and a page whose tier is still `unknown` counts as the owner\'s. A soft delete no longer lowers the page\'s tier. Before, the page became readable to remote callers and other members at once, and only a `lower_page` item waited for the owner. That item now records the requested labels: `gbrain trust confirm tp<id>` applies them if the page has not changed since, `gbrain trust drop tp<id>` keeps the earlier ones, and undoing a confirm restores them while the page is unchanged since. Narrowing writes, label-neutral writes, writers at or above the page\'s tier, synced owner edits and `gbrain repair visibility` are unchanged.' },
+ { since: 'TBD', id: 'stuck_head_released_9227', text: 'On a PGLite brain, the next gbrain process that opens the brain now releases at once every write claim an earlier, exited process left running and runs those writes again, instead of leaving later writes to the same source pending for the rest of the claim\'s lease (30 s, or about 6 minutes while publishing). `gbrain sources writer status` reports a claim whose owner has provably exited (that case on PGLite, a dead process on the same machine on Postgres) with `owner_exited: true`, `resumes_on_its_own: true`, `owner_live: false` and `next.code: claim_owner_exited`, where it said `claim_running`.' },
+ { since: 'TBD', id: 'receipt_git_state_live_9227', text: 'A write receipt read after the write (`gbrain write-request`, `get_write_request`, `list_write_requests`) for a page on a checkout that is not `canonical: git` now takes `persistence.git_state` from the page\'s Git effect: `committed` with the commit in `persistence.git_commit`, or `superseded`, `skipped`, `failed` or `retrying` with `persistence.git_reason`, where it always said `queued`. Stored receipts are not rewritten; the stored `outcome` still says `queued`.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/ops/persistence.ts b/src/core/ops/persistence.ts
index 1ec7eff214..48d39bad23 100644
--- a/src/core/ops/persistence.ts
+++ b/src/core/ops/persistence.ts
@@ -116,7 +116,7 @@ async function quotedOutcome(outcome: Record<string, unknown>, slug: string): Pr
async function publicReceipt(ctx: OperationContext, row: WriteRequest, facts?: WriteHealthFacts): Promise<Record<string, unknown>> {
const { receiptFor } = await import('../persistence/journal.ts');
- const { publicEffectsForRequest } = await import('../persistence/effect-journal.ts');
+ const { gitReceiptState, publicEffectsForRequest } = await import('../persistence/effect-journal.ts');
const { receiptDeliveredHint } = await import('../persistence/connector-errors.ts');
const { CHECKPOINT_VALIDATION_TIMEOUT, checkpointTimeoutHint } = await import('../persistence/checkpoint-validation.ts');
const { writeFailureDiagnostic } = await import('../persistence/verb-errors.ts');
@@ -136,7 +136,10 @@ async function publicReceipt(ctx: OperationContext, row: WriteRequest, facts?: W
const { cliRenderContext, renderAction } = await import('../agent-output.ts');
const render = ctx.remote === false ? cliRenderContext() : { transport: 'stdio' as const, isCallable: () => false, preapproved: () => false };
const { liveSearchable } = await import('../persistence/searchable.ts');
- const receipt = publicWriteReceipt(receiptFor(row, facts));
+ const stored = publicWriteReceipt(receiptFor(row, facts));
+ // #9227: a classic write's `git_state: queued` is what its publication stored; the Git effect row says where Git is now.
+ const git = stored.persistence?.git_state === 'queued' ? await gitReceiptState(ctx.engine, row.id) : null;
+ const receipt = git ? { ...stored, persistence: { ...stored.persistence!, ...git } } : stored;
// GBRA-217: a reason whose Git effect was superseded is reported as not committed.
const settled = await (await import('../persistence/change-reason.ts')).settledReasonRecorded(ctx.engine, row);
if (settled) ctx.emitNotice?.(settled.notice);
diff --git a/src/core/persistence/claim-phase.ts b/src/core/persistence/claim-phase.ts
index c239e0df27..6bdcb2166a 100644
--- a/src/core/persistence/claim-phase.ts
+++ b/src/core/persistence/claim-phase.ts
@@ -240,6 +240,8 @@ export interface ClaimState {
last_sql: ClaimLastSql | null;
/** Whether the same request continues without anyone acting (after its lease lapses or its transaction ends). */
resumes_on_its_own: boolean;
+ /** #9227: present when the owner provably exited: no process renews the claim and the next write to the brain releases it. */
+ owner_exited?: true;
why: string;
}
export interface ClaimRow {
@@ -249,6 +251,10 @@ export interface ClaimRow {
/** `claim_expires_at < now()`, read on the database clock. */
claim_lapsed?: boolean | null;
publication_started?: boolean | null;
+ /** #9227: last written before `abandonedClaimCutoff` (PGLite, read by a process that holds the brain); the owner has exited. */
+ abandoned?: boolean | null;
+ /** #9227: also prove the stamped owner dead the way the consumer's dead-owner scan does (`claimOwnerProvablyDead`). */
+ probe_owner?: boolean;
}
const age = (at: unknown, now: number): number | null => {
@@ -279,6 +285,11 @@ export function claimStateOf(row: ClaimRow, now = Date.now()): ClaimState | null
owner: own ? stampOwner(stamp!.owner) : null, last_sql: own ? stampLastSql(stamp!.last_sql, now) : null };
if (row.publication_started) return { phase: 'file_publication', claim_age_ms: claimAge, phase_age_ms: null, ...detail, resumes_on_its_own: false,
why: 'Its files were being published; the publication recovery path restores or finishes them when the owning process scans its roots again (after a restart if it hung).' };
+ if (row.abandoned === true || row.probe_owner && own && claimOwnerProvablyDead(stamp)) {
+ const recorded = own && (stamp!.phase === 'preparing' || stamp!.phase === 'publishing') ? stamp!.phase as 'preparing' | 'publishing' : 'unrecorded';
+ return { phase: recorded, claim_age_ms: claimAge, phase_age_ms: recorded === 'unrecorded' ? null : age(stamp!.since, now), ...detail, resumes_on_its_own: true, owner_exited: true,
+ why: `Its owner process has exited (${row.abandoned === true ? 'it last wrote the claim before this process opened the brain, which admits one process at a time' : 'same machine boot, its process is gone'}) and nothing renews the claim. The next write from this machine (or a gbrain process already running here, within seconds) releases it and runs the request again; nothing else is needed.` };
+ }
if (row.claim_lapsed) return { phase: 'publication_transaction', claim_age_ms: claimAge, phase_age_ms: null, ...detail, resumes_on_its_own: true,
why: 'Its claim lapsed while it was still running and nothing was published: a transaction holds the request row (renewals cannot reach it) or its owner is gone. It is requeued once that transaction ends or the owner process exits.' };
// #6650: an undispatched group follower's stamp names its token and owner but no phase yet.
@@ -295,6 +306,6 @@ function safeJson(text: string): unknown {
/** The running-overdue verdict: a `preparing` claim older than its budget. */
export interface ClaimStall { reason: 'preparation_overdue'; step: string | null; step_age_ms: number | null; waiting_on: WaitingOn; budget_ms: number }
export function claimStall(claim: ClaimState | null, budgetMs: number): ClaimStall | null {
- if (!claim || claim.phase !== 'preparing' || claim.phase_age_ms === null || claim.phase_age_ms < budgetMs) return null;
+ if (!claim || claim.owner_exited || claim.phase !== 'preparing' || claim.phase_age_ms === null || claim.phase_age_ms < budgetMs) return null;
return { reason: 'preparation_overdue', step: claim.step, step_age_ms: claim.step_age_ms, waiting_on: claim.waiting_on, budget_ms: budgetMs };
}
diff --git a/src/core/persistence/consumer.ts b/src/core/persistence/consumer.ts
index 0aa2cf5069..82490267c2 100644
--- a/src/core/persistence/consumer.ts
+++ b/src/core/persistence/consumer.ts
@@ -36,7 +36,7 @@ import { monitorEventLoopDelay, type IntervalHistogram } from 'node:perf_hooks';
import { maybeRefreshPlannerStats } from '../planner-stats.ts';
import { refreshFenceClear } from './worktree-refresh-schema.ts';
import { faultPoint } from './fault-points.ts';
-import { EFFECT_OWNER_KEY_SQL, releaseAbandonedClaims } from './effect-journal.ts';
+import { abandonedClaimCutoff, EFFECT_OWNER_KEY_SQL, PROCESS_STARTED_AT, releaseAbandonedClaims } from './effect-journal.ts';
import { readWriteSwitchSnapshot, writeSwitchOn } from './switches.ts';
import { consumerConnectionRoute, consumerStatementEngine, poolerExposureLine } from './consumer-lane.ts';
import { isConnectionLoss } from '../retry-matcher.ts';
@@ -64,8 +64,6 @@ const DRAIN_NOTICE_EVERY_MS = 10_000;
const CONNECTION_LOSS_RELEASE_MS = 30_000;
/** #6650: how often a consumer looks for claims held by an owner it can prove dead. */
export const DEAD_OWNER_SCAN_MS = 5_000;
-/** When this process started; on PGLite no claim written earlier can belong to a live owner. */
-const PROCESS_STARTED_AT = new Date(performance.timeOrigin);
/** #5801: the phase a connection checkout belongs to, carried through its async chain. */
const phaseScope = new AsyncLocalStorage<{ observation: PhaseObservation; startedAt: number }>();
@@ -414,6 +412,13 @@ export class PersistenceConsumer implements PersistenceConsumerLike {
const cursor = residentProjectionCursor(this.engine);
const params: unknown[] = [this.hostId, excluded, publicationConcurrency(this.engine) > 0, retryingTopologies, [...this.laneTasks.keys()]];
const windowCtes = residentWindowCtes(cursor, params, PROJECTION_RETRY_READY_SQL);
+ // #9227: on PGLite, claims an earlier process left behind count until the scan below has released them, so a one-shot
+ // write queued behind such a head clears it at once instead of waiting out its lease (30 s, or 365 s while publishing).
+ const cutoff = this.abandonedReleased || this.opts.oneShot?.waiting() === false ? null : abandonedClaimCutoff(this.engine);
+ if (cutoff) params.push(cutoff.toISOString());
+ const abandoned = cutoff ? `
+ OR EXISTS (SELECT 1 FROM persistence_requests r WHERE r.state='running' AND r.recovery IS NULL AND r.publication_started=false AND r.updated_at<$${params.length}::timestamptz)
+ OR EXISTS (SELECT 1 FROM persistence_effects e WHERE e.state='running' AND e.recovery IS NULL AND e.updated_at<$${params.length}::timestamptz)` : '';
const [row] = await this.phase('idle_probe', async signal => this.probeQuery<{ work: boolean; projection_work: boolean; projection_last: { source_incarnation: string; slug: string } | null; high_water: string | null }>(this.laneInUse || direct ? undefined : await this.acquireIdleLane(signal), `WITH ${windowCtes},
pwork AS (SELECT EXISTS (SELECT 1 FROM (SELECT source_incarnation,slug FROM recent UNION ALL SELECT source_incarnation,slug FROM win) j
CROSS JOIN LATERAL (SELECT s.id FROM sources s WHERE s.incarnation=j.source_incarnation AND NOT s.archived OFFSET 0) s
@@ -446,7 +451,7 @@ export class PersistenceConsumer implements PersistenceConsumerLike {
OR EXISTS (SELECT 1 FROM persistence_topology_changes c
JOIN persistence_worktrees w ON w.id=(c.recovery->>'worktreeId')::uuid
WHERE c.recovery IS NOT NULL AND w.owner_host_id=$1::uuid AND NOT (c.id::text=ANY($4::text[])))
- OR ${overdueDeferredWorkSql('$1')}
+ OR ${overdueDeferredWorkSql('$1')}${abandoned}
) AS work`, params, signal));
// A window holding work stays put, so the tick it wakes reads the same rows.
advanceProjectionCursor(cursor, row?.projection_work ? null : row?.projection_last, row?.high_water);
@@ -670,7 +675,7 @@ export class PersistenceConsumer implements PersistenceConsumerLike {
.catch(error => this.report(error)).finally(() => { this.projectionWorker = undefined; });
// Recover only our owner roots. Kernel exclusion, not elapsed heartbeat,
// proves that a previous process can no longer be publishing this root.
- if (scan && this.engine.kind === 'pglite' && !this.abandonedReleased) {
+ if (scan && abandonedClaimCutoff(this.engine) && !this.abandonedReleased) {
await this.phase('abandoned_claims', () => releaseAbandonedClaims(this.engine, PROCESS_STARTED_AT, this.policy.deadlines));
this.abandonedReleased = true;
}
diff --git a/src/core/persistence/diagnostics.ts b/src/core/persistence/diagnostics.ts
index 9e90a65128..5d6294b5e3 100644
--- a/src/core/persistence/diagnostics.ts
+++ b/src/core/persistence/diagnostics.ts
@@ -15,6 +15,7 @@ import { readWriteSwitchSnapshot } from './switches.ts';
import { DATABASE_REFUSAL_HINT, DATABASE_TRIGGER_HINT, ownerExceptionHint, unknownOriginHint } from './connector-errors.ts';
import { readQueuedHeadBlockers } from './head-blocker.ts';
import { SOURCE_CHANGED_ORIGINS } from './publication-failure.ts';
+import { abandonedClaimCutoff } from './effect-journal.ts';
export const WRITER_NEXT_ACTIONS: Record<string, string> = {
unexpected_staging_bytes: 'Keep the worktree blocked and retain its staging files and recovery capacity. Compare the recorded staging size and hash, then reconcile unexpected bytes explicitly before retrying; never discard unverified staging files.',
@@ -49,13 +50,17 @@ export function writerNextAction(reason: string | null | undefined): string {
* ceiling or on `restart_required` the host admin restarts the named owner (`tell_user_to_run`); a lapsed claim resumes
* on its own once its transaction ends or the owner exits, so the fix is the resume (`run`).
*/
-export interface ClaimNextAction { code: 'claim_running' | 'claim_overdue' | 'claim_lapsed'; why: string; retry_after_ms: number | null; fix: RenderedAction }
+export interface ClaimNextAction { code: 'claim_running' | 'claim_overdue' | 'claim_lapsed' | 'claim_owner_exited'; why: string; retry_after_ms: number | null; fix: RenderedAction }
export function claimNextAction(claim: ClaimState, stall: ClaimStall | null, policy: { budgetMs: number | null; ceilingMs: number }, sourceId: string | null,
owner: { row: ListedConsumer | null; live: boolean | null }, now = Date.now()): ClaimNextAction {
const ctx = cliRenderContext();
const status = ['gbrain', 'sources', 'writer', 'status', ...(sourceId ? ['--source', sourceId] : []), '--json'];
const who = claim.owner ? `${claim.owner.kind} pid ${claim.owner.pid}` : 'the owner';
const where = `${claim.step ? ` at step ${claim.step}` : ''}${claim.waiting_on !== 'unknown' ? ` (waiting on ${claim.waiting_on})` : ''}${claim.last_sql ? `, last statement ${claim.last_sql.label}` : ''}`;
+ if (claim.owner_exited) {
+ return { code: 'claim_owner_exited', retry_after_ms: null, why: `${who} has exited${where} and nothing renews the claim; the next write from this machine (or a gbrain process already running here, within seconds) releases it and runs the request again, so the source moves without anyone acting.`,
+ fix: renderAction({ argv: status, consent: [], actor: 'agent', requires_exclusive: false, why: 'Read-only: rerun after the next write from this machine to see the request requeued or committed.', verify: { argv: status } }, ctx) };
+ }
if (claim.phase === 'publication_transaction') {
return { code: 'claim_lapsed', retry_after_ms: null, why: `The claim lapsed while running; the request is requeued once the transaction holding its row ends or the owner exits, and the next consumer claims it.`,
fix: renderAction({ argv: status, consent: [], actor: 'agent', requires_exclusive: false, why: 'Read-only: rerun to see the request requeued or claimed again.', verify: { argv: status } }, ctx) };
@@ -163,8 +168,9 @@ export async function readWriterDiagnostics(engine: BrainEngine) {
const blockers = await engine.executeRaw<{ id: string; request_id: string; worktree_id: string | null; source_id: string | null; state: WriteRequestState; created_at: Date | string; blocked_reason: string | null; error_code: string | null;
operation: string; intent_kind: string | null; preparation_attempts: number | null } & Omit<ClaimRow, 'state'>>(
`SELECT request_id,worktree_id,source_id,state,blocked_reason,error_code,created_at,claim_phase,execution_token,claim_expires_at<now() AS claim_lapsed,publication_started,
- operation,intent->>'kind' AS intent_kind,preparation_attempts,id::text AS id
- FROM persistence_requests WHERE state IN ('queued','deferred','running','recovering') OR blocked_reason IS NOT NULL ORDER BY sequence LIMIT 100`);
+ operation,intent->>'kind' AS intent_kind,preparation_attempts,id::text AS id,
+ ($1::timestamptz IS NOT NULL AND state='running' AND recovery IS NULL AND updated_at<$1::timestamptz) AS abandoned
+ FROM persistence_requests WHERE state IN ('queued','deferred','running','recovering') OR blocked_reason IS NOT NULL ORDER BY sequence LIMIT 100`, [abandonedClaimCutoff(engine)?.toISOString() ?? null]);
const queue = await engine.executeRaw(`SELECT state,COUNT(*)::integer AS count,COALESCE(SUM(intent_bytes),0)::text AS intent_bytes,
MIN(created_at) AS oldest_request_at,
MAX(EXTRACT(EPOCH FROM (now()-created_at))*1000)::bigint::text AS oldest_age_ms
@@ -189,7 +195,8 @@ export async function readWriterDiagnostics(engine: BrainEngine) {
const hostId = existingHostIdFor(engine);
const consumerRows = hostId ? await listHostConsumers(engine, hostId).catch(() => []) : [];
const tracked = (engine as { getPoolDiagnostics?: () => { tracked?: unknown } | null }).getPoolDiagnostics?.()?.tracked ?? null;
- const claimStates = blockers.map(row => claimStateOf({ state: row.state, claim_phase: row.claim_phase, execution_token: row.execution_token, claim_lapsed: row.claim_lapsed, publication_started: row.publication_started }));
+ const claimStates = blockers.map(row => claimStateOf({ state: row.state, claim_phase: row.claim_phase, execution_token: row.execution_token, claim_lapsed: row.claim_lapsed,
+ publication_started: row.publication_started, abandoned: row.abandoned, probe_owner: true }));
const backends = await readOwnerBackends(engine, claimStates.flatMap(state => state?.owner ? [state.owner] : []));
const headBlockers = await readQueuedHeadBlockers(engine, blockers, hostId);
// C-NEW-4: the consumer is per process, so this is the answering process's own ingress, never proof the brain's owner is down.
@@ -202,7 +209,7 @@ export async function readWriterDiagnostics(engine: BrainEngine) {
limits: Object.fromEntries(Object.entries(limits).filter(([, value]) => Number.isFinite(value))), capacity: capacityDiagnostics(counters, limits), preparation_policy,
recent_failures: failures.map(row => ({ ...row, next_action: recentFailureNextAction(row.error_detail?.origin, row.request_id) })),
backend_visibility: backends.visibility,
- blockers: blockers.map(({ id: rowId, claim_phase: _phase, execution_token: _token, claim_lapsed: _lapsed, publication_started: _started, ...row }, index) => {
+ blockers: blockers.map(({ id: rowId, claim_phase: _phase, execution_token: _token, claim_lapsed: _lapsed, publication_started: _started, abandoned: _abandoned, ...row }, index) => {
const health = writeHealth(row);
const advice = writerNextAction(row.blocked_reason ?? row.error_code);
// #6176: a running request names the phase its claim is in and how long it has held it; #6278: its step, wait cause, owner
@@ -216,7 +223,7 @@ export async function readWriterDiagnostics(engine: BrainEngine) {
const ownerBackends = state?.owner ? backends.byOwner.get(`gbrain ${state.owner.kind}:${state.owner.pid}:${(state.owner.nonce ?? '').slice(0, 8)}`) ?? [] : [];
const claim = state ? { ...state, ...(state.owner ? { owner: { ...state.owner, backend: ownerBackends, backend_visibility: backends.visibility } } : {}),
...(budget === null ? {} : { budget_ms: budget }), stall, ...claimPool(engine, state, ownerRow), pool_tracked_subset: tracked,
- owner_live: ownerRow ? ownerRow.liveness === 'live' : null, owner_mode: ownerRow?.mode ?? null,
+ owner_live: state.owner_exited ? false : ownerRow ? ownerRow.liveness === 'live' : null, owner_mode: ownerRow?.mode ?? null,
next: claimNextAction(state, stall, { budgetMs: budget, ceilingMs }, row.source_id, { row: ownerRow, live: ownerRow ? ownerRow.liveness === 'live' : null }) } : undefined;
// #6543 (B2): a queued head names what keeps it unclaimed, and the cancel command only when this caller may cancel the blocking request.
const head = headBlockers.get(rowId);
diff --git a/src/core/persistence/effect-git.ts b/src/core/persistence/effect-git.ts
index 09d26caa29..0ac43b364c 100644
--- a/src/core/persistence/effect-git.ts
+++ b/src/core/persistence/effect-git.ts
@@ -298,9 +298,13 @@ export function commitMessageArgs(paragraphs: string[], dir: string | (() => str
*
* `owner` is the caller's writer lock on this checkout: with it, an index lock
* that is provably abandoned is reclaimed and the step retried (#6932).
+ *
+ * #9227: `commits` receives, for each requested path this call committed, the commit's oid, which the Git effect
+ * records in its outcome so a receipt read later can name it.
*/
export async function commitGitTargets(root: string, relativePaths: string[], signal?: AbortSignal,
- notes?: ReadonlyMap<string, GitCommitNote>, owner?: GitEffectOwner, expected?: ReadonlyMap<string, string | null>): Promise<Map<string, GitOutcome | OperationError>> {
+ notes?: ReadonlyMap<string, GitCommitNote>, owner?: GitEffectOwner, expected?: ReadonlyMap<string, string | null>,
+ commits?: Map<string, string>): Promise<Map<string, GitOutcome | OperationError>> {
const results = new Map<string, GitOutcome | OperationError>();
await withSession(root, signal, owner, true, async session => {
// One ls-files, one status, one add and one diff for the group; targets
@@ -424,7 +428,11 @@ export async function commitGitTargets(root: string, relativePaths: string[], si
`Another commit changed ${moved.join(', ')} while this group of ${commitPaths.length} file(s) was being committed, so nothing was committed and the other commit stays; the effect retries against the new HEAD.`)
: gitFailure('Cannot commit the canonical Git target.',
`git commit failed for ${commitPaths.length} file(s); a missing Git identity (user.name, user.email) in that checkout is one cause to check.`);
- for (const c of changed) results.set(c.requested, reset && superseded.includes(c.path) ? { git: 'superseded', reason: 'committed_by_another_writer' } : outcome);
+ for (const c of changed) {
+ const result = reset && superseded.includes(c.path) ? { git: 'superseded', reason: 'committed_by_another_writer' } : outcome;
+ results.set(c.requested, result);
+ if (sha && result === outcome) commits?.set(c.requested, sha);
+ }
});
return results;
}
@@ -760,10 +768,10 @@ export const DURABILITY_NOT_ENABLED = { git: 'skipped', reason: 'durability_not_
*/
export async function publishGitEffect(root: string, relativePath: string, signal?: AbortSignal,
hardened?: boolean, note?: GitCommitNote, binding?: Pick<WorktreeBinding, 'git_durability'> | null, owner?: GitEffectOwner,
- expectedHash?: string | null): Promise<Record<string, unknown>> {
+ expectedHash?: string | null, commits?: Map<string, string>): Promise<Record<string, unknown>> {
if (!(hardened ?? (await gitDurabilityPolicy(binding, root)).durable)) return { ...DURABILITY_NOT_ENABLED };
const outcome = (await commitGitTargets(root, [relativePath], signal, note ? new Map([[relativePath, note]]) : undefined, owner,
- expectedHash === undefined ? undefined : new Map([[relativePath, expectedHash]]))).get(relativePath)!;
+ expectedHash === undefined ? undefined : new Map([[relativePath, expectedHash]]), commits)).get(relativePath)!;
if (outcome instanceof OperationError) throw outcome;
if (outcome.reason === 'target_absent' && !(await gitRootAhead(root, signal))) return outcome;
const pushed = await pushGitRoot(root, signal);
diff --git a/src/core/persistence/effect-journal.ts b/src/core/persistence/effect-journal.ts
index ea2dd72d2c..6cada79613 100644
--- a/src/core/persistence/effect-journal.ts
+++ b/src/core/persistence/effect-journal.ts
@@ -1,6 +1,7 @@
import { randomUUID } from 'node:crypto';
import { relative, sep } from 'node:path';
import type { BrainEngine } from '../engine.ts';
+import { ENGINE_ROOT } from '../engine-root.ts';
import type { PageSnapshot } from '../page-state/types.ts';
import { opError } from '../ops/contract.ts';
import { readFix } from '../ops/op-fix.ts';
@@ -281,9 +282,22 @@ export async function claimCoalescedGitEffects(engine: BrainEngine, hostId: stri
return rows.sort((a, b) => Number(a.id) - Number(b.id));
}
+/** When this process started; on PGLite no claim written earlier can belong to a live owner. */
+export const PROCESS_STARTED_AT = new Date(performance.timeOrigin);
/**
- * PGLite only: the datastore admits one process, and this one opened it at
- * `processStartedAt`, so a claim last written before then was held by an owner
+ * #9227: the instant before which every running claim on this brain is abandoned, or null when this process cannot
+ * prove it. Only a PGLite engine that holds its datastore lock proves it: no other process can have renewed a claim
+ * since this one started, so the owner of an older claim has exited. A process without the lock, and every Postgres
+ * brain, gets null and releases nothing here.
+ */
+export function abandonedClaimCutoff(engine: BrainEngine): Date | null {
+ if (engine.kind !== 'pglite') return null;
+ const root = (engine as { [ENGINE_ROOT]?: { holdsDatastoreLock?: () => boolean } })[ENGINE_ROOT];
+ return root?.holdsDatastoreLock?.() === true ? PROCESS_STARTED_AT : null;
+}
+/**
+ * PGLite only, and only while this engine holds the datastore lock (`abandonedClaimCutoff`): the datastore admits one
+ * process, and this one opened it at `processStartedAt`, so a claim last written before then was held by an owner
* that has exited (a crash or SIGKILL). Release those claims now instead of
* waiting out their leases (effects 2 minutes, requests 30 s); a withdrawal
* mirror left running would otherwise also hold back writes to its pages.
@@ -296,7 +310,7 @@ export async function claimCoalescedGitEffects(engine: BrainEngine, hostId: stri
* kill loop reaches `persistence.max_preparation_attempts`.
*/
export async function releaseAbandonedClaims(engine: BrainEngine, processStartedAt: Date, chargePreparing = false): Promise<number> {
- if (engine.kind !== 'pglite') return 0;
+ if (!abandonedClaimCutoff(engine)) return 0;
const effects = await engine.executeRaw(`UPDATE persistence_effects SET state='queued',execution_token=NULL,claim_expires_at=NULL,claim_owner=NULL,
next_attempt_at=LEAST(next_attempt_at,now()) WHERE state='running' AND recovery IS NULL AND updated_at<$1::timestamptz
AND ${PERSISTENCE_PROTOCOL_PREDICATE} RETURNING id`, [processStartedAt.toISOString()]);
@@ -401,6 +415,29 @@ export async function failEffect(engine: SqlEngine, effect: PersistenceEffect, r
WHERE id=$1 AND execution_token=$2::uuid AND recovery IS NULL AND ${PERSISTENCE_PROTOCOL_PREDICATE}`, [effect.id, effect.execution_token, reason]);
}
+/**
+ * #9227: what a classic write's receipt says about Git, read from its Git effect each time the receipt is read. Its
+ * publication stored `git_state: queued` (`queuePublicationEffects`) and nothing rewrites it, so the effect row is the
+ * live answer: `committed` with the commit's oid (an effect that recorded none, from an older build or a file Git
+ * already held, reads `committed` without one), `superseded`, `skipped` or `failed` with the reason, `retrying`
+ * with the last attempt's error, `running`, `recovering`, and `queued` only while the effect is queued. Null when the
+ * request has no Git effect row.
+ */
+export async function gitReceiptState(engine: SqlEngine, requestId: string): Promise<{ git_state: string; git_commit?: string; git_reason?: string } | null> {
+ const [row] = await engine.executeRaw<{ state: string; error_code: string | null; recovering: boolean; outcome: Record<string, unknown> | null }>(
+ `SELECT state,error_code,outcome,recovery IS NOT NULL AS recovering FROM persistence_effects WHERE request_id=$1::uuid AND kind='git' ORDER BY id DESC LIMIT 1`, [requestId]);
+ if (!row) return null;
+ const reason = (value: unknown) => typeof value === 'string' && /^[a-z_]{1,80}$/.test(value) ? { git_reason: value } : {};
+ if (row.recovering) return { git_state: 'recovering' };
+ if (row.state === 'failed') return { git_state: 'failed', ...reason(row.error_code) };
+ if (row.state === 'queued') return row.error_code ? { git_state: 'retrying', ...reason(row.error_code) } : { git_state: 'queued' };
+ if (row.state !== 'committed') return { git_state: row.state };
+ const git = row.outcome?.git;
+ if (git === 'superseded' || git === 'skipped') return { git_state: git, ...reason(row.outcome?.reason) };
+ const commit = row.outcome?.commit;
+ return { git_state: 'committed', ...(typeof commit === 'string' && /^[0-9a-f]{40,64}$/.test(commit) ? { git_commit: commit } : {}) };
+}
+
/** Only public kind/state/reason, aggregated so withdrawal page counts cannot leak. */
export async function publicEffectsForRequest(engine: SqlEngine, requestId: string): Promise<Array<{ kind: EffectKind; state: string; reason?: string; push?: string; added?: number; removed?: number;
invalid_targets?: Array<{ slug: string; reason: 'invalid_slug' }> }>> {
diff --git a/src/core/persistence/effects.ts b/src/core/persistence/effects.ts
index a8cb6fae13..89fcba62b1 100644
--- a/src/core/persistence/effects.ts
+++ b/src/core/persistence/effects.ts
@@ -346,8 +346,10 @@ async function publishGitPage(engine: BrainEngine, effect: PersistenceEffect, bi
const root = binding.local_path;
if (!targetedWithdrawalEffect(effect) && !effect.data.source_scan) {
const target = await singleFileGitTarget(engine, effect, { ...binding, local_path: root }, attempt);
- if (target !== null) await completeEffect(engine, effect, await publishGitEffect(root, target, opts.signal, hardened, await gitCommitNote(engine, effect), undefined, owner,
- effect.data.expected_hash));
+ if (target === null) return;
+ const commits = new Map<string, string>();
+ const outcome = await publishGitEffect(root, target, opts.signal, hardened, await gitCommitNote(engine, effect), undefined, owner, effect.data.expected_hash, commits);
+ await completeEffect(engine, effect, commits.has(target) ? { ...outcome, commit: commits.get(target) } : outcome);
return;
}
// Only a page walk reads snapshots; a single-file effect completes by its recorded hash.
@@ -981,9 +983,10 @@ async function runEffectPass(engine: BrainEngine, config: GBrainConfig, opts: Ef
}
const notes = new Map<string, GitCommitNote>();
for (const { effect, path } of targets) { const note = await gitCommitNote(engine, effect); if (note) notes.set(path, note); }
+ const commits = new Map<string, string>();
const outcomes = targets.length ? await commitGitTargets(binding.local_path, targets.map(t => t.path), opts.signal, notes,
{ writerLock: lock, onReclaim: () => { reclaimed = true; } },
- new Map(targets.flatMap(t => t.effect.data.expected_hash === undefined ? [] : [[t.path, t.effect.data.expected_hash] as const]))) : new Map();
+ new Map(targets.flatMap(t => t.effect.data.expected_hash === undefined ? [] : [[t.path, t.effect.data.expected_hash] as const])), commits) : new Map();
const pending = unpushed.get(binding.local_path) ?? { binding, items: [] };
// #7089: an absent target whose removal an earlier attempt committed (its push failed) joins the push like an
// unchanged one while the root is ahead of its upstream; on a root that is not ahead it completes without a push.
@@ -993,7 +996,7 @@ async function runEffectPass(engine: BrainEngine, config: GBrainConfig, opts: Ef
const outcome = outcomes.get(path)!;
if (outcome instanceof OperationError) await recordFailure(engine, effect, outcome, opts.signal, attempt.target);
else if (outcome.reason === 'target_absent' && !ahead) await completeEffect(engine, effect, outcome);
- else pending.items.push({ effect, outcome, target: attempt.target });
+ else pending.items.push({ effect, outcome: commits.has(path) ? { ...outcome, commit: commits.get(path) } : outcome, target: attempt.target });
}
for (const effect of committedAtPublication) pending.items.push({ effect, outcome: { git: 'committed', at: 'publication' } });
if (pending.items.length) unpushed.set(binding.local_path, pending);
diff --git a/src/core/persistence/params.ts b/src/core/persistence/params.ts
index 2689e7c01f..e240d8f5ca 100644
--- a/src/core/persistence/params.ts
+++ b/src/core/persistence/params.ts
@@ -117,6 +117,7 @@ export const WRITE_RECEIPT_SCHEMA = {
file_written: { type: 'boolean' },
git_state: { type: 'string' },
git_commit: { type: 'string' },
+ git_reason: { type: 'string' },
},
},
created_at: { type: 'string' },
diff --git a/src/core/persistence/types.ts b/src/core/persistence/types.ts
index 5ff5dd1544..dd745e4347 100644
--- a/src/core/persistence/types.ts
+++ b/src/core/persistence/types.ts
@@ -54,8 +54,10 @@ export interface WriteReceipt {
mode: 'filesystem' | 'database';
file_written?: boolean;
git_state?: string;
- /** G2: the commit a strict (canonical: git) source's write was committed in before its row. */
+ /** G2: the commit a strict (canonical: git) source's write was committed in before its row; #9227: a classic write's Git effect commit. */
git_commit?: string;
+ /** #9227: why a classic write's Git effect is superseded, skipped, failed or retrying. */
+ git_reason?: string;
};
created_at?: string;
updated_at?: string;
@@ -129,6 +131,7 @@ export function isWriteReceipt(value: unknown): value is WriteReceipt {
if (value.persistence.file_written !== undefined && typeof value.persistence.file_written !== 'boolean') return false;
if (value.persistence.git_state !== undefined && typeof value.persistence.git_state !== 'string') return false;
if (value.persistence.git_commit !== undefined && typeof value.persistence.git_commit !== 'string') return false;
+ if (value.persistence.git_reason !== undefined && typeof value.persistence.git_reason !== 'string') return false;
}
return true;
}
@@ -160,6 +163,7 @@ export function publicWriteReceipt(receipt: WriteReceipt): WriteReceipt {
...(receipt.persistence.file_written !== undefined ? { file_written: receipt.persistence.file_written } : {}),
...(receipt.persistence.git_state !== undefined ? { git_state: receipt.persistence.git_state } : {}),
...(receipt.persistence.git_commit !== undefined ? { git_commit: receipt.persistence.git_commit } : {}),
+ ...(receipt.persistence.git_reason !== undefined ? { git_reason: receipt.persistence.git_reason } : {}),
} } : {}),
...(receipt.created_at !== undefined ? { created_at: receipt.created_at } : {}),
...(receipt.updated_at !== undefined ? { updated_at: receipt.updated_at } : {}),
diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts
index 139a57c8fc..0b2b0a5a6f 100644
--- a/src/core/pglite-engine.ts
+++ b/src/core/pglite-engine.ts
@@ -775,6 +775,11 @@ export class PGLiteEngine implements BrainEngine {
*/
walRepairReceipt: WalRepairReceipt | null = null;
+ /** #9227: whether this engine has its datastore open under the lock that admits one process (always true for an open in-memory store). */
+ holdsDatastoreLock(): boolean {
+ return this._db !== null && this._lock?.acquired === true;
+ }
+
get db(): PGLiteDB {
if (!this._db) throw new Error('PGLite not connected. Call connect() first.');
return this._db;
diff --git a/test/fixtures/stuck-head-worker.ts b/test/fixtures/stuck-head-worker.ts
index d7814e94ab..104fa9e5fb 100644
--- a/test/fixtures/stuck-head-worker.ts
+++ b/test/fixtures/stuck-head-worker.ts
@@ -3,7 +3,9 @@
* waits there, so the parent can SIGKILL the process at that point and read what the death leaves behind.
* Argument: a JSON config file. Prints one JSON line per event on stdout: `{event:'boundary', point}` when it reaches
* the seam for its own request, then nothing more (the parent kills it); without a seam, `{event:'done', state, ms}` or
- * `{event:'failure', code, message}`.
+ * `{event:'failure', code, message}`. With `hold`, the worker stays alive at the seam (and past a `write_pending` reply,
+ * which it reports) until the parent writes a line to its stdin; it then resumes from the seam, replays the same
+ * request_id and reports how the write ended. With `status`, it writes nothing and prints the request's writer-status row.
* GBRAIN_HOME is the parent's, so both processes are the same host.
*/
import { readFileSync } from 'node:fs';
@@ -11,6 +13,7 @@ import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
import { operations } from '../../src/core/operations.ts';
import { installFaultHook, type FaultPoint } from '../../src/core/persistence/fault-points.ts';
+import { runPersistenceAdministration } from '../../src/core/persistence/administration.ts';
export interface StuckHeadWorkerConfig {
engine: 'pglite' | 'postgres';
@@ -26,6 +29,10 @@ export interface StuckHeadWorkerConfig {
waitMs?: number;
/** A remote OAuth caller instead of the local CLI writer. */
auth?: Record<string, unknown>;
+ /** Stay alive at the seam until the parent writes a line to stdin, then finish the write. */
+ hold?: boolean;
+ /** Write nothing: print `{event:'status', blocker}`, the `gbrain sources writer status` row of `requestId` as this new process reads it. */
+ status?: boolean;
}
const emit = (event: Record<string, unknown>) => process.stdout.write(`${JSON.stringify(event)}\n`);
@@ -33,18 +40,30 @@ const emit = (event: Record<string, unknown>) => process.stdout.write(`${JSON.st
if (import.meta.main) {
const config: StuckHeadWorkerConfig = JSON.parse(readFileSync(process.argv[2]!, 'utf8'));
const engine = config.engine === 'pglite' ? new PGLiteEngine() : new PostgresEngine();
+ const go = config.hold ? new Promise<void>(resume => process.stdin.once('data', () => resume())) : new Promise<void>(() => {});
try {
await engine.connect(config.engine === 'pglite' ? { engine: 'pglite', database_path: config.databasePath } : { database_url: config.databaseUrl, poolSize: 4 });
+ if (config.status) {
+ const status = await runPersistenceAdministration(engine, 'writer_status', { source_id: config.sourceId }) as { blockers: Array<{ request_id: string }> };
+ emit({ event: 'status', blocker: status.blockers.find(row => row.request_id === config.requestId) ?? null });
+ process.exit(0);
+ }
if (config.point) installFaultHook(async (point, detail) => {
if (point !== config.point || detail.requestId !== config.requestId) return;
emit({ event: 'boundary', point });
- await new Promise(() => {});
+ await go;
});
const put = operations.find(op => op.name === 'put_page')!;
const started = performance.now();
- const out = await put.handler({ engine, config: { engine: config.engine, embedding_disabled: true } as never, logger: { info() {}, warn() {}, error() {} },
+ const write = () => put.handler({ engine, config: { engine: config.engine, embedding_disabled: true } as never, logger: { info() {}, warn() {}, error() {} },
dryRun: false, remote: config.auth !== undefined, sourceId: config.sourceId, ...(config.auth ? { auth: config.auth as never } : {}) },
{ slug: config.slug, content: config.content, request_id: config.requestId, wait_ms: config.waitMs ?? 30_000 });
+ const out = await write().catch(async error => {
+ if (!config.hold || (error as { code?: string }).code !== 'write_pending') throw error;
+ emit({ event: 'failure', code: 'write_pending', message: (error as Error).message });
+ await go;
+ return write();
+ });
emit({ event: 'done', state: (out as { state?: string }).state ?? null, ms: Math.round(performance.now() - started) });
process.exit(0);
} catch (error) {
diff --git a/test/persistence-receipt-git-state.test.ts b/test/persistence-receipt-git-state.test.ts
index 7d1378f8c6..e23cd6c287 100644
--- a/test/persistence-receipt-git-state.test.ts
+++ b/test/persistence-receipt-git-state.test.ts
@@ -1,10 +1,10 @@
/**
- * A classic (not `canonical: git`) worktree-backed page write: what its receipt says about Git once the Git effect
- * has committed the file. Pins master's answer: the receipt's `persistence.git_state` is the value its publication
- * transaction stored (`queued`, effect-journal.ts queuePublicationEffects) and never moves, because the Git effect's
- * completion (completeEffect) writes only its own effect row, whose outcome is `{ git: 'committed' }` without the commit
- * oid; the request's outcome is not touched after the write commits. The live state is the receipt's `effects` entry
- * for `git`. No commit oid is recorded anywhere for such a write. PGLite and, with DATABASE_URL, Postgres.
+ * A classic (not `canonical: git`) worktree-backed page write: what its receipt says about Git. Its publication stores
+ * `persistence.git_state: queued` (effect-journal.ts queuePublicationEffects) and nothing rewrites the request row, so
+ * a receipt read later (#9227) takes `git_state` and `git_commit` from the Git effect row: queued while the effect is,
+ * `committed` with the oid the effect recorded once it lands, `committed` without one for an effect row from an older
+ * build, and `superseded`, `failed` or `retrying` with the effect's reason. The stored outcome stays as written.
+ * PGLite and, with DATABASE_URL, Postgres.
*/
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { randomUUID } from 'node:crypto';
@@ -60,11 +60,14 @@ for (const backend of testBackends()) describe(`[${backend}] a classic source's
rmSync(home, { recursive: true, force: true });
}, 60_000);
- test('git_state stays queued with no commit oid after the commit lands; only effects[git] reads committed', async () => withEnv(env, async () => {
+ test('git_state reads queued until the Git effect commits, then committed with its oid; the stored row is not rewritten', async () => withEnv(env, async () => {
const requestId = randomUUID();
const written = await op('put_page').handler(ctx(), { slug: 'notes/landed', request_id: requestId, wait_ms: 20_000,
content: '---\ntitle: Landed\ntype: note\n---\n# Landed\n\nA page whose commit lands later.\n' }) as Record<string, any>;
expect(written).toMatchObject({ state: 'committed', persistence: { git_state: 'queued' } });
+ const read = () => op('get_write_request').handler(ctx(), { request_id: requestId }) as Promise<Record<string, any>>;
+ await disposePersistenceConsumer(engine);
+ expect((await read()).persistence).toEqual({ mode: 'filesystem', file_written: true, git_state: 'queued' });
const before = git(repo, 'rev-parse', 'HEAD');
await disposePersistenceConsumer(engine);
await runPersistenceEffects(engine, { engine: backend, embedding_disabled: true }, { hostId: localHostId(), limit: 8 });
@@ -74,14 +77,22 @@ for (const backend of testBackends()) describe(`[${backend}] a classic source's
const [effect] = await engine.executeRaw<{ state: string; outcome: Record<string, unknown> }>(
`SELECT e.state,e.outcome FROM persistence_effects e JOIN persistence_requests r ON r.id=e.request_id WHERE r.request_id=$1::uuid AND e.kind='git'`, [requestId]);
expect(effect!.state).toBe('committed');
- expect(effect!.outcome).toMatchObject({ git: 'committed' });
- expect(JSON.stringify(effect!.outcome)).not.toContain(head);
- const receipt = await op('get_write_request').handler(ctx(), { request_id: requestId }) as Record<string, any>;
- expect(receipt.persistence).toEqual({ mode: 'filesystem', file_written: true, git_state: 'queued' });
- expect(receipt.persistence.git_commit).toBeUndefined();
+ expect(effect!.outcome).toMatchObject({ git: 'committed', commit: head });
+ const receipt = await read();
+ expect(receipt.persistence).toEqual({ mode: 'filesystem', file_written: true, git_state: 'committed', git_commit: head });
expect(receipt.effects).toContainEqual(expect.objectContaining({ kind: 'git', state: 'committed' }));
const [stored] = await engine.executeRaw<{ outcome: Record<string, any> }>('SELECT outcome FROM persistence_requests WHERE request_id=$1::uuid', [requestId]);
expect(stored!.outcome.persistence).toEqual({ mode: 'filesystem', file_written: true, git_state: 'queued' });
expect(JSON.stringify(stored!.outcome)).not.toContain(head);
+ // The same receipt over other Git effect rows, each set the way the effect worker leaves it.
+ const setEffect = (sql: string) => engine.executeRaw(`UPDATE persistence_effects e SET ${sql} FROM persistence_requests r WHERE r.id=e.request_id AND r.request_id=$1::uuid AND e.kind='git'`, [requestId]);
+ await setEffect(`outcome=e.outcome-'commit'`);
+ expect((await read()).persistence).toEqual({ mode: 'filesystem', file_written: true, git_state: 'committed' });
+ await setEffect(`outcome='{"git":"superseded","reason":"committed_by_another_writer"}'::jsonb`);
+ expect((await read()).persistence).toMatchObject({ git_state: 'superseded', git_reason: 'committed_by_another_writer' });
+ await setEffect(`state='queued',error_code='git_index_locked',outcome=NULL`);
+ expect((await read()).persistence).toMatchObject({ git_state: 'retrying', git_reason: 'git_index_locked' });
+ await setEffect(`state='failed',error_code='targets_parked'`);
+ expect((await read()).persistence).toMatchObject({ git_state: 'failed', git_reason: 'targets_parked' });
}), 120_000);
});
diff --git a/test/persistence-stuck-head-9227.test.ts b/test/persistence-stuck-head-9227.test.ts
index eb9b1beaa0..6542af7de3 100644
--- a/test/persistence-stuck-head-9227.test.ts
+++ b/test/persistence-stuck-head-9227.test.ts
@@ -1,7 +1,7 @@
/**
* gbrain#9227: a write killed at the head of a source's queue, or one whose principal is gone, must not wedge the
* source. Each case kills a real child process (test/fixtures/stuck-head-worker.ts) with SIGKILL at a fault seam and
- * reads what master does with the head it left, on PGLite and, with DATABASE_URL, on Postgres:
+ * reads what becomes of the head it left, on PGLite and, with DATABASE_URL, on Postgres:
*
* 1. Killed while preparing: the claim is `running`/`preparing` with a 30 s lease and no recovery record.
* 2. Killed inside the publication transaction of a database-only page: the transaction rolls back and the claim keeps
@@ -11,14 +11,15 @@
* 4. The submitting OAuth client is revoked while its write is stuck at the head: no other principal can see or
* cancel it (control.ts cancelWriteRequest is scoped to the principal).
*
- * Findings this file pins: on Postgres a later write's own one-shot consumer proves a same-host owner dead (#6650) and
- * reclaims cases 1, 2 and 4 at once; case 3 is recovered at once on both engines (the recovery scan); a revoked
- * principal's write ends `failed`/`permission_denied` when it is next prepared. Two cases wait out the lease: an owner
- * Postgres cannot prove dead (another boot or pid namespace), and every database-only head on PGLite, where the idle
- * probe never counts a claim left by an earlier process, so cases 1, 2 and 4 hold their source for the rest of the
- * lease (30 s or 365 s). Meanwhile later writes are accepted and wait (`write_pending`), and `gbrain sources writer
- * status` reads the dead claim as a live one (`claim_running`, `resumes_on_its_own: false`). A lapsed lease is
- * modelled by moving `claim_expires_at` into the past, the clock the reclaim reads.
+ * What this file pins: the next write to the source, from a new process, commits the head at once in cases 1, 2 and 4
+ * on both engines. On Postgres its one-shot consumer proves a same-host owner dead (#6650); on PGLite it holds the
+ * datastore lock, so every claim written before it started is abandoned and its first scan releases them (#9227).
+ * Case 3 is recovered at once on both engines (the recovery scan). A revoked principal's write ends
+ * `failed`/`permission_denied` when it is next prepared. Meanwhile `gbrain sources writer status`, run from a new
+ * process, reports the owner exited (`claim_owner_exited`, `resumes_on_its_own: true`). A live owner keeps its claim: a
+ * second PGLite process cannot open the brain at all, and Postgres cannot prove a live owner dead. An owner Postgres
+ * cannot prove dead (another boot or pid namespace) still holds the head for the rest of its lease (30 s, or 365 s
+ * while publishing); a lapsed lease is modelled by moving `claim_expires_at` into the past, the clock the reclaim reads.
*/
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
import { randomUUID } from 'node:crypto';
@@ -29,7 +30,6 @@ import { join } from 'node:path';
import type { BrainEngine } from '../src/core/engine.ts';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
-import { runPersistenceAdministration } from '../src/core/persistence/administration.ts';
import { claimWorktree } from '../src/core/persistence/ownership.ts';
import { cancelWriteRequest } from '../src/core/persistence/control.ts';
import { readLocalWriter } from '../src/core/persistence/identity.ts';
@@ -42,7 +42,7 @@ import { withEnv } from './helpers/with-env.ts';
const page = (title: string) => `---\ntitle: ${title}\ntype: note\n---\n# ${title}\n\nBody of ${title}.\n`;
const FOREIGN_BOOT = '00000000-0000-0000-0000-000000000000';
type Head = { state: string; error_code: string | null; preparation_attempts: number; recovery: boolean; phase: string | null; token: string | null; lease_s: number | null };
-type WorkerEvent = { event: 'boundary' | 'done' | 'failure'; state?: string; code?: string; ms?: number; message?: string };
+type WorkerEvent = { event: 'boundary' | 'done' | 'failure' | 'status'; point?: string; state?: string; code?: string; ms?: number; message?: string; blocker?: Record<string, any> | null };
for (const backend of testBackends()) describe(`[${backend}] a stuck head of a source's write queue (#9227)`, () => {
let engine: BrainEngine;
@@ -78,22 +78,36 @@ for (const backend of testBackends()) describe(`[${backend}] a stuck head of a s
rmSync(home, { recursive: true, force: true });
}, 60_000);
- /** Runs one put_page in a child process on this brain and host; with a seam, SIGKILLs it there. PGLite is closed meanwhile (one process at a time). */
- async function worker(config: Partial<StuckHeadWorkerConfig> & { slug: string }): Promise<WorkerEvent> {
- await disposePersistenceConsumer(engine);
- if (!postgres) await engine.disconnect();
+ /** Starts one put_page in a child process on this brain and host; `next()` resolves with its next event. */
+ function spawnWorker(config: Partial<StuckHeadWorkerConfig> & { slug: string }) {
const requestId = config.requestId ?? randomUUID();
const file = join(home, `worker-${requestId}.json`);
writeFileSync(file, JSON.stringify({ engine: backend, databasePath: dataDir, databaseUrl, sourceId: 'default', content: page(config.slug), requestId, ...config }));
- const child = spawn('bun', [join(import.meta.dir, 'fixtures/stuck-head-worker.ts'), file], { env: { ...process.env, HOME: home, GBRAIN_HOME: home }, stdio: ['ignore', 'pipe', 'pipe'] });
- let out = '', err = '';
- child.stderr.on('data', chunk => { err += chunk; });
+ const child = spawn('bun', [join(import.meta.dir, 'fixtures/stuck-head-worker.ts'), file], { env: { ...process.env, HOME: home, GBRAIN_HOME: home }, stdio: [config.hold ? 'pipe' : 'ignore', 'pipe', 'pipe'] });
+ let out = '', err = '', seen = 0;
+ const waiters: Array<() => void> = [];
+ child.stderr!.on('data', chunk => { err += chunk; });
+ child.stdout!.on('data', chunk => { out += chunk; waiters.splice(0).forEach(wake => wake()); });
const exited = new Promise<void>(resolve => child.once('exit', () => resolve()));
+ let done = false;
+ void exited.then(() => setTimeout(() => { done = true; waiters.splice(0).forEach(wake => wake()); }, 100));
+ const next = async (): Promise<WorkerEvent> => {
+ for (;;) {
+ const lines = out.split('\n').filter(text => text.startsWith('{') && text.endsWith('}'));
+ if (lines.length > seen) return JSON.parse(lines[seen++]!);
+ if (done) throw new Error(`worker exited without an event: ${out}${err.slice(-2000)}`);
+ await new Promise<void>(wake => waiters.push(wake));
+ }
+ };
+ return { child, exited, next };
+ }
+ /** Runs one put_page in a child process; with a seam, SIGKILLs it there. PGLite is closed meanwhile (one process at a time). */
+ async function worker(config: Partial<StuckHeadWorkerConfig> & { slug: string }): Promise<WorkerEvent> {
+ await disposePersistenceConsumer(engine);
+ if (!postgres) await engine.disconnect();
try {
- const event = await new Promise<WorkerEvent>((resolve, reject) => {
- child.stdout.on('data', chunk => { out += chunk; const line = out.split('\n').find(text => text.startsWith('{')); if (line) resolve(JSON.parse(line)); });
- child.once('exit', () => setTimeout(() => reject(new Error(`worker exited without an event: ${out}${err.slice(-2000)}`)), 100));
- });
+ const { child, exited, next } = spawnWorker(config);
+ const event = await next();
if (event.event === 'boundary') child.kill('SIGKILL');
await exited;
return event;
@@ -106,10 +120,8 @@ for (const backend of testBackends()) describe(`[${backend}] a stuck head of a s
const head = async (requestId: string): Promise<Head> => (await engine.executeRaw<Head>(`SELECT state,error_code,preparation_attempts,recovery IS NOT NULL AS recovery,
claim_phase->>'phase' AS phase,execution_token::text AS token,round(EXTRACT(EPOCH FROM claim_expires_at-updated_at))::int AS lease_s
FROM persistence_requests WHERE request_id=$1::uuid`, [requestId]))[0]!;
- const blocker = async (source: string, requestId: string) => {
- const status = await withEnv(env, () => runPersistenceAdministration(engine, 'writer_status', { source_id: source })) as { blockers: Array<Record<string, any>> };
- return status.blockers.find(row => row.request_id === requestId)!;
- };
+ /** The request's `gbrain sources writer status` row as a new process on this host reads it. */
+ const blocker = async (source: string, requestId: string) => (await worker({ slug: 'unused', sourceId: source, requestId, status: true })).blocker!;
const lapse = (requestId: string) => engine.executeRaw(`UPDATE persistence_requests SET claim_expires_at=now()-interval '1 second' WHERE request_id=$1::uuid AND state='running'`, [requestId]);
/**
@@ -117,37 +129,32 @@ for (const backend of testBackends()) describe(`[${backend}] a stuck head of a s
* owner dead and the head and the write both commit at once. PGLite: the write waits behind the head until its lease lapses.
*/
async function laterWrites(source: string, stuck: string, slug: string): Promise<Head> {
- if (postgres) {
- const later = await worker({ slug, sourceId: source });
- expect(later).toMatchObject({ event: 'done', state: 'committed' });
- expect(later.ms!).toBeLessThan(10_000);
- return head(stuck);
- }
- const waiting = await worker({ slug, sourceId: source, waitMs: 3_000 });
- expect(waiting).toMatchObject({ event: 'failure', code: 'write_pending' });
- const held = await head(stuck);
- expect(held.state).toBe('running');
- await lapse(stuck);
- expect(await worker({ slug: `${slug}-after-lapse`, sourceId: source })).toMatchObject({ event: 'done', state: 'committed' });
- const [pending] = await engine.executeRaw<{ state: string }>(`SELECT state FROM persistence_requests WHERE slug=$1 AND source_id=$2`, [slug, source]);
- expect(pending?.state).toBe('committed');
+ const later = await worker({ slug, sourceId: source });
+ expect(later).toMatchObject({ event: 'done', state: 'committed' });
+ expect(later.ms!).toBeLessThan(10_000);
return head(stuck);
}
- test('killed while preparing: a 30 s claim under a dead owner; writer status calls it a live claim; the head commits', async () => {
+ test('killed while preparing: a 30 s claim under a dead owner; writer status reports the owner exited; the next write commits the head at once', async () => {
const stuck = randomUUID();
await killAt('consumer:preparing', { slug: 'notes/killed-preparing', requestId: stuck });
expect(await head(stuck)).toMatchObject({ state: 'running', phase: 'preparing', recovery: false, preparation_attempts: 0, lease_s: 30 });
const status = await blocker('default', stuck);
- expect(status).toMatchObject({ state: 'running', claim: { phase: 'preparing', resumes_on_its_own: false, owner_live: postgres ? true : null, next: { code: 'claim_running' } } });
+ expect(status).toMatchObject({ state: 'running', claim: { phase: 'preparing', owner_exited: true, resumes_on_its_own: true, owner_live: false, stall: null, next: { code: 'claim_owner_exited', fix: { next: 'run' } } } });
expect(await laterWrites('default', stuck, 'notes/after-preparing')).toMatchObject({ state: 'committed', recovery: false });
}, 120_000);
- test('killed inside the publication transaction of a database-only page: the dead owner keeps a 365 s publication lease; the head commits', async () => {
+ test('killed inside the publication transaction of a database-only page: a 365 s lease under a dead owner; the next write commits the head at once', async () => {
const stuck = randomUUID();
await killAt('publication:before_commit', { slug: 'notes/killed-publishing', requestId: stuck });
expect(await head(stuck)).toMatchObject({ state: 'running', phase: 'publishing', recovery: false, lease_s: 365 });
- expect(await blocker('default', stuck)).toMatchObject({ claim: { phase: 'publishing', resumes_on_its_own: false, next: { code: 'claim_running' } } });
+ expect(await blocker('default', stuck)).toMatchObject({ claim: { phase: 'publishing', owner_exited: true, resumes_on_its_own: true, owner_live: false, next: { code: 'claim_owner_exited' } } });
+ // The page lock the killed transaction held ended with it; the claim is a row, not a lock. Extract's derived-link
+ // replacement (derived-links.ts) takes this same lock and gets it at once while the head is still stuck.
+ const started = performance.now();
+ await engine.transaction(tx => tx.lockPageKeys([{ sourceId: 'default', slug: 'notes/killed-publishing' }]));
+ expect(performance.now() - started).toBeLessThan(5_000);
+ expect(await head(stuck)).toMatchObject({ state: 'running' });
expect(await laterWrites('default', stuck, 'notes/after-publishing')).toMatchObject({ state: 'committed' });
}, 120_000);
@@ -175,6 +182,38 @@ for (const backend of testBackends()) describe(`[${backend}] a stuck head of a s
expect(await laterWrites('default', stuck, 'notes/after-revoked')).toMatchObject({ state: 'failed', error_code: 'permission_denied' });
}, 120_000);
+ test('a live owner keeps its claim: a process that cannot open the brain (PGLite) or cannot prove the owner dead (Postgres) never releases it', async () => {
+ await disposePersistenceConsumer(engine);
+ if (!postgres) await engine.disconnect();
+ let owner: ReturnType<typeof spawnWorker> | undefined;
+ try {
+ const live = randomUUID();
+ owner = spawnWorker({ slug: 'notes/live-owner', requestId: live, point: 'consumer:preparing', hold: true });
+ expect(await owner.next()).toEqual({ event: 'boundary', point: 'consumer:preparing' });
+ const before = postgres ? await head(live) : null;
+ // A later write from another process: on PGLite it cannot take the datastore lock, so it runs no statement at all.
+ const later = spawnWorker({ slug: 'notes/behind-live-owner', waitMs: 3_000 });
+ const blocked = await later.next();
+ await later.exited;
+ expect(blocked.event).toBe('failure');
+ if (postgres) {
+ expect(blocked.code).toBe('write_pending');
+ expect(await head(live)).toMatchObject({ state: 'running', phase: 'preparing', token: before!.token });
+ } else expect(blocked.code).toBe('pglite_busy');
+ expect(await owner.next()).toMatchObject({ event: 'failure', code: 'write_pending' });
+ owner.child.stdin!.write('go\n');
+ // The owner's own 30 s foreground budget may have cut the held preparation and started it again; either way it commits.
+ let settled = await owner.next();
+ while (settled.event === 'boundary') settled = await owner.next();
+ expect(settled).toMatchObject({ event: 'done', state: 'committed' });
+ await owner.exited;
+ } finally {
+ owner?.child.kill('SIGKILL');
+ await owner?.exited;
+ if (!postgres) await withEnv(env, reopen);
+ }
+ }, 120_000);
+
if (postgres) test('an owner Postgres cannot prove dead (another boot) holds the head until its lease lapses, then the head commits', async () => {
const stuck = randomUUID();
await killAt('consumer:preparing', { slug: 'notes/foreign-owner', requestId: stuck });
@@ -184,6 +223,9 @@ for (const backend of testBackends()) describe(`[${backend}] a stuck head of a s
await engine.executeRaw(`UPDATE persistence_effects SET claim_owner=jsonb_set(claim_owner,'{owner,boot_id}',to_jsonb($2::text)) WHERE claim_owner->'owner'->>'pid'=$1`, [pid, FOREIGN_BOOT]);
const before = await head(stuck);
expect(before).toMatchObject({ state: 'running', phase: 'preparing', lease_s: 30 });
+ const status = await blocker('default', stuck);
+ expect(status.claim).toMatchObject({ resumes_on_its_own: false, next: { code: 'claim_running' } });
+ expect(status.claim.owner_exited).toBeUndefined();
expect(await worker({ slug: 'notes/behind-foreign-owner', waitMs: 3_000 })).toMatchObject({ event: 'failure', code: 'write_pending' });
expect(await head(stuck)).toMatchObject({ state: 'running', token: before.token });
await lapse(stuck);