Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(persistence): index committed foreground receipts so cold write claims stop walking the journal (#7649)

garrytan27ac0ad7d611
diff --git a/docs/architecture/key-files/core-persistence-continued.md b/docs/architecture/key-files/core-persistence-continued.md
index b6dfaeb406..c689681b49 100644
--- a/docs/architecture/key-files/core-persistence-continued.md
+++ b/docs/architecture/key-files/core-persistence-continued.md
@@ -41,7 +41,7 @@ The #6188 fence check and fence repair live in [fence repair](core-fence-repair.
- `src/core/persistence/switches.ts` + `preadmit-cache.ts` + `config-snapshot.ts` — the single-write fast path for foreground page writes on Postgres. `switches.ts` reads every write-path kill switch (`persistence.single_write_group` / `GBRAIN_SINGLE_WRITE_GROUP`, `persistence.preadmit_cache` / `GBRAIN_PREADMIT_CACHE`, `sync.waive_batch` / `GBRAIN_SYNC_WAIVE_BATCH`, `sync.foreground_priority` / `GBRAIN_SYNC_FOREGROUND_PRIORITY`, and `persistence.preparation_deadlines` / `GBRAIN_PREPARATION_DEADLINES` (#6278: off restores the old preparation predicate, see persistence-stalls.md); on by default, `0`/`false` off, env over config) in one `SELECT key,value FROM config WHERE key = ANY(...)` per engine, kept 5 s (the consumer tick reads it under its phase deadline), together with the four preparation budget keys (`readWriteSwitchSnapshot` returns `{ switches, preparation }`; `readPreparationPolicy` is the budgets alone); read-through views register their engine (`registerEngineView`) to share it. With `single_write_group` on: the consumer publishes an eligible single page write (put_page, capture, edit/delete/restore/revert without an internal `intent.kind` or group key) as a group of one through `publishGroup` (`publishSingleWrite`; its intent carries no group marker, its effects queue without `deferBatchReconcile`, and a group that does not commit falls back to `publishMutation`, reclaiming a claim its file restoration released); PGLite and refused shapes keep `publishMutation`. The waiter's own wake (`wake(true)`) claims without the periodic scans (recovery records still block the claim through `claimableWriteSql`); preparation reads the config table once (`config-snapshot.ts`) and sends its independent reads together; the coordinated (`coordinated: true`) put_page apply reuses the publisher's locked read as the import base, pipelines its follow-up statements and reads the page once after its last write for the read-back check, seal, core notice and receipt. Publication, the own admission and the recovery transactions run on the consumer's reserved lane (`onLane`: the idle-probe connection, taken while two other pool connections stay free and kept across writes; never lent while an idle probe runs on it, and probes skip it while lent), so their prepared statements stay warm; the waiter is told of the commit before the recovery record is cleared. `preadmit-cache.ts` (with `preadmit_cache` on, Postgres) answers the brain id, source row, worktree binding and local-writer reads of `submitPageMutation` from a per-engine cache (30 s, keyed by brain id; absences never cached); admission rechecks each under lock and the brain id against its protocol statement, and a refusal the cache could explain (`source_changed`, `permission_denied`, a changed brain) drops the cache and runs the whole preparation once more uncached. Pinned by `test/e2e/single-write-fast-path-postgres.test.ts` (switches, cache rechecks, replay, restoration after a failure past the file rename, the recovery-blocked direct claim) and `test/single-write-group-parity.test.ts` + `test/e2e/single-write-group-parity-postgres.test.ts` (row-for-row parity with the switches off); measured by `scripts/bench/foreground-put-page.ts`.
-- Foreground priority (#5984 Phase 4.5, `foreground_priority` switch, `journal.ts`): `claimableWriteSql` lets a non-sync request pass earlier queued sync rows of its root that have not started and do not name its page (slug, page id or `intent.renameFrom.slug`), claims it first, and holds a passed sync row while that request runs; `claimNextLaneHead` starts no lane head while such a request is unfinished, except one head per foreground commit since the run's last head claim (`seen_sequence`). A consumer whose claimed write passed sync rows (`passed_sync`) claims plain FIFO on that root after it commits (`owesSyncTurn`), so a stream of writes cannot starve the drain. The sync side's pauses (no admit-ahead for 60 s after a write, no group while one is queued, the 25-page credit wait) apply only with the switch off. Receipt health judges a claimed write only against earlier started writes. Pinned by `test/managed-sync-foreground-priority.test.ts` (two processes) and `test/persistence-diagnostics.test.ts`.
+- Foreground priority (#5984 Phase 4.5, `foreground_priority` switch, `journal.ts`): `claimableWriteSql` lets a non-sync request pass earlier queued sync rows of its root that have not started and do not name its page (slug, page id or `intent.renameFrom.slug`), claims it first, and holds a passed sync row while that request runs; `claimNextLaneHead` starts no lane head while such a request is unfinished, except one head per foreground commit since the run's last head claim (`seen_sequence`). Both claims find that commit through `FOREGROUND_WAITING_ON`'s probe, served by `persistence_requests_foreground_committed` (#7649, a partial index on committed non-sync rows by worktree and sequence, see [request indexes](core-persistence-request-indexes.md)), so a consumer's first claims after a restart, when the bound is 0, stay index probes on a large request table. A consumer whose claimed write passed sync rows (`passed_sync`) claims plain FIFO on that root after it commits (`owesSyncTurn`), so a stream of writes cannot starve the drain. The sync side's pauses (no admit-ahead for 60 s after a write, no group while one is queued, the 25-page credit wait) apply only with the switch off. Receipt health judges a claimed write only against earlier started writes. Pinned by `test/managed-sync-foreground-priority.test.ts` (two processes) and `test/persistence-diagnostics.test.ts`.
- `src/core/postgres-engine/reserved-transactions.ts` with `isPersistenceDeclaration` (`src/core/persistence/protocol.ts`) — BEGIN pipelining. `isPersistenceDeclaration` (GBRA-156) accepts exactly the `protocol.ts` declarations (transaction-local `set_config` calls and the brain row FOR SHARE). Admission, the FIFO claim, the effect claims and source guards, the receipt true-up, the single publication (`coordinator.ts`) and the consumer's lent lane transactions (`consumer.ts` `onLane`, through `reservedTransactions`) pass it as `TransactionOptions.pipelineFirst`, so behind a transaction-mode pooler the declaration goes out with BEGIN (one round trip less per transaction) while every later statement still waits for BEGIN's answer; if BEGIN fails, only the declaration has run, in autocommit, where it changes nothing. A reserved transaction gates every statement its body can reach, savepoint and nested-transaction bodies and `file()` included. A sync lane group's publication passes none: its begun mark must follow BEGIN's answer (`sync-lanes.ts` `awaitLaneBegin`). Tests: `test/persistence-pipeline-first.test.ts` (the predicate and the reviewed call sites), `test/e2e/pgbouncer-transaction-pipeline-6930.test.ts`.
- `src/core/persistence/checkout-state.ts` (`readCheckoutState`) — Q1: a canonical checkout against its last fetched upstream, read-only and time-bounded (no fetch): ahead/behind, `diverged`, each commit the upstream lacks (sha, author, date, subject, paths; at most 20) and porcelain counts. `sources writer status` reports it as `checkouts`; doctor `git_convergence` adds the commits and counts for a managed checkout and warns at once on a diverged one.
diff --git a/docs/architecture/key-files/core-persistence-request-indexes.md b/docs/architecture/key-files/core-persistence-request-indexes.md
index 0cf6b662ab..697e455eda 100644
--- a/docs/architecture/key-files/core-persistence-request-indexes.md
+++ b/docs/architecture/key-files/core-persistence-request-indexes.md
@@ -2,7 +2,7 @@
[Subsystem index](../KEY_FILES.md) · [Core Persistence](core-persistence.md) · [continued](core-persistence-continued.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-The online-built request-table indexes (the managed sync checkpoint pair, the page purge's slug index, dream postprocessing's request-id index), the checkpoint validation they serve, the named fault seams the persistence gate and the purge tests inject at, and the stuck-recovery resolution (`gbrain repair stuck-recovery`) that uses them.
+The online-built request-table indexes (the managed sync checkpoint pair, the page purge's slug index, dream postprocessing's request-id index, the write claims' foreground-commit index), the checkpoint validation they serve, the named fault seams the persistence gate and the purge tests inject at, and the stuck-recovery resolution (`gbrain repair stuck-recovery`) that uses them.
- `src/core/persistence/checkpoint-validation.ts` — the managed sync checkpoint's incomplete-receipt validation (#5762): `INCOMPLETE_SYNC_RECEIPT_SQL` is three `LIMIT 1` probes (recovery; an open page receipt of the run; a non-committed page receipt with no committed sibling at its index) served by `persistence_requests_recovery`, `persistence_requests_sync_run_open` and `persistence_requests_sync_run_committed`, with the same result as the former single predicate. `findIncompleteSyncReceipt` returns the blocking receipt's request id and maps only this statement's 57014 statement timeout to the terminal, non-transient `checkpoint_validation_timeout`. `readRequestIndexStates` reads each index of `PERSISTENCE_REQUEST_ONLINE_INDEXES` (or the list passed) as valid, missing, INVALID or building (Postgres `pg_stat_progress_create_index`); `formatCheckpointTimeoutHint` / `checkpointTimeoutHint` is the one formatter for the three-state hint and the filled retry (`gbrain sync --source <id> --no-pull --retry-failed` plus the cursor-selecting options the checkpoint intent records in `syncOptions` and the saved processing flags), used by the sync summary and `get_write_request`. Tests: `test/persistence-sync-checkpoint-validation.test.ts`, `test/e2e/persistence-request-indexes-postgres.test.ts`.
- `src/core/persistence/fault-points.ts` — named crash seams for the persistence gate. `faultPoint(point, detail)` is a no-op unless a gate worker installed a hook with `installFaultHook`; production never does. Seams: `publication:<boundary>` (the five `PublicationHooks.boundary` names, routed through `withFaultPoints` in `publishMutation`), `effect:<kind>:mid` for every `EffectKind` in `completeEffect` (after the side effect, before its completion is recorded; `EFFECT_FAULT_POINTS` is a `Record<EffectKind, FaultPoint>`), `effect_recovery:before_clear` (`effect-recovery.ts`), `large_file:stage_streaming` (after each piece `stageStreamedContent` writes to a stage's in-progress name), `effect_recovery:before_stage` / `effect_recovery:staged` (a streamed, version 2 withdrawal mirror: once its record is durable, before its file is streamed into the publication stage, and after that step, before the recovery transaction renames the stage), `publication_recovery:before_clear` (`recoverPublication`), `sync:mid_checkpoint` (after each committed cursor step in `sync-run.ts` `saveCursor`), `sync:after_freeze` (#6712 PR 0: at the end of `freezeEntry` for an import entry, after its bytes are read and before the request is admitted; detail carries `slug` and `path`), `consumer:prepared` (between preparing a claimed request and publishing it), `stuck_recovery:*` (between the steps of `recovery-resolve.ts`) and the purge lookups `purge:*` (page-purge.ts, intent-redaction.ts, facts/purge.ts; a hook throwing a 57014 models a statement timeout). Used by `scripts/persistence/crash-robot.ts`, the #6511 tests and `test/sync-convergence.serial.test.ts`, which installs it in real `serve`/`sync` processes through the `test/fixtures/convergence-fault-kill.ts` preload (one action registry: kill, uncommitted edit, stall until released, or throw a given SQLSTATE, at the nth hit).
diff --git a/docs/guides/repair.md b/docs/guides/repair.md
index 6fc13e308b..bbb4170780 100644
--- a/docs/guides/repair.md
+++ b/docs/guides/repair.md
@@ -145,7 +145,7 @@ work behind exits 0, so scripts should also check `results[].complete`.
| --- | --- | --- | --- |
| `timeline` | `timeline_history` | Re-saves each page with its current body through a revision-bound `put_page`. The save writes each database-only timeline entry back into the page as a bullet preceded by `<!-- gbrain:materialized v1 <hash> -->`. | `kept_unrenderable_rows`: entries that would change if written as a bullet (for example an empty source). They stay in the database. |
| `visibility` | `derived_visibility` | Stamps an explicit `visibility` on extracted atoms and synthesized concepts. An atom takes its origin page's visibility; transcript atoms and atoms whose origin is gone become `private`; a concept takes the strictest visibility of its input atoms. A concept input found only through an atom's `concepts:` list counts as private. Atoms are repaired before concepts. It never loosens an explicit value: `private` stays `private`, and `world` can only become `private`. A missing value is stamped with the origin's value, which is `world` when the origin page is public. | `concepts_without_lineage`: concepts whose inputs cannot be found. They stay as they are, and remote readers already treat a missing visibility as private. `atoms_origin_gone_to_private` counts atoms made private because their origin page no longer exists. |
-| `request-indexes` | `persistence_request_indexes` | Creates a missing request-table index (the managed sync checkpoint pair, the purge slug index, the dream postprocess request-id index), or drops an INVALID one (left by an interrupted concurrent build) and rebuilds it, on a brain whose schema version is already current. Postgres builds each index `CONCURRENTLY`, one at a time, so writes continue; PGLite builds inline. It changes no user data, takes no journal admission and runs brain-wide (`--source` does not narrow it). See [managed sync request indexes](#request-indexes). | `building`: an index another session is still building; it is never dropped. |
+| `request-indexes` | `persistence_request_indexes` | Creates a missing request-table index (the managed sync checkpoint pair, the purge slug index, the dream postprocess request-id index, the write claims' foreground-commit index), or drops an INVALID one (left by an interrupted concurrent build) and rebuilds it, on a brain whose schema version is already current. Postgres builds each index `CONCURRENTLY`, one at a time, so writes continue; PGLite builds inline. It changes no user data, takes no journal admission and runs brain-wide (`--source` does not narrow it). See [managed sync request indexes](#request-indexes). | `building`: an index another session is still building; it is never dropped. |
| `connector-fences` | none (a `connector_fence_below_timeline` refusal names it) | Moves a facts or takes fence that sits below the timeline sentinel of a Google or GitHub page into the page body through a revision-bound `put_page`, so the connector's next re-render carries the fence instead of refusing. Then re-attempt the held item with `gbrain sources retry-held <source>`. | `kept_ambiguous_pages`: pages whose fences are duplicated, unbalanced or unparseable. Move the fence by hand: read the page, place the fence above the `<!-- timeline -->` line, and save it with the current `expected_revision`. |
| `take-supersession` | none | Rebuilds `takes.superseded_by` for supersession chains written before the pointer moved onto the old fence row (#5886). Each struck take is linked to the row that replaced it only from evidence: a committed `takes_supersede` receipt, a stored `superseded_by`, or a row carrying the old `superseded by #<own row>` pointer with exactly one possible predecessor. The pointer is written onto the old row (`<source>; superseded by #N`) and stale self-pointers are dropped through a revision-bound `put_page`; a page whose fence is already right but whose stored pointers differ is reprojected without a page write. See [take supersession](#take-supersession). | `ambiguous_pages` / `ambiguous_rows`: pages where a new row has two or more possible predecessors, or where evidence conflicts. The preview lists each with its candidates and the manual edit; nothing on them changes. `unparsed_pages`: pages whose takes fence does not parse cleanly. |
| `accepted-takes` | none | Explicit-only, preview-bound. Restores takes that accepting a held strike-through or kind change removed on v0.60.253.0 (`gbrain trust confirm tp<id>` ran a `takes_remove` instead of applying the change). It restores only what the records prove: the accepted proposal, the `takes_remove` receipt bound to it, the page version that receipt archived (the row the accept should have applied) and the reservation row still holding its number. `--apply --expect <hash>` republishes each page with that row through a revision-bound `put_page`. See [accepted takes](#accepted-takes). | `version_missing` (the archived version was pruned), `row_number_in_use` (the number holds a take again), `page_missing`, `fence_unreadable`. A grade or resolution the database alone held is not in any record and is not restored. |
@@ -258,7 +258,11 @@ is missing or INVALID. After the rebuild, run the retry the refusal printed
processing flags).
The same command builds `persistence_requests_slug_intent` (the page purge's
-lookup) and `persistence_requests_request_id`, which dream synthesis uses to
+lookup), `persistence_requests_foreground_committed`, which every write claim
+reads to find the newest foreground commit of a worktree (without it the
+claims after a restart can overrun their claim phase on a large request table,
+so the write queue grows and sync never commits), and
+`persistence_requests_request_id`, which dream synthesis uses to
find each child's committed page. When that lookup, or a postprocess request
read, still hits the statement timeout, the synthesis pass is deferred rather
than failed: the phase warns `postprocessing deferred`, the pages stay
diff --git a/src/commands/doctor/checks/persistence-requests.ts b/src/commands/doctor/checks/persistence-requests.ts
index 6b96d788cd..99f4635414 100644
--- a/src/commands/doctor/checks/persistence-requests.ts
+++ b/src/commands/doctor/checks/persistence-requests.ts
@@ -43,7 +43,7 @@ export async function requestIndexesCheck(engine: BrainEngine): Promise<Check> {
const deferral = postprocess.length ? ` Dream synthesis postprocessing deferred on a statement timeout ${postprocess.map(d => `${d.consecutive_passes} passes in a row on source ${d.source_id} `
+ `(${d.read ? DEFERRED_READ_LABEL[d.read] : 'a request read'}, ${d.elapsed_ms ?? '?'} ms; ${d.pending_outputs} output(s) pending)`).join('; ')}.` : '';
if (broken.length) return { name: 'persistence_request_indexes', status: 'warn', details,
- message: `${broken.map(index => `${index.name} is ${index.state === 'invalid' ? 'INVALID' : 'missing'}`).join('; ')}: request-table reads (managed sync checkpoints, page purges, `
+ message: `${broken.map(index => `${index.name} is ${index.state === 'invalid' ? 'INVALID' : 'missing'}`).join('; ')}: request-table reads (managed sync checkpoints, write claims, page purges, `
+ `dream synthesis postprocessing) can time out on a large request table. Rebuild on the brain host: ${REQUEST_INDEXES_REPAIR_COMMAND}${deferral}` };
if (building.length) return { name: 'persistence_request_indexes', status: 'warn', details,
message: `Still building: ${building.map(index => `${index.name} (${index.progress?.phase ?? 'building'}${index.progress?.blocks_total
diff --git a/src/core/persistence/schema.ts b/src/core/persistence/schema.ts
index d47780a834..60a6c693aa 100644
--- a/src/core/persistence/schema.ts
+++ b/src/core/persistence/schema.ts
@@ -35,8 +35,19 @@ export const PERSISTENCE_REQUEST_SLUG_INTENT_INDEX = { name: 'persistence_reques
export const PERSISTENCE_REQUEST_ID_INDEX_SQL = `CREATE INDEX IF NOT EXISTS persistence_requests_request_id
ON persistence_requests(source_id,request_id)`;
export const PERSISTENCE_REQUEST_ID_INDEX = { name: 'persistence_requests_request_id', table: 'persistence_requests', sql: PERSISTENCE_REQUEST_ID_INDEX_SQL } as const;
+/**
+ * #7649: the write claims' foreground-commit probe (journal.ts `FOREGROUND_WAITING_ON`: a committed non-sync request on the
+ * worktree after the `syncSeen` bound). After a restart the bound is 0, and with no index on (worktree, sequence) for committed
+ * rows the probe walked the sequence key (or the heap) through every receipt, sync receipts included, on each claim. The
+ * predicate is the probe's `state='committed' AND NOT SYNC_KIND(c)` written the same way, so the planner can use it. Postgres
+ * builds it CONCURRENTLY in the persistence_foreground_committed_index migration (`transaction: false`); PGLite inline.
+ */
+export const PERSISTENCE_FOREGROUND_COMMITTED_INDEX_SQL = `CREATE INDEX IF NOT EXISTS persistence_requests_foreground_committed
+ ON persistence_requests(worktree_id,sequence) WHERE state='committed' AND NOT COALESCE(intent->>'kind','') LIKE 'managed_sync_%'`;
+export const PERSISTENCE_FOREGROUND_COMMITTED_INDEX = { name: 'persistence_requests_foreground_committed', table: 'persistence_requests', sql: PERSISTENCE_FOREGROUND_COMMITTED_INDEX_SQL } as const;
/** Every request-table index built online (a migration, then `gbrain repair request-indexes` and doctor `persistence_request_indexes`). */
-export const PERSISTENCE_REQUEST_ONLINE_INDEXES = [...PERSISTENCE_SYNC_RUN_INDEXES, PERSISTENCE_REQUEST_SLUG_INTENT_INDEX, PERSISTENCE_REQUEST_ID_INDEX] as const;
+export const PERSISTENCE_REQUEST_ONLINE_INDEXES = [...PERSISTENCE_SYNC_RUN_INDEXES, PERSISTENCE_REQUEST_SLUG_INTENT_INDEX, PERSISTENCE_REQUEST_ID_INDEX,
+ PERSISTENCE_FOREGROUND_COMMITTED_INDEX] as const;
/**
* #6317: the movement watermark (the newest committed receipt of a worktree)
* that `writer movement`, `data_moving` and doctor `managed_sync_not_moving`
@@ -107,7 +118,7 @@ export const PERSISTENCE_CONSUMERS_TABLE_SQL = `CREATE TABLE IF NOT EXISTS persi
/** Indexes the Postgres blob omits because a migration builds them CONCURRENTLY. */
export const POSTGRES_CONCURRENT_PERSISTENCE_INDEXES: ReadonlySet<string> = new Set([
PERSISTENCE_DATABASE_PENDING_INDEX_SQL, PERSISTENCE_SYNC_RUN_OPEN_INDEX_SQL, PERSISTENCE_SYNC_RUN_COMMITTED_INDEX_SQL, PERSISTENCE_SYNC_WATERMARK_INDEX_SQL, PERSISTENCE_COMPACTABLE_INDEX_SQL,
- PERSISTENCE_REQUEST_SLUG_INTENT_INDEX_SQL, PERSISTENCE_REQUEST_ID_INDEX_SQL,
+ PERSISTENCE_REQUEST_SLUG_INTENT_INDEX_SQL, PERSISTENCE_REQUEST_ID_INDEX_SQL, PERSISTENCE_FOREGROUND_COMMITTED_INDEX_SQL,
]);
/** Durable infrastructure: never reconstruct or discard these rows during page reindexing. */
export const PERSISTENCE_SCHEMA_STATEMENTS = [
@@ -203,6 +214,7 @@ export const PERSISTENCE_SCHEMA_STATEMENTS = [
PERSISTENCE_COMPACTABLE_INDEX_SQL,
PERSISTENCE_REQUEST_SLUG_INTENT_INDEX_SQL,
PERSISTENCE_REQUEST_ID_INDEX_SQL,
+ PERSISTENCE_FOREGROUND_COMMITTED_INDEX_SQL,
`CREATE INDEX IF NOT EXISTS persistence_requests_principal ON persistence_requests(principal_kind,principal_id,sequence DESC)`,
`CREATE TABLE IF NOT EXISTS persistence_effects (
id bigserial PRIMARY KEY,
diff --git a/src/core/pglite-schema.generated.ts b/src/core/pglite-schema.generated.ts
index ba667bf4dd..15ae0cd7d3 100644
--- a/src/core/pglite-schema.generated.ts
+++ b/src/core/pglite-schema.generated.ts
@@ -1664,6 +1664,8 @@ CREATE INDEX IF NOT EXISTS persistence_requests_slug_intent
ON persistence_requests(source_id,slug,sequence) WHERE intent IS NOT NULL AND NOT compacted;
CREATE INDEX IF NOT EXISTS persistence_requests_request_id
ON persistence_requests(source_id,request_id);
+CREATE INDEX IF NOT EXISTS persistence_requests_foreground_committed
+ ON persistence_requests(worktree_id,sequence) WHERE state='committed' AND NOT COALESCE(intent->>'kind','') LIKE 'managed_sync_%';
CREATE INDEX IF NOT EXISTS persistence_requests_principal ON persistence_requests(principal_kind,principal_id,sequence DESC);
CREATE TABLE IF NOT EXISTS persistence_effects (
id bigserial PRIMARY KEY,
diff --git a/src/core/repair/request-indexes.ts b/src/core/repair/request-indexes.ts
index 4cd9750a30..1093283763 100644
--- a/src/core/repair/request-indexes.ts
+++ b/src/core/repair/request-indexes.ts
@@ -1,9 +1,9 @@
/**
- * `gbrain repair request-indexes` (#5762, #6511): create a missing online-built
- * request index (the managed sync checkpoint pair, the purge slug index), or
- * drop an INVALID one (an interrupted concurrent build) and rebuild it, on a
- * brain whose schema version is already current, where the v179 and slug-index
- * migrations never run again. Postgres builds CONCURRENTLY, one index at
+ * `gbrain repair request-indexes` (#5762, #6511, #7649): create a missing online-built
+ * request index (the managed sync checkpoint pair, the purge slug index, the
+ * request-id index, the write claims' foreground-commit index), or drop an
+ * INVALID one (an interrupted concurrent build) and rebuild it, on a brain
+ * whose schema version is already current, where their migrations never run again. Postgres builds CONCURRENTLY, one index at
* a time; PGLite builds inline. An index another session is still building is
* reported, never dropped. Brain-wide; no journal admission and no user data
* changes, so `--all` and doctor remediation include it.
diff --git a/src/core/schema-migrations/latest.generated.ts b/src/core/schema-migrations/latest.generated.ts
index 4f1b5b8a21..4059749ed5 100644
--- a/src/core/schema-migrations/latest.generated.ts
+++ b/src/core/schema-migrations/latest.generated.ts
@@ -4,4 +4,4 @@
// The highest schema migration version (migrate.ts LATEST_VERSION), importable without
// loading the migrations themselves.
-export const LATEST_SCHEMA_VERSION = 275;
+export const LATEST_SCHEMA_VERSION = 276;
diff --git a/src/core/schema-migrations/registry.generated.ts b/src/core/schema-migrations/registry.generated.ts
index fa2fc51126..f09d46ac19 100644
--- a/src/core/schema-migrations/registry.generated.ts
+++ b/src/core/schema-migrations/registry.generated.ts
@@ -276,6 +276,7 @@ import { v272 } from './v272-git-canonical-declared.ts';
import { v273 } from './v273-segment-derivations.ts';
import { v274 } from './v274-pages-withheld-source-path.ts';
import { v275 } from './v275-rls-backfill-unguarded-tables.ts';
+import { v276 } from './v276-persistence-foreground-committed-index.ts';
// Array order is historical (see HISTORICAL_ARRAY_ORDER in the generator); the
// runner sorts by version before applying.
@@ -550,4 +551,5 @@ export const MIGRATIONS: Migration[] = [
v273,
v274,
v275,
+ v276,
];
diff --git a/src/core/schema-migrations/v276-persistence-foreground-committed-index.ts b/src/core/schema-migrations/v276-persistence-foreground-committed-index.ts
new file mode 100644
index 0000000000..7387489204
--- /dev/null
+++ b/src/core/schema-migrations/v276-persistence-foreground-committed-index.ts
@@ -0,0 +1,25 @@
+import type { Migration } from './types.ts';
+import { PERSISTENCE_FOREGROUND_COMMITTED_INDEX } from '../persistence/schema.ts';
+import { buildIndexOnline, migrationNotice } from './helpers.ts';
+
+// Applied by src/core/migrate.ts on the next initSchema(); see docs/ENGINES.md
+// ("Canonical schema sources") for when schema.sql or a TS fragment also changes.
+//
+// #7649: after a restart every write claim's foreground-commit probe (journal.ts
+// FOREGROUND_WAITING_ON) starts at sequence 0, and no index served committed
+// rows by (worktree, sequence), so on a large request table each claim walked
+// every receipt and overran the claim phase. Postgres builds the partial index
+// CONCURRENTLY outside a transaction, after dropping an INVALID leftover of an
+// interrupted build (the blob omits it); PGLite builds it inline. `gbrain repair
+// request-indexes` runs the same helper on a brain whose schema is already
+// current. DDL only: no column, no backfill.
+export const v276: Migration = {
+ version: 276,
+ name: 'persistence_foreground_committed_index',
+ idempotent: true,
+ transaction: false,
+ sql: '',
+ handler: async engine => {
+ await buildIndexOnline(engine, 276, PERSISTENCE_FOREGROUND_COMMITTED_INDEX, { notice: migrationNotice });
+ },
+};
diff --git a/test/e2e/persistence-claim-committed-probe-7649-postgres.test.ts b/test/e2e/persistence-claim-committed-probe-7649-postgres.test.ts
new file mode 100644
index 0000000000..b7e603d51d
--- /dev/null
+++ b/test/e2e/persistence-claim-committed-probe-7649-postgres.test.ts
@@ -0,0 +1,184 @@
+/**
+ * #7649 on Postgres: after a restart a resident consumer claims with an empty `syncSeen`, so the foreground-commit probe
+ * of both write claims (journal.ts `FOREGROUND_WAITING_ON`: a committed non-sync request on the worktree after the bound)
+ * starts at sequence 0. With no index on committed rows by (worktree, sequence) the planner walked the sequence key or the
+ * heap through every receipt, sync receipts included, on every claim; on a large request table each claim overran the 5 s
+ * claim phase, so the write queue grew and sync never committed.
+ *
+ * The plans here are of the statements `claimNextWrite` and `claimNextLaneHead` send, with their own parameters: the
+ * claim's transaction is intercepted at the statement, which is EXPLAINed instead of run. The journal holds 160k receipts,
+ * nearly all sync imports, with the foreground commits at its end (a big import, then ordinary writes), and a queued sync
+ * row ahead of a queued foreground write on one worktree, so the claim evaluates the probe. The migration that builds the
+ * index is found by its name: the number is provisional until the merge slot.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import postgres from '#postgres';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../../src/core/engine.ts';
+import type { PostgresEngine } from '../../src/core/postgres-engine.ts';
+import { claimNextLaneHead, claimNextWrite } from '../../src/core/persistence/journal.ts';
+import { readRequestIndexStates } from '../../src/core/persistence/checkpoint-validation.ts';
+import { PERSISTENCE_FOREGROUND_COMMITTED_INDEX } from '../../src/core/persistence/schema.ts';
+import { requestIndexesCheck } from '../../src/commands/doctor/checks/persistence-requests.ts';
+import { MIGRATIONS } from '../../src/core/schema-migrations/registry.generated.ts';
+import { requirePostgresTestDatabase } from '../helpers/test-backends.ts';
+import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';
+
+const INDEX = PERSISTENCE_FOREGROUND_COMMITTED_INDEX;
+const migration = MIGRATIONS.find(m => m.name === 'persistence_foreground_committed_index');
+const HOST = randomUUID(), WA = randomUUID(), WB = randomUUID();
+const RECEIPTS = 160_000;
+const RUN = 'run-7649';
+
+type PlanNode = Record<string, unknown> & { Plans?: PlanNode[] };
+const nodes = (plan: PlanNode): PlanNode[] => [plan, ...(plan.Plans ?? []).flatMap(nodes)];
+
+let engine: PostgresEngine;
+let databaseUrl: string;
+let close: () => Promise<void>;
+
+beforeAll(async () => {
+ const pg = await isolatedPersistencePostgres(requirePostgresTestDatabase());
+ engine = pg.engine; databaseUrl = pg.databaseUrl; close = pg.close;
+ await engine.executeRaw(`INSERT INTO persistence_worktrees(id,owner_host_id,state) VALUES ($1::uuid,$3::uuid,'active'),($2::uuid,$3::uuid,'active')`, [WA, WB, HOST]);
+ await engine.executeRaw(`INSERT INTO persistence_requests(principal_kind,principal_id,request_id,operation,source_id,source_incarnation,
+ worktree_id,slug,digest,intent,authority,intent_bytes,terminal_reservation,state,completed_at)
+ SELECT 'local_cli','seed',gen_random_uuid(),'put_page','default',(SELECT incarnation FROM sources WHERE id='default'),
+ CASE WHEN g%2=0 THEN $1::uuid ELSE $2::uuid END,'seed-'||g,'d',
+ CASE WHEN g%20=0 AND g>$3::int*0.98 THEN jsonb_build_object('kind','put_page','slug','seed-'||g)
+ ELSE jsonb_build_object('kind','managed_sync_import','runId',md5((g/1000)::text),'index',g%1000,'slug','seed-'||g) END,
+ '{}'::jsonb,1,64,'committed',now()
+ FROM generate_series(1,$3::int) g`, [WA, WB, RECEIPTS]);
+ for (const [slug, intent] of [['queued-sync', { kind: 'managed_sync_import', runId: 'live', index: 0, slug: 'queued-sync', lane: RUN }],
+ ['queued-fg', { kind: 'put_page', slug: 'queued-fg' }]] as const) {
+ await engine.executeRaw(`INSERT INTO persistence_requests(principal_kind,principal_id,request_id,operation,source_id,source_incarnation,
+ worktree_id,slug,digest,intent,authority,intent_bytes,terminal_reservation,state)
+ VALUES ('local_cli','seed',gen_random_uuid(),'put_page','default',(SELECT incarnation FROM sources WHERE id='default'),$1::uuid,$2,'d',$3::text::jsonb,'{}'::jsonb,1,64,'queued')`,
+ [WA, slug, JSON.stringify(intent)]);
+ }
+ await engine.executeRaw(`UPDATE persistence_requests SET intent=intent||jsonb_build_object('group',request_id::text) WHERE slug='queued-sync'`);
+ await engine.executeRaw('ANALYZE persistence_requests');
+}, 180_000);
+afterAll(async () => { await close?.(); }, 60_000);
+
+/** EXPLAIN (ANALYZE) of the claim statement each claim sends, in its own transaction with its own parameters; the claim then finds nothing. */
+async function claimPlans(): Promise<Record<'write' | 'laneHead', PlanNode>> {
+ const plans: Partial<Record<'write' | 'laneHead', PlanNode>> = {};
+ const original = engine.transactionDirect;
+ engine.transactionDirect = ((fn: (tx: BrainEngine) => Promise<unknown>, opts?: unknown) => original.call(engine, async (tx: BrainEngine) => {
+ const raw = tx.executeRaw.bind(tx);
+ tx.executeRaw = (async (sql: string, params?: unknown[], options?: unknown) => {
+ const label = sql.startsWith('WITH waiting AS MATERIALIZED') ? 'write'
+ : /^SELECT r\.\* FROM persistence_requests r\s+JOIN persistence_worktrees/.test(sql) ? 'laneHead' : null;
+ if (!label) return raw(sql, params as never, options as never);
+ const [row] = await raw<{ 'QUERY PLAN': Array<{ Plan: PlanNode }> }>(`EXPLAIN (ANALYZE, FORMAT JSON) ${sql}`, params as never, options as never);
+ plans[label] = row!['QUERY PLAN'][0]!.Plan;
+ return [];
+ }) as BrainEngine['executeRaw'];
+ return fn(tx);
+ }, opts as never)) as PostgresEngine['transactionDirect'];
+ try {
+ expect(await claimNextWrite(engine, HOST, 30_000, [], [], [], {})).toBeNull();
+ expect(await claimNextLaneHead(engine, HOST, WA, RUN, 30_000, null)).toBeNull();
+ } finally {
+ engine.transactionDirect = original;
+ }
+ expect(Object.keys(plans).sort()).toEqual(['laneHead', 'write']);
+ return plans as Record<'write' | 'laneHead', PlanNode>;
+}
+
+/** The committed probe's scans (alias `c` in FOREGROUND_WAITING_ON): node type, indexes read and rows read then discarded. */
+const committedProbe = (plan: PlanNode) => nodes(plan)
+ .filter(node => node.Alias === 'c' && node['Relation Name'] === 'persistence_requests')
+ .map(node => ({ type: node['Node Type'], indexes: [...new Set(nodes(node).map(n => n['Index Name']).filter(Boolean))],
+ removed: Number(node['Rows Removed by Filter'] ?? 0) }));
+
+describe('#7649 the cold claim committed probe', () => {
+ test('with an empty syncSeen both claims read the committed probe through the new index, and nothing is filtered out', async () => {
+ const plans = await claimPlans();
+ for (const plan of [plans.write, plans.laneHead]) {
+ const probe = committedProbe(plan);
+ expect(probe.length).toBeGreaterThan(0);
+ for (const scan of probe) {
+ expect(scan.type).not.toBe('Seq Scan');
+ expect(scan.indexes).toEqual([INDEX.name]);
+ expect(scan.removed).toBe(0);
+ }
+ }
+ }, 120_000);
+
+ test('without the index the probe walks the sync receipts; the migration builds it back and the probe reads it again', async () => {
+ await engine.executeRaw(`DROP INDEX ${INDEX.name}`);
+ const cold = await claimPlans();
+ for (const plan of [cold.write, cold.laneHead]) {
+ const probe = committedProbe(plan);
+ expect(probe.length).toBeGreaterThan(0);
+ expect(probe.flatMap(scan => scan.indexes)).not.toContain(INDEX.name);
+ expect(Math.max(...probe.map(scan => scan.removed))).toBeGreaterThan(RECEIPTS / 2);
+ }
+ await migration!.handler!(engine);
+ expect((await readRequestIndexStates(engine, [INDEX]))[0]!.state).toBe('valid');
+ const warm = await claimPlans();
+ for (const plan of [warm.write, warm.laneHead]) expect(committedProbe(plan).flatMap(scan => scan.indexes)).toEqual([INDEX.name]);
+ }, 180_000);
+
+ test('the real claim still claims the queued foreground write ahead of the sync row it passes', async () => {
+ const row = await claimNextWrite(engine, HOST, 30_000, [], [], [], {});
+ expect(row?.slug).toBe('queued-fg');
+ expect(row?.passed_sync).toBe(true);
+ }, 60_000);
+});
+
+describe('#7649 an interrupted CONCURRENTLY build', () => {
+ test('the migration runs outside a transaction and is idempotent', () => {
+ expect(MIGRATIONS.filter(m => m.name === 'persistence_foreground_committed_index')).toHaveLength(1);
+ expect(migration).toMatchObject({ idempotent: true, transaction: false, sql: '' });
+ });
+
+ test('a build cancelled while it waits for a writer leaves an INVALID index, which the migration drops and rebuilds CONCURRENTLY', async () => {
+ await engine.executeRaw(`DROP INDEX ${INDEX.name}`);
+ const writer = postgres(databaseUrl, { max: 1, prepare: false });
+ const builder = postgres(databaseUrl, { max: 1, prepare: false });
+ try {
+ let release!: () => void;
+ const released = new Promise<void>(resolve => { release = resolve; });
+ let locked!: () => void;
+ const holding = new Promise<void>(resolve => { locked = resolve; });
+ const writerTx = writer.begin(async sql => {
+ await sql.unsafe('LOCK TABLE persistence_requests IN ROW EXCLUSIVE MODE');
+ locked();
+ await released;
+ });
+ await holding;
+ await builder.unsafe(`SET statement_timeout = '1s'`);
+ // CREATE INDEX CONCURRENTLY commits the catalog entry, then waits for the writer: the timeout leaves it INVALID.
+ await expect(builder.unsafe(INDEX.sql.replace('CREATE INDEX', 'CREATE INDEX CONCURRENTLY'))).rejects.toThrow(/statement timeout/);
+ release();
+ await writerTx;
+ } finally {
+ await writer.end();
+ await builder.end();
+ }
+ expect((await readRequestIndexStates(engine, [INDEX]))[0]!.state).toBe('invalid');
+ expect((await requestIndexesCheck(engine)).message).toContain(`${INDEX.name} is INVALID`);
+ const [before] = await engine.executeRaw<{ oid: string }>('SELECT to_regclass($1)::oid::text AS oid', [INDEX.name]);
+
+ const statements: string[] = [];
+ const reserve = engine.withReservedConnection.bind(engine);
+ engine.withReservedConnection = (async (fn: Parameters<PostgresEngine['withReservedConnection']>[0]) => reserve(conn => fn({
+ executeRaw: async (sql: string, params?: unknown[]) => { statements.push(sql); return conn.executeRaw(sql, params); },
+ }))) as PostgresEngine['withReservedConnection'];
+ try {
+ await migration!.handler!(engine);
+ } finally { engine.withReservedConnection = reserve; }
+ expect(statements.filter(sql => sql.startsWith('CREATE INDEX CONCURRENTLY IF NOT EXISTS')).map(sql => sql.split(/\s+/)[6])).toEqual([INDEX.name]);
+ expect((await readRequestIndexStates(engine, [INDEX]))[0]!.state).toBe('valid');
+ const [after] = await engine.executeRaw<{ oid: string }>('SELECT to_regclass($1)::oid::text AS oid', [INDEX.name]);
+ expect(after!.oid).not.toBe(before!.oid);
+ expect(await requestIndexesCheck(engine)).toMatchObject({ status: 'ok' });
+
+ await migration!.handler!(engine);
+ const [again] = await engine.executeRaw<{ oid: string }>('SELECT to_regclass($1)::oid::text AS oid', [INDEX.name]);
+ expect(again!.oid).toBe(after!.oid);
+ }, 120_000);
+});
diff --git a/test/e2e/persistence-request-indexes-postgres.test.ts b/test/e2e/persistence-request-indexes-postgres.test.ts
index a8e4e6b830..0a62d8c5ac 100644
--- a/test/e2e/persistence-request-indexes-postgres.test.ts
+++ b/test/e2e/persistence-request-indexes-postgres.test.ts
@@ -107,23 +107,23 @@ test('#5762 the Postgres blob never builds the indexes; v179 builds them concurr
try { await v179.handler!(engine); } finally { engine.withReservedConnection = reserve; }
expect(statements.filter(sql => sql.startsWith('CREATE INDEX CONCURRENTLY IF NOT EXISTS')).map(sql => sql.split(/\s+/)[6])).toEqual(NAMES);
expect(statements.filter(sql => /^(SET|RESET)\b/i.test(sql))).toEqual([]);
- expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid']);
+ expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
}), 120_000);
test('#5762 gbrain repair request-indexes rebuilds a dropped and an INVALID index on a current schema; a second run is a no-op', async () => withEnv(env, async () => {
await engine.executeRaw('DROP INDEX persistence_requests_sync_run_open');
await engine.executeRaw("UPDATE pg_index SET indisvalid=false WHERE indexrelid='persistence_requests_sync_run_committed'::regclass");
- expect(await readRequestIndexStates(engine)).toEqual([{ name: NAMES[0], state: 'missing' }, { name: NAMES[1], state: 'invalid' }, { name: ALL_NAMES[2], state: 'valid' }, { name: ALL_NAMES[3], state: 'valid' }]);
+ expect(await readRequestIndexStates(engine)).toEqual([{ name: NAMES[0], state: 'missing' }, { name: NAMES[1], state: 'invalid' }, { name: ALL_NAMES[2], state: 'valid' }, { name: ALL_NAMES[3], state: 'valid' }, { name: ALL_NAMES[4], state: 'valid' }]);
const doctor = await requestIndexesCheck(engine);
expect(doctor).toMatchObject({ name: 'persistence_request_indexes', status: 'warn', details: { count: 2, command: 'gbrain repair request-indexes --apply' } });
expect(doctor.message).toContain('persistence_requests_sync_run_committed is INVALID');
const scope = await resolveRepairScope(engine);
const preview = await runRepair(ctx(), requestIndexesRepair, scope, { apply: false });
expect(preview).toMatchObject({ affected: 2, cost: { lifetime_ids: 0 }, apply_command: 'gbrain repair request-indexes --apply' });
- expect(await states()).toEqual(['missing', 'invalid', 'valid', 'valid']);
+ expect(await states()).toEqual(['missing', 'invalid', 'valid', 'valid', 'valid']);
const applied = await runRepair(ctx(), requestIndexesRepair, scope, { apply: true });
expect(applied).toMatchObject({ applied: 2, complete: true });
- expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid']);
+ expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
expect((await requestIndexesCheck(engine)).status).toBe('ok');
expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: true })).toMatchObject({ affected: 0, applied: 0, complete: true });
}), 120_000);
@@ -142,7 +142,7 @@ test('#5762 a build that leaves the index INVALID (a concurrent caller failed fi
try { await expect(buildIndexOnline(engine, 0, index)).rejects.toThrow(`Index ${index.name} is not valid after its concurrent build`); }
finally { engine.withReservedConnection = reserve; }
expect(await buildIndexOnline(engine, 0, index)).toBe('rebuilt');
- expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid']);
+ expect(await states()).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
}), 60_000);
test('#5762 recovery journey: a checkpoint timeout with a dropped index is fixed by the two printed commands', async () => withEnv(env, async () => {
diff --git a/test/fix-wave-4-integration.test.ts b/test/fix-wave-4-integration.test.ts
index 878ff4bf15..4117c5a5ca 100644
--- a/test/fix-wave-4-integration.test.ts
+++ b/test/fix-wave-4-integration.test.ts
@@ -77,8 +77,8 @@ test('X1: an upgrade from v178 builds the request indexes; managed working-tree
await engine.executeRaw('DROP INDEX IF EXISTS persistence_requests_sync_run_committed');
await engine.setConfig('version', '178');
expect(await runMigrations(engine)).toEqual({ applied: LATEST_VERSION - 178, current: LATEST_VERSION });
- // #6511/#6757: the online-built set is the checkpoint pair, the purge slug index and the request-id index.
- expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid']);
+ // #6511/#6757/#7649: the online-built set is the checkpoint pair, the purge slug index, the request-id index and the claims' foreground-commit index.
+ expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
expect((await requestIndexesCheck(engine)).status).toBe('ok');
const id = `x1-${randomUUID().slice(0, 8)}`, root = join(home, id);
@@ -230,7 +230,7 @@ test('X11: the remediation run reaches every new repair kind, runs the free ones
const byKind = Object.fromEntries(run.repairs.map((r: { kind: string }) => [r.kind, r]));
expect(byKind['request-indexes']).toMatchObject({ status: 'completed' });
expect(byKind['orphan-bindings']).toMatchObject({ status: 'completed' });
- expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid']);
+ expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
expect(await engine.executeRaw("SELECT 1 FROM persistence_source_bindings WHERE source_id='x11-orphan'")).toEqual([]);
const replanned = (await planRepairSteps(engine, { noEmbed: true })).map(step => step.kind);
expect(replanned).not.toContain('request-indexes');
diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-default.json b/test/fixtures/goldens/catalog/pglite-engine-init-default.json
index bdcf3843c3..217c5e7be0 100644
--- a/test/fixtures/goldens/catalog/pglite-engine-init-default.json
+++ b/test/fixtures/goldens/catalog/pglite-engine-init-default.json
@@ -1425,6 +1425,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json b/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json
index 9a6a8ada26..e0f02ad7ce 100644
--- a/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json
+++ b/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json
@@ -1425,6 +1425,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json b/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json
index 5b10e6aafd..eb319773c4 100644
--- a/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json
+++ b/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json
@@ -1421,6 +1421,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/pglite-schema-blob-default.json b/test/fixtures/goldens/catalog/pglite-schema-blob-default.json
index efd7faefbb..7e4abc0c3f 100644
--- a/test/fixtures/goldens/catalog/pglite-schema-blob-default.json
+++ b/test/fixtures/goldens/catalog/pglite-schema-blob-default.json
@@ -1064,6 +1064,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/pglite-schema-blob-fts-portuguese.json b/test/fixtures/goldens/catalog/pglite-schema-blob-fts-portuguese.json
index d8bb6ddfc8..8cbd4d316f 100644
--- a/test/fixtures/goldens/catalog/pglite-schema-blob-fts-portuguese.json
+++ b/test/fixtures/goldens/catalog/pglite-schema-blob-fts-portuguese.json
@@ -1064,6 +1064,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/pglite-schema-blob-high-dims.json b/test/fixtures/goldens/catalog/pglite-schema-blob-high-dims.json
index 9556d6703c..a38ceaa10f 100644
--- a/test/fixtures/goldens/catalog/pglite-schema-blob-high-dims.json
+++ b/test/fixtures/goldens/catalog/pglite-schema-blob-high-dims.json
@@ -1063,6 +1063,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-default.json b/test/fixtures/goldens/catalog/postgres-engine-init-default.json
index 4237d592cb..d19b17f792 100644
--- a/test/fixtures/goldens/catalog/postgres-engine-init-default.json
+++ b/test/fixtures/goldens/catalog/postgres-engine-init-default.json
@@ -1429,6 +1429,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json b/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json
index 51e563462c..b8d3e9db34 100644
--- a/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json
+++ b/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json
@@ -1429,6 +1429,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json b/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json
index 143e2def72..a0097c3a3d 100644
--- a/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json
+++ b/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json
@@ -1425,6 +1425,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/fixtures/goldens/doctor/hot-checks-postgres-json.json b/test/fixtures/goldens/doctor/hot-checks-postgres-json.json
index 2e94746121..c206a3da22 100644
--- a/test/fixtures/goldens/doctor/hot-checks-postgres-json.json
+++ b/test/fixtures/goldens/doctor/hot-checks-postgres-json.json
@@ -1475,6 +1475,10 @@
{
"name": "persistence_requests_request_id",
"state": "valid"
+ },
+ {
+ "name": "persistence_requests_foreground_committed",
+ "state": "valid"
}
],
"repair": "request-indexes"
@@ -6211,6 +6215,10 @@
{
"name": "persistence_requests_request_id",
"state": "valid"
+ },
+ {
+ "name": "persistence_requests_foreground_committed",
+ "state": "valid"
}
],
"repair": "request-indexes"
diff --git a/test/fixtures/goldens/doctor/json-pglite-degraded.json b/test/fixtures/goldens/doctor/json-pglite-degraded.json
index e0d52042c0..6019b17b99 100644
--- a/test/fixtures/goldens/doctor/json-pglite-degraded.json
+++ b/test/fixtures/goldens/doctor/json-pglite-degraded.json
@@ -1263,6 +1263,10 @@
{
"name": "persistence_requests_request_id",
"state": "valid"
+ },
+ {
+ "name": "persistence_requests_foreground_committed",
+ "state": "valid"
}
],
"repair": "request-indexes"
diff --git a/test/fixtures/goldens/doctor/json-pglite-fresh.json b/test/fixtures/goldens/doctor/json-pglite-fresh.json
index 77340d704a..c8bc4dcac6 100644
--- a/test/fixtures/goldens/doctor/json-pglite-fresh.json
+++ b/test/fixtures/goldens/doctor/json-pglite-fresh.json
@@ -1319,6 +1319,10 @@
{
"name": "persistence_requests_request_id",
"state": "valid"
+ },
+ {
+ "name": "persistence_requests_foreground_committed",
+ "state": "valid"
}
],
"repair": "request-indexes"
diff --git a/test/fixtures/goldens/doctor/json-postgres-fresh.json b/test/fixtures/goldens/doctor/json-postgres-fresh.json
index feaf1fa688..aabd58754f 100644
--- a/test/fixtures/goldens/doctor/json-postgres-fresh.json
+++ b/test/fixtures/goldens/doctor/json-postgres-fresh.json
@@ -1311,6 +1311,10 @@
{
"name": "persistence_requests_request_id",
"state": "valid"
+ },
+ {
+ "name": "persistence_requests_foreground_committed",
+ "state": "valid"
}
],
"repair": "request-indexes"
diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.json
index 64bda389d1..87a755dc51 100644
--- a/test/fixtures/goldens/migrations/records.json
+++ b/test/fixtures/goldens/migrations/records.json
@@ -1,6 +1,6 @@
{
"golden": {
- "count": 270,
+ "count": 271,
"gaps": [
17,
18,
@@ -38,6 +38,7 @@
"PERSISTENCE_COMPACTABLE_INDEX": "25ee657878efb41ba9ea3a7624464ddcf66407d861659456c230366e57f230a0",
"PERSISTENCE_DATABASE_PENDING_INDEX_SQL": "6e0f618be33d037454933ea4dea11438413d45aef35d3bd4f33e89624d6e199e",
"PERSISTENCE_EFFECT_PARKED_INDEX_SQL": "0107863883415378d7c0dce7a0cbd5311c2c5a59fc0f81917cc8b767fd235cf2",
+ "PERSISTENCE_FOREGROUND_COMMITTED_INDEX": "cb6bb9f8e9bcfd2524a04d05e5da865e92c69442bbcd438d8191d311f76d0663",
"PERSISTENCE_REQUEST_ID_INDEX": "2c5325e86cf395f617d93a39a45593931639cd8b14fd513b6d9e7d125f16c78f",
"PERSISTENCE_REQUEST_SLUG_INTENT_INDEX": "5f2e4a3dde452e610fe7eaae949642c00d7c61a363b1cd344d7f0917db233aa5",
"PERSISTENCE_SYNC_RUN_INDEXES": "e1b12613c5fd2facb850e6857b0bddcf49b2478ec258fa226fe15e71e55df9c1",
@@ -87,7 +88,7 @@
"verifyProjectionStatistics": "0d231b04ecfbc5a4109d6fca9567bf4e8c9e9db6c3fbbb35212e026579ed6f67",
"withSchemaLockTimeout": "95605bd3004955e11e936c7a4aa6889319b401098abf9db18ef3e48da2c3fc59"
},
- "latest": 275,
+ "latest": 276,
"records": [
{
"handler": {
@@ -2281,8 +2282,8 @@
"idempotent": true,
"name": "durable_concurrent_persistence",
"sql": {
- "length": 13332,
- "sha256": "e0e36fbcff5d72d7cffb6c290f5fce2fdc7b2f014c4bb20887240821f0408d29"
+ "length": 13533,
+ "sha256": "44196cf3f167069f378f222b703225d8f5348a4c72b7e72712f7963cbe259a5b"
},
"sqlFor": "absent",
"transaction": "absent",
@@ -4445,6 +4446,35 @@
"transaction": false,
"verify": "absent",
"version": 275
+ },
+ {
+ "handler": {
+ "form": "arrow",
+ "helpers": [
+ "PERSISTENCE_FOREGROUND_COMMITTED_INDEX",
+ "PLANNER_REPAIR_TABLES",
+ "analyzeAfterIndexBuild",
+ "analyzeDeferred",
+ "analyzeYieldingToVacuum",
+ "buildIndexOnline",
+ "dropInvalidConcurrentIndex",
+ "lockTableForAnalyze",
+ "migrationNotice",
+ "quietMigrationNotices",
+ "tableRows"
+ ],
+ "sha256": "6216cb939d9e686a3386b406051d172d78e86f55edb629f7672b9602926d0484"
+ },
+ "idempotent": true,
+ "name": "persistence_foreground_committed_index",
+ "sql": {
+ "length": 0,
+ "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+ },
+ "sqlFor": "absent",
+ "transaction": false,
+ "verify": "absent",
+ "version": 276
}
],
"versions": [
@@ -4717,7 +4747,8 @@
272,
273,
274,
- 275
+ 275,
+ 276
]
},
"normalizer": "migrations-record-v1"
diff --git a/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json b/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json
index 8925e195fb..e5d2841aa9 100644
--- a/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json
+++ b/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json
@@ -1425,6 +1425,7 @@
"persistence_local_writers_pkey": "CREATE UNIQUE INDEX persistence_local_writers_pkey ON public.persistence_local_writers USING btree (id)",
"persistence_requests_compactable": "CREATE INDEX persistence_requests_compactable ON public.persistence_requests USING btree (completed_at) WHERE ((recovery IS NULL) AND (NOT compacted) AND (state = ANY (ARRAY['committed'::text, 'conflict'::text, 'failed'::text, 'cancelled'::text])))",
"persistence_requests_database_pending": "CREATE INDEX persistence_requests_database_pending ON public.persistence_requests USING btree (source_incarnation, sequence) WHERE ((worktree_id IS NULL) AND (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text])))",
+ "persistence_requests_foreground_committed": "CREATE INDEX persistence_requests_foreground_committed ON public.persistence_requests USING btree (worktree_id, sequence) WHERE ((state = 'committed'::text) AND (NOT (COALESCE((intent ->> 'kind'::text), ''::text) ~~ 'managed_sync_%'::text)))",
"persistence_requests_pending": "CREATE INDEX persistence_requests_pending ON public.persistence_requests USING btree (worktree_id, sequence) WHERE (state = ANY (ARRAY['queued'::text, 'running'::text, 'recovering'::text]))",
"persistence_requests_pkey": "CREATE UNIQUE INDEX persistence_requests_pkey ON public.persistence_requests USING btree (id)",
"persistence_requests_principal": "CREATE INDEX persistence_requests_principal ON public.persistence_requests USING btree (principal_kind, principal_id, sequence DESC)",
diff --git a/test/migration-persistence-foreground-committed-index.test.ts b/test/migration-persistence-foreground-committed-index.test.ts
new file mode 100644
index 0000000000..82a01c1d6f
--- /dev/null
+++ b/test/migration-persistence-foreground-committed-index.test.ts
@@ -0,0 +1,134 @@
+/**
+ * #7649: a resident consumer's write claims after a restart run with an empty `syncSeen`, so their foreground-commit probe
+ * (journal.ts `FOREGROUND_WAITING_ON`) starts at sequence 0 and, with no index on committed rows by (worktree, sequence),
+ * walked every receipt on each claim. `persistence_requests_foreground_committed` serves it; its predicate must be the
+ * probe's own (`state='committed' AND NOT SYNC_KIND(c)`) or the planner cannot use it. The migration has the v179
+ * online-index shape: the Postgres blob omits the index and the migration builds it CONCURRENTLY, PGLite's blob carries
+ * it, and `gbrain repair request-indexes` rebuilds a dropped or INVALID one on a current schema. The Postgres access path
+ * on a large journal and an interrupted concurrent build are in test/e2e/persistence-claim-committed-probe-7649-postgres.test.ts.
+ *
+ * The migration is found by its name, never by its number: the number is provisional until the merge slot.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { randomUUID } from 'node:crypto';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import type { OperationContext } from '../src/core/ops/contract.ts';
+import { readRequestIndexStates } from '../src/core/persistence/checkpoint-validation.ts';
+import { claimNextWrite, SYNC_KIND } from '../src/core/persistence/journal.ts';
+import { PERSISTENCE_FOREGROUND_COMMITTED_INDEX, PERSISTENCE_REQUEST_ONLINE_INDEXES, PERSISTENCE_SCHEMA_STATEMENTS,
+ POSTGRES_CONCURRENT_PERSISTENCE_INDEXES } from '../src/core/persistence/schema.ts';
+import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.generated.ts';
+import { SCHEMA_SQL } from '../src/core/schema-embedded.generated.ts';
+import { MIGRATIONS } from '../src/core/schema-migrations/registry.generated.ts';
+import { LATEST_SCHEMA_VERSION } from '../src/core/schema-migrations/latest.generated.ts';
+import { runMigrations } from '../src/core/migrate.ts';
+import { runRepair, resolveRepairScope } from '../src/core/repair/core.ts';
+import { requestIndexesRepair } from '../src/core/repair/request-indexes.ts';
+import { requestIndexesCheck } from '../src/commands/doctor/checks/persistence-requests.ts';
+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const INDEX = PERSISTENCE_FOREGROUND_COMMITTED_INDEX;
+const migration = MIGRATIONS.find(m => m.name === 'persistence_foreground_committed_index');
+
+describe('#7649 the foreground-commit index migration module', () => {
+ test('is registered once in the v179 online-index shape, after the request-id index', () => {
+ expect(MIGRATIONS.filter(m => m.name === 'persistence_foreground_committed_index')).toHaveLength(1);
+ expect(migration).toMatchObject({ idempotent: true, transaction: false, sql: '' });
+ expect(migration!.version).toBeGreaterThan(MIGRATIONS.find(m => m.name === 'persistence_request_id_index')!.version);
+ expect(migration!.version).toBeLessThanOrEqual(LATEST_SCHEMA_VERSION);
+ expect(typeof migration!.handler).toBe('function');
+ });
+ test("its predicate is the claim probe's committed non-sync condition, keyed by worktree and sequence; DDL only", () => {
+ const sql: string = INDEX.sql;
+ expect(sql).toBe(`CREATE INDEX IF NOT EXISTS persistence_requests_foreground_committed
+ ON persistence_requests(worktree_id,sequence) WHERE state='committed' AND NOT ${SYNC_KIND('probe').replaceAll('probe.', '')}`);
+ expect(INDEX.sql).not.toMatch(/ALTER TABLE|ADD COLUMN|UPDATE /i);
+ });
+ test('the Postgres blob omits the index (the migration builds it CONCURRENTLY); the PGLite blob and v151 carry it; repair and doctor cover it', () => {
+ expect(POSTGRES_CONCURRENT_PERSISTENCE_INDEXES.has(INDEX.sql)).toBe(true);
+ expect(SCHEMA_SQL).not.toContain(INDEX.name);
+ expect(PGLITE_SCHEMA_SQL).toContain(INDEX.name);
+ expect(PERSISTENCE_SCHEMA_STATEMENTS).toContain(INDEX.sql);
+ expect(PERSISTENCE_REQUEST_ONLINE_INDEXES.map(index => index.name)).toContain(INDEX.name);
+ });
+});
+
+for (const backend of testBackends()) describe(`[${backend}] #7649 the foreground-commit index on a brain`, () => {
+ let engine: BrainEngine;
+ let closePostgres: (() => Promise<void>) | undefined;
+ const ctx = () => ({ engine, config: { engine: backend }, remote: false, dryRun: false, logger: { info() {}, warn() {}, error() {} } }) as unknown as OperationContext;
+ const state = async () => (await readRequestIndexStates(engine, [INDEX]))[0]!.state;
+ const oid = async () => (await engine.executeRaw<{ oid: string }>('SELECT to_regclass($1)::oid::text AS oid', [INDEX.name]))[0]!.oid;
+
+ beforeAll(async () => {
+ if (backend === 'pglite') { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }
+ else { const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL!); engine = pg.engine; closePostgres = pg.close; }
+ }, 120_000);
+ afterAll(async () => { await engine.disconnect(); await closePostgres?.(); }, 60_000);
+
+ test('a fresh brain has the index valid; the migration handler leaves it untouched', async () => {
+ expect(await state()).toBe('valid');
+ const before = await oid();
+ await migration!.handler!(engine);
+ expect(await oid()).toBe(before);
+ });
+
+ if (backend === 'pglite') test("on PGLite's journal of 20k sync receipts the cold claim's committed probe reads the index", async () => {
+ const host = randomUUID(), worktree = randomUUID();
+ await engine.executeRaw(`INSERT INTO persistence_worktrees(id,owner_host_id,state) VALUES ($1::uuid,$2::uuid,'active')`, [worktree, host]);
+ await engine.executeRaw(`INSERT INTO persistence_requests(principal_kind,principal_id,request_id,operation,source_id,source_incarnation,
+ worktree_id,slug,digest,intent,authority,intent_bytes,terminal_reservation,state,completed_at)
+ SELECT 'local_cli','seed',gen_random_uuid(),'put_page','default',(SELECT incarnation FROM sources WHERE id='default'),$1::uuid,'seed-'||g,'d',
+ CASE WHEN g>19980 THEN jsonb_build_object('kind','put_page') ELSE jsonb_build_object('kind','managed_sync_import','runId','r','index',g) END,
+ '{}'::jsonb,1,64,'committed',now()
+ FROM generate_series(1,20000) g`, [worktree]);
+ for (const [slug, kind] of [['queued-sync', 'managed_sync_import'], ['queued-fg', 'put_page']]) {
+ await engine.executeRaw(`INSERT INTO persistence_requests(principal_kind,principal_id,request_id,operation,source_id,source_incarnation,
+ worktree_id,slug,digest,intent,authority,intent_bytes,terminal_reservation,state)
+ VALUES ('local_cli','seed',gen_random_uuid(),'put_page','default',(SELECT incarnation FROM sources WHERE id='default'),$1::uuid,$2,'d',
+ jsonb_build_object('kind',$3::text),'{}'::jsonb,1,64,'queued')`, [worktree, slug, kind]);
+ }
+ await engine.executeRaw('ANALYZE persistence_requests');
+ let plan = '';
+ const original = engine.transactionDirect;
+ engine.transactionDirect = ((fn: (tx: BrainEngine) => Promise<unknown>, opts?: unknown) => original.call(engine, async (tx: BrainEngine) => {
+ const raw = tx.executeRaw.bind(tx);
+ tx.executeRaw = (async (sql: string, params?: unknown[], options?: unknown) => {
+ if (!sql.startsWith('WITH waiting AS MATERIALIZED')) return raw(sql, params as never, options as never);
+ plan = (await raw<{ 'QUERY PLAN': string }>(`EXPLAIN ${sql}`, params as never, options as never)).map(row => row['QUERY PLAN']).join('\n');
+ return [];
+ }) as BrainEngine['executeRaw'];
+ return fn(tx);
+ }, opts as never)) as BrainEngine['transactionDirect'];
+ try { expect(await claimNextWrite(engine, host, 30_000, [], [], [], {})).toBeNull(); }
+ finally { engine.transactionDirect = original; }
+ expect(plan).toContain(`Index Only Scan using ${INDEX.name} on persistence_requests c`);
+ expect(plan).not.toContain('Seq Scan on persistence_requests c');
+ expect((await claimNextWrite(engine, host, 30_000, [], [], [], {}))?.slug).toBe('queued-fg');
+ }, 60_000);
+
+ test('a populated brain upgrading from the previous schema version builds it in the migration', async () => {
+ await engine.executeRaw(`DROP INDEX ${INDEX.name}`);
+ expect(await state()).toBe('missing');
+ await engine.setConfig('version', String(migration!.version - 1));
+ const result = await runMigrations(engine);
+ expect(result.current).toBe(LATEST_SCHEMA_VERSION);
+ expect(await state()).toBe('valid');
+ }, 120_000);
+
+ test('a dropped index is named by doctor and rebuilt by gbrain repair request-indexes; a second repair is a no-op', async () => {
+ await engine.executeRaw(`DROP INDEX ${INDEX.name}`);
+ const check = await requestIndexesCheck(engine);
+ expect(check.status).toBe('warn');
+ expect(check.message).toContain(`${INDEX.name} is missing: request-table reads (managed sync checkpoints, write claims,`);
+ expect(check.message).toContain('gbrain repair request-indexes');
+ const scope = await resolveRepairScope(engine);
+ expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: false })).toMatchObject({ affected: 1, apply_command: 'gbrain repair request-indexes --apply' });
+ expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: true })).toMatchObject({ applied: 1, complete: true });
+ expect(await state()).toBe('valid');
+ expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: true })).toMatchObject({ affected: 0, applied: 0, complete: true });
+ expect(await requestIndexesCheck(engine)).toMatchObject({ status: 'ok', message: 'Request-table indexes are valid.' });
+ }, 60_000);
+});
diff --git a/test/persistence-request-indexes.test.ts b/test/persistence-request-indexes.test.ts
index e57c1a8d5f..b84ada9a91 100644
--- a/test/persistence-request-indexes.test.ts
+++ b/test/persistence-request-indexes.test.ts
@@ -42,12 +42,12 @@ test('#5762 gbrain repair request-indexes rebuilds a dropped index inline on PGL
await engine.executeRaw(`DROP INDEX ${NAMES[1]}`);
const doctor = await requestIndexesCheck(engine);
expect(doctor).toMatchObject({ status: 'warn', details: { count: 1, repair: 'request-indexes', docs: 'docs/guides/repair.md#request-indexes' } });
- expect(doctor.message).toBe(`${NAMES[1]} is missing: request-table reads (managed sync checkpoints, page purges, dream synthesis postprocessing) can time out on a large request table. Rebuild on the brain host: gbrain repair request-indexes --apply`);
+ expect(doctor.message).toBe(`${NAMES[1]} is missing: request-table reads (managed sync checkpoints, write claims, page purges, dream synthesis postprocessing) can time out on a large request table. Rebuild on the brain host: gbrain repair request-indexes --apply`);
const scope = await resolveRepairScope(engine);
expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: false })).toMatchObject({ affected: 1, sample: [`(brain):${NAMES[1]}`], cost: { lifetime_ids: 0, embedding_pages: 0 } });
expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: true })).toMatchObject({ applied: 1, complete: true });
- // #6511: the online-built set is the checkpoint pair plus the purge slug index.
- expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid']);
+ // #6511/#6757/#7649: the online-built set is the checkpoint pair, the purge slug, request-id and foreground-commit indexes.
+ expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
expect(await runRepair(ctx(), requestIndexesRepair, scope, { apply: true })).toMatchObject({ affected: 0, applied: 0 });
}, 60_000);
diff --git a/test/persistence-sync-checkpoint-validation.test.ts b/test/persistence-sync-checkpoint-validation.test.ts
index 7af1b257f0..8df3514492 100644
--- a/test/persistence-sync-checkpoint-validation.test.ts
+++ b/test/persistence-sync-checkpoint-validation.test.ts
@@ -311,13 +311,13 @@ test('#5762 the hint names one of three index states and keeps the saved process
});
test('#5762 a dropped index reads as missing and the hint then names the rebuild command', async () => each(async engine => {
- // #6511 and #6757 added the purge slug and request-id indexes to the online-built set; the default read lists them after the checkpoint pair.
- expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid']);
+ // #6511, #6757 and #7649 added the purge slug, request-id and foreground-commit indexes to the online-built set; the default read lists them after the checkpoint pair.
+ expect((await readRequestIndexStates(engine)).map(index => index.state)).toEqual(['valid', 'valid', 'valid', 'valid', 'valid']);
await engine.executeRaw('DROP INDEX persistence_requests_sync_run_committed');
try {
expect(await readRequestIndexStates(engine)).toEqual([{ name: 'persistence_requests_sync_run_open', state: 'valid' },
{ name: 'persistence_requests_sync_run_committed', state: 'missing' }, { name: 'persistence_requests_slug_intent', state: 'valid' },
- { name: 'persistence_requests_request_id', state: 'valid' }]);
+ { name: 'persistence_requests_request_id', state: 'valid' }, { name: 'persistence_requests_foreground_committed', state: 'valid' }]);
} finally {
const { PERSISTENCE_SYNC_RUN_COMMITTED_INDEX_SQL } = await import('../src/core/persistence/schema.ts');
await engine.executeRaw(PERSISTENCE_SYNC_RUN_COMMITTED_INDEX_SQL);
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index 3c2ed50bec..86c0e71b3b 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -252,6 +252,7 @@ test/managed-sync-preparation-stalled-holds.test.ts
test/managed-sync-row-parity.test.ts
test/managed-sync-stale-group-head.test.ts
test/mcp-unreadable-sources-notice.test.ts
+test/migration-persistence-foreground-committed-index.test.ts
test/migration-persistence-request-id-index.test.ts
test/migration-persistence-request-slug-index.test.ts
test/on-commit-embedding.serial.test.ts