Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Keep the search readiness check fast while upgraded pages wait for their rebuild (#9192) (#9490)
Garry Tane7864f6cf020
diff --git a/changelog.d/capy-projection-waiting-index-9192.md b/changelog.d/capy-projection-waiting-index-9192.mdnew file mode 100644index 0000000000..64245ea87b--- /dev/null+++ b/changelog.d/capy-projection-waiting-index-9192.md@@ -0,0 +1,4 @@+**The check that runs before every search stays fast while a large brain's pages wait to be rebuilt after an upgrade from before v0.51.**++### Fixed+- After an upgrade from before v0.51, pages still answering from their old index were walked one by one by the readiness check that runs before every search, so the check cost 21 ms on a test brain with 20,000 such pages and 126 to 149 ms with 200,000. A new partial index (migration v317) holds only the pages that are out of search, so the check now takes about 2 ms and the index is empty once the rebuild finishes. On Postgres the index builds without blocking writes; nothing is rewritten. Follow-up to #9472 (#9192).diff --git a/docs/architecture/key-files/core-search-1.md b/docs/architecture/key-files/core-search-1.mdindex 1eb3f8e74b..7e37f87225 100644--- a/docs/architecture/key-files/core-search-1.md+++ b/docs/architecture/key-files/core-search-1.md@@ -67,7 +67,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/search/private-visibility.ts` — shared page-visibility resolver and SQL predicates. Explicit local trust disables page filtering; operator env/config opt-outs keep their semantics; failed config reads enforce privacy. `privatePagesFilterFragment` authorizes concrete page rows and `isPrivatePage` is its row-side twin: explicit `private` hides a page, absent visibility is world for ordinary pages and private for extracted atoms (`type = 'atom'`) and synthesized concepts (`type = 'concept'` with `synthesized_by`), because their origin may be a private page or transcript. The page predicate also follows derived origins: an atom whose `source_slug` page is explicitly private, and a synthesized concept with a private input atom (or an input whose origin is private, including an origin a rename moved after a later rename onto the old slug dropped its alias: `slug_rename_history`, GST-2687) over `concept-provenance` edges, are private whatever their own field says, so a later private flip of an origin reaches derived pages before any repair. The concept-input test is the SQL function `gbrain_concept_inputs_private(page_id)` (migration v311, never inlined, so a read plans one call per page instead of the subplans); a read uses it when its schema version is v311 or later: the one a remote, scope or attested record carries (`RemotePolicy.schemaVersion`), or for a local read the one its scoped engine recorded at connect or migration (`schemaVersionSeen`, `migrate-pending.ts`); otherwise the same test runs inline, so reads keep working before the migration applies. Chronicle events follow `event.depth` (fail closed). Declared lineage matches `derived_from` slugs with `= ANY(ARRAY(...))`, not `IN (SELECT ... jsonb_array_elements_text(...))`: same rows, but the planner priced the IN form as a 100-row semi-join per page and pushed every read carrying the rule past `jit_above_cost` (`test/e2e/private-visibility-plan-postgres.test.ts`, which measures with `links` and `slug_rename_history` both unanalyzed and analyzed; test `test/concept-rename-history-gst2687.test.ts`, both engines). `privateSnapshotFilterFragment` is the same rule on a row's own frontmatter for `page_versions`. `effectiveVisibility(origin)` (transcript or missing origin private, page origin per the read rule) and `strictestVisibility` are used by both `extract_atoms` paths, concept synthesis and `gbrain repair visibility`. Remote publication rechecks (`persistence/page-visibility.ts`) and the alias-inject recheck in `hybrid.ts` use the same rule. Tests: `test/private-visibility.test.ts`, `test/derived-page-visibility.test.ts` (PGLite and Postgres); link origins and timeline projections use independent ID-based predicates. `privateProvenanceFilterFragment` hides a fact whose provenance page (`source_markdown_slug`) is private; the fact-list reads in `engine-sql/facts.ts` (by entity, since, keyset, session, supersessions, trajectory) apply it behind `FactListOpts.excludePrivate` / `TrajectoryOpts.excludePrivate`, which every remote fact surface resolves with `resolveExcludePrivatePages` (`test/remote-fact-list-private-provenance.test.ts`, both engines). Remote fact writes apply the same predicate behind an `excludePrivate` option: `forget` (`guardRemoteForget`, the row lock in `persistence/memory-mutations.ts`, `recordFactWithdrawal`, `similarActiveAfterForget`) and `remember` (`decideSingleFact`'s exact and cosine arms, `decideReplacement`), so a fact remote reads hide is `not_found` to a remote write and never matched as its duplicate (`test/remote-fact-write-private-provenance.test.ts`, both engines). Legacy slug-set helpers only enumerate names; they never authorize another row's data. `ops/context.ts:readPolicyOpts` combines canonical source precedence, resolved privacy and holder permissions. Both engines filter content/history/graph/identity/analytics before limits or aggregation; current and historical visibility are checked independently while page privacy is enforced. Semantic response caching remains disabled, so visibility-key folding alone never authorizes reuse.-- `src/core/search/projection-readiness.ts` — one JIT-off statement probes visible unsealed pages: ready, projection_pending, by_design (only protected/archived unsealed) or unknown; LIMIT 1 arms use the v161 pending index. Cached per scope by DB generation (page clock, pages filenode, archive flags, no open tx); `GBRAIN_PROJECTION_READINESS_CACHE=0` disables.+- `src/core/search/projection-readiness.ts` — one JIT-off statement probes visible unsealed pages: ready, projection_pending, by_design (only protected/archived unsealed) or unknown; LIMIT 1 arms use the v317 `idx_pages_projection_waiting` partial index (only pages out of search, so pages still served from their pre-v153 index are not walked; #9192). Cached per scope by DB generation (page clock, pages filenode, archive flags, no open tx); `GBRAIN_PROJECTION_READINESS_CACHE=0` disables.- `src/core/search/projection-statistics.ts` — current-projection expression statistics, strict migration postconditions and owner-authorized batch refresh. Migration 160 verifies object identity and populated samples before advancing the ledger, distinguishing empty tables from statistics this role cannot see. Import, sync, reindex and drained recovery refresh outside page locks; failures preserve completed writes and give a maintenance hint. A pass that changed fewer than 50 + 10% of the sampled pages leaves the statistics alone once they exist (the projection statistic and, on Postgres, every `SEARCH_STATISTICS_COLUMNS` column: `missingSearchStatistics`), and PGLite then analyzes only the hot tables its row deltas mark stale. On Postgres it also samples the page columns search filters read (`SEARCH_PAGE_STATISTICS_SQL`) and the chunk `SEARCH_STATISTICS_COLUMNS`; a vacuum holding a table defers it (analyze-lock.ts). `refreshChunkStatistics` runs the chunk step after a drain that changed 50 + 10% of chunks (any sans statistics): without them the vector candidate statement sorts every eligible chunk instead of walking the HNSW index (`test/e2e/vector-chunk-statistics-postgres.test.ts`). Absent search statistics (pg_upgrade, a deleted pg_statistic; 50-60 s searches at 5k pages) get `ensureSearchStatistics`: PostgresEngine runs it once per engine in the background on its first pool search (never awaited; disconnect waits), and doctor `planner_stats_stale` plus `repair planner-stats` report and apply the same refresh (`test/search-statistics-guard.test.ts`).diff --git a/src/core/schema-migrations/latest.generated.ts b/src/core/schema-migrations/latest.generated.tsindex cb8e941e5c..8c86b99565 100644--- a/src/core/schema-migrations/latest.generated.ts+++ b/src/core/schema-migrations/latest.generated.ts@@ -4,4 +4,4 @@// The highest schema migration version (migrate.ts LATEST_VERSION), importable without// loading the migrations themselves.-export const LATEST_SCHEMA_VERSION = 316;+export const LATEST_SCHEMA_VERSION = 317;diff --git a/src/core/schema-migrations/registry.generated.ts b/src/core/schema-migrations/registry.generated.tsindex c6761f0918..ed8ba44003 100644--- a/src/core/schema-migrations/registry.generated.ts+++ b/src/core/schema-migrations/registry.generated.ts@@ -317,6 +317,7 @@ import { v313 } from './v313-wake-events.ts';import { v314 } from './v314-guest-notes-fail-closed.ts';import { v315 } from './v315-facts-context-head-index.ts';import { v316 } from './v316-pages-source-type-index.ts';+import { v317 } from './v317-pages-projection-waiting-index.ts';// Array order is historical (see HISTORICAL_ARRAY_ORDER in the generator); the// runner sorts by version before applying.@@ -632,4 +633,5 @@ export const MIGRATIONS: Migration[] = [v314,v315,v316,+ v317,];diff --git a/src/core/schema-migrations/v317-pages-projection-waiting-index.ts b/src/core/schema-migrations/v317-pages-projection-waiting-index.tsnew file mode 100644index 0000000000..de12ee2c9c--- /dev/null+++ b/src/core/schema-migrations/v317-pages-projection-waiting-index.ts@@ -0,0 +1,25 @@+import type { Migration } from './types.ts';+import { buildIndexOnline, migrationNotice } from './helpers.ts';+import { ACTIVATION_SERVED_REVISION } from '../search/safe-chunks.ts';++// Applied by src/core/migrate.ts on the next initSchema(); see docs/ENGINES.md+// ("Canonical schema sources") for when schema.sql or a TS fragment also changes.+//+// #9192: the search readiness probe asks whether any visible page is out of search, in idx_pages_projection_pending's+// order under LIMIT 1. A page v153 left serving its pre-upgrade index is in that index too, so while a large brain+// rebuilds, the probe stepped over every such page on every search (about 0.8 us each, about 0.5 s at 600k pages).+// idx_pages_projection_waiting holds only the pages out of search, so the walk stops at its first row and the index is+// empty once the rebuild finishes. Postgres builds it CONCURRENTLY outside the migration transaction (a handler);+// PGLite builds inline. Nothing is rewritten. No schema.sql mirror, like v316.+export const v317: Migration = {+ version: 317,+ name: 'pages_projection_waiting_index',+ idempotent: true,+ sql: '',+ handler: async (engine) => {+ await buildIndexOnline(engine, 317, { name: 'idx_pages_projection_waiting', table: 'pages',+ sql: `CREATE INDEX IF NOT EXISTS idx_pages_projection_waiting ON pages (source_id, page_kind, slug)+ WHERE deleted_at IS NULL AND text_projection_revision IS DISTINCT FROM knowledge_revision+ AND text_projection_revision IS DISTINCT FROM '${ACTIVATION_SERVED_REVISION}'::uuid` }, { notice: migrationNotice });+ },+};diff --git a/src/core/search/projection-readiness.ts b/src/core/search/projection-readiness.tsindex d7fe23f9fb..0ebfc2d2f7 100644--- a/src/core/search/projection-readiness.ts+++ b/src/core/search/projection-readiness.ts@@ -91,8 +91,9 @@ async function probeUncached(// pending its LIMIT 1 finds no protected page to stop at, so evaluating it anyway read every unsealed page (#8000).// The planner has no statistics for a column-to-column comparison, so it believes most pages are unsealed and an// EXISTS chose a sequential scan of pages that expected to stop at once; on a sealed brain it read every page- // (~600 ms at 600k pages, on every cold call). Ordering by idx_pages_projection_pending's columns under LIMIT 1- // makes that partial index the cheap first row, so the probe reads only the unsealed pages.+ // (~600 ms at 600k pages, on every cold call). Ordering by the partial index's columns under LIMIT 1 makes it the+ // cheap first row. The index is idx_pages_projection_waiting (v317, #9192): its predicate is this one, so pages v153+ // left serving their pre-upgrade index are not in it and the probe no longer steps over each of them.// JIT off (#8033): a remote caller's private-visibility subplans lift the estimate past jit_above_cost, and compiling// them cost ~66 of ~67 ms at 20k pages while the probe itself reads one index row.const unsealed = (cond: string) => `(SELECT true FROM pages p JOIN sources s ON s.id = p.source_iddiff --git a/test/e2e/projection-readiness-plan-postgres.test.ts b/test/e2e/projection-readiness-plan-postgres.test.tsindex 92d9fa1339..8e1d9f2013 100644--- a/test/e2e/projection-readiness-plan-postgres.test.ts+++ b/test/e2e/projection-readiness-plan-postgres.test.ts@@ -5,13 +5,15 @@* so an EXISTS over it chose a sequential scan of pages that expected to stop* at the first row; on a sealed brain it read every page (~600 ms at 600k* pages). Pins the plan on real Postgres with analyzed, fully sealed filler- * pages: the probe walks the partial idx_pages_projection_pending and no- * sequential scan of pages runs.+ * pages: the probe walks a partial index and no sequential scan of pages runs.+ * Since #9192 that index is idx_pages_projection_waiting (v317), which also+ * leaves out pages still served from their pre-upgrade index.*/import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import type { BrainEngine } from '../../src/core/engine.ts';import { PostgresEngine } from '../../src/core/postgres-engine.ts';import { probeProjectionReadiness, type ProjectionReadinessScope } from '../../src/core/search/projection-readiness.ts';+import { ACTIVATION_SERVED_REVISION } from '../../src/core/search/safe-chunks.ts';import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';import { withEnv } from '../helpers/with-env.ts';import { hasDatabase } from './helpers.ts';@@ -62,7 +64,7 @@ describePostgres('projection readiness probe plan on Postgres (GBRA-156)', () =>const { result, probe } = await probeStatement(scope);expect(result).toEqual({ status: 'ready', ready: true });const text = await plan(probe.sql, probe.params);- expect(text).toMatch(/Index Scan using idx_pages_projection_pending on pages p\b/);+ expect(text).toMatch(/Index Scan using idx_pages_projection_waiting on pages p\b/);// The private-visibility clause carries lazily evaluated subplans over pages; none may run.expect(text.split('\n').filter(line => line.includes('Seq Scan on pages') && !line.includes('never executed'))).toEqual([]);});@@ -105,6 +107,23 @@ describePostgres('projection readiness probe plan on Postgres (GBRA-156)', () =>}});+ // #9192: pages v153 left serving their pre-upgrade index are unsealed but not pending. The probe reads+ // idx_pages_projection_waiting, which leaves them out, instead of stepping over each one in idx_pages_projection_pending.+ test('a brain of pages still served from their pre-upgrade index answers ready without walking them (#9192)', async () => {+ await engine.executeRaw(`UPDATE pages SET text_projection_revision='${ACTIVATION_SERVED_REVISION}'::uuid`);+ try {+ await engine.executeRaw('ANALYZE pages');+ const { result, probe } = await probeStatement({});+ expect(result).toEqual({ status: 'ready', ready: true });+ const text = await plan(probe.sql, probe.params);+ expect(text).toMatch(/Index Scan using idx_pages_projection_waiting on pages p\b/);+ expect(text).not.toContain('Rows Removed by Filter');+ } finally {+ await engine.executeRaw('UPDATE pages SET text_projection_revision=knowledge_revision');+ await engine.executeRaw('ANALYZE pages');+ }+ });+test('a brain with many pending pages answers pending without the by_design walk (#8000)', async () => {await engine.executeRaw("UPDATE pages SET text_projection_revision=NULL WHERE source_id='source-a'");try {diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-default.json b/test/fixtures/goldens/catalog/pglite-engine-init-default.jsonindex c19734fd4d..aefbd11828 100644--- a/test/fixtures/goldens/catalog/pglite-engine-init-default.json+++ b/test/fixtures/goldens/catalog/pglite-engine-init-default.json@@ -1465,6 +1465,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json b/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.jsonindex ab9f75c6c7..c740019d4c 100644--- a/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json+++ b/test/fixtures/goldens/catalog/pglite-engine-init-fts-portuguese.json@@ -1465,6 +1465,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json b/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.jsonindex 0a71f909f7..fe21554b09 100644--- a/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json+++ b/test/fixtures/goldens/catalog/pglite-engine-init-high-dims.json@@ -1463,6 +1463,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-default.json b/test/fixtures/goldens/catalog/postgres-engine-init-default.jsonindex 93651da54d..471cf0a697 100644--- a/test/fixtures/goldens/catalog/postgres-engine-init-default.json+++ b/test/fixtures/goldens/catalog/postgres-engine-init-default.json@@ -1468,6 +1468,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json b/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.jsonindex 515ed181fc..db15a1783e 100644--- a/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json+++ b/test/fixtures/goldens/catalog/postgres-engine-init-fts-portuguese.json@@ -1468,6 +1468,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json b/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.jsonindex b63c09106a..f4879910fc 100644--- a/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json+++ b/test/fixtures/goldens/catalog/postgres-engine-init-high-dims.json@@ -1466,6 +1466,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.jsonindex 63475f7c1e..e99d61f5ec 100644--- a/test/fixtures/goldens/migrations/records.json+++ b/test/fixtures/goldens/migrations/records.json@@ -1,6 +1,6 @@{"golden": {- "count": 311,+ "count": 312,"gaps": [17,18,@@ -104,7 +104,7 @@"widenIdentityIndex": "86a6176bd7637d964dbc1d3cd3b1bdfa3ca0161e1ee248c37d7380a1471282e6","withSchemaLockTimeout": "95605bd3004955e11e936c7a4aa6889319b401098abf9db18ef3e48da2c3fc59"},- "latest": 316,+ "latest": 317,"records": [{"handler": {@@ -5154,6 +5154,35 @@"transaction": "absent","verify": "absent","version": 316+ },+ {+ "handler": {+ "form": "arrow",+ "helpers": [+ "ACTIVATION_SERVED_REVISION",+ "PLANNER_REPAIR_TABLES",+ "analyzeAfterIndexBuild",+ "analyzeDeferred",+ "analyzeYieldingToVacuum",+ "buildIndexOnline",+ "dropInvalidConcurrentIndex",+ "lockTableForAnalyze",+ "migrationNotice",+ "quietMigrationNotices",+ "tableRows"+ ],+ "sha256": "247e56b25c60144716ebde796118016cc9e24c1a17e338d47df750101917e8a7"+ },+ "idempotent": true,+ "name": "pages_projection_waiting_index",+ "sql": {+ "length": 0,+ "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"+ },+ "sqlFor": "absent",+ "transaction": "absent",+ "verify": "absent",+ "version": 317}],"versions": [@@ -5467,7 +5496,8 @@313,314,315,- 316+ 316,+ 317]},"normalizer": "migrations-record-v1"diff --git a/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json b/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.jsonindex 70c823596c..bb3e06f9e4 100644--- a/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json+++ b/test/fixtures/goldens/pglite-upgrade-replay/catalog-after-boot.json@@ -1465,6 +1465,7 @@"idx_oauth_tokens_expiry": "CREATE INDEX idx_oauth_tokens_expiry ON public.oauth_tokens USING btree (expires_at)","idx_pages_frontmatter": "CREATE INDEX idx_pages_frontmatter ON public.pages USING gin (frontmatter)","idx_pages_projection_pending": "CREATE INDEX idx_pages_projection_pending ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision))",+ "idx_pages_projection_waiting": "CREATE INDEX idx_pages_projection_waiting ON public.pages USING btree (source_id, page_kind, slug) WHERE ((deleted_at IS NULL) AND (text_projection_revision IS DISTINCT FROM knowledge_revision) AND (text_projection_revision IS DISTINCT FROM '00000000-0000-0000-0000-000000000153'::uuid))","idx_pages_search": "CREATE INDEX idx_pages_search ON public.pages USING gin (search_vector)","idx_pages_slug_basename": "CREATE INDEX idx_pages_slug_basename ON public.pages USING btree (source_id, regexp_replace(slug, '^.*/'::text, ''::text))","idx_pages_slug_trgm": "CREATE INDEX idx_pages_slug_trgm ON public.pages USING gin (slug gin_trgm_ops)",diff --git a/test/migrations-notice.test.ts b/test/migrations-notice.test.tsindex 61e6fccc30..7a0b6784a9 100644--- a/test/migrations-notice.test.ts+++ b/test/migrations-notice.test.ts@@ -101,6 +101,7 @@ describe('pendingUpgradeMigrationsNotice', () => {test('an empty ledger on a brain created in this run (schema version 1 at connect) gets no notice', async () => {const created = { getConfig: async () => null } as never;noteSchemaVersion(created, 1);+ // pregate-literal: LATEST_SCHEMA_VERSION=316 any version above 1 stands for the brain after this run migrated itnoteSchemaVersion(created, 316);expect(await inHome(() => pendingUpgradeMigrationsNotice(created, 'query', latest))).toBeNull();});diff --git a/test/search-during-upgrade-rebuild-9192.test.ts b/test/search-during-upgrade-rebuild-9192.test.tsindex f4fa7b44a1..571269a980 100644--- a/test/search-during-upgrade-rebuild-9192.test.ts+++ b/test/search-during-upgrade-rebuild-9192.test.ts@@ -19,6 +19,7 @@ import { dispatchToolCall } from '../src/mcp/dispatch.ts';import { installFixtureChunks } from './helpers/page-projection.ts';import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';import { testBackends } from './helpers/test-backends.ts';+import { withEnv } from './helpers/with-env.ts';const TEXT = 'The okapi theremin appears in this synthetic note.';const SLUGS = ['notes/okapi-one', 'notes/okapi-two'];@@ -210,6 +211,19 @@ for (const backend of testBackends()) {expect((await projectionBacklog(engine)).pending).toBe(2);}, url), 120_000);+ test('the readiness probe reads the index of pages waiting for a rebuild, which leaves out pages served from their old index (v317)', () => activated(async engine => {+ const captured: Array<{ sql: string; params?: unknown[] }> = [];+ const recording = { executeRaw: (sql: string, params?: unknown[]) => { captured.push({ sql, params }); return engine.executeRaw(sql, params); } } as unknown as BrainEngine;+ expect((await withEnv({ GBRAIN_PROJECTION_READINESS_CACHE: '0' }, () => probeProjectionReadiness(recording))).status).toBe('ready');+ const probe = captured.find(call => call.sql.includes('IS DISTINCT FROM'));+ expect(probe).toBeDefined();+ const plan = await engine.transaction(async tx => {+ await tx.executeRaw('SET LOCAL enable_seqscan = off');+ return (await tx.executeRaw<Record<string, string>>(`EXPLAIN ${probe!.sql}`, probe!.params)).map(row => Object.values(row)[0]).join('\n');+ });+ expect(plan).toContain('idx_pages_projection_waiting');+ }, url), 120_000);+test('the v153 handler says which pages stay served and which wait', () => activated(async engine => {const writes: string[] = [];const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);