Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Keep unfenced pages searchable while an upgrade from before v0.51 rebuilds the index (#9192) (#9472)

Garry Tan0833e68b7517
diff --git a/changelog.d/capy-search-during-upgrade-rebuild-9192.md b/changelog.d/capy-search-during-upgrade-rebuild-9192.md
new file mode 100644
index 0000000000..e9ab7ab83f
--- /dev/null
+++ b/changelog.d/capy-search-during-upgrade-rebuild-9192.md
@@ -0,0 +1,8 @@
+---
+important: regression
+---
+**An upgrade from before v0.51 keeps a page in search while the index rebuilds, when its old index can't hold withdrawn, held or redacted text, and says how long the rebuild will take.**
+
+### Fixed
+- Upgrading from a release before v0.51 used to take every page out of keyword and vector search until `gbrain projections drain` finished: about 9 hours on a 50,000-page brain behind an 18 ms database round trip. Now a Markdown or code page indexed at or after the safe-chunk fence keeps answering from its old index until its own rebuild, as long as it has no facts or takes fence, no private-key text and no quarantine marker. Pages with a facts or takes fence, pages with private-key text, quarantined pages, images and pages indexed before the safe-chunk fence still wait for the rebuild, so a brain made mostly of entity pages with a Facts fence gets little from this change. Access checks are unchanged, a page that changes after the upgrade leaves search until it is rebuilt, and no embedding calls are made. Reported by @jonahberg (#9192).
+- The upgrade's `[153]` line says how many pages keep answering, how many wait and why, and how long the rebuild takes at the round trip it measured to the database. `gbrain doctor` still counts every page waiting for the rebuild, and says results may be incomplete only while some page is out of search.
diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.md
index 40c08c1bc1..5759d23230 100644
--- a/docs/architecture/RETRIEVAL.md
+++ b/docs/architecture/RETRIEVAL.md
@@ -674,7 +674,8 @@ take-holder permissions.
**Say to your agent:** *"Check whether my search index is ready, and repair
code metadata without spending on embeddings."* Search and query report
`projection_readiness` for empty and nonempty results. `projection_pending`
-means visible pages lack a current revision seal; `projection_status_unknown`
+means visible pages lack a current revision seal and are not still served from
+the index they had before migration 153's rebuild; `projection_status_unknown`
means the diagnostic could not establish readiness. Neither is a clean miss.
`projection_by_design` means the only unsealed pages in scope are
protected-OCR images (or pages of an archived source), which stay out of the
diff --git a/docs/architecture/canonical-writers.tsv b/docs/architecture/canonical-writers.tsv
index c4e90f02d4..4cab43ac6a 100644
--- a/docs/architecture/canonical-writers.tsv
+++ b/docs/architecture/canonical-writers.tsv
@@ -116,7 +116,7 @@ src/core/ops/links.ts 3 coordinator Unmanaged add_link/remove_link keep the engi
src/core/ops/pages.ts 0 coordinator Page mutations delegate to persistence modules; purge_deleted_pages runs purgeDeletedPagesCoordinated.
src/core/output/writer.ts 6 engine_guard Canonical table SQL trigger requires coordinator capability; legacy calls fail before canonical DB publication.
src/core/page-state/materialize.ts 1 coordinator Publication is transaction-scoped under admitted authority, source/page guards and revision checks.
-src/core/page-state/projection-schema.ts 1 schema Schema installation and forward-reference DDL; coordinated activation quiesces canonical writers.
+src/core/page-state/projection-schema.ts 2 schema Schema installation and forward-reference DDL; coordinated activation quiesces canonical writers. #9192: the activation also assigns a first knowledge_revision to pages written before v150 (no content column changes).
src/core/page-state/projections.ts 3 projection Derived indexing, retrieval telemetry, or source settings; canonical fields remain guarded by SQL triggers.
src/core/page-state/versions.ts 2 projection GBRA-217: stamps only pages.version_history_gap_at (history metadata outside the canonical trigger's guarded fields) in the caller's versioning transaction, when an import reports a multi-commit gap or a prune drops snapshots; no canonical field or body is written.
src/core/page-state/image-seal.ts 1 projection The explicit image seal (#6803) sets only text_projection_revision on an importer-attested image page, under its page guard with a revision compare-and-set; no canonical field or body is written.
diff --git a/docs/architecture/key-files/core-search-2.md b/docs/architecture/key-files/core-search-2.md
index f36999da0a..d8152ab66f 100644
--- a/docs/architecture/key-files/core-search-2.md
+++ b/docs/architecture/key-files/core-search-2.md
@@ -29,7 +29,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/search/return-policy.ts` (default OFF) — intent-aware adaptive return-sizing. Pure, dependency-light module that trims the final ranked candidate set to an intent-driven cap instead of returning the full top-K. `entity` intent gets a tight cap; `temporal`/`event`/`general` get a recall-preserving cap (`concept` is coerced to `general` by `hybrid.ts` before the call — concept queries want breadth). A `minKeep` failsafe (≥1) guarantees a human never gets a silent blank when candidates exist. WHY a cap, not a score-cliff detector: PrecisionMemBench instrumentation (gbrain-evals) measured the rank1→rank2 RRF gap is ~identical whether rank-1 is correct (0.602) or wrong (0.569) — mechanical decay, not a separatrix; rank-1 is right in 94% of single-answer cases, so "return a tight set" is the whole win and cliff-cutting just adds noise. Exports `AdaptiveReturnConfig`, `DEFAULT_ADAPTIVE_RETURN` (frozen: `enabled=false`, `entityMax=2`, `otherMax=6`, `minKeep=1`), `AdaptiveReturnDecision` (`{applied, intent, cap, kept, total}`), `AdaptiveReturnInput` (`boolean | Partial<AdaptiveReturnConfig> | undefined`), `adaptiveReturnFromConfig(cfg)`, `resolveAdaptiveReturn(perCall, fromConfig)` (defaults → config → per-call merge), `adaptiveReturnEnabled(...)` (cache-skip gate check), `applyAdaptiveReturn(results, intent, cfg)` (the trim). Config knobs (DB or file plane): `search.adaptive_return` (master switch), `search.adaptive_return_entity_max`, `search.adaptive_return_other_max`, `search.adaptive_return_min_keep` (each clamped ≥1). Wired into `hybridSearch` AFTER `applyReranker`, BEFORE the `limit` slice, and ONLY on the first page (`offset===0`) — paginating a confidence-gated set is incoherent, so paginated calls fall through to the fixed limit. Stamps the decision onto `HybridSearchMeta.adaptive_return` for `gbrain search --explain`. `hybridSearchCached` SKIPS the cache when the gate is on (a trimmed set must not be served to a gate-off lookup and vice versa). `SearchOpts.adaptiveReturn` + `HybridSearchMeta.adaptive_return` declared in `src/core/types.ts`. Agent-facing: the `query` op (`src/core/operations.ts`) exposes an `adaptive_return` boolean param whose description instructs the agent WHEN to set it (single-answer → on; breadth/exploration → off; pass `limit:1` for a hard single-answer cap), threaded into `hybridSearchCached` — end users never touch the config knob; their agent decides per query (same pattern as `salience`/`recency`). Pinned by `test/search/return-policy.test.ts` (mechanism) + `test/search/query-op-adaptive-return.test.ts` (agent surface: param exists + description teaches both directions + the never-empty contract).
-- `src/core/search/safe-chunks.ts` — shared current-revision and remote safe-chunk predicates. `currentTextProjectionFilter` emits `(text_projection_revision = knowledge_revision) IS TRUE`, preserving NULL exclusion while allowing expression statistics to guide the planner. Visibility and revision predicates stay inside candidate selection. Body/chunk changes invalidate the safe index until a guarded rebuild. `belowSafeChunkFence` is the page-level predicate shared by the importers' unchanged-content re-seal and the drain. `chunkWriteInvalidations(pageId, chunks)` yields one guarded `chunker_version = -1` invalidation per row batch of a chunk replacement (`batch-rows.ts` `jsonRowBatches`); the caller stops at the first that returns a row. Wave B: `chunkVisibleFilter(p, c)` (current projection plus a row at or below the page's body generation; sealed pages hold no retired rows, and no subplan keeps vector search on HNSW) is the search predicate; `chunkGenerationFilter(p, c)` serves generation readers (an inline row of a non-segmented page, or a row whose occurrence has a live manifest row, hiding rows a deferred retirement or demotion left; `p` needs `body_storage`), `chunkGenerationProbe(c)` the same without a joined page, and `chunkManifestJoin` / `chunkDocumentOrder` give segmented rows document order (manifest byte_start, local_ordinal); `scripts/check-chunk-visibility.ts` requires one of them or a reason on every `content_chunks` reader (rules in its header).
+- `src/core/search/safe-chunks.ts` — shared current-revision and remote safe-chunk predicates. `currentTextProjectionFilter` emits `(text_projection_revision = knowledge_revision) IS TRUE`, preserving NULL exclusion while allowing expression statistics to guide the planner. Visibility and revision predicates stay inside candidate selection. Body/chunk changes invalidate the safe index until a guarded rebuild. `belowSafeChunkFence` is the page-level predicate shared by the importers' unchanged-content re-seal and the drain. `chunkWriteInvalidations(pageId, chunks)` yields one guarded `chunker_version = -1` invalidation per row batch of a chunk replacement (`batch-rows.ts` `jsonRowBatches`); the caller stops at the first that returns a row. Wave B: `chunkVisibleFilter(p, c)` (served projection (`servedTextProjectionFilter`) plus a row at or below the page's body generation; sealed pages hold no retired rows, and no subplan keeps vector search on HNSW) is the search predicate; `chunkGenerationFilter(p, c)` serves generation readers (an inline row of a non-segmented page, or a row whose occurrence has a live manifest row, hiding rows a deferred retirement or demotion left; `p` needs `body_storage`), `chunkGenerationProbe(c)` the same without a joined page, and `chunkManifestJoin` / `chunkDocumentOrder` give segmented rows document order (manifest byte_start, local_ordinal); `scripts/check-chunk-visibility.ts` requires one of them or a reason on every `content_chunks` reader (rules in its header).
- `src/core/search/source-boost.ts` — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, <fork>/chat/ 0.5, archive/ 0.5, extracts/ 0.3) and `DEFAULT_HARD_EXCLUDES` (test/, attachments/, .raw/). `archive/` is DEMOTED (findable, ranked below curated), not hard-excluded — archive holds high-signal history users expect to retrieve; the demote is a prior at the SQL/fusion layer and the cross-encoder reranker can still promote a strongly-matching archive page. `parseSourceBoostEnv`/`parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST`/`GBRAIN_SEARCH_EXCLUDE`. `resolveBoostMap(env, config)` layers defaults (dropped by a `none` entry), the brain's `search.source_boosts` config and the env override; hybridSearch resolves it once from the mode config snapshot and passes it to the engines as `SearchOpts.source_boosts` (engines fall back to defaults + env when absent; the keyword-only `search` op branch reads the key itself). `resolveHardExcludes` merges defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. The surviving exclude policy is auditable via the `hidden_by_search_policy` doctor check (`src/commands/doctor/checks/calibration.ts`, local + remote paths) which counts chunked pages withheld per active exclude prefix, reusing `resolveHardExcludes` + `buildVisibilityClause` + the exported `escapeLikePattern`.
- `src/core/link-ranking-policy.ts` — the link-type ranking participation policy: `NON_RANKING_LINK_TYPES` (`amends`), `linkTypeRanks`, and `rankingLinkSql(alias)`, the inlined predicate every link-counting ranking reader applies (read-enrichment backlinks, adjacency, fanout, chain hops and degree; enrich inbound order; entity-resolution connection count; generated-page demotion; entity anchoring). Reference reads (graph ops, cards, `backlink_count`, referrers) keep these edges. Tests: `test/amends-link.test.ts` (+ Postgres twin), one invariance row per consumer.
diff --git a/docs/architecture/key-files/core-services-2-continued.md b/docs/architecture/key-files/core-services-2-continued.md
index c80fcbaf7d..c7107924b2 100644
--- a/docs/architecture/key-files/core-services-2-continued.md
+++ b/docs/architecture/key-files/core-services-2-continued.md
@@ -6,12 +6,12 @@ The page-state concurrency foundation under `src/core/page-state/` and the share
- `src/core/ops/get-page-slice.ts` — get_page `section` / `grep` / `context` (#7180), params `GET_PAGE_SLICE_PARAMS` (full surface only; the get_page description names them on starter, and dispatch accepts them there). `ops/get-page-body.ts` routes a slice here: `parseSliceRequest` makes it a window-mode request carrying `slice`, and `readPartialBody`, after authorization and the quarantine `body_omitted` check, passes its read document's `stream()` (the inline bytes, or `SegmentedDocument.stream()` one segment at a time) with the window's metadata and size. Both scan the read projection line by line, so a segmented page is never held whole (memory: the returned text, one segment, at most `context` paragraphs; a line over 1 MiB is scanned in pieces). `readSection`: the first ATX heading (`HEADING`, outside `FENCE` blocks, as the outline sees them) whose whitespace-folded text equals the request case-insensitively (a leading `##` is ignored), down to the next heading of the same or higher level or the timeline separator; returns `section {heading, level, heading_path, line_start, line_end, text}` and stops reading at the section's end; a miss is `invalid_params` naming up to 20 headings with `detail.total_headings` and an outline fix. `readGrep`: blank-line paragraphs with a line containing the literal term (case-insensitive, no regex), `context` paragraphs either side (default 1, clamped to 5), windows that overlap or touch merged; `grep {term, context, total_matches, returned_matches, omitted_matches, blocks[{line_start, line_end, match_starts, text}]}`. Caps: 50 matching paragraphs and `SLICE_MAX_CHARS` (100,000), fewer when the egress cap is smaller (`sliceCharBudget`); `truncated` says a cap was hit, and a cut section adds `omitted_chars` and a `next` window read. Line numbers are the read view's, the ones `body: 'window'` `lines` reads back. A slice with a selector, include_content, content_only, `view: markdown`, a body other than auto/window, or known_as_of is `invalid_params`. Every get_page shape (whole, content_only, partial, slice) carries `chars`, the characters of page text it holds (`projectGetPage`, `readPartialBody`). Test: `test/get-page-slices.test.ts` (both engines).
-- `src/core/page-state/` — Canonical page concurrency foundation shared by both engines. `schema.ts` (migration 150) adds source incarnation UUIDs, logical page revisions, text-projection revisions, and persistent guards for absent pages. `guards.ts` locks sorted sources then page identities in an existing transaction; a transaction or savepoint never re-acquires a key it holds, which covers absent pages because the only pages INSERT runs after `putPage` takes the same guard (`test/e2e/page-guards-postgres.test.ts`); callers lock sources before authority/request/fact rows. `snapshot.ts` computes the normalized body/timeline fingerprints only for a page with fact withdrawals, page-subject purges or a source '*' purge marker (they feed only the withdrawal overlay), as `page-snapshot-batch.ts` does. `snapshot.ts` reads page/tags/withdrawals in one MVCC statement and applies withdrawals using database-normalized text. `transactions.ts` makes helper transaction APIs compose through savepoints. `versions.ts` records the complete canonical snapshot; `rename-alias.ts` records `slug_aliases(old -> new)` inside both engines' `updateSlug` transaction, repoints aliases that named the old slug and drops one spelled like the new slug, and keeps every move (and any alias it drops) in `slug_rename_history` (v285), which only the folder-rule lineage test reads; `moveSlugBindings` moves the page's facts (`entity_slug`, `source_markdown_slug`) and `page_aliases` in the same transaction, letting a purged page's leftovers at the new slug yield (`test/rename-slug-alias.test.ts`); `movePageIdReferences` (#5431) moves the page-id-keyed rows of a stale rename row onto the surviving row (`links` from/to/origin, `timeline_entries` page and event, `page_versions`), dropping stale-side duplicates on the `links` composite key and the timeline dedup indexes (`test/rename-slug-alias.test.ts`); `materialize.ts` persists only verified canonical overlay bytes without another logical revision. `types.ts` defines revision preconditions: no revision is create-only (an existing page refuses `revision_required`, #5385, its own code since the omission is the caller's and not a race), a stale revision or a page another writer created where an internal writer asserted none (`expectedRevision: null`) is `revision_conflict`, explicit force overwrites. Plain legacy `putPage` calls keep compatibility until their owning operation adopts the managed write protocol. `projection-schema.ts` (migration 153) invalidates unverified prior projections once and queues durable work; `projections.ts` rebuilds Markdown without a worktree/provider and installs only matching canonical revisions. Embedding installs also compare exact chunk identities/text. Source importers seal code/image projections; all read arms hide unsealed projections.
+- `src/core/page-state/` — Canonical page concurrency foundation shared by both engines. `schema.ts` (migration 150) adds source incarnation UUIDs, logical page revisions, text-projection revisions, and persistent guards for absent pages. `guards.ts` locks sorted sources then page identities in an existing transaction; a transaction or savepoint never re-acquires a key it holds, which covers absent pages because the only pages INSERT runs after `putPage` takes the same guard (`test/e2e/page-guards-postgres.test.ts`); callers lock sources before authority/request/fact rows. `snapshot.ts` computes the normalized body/timeline fingerprints only for a page with fact withdrawals, page-subject purges or a source '*' purge marker (they feed only the withdrawal overlay), as `page-snapshot-batch.ts` does. `snapshot.ts` reads page/tags/withdrawals in one MVCC statement and applies withdrawals using database-normalized text. `transactions.ts` makes helper transaction APIs compose through savepoints. `versions.ts` records the complete canonical snapshot; `rename-alias.ts` records `slug_aliases(old -> new)` inside both engines' `updateSlug` transaction, repoints aliases that named the old slug and drops one spelled like the new slug, and keeps every move (and any alias it drops) in `slug_rename_history` (v285), which only the folder-rule lineage test reads; `moveSlugBindings` moves the page's facts (`entity_slug`, `source_markdown_slug`) and `page_aliases` in the same transaction, letting a purged page's leftovers at the new slug yield (`test/rename-slug-alias.test.ts`); `movePageIdReferences` (#5431) moves the page-id-keyed rows of a stale rename row onto the surviving row (`links` from/to/origin, `timeline_entries` page and event, `page_versions`), dropping stale-side duplicates on the `links` composite key and the timeline dedup indexes (`test/rename-slug-alias.test.ts`); `materialize.ts` persists only verified canonical overlay bytes without another logical revision. `types.ts` defines revision preconditions: no revision is create-only (an existing page refuses `revision_required`, #5385, its own code since the omission is the caller's and not a race), a stale revision or a page another writer created where an internal writer asserted none (`expectedRevision: null`) is `revision_conflict`, explicit force overwrites. Plain legacy `putPage` calls keep compatibility until their owning operation adopts the managed write protocol. `projection-schema.ts` (migration 153) invalidates unverified prior projections once and queues durable work, but a Markdown or code page at or after the safe-chunk fence with no facts or takes fence, private-key text or quarantine marker keeps its old index in search until rebuilt (`servedTextProjectionFilter`); `projections.ts` rebuilds Markdown without a worktree/provider and installs only matching canonical revisions. Embedding installs also compare exact chunk identities/text. Source importers seal code/image projections; all read arms hide unsealed projections.
- `src/core/page-state/projections.ts` — `sealPageTextProjection` and `sealImportedPage` write the page keyword vector through `pageSearchVectorSql` (`fts-language.ts`: title weight A, sanitized timeline weight C, both through `gbrain_fts_input`), the same expression `search-vector-backfill.ts` uses. Captures canonical revision, source incarnation, page identity, the complete raw chunk inventory and resolved indexing inputs before preparing derived work. Embedding reads require a current text seal; healing/rebuild capture can explicitly include unsealed pages. Projection installation compares its originating seal, chunk identities/text/source/completion state, chunker version and indexing context (including stored corpus generation) under the page guard before replacing any row. A same-revision re-chunk may replace the snapshot it actually captured; a delayed older installation cannot discard newer chunks or vectors. Rebuild also rechecks its queued job before capture so a completed job from an old listing stays completed. Chunkless healing sanitizes complete body/timeline fences before splitting, preserving withdrawn history in direct page reads while excluding it from retrieval. Tests: `page-projection-origin.test.ts`, `page-projection-vector-preservation.test.ts` (PGLite and isolated Postgres), `embed-stale-chunkless-pages.serial.test.ts` (public healing entry). Shared `markdown-chunks.ts` and `code-chunks.ts` preserve fenced code and source-language metadata. A preserving rebuild keeps a vector only when its `content_chunks.embedding_input_hash` (`src/core/embedding-input-hash.ts`) equals one recomputed from the current page, or, with no recorded hash, when the page is non-contextual and the legacy model/text-hash checks pass; a contextual chunk without provenance is nulled once. `installPageEmbeddings` records the hash in the vector's UPDATE (the plain re-embed tier by default, or the caller's `built` tier and corpus generation), and returns `{ installed, complete }`: it matches each offered vector to the page's current chunk by source and exact text (not index) and writes it only when its input hash for the captured context is one the current context accepts, in batched `jsonb_to_recordset` UPDATEs, so a page that changed under the provider call keeps the vectors of its unchanged chunks; the embedding signature is stamped only when `complete` (same chunk sequence, nothing dropped). A preserving rebuild likewise keys kept vectors by chunk content, so a top insert shifts vectors with their chunks (tests: `page-embedding-content-validity.test.ts`). Also `stampEmbeddingInputs` does the same for full imports. `resealSafeChunks` re-seals a page below the safe-chunk fence from its unchanged body with no canonical write; `projectionBelowSafeFence` is its predicate. Tests add `projection-embedding-input-hash.test.ts` and `safe-chunk-reseal.test.ts`. It replaces outgoing code edges without deleting incoming edges to retained chunks and clears backfill watermarks. Failed jobs rotate with a retained diagnostic; successful drain refreshes planner statistics. `rebuildPendingPageProjections(engine, limit, { deadlineMs, now, notAfter, retryCooldown, onFailure })` (#5401) checks an injectable clock between pages, bounds a run to rows queued at or before its start (`notAfter`, cast from text in SQL so microseconds survive the driver), applies the 30 s `rebuild_failed` cooldown for the resident, and treats a lock or snapshot error as that page's failure instead of aborting the batch; `projectionBacklog` returns counts only (pending, failed, oldest age) for the drain command and the resident's `projection_status` IPC reply. A same-revision conflict throws `PageProjectionConflictError` naming the fields that changed (`text_projection_revision`, `chunk_digest`, `indexing_context`), the slug and source; `retryProjectionConflict(attempt)` re-runs an attempt that captures a fresh snapshot at most three more times with jitter, then rethrows (`page_projection_conflict` on the wire). Tests: `page-projection-origin.test.ts` (resident rebuild race, two-writer progress, exhausted-retry CLI/JSON output).
- `src/core/page-state/chunk-moves.ts` — #7172: `CHUNK_VECTOR_WRITE_BUDGET` (5), the most new `content_chunks` row versions carrying a vector (inserts, and non-HOT updates of `chunk_index`, text or vector; each adds an HNSW entry) one transaction of a managed chunk replacement makes, with its derivation from the publication's 5 s statement bound (a version without a vector is not counted); `planMoveWaves` plans the in-place renumbering of kept rows within a budget (one UPDATE per wave, a cycle parks one row at `-1 - target`), `applyMoveWaves` runs it. `import-file.ts` (with `boundedChunkWrites`, set by page-prepare, import-prepare and sync-prepare) commits a page whose replacement exceeds the budget with its projection pending (queue reason `chunk_replacement_deferred`), and `projections.ts` `continueDeferredProjection` finishes it after commit in steps of at most the budget; `rebuildPendingPageProjections` resumes an interrupted one.
- `src/core/page-state/projection-class.ts` — #6803: `projectionClassSql(p, s)`, the one CASE that puts each queued page in exactly one class (`archived`, `text` for Markdown/code and segmented pages, `protected` for a protected fence, `sealable` per `sealableImageSql`, else `reimport`), shared by `projectionBacklog`, the drain probe, search readiness and `embed --stale --images`. `sealableImageSql`: an image page at or after the safe-chunk fence whose one chunk (index 0, `image_asset`) carries the page's own text.
-- `src/core/page-state/image-seal.ts` — #6803: `sealImageProjections` restores the importer's seal on `sealable` images in groups of 500: a primary-key window of queue rows probed by each page's unique key (a statistics-less PGLite planner otherwise probes through `pages_dedup_idx`), then under the page guard a revision compare-and-set with the class rechecked; no chunk write, the visual vector untouched, a text vector whose recorded input no longer matches nulled for `embed --stale`. `IMAGE_SEAL_PAGES_PER_SECOND` is the measured rate (its comment names the run); `projectionDrainLimit(textPending)` sizes doctor's `--limit` to half `AGENT_TOOL_LIMIT_MS` at that rate, or at the measured `TEXT_REBUILD_PAGES_PER_SECOND` while queued text waits (the drain takes text first).
+- `src/core/page-state/image-seal.ts` — #6803: `sealImageProjections` restores the importer's seal on `sealable` images in groups of 500: a primary-key window of queue rows probed by each page's unique key (a statistics-less PGLite planner otherwise probes through `pages_dedup_idx`), then under the page guard a revision compare-and-set with the class rechecked; no chunk write, the visual vector untouched, a text vector whose recorded input no longer matches nulled for `embed --stale`. `IMAGE_SEAL_PAGES_PER_SECOND` is the measured rate (its comment names the run); `projectionDrainLimit(textPending)` sizes doctor's `--limit` to half `AGENT_TOOL_LIMIT_MS` at that rate, or at the measured `TEXT_REBUILD_PAGES_PER_SECOND` (`rebuild-estimate.ts`; it also times v153's notice) while queued text waits (the drain takes text first).
- `src/core/page-state/projection-window.ts` — #6803: the resident's bounded queue read: per pass at most `RESIDENT_PROBE_BUDGET` (500) rows, the 100 newest by `updated_at` plus a primary-key window from a per-engine cursor that starts at a random key and wraps; a negative cache (100k keys) skips image rows the resident never builds. The consumer's idle probe walks the same window and advances the same cursor. Tests: `projection-resident-window.test.ts`.
- `src/core/page-state/projection-status.ts` — #6803: the resident's cached `projectionBacklog` for the `projection_status` IPC op, refreshed when a cheap signal (queue count, max `updated_at`, source archive flags) moves, checked every 60 s and at most 10 minutes old.
- `src/core/page-state/projection-upgrade-notice.ts` — #6803: `gbrain post-upgrade`'s read-only `[AGENT]` block while `sealable` or `reimport` image pages wait, naming `gbrain projections drain --dry-run` with an estimate at the measured seal rate.
diff --git a/docs/guides/concurrent-writes.md b/docs/guides/concurrent-writes.md
index 5c28e1b268..605e477ac8 100644
--- a/docs/guides/concurrent-writes.md
+++ b/docs/guides/concurrent-writes.md
@@ -42,7 +42,8 @@ Separate calls can observe different committed revisions. Search ranking,
embeddings, and direct filesystem reads are outside this snapshot guarantee.
Search reports `projection_pending` while visible canonical revisions still
-need a derived-text rebuild, including when other pages already produce hits.
+need a derived-text rebuild, including when other pages already produce hits
+(a page migration 153 left serving its pre-upgrade index does not count).
The resident owner rebuilds queued Markdown and code using revision-checked
snapshots. `gbrain reindex-code --force --no-embed` repairs code metadata through
that owner without changing canonical files or revisions. Exact compatible
diff --git a/docs/guides/repair.md b/docs/guides/repair.md
index c8e67f1a45..8ab18d6a9d 100644
--- a/docs/guides/repair.md
+++ b/docs/guides/repair.md
@@ -644,7 +644,12 @@ Verify with `gbrain doctor --only persistence_capacity --json`: the
After an upgrade that re-queues text projections (for example the 0.51
protocol activation, which also queued every image page), search and code
-results stay incomplete until every queued page is sealed. Every queued row
+results stay incomplete until every queued page is sealed. That activation
+(migration 153) keeps a Markdown or code page indexed at or after the
+safe-chunk fence in search from its old index until its own rebuild, when the
+page has no facts or takes fence, no private-key text and no quarantine marker
+and has not changed since; every other queued page stays out of search until
+it is sealed. Every queued row
falls in exactly one class, and doctor `text_projection_readiness` counts each
one (counts only, never page names):
diff --git a/scripts/check-engine-sql-dynamic.ts b/scripts/check-engine-sql-dynamic.ts
index 9770f8e4a4..626c7044cd 100644
--- a/scripts/check-engine-sql-dynamic.ts
+++ b/scripts/check-engine-sql-dynamic.ts
@@ -67,6 +67,7 @@ const VETTED_BUILDERS: Record<string, string> = {
currentCodeEdgeFilter: 'src/core/code-intel/read-scope.ts: constant current-edge predicate over a caller alias',
buildCJKKeywordSql: 'src/core/search/cjk-keyword-sql.ts: binds the query as params; shared by both engines',
currentTextProjectionFilter: 'src/core/search/safe-chunks.ts: constant projection predicate over a caller alias',
+ servedTextProjectionFilter: 'src/core/search/safe-chunks.ts: constant projection predicate over a caller alias, ORed with the constant v153 served-index marker (#9192)',
chunkVisibleFilter: 'src/core/search/safe-chunks.ts: constant projection + live body generation predicate over caller page/chunk aliases (Wave B)',
chunkGenerationFilter: 'src/core/search/safe-chunks.ts: constant live body generation predicate over caller page/chunk aliases (Wave B)',
chunkManifestJoin: 'src/core/search/safe-chunks.ts: constant LEFT JOIN onto the live page_body_manifest row over caller aliases (Wave B)',
diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsv
index 9446700b61..11e1829ba8 100644
--- a/scripts/page-body-readers-baseline.tsv
+++ b/scripts/page-body-readers-baseline.tsv
@@ -411,6 +411,7 @@ src/core/page-state/import-guard.ts sameContentAnyKeyOrder field META
src/core/page-state/materialize.ts materializePageSnapshot call:contentHash META inline pages only
src/core/page-state/materialize.ts materializePageSnapshot field SEGW assertInlineBody before the UPDATE
src/core/page-state/materialize.ts materializePageSnapshot sql SEGW assertInlineBody before the UPDATE
+src/core/page-state/projection-schema.ts PAGE_PROJECTION_ACTIVATION_SQL sql NONE not a live reader: v153 runs once at activation, over inline pages only (segmented pages did not exist); the regex scans for fence and private-key markers (#9192)
src/core/page-state/projection-schema.ts PAGE_SEARCH_VECTOR_TRIGGER_SQL sql STREAM trigger DDL: the search vector reads timeline for inline rows and timeline_lead for segmented rows (lane 1a)
src/core/page-state/projections.ts preparePageProjection call:prepareMarkdownChunks STREAM
src/core/page-state/projections.ts readGuardedProjectionSnapshot field STREAM
diff --git a/src/commands/doctor/checks/projection-readiness.ts b/src/commands/doctor/checks/projection-readiness.ts
index 26ef7db35f..5b83e8c21c 100644
--- a/src/commands/doctor/checks/projection-readiness.ts
+++ b/src/commands/doctor/checks/projection-readiness.ts
@@ -101,7 +101,8 @@ export async function checkProjectionReadiness(engine: BrainEngine, scope: PageR
const heldText = held.length ? `${held.length} page(s) holding a private key failed their credential-safe re-chunk and stay withheld from search${opts.remote ? '' : ` (${held.slice(0, 3)
.map(hold => `${hold.source_id}:${hold.slug} ${hold.code}`).join(', ')}${held.length > 3 ? ', ...' : ''})`}; retry them with ${[...new Set(held.map(hold => `\`${hold.fix.argv!.join(' ')}\``))].join(', ')}.` : '';
const text = backlog.pending || (work === 0 && readiness.status === 'projection_pending')
- ? `Visible pages have missing or stale text projections${backlog.pending ? ` (${backlog.pending} queued Markdown/code page(s)${backlog.failed ? `, ${backlog.failed} last failed` : ''})` : ''}; search and code results may be incomplete. ${opts.resident
+ ? `Visible pages have missing or stale text projections${backlog.pending ? ` (${backlog.pending} queued Markdown/code page(s)${backlog.failed ? `, ${backlog.failed} last failed` : ''})` : ''}; ${readiness.status === 'projection_pending'
+ ? 'search and code results may be incomplete.' : 'search keeps answering from the index these pages had before the upgrade until each is rebuilt.'} ${opts.resident
? 'The resident gbrain process that holds this PGLite brain is rebuilding queued Markdown and code projections (up to 100 pages per pass, fewer while writes wait). On the brain host, `gbrain doctor --only text_projection_readiness --json` shows the counts and how to drain faster.'
: 'Run `gbrain projections drain` to rebuild queued Markdown and code projections now; it is safe while `gbrain serve` runs on Postgres.'}${segmentedNote} ${CODE_REPAIR}`
: '';
diff --git a/src/core/chunkers/symbol-resolver.ts b/src/core/chunkers/symbol-resolver.ts
index d3fa007e01..c4402384b5 100644
--- a/src/core/chunkers/symbol-resolver.ts
+++ b/src/core/chunkers/symbol-resolver.ts
@@ -55,7 +55,7 @@
*/
import type { BrainEngine } from '../engine.ts';
-import { chunkVisibleFilter } from '../search/safe-chunks.ts';
+import { chunkVisibleFilter, currentTextProjectionFilter } from '../search/safe-chunks.ts';
/**
* Bump this ISO timestamp whenever the extractor or resolver shape
@@ -153,7 +153,7 @@ export async function resolveSymbolEdgesIncremental(
JOIN pages p ON p.id = cc.page_id
JOIN sources s ON s.id = p.source_id
WHERE p.source_id = $1 AND p.deleted_at IS NULL AND NOT s.archived
- AND ${chunkVisibleFilter('p', 'cc')}
+ AND ${chunkVisibleFilter('p', 'cc', currentTextProjectionFilter)}
AND (cc.edges_backfilled_at IS NULL
OR cc.edges_backfilled_at < $2::timestamptz)
AND ${OWNS_EDGE}
@@ -168,7 +168,7 @@ export async function resolveSymbolEdgesIncremental(
`SELECT cc.id, cc.page_id, p.slug FROM content_chunks cc
JOIN pages p ON p.id=cc.page_id JOIN sources s ON s.id=p.source_id
WHERE cc.id=ANY($1::int[]) AND p.source_id=$2 AND p.deleted_at IS NULL AND NOT s.archived
- AND ${chunkVisibleFilter('p', 'cc')}
+ AND ${chunkVisibleFilter('p', 'cc', currentTextProjectionFilter)}
AND (cc.edges_backfilled_at IS NULL OR cc.edges_backfilled_at < $3::timestamptz)
AND ${OWNS_EDGE}
ORDER BY cc.id`, [chunks.map(chunk => chunk.id), opts.sourceId, EDGE_EXTRACTOR_VERSION_TS],
diff --git a/src/core/code-intel/read-scope.ts b/src/core/code-intel/read-scope.ts
index 2efc1cad7e..b15c9de14f 100644
--- a/src/core/code-intel/read-scope.ts
+++ b/src/core/code-intel/read-scope.ts
@@ -1,6 +1,6 @@
import type { PageReadScope } from '../types.ts';
import { pageReadFilter } from '../search/read-policy-sql.ts';
-import { currentTextProjectionFilter, requiresSafeChunks, safeChunksFilter } from '../search/safe-chunks.ts';
+import { requiresSafeChunks, safeChunksFilter, servedTextProjectionFilter } from '../search/safe-chunks.ts';
export interface CodeReadScope extends PageReadScope {
allSources?: boolean;
@@ -11,7 +11,7 @@ export function codeReadFilter(params: unknown[], scope: CodeReadScope): string
return [
pageReadFilter('p', policy, params, true),
scope.sourceIds?.length === 0 ? 'FALSE' : 'TRUE',
- currentTextProjectionFilter('p'),
+ servedTextProjectionFilter('p'),
...(requiresSafeChunks(scope) ? [safeChunksFilter('p')] : []),
].join(' AND ');
}
diff --git a/src/core/embed-stale.ts b/src/core/embed-stale.ts
index 42196e9a76..cf8b14e08f 100644
--- a/src/core/embed-stale.ts
+++ b/src/core/embed-stale.ts
@@ -37,7 +37,7 @@ import { AbortError } from './abort-check.ts';
import { contentDriftSweepLine, runContentDriftSweep, type ContentDriftSweepStatus } from './content-drift-sweep.ts';
import { meterPageBatch, spendGrant } from './page-embedding-budget.ts';
import { acquirePageEmbeddingLease, stillPendingChunks } from './page-embedding-lease.ts';
-import { chunkGenerationFilter, chunkVisibleFilter } from './search/safe-chunks.ts';
+import { chunkGenerationFilter, chunkVisibleFilter, currentTextProjectionFilter } from './search/safe-chunks.ts';
import { embedSegmentedPage, embedSegmentedRows, readSegmentedEmbedPage, type SegmentedEmbedOpts } from './page-body/segmented-embedding.ts';
/**
@@ -201,7 +201,7 @@ export async function stampIfPageProvenanceComplete(
`SELECT count(*) > 0
AND bool_and(${currentSpaceChunkPredicate(colId, 1, 4)}) AS complete
FROM content_chunks cc JOIN pages p ON p.id = cc.page_id
- WHERE p.slug = $2 AND p.source_id = $3 AND ${chunkVisibleFilter('p', 'cc')}
+ WHERE p.slug = $2 AND p.source_id = $3 AND ${chunkVisibleFilter('p', 'cc', currentTextProjectionFilter)}
AND EXISTS (SELECT 1 FROM sources s WHERE s.id=p.source_id AND NOT s.archived)`,
[model, slug, sourceId, dims],
);
diff --git a/src/core/engine-sql/chunks.ts b/src/core/engine-sql/chunks.ts
index 1ce93ebb97..ee1f00f4fb 100644
--- a/src/core/engine-sql/chunks.ts
+++ b/src/core/engine-sql/chunks.ts
@@ -33,7 +33,7 @@ import {
COLUMN_NAME_REGEX,
EmbeddingColumnNotRegisteredError,
} from '../search/embedding-column.ts';
-import { chunkDocumentOrder, chunkGenerationFilter, chunkManifestJoin, chunkVisibleFilter, chunkWriteInvalidations, currentTextProjectionFilter, movableChunkSql, requiresSafeChunks, safeChunksFilter } from '../search/safe-chunks.ts';
+import { chunkDocumentOrder, chunkGenerationFilter, chunkManifestJoin, chunkVisibleFilter, chunkWriteInvalidations, movableChunkSql, requiresSafeChunks, safeChunksFilter, servedTextProjectionFilter } from '../search/safe-chunks.ts';
import { privatePagesFilterForCaller } from '../search/private-visibility.ts';
import { currentSpaceChunkPredicate, splitEmbeddingSignature, lockEmbeddingSources } from '../embedding-invalidation.ts';
import { EMBED_SKIP_FILTER_FRAGMENT } from '../embed-skip.ts';
@@ -1163,7 +1163,7 @@ export async function getChunkWindows(exec: ScopedRead, requests: ChunkWindowReq
FROM pages p
JOIN sources s ON s.id = p.source_id
WHERE p.id = ANY(${[...new Set(requests.map(r => r.page_id))]}::int[]) ${scopeSql}
- AND p.deleted_at IS NULL AND ${trustedSql(currentTextProjectionFilter('p'))} AND NOT s.archived
+ AND p.deleted_at IS NULL AND ${trustedSql(servedTextProjectionFilter('p'))} AND NOT s.archived
AND ${trustedSql(QUARANTINE_FILTER_FRAGMENT)} ${privateSql} ${safeSql}
), chunk_rows AS (
SELECT cc.page_id, cc.id AS chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.projected_byte_start, cc.projected_byte_end, cc.occurrence_id, a.prio
diff --git a/src/core/page-state/image-seal.ts b/src/core/page-state/image-seal.ts
index 13989f1ae9..9269f30d73 100644
--- a/src/core/page-state/image-seal.ts
+++ b/src/core/page-state/image-seal.ts
@@ -6,6 +6,7 @@
* moved since. Explicit only: the resident never seals images.
*/
import type { BrainEngine } from '../engine.ts';
+import { TEXT_REBUILD_PAGES_PER_SECOND } from './rebuild-estimate.ts';
import { quoteIdentifier } from '../search/embedding-column.ts';
import { chunkGenerationFilter, chunkGenerationProbe, movableChunkSql } from '../search/safe-chunks.ts';
import { sealableImageSql } from './projection-class.ts';
@@ -23,12 +24,7 @@ import { AGENT_TOOL_LIMIT_MS } from '../agent-output.ts';
*/
export const IMAGE_SEAL_PAGES_PER_SECOND = 1500;
-/**
- * Measured Markdown rebuild throughput of the same drain (no embedding): 2,000
- * queued notes of about 3 KB, 117 pages/s on PGLite and 87 pages/s on local
- * Postgres 16, same machine and day; the slower, rounded down.
- */
-export const TEXT_REBUILD_PAGES_PER_SECOND = 80;
+export { TEXT_REBUILD_PAGES_PER_SECOND } from './rebuild-estimate.ts';
/**
* Pages one drain run takes in doctor's fix: half the agent tool limit at the
diff --git a/src/core/page-state/projection-schema.ts b/src/core/page-state/projection-schema.ts
index ffee981e7e..f24d18f436 100644
--- a/src/core/page-state/projection-schema.ts
+++ b/src/core/page-state/projection-schema.ts
@@ -1,4 +1,5 @@
import { requireMaintainerPoliciesSql } from '../derived-state-maintainers.ts';
+import { ACTIVATION_SERVED_REVISION, protectedTextFilter, SAFE_FENCE_CHUNKER_VERSION } from '../search/safe-chunks.ts';
/**
* The page search-vector trigger fires on the columns its function reads:
@@ -62,9 +63,22 @@ export const PAGE_PROJECTION_SCHEMA_STATEMENTS = [
] as const;
export const PAGE_PROJECTION_SCHEMA_SQL = PAGE_PROJECTION_SCHEMA_STATEMENTS.join(';\n') + ';\n';
-/** Run once at protocol activation, never on schema replay: old indexes are unverified. */
+/**
+ * Run once at protocol activation, never on schema replay: old indexes are unverified, so every page is queued for a
+ * rebuild. #9192: a Markdown or code page indexed at or after the safe-chunk fence keeps serving that index until its
+ * rebuild (ACTIVATION_SERVED_REVISION) when nothing the rebuild's projection removes can be in it: no facts or takes
+ * fence (withdrawn, held, purged and lower-tier rows), no private-key text (credential-safe projection) and no
+ * quarantine marker. Every other page waits for the drain. A page written before v150 gets its first revision in its
+ * own statement, whose trigger clears text_projection_revision.
+ */
export const PAGE_PROJECTION_ACTIVATION_SQL = `
- UPDATE pages SET text_projection_revision=NULL,embedding_signature=NULL;
+ UPDATE pages SET knowledge_revision=gen_random_uuid() WHERE knowledge_revision IS NULL;
+ UPDATE pages SET text_projection_revision=CASE WHEN deleted_at IS NULL AND page_kind IN ('markdown','code')
+ AND chunker_version >= ${SAFE_FENCE_CHUNKER_VERSION}
+ AND NOT (${protectedTextFilter('compiled_truth')} OR ${protectedTextFilter('timeline')})
+ AND strpos(compiled_truth || COALESCE(timeline,''),'PRIVATE KEY-----') = 0
+ AND NOT (COALESCE(frontmatter,'{}'::jsonb) ? 'quarantine')
+ THEN '${ACTIVATION_SERVED_REVISION}'::uuid END,embedding_signature=NULL;
INSERT INTO page_projection_jobs(source_incarnation,slug,revision,reason)
SELECT s.incarnation,p.slug,p.knowledge_revision,'protocol_activation'
FROM pages p JOIN sources s ON s.id=p.source_id WHERE p.deleted_at IS NULL
diff --git a/src/core/page-state/rebuild-estimate.ts b/src/core/page-state/rebuild-estimate.ts
new file mode 100644
index 0000000000..c576246876
--- /dev/null
+++ b/src/core/page-state/rebuild-estimate.ts
@@ -0,0 +1,31 @@
+import type { BrainEngine } from '../engine.ts';
+
+/**
+ * Measured Markdown rebuild throughput of the same drain (no embedding): 2,000
+ * queued notes of about 3 KB, 117 pages/s on PGLite and 87 pages/s on local
+ * Postgres 16, same machine and day; the slower, rounded down.
+ */
+export const TEXT_REBUILD_PAGES_PER_SECOND = 80;
+
+/**
+ * #9192: database round trips one queued Markdown page's rebuild makes, fitted from a 50,000-page drain on Postgres 16
+ * at a 0.1 ms and a 17.3 ms round trip (85 and 1.6 pages/s): the time behind a remote pooler is round trips, not work.
+ */
+export const TEXT_REBUILD_ROUND_TRIPS_PER_PAGE = 36;
+
+/** How long a text rebuild of `pages` takes at a measured round trip of `rttMs`, in words. */
+export function textRebuildDuration(pages: number, rttMs: number): string {
+ const minutes = Math.max(1, Math.ceil(pages * (1 / TEXT_REBUILD_PAGES_PER_SECOND + TEXT_REBUILD_ROUND_TRIPS_PER_PAGE * rttMs / 1000) / 60));
+ return minutes >= 120 ? `${Math.round(minutes / 60)} hour(s)` : `${minutes} minute(s)`;
+}
+
+/** The median of five `SELECT 1` round trips to the engine, in milliseconds. */
+export async function measuredRoundTripMs(engine: Pick<BrainEngine, 'executeRaw'>): Promise<number> {
+ const samples: number[] = [];
+ for (let i = 0; i < 5; i++) {
+ const started = performance.now();
+ await engine.executeRaw('SELECT 1');
+ samples.push(performance.now() - started);
+ }
+ return Math.round(samples.sort((a, b) => a - b)[2]! * 10) / 10;
+}
diff --git a/src/core/persistence/sync-freshness.ts b/src/core/persistence/sync-freshness.ts
index 2303f77482..7887182cbd 100644
--- a/src/core/persistence/sync-freshness.ts
+++ b/src/core/persistence/sync-freshness.ts
@@ -22,7 +22,7 @@ import type { BrainEngine } from '../engine.ts';
import { execFileBounded } from '../bounded-child-exec.ts';
import { gitChildEnv } from '../git-env.ts';
import { SYNC_FRESHNESS_TARGET_KEY } from '../sync-config-keys.ts';
-import { chunkVisibleFilter } from '../search/safe-chunks.ts';
+import { chunkVisibleFilter, currentTextProjectionFilter } from '../search/safe-chunks.ts';
export const FRESHNESS_WINDOW_HOURS = 24;
/** The newest distinct targets dated per source; older ones in the window are left out of the sample. */
@@ -132,7 +132,7 @@ export async function readSyncFreshness(engine: BrainEngine, opts: { sourceIds?:
FROM sample r LEFT JOIN later ON later.source_id=r.source_id AND later.slug=r.slug),
e AS (SELECT s.completed_at, s.committed_at, s.superseded, ch.n, ch.missing, ch.last_at FROM s
LEFT JOIN LATERAL (SELECT count(*) AS n, count(*) FILTER (WHERE cc.embedded_at IS NULL) AS missing, max(cc.embedded_at) AS last_at FROM content_chunks cc JOIN pages p ON p.id=cc.page_id
- WHERE cc.page_id=s.page_id AND ${chunkVisibleFilter('p', 'cc')}) ch ON true),
+ WHERE cc.page_id=s.page_id AND ${chunkVisibleFilter('p', 'cc', currentTextProjectionFilter)}) ch ON true),
l AS (SELECT GREATEST(0, extract(epoch FROM completed_at - committed_at)) AS searchable,
CASE WHEN NOT superseded AND n > 0 AND missing = 0 AND last_at >= completed_at THEN GREATEST(0, extract(epoch FROM last_at - committed_at)) END AS embedded,
(NOT superseded AND n > 0 AND missing > 0) AS pending FROM e)
diff --git a/src/core/schema-migrations/v153-verified-text-projection-activation.ts b/src/core/schema-migrations/v153-verified-text-projection-activation.ts
index 6f2eaf150e..0d772639f3 100644
--- a/src/core/schema-migrations/v153-verified-text-projection-activation.ts
+++ b/src/core/schema-migrations/v153-verified-text-projection-activation.ts
@@ -1,13 +1,22 @@
import type { Migration } from './types.ts';
import { PAGE_PROJECTION_ACTIVATION_SQL, PAGE_PROJECTION_SCHEMA_SQL } from '../page-state/projection-schema.ts';
+import { ACTIVATION_SERVED_REVISION } from '../search/safe-chunks.ts';
+import { measuredRoundTripMs, textRebuildDuration } from '../page-state/rebuild-estimate.ts';
import { migrationNotice } from './helpers.ts';
export const v153: Migration = {
version: 153, name: 'verified_text_projection_activation', idempotent: true, sql: PAGE_PROJECTION_SCHEMA_SQL + PAGE_PROJECTION_ACTIVATION_SQL,
- // #9192: the activation takes every page out of the search index until the drain rebuilds it, so say so once.
+ // #9192: say once what search keeps serving, what waits for the rebuild, and how long it takes on this database.
handler: async engine => {
- const [row] = await engine.executeRaw<{ n: number }>("SELECT count(*)::int AS n FROM page_projection_jobs WHERE reason = 'protocol_activation'");
- const n = Number(row?.n ?? 0);
- if (n > 0) migrationNotice(` [153] ${n} page(s) queued for a search index rebuild; search and query results stay partial and say so until it finishes. Progress: gbrain doctor --only text_projection_readiness\n`);
+ const [row] = await engine.executeRaw<{ n: number; served: number }>(`SELECT count(*)::int AS n,
+ count(*) FILTER (WHERE p.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid)::int AS served
+ FROM page_projection_jobs j JOIN sources s ON s.incarnation = j.source_incarnation JOIN pages p ON p.source_id = s.id AND p.slug = j.slug
+ WHERE j.reason = 'protocol_activation'`);
+ const n = Number(row?.n ?? 0), served = Number(row?.served ?? 0);
+ if (n === 0) return;
+ const rtt = await measuredRoundTripMs(engine);
+ const waiting = n - served ? `; ${n - served} (images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker) stay out of search until rebuilt` : '';
+ migrationNotice(` [153] ${n} page(s) queued for a keyless search index rebuild (no embedding calls). Search keeps answering from the index ${served} of them already had${waiting}. ` +
+ `At the ${rtt} ms round trip measured to this database the rebuild takes about ${textRebuildDuration(n, rtt)}: gbrain projections drain. Progress: gbrain doctor --only text_projection_readiness\n`);
},
};
diff --git a/src/core/search/evidence-delivery.ts b/src/core/search/evidence-delivery.ts
index 0f966df6f9..5ba448822c 100644
--- a/src/core/search/evidence-delivery.ts
+++ b/src/core/search/evidence-delivery.ts
@@ -31,7 +31,7 @@ import { buildSnippetMarker } from './snippet-cap.ts';
import { deliveredQuoteText, deliveredQuoteTitle, fenceTranscriptRows, neutralizeTranscriptRows } from './transcript-quote.ts';
import { stampPageTrust } from '../eligibility/stamp.ts';
import { pageReadFilter } from './read-policy-sql.ts';
-import { chunkGenerationFilter, chunkManifestReads, currentTextProjectionFilter, safeChunksFilter, requiresSafeChunks } from './safe-chunks.ts';
+import { chunkGenerationFilter, chunkManifestReads, safeChunksFilter, requiresSafeChunks, servedTextProjectionFilter } from './safe-chunks.ts';
import { resolveExcludePrivatePages } from './private-visibility.ts';
import { safeSplitIndex } from '../text-safe.ts';
import { loadFenceChunkOverlay, type FenceChunkOverlay } from '../eligibility/fence-overlay.ts';
@@ -416,7 +416,7 @@ export async function selectLongPages(engine: BrainEngine, plan: EvidencePlan |
const rows = await withTimeout(engine.executeRaw<{ id: number; chars: number; bytes: number }>(
`SELECT p.id, char_length(COALESCE(p.compiled_truth, ''))${tl ? ` + ${tl}char_length(p.timeline) END` : ''} AS chars,
octet_length(COALESCE(p.compiled_truth, ''))${tl ? ` + ${tl}octet_length(p.timeline) END` : ''} AS bytes FROM pages p
- WHERE p.id = ANY($1::int[]) AND ${filter} AND ${currentTextProjectionFilter('p')}${requiresSafeChunks(scope) ? ` AND ${safeChunksFilter('p')}` : ''}`,
+ WHERE p.id = ANY($1::int[]) AND ${filter} AND ${servedTextProjectionFilter('p')}${requiresSafeChunks(scope) ? ` AND ${safeChunksFilter('p')}` : ''}`,
params), LONG_PAGE_DEADLINE_MS);
sizes = new Map(rows.map(r => [Number(r.id), { chars: Number(r.chars), bytes: Number(r.bytes) }]));
} catch (e) {
@@ -1657,7 +1657,7 @@ export async function resolveFrozenHits(
JOIN pages p ON p.source_id = h.source_id AND p.slug = h.slug
LEFT JOIN content_chunks cc ON cc.page_id = p.id AND ${chunkGenerationFilter('p', 'cc', await chunkManifestReads(engine))}
AND CASE WHEN h.chunk_id > 0 THEN cc.id = h.chunk_id ELSE cc.chunk_index = 0 END
- WHERE ${filter} AND ${currentTextProjectionFilter('p')} ${safe}`,
+ WHERE ${filter} AND ${servedTextProjectionFilter('p')} ${safe}`,
params,
);
const byOrd = new Map<number, Record<string, unknown>>();
diff --git a/src/core/search/exact-scan.ts b/src/core/search/exact-scan.ts
index 4caa1d3e13..db2dc0891d 100644
--- a/src/core/search/exact-scan.ts
+++ b/src/core/search/exact-scan.ts
@@ -38,7 +38,7 @@ import { pageEligibleSql } from '../eligibility/sql.ts';
import { buildHardExcludeClause } from './sql-ranking.ts';
import { escapeLikePattern } from './sql-ranking.ts';
import { isStatementTimeoutError } from '../retry-matcher.ts';
-import { chunkVisibleFilter, currentTextProjectionFilter, safeChunksFilter, SAFE_FENCE_CHUNKER_VERSION } from './safe-chunks.ts';
+import { chunkVisibleFilter, safeChunksFilter, SAFE_FENCE_CHUNKER_VERSION, servedTextProjectionFilter } from './safe-chunks.ts';
export const EXACT_DEFAULT_LIMIT = 100;
export const EXACT_MAX_LIMIT = 1000;
@@ -160,7 +160,7 @@ export function buildExactBatchStatement(opts: ExactScanOpts, after: number, bat
)
SELECT p.id AS page_id, p.slug, p.source_id, p.title, p.type, pp.matching_chunks, pp.snippets,
CASE WHEN pp.page_id IS NOT NULL THEN NULL ${unsealed}
- WHEN NOT ${currentTextProjectionFilter('p')} THEN 'projection_pending'
+ WHEN NOT ${servedTextProjectionFilter('p')} THEN 'projection_pending'
WHEN NOT EXISTS (SELECT 1 FROM content_chunks vc WHERE vc.page_id = p.id AND ${chunkVisibleFilter('p', 'vc')}) AND ${hasBody} THEN 'unchunked'
ELSE NULL END AS gap
FROM batch b JOIN pages p ON p.id = b.id LEFT JOIN per_page pp ON pp.page_id = b.id
diff --git a/src/core/search/projection-readiness.ts b/src/core/search/projection-readiness.ts
index a7f12b7afc..d7fe23f9fb 100644
--- a/src/core/search/projection-readiness.ts
+++ b/src/core/search/projection-readiness.ts
@@ -1,6 +1,7 @@
import type { BrainEngine } from '../engine.ts';
import type { PageReadScope } from '../types.ts';
import { projectionClassSql } from '../page-state/projection-class.ts';
+import { ACTIVATION_SERVED_REVISION } from './safe-chunks.ts';
import { coverageScopeFilters } from './retrieval-coverage.ts';
export interface ProjectionReadinessScope extends PageReadScope {
@@ -97,6 +98,7 @@ async function probeUncached(
const unsealed = (cond: string) => `(SELECT true FROM pages p JOIN sources s ON s.id = p.source_id
WHERE p.deleted_at IS NULL
AND p.text_projection_revision IS DISTINCT FROM p.knowledge_revision
+ AND p.text_projection_revision IS DISTINCT FROM '${ACTIVATION_SERVED_REVISION}'::uuid
AND ${filters.join(' AND ')} AND ${cond}
ORDER BY p.source_id, p.page_kind, p.slug LIMIT 1) IS NOT NULL`;
const cls = projectionClassSql('p', 's');
diff --git a/src/core/search/retrieval-coverage.ts b/src/core/search/retrieval-coverage.ts
index eaf912f088..63ed1e63b5 100644
--- a/src/core/search/retrieval-coverage.ts
+++ b/src/core/search/retrieval-coverage.ts
@@ -25,6 +25,7 @@ import type { BrainEngine } from '../engine.ts';
import { budgetRowAppliesFilter } from '../page-size-policy.ts';
import { projectionClassSql } from '../page-state/projection-class.ts';
import { pageReadFilter } from './read-policy-sql.ts';
+import { servedTextProjectionFilter } from './safe-chunks.ts';
import type { ProjectionReadinessScope } from './projection-readiness.ts';
/** Counts stop here (one bounded scan per call); a capped count reports `capped`. */
@@ -146,7 +147,7 @@ async function countInScope(engine: Pick<BrainEngine, 'executeRaw'>, scope: Proj
/** The pending pages the readiness probe found (unsealed, needing work) in the same scope. */
export function countPendingProjections(engine: Pick<BrainEngine, 'executeRaw'>, scope: ProjectionReadinessScope, relevance: CoverageRelevance): Promise<ScopedCount | null> {
return countInScope(engine, scope, relevance, '',
- `p.text_projection_revision IS DISTINCT FROM p.knowledge_revision AND ${projectionClassSql('p', 's')} NOT IN ('protected','archived')`, true);
+ `NOT ${servedTextProjectionFilter('p')} AND ${projectionClassSql('p', 's')} NOT IN ('protected','archived')`, true);
}
/** Pages in scope held from embedding: a held or capped budget row that still applies, on a page no `embed_skip` marker keeps out of embedding for good. */
diff --git a/src/core/search/safe-chunks.ts b/src/core/search/safe-chunks.ts
index 09d35729d3..8379765936 100644
--- a/src/core/search/safe-chunks.ts
+++ b/src/core/search/safe-chunks.ts
@@ -45,6 +45,23 @@ export function currentTextProjectionFilter(alias: string): string {
return `(${alias}.text_projection_revision = ${alias}.knowledge_revision) IS TRUE`;
}
+/**
+ * #9192: the `text_projection_revision` v153's activation leaves on a page whose pre-upgrade index it keeps serving
+ * (PAGE_PROJECTION_ACTIVATION_SQL says which). The page stays unsealed (its rebuild is queued); any revision change
+ * clears the column (pages_knowledge_revision trigger), and so does the rebuild. A version-0 UUID: every revision is
+ * gen_random_uuid() (version 4), so the marker never equals one and every strict `= knowledge_revision` read treats
+ * the page as not current.
+ */
+export const ACTIVATION_SERVED_REVISION = '00000000-0000-0000-0000-000000000153';
+
+/**
+ * What search serves: the current projection, or the pre-activation index of a page unchanged since (#9192). The
+ * current-projection term stays its own disjunct so the planner keeps reading pages_text_projection_current_stats.
+ */
+export function servedTextProjectionFilter(alias: string): string {
+ return `(${currentTextProjectionFilter(alias)} OR (${alias}.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid) IS TRUE)`;
+}
+
/**
* Wave B: the chunk row belongs to its page's live body: an inline row
* (`body_generation IS NULL`) of a page that is not segmented, or a segmented
@@ -95,13 +112,14 @@ export async function chunkManifestReads(engine: Pick<BrainEngine, 'kind' | 'exe
}
/**
- * Search-facing chunk visibility: the current text projection and the live body generation. A sealed page holds no
+ * Search-facing chunk visibility: the served text projection and the live body generation; maintenance reads pass
+ * currentTextProjectionFilter. A sealed page holds no
* rows of a retired occurrence (the flip deletes them in its transaction, or leaves the page unsealed until the
* projection job has), so the row predicate needs no manifest probe, whose per-row cost moved vector search off its
* HNSW plan (test/e2e/vector-chunk-statistics-postgres.test.ts).
*/
-export function chunkVisibleFilter(page: string, chunk: string): string {
- return `(${currentTextProjectionFilter(page)}
+export function chunkVisibleFilter(page: string, chunk: string, projection = servedTextProjectionFilter): string {
+ return `(${projection(page)}
AND (${chunk}.body_generation IS NULL OR ${chunk}.body_generation <= ${page}.body_generation))`;
}
diff --git a/src/core/search/sql-ranking.ts b/src/core/search/sql-ranking.ts
index 688241932b..eaf9531196 100644
--- a/src/core/search/sql-ranking.ts
+++ b/src/core/search/sql-ranking.ts
@@ -20,7 +20,7 @@
import { quarantineFilterFragment } from '../quarantine.ts';
import { unverifiedExtractionFragment } from '../extraction-review.ts';
import { privatePagesFilterForCaller } from './private-visibility.ts';
-import { chunkVisibleFilter, currentTextProjectionFilter, requiresSafeChunks, safeChunksFilter } from './safe-chunks.ts';
+import { chunkVisibleFilter, requiresSafeChunks, safeChunksFilter, servedTextProjectionFilter } from './safe-chunks.ts';
import { pageEligibleSql } from '../eligibility/sql.ts';
import type { TrustTier } from '../trust/tier.ts';
@@ -213,7 +213,7 @@ export function buildVisibilityClause(
const pinnedQuestion = ` AND NOT (COALESCE(${pageAlias}.frontmatter, '{}'::jsonb) ? 'pinned_question')`;
const trustClause = opts?.minTrust || opts?.suppressFlagged
? ` AND ${pageEligibleSql(pageAlias, { floor: opts.minTrust, suppressFlagged: opts.suppressFlagged })}` : '';
- const projection = opts?.chunkAlias ? chunkVisibleFilter(pageAlias, opts.chunkAlias) : currentTextProjectionFilter(pageAlias);
+ const projection = opts?.chunkAlias ? chunkVisibleFilter(pageAlias, opts.chunkAlias) : servedTextProjectionFilter(pageAlias);
return `AND ${pageAlias}.deleted_at IS NULL AND ${projection} AND NOT ${sourceAlias}.archived AND ${quarantine}${pinnedQuestion}${privateClause}${chunksClause}${trustClause}`;
}
diff --git a/test/fixtures/fold-hook-noop-6712.json b/test/fixtures/fold-hook-noop-6712.json
index 74269770e5..164355cab4 100644
--- a/test/fixtures/fold-hook-noop-6712.json
+++ b/test/fixtures/fold-hook-noop-6712.json
@@ -87,7 +87,7 @@
"SELECT key,value FROM config WHERE key IN ('search_embedding_column','embedding_columns','embedding_model','embedding_dimensions','contextual_retrieval.mode') ORDER BY key",
"SELECT c.chunk_index, c.embedding_input_hash, p.contextual_retrieval_mode, c.embedding_image IS NULL AND c.embedding_multimodal IS NULL AND c.embedded_text_hash = md5(c.chunk_text) AS text_only, c.embedding IS NULL AND c.embedding_image IS NULL AND c.embedding_multimodal IS NULL AND c.embedding_input_hash IS NULL AND c.embedded_text_hash IS NULL AS unembedded FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE p.source_id = $1 AND p.slug = $2",
"SELECT key, value FROM config WHERE key IN ('search_embedding_column', 'embedding_columns')",
- "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
"INSERT INTO page_versions (page_id,compiled_truth,frontmatter,knowledge_revision,timeline,title,type,tags,is_deleted,source_path,change_reason) VALUES ($1,$2,$3::text::jsonb,$4::uuid,$5,$6,$7,$8::text::jsonb,$9,$10,$11) RETURNING *",
"SELECT id, slug, source_path, deleted_at FROM pages WHERE source_id=$1 AND slug=$2",
"SELECT local_path, config->>'slug_root_mode' AS mode FROM sources WHERE id=$1",
@@ -219,7 +219,7 @@
"SELECT c.chunk_index, c.embedding_input_hash, p.contextual_retrieval_mode, c.embedding_image IS NULL AND c.embedding_multimodal IS NULL AND c.embedded_text_hash = md5(c.chunk_text) AS text_only, c.embedding IS NULL AND c.embedding_image IS NULL AND c.embedding_multimodal IS NULL AND c.embedding_input_hash IS NULL AND c.embedded_text_hash IS NULL AS unembedded FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE p.source_id = $1 AND p.slug = $2",
"SELECT key, value FROM config WHERE key IN ('search_embedding_column', 'embedding_columns')",
"-- chunk-visibility-ok: whether any chunk row of a segmented body exists, no row is returned SELECT has_table_privilege('page_body_manifest', 'SELECT') AS readable, NOT has_table_privilege('page_body_manifest', 'SELECT') AND (EXISTS (SELECT 1 FROM content_chunks WHERE body_generation IS NOT NULL) OR EXISTS (SELECT 1 FROM content_chunks WHERE occurrence_id IS NOT NULL)) AS segmented",
- "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
"INSERT INTO page_versions (page_id,compiled_truth,frontmatter,knowledge_revision,timeline,title,type,tags,is_deleted,source_path,change_reason) VALUES ($1,$2,$3::text::jsonb,$4::uuid,$5,$6,$7,$8::text::jsonb,$9,$10,$11) RETURNING *",
"WITH page AS (SELECT id FROM pages WHERE slug = $1 AND source_id = $2 FOR UPDATE), gone AS ( DELETE FROM content_chunks WHERE page_id = (SELECT id FROM page) RETURNING page_id ) UPDATE pages SET text_projection_revision = NULL WHERE id = (SELECT page_id FROM gone LIMIT 1) AND text_projection_revision IS NOT NULL",
"DELETE FROM page_aliases WHERE source_id = $1 AND slug = $2 AND origin IN ('declared','subject')",
diff --git a/test/fixtures/goldens/migrations/records.json b/test/fixtures/goldens/migrations/records.json
index 60445a18a9..7b364b7b99 100644
--- a/test/fixtures/goldens/migrations/records.json
+++ b/test/fixtures/goldens/migrations/records.json
@@ -8,6 +8,7 @@
100
],
"helpers": {
+ "ACTIVATION_SERVED_REVISION": "4ae2affe24656313c50011f09ed465dd8a9bc48b6d5af255ff39a06070d82e66",
"ALIAS_TABLES": "cddd37c636af42d862c08ca7754a57dfe9d12576b769fa8f74a5f02c803f67a0",
"BELIEF_FACTS_STREAM_INDEX": "9e4491eb9e0b94924a58deea3feff1b315ec0efd25a77a4e6515b362beffbc92",
"BOUNDED_TIMELINE_CHECKPOINT": "385c4da3e28732343148795d6e9a86670813a7a7cfa1e52f8a5bb494680a3c0d",
@@ -75,6 +76,7 @@
"identityIndexHasSource": "ab7e4cad64ebf233f3a884816c114b0c582f4f2166c3bcf19ed8d92bcfcf3807",
"isLockTimeoutError": "52a3720e1ffb74ce381cc3269f78130c38a89c807e308283ec3d557bae2b89b1",
"lockTableForAnalyze": "b6e4f173cffd877d2cfd85af813aa694a6d2c32907eaea613c6879800d3a7b44",
+ "measuredRoundTripMs": "8a51ca0f3fe1b930929b64efe81541160bfc3fedc24f90c2af35dfd74ab1687d",
"migrateAutoThinkToPins": "34267fe3b1cc4ab2850ddcf9ba18ce73d959678233a53de238c85231e6b71470",
"migrateConnectorCheckpoints": "5c13e69a334bfc5275c7d555c4e7b28e034569f086dd4bd70b4b78fa4770dc7a",
"migrateLegacyTokens": "6e4abe3670127e7a495b56ecbf9f5789edd3a7362d45704656c2210a85b27f4c",
@@ -96,6 +98,7 @@
"sqlQueryForEngine": "71fb1ccec1b22cdc1662876dbd8c17ed81e9ab83485c6965cdf5caa4fc4a4ab5",
"swapIdentityIndex": "84ac62b697a16d244b2e0eecc962fb0558a24c2a26123bc7e77f1c7bec7c1066",
"tableRows": "906d2413b13d76a06d5910885e47ed5c544be4ff3a2266e83146715721b24270",
+ "textRebuildDuration": "cffe5e296f8dbdd2552a4993d0e6dbf6ddef5604fef75dc1d327d4172b66211e",
"verifyProjectionStatistics": "0d231b04ecfbc5a4109d6fca9567bf4e8c9e9db6c3fbbb35212e026579ed6f67",
"widenIdentityIndex": "86a6176bd7637d964dbc1d3cd3b1bdfa3ca0161e1ee248c37d7380a1471282e6",
"withSchemaLockTimeout": "95605bd3004955e11e936c7a4aa6889319b401098abf9db18ef3e48da2c3fc59"
@@ -2319,16 +2322,19 @@
"handler": {
"form": "arrow",
"helpers": [
+ "ACTIVATION_SERVED_REVISION",
+ "measuredRoundTripMs",
"migrationNotice",
- "quietMigrationNotices"
+ "quietMigrationNotices",
+ "textRebuildDuration"
],
- "sha256": "4fd47425063a5a78da6ded62f86e56cb1d0587604a99289e4dce3d1834105df6"
+ "sha256": "4db3224e2165b8d654bb19cdc8653840ac363f2d9ac1ea6f910304784ceb4cf2"
},
"idempotent": true,
"name": "verified_text_projection_activation",
"sql": {
- "length": 7430,
- "sha256": "0a1e4fc42f20571946543865c49e15503168a124a309aa7f04f3e0ff2025493f"
+ "length": 7981,
+ "sha256": "e3d42430bc12655abf281a9e75ec2b68bace3e99ae30ff84fe28db8d0f8a33a8"
},
"sqlFor": "absent",
"transaction": "absent",
diff --git a/test/fixtures/goldens/sql-text/chunks.json b/test/fixtures/goldens/sql-text/chunks.json
index 3540969f7c..bafe6ce433 100644
--- a/test/fixtures/goldens/sql-text/chunks.json
+++ b/test/fixtures/goldens/sql-text/chunks.json
@@ -104,8 +104,8 @@
"string",
"string"
],
- "sql": "SELECT COALESCE(sum(t.n), 0) AS total, max(t.max_index) AS max_index, bool_or(t.segmented) AS segmented, max(t.n) AS max_page_chunks FROM ( SELECT count(*) AS n, max(cc.chunk_index) AS max_index, bool_or(cc.body_generation IS NOT NULL) AS segmented FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) GROUP BY cc.page_id ) t",
- "textSha256": "c29af72128a53d14d0d77ce09564344c7df8b84404c24ef22980a0181983359b"
+ "sql": "SELECT COALESCE(sum(t.n), 0) AS total, max(t.max_index) AS max_index, bool_or(t.segmented) AS segmented, max(t.n) AS max_page_chunks FROM ( SELECT count(*) AS n, max(cc.chunk_index) AS max_index, bool_or(cc.body_generation IS NOT NULL) AS segmented FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) GROUP BY cc.page_id ) t",
+ "textSha256": "46f87852b6ca117ac1f5cea96782a63987d8bcd5920da127953ae354d682f622"
},
{
"lane": "pool",
@@ -115,8 +115,8 @@
"number",
"number"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND cc.chunk_index >= $3 AND cc.chunk_index < $4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "dfe772afa8fb395d035e32b1483c97558a7121379ee8818703d5bcdffea2717b"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND cc.chunk_index >= $3 AND cc.chunk_index < $4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "747b2b79b1c4ee0c109af834a60762090033f01fcd70e74bd02a779d735110b5"
}
],
"sourceIds+excludePrivate": [
@@ -138,8 +138,8 @@
"string",
"array<string>"
],
- "sql": "SELECT COALESCE(sum(t.n), 0) AS total, max(t.max_index) AS max_index, bool_or(t.segmented) AS segmented, max(t.n) AS max_page_chunks FROM ( SELECT count(*) AS n, max(cc.chunk_index) AS max_index, bool_or(cc.body_generation IS NOT NULL) AS segmented FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 GROUP BY cc.page_id ) t",
- "textSha256": "a6f19b17a4be59f161dfa7d43355a5cb7b890ec64723623a42df2941bd0b50f8"
+ "sql": "SELECT COALESCE(sum(t.n), 0) AS total, max(t.max_index) AS max_index, bool_or(t.segmented) AS segmented, max(t.n) AS max_page_chunks FROM ( SELECT count(*) AS n, max(cc.chunk_index) AS max_index, bool_or(cc.body_generation IS NOT NULL) AS segmented FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 GROUP BY cc.page_id ) t",
+ "textSha256": "27a4eabe9b6bdd41a15c20ab492883e2681a1e20bbcd70c268d910f5d3f72e92"
},
{
"lane": "pool",
@@ -149,8 +149,8 @@
"number",
"number"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 AND cc.chunk_index >= $3 AND cc.chunk_index < $4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "5f2758a8c4d70bcfaddae4867872af318dfd87b70f70cc5da56e51c1bf2f175a"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 AND cc.chunk_index >= $3 AND cc.chunk_index < $4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "59ef83c64302bf1a508c1e2563b7a2b615787dc7d5b9469594df47c45523d459"
}
]
},
@@ -174,8 +174,8 @@
"array<string>",
"number"
],
- "sql": "WITH req AS ( SELECT * FROM unnest($1::int[], $2::int[], $3::int[], $4::int[]) AS r(page_id, lo, hi, prio) ), auth AS ( SELECT p.id, p.slug, p.source_id, p.type, p.knowledge_revision::text AS revision, p.compiled_truth, p.timeline, p.body_storage, p.body_generation, p.body_storage = 'segmented' AS segmented, (COALESCE(p.chunker_version, 0) >= $5) AS sealed, (SELECT MIN(req.prio) FROM req WHERE req.page_id = p.id) AS prio FROM pages p JOIN sources s ON s.id = p.source_id WHERE p.id = ANY($6::int[]) AND p.deleted_at IS NULL AND (p.text_projection_revision = p.knowledge_revision) IS TRUE AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') ), chunk_rows AS ( SELECT cc.page_id, cc.id AS chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.projected_byte_start, cc.projected_byte_end, cc.occurrence_id, a.prio FROM auth a JOIN content_chunks cc ON cc.page_id = a.id WHERE a.sealed AND ((cc.body_generation IS NULL AND a.body_storage <> 'segmented') OR (cc.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation)))) AND cc.chunk_source = ANY($7::text[]) AND COALESCE(cc.modality, 'text') = 'text' AND EXISTS (SELECT 1 FROM req r WHERE r.page_id = cc.page_id AND cc.chunk_index BETWEEN r.lo AND r.hi) ORDER BY a.prio, cc.page_id, cc.chunk_index LIMIT $8 ) SELECT 'page' AS kind, a.id AS page_id, a.slug, a.source_id, a.type, a.revision, a.sealed, a.segmented, a.prio, (SELECT MAX(c2.chunk_index) FROM content_chunks c2 WHERE c2.page_id = a.id AND ((c2.body_generation IS NULL AND a.body_storage <> 'segmented') OR (c2.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = c2.page_id AND lm.occurrence_id = c2.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation))))) AS max_chunk_index, a.compiled_truth, a.timeline, NULL::int AS chunk_id, NULL::int AS chunk_index, NULL::text AS chunk_text, NULL::text AS chunk_source, NULL::bigint AS projected_byte_start, NULL::bigint AS projected_byte_end, NULL::bigint AS occurrence_id FROM auth a UNION ALL SELECT 'chunk' AS kind, c.page_id, NULL, NULL, NULL, NULL, NULL, NULL, c.prio, NULL, NULL, NULL, c.chunk_id, c.chunk_index, c.chunk_text, c.chunk_source, c.projected_byte_start, c.projected_byte_end, c.occurrence_id FROM chunk_rows c",
- "textSha256": "f82321f0db92ba814b7a8aff075d13147d17d2669cfac4d3fa4cb1f2df59e51d"
+ "sql": "WITH req AS ( SELECT * FROM unnest($1::int[], $2::int[], $3::int[], $4::int[]) AS r(page_id, lo, hi, prio) ), auth AS ( SELECT p.id, p.slug, p.source_id, p.type, p.knowledge_revision::text AS revision, p.compiled_truth, p.timeline, p.body_storage, p.body_generation, p.body_storage = 'segmented' AS segmented, (COALESCE(p.chunker_version, 0) >= $5) AS sealed, (SELECT MIN(req.prio) FROM req WHERE req.page_id = p.id) AS prio FROM pages p JOIN sources s ON s.id = p.source_id WHERE p.id = ANY($6::int[]) AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') ), chunk_rows AS ( SELECT cc.page_id, cc.id AS chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.projected_byte_start, cc.projected_byte_end, cc.occurrence_id, a.prio FROM auth a JOIN content_chunks cc ON cc.page_id = a.id WHERE a.sealed AND ((cc.body_generation IS NULL AND a.body_storage <> 'segmented') OR (cc.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation)))) AND cc.chunk_source = ANY($7::text[]) AND COALESCE(cc.modality, 'text') = 'text' AND EXISTS (SELECT 1 FROM req r WHERE r.page_id = cc.page_id AND cc.chunk_index BETWEEN r.lo AND r.hi) ORDER BY a.prio, cc.page_id, cc.chunk_index LIMIT $8 ) SELECT 'page' AS kind, a.id AS page_id, a.slug, a.source_id, a.type, a.revision, a.sealed, a.segmented, a.prio, (SELECT MAX(c2.chunk_index) FROM content_chunks c2 WHERE c2.page_id = a.id AND ((c2.body_generation IS NULL AND a.body_storage <> 'segmented') OR (c2.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = c2.page_id AND lm.occurrence_id = c2.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation))))) AS max_chunk_index, a.compiled_truth, a.timeline, NULL::int AS chunk_id, NULL::int AS chunk_index, NULL::text AS chunk_text, NULL::text AS chunk_source, NULL::bigint AS projected_byte_start, NULL::bigint AS projected_byte_end, NULL::bigint AS occurrence_id FROM auth a UNION ALL SELECT 'chunk' AS kind, c.page_id, NULL, NULL, NULL, NULL, NULL, NULL, c.prio, NULL, NULL, NULL, c.chunk_id, c.chunk_index, c.chunk_text, c.chunk_source, c.projected_byte_start, c.projected_byte_end, c.occurrence_id FROM chunk_rows c",
+ "textSha256": "f24da3c762df787ffdf6f66e172590825b5eceaa2b533540ec492bb1422d2459"
}
],
"scopedPrivateSafe": [
@@ -198,8 +198,8 @@
"array<string>",
"number"
],
- "sql": "WITH req AS ( SELECT * FROM unnest($1::int[], $2::int[], $3::int[], $4::int[]) AS r(page_id, lo, hi, prio) ), auth AS ( SELECT p.id, p.slug, p.source_id, p.type, p.knowledge_revision::text AS revision, p.compiled_truth, p.timeline, p.body_storage, p.body_generation, p.body_storage = 'segmented' AS segmented, (COALESCE(p.chunker_version, 0) >= $5) AS sealed, (SELECT MIN(req.prio) FROM req WHERE req.page_id = p.id) AS prio FROM pages p JOIN sources s ON s.id = p.source_id WHERE p.id = ANY($6::int[]) AND p.source_id = ANY($7::text[]) AND p.deleted_at IS NULL AND (p.text_projection_revision = p.knowledge_revision) IS TRUE AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND p.chunker_version >= 4 ), chunk_rows AS ( SELECT cc.page_id, cc.id AS chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.projected_byte_start, cc.projected_byte_end, cc.occurrence_id, a.prio FROM auth a JOIN content_chunks cc ON cc.page_id = a.id WHERE a.sealed AND ((cc.body_generation IS NULL AND a.body_storage <> 'segmented') OR (cc.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation)))) AND cc.chunk_source = ANY($8::text[]) AND COALESCE(cc.modality, 'text') = 'text' AND EXISTS (SELECT 1 FROM req r WHERE r.page_id = cc.page_id AND cc.chunk_index BETWEEN r.lo AND r.hi) ORDER BY a.prio, cc.page_id, cc.chunk_index LIMIT $9 ) SELECT 'page' AS kind, a.id AS page_id, a.slug, a.source_id, a.type, a.revision, a.sealed, a.segmented, a.prio, (SELECT MAX(c2.chunk_index) FROM content_chunks c2 WHERE c2.page_id = a.id AND ((c2.body_generation IS NULL AND a.body_storage <> 'segmented') OR (c2.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = c2.page_id AND lm.occurrence_id = c2.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation))))) AS max_chunk_index, a.compiled_truth, a.timeline, NULL::int AS chunk_id, NULL::int AS chunk_index, NULL::text AS chunk_text, NULL::text AS chunk_source, NULL::bigint AS projected_byte_start, NULL::bigint AS projected_byte_end, NULL::bigint AS occurrence_id FROM auth a UNION ALL SELECT 'chunk' AS kind, c.page_id, NULL, NULL, NULL, NULL, NULL, NULL, c.prio, NULL, NULL, NULL, c.chunk_id, c.chunk_index, c.chunk_text, c.chunk_source, c.projected_byte_start, c.projected_byte_end, c.occurrence_id FROM chunk_rows c",
- "textSha256": "89c36c375dfa71e17159ad7b8b2ecb95a8d0332d060800d6fc2b86c7801a1881"
+ "sql": "WITH req AS ( SELECT * FROM unnest($1::int[], $2::int[], $3::int[], $4::int[]) AS r(page_id, lo, hi, prio) ), auth AS ( SELECT p.id, p.slug, p.source_id, p.type, p.knowledge_revision::text AS revision, p.compiled_truth, p.timeline, p.body_storage, p.body_generation, p.body_storage = 'segmented' AS segmented, (COALESCE(p.chunker_version, 0) >= $5) AS sealed, (SELECT MIN(req.prio) FROM req WHERE req.page_id = p.id) AS prio FROM pages p JOIN sources s ON s.id = p.source_id WHERE p.id = ANY($6::int[]) AND p.source_id = ANY($7::text[]) AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND p.chunker_version >= 4 ), chunk_rows AS ( SELECT cc.page_id, cc.id AS chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.projected_byte_start, cc.projected_byte_end, cc.occurrence_id, a.prio FROM auth a JOIN content_chunks cc ON cc.page_id = a.id WHERE a.sealed AND ((cc.body_generation IS NULL AND a.body_storage <> 'segmented') OR (cc.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = cc.page_id AND lm.occurrence_id = cc.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation)))) AND cc.chunk_source = ANY($8::text[]) AND COALESCE(cc.modality, 'text') = 'text' AND EXISTS (SELECT 1 FROM req r WHERE r.page_id = cc.page_id AND cc.chunk_index BETWEEN r.lo AND r.hi) ORDER BY a.prio, cc.page_id, cc.chunk_index LIMIT $9 ) SELECT 'page' AS kind, a.id AS page_id, a.slug, a.source_id, a.type, a.revision, a.sealed, a.segmented, a.prio, (SELECT MAX(c2.chunk_index) FROM content_chunks c2 WHERE c2.page_id = a.id AND ((c2.body_generation IS NULL AND a.body_storage <> 'segmented') OR (c2.body_generation <= a.body_generation AND EXISTS (SELECT 1 FROM page_body_manifest lm WHERE lm.page_id = c2.page_id AND lm.occurrence_id = c2.occurrence_id AND lm.staging_id IS NULL AND lm.gen_from <= a.body_generation AND (lm.gen_to IS NULL OR lm.gen_to > a.body_generation))))) AS max_chunk_index, a.compiled_truth, a.timeline, NULL::int AS chunk_id, NULL::int AS chunk_index, NULL::text AS chunk_text, NULL::text AS chunk_source, NULL::bigint AS projected_byte_start, NULL::bigint AS projected_byte_end, NULL::bigint AS occurrence_id FROM auth a UNION ALL SELECT 'chunk' AS kind, c.page_id, NULL, NULL, NULL, NULL, NULL, NULL, c.prio, NULL, NULL, NULL, c.chunk_id, c.chunk_index, c.chunk_text, c.chunk_source, c.projected_byte_start, c.projected_byte_end, c.occurrence_id FROM chunk_rows c",
+ "textSha256": "5bcd45cd04bd8e67e1fa997f75418198809174197fa1767ceb6f8f7923e41eec"
}
]
},
@@ -223,8 +223,8 @@
"string",
"string"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "50b5504e4acfcff7895290a159757f56e42252349975c3d87e427d61290ab446"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "5f99c360a5fc587d74b7e92627b2488cd1aae3e7a7b8938371836c8f3118e897"
}
],
"excludePrivate": [
@@ -246,8 +246,8 @@
"string",
"string"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "b36f0fad328b986f0004b91e867c7a6ba4cfdd187319089f8e3c95abbe35b790"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((((p.page_readers IS NULL AND NOT (p.frontmatter ? 'readable_by'))) AND (p.folder_rule_ids IS NULL) AND NOT EXISTS (SELECT 1 FROM access_pending_coverage pc WHERE pc.source_id = p.source_id AND (starts_with(p.slug, pc.prefix) AND (pc.prefix = '' OR right(pc.prefix, 1) = '/' OR p.slug = pc.prefix))) AND NOT (SELECT gbrain_access_state_unreadable()) AND ((p.type NOT IN ('atom', 'question', 'concept', 'event') AND NOT (p.frontmatter ? 'derived_from')) OR NOT gbrain_access_origin_restricted(p.id, p.source_id, p.type, p.frontmatter))) AND ((COALESCE(p.frontmatter->>'visibility', CASE WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(p.frontmatter ? 'pinned_question', false) AND NOT (p.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(p.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(p.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN p.type = 'atom' THEN (EXISTS (SELECT 1 FROM pages derived_origin WHERE derived_origin.source_id = p.source_id AND derived_origin.slug = ANY(ARRAY[(p.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=p.source_id AND renamed.alias_slug=(p.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=p.source_id AND moved.old_slug=(p.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private')) WHEN (p.type IN ('atom', 'question') OR (p.type = 'concept' AND p.frontmatter->>'synthesized_by' IS NOT NULL)) THEN EXISTS (SELECT 1 FROM links derived_input_link JOIN pages derived_input ON derived_input.id = derived_input_link.to_page_id WHERE derived_input_link.from_page_id = p.id AND derived_input_link.link_source = 'concept-provenance' AND derived_input_link.link_type = 'synthesized_from' AND (COALESCE(derived_input.frontmatter->>'visibility', CASE WHEN derived_input.type = 'atom' THEN 'private' ELSE 'world' END) = 'private' OR EXISTS (SELECT 1 FROM pages derived_input_origin WHERE derived_input_origin.source_id = derived_input.source_id AND derived_input_origin.slug = ANY(ARRAY[(derived_input.frontmatter->>'source_slug'), (SELECT renamed.canonical_slug FROM slug_aliases renamed WHERE renamed.source_id=derived_input.source_id AND renamed.alias_slug=(derived_input.frontmatter->>'source_slug') AND renamed.notes='rename')]) AND derived_input_origin.frontmatter->>'visibility' = 'private') OR EXISTS (SELECT 1 FROM slug_rename_history moved JOIN pages moved_to ON moved_to.source_id=moved.source_id AND moved_to.slug=moved.new_slug WHERE moved.source_id=derived_input.source_id AND moved.old_slug=(derived_input.frontmatter->>'source_slug') AND moved_to.frontmatter->>'visibility'='private'))) WHEN p.type = 'event' AND p.frontmatter->>'captured_via' LIKE 'life-chronicle:%' THEN NOT EXISTS (SELECT 1 FROM pages chronicle_origin WHERE chronicle_origin.source_id = p.source_id AND chronicle_origin.slug = p.frontmatter->'event'->>'depth' AND (COALESCE(chronicle_origin.frontmatter->>'visibility', CASE WHEN (chronicle_origin.type IN ('atom', 'question') OR (chronicle_origin.type = 'concept' AND chronicle_origin.frontmatter->>'synthesized_by' IS NOT NULL)) THEN 'private' ELSE 'world' END) <> 'private' AND NOT COALESCE(chronicle_origin.frontmatter ? 'pinned_question', false) AND NOT (chronicle_origin.frontmatter->>'visibility' IS NULL AND (COALESCE(translate(btrim(chronicle_origin.frontmatter->>'access', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('private','restricted','confidential'), false) OR COALESCE(translate(btrim(chronicle_origin.frontmatter->>'confidentiality', (E'\\u0020\\u0009\\u000a\\u000b\\u000c\\u000d' || CASE WHEN current_setting('server_encoding') = 'UTF8' THEN convert_from(decode('c2a0e19a80e28080e28081e28082e28083e28084e28085e28086e28087e28088e28089e2808ae280a8e280a9e280afe2819fe38080efbbbf', 'hex'), 'UTF8') ELSE '' END)), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz') IN ('confidential','secret','private','restricted'), false)))) AND NOT (CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = chronicle_origin.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(chronicle_origin.frontmatter->'derived_from') = 'array' THEN chronicle_origin.frontmatter->'derived_from' ELSE jsonb_build_array(chronicle_origin.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END)) ELSE false END) AND NOT (CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') IN ('array', 'string') THEN EXISTS ( SELECT 1 FROM pages declared_origin WHERE declared_origin.source_id = p.source_id AND declared_origin.frontmatter->>'visibility' = 'private' AND declared_origin.slug = ANY(ARRAY( SELECT regexp_replace(declared.slug, '\\.md$', '') FROM jsonb_array_elements_text( CASE WHEN jsonb_typeof(p.frontmatter->'derived_from') = 'array' THEN p.frontmatter->'derived_from' ELSE jsonb_build_array(p.frontmatter->'derived_from') END) AS declared(slug)))) ELSE false END))) AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "723e136bddc29691b07ed55be488914b52e5531271c741645a8be03d4a36169b"
}
],
"includeEmbedding": [
@@ -269,8 +269,8 @@
"string",
"string"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "f25b0af8fc1b229c537e242c5eddfa3d9254c2dc44b338901b4edf8510f68c01"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null , cc.\"embedding\" AS embedding FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "5ccb56000391b8d9cdac00eea5d4d4396ad344d3673b997ccd71638d7eff4e6d"
}
],
"includeUnsealed": [
@@ -315,8 +315,8 @@
"string",
"string"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "41ce26c59b8099481552ac41ad14890c771fde32556df938ec6cbb95b8af984c"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = $2 AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.chunker_version >= 4 ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "7b5ec556e4e4d7c87fd88bebf6de93e2cfd6891d57af3706be28a112a776af22"
}
],
"sourceIds": [
@@ -338,8 +338,8 @@
"string",
"array<string>"
],
- "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
- "textSha256": "1d6acb2f10cc903eae3af683074ab9616b8113db68ed99a4433cbe997f63401d"
+ "sql": "SELECT cc.id, cc.page_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, cc.model, cc.token_count, cc.embedded_at, cc.language, cc.symbol_name, cc.symbol_type, cc.start_line, cc.end_line, cc.parent_symbol_path, cc.doc_comment, cc.symbol_name_qualified, cc.modality, cc.projected_byte_start, cc.projected_byte_end, (cc.\"embedding\" IS NULL) AS embedding_is_null FROM content_chunks cc JOIN pages p ON p.id = cc.page_id LEFT JOIN page_body_manifest m ON cc.occurrence_id IS NOT NULL AND m.page_id = cc.page_id AND m.occurrence_id = cc.occurrence_id AND m.staging_id IS NULL AND m.gen_from <= p.body_generation AND (m.gen_to IS NULL OR m.gen_to > p.body_generation) WHERE p.slug = $1 AND p.source_id = ANY($2::text[]) AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) ORDER BY m.byte_start, cc.local_ordinal, cc.chunk_index",
+ "textSha256": "a60680674b46ee590ed20a9b878c10e069dbcc63439ab68559ce580f50aaca00"
}
]
},
@@ -350,8 +350,8 @@
"params": [
"string"
],
- "sql": "SELECT cc.* FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.slug = $1 ORDER BY cc.chunk_index",
- "textSha256": "f4117c83ef5df061c6d44baff7af14a9a7dc16b468560a1338f3141c24b546d7"
+ "sql": "SELECT cc.* FROM content_chunks cc JOIN pages p ON p.id = cc.page_id WHERE (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND p.slug = $1 ORDER BY cc.chunk_index",
+ "textSha256": "198cb67722946551d610bff7bac9876250dd5eb0d6d6ff614c2f9f061760583b"
}
],
"sourceId+includeUnsealed": [
@@ -373,8 +373,8 @@
"params": [
"array<number>"
],
- "sql": "SELECT cc.id, vector_send(cc.\"embedding_voyage\"::vector) AS embedding FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id = ANY($1::int[]) AND cc.\"embedding_voyage\" IS NOT NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation))",
- "textSha256": "67cbaa36ba5cf2f6528734247df1db73005371251b757ef6fc5a5efdcd756dc3"
+ "sql": "SELECT cc.id, vector_send(cc.\"embedding_voyage\"::vector) AS embedding FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id = ANY($1::int[]) AND cc.\"embedding_voyage\" IS NOT NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation))",
+ "textSha256": "d3a5cfbd6948b8e74682c62e7056500b952cbedd19155196a4d33975d7114569"
}
],
"default": [
@@ -383,8 +383,8 @@
"params": [
"array<number>"
],
- "sql": "SELECT cc.id, vector_send(cc.\"embedding\"::vector) AS embedding FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id = ANY($1::int[]) AND cc.\"embedding\" IS NOT NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation))",
- "textSha256": "7e78d658816597146af750ec180ac76fb8f819e5e636ef50a93d2cf2ca76d532"
+ "sql": "SELECT cc.id, vector_send(cc.\"embedding\"::vector) AS embedding FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id = ANY($1::int[]) AND cc.\"embedding\" IS NOT NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation))",
+ "textSha256": "5b661b995104658500299be38bb9609bdb3c285d0ae79a9dcc3961d2d4ed3eae"
}
]
},
diff --git a/test/fixtures/goldens/sql-text/cjk-search.json b/test/fixtures/goldens/sql-text/cjk-search.json
index c5d9ad3a55..b706802bcf 100644
--- a/test/fixtures/goldens/sql-text/cjk-search.json
+++ b/test/fixtures/goldens/sql-text/cjk-search.json
@@ -49,8 +49,8 @@
"number",
"number"
],
- "sql": "WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') AND cc.modality = 'text' ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $10 ), best_per_page AS ( SELECT DISTINCT ON (COALESCE(source_id, 'default'), slug) * FROM ranked ORDER BY COALESCE(source_id, 'default'), slug, score DESC, page_id ASC, chunk_id ASC ) SELECT * FROM best_per_page ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $11 OFFSET $12",
- "textSha256": "19f4c98c6cbcd6e50e16ef9824423c39334cb4b68bc4e057d850dd1c1422a7e0"
+ "sql": "WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') AND cc.modality = 'text' ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $10 ), best_per_page AS ( SELECT DISTINCT ON (COALESCE(source_id, 'default'), slug) * FROM ranked ORDER BY COALESCE(source_id, 'default'), slug, score DESC, page_id ASC, chunk_id ASC ) SELECT * FROM best_per_page ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $11 OFFSET $12",
+ "textSha256": "73b18859be0ba27a6161fd644f7d14ef0a7964a17532d4c56a4742303f4e1c19"
},
{
"event": "commit",
@@ -105,8 +105,8 @@
"number",
"number"
],
- "sql": "WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND cc.chunk_source = 'compiled_truth' AND p.type = $10 AND p.source_id = ANY($11::text[]) AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') AND cc.modality = 'text' ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $12 ), best_per_page AS ( SELECT DISTINCT ON (COALESCE(source_id, 'default'), slug) * FROM ranked ORDER BY COALESCE(source_id, 'default'), slug, score DESC, page_id ASC, chunk_id ASC ) SELECT * FROM best_per_page ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $13 OFFSET $14",
- "textSha256": "31ac33411b3a569999e4b1c643d88d381c4ecad49c56dcd772335baebc27fb55"
+ "sql": "WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND cc.chunk_source = 'compiled_truth' AND p.type = $10 AND p.source_id = ANY($11::text[]) AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') AND cc.modality = 'text' ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $12 ), best_per_page AS ( SELECT DISTINCT ON (COALESCE(source_id, 'default'), slug) * FROM ranked ORDER BY COALESCE(source_id, 'default'), slug, score DESC, page_id ASC, chunk_id ASC ) SELECT * FROM best_per_page ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $13 OFFSET $14",
+ "textSha256": "2f690b32b65601bef67b25c3dd99915717401878882c41de66735c71caad4eef"
},
{
"event": "commit",
@@ -150,8 +150,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $10 OFFSET $11",
- "textSha256": "80fa25aa4c73070ddeaf034ab33184847d1b8a91203b9743a959013532e9f5ed"
+ "sql": "SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $10 OFFSET $11",
+ "textSha256": "84e9b48ebea8167c0ed1113c23e2c4af7e1772af95dbfc42f4ee2dce5aa7099a"
},
{
"event": "commit",
@@ -194,8 +194,8 @@
"number",
"number"
],
- "sql": "SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND p.source_id = $10 AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $11 OFFSET $12",
- "textSha256": "c1851c6f30af59f523fa02f8badedc034c39b866dadd9798fffa1c452fa2895b"
+ "sql": "SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.effective_date, p.effective_date_source, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN p.frontmatter->>'message_id' END AS message_id, p.frontmatter->>'thread_id' AS thread_id, CASE WHEN NULLIF(regexp_replace(p.frontmatter->>'message_id', '^[[:space:]]+|[[:space:]]+$', '', 'g'), '') IS NOT NULL THEN NULLIF(p.frontmatter->>'subject', '') END AS source_subject, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ((((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $1, ''))) / NULLIF(LENGTH($2), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $3, ''))) / NULLIF(LENGTH($4), 0)::real) + ((LENGTH(cc.chunk_text) - LENGTH(REPLACE(cc.chunk_text, $5, ''))) / NULLIF(LENGTH($6), 0)::real) + COALESCE(1.0 / NULLIF(POSITION($7 IN cc.chunk_text), 0)::real, 0.0)) * (CASE WHEN (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'provenance') = 'auto-extracted' AND (COALESCE(p.frontmatter, '{}'::jsonb) ->> 'status') = 'unverified' THEN 1.0 WHEN p.slug LIKE 'media/articles/%' THEN 1.1 WHEN p.slug LIKE 'openclaw/chat/%' THEN 0.5 WHEN p.slug LIKE 'media/repos/%' THEN 1.1 WHEN p.slug LIKE 'originals/%' THEN 1.5 WHEN p.slug LIKE 'companies/%' THEN 1.2 WHEN p.slug LIKE 'concepts/%' THEN 1.3 WHEN p.slug LIKE 'meetings/%' THEN 1.1 WHEN p.slug LIKE 'extracts/%' THEN 0.3 WHEN p.slug LIKE 'writing/%' THEN 1.4 WHEN p.slug LIKE 'media/x/%' THEN 0.7 WHEN p.slug LIKE 'archive/%' THEN 0.5 WHEN p.slug LIKE 'people/%' THEN 1.2 WHEN p.slug LIKE 'deals/%' THEN 1.2 WHEN p.slug LIKE 'civic/%' THEN 1 WHEN p.slug LIKE 'daily/%' THEN 0.8 WHEN p.slug LIKE 'yc/%' THEN 1 ELSE 1.0 END)) AS score, CASE WHEN p.updated_at < ( SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id JOIN sources s ON s.id = p.source_id WHERE cc.chunk_text LIKE $8 ESCAPE '\\' AND cc.chunk_text LIKE $9 ESCAPE '\\' AND p.source_id = $10 AND NOT (p.slug LIKE 'test/%' OR p.slug LIKE 'attachments/%' OR p.slug LIKE '.raw/%') AND p.deleted_at IS NULL AND (((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND (cc.body_generation IS NULL OR cc.body_generation <= p.body_generation)) AND NOT s.archived AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'pinned_question') ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $11 OFFSET $12",
+ "textSha256": "428b1ae1103da278615829359819f63ed15c31c47d83c092fb2ecf06a80c670f"
},
{
"event": "commit",
diff --git a/test/fixtures/goldens/sql-text/code-edges.json b/test/fixtures/goldens/sql-text/code-edges.json
index 1956908a63..7b12eadf23 100644
--- a/test/fixtures/goldens/sql-text/code-edges.json
+++ b/test/fixtures/goldens/sql-text/code-edges.json
@@ -63,8 +63,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) LIMIT $3",
- "textSha256": "ca9ab8f50f42835faf2b57c56dbdb28da474e95e561cbb768e1fbd5faa12f573"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) LIMIT $3",
+ "textSha256": "8257ae13d27588c4fbdc20cb8087ba8761904f8b29f9a427c4cea6ae7230c832"
},
{
"lane": "pool",
@@ -73,8 +73,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_chunk_id IN (SELECT id FROM content_chunks WHERE symbol_name = $1) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_chunk_id IN (SELECT id FROM content_chunks WHERE symbol_name = $2) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) LIMIT $3",
- "textSha256": "8e168723e3839daeec0e83980381f4418b39245a7dac73c11fbe04375bd79d95"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_chunk_id IN (SELECT id FROM content_chunks WHERE symbol_name = $1) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_chunk_id IN (SELECT id FROM content_chunks WHERE symbol_name = $2) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) LIMIT $3",
+ "textSha256": "1411a1342b4e96c6c7695d18e79945b8c9cc0a20ad10ffdde2f2c872bd7c1d18"
}
],
"chunkId": [
@@ -87,8 +87,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_chunk_id = $1::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_chunk_id = $3::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $4 LIMIT $5",
- "textSha256": "c9bfbe76c75e1f0e2879e7ae6396dbbfafa1cb409a633a2f6679497b5f8e5a92"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_chunk_id = $1::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_chunk_id = $3::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $4 LIMIT $5",
+ "textSha256": "abff1c53f2d7659a5b6da4092d74b81fe3349053fce7c523d764e86ac33e7afa"
}
],
"default": [
@@ -99,8 +99,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) LIMIT $3",
- "textSha256": "ca9ab8f50f42835faf2b57c56dbdb28da474e95e561cbb768e1fbd5faa12f573"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) LIMIT $3",
+ "textSha256": "8257ae13d27588c4fbdc20cb8087ba8761904f8b29f9a427c4cea6ae7230c832"
}
],
"sourceId": [
@@ -113,8 +113,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $3 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $4 LIMIT $5",
- "textSha256": "0cc6d3fe0897c0b4a85af4f6802dc424f6f9562df52769471a3bc8e1e87c82b6"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE from_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE from_symbol_qualified = $3 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $4 LIMIT $5",
+ "textSha256": "8cf8964f4224ffe25be30cf561f9ee7a6fc411572740c0a4518eadf398d778e6"
}
]
},
@@ -129,8 +129,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_chunk_id = $1::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE (edge_metadata->>'resolved_chunk_id')::int = $3::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $4 LIMIT $5",
- "textSha256": "ac70ef2296e47ecbbbde75fd59f4e0fbea9f77aee36ba46709d1db6a8b4780ba"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_chunk_id = $1::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE (edge_metadata->>'resolved_chunk_id')::int = $3::int AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $4 LIMIT $5",
+ "textSha256": "87de61ac21eff91a768295896c0a96520b37a84fe3f8cc777b7dec707c71fe09"
}
],
"default": [
@@ -141,8 +141,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE to_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) LIMIT $3",
- "textSha256": "035d362219e8718da7312c66a55c8ba87c30816e9ca2690ae9e451d227003f47"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE to_symbol_qualified = $2 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) LIMIT $3",
+ "textSha256": "ef039fc413baa8d569d0de5ea7e12c482a68786bfc751964f2e1785eab9c35c8"
}
],
"sourceId": [
@@ -155,8 +155,8 @@
"string",
"number"
],
- "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE to_symbol_qualified = $3 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND (p.text_projection_revision = p.knowledge_revision) IS TRUE ) AND source_id = $4 LIMIT $5",
- "textSha256": "66d18e0cbad1c5b661a33bada899c5af1b47883c7dd6aea9c1d4a65e1603ea85"
+ "sql": "SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, true as resolved FROM code_edges_chunk WHERE to_symbol_qualified = $1 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_chunk.to_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $2 UNION ALL SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id, false as resolved FROM code_edges_symbol WHERE to_symbol_qualified = $3 AND EXISTS ( SELECT 1 FROM content_chunks cc JOIN pages p ON p.id=cc.page_id WHERE cc.id=code_edges_symbol.from_chunk_id AND p.deleted_at IS NULL AND NOT (COALESCE(p.frontmatter, '{}'::jsonb) ? 'quarantine') AND EXISTS (SELECT 1 FROM sources read_source WHERE read_source.id = p.source_id AND NOT read_source.archived) AND TRUE AND ((p.text_projection_revision = p.knowledge_revision) IS TRUE OR (p.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) ) AND source_id = $4 LIMIT $5",
+ "textSha256": "b43463ca3a5227ac1d4db9fee0d1dcf33a715b4804b981d187c42a6cf41076e5"
}
]
},
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index cc3996a84d..786a457bab 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -620,6 +620,7 @@ test/retrieval-coverage-notices.test.ts
test/schema-missing-tables.test.ts
test/schema-replay-deadlock-6143.test.ts
test/scripts/test-stress-postgres.test.ts
+test/search-during-upgrade-rebuild-9192.test.ts
test/search-entity-facts-eligibility.test.ts
test/search-keyword-evidence-label-gbra449.test.ts
test/search-read-keyword-bound.test.ts
diff --git a/test/projection-rebuild-progress-9192.test.ts b/test/projection-rebuild-progress-9192.test.ts
index a16add213f..6d31b81aa6 100644
--- a/test/projection-rebuild-progress-9192.test.ts
+++ b/test/projection-rebuild-progress-9192.test.ts
@@ -1,5 +1,5 @@
/**
- * #9192: v153's activation takes every page out of the search index until the projection drain rebuilds it. While that
+ * #9192: v153's activation takes a page indexed before the safe-chunk fence out of search until the projection drain rebuilds it. While that
* rebuild runs, search and query say how far it is (pages done of total, the rate measured since it started) and which
* command shows progress, and the upgrade says so once when the activation runs.
*/
@@ -22,6 +22,9 @@ async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUr
await engine.putPage(slug, { type: 'note', title: 'Quokka harmonica', compiled_truth: TEXT });
await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT }]);
}
+ // Indexed before the safe-chunk fence, so activation takes them out of search (an index built after it stays
+ // served: search-during-upgrade-rebuild-9192.test.ts).
+ await engine.executeRaw('UPDATE pages SET chunker_version = 3');
for (const statement of PAGE_PROJECTION_ACTIVATION_SQL.split(';').map(s => s.trim()).filter(Boolean)) await engine.executeRaw(statement);
await run(engine);
} finally { await close(); }
@@ -47,13 +50,13 @@ for (const backend of testBackends()) {
expect(await answer(engine, 'search')).not.toContain('the search index rebuild an upgrade started');
}, url), 120_000);
- test('the v153 handler prints the queued count and the progress command once', () => activated(async engine => {
+ test('the v153 handler prints what stays served, what waits, the measured estimate and the progress command once', () => activated(async engine => {
const writes: string[] = [];
const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);
try { await v153.handler!(engine); } finally { spy.mockRestore(); }
- expect(writes.filter(w => w.includes('[153]'))).toEqual([
- ' [153] 2 page(s) queued for a search index rebuild; search and query results stay partial and say so until it finishes. Progress: gbrain doctor --only text_projection_readiness\n',
- ]);
+ const lines = writes.filter(w => w.includes('[153]'));
+ expect(lines).toHaveLength(1);
+ expect(lines[0]).toMatch(/^ {2}\[153\] 2 page\(s\) queued for a keyless search index rebuild \(no embedding calls\)\. Search keeps answering from the index 0 of them already had; 2 \(images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker\) stay out of search until rebuilt\. At the [\d.]+ ms round trip measured to this database the rebuild takes about 1 minute\(s\): gbrain projections drain\. Progress: gbrain doctor --only text_projection_readiness\n$/);
}, url), 120_000);
});
}
diff --git a/test/search-during-upgrade-rebuild-9192.test.ts b/test/search-during-upgrade-rebuild-9192.test.ts
new file mode 100644
index 0000000000..dc5eedfc2a
--- /dev/null
+++ b/test/search-during-upgrade-rebuild-9192.test.ts
@@ -0,0 +1,219 @@
+/**
+ * #9192: v153's activation queues every page for a text-projection rebuild. Search keeps answering from the index an
+ * inline Markdown or code page had (built at or after the safe-chunk fence) until that page's rebuild, and stops
+ * serving it the moment the page changes. A page indexed before the fence, and the queued rebuild itself, are unchanged.
+ */
+import { describe, expect, spyOn, test } from 'bun:test';
+import type { BrainEngine } from '../src/core/engine.ts';
+import { addMember, memberIdentity } from '../src/core/access/members.ts';
+import { stampIfPageProvenanceComplete } from '../src/core/embed-stale.ts';
+import { importFromContent } from '../src/core/import-file.ts';
+import { PAGE_PROJECTION_ACTIVATION_SQL } from '../src/core/page-state/projection-schema.ts';
+import { projectionBacklog, rebuildPendingPageProjections } from '../src/core/page-state/projections.ts';
+import { textRebuildDuration } from '../src/core/page-state/rebuild-estimate.ts';
+import { probeProjectionReadiness } from '../src/core/search/projection-readiness.ts';
+import { withCallerAccess, __resetPrivateVisibilityCacheForTests } from '../src/core/search/private-visibility.ts';
+import { ACTIVATION_SERVED_REVISION, currentTextProjectionFilter, servedTextProjectionFilter } from '../src/core/search/safe-chunks.ts';
+import { v153 } from '../src/core/schema-migrations/v153-verified-text-projection-activation.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { installFixtureChunks } from './helpers/page-projection.ts';
+import { isolatedSharedSkillsEngine } from './helpers/shared-skills-engine.ts';
+import { testBackends } from './helpers/test-backends.ts';
+
+const TEXT = 'The okapi theremin appears in this synthetic note.';
+const SLUGS = ['notes/okapi-one', 'notes/okapi-two'];
+const vector = (seed: number) => { const v = new Float32Array(1536); v[seed] = 1; return v; };
+
+const statements = (sql: string) => sql.split(';').map(s => s.trim()).filter(Boolean);
+/** The activation as v0.51.0.0 through master shipped it: every page out of search until rebuilt. */
+const SHIPPED_ACTIVATION = ['UPDATE pages SET text_projection_revision=NULL,embedding_signature=NULL', statements(PAGE_PROJECTION_ACTIVATION_SQL).at(-1)!];
+const drain = async (engine: BrainEngine) => { while ((await rebuildPendingPageProjections(engine, 20)).rebuilt > 0) { /* drain */ } };
+const count = async (engine: BrainEngine, where: string) =>
+ Number((await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM pages p WHERE ${where}`))[0]?.n);
+
+async function activated(run: (engine: BrainEngine) => Promise<void>, databaseUrl?: string, opts: { belowFence?: string; preRevision?: boolean; code?: string; extra?: Record<string, { body: string; frontmatter?: Record<string, unknown> }> } = {}) {
+ const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);
+ try {
+ for (const [i, slug] of SLUGS.entries()) {
+ await engine.putPage(slug, { type: 'note', title: 'Okapi theremin', compiled_truth: TEXT });
+ await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT, embedding: vector(i) }]);
+ }
+ for (const [slug, page] of Object.entries(opts.extra ?? {})) {
+ await engine.putPage(slug, { type: 'note', title: 'Okapi theremin', compiled_truth: page.body, ...(page.frontmatter ? { frontmatter: page.frontmatter } : {}) });
+ await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_source: 'compiled_truth', chunk_text: TEXT }]);
+ }
+ if (opts.belowFence) await engine.executeRaw('UPDATE pages SET chunker_version = 3 WHERE slug = $1', [opts.belowFence]);
+ if (opts.code) await engine.executeRaw("UPDATE pages SET page_kind = 'code', source_path = NULL WHERE slug = $1", [opts.code]);
+ if (opts.preRevision) {
+ // Pages written before v150 reach v153 with no knowledge_revision (the upgrade #9192 reported, from v145).
+ await engine.executeRaw('ALTER TABLE pages ALTER COLUMN knowledge_revision DROP NOT NULL');
+ await engine.executeRaw('ALTER TABLE pages DISABLE TRIGGER USER');
+ await engine.executeRaw('UPDATE pages SET knowledge_revision = NULL, text_projection_revision = NULL');
+ await engine.executeRaw('ALTER TABLE pages ENABLE TRIGGER USER');
+ }
+ for (const statement of statements(PAGE_PROJECTION_ACTIVATION_SQL)) await engine.executeRaw(statement);
+ await run(engine);
+ } finally { await close(); }
+}
+
+const SRC = 'source-a';
+const ACCESS_PAGES = {
+ alice: { slug: 'notes/alice-plan', canary: 'canaryinasalice', fm: 'readable_by: [alice-example]' },
+ open: { slug: 'notes/open-plan', canary: 'canaryinasopen', fm: '' },
+ gone: { slug: 'notes/gone-plan', canary: 'canaryinasgone', fm: '' },
+} as const;
+const accessPage = (p: { slug: string; canary: string }, fm: string) => `---\ntype: note\ntitle: ${p.slug}\n${fm}\n---\n# Plan\n\nPlanning notes ${p.canary}.`;
+
+/** Imported pages with readers, activated; `seen(member)` is what that member's search returns (null: the local owner). */
+async function accessActivated(run: (engine: BrainEngine, seen: (member: string | null) => Promise<string[]>) => Promise<void>, databaseUrl?: string) {
+ const { engine, close } = await isolatedSharedSkillsEngine(databaseUrl);
+ try {
+ __resetPrivateVisibilityCacheForTests();
+ await engine.executeRaw('INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING', [SRC]);
+ for (const p of Object.values(ACCESS_PAGES)) await importFromContent(engine, p.slug, accessPage(p, p.fm), { sourceId: SRC, noEmbed: true } as never);
+ await addMember(engine, 'alice-example');
+ await addMember(engine, 'bob-example');
+ for (const statement of statements(PAGE_PROJECTION_ACTIVATION_SQL)) await engine.executeRaw(statement);
+ const seen = async (member: string | null) => {
+ const identity = member ? (await memberIdentity(engine, member)).identity : null;
+ const out: string[] = [];
+ for (const [key, page] of Object.entries(ACCESS_PAGES)) {
+ const call = () => dispatchToolCall(engine, 'search', { query: page.canary, match: 'keyword', fields: 'full' }, { remote: false, sourceId: SRC });
+ const reply = identity ? await withCallerAccess({ identity, applyRemoteReadPolicy: true, hidePrivate: true, privateReadableSources: [] }, call) : await call();
+ if (reply.content.map(c => c.text).join('\n').includes(page.canary)) out.push(key);
+ }
+ return out.sort();
+ };
+ await run(engine, seen);
+ } finally { await close(); }
+}
+const keywordSlugs = async (engine: BrainEngine, query = 'okapi theremin', remote = false) =>
+ (await engine.searchKeyword(query, { limit: 10, requireSafeChunks: remote })).map(r => r.slug).sort();
+const vectorSlugs = async (engine: BrainEngine, seed: number) => (await engine.searchVector(vector(seed), { limit: 1 })).map(r => r.slug);
+
+test('the estimate follows the measured round trip: 50,000 pages take minutes locally and hours behind a 17 ms pooler', () => {
+ expect(textRebuildDuration(50_000, 0.1)).toBe('14 minute(s)');
+ expect(textRebuildDuration(50_000, 17.3)).toBe('9 hour(s)');
+});
+
+for (const backend of testBackends()) {
+ const url = backend === 'postgres' ? process.env.DATABASE_URL : undefined;
+ describe(`#9192 search keeps answering while the upgrade's rebuild drains (${backend})`, () => {
+ test('keyword, safe-chunk and vector search find every page right after activation, with no partial notice', () => activated(async engine => {
+ expect((await projectionBacklog(engine)).pending).toBe(2);
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ expect(await keywordSlugs(engine, 'okapi theremin', true)).toEqual(SLUGS);
+ expect(await vectorSlugs(engine, 0)).toEqual([SLUGS[0]!]);
+ expect(await vectorSlugs(engine, 1)).toEqual([SLUGS[1]!]);
+ expect((await probeProjectionReadiness(engine)).status).toBe('ready');
+ const reply = JSON.stringify(await dispatchToolCall(engine, 'search', { query: 'okapi theremin' }, { remote: false }));
+ expect(reply).toContain(SLUGS[0]!);
+ expect(reply).not.toContain('results may be incomplete');
+ }, url), 120_000);
+
+ test('pages from before v150, which reach the activation with no revision, keep answering too', () => activated(async engine => {
+ expect((await projectionBacklog(engine)).pending).toBe(2);
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ expect(await vectorSlugs(engine, 1)).toEqual([SLUGS[1]!]);
+ }, url, { preRevision: true }), 120_000);
+
+ test('a page edited after activation leaves search until its rebuild, then answers with the new text', () => activated(async engine => {
+ await engine.putPage(SLUGS[0]!, { type: 'note', title: 'Okapi theremin', compiled_truth: 'The okapi now plays a zither.' });
+ expect(await keywordSlugs(engine)).toEqual([SLUGS[1]!]);
+ while ((await rebuildPendingPageProjections(engine, 20)).rebuilt > 0) { /* drain */ }
+ expect(await keywordSlugs(engine, 'zither')).toEqual([SLUGS[0]!]);
+ expect((await projectionBacklog(engine)).pending).toBe(0);
+ const [row] = await engine.executeRaw<{ sealed: number }>('SELECT count(*)::int AS sealed FROM pages WHERE text_projection_revision = knowledge_revision');
+ expect(row?.sealed).toBe(2);
+ }, url), 120_000);
+
+ test('a page indexed before the safe-chunk fence stays out of search until it is rebuilt', () => activated(async engine => {
+ expect(await keywordSlugs(engine)).toEqual([SLUGS[1]!]);
+ expect((await probeProjectionReadiness(engine)).status).toBe('projection_pending');
+ while ((await rebuildPendingPageProjections(engine, 20)).rebuilt > 0) { /* drain */ }
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ }, url, { belowFence: SLUGS[0] }), 120_000);
+
+ test('a page whose rebuild would remove something keeps waiting: facts or takes fences, private-key text, a quarantine marker', () => activated(async engine => {
+ const served = await engine.executeRaw<{ slug: string }>(`SELECT slug FROM pages WHERE text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid ORDER BY slug`);
+ expect(served.map(r => r.slug)).toEqual(SLUGS);
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ expect((await probeProjectionReadiness(engine)).status).toBe('projection_pending');
+ expect((await projectionBacklog(engine)).pending).toBe(6);
+ }, url, { extra: {
+ 'notes/facts': { body: `${TEXT}\n\n<!--- gbrain:facts:begin -->\n| # | claim |\n|---|---|\n| 1 | okapi |\n<!--- gbrain:facts:end -->` },
+ 'notes/takes': { body: `${TEXT}\n\n<!--- gbrain:takes:begin -->\nokapi take\n<!--- gbrain:takes:end -->` },
+ 'notes/key': { body: [TEXT, ['-----BEGIN', ' RSA PRIVATE KEY', '-----'].join(''), 'MIIEexample', ['-----END', ' RSA PRIVATE KEY', '-----'].join('')].join('\n') },
+ 'notes/quarantined': { body: TEXT, frontmatter: { quarantine: 'junk' } },
+ } }), 120_000);
+
+ test('a restricted caller gets no marker page it cannot read, and a soft-deleted marker page is not returned', () => accessActivated(async (engine, seen) => {
+ expect(await count(engine, `p.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid`)).toBe(3);
+ expect(await seen(null)).toEqual(['alice', 'gone', 'open']);
+ expect(await seen('alice-example')).toEqual(['alice', 'gone', 'open']);
+ expect(await seen('bob-example')).toEqual(['gone', 'open']);
+ await engine.softDeletePage(ACCESS_PAGES.gone.slug, { sourceId: SRC });
+ expect(await seen(null)).toEqual(['alice', 'open']);
+ expect(await seen('bob-example')).toEqual(['open']);
+ }, url), 120_000);
+
+ test('a page whose readers change before its rebuild leaves search, and afterwards answers only its new readers', () => accessActivated(async (engine, seen) => {
+ await importFromContent(engine, ACCESS_PAGES.open.slug, accessPage(ACCESS_PAGES.open, 'readable_by: [alice-example]'), { sourceId: SRC, noEmbed: true } as never);
+ const [open] = await engine.executeRaw<{ tpr: string | null }>('SELECT text_projection_revision::text AS tpr FROM pages WHERE slug = $1', [ACCESS_PAGES.open.slug]);
+ expect(open?.tpr).not.toBe(ACTIVATION_SERVED_REVISION);
+ expect(await seen('bob-example')).toEqual(['gone']);
+ await drain(engine);
+ expect(await seen('bob-example')).toEqual(['gone']);
+ expect(await seen('alice-example')).toEqual(['alice', 'gone', 'open']);
+ }, url), 120_000);
+
+ test('a page whose rebuild fails keeps serving its old index and is still counted as waiting', () => activated(async engine => {
+ await rebuildPendingPageProjections(engine, 20);
+ expect(await engine.executeRaw('SELECT reason FROM page_projection_jobs WHERE slug = $1', [SLUGS[0]])).toEqual([{ reason: 'rebuild_failed' }]);
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ const backlog = await projectionBacklog(engine);
+ expect(backlog.pending).toBe(1);
+ expect(backlog.failed).toBe(1);
+ }, url, { code: SLUGS[0] }), 120_000);
+
+ test('a brain that already ran the shipped activation moves to this build part-way through its rebuild with no re-run and no refusal', () => activated(async engine => {
+ for (const statement of SHIPPED_ACTIVATION) await engine.executeRaw(statement);
+ expect((await rebuildPendingPageProjections(engine, 1)).rebuilt).toBe(1);
+ const writes: string[] = [];
+ const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);
+ try { await engine.initSchema(); } finally { spy.mockRestore(); }
+ expect(writes.some(w => w.includes('[153]'))).toBe(false);
+ expect(await count(engine, `p.text_projection_revision = '${ACTIVATION_SERVED_REVISION}'::uuid`)).toBe(0);
+ expect((await keywordSlugs(engine)).length).toBe(1);
+ expect((await projectionBacklog(engine)).pending).toBe(1);
+ await drain(engine);
+ expect(await keywordSlugs(engine)).toEqual(SLUGS);
+ }, url), 120_000);
+
+ test('the marker never equals a real revision, and the strict current-projection reads treat a marker page as not current', () => activated(async engine => {
+ expect(ACTIVATION_SERVED_REVISION[14]).toBe('0');
+ expect(await count(engine, "substr(p.knowledge_revision::text, 15, 1) <> '4'")).toBe(0);
+ expect(await count(engine, servedTextProjectionFilter('p'))).toBe(2);
+ expect(await count(engine, currentTextProjectionFilter('p'))).toBe(0);
+ const stamp = { signature: 'openai:text-embedding-3-small:1536', column: 'embedding' };
+ expect(await stampIfPageProvenanceComplete(engine, SLUGS[0]!, 'default', stamp)).toBe(false);
+ await drain(engine);
+ expect(await count(engine, currentTextProjectionFilter('p'))).toBe(2);
+ }, url), 120_000);
+
+ test('an older build reads a marker page as waiting: its search predicate drops it and its readiness predicate counts it', () => activated(async engine => {
+ expect(await count(engine, '(p.text_projection_revision = p.knowledge_revision) IS TRUE')).toBe(0);
+ expect(await count(engine, 'p.deleted_at IS NULL AND p.text_projection_revision IS DISTINCT FROM p.knowledge_revision')).toBe(2);
+ expect((await projectionBacklog(engine)).pending).toBe(2);
+ }, url), 120_000);
+
+ test('the v153 handler says which pages stay served and which wait', () => activated(async engine => {
+ const writes: string[] = [];
+ const spy = spyOn(process.stderr, 'write').mockImplementation(((chunk: string) => { writes.push(String(chunk)); return true; }) as typeof process.stderr.write);
+ try { await v153.handler!(engine); } finally { spy.mockRestore(); }
+ const line = writes.find(w => w.includes('[153]')) ?? '';
+ expect(line).toContain('Search keeps answering from the index 1 of them already had; 1 (images, pages indexed before the safe-chunk fence, and pages with a facts or takes fence, private-key text or a quarantine marker) stay out of search until rebuilt.');
+ expect(line).toContain('gbrain projections drain');
+ }, url, { belowFence: SLUGS[0] }), 120_000);
+ });
+}
diff --git a/test/sql-ranking.test.ts b/test/sql-ranking.test.ts
index 082932f603..a92cb5c133 100644
--- a/test/sql-ranking.test.ts
+++ b/test/sql-ranking.test.ts
@@ -311,7 +311,7 @@ describe('buildVisibilityClause (v0.26.5)', () => {
test('uses the supplied aliases verbatim', () => {
expect(buildVisibilityClause('pp', 'src')).toBe(
- "AND pp.deleted_at IS NULL AND (pp.text_projection_revision = pp.knowledge_revision) IS TRUE AND NOT src.archived AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'pinned_question')",
+ "AND pp.deleted_at IS NULL AND ((pp.text_projection_revision = pp.knowledge_revision) IS TRUE OR (pp.text_projection_revision = '00000000-0000-0000-0000-000000000153'::uuid) IS TRUE) AND NOT src.archived AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'quarantine') AND NOT (COALESCE(pp.frontmatter, '{}'::jsonb) ? 'pinned_question')",
);
});