Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Local searches cap the title step at 5,000 pages for very common words (#7082) (#9502)
Garry Tanad6da0b72479
diff --git a/behavior-changes.d/title_arm_candidate_cap_7082.md b/behavior-changes.d/title_arm_candidate_cap_7082.mdnew file mode 100644index 0000000000..b0ce24aaf0--- /dev/null+++ b/behavior-changes.d/title_arm_candidate_cap_7082.md@@ -0,0 +1,4 @@+---+kind: fix+---+On a local search (the CLI and gbrain serve's local callers), the title step now ranks at most 5,000 of the pages that match the query in their title or timeline: the 5,000 with the lowest page ids, so the same pages on every call for the same data. Before, a very common word made it rank every matching page and read every matching timeline until the keyword time limit. A query that matches 5,000 pages or fewer gets the same result as before, timeline-only matches included. When the cut applies, the pages whose title holds the words come first (the rows remote callers get), then the other matching pages among the 5,000 kept, and hybrid search reports `keyword_candidates_incomplete`. A query of several words is cut on the pages that match the whole query, not word by word. Remote callers' results do not change.diff --git a/changelog.d/capy-title-arm-candidate-cap.md b/changelog.d/capy-title-arm-candidate-cap.mdnew file mode 100644index 0000000000..5b4bf2c868--- /dev/null+++ b/changelog.d/capy-title-arm-candidate-cap.md@@ -0,0 +1,6 @@+**Local searches on a very common word no longer spend the keyword time limit in the title step.**++### Fixed+- The title step of hybrid search now ranks at most 5,000 matching pages for local callers, the ones with the lowest page ids, so the cut keeps the same pages on every call. A word that sits in many page timelines used to make it rank every matching page and run into the 8 s keyword statement limit on a large brain, a second time after the keyword step (#7082). When the cut applies, pages whose title holds the words come first and hybrid search reports `keyword_candidates_incomplete`. A query that matches 5,000 pages or fewer keeps its exact result, including a page found only through a timeline row that no chunk holds. Remote callers' results do not change.++<!-- behavior-change: title_arm_candidate_cap_7082 -->diff --git a/docs/architecture/key-files/engines-2.md b/docs/architecture/key-files/engines-2.mdindex a3b88272da..70245b2c36 100644--- a/docs/architecture/key-files/engines-2.md+++ b/docs/architecture/key-files/engines-2.md@@ -31,7 +31,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/engine-sql/keyword-pages.ts` + `src/core/search/keyword-statement.ts` — the strict keyword count (`BrainEngine.countKeywordPages`) and the keyword-mode page query (`BrainEngine.searchKeywordPages`) for both engines. `buildKeywordMatch` is the ONE WHERE builder both statements read: `searchKeyword`'s strict match (websearch FTS on `content_chunks.search_vector`, or the CJK LIKE/ILIKE terms), never its OR retry, with the same page filters, hard excludes and `buildVisibilityClause` (private pages, safe chunks, so no private fence text). A query with slug handles (`search/keyword-handles.ts`) adds a second arm under the same filters, the chunks of the pages whose slug basename equals a handle (`idx_pages_slug_basename`), scored 1 + rank so they lead; the count is a `UNION` of both arms and `buildKeywordHandlePagesStatement` reads the handle arm alone for `searchKeyword`. The match returns its WHERE params first and the score-only rank params after them (`whereParamCount`), so the count binds only what its WHERE reads (a CJK count bound the term-frequency params and failed on both engines; `test/sql-bind-placeholders.test.ts`). The count is `count(*)` over `SELECT DISTINCT page_id … LIMIT 10001`; a non-timeout count failure logs its SQLSTATE once per process (`keyword-paging.ts`); the page query takes each page's best chunk before paging (no global chunk cap), orders by `(score DESC, page_id ASC)` with float8 scores so the keyset cursor round-trips, and pages those (page_id, chunk_id, score) rows before it reads the page and chunk rows it returns (#7082; `test/search/keyword-pages-limit-first.test.ts`, Postgres arm in `test/e2e/keyword-pages-limit-first-postgres.test.ts`). Postgres runs both inside `withScopedReadTransaction` with a statement timeout (the count's set from the caller's deadline); PGLite has none.- `src/core/search/exact-scan.ts` + `src/core/ops/search-exact.ts` + `src/cli/search-exact-format.ts` — #7191: `search_exact` (CLI `gbrain search --exact`, routed in `cli/main.ts` before ranked-search flags parse): an unranked literal scan of `content_chunks.chunk_text` (words in order, any whitespace between, case-insensitive unless `case_sensitive`; the ARE comes from `phrasePattern`, with a LIKE prefilter on the longest word) in keyset batches of pages by id (`buildExactBatchStatement`), under ranked search's visibility (live page, unarchived source, quarantine, pinned questions, `chunkVisibleFilter`, source scope, private pages and the trust floor for untrusted callers, `safeChunksFilter` for remote) and the hard-excluded prefixes. `complete` is true only when the scope was scanned to the end with no `limit` or `deadline` stop and no unreadable page (`unchunked`, `projection_pending`, remote `unsealed`, `scan_timeout`); a Postgres statement timeout halves the batch, down to one page. `resume_after` continues a bounded scan. Registered `filtered` (`min_trust`, `stampPageTrust`), output redaction `retrieval`. Guide: [exact search](../../guides/exact-search.md). Tests: `test/search-exact.test.ts`, `test/e2e/search-exact-postgres.test.ts`.-- `src/core/engine-sql/titles.ts` + `src/core/search/title-statement.ts` — the page-grain title arm (`BrainEngine.searchTitles`) for both engines: `buildSearchTitlesStatement` in `search/` (SQL composed outside engine-sql like `cjk-keyword-sql.ts`; also used by the plan-proof E2E `test/e2e/title-arm-postgres.test.ts`) and `searchTitles(scoped, query, opts, dialect)` in engine-sql, where `dialect` carries the engine differences kept from master (Postgres: 8s per-attempt `statement_timeout`, `enable_seqscan` off for the relaxed local re-run; PGLite: the timeline `stale` probe). Both orderings sort an exact-title key first (`lower(btrim(title, ' "''' || chr(160))) = normalizeAlias(query)`, bound separately from the websearch text so the OR fallback never rebinds it; #5889). The websearch text is read through `gbrain_fts_input($1)` (#6370), the same CJK/ASCII boundary split the index applies, so a glued title query matches. Safe-chunks callers and every OR re-run match `ts_filter(search_vector, '{a}') @@ q` (weight A is the title alone) and rank on the same slice: served by the expression index `idx_pages_title_search` (GBRA-156, v267), which holds title lexemes only, so a term in timeline text never makes its page a candidate; title-only for negated and phrase terms, following the stored `search_vector` language. Postgres runs each attempt with the keyword statement's `work_mem` (`statementSettings`). Pages with a NULL `search_vector` are absent from the remote arm. #7471: a 2-4 word name query ORs in its given-name variants (`titleNameVariants`: "Matt Example" also matches the title "Matthew Example"), each its own bound websearch query; the OR fallback of a query of at most `RARITY_MAX_TERMS` words runs once per word in one scoped read and `rankByRarity` puts titles holding the rarest word first (ts_rank_cd knows no term rarity, so a surname's one title used to tie thousands of first-name titles and lose on page id). #7491: OR-retry titles holding every identifier the query names (`holdsIdentifiers`, search/keyword-identifier-probe.ts) lead untagged; the rest stay `keyword_relaxed`.+- `src/core/engine-sql/titles.ts` + `src/core/search/title-statement.ts` — the page-grain title arm (`BrainEngine.searchTitles`) for both engines: `buildSearchTitlesStatement` in `search/` (SQL composed outside engine-sql like `cjk-keyword-sql.ts`; also used by the plan-proof E2E `test/e2e/title-arm-postgres.test.ts`) and `searchTitles(scoped, query, opts, dialect)` in engine-sql, where `dialect` carries the engine differences kept from master (Postgres: 8s per-attempt `statement_timeout`, `enable_seqscan` off for the relaxed local re-run; PGLite: the timeline `stale` probe). Both orderings sort an exact-title key first (`lower(btrim(title, ' "''' || chr(160))) = normalizeAlias(query)`, bound separately from the websearch text so the OR fallback never rebinds it; #5889). The websearch text is read through `gbrain_fts_input($1)` (#6370), the same CJK/ASCII boundary split the index applies, so a glued title query matches. Safe-chunks callers and every OR re-run match `ts_filter(search_vector, '{a}') @@ q` (weight A is the title alone) and rank on the same slice: served by the expression index `idx_pages_title_search` (GBRA-156, v267), which holds title lexemes only, so a term in timeline text never makes its page a candidate; title-only for negated and phrase terms, following the stored `search_vector` language. Postgres runs each attempt with the keyword statement's `work_mem` (`statementSettings`). Pages with a NULL `search_vector` are absent from the remote arm. #7471: a 2-4 word name query ORs in its given-name variants (`titleNameVariants`: "Matt Example" also matches the title "Matthew Example"), each its own bound websearch query; the OR fallback of a query of at most `RARITY_MAX_TERMS` words runs once per word in one scoped read and `rankByRarity` puts titles holding the rarest word first (ts_rank_cd knows no term rarity, so a surname's one title used to tie thousands of first-name titles and lose on page id). #7491: OR-retry titles holding every identifier the query names (`holdsIdentifiers`, search/keyword-identifier-probe.ts) lead untagged; the rest stay `keyword_relaxed`. #7082: a local caller's strict attempt matches the whole vector but ranks at most `titleWholeVectorCandidates()` (5,000) matching pages, the lowest page ids (the `candidates` CTE, ordered by id); a cut attempt reports `keyword_candidates_incomplete` (reason `candidate_budget`) and puts the title-only statement's rows first.- `src/core/engine-sql/page-lookup.ts` + `src/core/page-find.ts` + `src/core/page-counts.ts` + `src/core/ops/find-pages.ts` + `src/core/ops/stats-by.ts` + `src/cli/format-page-lookup.ts` — find-by-name and grouped counts (`gbrain find` / `find_pages`, `gbrain stats --by` / `get_stats` `by`, `since`, `collapse_aliases`). `parseFindPattern` turns a glob (`*`/`?`, whole slug or source path), a slug prefix (contains `/`) or a name (title or last slug segment substring) into one `escapeLikePattern`-escaped ILIKE pattern; names are served by `idx_pages_trgm` + `idx_pages_slug_trgm`, while globs and prefixes also match `source_path` (no trigram index). Rows sort by slug then source in the C collation, carry `count(*) OVER ()` as `total`, and the op reports `complete` (limit default 50, cap 500 local, 100 remote). Find hides the ranked-search hard excludes (`resolveHardExcludes` + `buildHardExcludeClause`) unless the pattern's literal lead is inside one, and names them in `hidden_prefixes`. `countPagesBy` returns per (source, type) counts plus raw and distinct-slug totals (per type group through an `unnest` mapping and GROUPING SETS), so cross-source twins are never silently double counted; `since` is the query effective-date bound (`resolveDateBoundary`). Every caller sees live pages in unarchived sources; a remote caller's `readPolicyOpts` policy adds `pageReadFilter(..., live)`. `collapseTypes` lets the active pack decide (declared stays, alias joins canonical), else normalizes (last segment, lower case, singular) and joins a `-<name>` suffix to an existing group; each group lists `member_types` with its rule (`exact`, `pack_alias`, `normalized`, `suffix`). Without the grouping params `get_stats` returns its unchanged counters. Tests: `test/page-find-pattern-7191.test.ts`, `test/page-lookup-7191.test.ts` (Postgres arm: `test/e2e/page-lookup-7191.test.ts`).- `src/core/engine-sql/health.ts` — `BrainEngine.getHealth` once for both engines (`migrated health`). Three statements, no per-page rows returned: the counters (pages, chunks, links, embedding coverage, missing embeddings, dead links, entity link/timeline coverage), the linkable scope grouped by page type with islanded and with-timeline counts (the orphan policy rendered in SQL by `orphanExclusionSql`; the brain_score timeline grading applies per type), and `most_connected` as live, in-scope link endpoints grouped per entity page (ties break by slug in code-point order; a link whose far endpoint is soft-deleted does not count, #6940). The linkable groups are a separate statement because folding them into the counters crosses Postgres's JIT cost thresholds at about 20k pages. Per-page and per-link checks that misplan without planner statistics (PGLite before its first ANALYZE) are primary-key lookups the planner cannot reorder (scalar subqueries, a LATERAL fenced with `OFFSET 0`), so the statements stay linear with or without statistics. Scope rules: `sourceIds` > `sourceId` > brain-wide, duplicate ids count once, a link counts only when both endpoints are in scope, endpoint liveness in both directions, quarantined pages out of the entity denominators and the linkable scope. Entity pages (#4772) are the types the active schema pack(s) declare as entities, resolved per source by `HealthDeps.entityTypes` (`src/core/schema-pack/entity-types.ts`) and bound as `text[]` parameters (`entityTypePredicateSql`: one list, or a `CASE p.source_id` when sources' packs disagree); a pack that does not load binds an empty list and the result carries `entity_types_status: 'pack_unavailable'`, never the old hardcoded list. Reads take `LegacyUnscopedRead` (pool, no scope transaction). Engine deps are four functions (`embeddingColumn`, `countStalePagesForExtraction`, `getConfig`, `entityTypes`). Pinned field by field against the pre-F4a implementation (`test/helpers/legacy-get-health.ts`, whose `most_connected` was corrected for #6940 as an intended change) by `test/engine-sql-health-equality.test.ts` and `test/e2e/engine-sql-health-parity.test.ts` (direct Postgres and PgBouncer); the pack-driven entity predicate by `test/health-entity-types-pack.test.ts` and `test/e2e/health-entity-types-pack.test.ts`; `most_connected` liveness by `test/health-most-connected-liveness-6940.test.ts` (+ Postgres twin).diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.mdindex 302c6e1073..aab59575f6 100644--- a/docs/guides/troubleshooting.md+++ b/docs/guides/troubleshooting.md@@ -536,8 +536,12 @@ seconds and comes back with only vector hits; search metadata carriesstatement timeout` (SQLSTATE 57014). Keyword-mode paging and the strict pagecount rank every matching page, so their cost grows with how common the termsare. Hybrid search's keyword arm ranks at most 50,000 matching chunks (its metadata-carries `keyword_candidates_incomplete` when that bound applies). A local-caller's statement usually stops reading matches at that bound. A remote MCP+carries `keyword_candidates_incomplete` when that bound applies). For a local+caller, hybrid search's title step ranks at most 5,000 of the pages whose title+or timeline matches (the lowest page ids) and reports the same+`keyword_candidates_incomplete` when it cuts; a remote caller's title step+matches titles only. A local+caller's keyword statement usually stops reading matches at the 50,000 bound. A remote MCPcaller's arm stops there when the call sets no type, frontmatter or date filterand no source scope narrower than every live source; with such a filter itcan still read every match before ranking. (A remote caller's arm used to readdiff --git a/llms-full.txt b/llms-full.txtindex 03715c301a..df1d06b0c2 100644--- a/llms-full.txt+++ b/llms-full.txt@@ -9796,8 +9796,12 @@ seconds and comes back with only vector hits; search metadata carriesstatement timeout` (SQLSTATE 57014). Keyword-mode paging and the strict pagecount rank every matching page, so their cost grows with how common the termsare. Hybrid search's keyword arm ranks at most 50,000 matching chunks (its metadata-carries `keyword_candidates_incomplete` when that bound applies). A local-caller's statement usually stops reading matches at that bound. A remote MCP+carries `keyword_candidates_incomplete` when that bound applies). For a local+caller, hybrid search's title step ranks at most 5,000 of the pages whose title+or timeline matches (the lowest page ids) and reports the same+`keyword_candidates_incomplete` when it cuts; a remote caller's title step+matches titles only. A local+caller's keyword statement usually stops reading matches at the 50,000 bound. A remote MCPcaller's arm stops there when the call sets no type, frontmatter or date filterand no source scope narrower than every live source; with such a filter itcan still read every match before ranking. (A remote caller's arm used to readdiff --git a/src/core/engine-sql/titles.ts b/src/core/engine-sql/titles.tsindex d2ae32b188..cf79026ab0 100644--- a/src/core/engine-sql/titles.ts+++ b/src/core/engine-sql/titles.ts@@ -41,7 +41,13 @@* A dominates); their OR re-run matches titles only, like remote callers,* because matching the whole vector let a date in a question pull every dated* timeline into `ts_rank_cd`: 3-5 s on a large brain (#7135, diagnosed by- * @veltri-23 in #7136).+ * @veltri-23 in #7136). The strict attempt ranks at most+ * `titleWholeVectorCandidates()` matching pages (#7082: a very common word+ * otherwise read every matching vector until the statement timeout). The whole+ * vector is the only lexical path to timeline_entries rows no chunk holds, so a+ * rarer term keeps its exact result; a cut attempt reports+ * `keyword_candidates_incomplete` and puts the title-only statement's rows+ * first, so title and exact-title matches never depend on which pages the cut kept.*/import type { SearchOpts, SearchResult } from '../types.ts';import { searchLimitCap } from '../search/eval-pool-depth.ts';@@ -80,7 +86,7 @@ export async function searchTitles(const statementFor = (chunkManifest: boolean, relaxed = false) => buildSearchTitlesStatement(query, opts, { ...dialect, chunkManifest, relaxed });const statement = statementFor(true);/** One scoped read running the statement once per query text ($1), in order. */- const run = (queryTexts: string[], relaxed: boolean) => {+ const run = (queryTexts: string[], relaxed: boolean, top?: number) => {const preferIndex = relaxed && dialect.relaxedPrefersIndex && !statement.safeChunks;const settings = {...(dialect.statementTimeout ? { statement_timeout: dialect.statementTimeout } : {}),@@ -90,14 +96,21 @@ export async function searchTitles(return scoped(async exec => {const sql = statementFor(await dialect.chunkManifest?.(exec) ?? true, relaxed).sql;const lists = [];- for (const text of queryTexts) lists.push((await exec.unsafe(sql, [text, ...statement.params.slice(1)])).rows);+ for (const text of queryTexts) lists.push((await exec.unsafe(sql, [text, statement.params[1], ...(top === undefined ? statement.params.slice(2, 4) : [top, 0]), ...statement.params.slice(4)])).rows);return lists;}, Object.keys(settings).length ? settings : undefined);};const strictQuery = statement.params[0] as string;- const [rows] = await run([strictQuery], false);- if (rows.length > 0) return rows.map(rowToSearchResult);+ const limit = statement.params[2] as number, offset = statement.params[3] as number;+ const [rows] = await run([strictQuery], false, offset + limit);+ if (rows.length > 0 && rows[0].candidates_cut === true) {+ opts?.onKeywordIncomplete?.('candidate_budget');+ const [titleRows] = await run([strictQuery], true, offset + limit);+ const titled = new Set(titleRows.map(r => r.page_id));+ return [...titleRows, ...rows.filter(r => !titled.has(r.page_id))].slice(offset, offset + limit).map(rowToSearchResult);+ }+ if (rows.length > offset) return rows.slice(offset).map(rowToSearchResult);const orQuery = buildOrFallbackWebsearchQuery(strictQuery);if (!orQuery) return [];const terms = orFallbackTokens(strictQuery);diff --git a/src/core/search/title-statement.ts b/src/core/search/title-statement.tsindex 07341136ce..9f9921282e 100644--- a/src/core/search/title-statement.ts+++ b/src/core/search/title-statement.ts@@ -31,10 +31,24 @@ export interface SearchTitlesDialect {staleProbe: boolean;/** chunkManifestReads; unset reads the manifest. */chunkManifest?: boolean;- /** The relaxed (OR) re-run: match and rank title lexemes only, for local callers too (#7135). */+ /** Match and rank title lexemes only, for local callers too: the OR re-run (#7135) and the cut strict attempt (#7082). */relaxed?: boolean;}+/**+ * #7082: a local caller's strict attempt matches the whole vector (title plus+ * timelines, timeline_entries rows included), and a very common word matched+ * most pages, so ranking read every matching vector until the statement timeout.+ * It now ranks at most this many matching pages, the ones with the lowest page+ * ids, so a cut keeps the same pages on every call for the same data; a term+ * that matches fewer is ranked exactly as before, and a cut attempt is+ * completed by the title-only statement (engine-sql/titles.ts).+ */+let wholeVectorCandidates = 5000;+export function titleWholeVectorCandidates(): number { return wholeVectorCandidates; }+/** Test seam: lower the cap so a small fixture can be cut. */+export function __setTitleWholeVectorCandidatesForTests(n: number): void { wholeVectorCandidates = n; }+export interface SearchTitlesStatement {sql: string;/** `$1` is the websearch text (rebound by the OR fallback); `$2` the exact-title key. */@@ -110,18 +124,28 @@ export function buildSearchTitlesStatement(query: string, opts: SearchOpts | und? `CASE WHEN tp.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries teWHERE te.page_id = tp.page_id) THEN true ELSE false END`: 'false';+ const pageFilters = `${filters}+ ${hardExcludeClause}+ ${visibilityClause}`;+ const capped = !titleOnly;const sql = `- WITH top_pages AS (+ WITH ${capped ? `candidates AS MATERIALIZED (+ SELECT p.id FROM pages p+ JOIN sources s ON s.id = p.source_id+ WHERE ${matchSql}+ ${pageFilters}+ ORDER BY p.id+ LIMIT ${wholeVectorCandidates + 1}+ ),+ ` : ''}top_pages AS (SELECT p.slug, p.id AS page_id, p.title, p.type, p.source_id,p.effective_date, p.effective_date_source, p.updated_at, p.body_storage, p.body_generation,(lower(btrim(p.title, ' "''' || chr(160))) = $2) AS exact_title,${rankSql} * ${sourceFactorCase} AS score- FROM pages p+ FROM ${capped ? `(SELECT id FROM candidates ORDER BY id LIMIT ${wholeVectorCandidates}) c JOIN pages p ON p.id = c.id` : 'pages p'}JOIN sources s ON s.id = p.source_id- WHERE ${matchSql}- ${filters}- ${hardExcludeClause}- ${visibilityClause}+ WHERE ${capped ? 'true' : `${matchSql}+ ${pageFilters}`}ORDER BY exact_title DESC, score DESC, p.id ASCLIMIT $3 OFFSET $4)@@ -132,7 +156,8 @@ export function buildSearchTitlesStatement(query: string, opts: SearchOpts | undCOALESCE(rep.chunk_text, '') AS chunk_text,COALESCE(rep.chunk_source, 'compiled_truth') AS chunk_source,tp.score,- ${staleSql} AS stale+ ${staleSql} AS stale,+ ${capped ? `(SELECT count(*) FROM candidates) > ${wholeVectorCandidates}` : 'false'} AS candidates_cutFROM top_pages tpLEFT JOIN LATERAL (SELECT cc.id, cc.chunk_index, cc.chunk_text, cc.chunk_sourcediff --git a/test/e2e/title-arm-postgres.test.ts b/test/e2e/title-arm-postgres.test.tsindex a0bfd3b8c1..7bd6cd918a 100644--- a/test/e2e/title-arm-postgres.test.ts+++ b/test/e2e/title-arm-postgres.test.ts@@ -12,7 +12,7 @@*/import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import type { PostgresEngine } from '../../src/core/postgres-engine.ts';-import { buildSearchTitlesStatement } from '../../src/core/search/title-statement.ts';+import { __setTitleWholeVectorCandidatesForTests, buildSearchTitlesStatement, titleWholeVectorCandidates } from '../../src/core/search/title-statement.ts';import { installFixtureChunks } from '../helpers/page-projection.ts';import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';import { KEYWORD_STATEMENT_TIMEOUT_DEFAULT_MS } from '../../src/core/postgres-engine/keyword-statement-timeout.ts';@@ -22,6 +22,7 @@ await registerPostgresTests(() => import('../search/title-arm-exact.test.ts'));const POSTGRES_DIALECT = { statementTimeout: `${KEYWORD_STATEMENT_TIMEOUT_DEFAULT_MS}ms`, relaxedPrefersIndex: true, staleProbe: false };const REMOTE = { requireSafeChunks: true, excludePrivate: true, sourceId: 'default', limit: 50 };+const DEFAULT_CAP = titleWholeVectorCandidates();describe('remote title arm plan (postgres)', () => {let engine: PostgresEngine;@@ -86,4 +87,28 @@ describe('remote title arm plan (postgres)', () => {expect(viaEngine.map((r) => r.slug)).toEqual(direct.map((r) => r.slug));expect(viaEngine.length).toBe(4);});++ // #7082: a local caller's whole-vector attempt ranks at most titleWholeVectorCandidates() pages, chosen by page id.+ test('a cut local attempt keeps the lowest page ids, reports candidate_budget and leads with the title rows', async () => {+ const local = async () => {+ const cut: string[] = [];+ const rows = await engine.searchTitles('acme', { limit: 50, onKeywordIncomplete: r => { cut.push(r); } });+ return { slugs: rows.map(r => r.slug), cut };+ };+ // Fold page-1's timeline_entries row (it names acme) into its search_vector, as the next title write does.+ await engine.executeRaw("UPDATE pages SET title = title WHERE slug = 'notes/page-1'");+ const titled = (await engine.searchTitles('acme', REMOTE)).map(r => r.slug);+ const whole = await local();+ expect(whole.cut).toEqual([]);+ expect(whole.slugs.sort()).toEqual([...titled, 'notes/page-1'].sort());+ __setTitleWholeVectorCandidatesForTests(2);+ try {+ const first = await local();+ expect(first.cut).toEqual(['candidate_budget']);+ expect(first.slugs).toEqual([...titled, 'notes/page-1']);+ for (let i = 0; i < 3; i++) expect((await local()).slugs).toEqual(first.slugs);+ } finally {+ __setTitleWholeVectorCandidatesForTests(DEFAULT_CAP);+ }+ });});diff --git a/test/fixtures/retrieval-quality/timeline-titles/corpus.ts b/test/fixtures/retrieval-quality/timeline-titles/corpus.tsnew file mode 100644index 0000000000..ba3aee9a72--- /dev/null+++ b/test/fixtures/retrieval-quality/timeline-titles/corpus.ts@@ -0,0 +1,55 @@+/**+ * Pages with titles and dated timelines for the title arm's local-caller+ * predicate (#7082). Every name is a placeholder. Several pages hold a query's+ * words only in their timeline, so the title arm (which reads title lexemes)+ * and the keyword arm (which reads compiled-truth and timeline chunks) find+ * different pages for the same query. Data only: the tests seed it.+ */+export interface TimelineTitlePage {+ slug: string;+ type: string;+ title: string;+ compiled_truth: string;+ timeline: string;+}++export const TIMELINE_TITLE_PAGES: TimelineTitlePage[] = [+ { slug: 'people/alice-example', type: 'person', title: 'Alice Example', compiled_truth: 'Founder and chief executive of Acme Example.',+ timeline: '- 2026-03-02: board meeting at Acme Example\n- 2026-05-14: zircon quarry visit with Bob Example\n- 2026-09-01: quarterly planning review' },+ { slug: 'people/bob-example', type: 'person', title: 'Bob Example', compiled_truth: 'Geologist who advises on quarry sites.',+ timeline: '- 2026-05-14: zircon quarry visit\n- 2026-07-20: shared the board meeting notes' },+ { slug: 'people/carol-example', type: 'person', title: 'Carol Example', compiled_truth: 'Investor at Fund A.',+ timeline: '- 2026-04-11: fundraise call with Alice Example\n- 2026-08-30: board meeting observer' },+ { slug: 'people/dan-example', type: 'person', title: 'Dan Example', compiled_truth: 'Engineer on the settlement team.',+ timeline: '- 2026-06-03: kelmit protocol rollout\n- 2026-10-07: on call for the launch' },+ { slug: 'companies/acme-example', type: 'company', title: 'Acme Example', compiled_truth: 'Industrial company that runs survey projects.',+ timeline: '- 2026-03-02: board meeting approved the zircon quarry budget\n- 2026-10-07: launch of the survey program' },+ { slug: 'projects/zircon-quarry-survey', type: 'project', title: 'Zircon Quarry Survey', compiled_truth: 'Survey of the northern quarry for zircon deposits.',+ timeline: '- 2026-05-14: site visit' },+ { slug: 'meetings/2026-03-02-acme-board-meeting', type: 'meeting', title: 'Acme Board Meeting, March', compiled_truth: 'Minutes of the March board meeting.', timeline: '' },+ { slug: 'meetings/2026-07-20-board-meeting', type: 'meeting', title: 'Board Meeting, July', compiled_truth: 'Minutes: hiring and the planning calendar.', timeline: '' },+ { slug: 'concepts/quarterly-planning', type: 'concept', title: 'Quarterly Planning', compiled_truth: 'How the team runs its quarterly planning review.',+ timeline: '- 2026-09-01: planning review held' },+ { slug: 'deals/acme-example-fundraise', type: 'deal', title: 'Acme Example Fundraise', compiled_truth: 'Series A fundraise led by Fund A.',+ timeline: '- 2026-04-11: first call' },+ { slug: 'notes/nastel-ledger', type: 'note', title: 'Nastel Ledger', compiled_truth: 'Ledger design notes for settlement.',+ timeline: '- 2026-06-03: adopted the kelmit protocol for settlement' },+ { slug: 'notes/october-launch-checklist', type: 'note', title: 'October Launch Checklist', compiled_truth: 'Checklist for the survey program launch.',+ timeline: '- 2026-10-07: launch day' },+ { slug: 'notes/hiring-plan', type: 'note', title: 'Hiring Plan', compiled_truth: 'Roles to fill before the next planning cycle.', timeline: '' },+ { slug: 'notes/settlement-design', type: 'note', title: 'Settlement Design', compiled_truth: 'How payments settle between ledgers.', timeline: '' },+];++/** Queries whose words sit in titles, in timelines, or in both. */+export const TIMELINE_TITLE_QUERIES: string[] = [+ 'zircon quarry',+ 'board meeting',+ 'Acme Example',+ 'kelmit protocol',+ 'quarterly planning review',+ 'launch',+ 'Alice Example',+ 'fundraise',+ 'Nastel Ledger',+ '2026-10-07 launch',+];diff --git a/test/search/title-arm-candidate-cap-7082.test.ts b/test/search/title-arm-candidate-cap-7082.test.tsnew file mode 100644index 0000000000..5e0909531c--- /dev/null+++ b/test/search/title-arm-candidate-cap-7082.test.ts@@ -0,0 +1,141 @@+/**+ * #7082: a local caller's strict title attempt matches the whole+ * pages.search_vector (title plus timelines, timeline_entries rows included),+ * and a very common word matched most pages, so ranking read every matching+ * vector until the statement timeout. The attempt now ranks at most+ * titleWholeVectorCandidates() matching pages. A rarer term keeps its exact+ * whole-vector result, which is the only lexical path to a timeline_entries row+ * no chunk holds; a cut attempt reports candidate_budget and leads with the+ * title-only rows remote callers get.+ *+ * Hermetic PGLite with an empty gateway env, so hybridSearch never embeds.+ */+import { describe, test, expect, beforeAll, afterAll, afterEach } from 'bun:test';+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';+import { hybridSearch } from '../../src/core/search/hybrid.ts';+import { configureGateway, resetGateway } from '../../src/core/ai/gateway.ts';+import { __setTitleWholeVectorCandidatesForTests, titleWholeVectorCandidates } from '../../src/core/search/title-statement.ts';+import { installFixtureChunks } from '../helpers/page-projection.ts';+import { TIMELINE_TITLE_PAGES, TIMELINE_TITLE_QUERIES } from '../fixtures/retrieval-quality/timeline-titles/corpus.ts';++let engine: PGLiteEngine;+const defaultCap = titleWholeVectorCandidates();++beforeAll(async () => {+ configureGateway({ embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: 1536, env: {} });+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ for (const page of TIMELINE_TITLE_PAGES) {+ await engine.putPage(page.slug, { type: page.type, title: page.title, compiled_truth: page.compiled_truth, timeline: page.timeline });+ await installFixtureChunks(engine, page.slug, [+ { chunk_index: 0, chunk_text: page.compiled_truth, chunk_source: 'compiled_truth' },+ ...(page.timeline ? [{ chunk_index: 1, chunk_text: page.timeline, chunk_source: 'timeline' as const }] : []),+ ]);+ }+ // A row written straight to timeline_entries (meeting extraction, enrichment): no chunk holds its text,+ // and the page vector folds it in on the next title or timeline write.+ await engine.putPage('people/erin-example', { type: 'person', title: 'Erin Example', compiled_truth: 'Plant engineer.', timeline: '' });+ await installFixtureChunks(engine, 'people/erin-example', [{ chunk_index: 0, chunk_text: 'Plant engineer.', chunk_source: 'compiled_truth' }]);+ await engine.addTimelineEntry('people/erin-example', { date: '2026-10-08', summary: 'Toured the qendril smelter', source: 'meetings/2026-10-08-plant-tour' });+ await engine.executeRaw("UPDATE pages SET title = title WHERE slug = 'people/erin-example'");+});++afterEach(() => __setTitleWholeVectorCandidatesForTests(defaultCap));++afterAll(async () => {+ await engine.disconnect();+ resetGateway();+});++const titles = async (q: string, remote: boolean) => {+ const cut: string[] = [];+ const rows = await engine.searchTitles(q, { limit: 10, onKeywordIncomplete: r => { cut.push(r); }, ...(remote ? { excludePrivate: true } : {}) });+ return { slugs: rows.map(r => r.slug), cut };+};++const words = (q: string) => q.toLowerCase().split(/\s+/);+const timelineOnly = (q: string) => TIMELINE_TITLE_PAGES+ .filter(p => words(q).every(w => p.timeline.toLowerCase().includes(w)) && !words(q).every(w => p.title.toLowerCase().includes(w)))+ .map(p => p.slug);++describe('title arm: whole-vector candidate cap for local callers (#7082)', () => {+ for (const q of TIMELINE_TITLE_QUERIES) {+ test(`under the cap, the local title arm keeps every timeline-only match: "${q}"`, async () => {+ const { slugs, cut } = await titles(q, false);+ for (const slug of timelineOnly(q)) expect(slugs).toContain(slug);+ expect(cut).toEqual([]);+ });++ test(`hybrid search returns every timeline-only match: "${q}"`, async () => {+ const top = (await hybridSearch(engine, q, { limit: 10 })).map(r => r.slug);+ for (const slug of timelineOnly(q)) expect(top).toContain(slug);+ });+ }++ test('a term under the cap keeps its timeline matches and reports no cut', async () => {+ const { slugs, cut } = await titles('zircon quarry', false);+ expect(slugs[0]).toBe('projects/zircon-quarry-survey');+ for (const slug of ['people/alice-example', 'people/bob-example', 'companies/acme-example']) expect(slugs).toContain(slug);+ expect(cut).toEqual([]);+ });++ test('a timeline_entries row no chunk holds is still found by a local search', async () => {+ expect((await titles('qendril smelter', false)).slugs).toEqual(['people/erin-example']);+ expect((await hybridSearch(engine, 'qendril smelter', { limit: 10 })).map(r => r.slug)).toContain('people/erin-example');+ });++ test('remote callers keep the title-only match', async () => {+ expect((await titles('qendril smelter', true)).slugs).toEqual([]);+ expect((await titles('zircon quarry', true)).slugs).toEqual(['projects/zircon-quarry-survey']);+ });++ test('a cut attempt reports candidate_budget and leads with every title match', async () => {+ const remote = await titles('board meeting', true);+ expect(remote.slugs.length).toBe(2);+ __setTitleWholeVectorCandidatesForTests(2);+ const { slugs, cut } = await titles('board meeting', false);+ expect(cut).toEqual(['candidate_budget']);+ expect(slugs.slice(0, remote.slugs.length)).toEqual(remote.slugs);+ expect(new Set(slugs).size).toBe(slugs.length);+ expect(slugs.length).toBeLessThanOrEqual(remote.slugs.length + 3);+ });++ test('a cut keeps the matching pages with the lowest page ids, the same pages on every call', async () => {+ const titled = (await titles('board meeting', true)).slugs;+ const lowest = (await engine.executeRaw<{ slug: string }>(+ "SELECT slug FROM pages WHERE search_vector @@ websearch_to_tsquery('english', 'board meeting') ORDER BY id LIMIT 2")).map(r => r.slug);+ __setTitleWholeVectorCandidatesForTests(2);+ const first = (await titles('board meeting', false)).slugs;+ expect(first.slice(titled.length).sort()).toEqual(lowest.filter(s => !titled.includes(s)).sort());+ for (let i = 0; i < 3; i++) expect((await titles('board meeting', false)).slugs).toEqual(first);+ });++ test('a cut attempt pages through the merged rows once', async () => {+ __setTitleWholeVectorCandidatesForTests(2);+ const all = (await engine.searchTitles('board meeting', { limit: 10 })).map(r => r.slug);+ expect(all.length).toBeGreaterThan(2);+ for (const offset of [1, 2, all.length - 1]) {+ const cut: string[] = [];+ const page = (await engine.searchTitles('board meeting', { limit: 2, offset, onKeywordIncomplete: r => { cut.push(r); } })).map(r => r.slug);+ expect({ offset, page, cut }).toEqual({ offset, page: all.slice(offset, offset + 2), cut: ['candidate_budget'] });+ }+ });++ test('hybrid search reports the cut as keyword_candidates_incomplete', async () => {+ __setTitleWholeVectorCandidatesForTests(2);+ let degraded: unknown[] = [];+ await hybridSearch(engine, 'board meeting', { limit: 10, onMeta: m => { degraded = m.degraded ?? []; } });+ expect(degraded).toContainEqual({ stage: 'keyword_candidates_incomplete', reason: 'candidate_budget' });+ __setTitleWholeVectorCandidatesForTests(defaultCap);+ await hybridSearch(engine, 'board meeting', { limit: 10, onMeta: m => { degraded = m.degraded ?? []; } });+ expect(degraded).not.toContainEqual({ stage: 'keyword_candidates_incomplete', reason: 'candidate_budget' });+ });++ test('the cap never touches the title-only statement', async () => {+ __setTitleWholeVectorCandidatesForTests(1);+ const { slugs, cut } = await titles('board meeting', true);+ expect(slugs.length).toBe(2);+ expect(cut).toEqual([]);+ });+});