Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
Broad keyword searches from MCP clients stop at the candidate bound instead of timing out (#7082)
garrytana84df80bbf52
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 927b2836bc..8214bb6555 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,19 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++### Broad keyword searches from MCP clients answer again instead of timing out++**A remote caller's hybrid search no longer times out its keyword arm on a common word.** Hybrid search ranks at most 50,000 matching chunks, but for a remote (MCP) caller the planner still read every matching chunk before that bound applied, because the private-page checks made the page join look expensive. On a large Postgres brain a word in most chunks then ran past the 8 s keyword statement timeout, the search answered from vectors alone (`keyword_arm_failed`), and the serving process skipped the keyword arm for that query for the next 10 minutes.++- When a remote caller sets no type, frontmatter or date filter and its source scope covers every live source (the usual MCP scope), the keyword statement now reads matching chunks straight from the full-text index and checks each one's page by key, with every visibility, member-access, quarantine, deletion and archived-source filter applied before the bound. It stops reading once it has 50,000 candidates.+- Measured on a synthetic 2.1M-chunk Postgres brain (warm, median): a word in 95% of chunks went from 9.9 s to 0.65 s, a word in a third of chunks from 1.5 s to 0.76 s, and a word in 5% of chunks from 0.93 s to 0.70 s. Rare terms are unchanged (about 40 ms), and their results are identical.+- Results are identical whenever a query matches fewer than 50,000 chunks. Above that, the 50,000-chunk sample is drawn in index order instead of another plan's order, so which pages of a very broad term rank can change; the result was already a sample there (`keyword_candidates_incomplete`).+- Local callers, typed or dated searches and narrower source scopes run the same statement as before.+- The keyword-mode page statement and the strict page count now run with the same 32 MB `work_mem` as the keyword arm, so a common word's bitmap no longer goes lossy: 5.8 s to 3.7 s for keyword paging and 0.68 s to 0.26 s for a count on the same brain.+- `scripts/bench/retrieval-scale/keyword-broad.ts` times the keyword statements for broad, medium and rare terms and can diff two checkouts' results; `run.ts` takes `--queries` and `--no-query-cache`.+## [0.60.699.0] - 2026-11-12### The Ubicloud gate VMs run a current gitdiff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.mdindex 52f553f07a..9332cfc5b5 100644--- a/docs/guides/troubleshooting.md+++ b/docs/guides/troubleshooting.md@@ -509,11 +509,15 @@ when the inert setting is still present.broad `gbrain search` or `gbrain query` with common words takes about 8seconds and comes back with only vector hits; search metadata carries`keyword_arm_failed`, and the server log says `canceling statement due to-statement timeout` (SQLSTATE 57014). The strict keyword statement ranks every-matching chunk before its `LIMIT`, so its cost grows with how common the terms-are, and the commonest terms cannot finish inside the keyword statement bound.-An arm that timed out is not evidence of absence: the pages exist, the arm-could not rank them in time.+statement timeout` (SQLSTATE 57014). Keyword-mode paging and the strict page+count rank every matching page, so their cost grows with how common the terms+are. Hybrid search's keyword arm ranks at most 50,000 matching chunks (its metadata+carries `keyword_candidates_incomplete` when that bound applies), and when the+call sets no type, frontmatter or date filter and no source scope narrower than+every live source, it stops reading matches at that bound. (A remote MCP+caller's arm used to read every match before the bound applied, so a term in+most chunks timed out.) An arm that timed out is not evidence of absence: the+pages exist, the arm could not rank them in time.**Say to your agent:** *"Check whether keyword search is hitting its timelimit"* — the agent runs `gbrain doctor` and reads the `keyword_plan` check.@@ -546,12 +550,10 @@ range (exit 2, nothing written). It is read once per connection, so:3. Confirm with `gbrain doctor --only keyword_plan --json`: `details.timeout_ms`shows the value in effect.-A longer bound buys time, not a cheaper statement: a query whose terms match-most of the corpus still reads and ranks every match, so prefer narrowing with-`--source-id` (MCP `source_id`) or `match: "keyword"` paging, and follow-[#6495](https://github.com/garrytan/gbrain/issues/6495) for the bounded-candidate set that will answer broad terms from a capped set instead of-failing open.+A longer bound buys time, not a cheaper statement: keyword-mode paging and the+strict count still read every match of a term that matches most of the corpus,+so prefer narrowing with `--source-id` (MCP `source_id`), `--types` or+`--since`.**A timed-out keyword search is skipped for 10 minutes.** After hybrid search'skeyword statement hits the bound, a serving process (`gbrain serve`, an MCPdiff --git a/llms-full.txt b/llms-full.txtindex 17bc03954f..67d8f0e949 100644--- a/llms-full.txt+++ b/llms-full.txt@@ -9798,11 +9798,15 @@ when the inert setting is still present.broad `gbrain search` or `gbrain query` with common words takes about 8seconds and comes back with only vector hits; search metadata carries`keyword_arm_failed`, and the server log says `canceling statement due to-statement timeout` (SQLSTATE 57014). The strict keyword statement ranks every-matching chunk before its `LIMIT`, so its cost grows with how common the terms-are, and the commonest terms cannot finish inside the keyword statement bound.-An arm that timed out is not evidence of absence: the pages exist, the arm-could not rank them in time.+statement timeout` (SQLSTATE 57014). Keyword-mode paging and the strict page+count rank every matching page, so their cost grows with how common the terms+are. Hybrid search's keyword arm ranks at most 50,000 matching chunks (its metadata+carries `keyword_candidates_incomplete` when that bound applies), and when the+call sets no type, frontmatter or date filter and no source scope narrower than+every live source, it stops reading matches at that bound. (A remote MCP+caller's arm used to read every match before the bound applied, so a term in+most chunks timed out.) An arm that timed out is not evidence of absence: the+pages exist, the arm could not rank them in time.**Say to your agent:** *"Check whether keyword search is hitting its timelimit"* — the agent runs `gbrain doctor` and reads the `keyword_plan` check.@@ -9835,12 +9839,10 @@ range (exit 2, nothing written). It is read once per connection, so:3. Confirm with `gbrain doctor --only keyword_plan --json`: `details.timeout_ms`shows the value in effect.-A longer bound buys time, not a cheaper statement: a query whose terms match-most of the corpus still reads and ranks every match, so prefer narrowing with-`--source-id` (MCP `source_id`) or `match: "keyword"` paging, and follow-[#6495](https://github.com/garrytan/gbrain/issues/6495) for the bounded-candidate set that will answer broad terms from a capped set instead of-failing open.+A longer bound buys time, not a cheaper statement: keyword-mode paging and the+strict count still read every match of a term that matches most of the corpus,+so prefer narrowing with `--source-id` (MCP `source_id`), `--types` or+`--since`.**A timed-out keyword search is skipped for 10 minutes.** After hybrid search'skeyword statement hits the bound, a serving process (`gbrain serve`, an MCPdiff --git a/scripts/bench/retrieval-scale/keyword-broad.ts b/scripts/bench/retrieval-scale/keyword-broad.tsnew file mode 100644index 0000000000..44edc03080--- /dev/null+++ b/scripts/bench/retrieval-scale/keyword-broad.ts@@ -0,0 +1,113 @@+#!/usr/bin/env bun+/**+ * Broad-term keyword probe (#7082): times the Postgres keyword statements for+ * broad, medium and rare terms on a corpus.ts brain and prints their plans.+ * Opt-in bench code, never run in CI.+ *+ * bun scripts/bench/retrieval-scale/keyword-broad.ts --url <direct postgres url> [--terms a,b,c] [--n 10]+ * [--calls arm,arm_phrase,arm_remote,arm_all_sources,arm_small_source,arm_person,pages,pages_remote,count] [--explain <dir>] [--label <name>] [--out results.jsonl]+ * [--rows rows.jsonl] [--tree <checkout>] [--no-share]+ *+ * Without --terms it picks one term per band from a 1% sample of chunks+ * (broad: 20-40% of chunks, medium: 1-5%, rare: under 0.05%) and reports each+ * term's exact share. For every term it times, --n times each, the hybrid+ * keyword arm (searchKeyword with orFallback, limit 50: plain, with the+ * keyword phrase factor, and with an MCP caller's excludePrivate and+ * requireSafeChunks, alone, scoped to every live source as the search+ * operation scopes it, and narrowed to the smallest source or to person pages), the keyword-mode page statement (searchKeywordPages,+ * limit 20, local and MCP scope) and the strict page count, with the configured statement timeout+ * lifted to 120 s so a slow statement is measured instead of cancelled.+ * --explain writes EXPLAIN (ANALYZE, BUFFERS) of each captured statement,+ * run with the same transaction-local settings. --tree runs another+ * checkout's engine (for example master) against the same brain; --rows writes+ * each call's first result list (slug, chunk id, score) so two trees can be diffed.+ * --no-share skips counting each term's matches, which reads every match and+ * would warm the cache ahead of a cold measurement.+ */+import { appendFileSync, mkdirSync, writeFileSync } from 'node:fs';+import { join, resolve } from 'node:path';+import postgres from '#postgres';++const argv = process.argv.slice(2);+const flag = (name: string, fallback?: string) => { const i = argv.indexOf(name); return i >= 0 ? argv[i + 1] : fallback; };+const url = flag('--url')!;+const N = Number(flag('--n', '10'));+const explainDir = flag('--explain');+const out = flag('--out');+const tree = resolve(flag('--tree', resolve(import.meta.dir, '..', '..', '..'))!);+const label = flag('--label', 'tree');+const rowsOut = flag('--rows');+const sql = postgres(url, { max: 2, prepare: false, onnotice: () => {} });++async function pickTerms(): Promise<Array<{ band: string; term: string }>> {+ const stats = await sql.unsafe("SELECT word, ndoc FROM ts_stat('SELECT search_vector FROM content_chunks TABLESAMPLE SYSTEM (1) REPEATABLE (7)') ORDER BY ndoc DESC") as Array<{ word: string; ndoc: number }>;+ const [{ n }] = await sql.unsafe('SELECT count(*)::int AS n FROM content_chunks TABLESAMPLE SYSTEM (1) REPEATABLE (7)') as any;+ const share = (w: { ndoc: number }) => Number(w.ndoc) / n;+ const pick = (lo: number, hi: number) => stats.find(w => /^[a-z]{4,}$/.test(w.word) && share(w) >= lo && share(w) < hi)?.word;+ return [['broad', pick(0.2, 0.4)], ['medium', pick(0.01, 0.05)], ['rare', pick(0.00005, 0.0005)]]+ .filter(([, t]) => t).map(([band, term]) => ({ band: band!, term: term! }));+}++const terms = flag('--terms') ? flag('--terms')!.split(',').map(t => ({ band: 'given', term: t })) : await pickTerms();+const [{ chunks }] = await sql.unsafe('SELECT count(*)::int AS chunks FROM content_chunks') as any;+const liveSources = (await sql.unsafe('SELECT id FROM sources WHERE NOT archived ORDER BY id') as Array<{ id: string }>).map(r => r.id);++const { createEngine } = await import(join(tree, 'src/core/engine-factory.ts'));+const engine = await createEngine({ engine: 'postgres' });+await engine.connect({ database_url: url });+const target = engine as unknown as Record<string, any>;+target.keywordStatementTimeoutSetting = async () => '120s';+let captured: Array<{ sql: string; params: unknown[]; settings: Record<string, unknown> }> = [];+const original = target.withScopedReadTransaction;+target.withScopedReadTransaction = function (ids: unknown, id: unknown, callback: (tx: any) => unknown, opts: any) {+ return original.call(this, ids, id, (tx: any) => callback(new Proxy(tx, {+ get(t, prop, receiver) {+ if (prop !== 'unsafe') return Reflect.get(t, prop, receiver);+ return (text: string, params: unknown[]) => { captured.push({ sql: text, params, settings: opts?.settings ?? {} }); return t.unsafe(text, params); };+ },+ })), opts);+};++const pct = (xs: number[], q: number) => { const s = [...xs].sort((a, b) => a - b); return s[Math.min(s.length - 1, Math.ceil(q * s.length) - 1)]!; };+const calls: Record<string, (term: string) => Promise<unknown[] | number>> = {+ arm: term => engine.searchKeyword(term, { limit: 50, orFallback: true }),+ arm_phrase: term => engine.searchKeyword(term, { limit: 50, orFallback: true, keywordPhrase: true }),+ arm_remote: term => engine.searchKeyword(term, { limit: 50, orFallback: true, excludePrivate: true, requireSafeChunks: true }),+ arm_all_sources: term => engine.searchKeyword(term, { limit: 50, orFallback: true, excludePrivate: true, requireSafeChunks: true, sourceIds: liveSources }),+ arm_small_source: term => engine.searchKeyword(term, { limit: 50, orFallback: true, excludePrivate: true, requireSafeChunks: true, sourceIds: ['source-c'] }),+ arm_person: term => engine.searchKeyword(term, { limit: 50, orFallback: true, excludePrivate: true, requireSafeChunks: true, types: ['person'] }),+ pages: term => engine.searchKeywordPages(term, undefined, { limit: 20 }),+ pages_remote: term => engine.searchKeywordPages(term, { excludePrivate: true, requireSafeChunks: true }, { limit: 20 }),+ count: term => engine.countKeywordPages(term, {}),+};+const only = flag('--calls')?.split(',');+if (only) for (const name of Object.keys(calls)) if (!only.includes(name)) delete calls[name];+if (explainDir) mkdirSync(explainDir, { recursive: true });+for (const { band, term } of terms) {+ const [{ m }] = argv.includes('--no-share') ? [{ m: null }] : await sql.unsafe("SELECT count(*)::int AS m FROM content_chunks WHERE search_vector @@ websearch_to_tsquery('english', $1)", [term]) as any;+ for (const [name, call] of Object.entries(calls)) {+ const ms: number[] = [];+ let rows = 0, err: string | undefined, first: unknown[] | undefined;+ captured = [];+ for (let i = 0; i < N; i++) {+ const t = performance.now();+ try { const r = await call(term); rows = Array.isArray(r) ? r.length : Number(r); first ??= Array.isArray(r) ? r.map((x: any) => [x.slug, x.chunk_id, x.score]) : [r]; } catch (e) { err = (e as Error).message.slice(0, 80); }+ ms.push(performance.now() - t);+ }+ const row = { label, band, term, chunks, matched: m, share: m === null ? null : Math.round((1000 * m) / chunks) / 10, call: name, n: N, first_ms: Math.round(ms[0]!), p50_ms: Math.round(pct(ms, 0.5)), p95_ms: Math.round(pct(ms, 0.95)), rows, err };+ console.log(JSON.stringify(row));+ if (out) appendFileSync(out, JSON.stringify(row) + '\n');+ if (rowsOut) appendFileSync(rowsOut, JSON.stringify({ label, term, call: name, rows: first }) + '\n');+ if (explainDir && captured[0]) {+ const c = captured[0];+ const plan = await sql.begin(async tx => {+ await tx.unsafe("SET LOCAL jit = off; SET LOCAL statement_timeout = '120s'");+ for (const [k, v] of Object.entries(c.settings)) if (k !== 'statement_timeout') await tx.unsafe('SET LOCAL ' + k + " = '" + String(v) + "'");+ return tx.unsafe('EXPLAIN (ANALYZE, BUFFERS) ' + c.sql, c.params as any[]);+ }) as Array<Record<string, string>>;+ writeFileSync(join(explainDir, label + '-' + band + '-' + term.replace(/\W+/g, '_') + '-' + name + '.txt'), c.sql + '\n\n-- params: ' + JSON.stringify(c.params).slice(0, 400) + '\n\n' + plan.map(r => r['QUERY PLAN']).join('\n') + '\n');+ }+ }+}+await engine.disconnect();+await sql.end();diff --git a/scripts/bench/retrieval-scale/run.ts b/scripts/bench/retrieval-scale/run.tsindex 2e5270c085..a13b0f3a92 100644--- a/scripts/bench/retrieval-scale/run.ts+++ b/scripts/bench/retrieval-scale/run.ts@@ -8,7 +8,8 @@* [--direct-url postgresql://...:5434/bench_retrieval_300k] [--n 20] [--warmup 3] \* [--ops search,query,get,recall] [--modes cold,warm] [--search-mode tokenmax] \* [--embed-ms 0] [--rerank-ms 0] [--toxiproxy http://127.0.0.1:8474 --rtt-ms 40] [--trace] [--serve-cpu-prof <dir>] \- * [--tree <checkout>] [--keep] [--label <name>] [--out results.jsonl] [--results calls.jsonl]+ * [--tree <checkout>] [--keep] [--label <name>] [--out results.jsonl] [--results calls.jsonl] \+ * [--queries 'a|b c'] [--no-query-cache]** --toxiproxy adds --rtt-ms of round-trip latency to its `pooler` proxy (vm-setup.sh* creates it on 127.0.0.1:6544 in front of PgBouncer); point --url at 6544 to use it.@@ -30,7 +31,9 @@* the path a shell-out agent pays once a serve is running.** Queries are 2-4 generated words from a random page body plus, for half of- * them, a word of its title: synthetic text only. Rows carry timings and+ * them, a word of its title: synthetic text only. --queries replaces them with+ * your own (every op cycles through the list); --no-query-cache turns off the+ * semantic query cache so a repeated query runs its statements again. Rows carry timings and* counts. Prints a Markdown table and appends JSONL rows to --out.*/import { appendFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';@@ -97,6 +100,7 @@ async function main() {};const set = await runCli(['config', 'set', 'search.mode', searchMode]);if (set.code !== 0) log('config set search.mode failed: ' + set.stderr.slice(0, 300));+ if (has('--no-query-cache')) await runCli(['config', 'set', 'search.cache.enabled', 'false']);// Query set and get targets, drawn deterministically from the corpus.const sql = postgres(directUrl, { max: 1, prepare: false, onnotice: () => {} });@@ -108,7 +112,8 @@ async function main() {const titleWord = p.title.split(/\s+/).find(w => w.length > 3);return [...(i % 2 === 0 && titleWord ? [titleWord.toLowerCase()] : []), ...words.slice(at, at + k)].join(' ');});- const q = (i: number) => queries[i % queries.length]!;+ const given = flag('--queries')?.split('|');+ const q = (i: number) => given ? given[i % given.length]! : queries[i % queries.length]!;// Each op reads its own slice of the query set, so no op reuses another's per-process query embedding.const opBase = (op: string) => ops.indexOf(op) * 7 + 1;const pk = (i: number) => picks[i % picks.length]!;diff --git a/src/core/engine-sql/keyword-pages.ts b/src/core/engine-sql/keyword-pages.tsindex 49c734849e..c5efe0aa86 100644--- a/src/core/engine-sql/keyword-pages.ts+++ b/src/core/engine-sql/keyword-pages.ts@@ -13,8 +13,12 @@ import { rowToSearchResult } from '../utils.ts';import { buildKeywordCountStatement, buildKeywordHandlePagesStatement, buildKeywordPagesStatement, type KeywordPageWindow } from '../search/keyword-statement.ts';import type { ScopedReadRunner } from './cjk-search.ts';-/** Engine session settings: Postgres bounds each read with a statement timeout; PGLite has none. */-export interface KeywordPagesDialect { statementTimeout?: string }+/**+ * Engine session settings: Postgres bounds each read with a statement timeout and runs it at the keyword+ * statement's work_mem (KEYWORD_STATEMENT_SETTINGS: at the 4 MB default a common term's bitmap went lossy and+ * its sort spilled to disk); PGLite has neither.+ */+export interface KeywordPagesDialect { statementTimeout?: string; statementSettings?: Readonly<Record<string, string>> }/** Distinct pages matching the strict keyword query, at most KEYWORD_COUNT_CAP + 1. */export async function countKeywordPages(@@ -24,7 +28,7 @@ export async function countKeywordPages(if (!statement) return 0;const timeout = dialect.statementTimeout && dialect.timeoutMs ? `${Math.max(1, Math.floor(dialect.timeoutMs))}ms` : dialect.statementTimeout;const rows = await scoped(async exec => (await exec.unsafe<{ n: number }>(statement.sql, statement.params)).rows,- timeout ? { statement_timeout: timeout } : undefined);+ timeout ? { statement_timeout: timeout, ...dialect.statementSettings } : undefined);return Number(rows[0]?.n ?? 0);}@@ -45,6 +49,6 @@ export async function searchHandlePages(async function readPages(scoped: ScopedReadRunner, statement: { sql: string; params: unknown[] } | null, dialect: KeywordPagesDialect): Promise<SearchResult[]> {if (!statement) return [];const rows = await scoped(async exec => (await exec.unsafe(statement.sql, statement.params)).rows,- dialect.statementTimeout ? { statement_timeout: dialect.statementTimeout } : undefined);+ dialect.statementTimeout ? { statement_timeout: dialect.statementTimeout, ...dialect.statementSettings } : undefined);return rows.map(rowToSearchResult);}diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.tsindex a88d2376cb..403928aa34 100644--- a/src/core/postgres-engine.ts+++ b/src/core/postgres-engine.ts@@ -123,7 +123,7 @@ import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, bimport { commonOrTokens, narrowedOrFallbackText, readOrTokenShares } from './search/keyword-or-narrow.ts';import { keywordArmRows } from './search/keyword-identifier-probe.ts';import { RARE_TERM_CHUNK_CAP, rankOrFallbackByRarity, rareTermPagesSql, type RareTermPages } from './search/keyword-or-rarity.ts';-import { buildKeywordChunkMatch, KEYWORD_RANK_CANDIDATES, KEYWORD_RANKED_ROWS_CTE, KEYWORD_STATEMENT_SETTINGS } from './search/keyword-chunk-match.ts';+import { buildKeywordChunkMatch, keywordCandidatesSql, KEYWORD_RANK_CANDIDATES, KEYWORD_RANKED_ROWS_CTE, KEYWORD_STATEMENT_SETTINGS } from './search/keyword-chunk-match.ts';import { privatePagesFilterFragment, privateSnapshotFilterFragment, privateLinkOriginFilterFragment, privateTimelineEventFilterFragment, privateFactFilterForCaller, privatePagesFilterForCaller } from './search/private-visibility.ts';import { deletedPageProjectionHiddenSql } from './eligibility/sql.ts';import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';@@ -1162,6 +1162,7 @@ export class PostgresEngine implements BrainEngine {params.push(opts.sourceId);sourceClause = `AND p.source_id = $${params.length}`;}+ const scope = sourceClause ? { idx: params.length, many: !!opts?.sourceIds?.length } : null;params.push(innerLimit);const innerLimitParam = `$${params.length}`;params.push(limit);@@ -1180,11 +1181,9 @@ export class PostgresEngine implements BrainEngine {const rankIdx = opts?.rankQuery ? params.push(opts.rankQuery) - 1 : opts?.orFallback ? params.push(params[0]) - 1 : 0;const chunk = buildKeywordChunkMatch(ftsLang, opts?.excludePrivate);- const rawQuery = `- WITH ranked_chunks AS (- SELECT page_id, chunk_id, score, bounded_rows FROM (SELECT *, count(*) OVER () AS bounded_rows, row_number() OVER (PARTITION BY page_id ORDER BY score DESC, chunk_id ASC) AS page_rank FROM (SELECT- p.id as page_id, cc.id as chunk_id,- ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $${rankIdx + 1})) * ${keywordPhraseFactorSql(ftsLang, `$${rankIdx + 1}`, opts?.keywordPhrase === true)} * ${sourceFactorCase} AS score+ const chunkFilters = `${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} ${languageClause} ${symbolKindClause}`;+ const score = `ts_rank(cc.search_vector, websearch_to_tsquery('${ftsLang}', $${rankIdx + 1})) * ${keywordPhraseFactorSql(ftsLang, `$${rankIdx + 1}`, opts?.keywordPhrase === true)} * ${sourceFactorCase}`;+ const pageJoin = `SELECT p.id as page_id, cc.id as chunk_id, ${score} AS scoreFROM ${chunk.from}JOIN pages p ON p.id = cc.page_idJOIN sources s ON s.id = p.source_id@@ -1192,15 +1191,17 @@ export class PostgresEngine implements BrainEngine {${typeClause}${typesClause}${excludeSlugsClause}- ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}- ${languageClause}- ${symbolKindClause}+ ${chunkFilters}${afterDateClause}${beforeDateClause}${sourceClause}${hardExcludeClause}${visibilityClause}- ${chunk.modality}+ ${chunk.modality}`;+ const candidates = keywordCandidatesSql(ftsLang, { excludePrivate: opts?.excludePrivate, pageFiltered: !!(typeClause || typesClause || afterDateClause || beforeDateClause), score, pageJoin, chunkFilters, pageFilters: `${excludeSlugsClause} ${sourceClause} ${hardExcludeClause} ${visibilityClause}`, scope });+ const rawQuery = `+ WITH ranked_chunks AS (+ SELECT page_id, chunk_id, score, bounded_rows FROM (SELECT *, count(*) OVER () AS bounded_rows, row_number() OVER (PARTITION BY page_id ORDER BY score DESC, chunk_id ASC) AS page_rank FROM (${candidates}LIMIT ${KEYWORD_RANK_CANDIDATES}) bounded) per_page WHERE page_rank = 1ORDER BY score DESC, page_id ASC, chunk_id ASCLIMIT ${innerLimitParam}@@ -1221,8 +1222,8 @@ export class PostgresEngine implements BrainEngine {// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING) + search-only// timeout. alwaysTransaction: this method needed sql.begin() on master// already (a transaction-local statement_timeout can never leak onto a- // pooled connection). The scope, JIT off, the timeout and the relaxed- // attempt's enable_seqscan are one settings statement+ // pooled connection). The scope, JIT off, the timeout and the+ // enable_seqscan of a relaxed or chunk-driven attempt are one settings statement// (postgres-engine/search-settings.ts).const statementTimeout = await this.keywordStatementTimeoutSetting();const runKeyword = (queryText: string, relaxed = false, rankText = queryText) => this.keywordTimeouts.run(opts?.keywordTimeoutCache === true, this,@@ -1230,7 +1231,7 @@ export class PostgresEngine implements BrainEngine {this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, (tx) => {const boundParams = params.map((v, i) => i === 0 ? collapseWebsearchDashRuns(queryText) : i === rankIdx && !opts?.rankQuery ? collapseWebsearchDashRuns(rankText) : v);return tx.unsafe(rawQuery, boundParams as Parameters<typeof tx.unsafe>[1]);- }, { alwaysTransaction: true, jitOff: true, settings: relaxed ? { statement_timeout: statementTimeout, enable_seqscan: 'off', ...KEYWORD_STATEMENT_SETTINGS } : { statement_timeout: statementTimeout, ...KEYWORD_STATEMENT_SETTINGS } }));+ }, { alwaysTransaction: true, jitOff: true, settings: relaxed || candidates !== pageJoin ? { statement_timeout: statementTimeout, enable_seqscan: 'off', ...KEYWORD_STATEMENT_SETTINGS } : { statement_timeout: statementTimeout, ...KEYWORD_STATEMENT_SETTINGS } }));const strict = await runKeyword(query);if (Number(strict[0]?.bounded_rows) >= KEYWORD_RANK_CANDIDATES) opts?.onKeywordIncomplete?.('candidate_budget');// D2 fix (fix/title-retrieval-arm): when the strict AND finds nothing,@@ -1242,7 +1243,7 @@ export class PostgresEngine implements BrainEngine {// search/keyword-identifier-probe.ts.return keywordArmRows<Record<string, unknown>>({ query, strict, orFallback: opts?.orFallback, limit, offset, toResult: rowToSearchResult, onIncomplete: opts?.onKeywordIncomplete,runProbe: (match, rank) => runKeyword(match, true, rank),- runHandles: handles => keywordPagesImpl.searchHandlePages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, handles, opts, limit, { statementTimeout }),+ runHandles: handles => keywordPagesImpl.searchHandlePages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, handles, opts, limit, { statementTimeout, statementSettings: KEYWORD_STATEMENT_SETTINGS }),runRelaxed: async (orQuery) => {const shares = await readOrTokenShares((sql, p) => this.executeRaw(sql, p), query, ftsLang);const narrowed = narrowedOrFallbackText(shares), common = commonOrTokens(shares, RARE_TERM_CHUNK_CAP);@@ -1261,10 +1262,10 @@ export class PostgresEngine implements BrainEngine {*//** Cat 40 Hard F2 strict keyword count and keyword-mode pages; SQL in engine-sql/keyword-pages.ts. */async countKeywordPages(query: string, opts?: SearchOpts & { timeoutMs?: number }): Promise<number> {- return keywordPagesImpl.countKeywordPages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, opts, { statementTimeout: await this.keywordStatementTimeoutSetting(), timeoutMs: opts?.timeoutMs });+ return keywordPagesImpl.countKeywordPages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, opts, { statementTimeout: await this.keywordStatementTimeoutSetting(), statementSettings: KEYWORD_STATEMENT_SETTINGS, timeoutMs: opts?.timeoutMs });}async searchKeywordPages(query: string, opts: SearchOpts | undefined, page: KeywordPageWindow): Promise<SearchResult[]> {- return keywordPagesImpl.searchKeywordPages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, opts, page, { statementTimeout: await this.keywordStatementTimeoutSetting() });+ return keywordPagesImpl.searchKeywordPages((read, settings) => this.withScopedReadTransaction(opts?.sourceIds, opts?.sourceId, tx => read(scopedRead(this.engineSqlOn(tx))), { alwaysTransaction: true, jitOff: true, settings }), query, opts, page, { statementTimeout: await this.keywordStatementTimeoutSetting(), statementSettings: KEYWORD_STATEMENT_SETTINGS });}async searchTitles(query: string, opts?: SearchOpts): Promise<SearchResult[]> {diff --git a/src/core/search/keyword-chunk-match.ts b/src/core/search/keyword-chunk-match.tsindex 24392523a4..96d3480941 100644--- a/src/core/search/keyword-chunk-match.ts+++ b/src/core/search/keyword-chunk-match.ts@@ -87,3 +87,62 @@ export function buildKeywordChunkMatch(ftsLang: string, excludePrivate: booleanmodality: '',};}++/**+ * #7082: the candidate rows of the Postgres keyword statement, before the+ * KEYWORD_RANK_CANDIDATES bound. `pageJoin` is the statement that joins every+ * matching chunk to its page under all filters. The bound only saves work when+ * the statement stops reading after it, and for excludePrivate (remote)+ * callers it did not: the private-visibility subplans priced the pages side so+ * high that the planner hashed the entire OFFSET 0 match before the first row+ * passed (680k rows for a term in a third of 2.1M chunks; about 10 s, past the+ * keyword statement timeout, for a term in nearly all of them).+ *+ * For those callers the GIN match streams through an OFFSET 0 subquery (only+ * the columns the score and the visibility clause read), and each matching+ * chunk's page is probed by primary key in a LATERAL subquery whose LIMIT 1+ * keeps it from being pulled up into a join. Every page filter, the visibility+ * clause included, runs in that probe, before the bound, so the rows that+ * reach the bound are exactly the ones the join admits, and the outer LIMIT+ * stops the scan at the bound. Under the bound the ranked set is identical; at+ * the bound the sample is the first matches in index order instead of another+ * plan's.+ *+ * The statement runs with enable_seqscan off (PostgresEngine.searchKeyword), as+ * the relaxed attempt does: for a term in nearly every chunk the planner picks a+ * seq scan, whose ring buffer leaves the heap uncached, so reaching the bound+ * took 4.5 s after a restart; the bitmap scan reaches it in about 0.3 s and+ * keeps the pages it reads in shared buffers.+ *+ * Probing every match costs more than hashing a small page set, so this runs+ * only when no page filter can be selective: no type, frontmatter or date+ * filter (`pageFiltered`), and a source scope (`scope`: the parameter index of+ * the source id or ids) that leaves no live source out, decided per execution+ * by a one-time filter over the UNION ALL. Local callers keep the join, which+ * plans well without the visibility subplans.+ */+export function keywordCandidatesSql(ftsLang: string, c: {+ excludePrivate?: boolean; pageFiltered: boolean; score: string; pageJoin: string; chunkFilters: string; pageFilters: string;+ scope: { idx: number; many: boolean } | null;+}): string {+ if (!c.excludePrivate || c.pageFiltered) return c.pageJoin;+ // chunk-visibility-ok: pageFilters carries buildVisibilityClause(..., { chunkAlias: 'cc' }), applied to every candidate in the page probe+ const chunkDriven = `SELECT p.id AS page_id, cc.id AS chunk_id, ${c.score} AS score+ FROM (+ SELECT cc.id, cc.page_id, cc.search_vector, cc.body_generation+ FROM content_chunks cc+ WHERE cc.search_vector @@ websearch_to_tsquery('${ftsLang}', $1) AND cc.modality = 'text' ${c.chunkFilters}+ OFFSET 0+ ) cc+ CROSS JOIN LATERAL (+ SELECT p.id, p.slug, p.frontmatter+ FROM pages p JOIN sources s ON s.id = p.source_id+ WHERE p.id = cc.page_id ${c.pageFilters}+ LIMIT 1+ ) p`;+ if (!c.scope) return chunkDriven;+ const allSources = `NOT EXISTS (SELECT 1 FROM sources ls WHERE NOT ls.archived AND ls.id <> ${c.scope.many ? `ALL($${c.scope.idx}::text[])` : `$${c.scope.idx}`})`;+ return `SELECT * FROM (${chunkDriven} WHERE ${allSources}+ UNION ALL+ ${c.pageJoin} AND NOT ${allSources}) arms`;+}diff --git a/test/e2e/keyword-arm-ranking-postgres.test.ts b/test/e2e/keyword-arm-ranking-postgres.test.tsindex d33b3b8254..f3e141a947 100644--- a/test/e2e/keyword-arm-ranking-postgres.test.ts+++ b/test/e2e/keyword-arm-ranking-postgres.test.ts@@ -1,15 +1,20 @@/**- * Keyword arm on PostgreSQL: the dual-backend ranking pin in- * test/search/keyword-arm-ranking.test.ts, plus a proof that the statement- * searchKeyword runs for excludePrivate callers keeps its full-text match in- * an OFFSET 0 subquery, which the planner plans on its own. Joined with pages,- * the private-visibility subplans inflate every candidate plan's cost until a+ * Keyword arm on PostgreSQL: the dual-backend ranking and access pins in+ * test/search/keyword-arm-ranking.test.ts and keyword-arm-access.test.ts, plus+ * proofs of the statement shapes searchKeyword runs. For excludePrivate callers+ * with no type, frontmatter or date filter and no source scope that leaves a+ * live source out, the candidates stream from the full-text match through an OFFSET 0 subquery,+ * each probing its page in a LATERAL subquery that runs every page filter,+ * so the KEYWORD_RANK_CANDIDATES bound ends the scan (#7082; the scan itself+ * is pinned in keyword-arm-bounded-scan-postgres.test.ts). Otherwise the+ * statement joins pages: for excludePrivate callers the full-text match+ * stays in an OFFSET 0 subquery, which the planner plans on its own, because+ * the private-visibility subplans inflate every join plan's cost until a* cheaper bitmap scan falls within the planner's 1% fuzz of a seq scan and* loses on startup cost; that seq scan detoasts every chunk's search_vector.- * Without excludePrivate the statement stays unfenced, so its join can still- * run in parallel. (The planner drops the trivial Subquery Scan node from- * EXPLAIN, so the proof reads the statement, then checks it plans and returns- * rows.)+ * Without excludePrivate that join stays unfenced, so it can still run in+ * parallel. (The planner drops the trivial Subquery Scan node from EXPLAIN,+ * so the proofs read the statement, then check it plans and returns rows.)** The per-page window carries ids and scores only; display columns are read* for the ranked rows (KEYWORD_RANKED_ROWS_CTE). Standard for identical@@ -26,7 +31,7 @@ import { installFixtureChunks } from '../helpers/page-projection.ts';import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';import { registerPostgresTests, requirePostgresTestDatabase } from '../helpers/test-backends.ts';-await registerPostgresTests(() => import('../search/keyword-arm-ranking.test.ts'));+await registerPostgresTests(() => import('../search/keyword-arm-ranking.test.ts'), () => import('../search/keyword-arm-access.test.ts'));describe('keyword arm plan (postgres)', () => {let engine: PostgresEngine;@@ -46,7 +51,7 @@ describe('keyword arm plan (postgres)', () => {afterAll(async () => { await close?.(); });- async function capturedKeywordStatement(query: string, excludePrivate: boolean): Promise<{ sql: string; params: unknown[]; rows: number }> {+ async function capturedKeywordStatement(query: string, opts: { excludePrivate: boolean; types?: string[]; sourceIds?: string[] }): Promise<{ sql: string; params: unknown[]; rows: number }> {const target = engine as unknown as Record<string, any>;const original = target.withScopedReadTransaction;let captured: { sql: string; params: unknown[] } | undefined;@@ -62,16 +67,33 @@ describe('keyword arm plan (postgres)', () => {})), opts);};try {- const rows = await engine.searchKeyword(query, { limit: 10, excludePrivate });+ const rows = await engine.searchKeyword(query, { limit: 10, ...opts });return { ...captured!, rows: rows.length };} finally {target.withScopedReadTransaction = original;}}+ const TYPED = ['note'];for (const query of ['acme', 'globex weekly', 'acme or globex']) {- test(`\`${query}\` with excludePrivate runs the full-text match inside the OFFSET 0 subquery`, async () => {- const { sql, params, rows } = await capturedKeywordStatement(query, true);+ test(`\`${query}\` with excludePrivate and no page filter reads candidates from the chunk match and probes each one's page`, async () => {+ const chunkDriven = /FROM \(\s*SELECT cc\.id, cc\.page_id, cc\.search_vector, cc\.body_generation\s+FROM content_chunks cc\s+WHERE cc\.search_vector @@ websearch_to_tsquery\('[a-z_]+', \$1\) AND cc\.modality = 'text'[^)]*OFFSET 0\s*\) cc\s+CROSS JOIN LATERAL \(\s*SELECT p\.id, p\.slug, p\.frontmatter\s+FROM pages p JOIN sources s ON s\.id = p\.source_id\s+WHERE p\.id = cc\.page_id[\s\S]*?LIMIT 1\s*\) p/;+ const unscoped = await capturedKeywordStatement(query, { excludePrivate: true });+ expect(unscoped.rows).toBeGreaterThan(0);+ expect(unscoped.sql).toMatch(chunkDriven);+ expect(unscoped.sql).not.toMatch(/JOIN pages p ON p\.id = cc\.page_id/);+ const scoped = await capturedKeywordStatement(query, { excludePrivate: true, sourceIds: ['default'] });+ expect(scoped.rows).toBe(unscoped.rows);+ expect(scoped.sql).toMatch(chunkDriven);+ expect(scoped.sql).toMatch(/WHERE NOT EXISTS \(SELECT 1 FROM sources ls WHERE NOT ls\.archived AND ls\.id <> ALL\(\$\d+::text\[\]\)\)\s+UNION ALL/);+ for (const { sql, params } of [unscoped, scoped]) {+ const plan = (await engine.executeRaw<{ 'QUERY PLAN': string }>(`EXPLAIN ${sql}`, params)).map((r) => r['QUERY PLAN']).join('\n');+ expect(plan).toContain('content_chunks');+ }+ });++ test(`\`${query}\` with a type filter and excludePrivate runs the full-text match inside the OFFSET 0 subquery`, async () => {+ const { sql, params, rows } = await capturedKeywordStatement(query, { excludePrivate: true, types: TYPED });expect(rows).toBeGreaterThan(0);expect(sql).toMatch(/FROM \(\s*SELECT id, page_id, chunk_index, chunk_text, chunk_source, search_vector, language, symbol_type, body_generation, occurrence_id\s+FROM content_chunks\s+WHERE search_vector @@ websearch_to_tsquery\('[a-z_]+', \$1\) AND modality = 'text'\s+OFFSET 0\s*\) cc\s+JOIN pages p/);expect(sql).not.toMatch(/cc\.search_vector @@/);@@ -80,8 +102,8 @@ describe('keyword arm plan (postgres)', () => {});test(`\`${query}\`: the per-page window and the bounded-row count carry ids and scores only; display columns are read for the ranked rows`, async () => {- for (const excludePrivate of [true, false]) {- const { sql, params, rows } = await capturedKeywordStatement(query, excludePrivate);+ for (const opts of [{ excludePrivate: true }, { excludePrivate: false }, { excludePrivate: true, types: TYPED }, { excludePrivate: false, types: TYPED }]) {+ const { sql, params, rows } = await capturedKeywordStatement(query, opts);expect(rows).toBeGreaterThan(0);type Node = { 'Node Type': string; Output?: string[]; Plans?: Node[] };const nodes = (node: Node): Node[] => [node, ...(node.Plans ?? []).flatMap(nodes)];@@ -96,12 +118,14 @@ describe('keyword arm plan (postgres)', () => {});test(`\`${query}\` without excludePrivate keeps the unfenced join`, async () => {- const { sql, rows } = await capturedKeywordStatement(query, false);- expect(rows).toBeGreaterThan(0);- expect(sql).toMatch(/FROM content_chunks cc\s+JOIN pages p/);- expect(sql).toMatch(/WHERE cc\.search_vector @@ websearch_to_tsquery\('[a-z_]+', \$1\)/);- expect(sql).toMatch(/AND cc\.modality = 'text'/);- expect(sql).not.toContain('OFFSET 0');+ for (const types of [undefined, TYPED]) {+ const { sql, rows } = await capturedKeywordStatement(query, { excludePrivate: false, types });+ expect(rows).toBeGreaterThan(0);+ expect(sql).toMatch(/FROM content_chunks cc\s+JOIN pages p/);+ expect(sql).toMatch(/WHERE cc\.search_vector @@ websearch_to_tsquery\('[a-z_]+', \$1\)/);+ expect(sql).toMatch(/AND cc\.modality = 'text'/);+ expect(sql).not.toContain('OFFSET 0');+ }});}});diff --git a/test/search/keyword-arm-access.test.ts b/test/search/keyword-arm-access.test.tsnew file mode 100644index 0000000000..fb9bbfe70f--- /dev/null+++ b/test/search/keyword-arm-access.test.ts@@ -0,0 +1,91 @@+/**+ * Keyword arm access pin, on PGLite here and on PostgreSQL through+ * test/e2e/keyword-arm-ranking-postgres.test.ts.+ *+ * Every page that matches the query term ranks above the open pages (the+ * restricted ones repeat it), so a filter that stopped running would put a+ * restricted page at the top. The term matches every page for a local caller+ * that may read private pages, which proves the matches exist; a remote+ * (excludePrivate) caller reads only the open ones. Soft-deleted, quarantined+ * and archived-source pages never surface for anyone.+ *+ * On Postgres the calls cover each shape of the keyword statement+ * (PostgresEngine.searchKeyword): no source scope and a scope that covers every+ * live source (candidates driven by the chunk match, one page probe per+ * match), a type filter and a scope that leaves a live source out (the page+ * join), and the keyword-mode count and pages statements.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import type { BrainEngine } from '../../src/core/engine.ts';+import { PGLiteEngine } from '../../src/core/pglite-engine.ts';+import type { SearchOpts } from '../../src/core/types.ts';+import { installFixtureChunks } from '../helpers/page-projection.ts';+import { isolatedPersistencePostgres } from '../helpers/persistence-postgres.ts';+import { testBackends } from '../helpers/test-backends.ts';++const TERM = 'quorvanel';+const OPEN = ['notes/open-a', 'notes/open-b'];+const SIDE_OPEN = 'side/open-c';+const READABLE_LOCALLY = ['notes/private', 'notes/readers', 'side/private'];++for (const kind of testBackends()) {+ describe(`keyword arm access (${kind})`, () => {+ let engine: BrainEngine;+ let close: () => Promise<void>;++ async function page(slug: string, sourceId: string, frontmatter: Record<string, unknown>, hits: number) {+ const body = `${Array.from({ length: hits }, () => TERM).join(' ')} planning notes for ${slug}.`;+ await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: body, frontmatter }, { sourceId });+ await installFixtureChunks(engine, slug, [{ chunk_index: 0, chunk_text: body, chunk_source: 'compiled_truth' }], { sourceId });+ }++ beforeAll(async () => {+ if (kind === 'postgres') {+ ({ engine, close } = await isolatedPersistencePostgres(process.env.DATABASE_URL!));+ } else {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ close = () => engine.disconnect();+ }+ await engine.executeRaw(`INSERT INTO sources (id, name) VALUES ('side', 'side'), ('gone', 'gone') ON CONFLICT DO NOTHING`);+ for (const slug of OPEN) await page(slug, 'default', {}, 1);+ await page(SIDE_OPEN, 'side', {}, 1);+ await page('notes/private', 'default', { visibility: 'private' }, 6);+ await page('notes/readers', 'default', { readable_by: ['alice-example'] }, 6);+ await page('side/private', 'side', { visibility: 'private' }, 6);+ await page('notes/deleted', 'default', {}, 6);+ await page('notes/quarantined', 'default', { quarantine: true }, 6);+ await page('gone/archived', 'gone', {}, 6);+ await engine.executeRaw(`UPDATE pages SET deleted_at = now() WHERE slug = 'notes/deleted'`);+ await engine.executeRaw(`UPDATE sources SET archived = true WHERE id = 'gone'`);+ }, 120_000);++ afterAll(async () => { await close?.(); });++ const scopes: Array<[string, SearchOpts, string[]]> = [+ ['no source scope', {}, [...OPEN, SIDE_OPEN]],+ ['a type filter', { types: ['note'] }, [...OPEN, SIDE_OPEN]],+ ['a scope covering every live source', { sourceIds: ['default', 'side'] }, [...OPEN, SIDE_OPEN]],+ ['a scope leaving a live source out', { sourceIds: ['default'] }, OPEN],+ ['one source', { sourceId: 'side' }, [SIDE_OPEN]],+ ];++ test('a local caller that reads private pages finds every live match, and never a deleted, quarantined or archived-source page', async () => {+ const slugs = (await engine.searchKeyword(TERM, { limit: 50 })).map(r => r.slug).sort();+ expect(slugs).toEqual([...OPEN, SIDE_OPEN, ...READABLE_LOCALLY].sort());+ });++ for (const [name, scope, open] of scopes) {+ test(`remote keyword arm with ${name} returns only open pages`, async () => {+ const remote = { ...scope, excludePrivate: true, requireSafeChunks: true };+ const arm = (await engine.searchKeyword(TERM, { ...remote, limit: 50 })).map(r => r.slug);+ expect([...arm].sort()).toEqual([...open].sort());+ const pages = (await engine.searchKeywordPages(TERM, remote, { limit: 50 })).map(r => r.slug);+ expect([...pages].sort()).toEqual([...open].sort());+ expect(await engine.countKeywordPages(TERM, remote)).toBe(open.length);+ });+ }++ });+}