Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Keyword arm: a NULL in the source scope picks the page join; tests read the production plan (#7082)

garrytanf0bbcf68f024
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8214bb6555..93b6ef87b0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,9 +17,10 @@ identifiers and attribution are available in the pre-removal Git revision
**A remote caller's hybrid search no longer times out its keyword arm on a common word.** Hybrid search ranks at most 50,000 matching chunks, but for a remote (MCP) caller the planner still read every matching chunk before that bound applied, because the private-page checks made the page join look expensive. On a large Postgres brain a word in most chunks then ran past the 8 s keyword statement timeout, the search answered from vectors alone (`keyword_arm_failed`), and the serving process skipped the keyword arm for that query for the next 10 minutes.
- When a remote caller sets no type, frontmatter or date filter and its source scope covers every live source (the usual MCP scope), the keyword statement now reads matching chunks straight from the full-text index and checks each one's page by key, with every visibility, member-access, quarantine, deletion and archived-source filter applied before the bound. It stops reading once it has 50,000 candidates.
-- Measured on a synthetic 2.1M-chunk Postgres brain (warm, median): a word in 95% of chunks went from 9.9 s to 0.65 s, a word in a third of chunks from 1.5 s to 0.76 s, and a word in 5% of chunks from 0.93 s to 0.70 s. Rare terms are unchanged (about 40 ms), and their results are identical.
+- Measured on a synthetic 2.1M-chunk Postgres brain (warm, median), the remote keyword statement for a word in 95% of chunks went from 9.6 s to 0.89 s, a word in a third of chunks from 1.4 s to 0.72 s, and a word in 5% of chunks from 0.88 s to 0.64 s. Rare terms are unchanged (about 40 ms), with identical results. A whole MCP search for the 95% word now answers with keyword hits in about 1 s, where it used to time out and fall back to vectors.
- Results are identical whenever a query matches fewer than 50,000 chunks. Above that, the 50,000-chunk sample is drawn in index order instead of another plan's order, so which pages of a very broad term rank can change; the result was already a sample there (`keyword_candidates_incomplete`).
-- Local callers, typed or dated searches and narrower source scopes run the same statement as before.
+- A remote caller scoped to only some sources runs the old page join through the same statement, now with sequential scans off as the OR retry already ran: a word in 95% of chunks went from 9.1 s to 1.7 s there, and its other results and timings are unchanged. Local callers and typed or dated searches run exactly the statement they ran before.
+- For a word in nearly every chunk, the statement turns sequential scans off, as the OR retry already did: the planner otherwise picked a seq scan, which does not keep the pages it reads in cache and took about 5 s to reach the bound after a restart, against under 1 s for the index scan.
- The keyword-mode page statement and the strict page count now run with the same 32 MB `work_mem` as the keyword arm, so a common word's bitmap no longer goes lossy: 5.8 s to 3.7 s for keyword paging and 0.68 s to 0.26 s for a count on the same brain.
- `scripts/bench/retrieval-scale/keyword-broad.ts` times the keyword statements for broad, medium and rare terms and can diff two checkouts' results; `run.ts` takes `--queries` and `--no-query-cache`.
diff --git a/src/core/search/keyword-chunk-match.ts b/src/core/search/keyword-chunk-match.ts
index 96d3480941..ce2541b247 100644
--- a/src/core/search/keyword-chunk-match.ts
+++ b/src/core/search/keyword-chunk-match.ts
@@ -118,8 +118,9 @@ export function buildKeywordChunkMatch(ftsLang: string, excludePrivate: boolean
* only when no page filter can be selective: no type, frontmatter or date
* filter (`pageFiltered`), and a source scope (`scope`: the parameter index of
* the source id or ids) that leaves no live source out, decided per execution
- * by a one-time filter over the UNION ALL. Local callers keep the join, which
- * plans well without the visibility subplans.
+ * by a one-time filter over the UNION ALL (a NULL id in the scope counts as
+ * leaving sources out). Local callers keep the join, which plans well without
+ * the visibility subplans.
*/
export function keywordCandidatesSql(ftsLang: string, c: {
excludePrivate?: boolean; pageFiltered: boolean; score: string; pageJoin: string; chunkFilters: string; pageFilters: string;
@@ -141,7 +142,7 @@ export function keywordCandidatesSql(ftsLang: string, c: {
LIMIT 1
) p`;
if (!c.scope) return chunkDriven;
- const allSources = `NOT EXISTS (SELECT 1 FROM sources ls WHERE NOT ls.archived AND ls.id <> ${c.scope.many ? `ALL($${c.scope.idx}::text[])` : `$${c.scope.idx}`})`;
+ const allSources = `NOT EXISTS (SELECT 1 FROM sources ls WHERE NOT ls.archived AND (ls.id <> ${c.scope.many ? `ALL($${c.scope.idx}::text[])` : `$${c.scope.idx}`}) IS NOT FALSE)`;
return `SELECT * FROM (${chunkDriven} WHERE ${allSources}
UNION ALL
${c.pageJoin} AND NOT ${allSources}) arms`;
diff --git a/test/e2e/keyword-arm-bounded-scan-postgres.test.ts b/test/e2e/keyword-arm-bounded-scan-postgres.test.ts
index 549b76b892..aa482aefa8 100644
--- a/test/e2e/keyword-arm-bounded-scan-postgres.test.ts
+++ b/test/e2e/keyword-arm-bounded-scan-postgres.test.ts
@@ -74,6 +74,7 @@ describe('bounded keyword arm (postgres)', () => {
expect(incomplete).toBe(true);
const [row] = await engine.transaction(async (tx) => {
await tx.executeRaw(`SET LOCAL work_mem = '32MB'`);
+ await tx.executeRaw(`SET LOCAL enable_seqscan = off`);
return tx.executeRaw<{ 'QUERY PLAN': Array<{ Plan: Node }> }>(`EXPLAIN (ANALYZE, FORMAT JSON) ${sql}`, params);
});
const read = nodes(row!['QUERY PLAN'][0]!.Plan)
diff --git a/test/e2e/keyword-arm-ranking-postgres.test.ts b/test/e2e/keyword-arm-ranking-postgres.test.ts
index f3e141a947..7027f74a40 100644
--- a/test/e2e/keyword-arm-ranking-postgres.test.ts
+++ b/test/e2e/keyword-arm-ranking-postgres.test.ts
@@ -85,7 +85,12 @@ describe('keyword arm plan (postgres)', () => {
const scoped = await capturedKeywordStatement(query, { excludePrivate: true, sourceIds: ['default'] });
expect(scoped.rows).toBe(unscoped.rows);
expect(scoped.sql).toMatch(chunkDriven);
- expect(scoped.sql).toMatch(/WHERE NOT EXISTS \(SELECT 1 FROM sources ls WHERE NOT ls\.archived AND ls\.id <> ALL\(\$\d+::text\[\]\)\)\s+UNION ALL/);
+ expect(scoped.sql).toMatch(/WHERE NOT EXISTS \(SELECT 1 FROM sources ls WHERE NOT ls\.archived AND \(ls\.id <> ALL\(\$\d+::text\[\]\)\) IS NOT FALSE\)\s+UNION ALL/);
+ type Node = { 'Node Type': string; 'Actual Rows'?: number; Plans?: Node[] };
+ const nodes = (node: Node): Node[] => [node, ...(node.Plans ?? []).flatMap(nodes)];
+ const [analyzed] = await engine.executeRaw<{ 'QUERY PLAN': Array<{ Plan: Node }> }>(`EXPLAIN (ANALYZE, FORMAT JSON) ${scoped.sql}`, scoped.params);
+ const [arms] = nodes(analyzed!['QUERY PLAN'][0]!.Plan).filter(node => node['Node Type'] === 'Append');
+ expect(arms!.Plans!.map(arm => arm['Actual Rows']! > 0)).toEqual([true, false]);
for (const { sql, params } of [unscoped, scoped]) {
const plan = (await engine.executeRaw<{ 'QUERY PLAN': string }>(`EXPLAIN ${sql}`, params)).map((r) => r['QUERY PLAN']).join('\n');
expect(plan).toContain('content_chunks');