Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(search): saved facts match stemmed whole words; correction-cue ordering behind search.saved_facts_supersede (default off)
garrytan9c7458812332
diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.mdindex a83e0932e5..421a49edb2 100644--- a/docs/architecture/RETRIEVAL.md+++ b/docs/architecture/RETRIEVAL.md@@ -337,6 +337,24 @@ Two cross-cutting seams sit around the pipeline rather than inside it:with correction-dated writes it lifted forget-then-remember from 92% to 100%correct. `search.query_facts_arm=false` turns it off. Verdict and gates:`docs/eval/decisions/query-facts-arm/`.+- **Saved facts beside `search` and `query` results.** Both ops return, in+ `retrieval.saved_facts` and a text block after the rows, up to five active+ saved facts: first those about the entity page the query names, then facts+ that share the query's words (`matchingSavedFacts` in+ `src/core/ops/search.ts`). Words match whole after stemming with the brain's+ full-text language, the facts keyword index's expression, so "renewal" finds+ "renews" and "count" does not match inside "countersigned"; words of a CJK+ script match as substrings. A fact needs two words of a two- or three-word+ query and three quarters of a longer one; the most matched words come first,+ then the newest. This is the only place `search` returns saved facts: the+ facts arm, the facts block and the temporal reserve run in `query` alone.+ With `search.saved_facts_supersede=true` (default off), a newer fact with a+ correction cue ("correction", "ignore the ...", "not what I posted") that+ shares at least two matched words with a returned fact is returned too, every+ fact it corrects moves below the facts nothing corrects, and those carry+ `corrected_by` (the correcting fact's id). The cue is a fixed word list, so+ an unmarked correction is not caught and a marked unrelated post can relabel+ a fact; nothing stored changes.- **Temporal fact reserve (default off).** With `search.temporal_fact_reserve=true`,a `query` with a deterministic temporal cue (when, before, after, how long,since, until, ordinal-time words, ISO dates, month names) and a token budgetdiff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsvindex bfa01bceb3..11682582a5 100644--- a/scripts/module-size-limits.tsv+++ b/scripts/module-size-limits.tsv@@ -47,4 +47,4 @@ src/commands/doctor/checks/extraction-sync.ts 1500 ratchet +7 GBRA-130 A-1: linksrc/core/search/evidence-delivery.ts 1733 ratchet +5 W5: EvidencePlan.rankCut resolved with the plan (logic in search/page-rank-cut.ts); +20 W5 whole_pages: wholeCost prices a block as delivery would emit it whole (needs the private Block/enrichBlock internals) and the cut is wired before allocate (cutWholePages lives in search/page-rank-cut.ts); new row: crossed the 1500 unlisted cap at the #6367 merge (master 1479, its explicit-budget cap) by the C1 date header threaded through every block path, cap included; pagePlanHits (search/page-plan-hits.ts) and the header loader (evidence-date.ts blockDateHeaders) were peeled first; peel candidate: allocateCapped; +8 at the #6396 merge: page evidence text cut through the fence-chunk trust overlay and the caller minTrust floor (re-summed: our 1504 + master +8); +119 Cat 40 next wave F2: the search.long_page_unit policy (withLongPagePolicy, selectLongPages size preflight with page/byte/deadline limits) and the long-page fetch and fallback in deliverEvidence, kept in the shared seam the plan names (gbrain-evals docs/plans/2026-10-11-cat40-next-wave, F2); +17 F2 rework to the frozen-comparison policy: search.long_page_min_chars (rendered-body characters, default 20,000) read in withLongPagePolicy, a character count in the preflight, the rendered-body confirmation in deliverEvidence, and the long-page row exemption in capDeliveredSnippets (re-summed at the 0.60.225.0 merge: master 1537 + F2 136); +39 Wave B PR 2 lane 2b: planSegmentedBlock joins the windows search/segmented-evidence.ts reads around a segmented page's hits into one run-marked document (needs the private Doc/Piece/buildDoc/locateAnchor internals), planBlock's unit cut split into blockFromDoc for both, sections bounded by their run, selectionCost counts a gap between runs, and the fetch skips the whole-body fence overlay for a segmented page; +4 segmented pages read at once before planning (segmentedReads; the read is search/segmented-evidence.ts, the statements page-body/read-range.ts); +17 GBRA-156: tail trim plan flag, DeliverOptions.query, DeliveryMeta.tail_trim, the trim call and the frozen-hit score/query library fields (logic in search/evidence-tail-trim.ts; interface fields cannot move)src/commands/config.ts 1524 ratchet new row: crossed the 1500 unlisted cap in wave 15 P7.4 #5461: set-time validation of think.pages_budget_chars / think.excerpt_ceiling_chars (range + parser live in core/think/pages-budget.ts), mirroring the cycle.consolidate.cluster_threshold hook; +1 wave 16 G5.1 (#6495): set-time refusal hook for search.keyword_statement_timeout_ms (check lives in commands/config/keyword-timeout-value.ts); +1 wave 18b lane 5 (#5012 item 6): search.contextual_retrieval prints the existing-pages hint after the write (text lives in commands/config/contextual-mode-hint.ts); +1 #6712 PR 2: retired/rollback sync key dispatch line (logic in core/sync-config-keys.ts handleSyncKeyConfig); +1 #6865: sync.slug_conflicts.auto_remove boolean validation at set time (refusal text in repair/slug-autoheal.ts); +1 #6948: sync.origin_twins.auto_resolve boolean validation at set time (refusal text in repair/origin-twins-heal.ts); +2 #6795: config set chat_model/expansion_model help line and the model-pin import (shadow report and file-pin unset live in commands/config/model-pin-shadow.ts); +1 #6712 PR 4b: sync.admission set-time hook (value logic in commands/config/sync-admission-value.ts); +4 #7130: set-time refusal hook for sync.max_deletes / sync.max_delete_ratio (and .source.<id>) / purge.retention_hours (check lives in core/sync-reconcile.ts deleteSafetyConfigRefusal); +1 GBRA-150: sync.metafile_pages set-time route (body in config/sync-metafile-pages-value.ts); re-summed at the v0.60.330.0 merge (1520 measured); +2 #7242 minions.terminal_hook file-plane key and its set branch (logic in minions/terminal-hook.ts); +1 GBRA-161: generations.* set-time route (refusal in commands/config/generations-values.ts); +1 GBRA-207 #7630: sync.repo_path set-time refusal hook (refusal in core/sync-config-keys.ts syncRepoPathRefusal)src/commands/mcp-expose.ts 1512 ratchet hosted_option: Session carries the gbrain.io coaching notice on the first-receipt run and Tailscale/service friction exits (copy, refs and gating live in core/onboard/hosted-option.ts)-src/core/behavior-change-notice.ts 1518 ratchet the BEHAVIOR_CHANGES table grows by one row per disclosed change: +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2); the behavior-change table split (#7607) retires this row+src/core/behavior-change-notice.ts 1520 ratchet +2 the saved_facts_stems row (search saved facts match stemmed whole words); the BEHAVIOR_CHANGES table grows by one row per disclosed change: +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2); the behavior-change table split (#7607) retires this rowdiff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex 59b8c1aa07..2eb414b316 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1320,6 +1320,8 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;{ since: '0.60.471.0', id: 'door_planned_state', text: 'On a source with `canonical: git`, a page write now commits exactly the bytes it published, or nothing. A file edited on disk while the write was publishing refuses `git_target_changed`: nothing is committed, the edit stays on disk and the source\'s writes pause until `gbrain repair stuck-recovery` resolves it (before, Git committed the edit while the database recorded the write\'s content). A staged change of the page\'s file refuses `git_index_conflict`, with the index untouched; commit or unstage it, then retry (before, the write replaced the staged change in the index). When the index still holds the version from before an earlier gbrain commit whose index update failed, the next write restores it and says so with a `git_index_restored` notice.' },// #7679 / #7685 orphaned worktree refresh. `since` is stamped at the merge slot.{ since: '0.60.472.0', id: 'refresh_orphan_7679', text: 'A managed `gbrain sources refresh` whose process died (for example a container replaced mid-refresh) no longer wedges the checkout: the next plain `gbrain sources refresh` on the owner host finishes the orphaned refresh through the `--resume` path before its own refresh and reports it as `recovered`, and the owner completes an orphaned `syncing` refresh once every member has synced to its target. A refresh whose holder is still running, or is recorded on another host, is never taken over. `gbrain sources writer status` and `gbrain sync status` now name the holder and its `liveness` (`live`, `orphaned`, `unknown`) with a fix naming `--resume`. With `--json`, a refusal that carries a structured fix (`refresh_in_progress`, `refresh_diverged`, a git read failure) now prints `fix` as an object (argv, actor, next) instead of a string; the prose stays in `suggestion`, so read that when a script needs text. A plain refresh of a checkout whose interrupted refresh needs a reset decision now refuses `refresh_recovery_required` (it refused `refresh_in_progress`).' },+ // GBRA-205 saved-facts wave (harness case cr-05). `since` is stamped at the merge slot.+ { since: 'TBD', id: 'saved_facts_stems', text: 'Search `saved_facts` matches a saved fact\'s words the way the facts keyword index does: whole words after stemming with the brain\'s full-text language, so a query for "seats renewal" finds a fact that says "renews at 52 seats", and "count" no longer matches inside "countersigned". A three-word query now needs two of its words (it needed all three); other lengths keep their thresholds (two of two, three quarters of four or more), and words of a CJK script still match as substrings. Among facts matching as many words, the newest comes first, and at most five facts ride a reply. New and default off: `search.saved_facts_supersede` returns a newer fact with a correction cue ahead of the older fact it corrects and labels that one `corrected_by`.' },];/** The newest disclosed change's release: the notice id moves only when a release adds rows. */diff --git a/src/core/config.ts b/src/core/config.tsindex 0a95052dce..4726fd2dee 100644--- a/src/core/config.ts+++ b/src/core/config.ts@@ -1437,6 +1437,7 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = ['search.evidence_tail_trim', // GBRA-156: protected top-3 tail trim of auto conversation blocks (default off; search/evidence-tail-trim.ts)'search.entity_anchoring', // entity-anchored retrieval for entity-scoped current-state query/search (default off; search/entity-anchor.ts)'search.query_facts_arm', // query adds matching active facts as rows in spare capacity (default on; search/facts-arm.ts)+ 'search.saved_facts_supersede', // search saved_facts: a newer fact with a correction cue goes ahead of the fact it corrects, labelled corrected_by (default off; ops/search.ts)'search.temporal_fact_reserve', // temporal-cue queries reserve up to 15% of the token budget for question-ranked dated facts (default off; search/facts-arm.ts)'search.page_rank_cut', // W5: whole_pages (default) | off — page unit + caller token_budget stops adding pages once their whole pages fill factor x the budget (search/page-rank-cut.ts)'search.page_rank_cut_factor', // W5: whole_pages target = factor x the budget (default 1.5)diff --git a/src/core/ops/search.ts b/src/core/ops/search.tsindex 6f11a42d0b..f8acae43e3 100644--- a/src/core/ops/search.ts+++ b/src/core/ops/search.ts@@ -65,6 +65,8 @@ import { heldFilesNotice, stampHeldHits } from '../persistence/held-reads.ts';import { opError } from './contract.ts';import { MIN_TRUST_PARAM, resolveReadEligibility } from '../eligibility/policy.ts';import { projectionEligibleSql } from '../eligibility/sql.ts';+import { factsFtsDocument, getFtsLanguage } from '../fts-language.ts';+import { hasCJK } from '../cjk.ts';import { loadPageTrust, stampPageTrust } from '../eligibility/stamp.ts';import type { TrustTier } from '../trust/tier.ts';import type { Operation, OperationContext } from './contract.ts';@@ -368,15 +370,40 @@ function startEntityResolution(ctx: OperationContext, queryText: string, scope:export interface SavedFactMatch { id: number; fact: string; entity_slug: string | null; kind: string; valid_from: string; source: string;/** #7043: every entity of the copies this row stands for, and how many rows it stands for. */- entity_slugs?: string[]; collapsed_count?: number }+ entity_slugs?: string[]; collapsed_count?: number;+ /** With `search.saved_facts_supersede` on: the newer returned fact that corrects this one. */+ corrected_by?: number }++/** Default off: a newer saved fact carrying a correction cue is returned ahead of the older fact it corrects, which is labelled `corrected_by`. */+export const SAVED_FACTS_SUPERSEDE_KEY = 'search.saved_facts_supersede';+/** What marks a saved fact as a correction (Postgres ARE, matched case-insensitively). */+const CORRECTION_CUE = String.raw`\m(correction|corrected|correcting|retract(s|ed|ing|ion)?|disregard|scratch that|i misspoke|i was wrong|not what (i|we) (posted|said|wrote)|ignore (the|my|that|what))\M`;++async function savedFactsSupersedeEnabled(engine: { getConfig(key: string): Promise<string | null> }): Promise<boolean> {+ try {+ const raw = (await engine.getConfig(SAVED_FACTS_SUPERSEDE_KEY))?.trim().toLowerCase();+ return raw === 'true' || raw === 'on' || raw === '1' || raw === 'yes';+ } catch {+ return false;+ }+}/*** Facts saved with `remember` live in the facts table, not in page chunks, so* page search never returns them (gbrain-evals Cat 40: agents saved a* correction with remember and the next session's search missed it). This- * finds active facts whose text or entity shares at least three quarters of the query's words,- * under the same source scope and visibility rules recall applies, copies of one claim collapsed- * before the cap (#7043).+ * finds active facts that contain at least three quarters of the query's words+ * (two of a two- or three-word query), under the same source scope and+ * visibility rules recall applies, copies of one claim collapsed before the cap+ * (#7043).+ * Words match whole after stemming with the brain's full-text language, as the+ * facts keyword index does: "renews" finds "renewal", and "count" never matches+ * inside "countersigned". CJK words, which that parser does not split, match as+ * substrings. Most matched words first, then newest. With+ * `search.saved_facts_supersede` on, a newer fact with a correction cue that+ * shares at least two matched words with a returned fact is returned too, and+ * every fact it corrects moves below the facts nothing corrects, labelled+ * `corrected_by`.*/async function matchingSavedFacts(ctx: OperationContext, scope: SourceScope, queryText: string, aliases: AliasDeclaration[] = []): Promise<Array<SavedFactMatch & CollapseColumns>> {// A query naming one of two declared names also matches facts saved under the other.@@ -385,45 +412,67 @@ async function matchingSavedFacts(ctx: OperationContext, scope: SourceScope, que.filter(t => t.length >= 3 && !FACT_MATCH_STOPWORDS.has(t)).slice(0, 12);if (terms.length === 0 || !ctx.emitResponseMeta) return [];const sources = scope.sourceIds?.length ? scope.sourceIds : [scope.sourceId ?? ctx.sourceId ?? 'default'];- const excludePrivate = await resolveExcludePrivatePages(ctx.engine, ctx.remote);+ const [excludePrivate, supersede] = await Promise.all([resolveExcludePrivatePages(ctx.engine, ctx.remote), savedFactsSupersedeEnabled(ctx.engine)]);const active = `f.source_id = ANY($1::text[])AND f.expired_at IS NULL AND (f.valid_until IS NULL OR f.valid_until > now())AND f.source != ALL($2::text[])${ctx.remote !== false ? `AND f.visibility = 'world'` : ''}`;const eligible = `${excludePrivate ? `AND ${privateProvenanceFilterFragment('f')}` : ''}AND ${projectionEligibleSql('facts', 'f', { floor: scope.minTrust })}`;+ const lang = getFtsLanguage();+ const doc = factsFtsDocument('f');try {// Most searches have no saved fact to find: the uncorrelated EXISTS is an indexed probe- // (idx_facts_since) the planner runs once, before the LIKE ANY scan, and the scan is skipped+ // (idx_facts_since) the planner runs once, before the keyword scan, and the scan is skipped// when no fact qualifies; one statement, so one round trip either way. The probe reads the// fact's own columns only: every row the scan returns passes it, so it never changes the// result, and the page-eligibility filters stay on the scan, where they cost an index- // lookup per candidate. In the probe the planner joined them to the whole pages index.- // The term threshold is applied before the cut: a fact matching every term is never crowded- // out by newer facts that match only one.- const need = Math.max(Math.min(2, terms.length), Math.ceil(terms.length * 0.75));- const rows = await ctx.engine.executeRaw<SavedFactMatch & CollapseColumns & { haystack: string; at: unknown; hits: unknown }>(- `SELECT * FROM (+ // lookup per candidate. The any-word tsquery reads the facts keyword index's own expression.+ // A word the language drops as a stopword counts toward neither side of the threshold.+ // The threshold is applied before the cut: a fact matching every word is never crowded+ // out by newer facts that match fewer.+ const rows = await ctx.engine.executeRaw<SavedFactMatch & CollapseColumns & { at: unknown; matched: unknown[]; need: unknown; cue: boolean }>(+ `WITH w AS (+ SELECT t.i, t.pat, plainto_tsquery('${lang}'::regconfig, t.term) AS tq+ FROM unnest($3::text[], $4::text[]) WITH ORDINALITY AS t(term, pat, i)+ WHERE t.pat IS NOT NULL OR numnode(plainto_tsquery('${lang}'::regconfig, t.term)) > 0+ ), k AS (+ SELECT count(*)::int AS n,+ NULLIF(string_agg('(' || tq::text || ')', ' | ') FILTER (WHERE pat IS NULL), '')::tsquery AS anyq,+ COALESCE(array_agg(pat) FILTER (WHERE pat IS NOT NULL), '{}') AS pats+ FROM w+ )+ SELECT * FROM (SELECT f.id, f.fact, f.entity_slug, f.kind, f.valid_from::text AS valid_from, f.source, f.valid_from AS at, ${SAVED_FACT_COLLAPSE_COLUMNS},- lower(f.fact || ' ' || COALESCE(f.entity_slug, '')) AS haystack,- (SELECT count(*) FROM unnest($3::text[]) AS t(pattern) WHERE lower(f.fact || ' ' || COALESCE(f.entity_slug, '')) LIKE t.pattern) AS hits- FROM facts f+ ARRAY(SELECT w.i FROM w WHERE CASE WHEN w.pat IS NOT NULL THEN lower(f.fact || ' ' || COALESCE(f.entity_slug, '')) LIKE w.pat+ ELSE ${doc} @@ w.tq END ORDER BY w.i) AS matched,+ CASE WHEN k.n <= 3 THEN LEAST(2, k.n) ELSE ceil(k.n * 0.75)::int END AS need,+ ($6::boolean AND f.fact ~* $5) AS cue+ FROM facts f, kWHERE ${active}${eligible}- AND lower(f.fact || ' ' || COALESCE(f.entity_slug, '')) LIKE ANY($3::text[])+ AND (${doc} @@ k.anyq OR lower(f.fact || ' ' || COALESCE(f.entity_slug, '')) LIKE ANY(k.pats))AND EXISTS (SELECT 1 FROM facts f WHERE ${active} LIMIT 1)) m- WHERE m.hits >= $4::int- ORDER BY m.hits DESC, m.at DESC, m.id DESC+ WHERE cardinality(m.matched) >= CASE WHEN m.cue THEN LEAST(2, m.need) ELSE m.need END+ ORDER BY cardinality(m.matched) DESC, m.at DESC, m.id DESCLIMIT 200`,- [sources, [...AUDIT_ROW_SOURCES], terms.map(t => `%${t.replace(/[\\%_]/g, m => `\\${m}`)}%`), need],+ [sources, [...AUDIT_ROW_SOURCES], terms, terms.map(t => hasCJK(t) ? `%${t.replace(/[\\%_]/g, m => `\\${m}`)}%` : null), CORRECTION_CUE, supersede],);- const matched = rows- .map(r => ({ r, hits: terms.filter(t => r.haystack.includes(t)).length }))- .filter(x => x.hits >= need)- .sort((a, b) => b.hits - a.hits)- .map(({ r: { haystack: _h, at: _a, hits: _n, ...fact } }) => ({ ...fact, id: Number(fact.id) }));- return (await collapseFactReads(ctx.engine, matched)).slice(0, 5);+ const found = rows.map(({ at, matched, need, cue, ...fact }) => ({ fact: { ...fact, id: Number(fact.id) } as SavedFactMatch & CollapseColumns,+ at: new Date(at as string).getTime(), words: new Set(matched.map(Number)), need: Number(need), cue }));+ const admitted = found.filter(r => r.words.size >= r.need);+ if (supersede) {+ for (const older of admitted) {+ const correction = found.filter(n => n.cue && n.fact.id !== older.fact.id && n.at > older.at+ && [...n.words].filter(w => older.words.has(w)).length >= Math.min(2, n.need)).sort((a, b) => b.at - a.at)[0];+ if (correction) older.fact.corrected_by = correction.fact.id;+ }+ const correcting = new Set(admitted.flatMap(r => r.fact.corrected_by ? [r.fact.corrected_by] : []));+ admitted.push(...found.filter(r => correcting.has(r.fact.id) && r.words.size < r.need));+ }+ const ordered = supersede ? [...admitted.filter(r => !r.fact.corrected_by), ...admitted.filter(r => r.fact.corrected_by)] : admitted;+ return (await collapseFactReads(ctx.engine, ordered.map(r => r.fact))).slice(0, 5);} catch {return [];}diff --git a/src/mcp/dispatch.ts b/src/mcp/dispatch.tsindex 8007d03b52..f6d83ff844 100644--- a/src/mcp/dispatch.ts+++ b/src/mcp/dispatch.ts@@ -402,7 +402,7 @@ export function retrievalNoticeBlocks(result: unknown, retrieval: unknown): striconst r = retrieval as {type_filter_notice?: unknown;other_names?: Array<{ name: string; alias: string; slug: string }>;- saved_facts?: Array<{ fact: string; entity_slug: string | null; valid_from: string; source: string; linked_from?: string }>;+ saved_facts?: Array<{ fact: string; entity_slug: string | null; valid_from: string; source: string; linked_from?: string; corrected_by?: number }>;answer_id?: unknown;feedback?: { rateable?: boolean; how_to_rate?: string };};@@ -415,7 +415,7 @@ export function retrievalNoticeBlocks(result: unknown, retrieval: unknown): stri}if (r.saved_facts?.length) {const { text, more } = wholeItemsWithin('Saved facts (remember): about the entity this query names (newest first), then facts those point to, then matches for its words; recall returns more:\n',- r.saved_facts.map(f => `- ${f.fact} [entity: ${f.entity_slug ?? 'none'}; saved ${String(f.valid_from).slice(0, 10)}; provenance: ${f.source}${f.linked_from ? `; named in a saved fact about ${f.linked_from}` : ''}]`),+ r.saved_facts.map(f => `- ${f.fact} [entity: ${f.entity_slug ?? 'none'}; saved ${String(f.valid_from).slice(0, 10)}; provenance: ${f.source}${f.linked_from ? `; named in a saved fact about ${f.linked_from}` : ''}${f.corrected_by ? '; corrected by a newer saved fact above' : ''}]`),'\n', SAVED_FACTS_NOTICE_MAX_CHARS);blocks.push(more ? `${text}\n(+${more} more; recall returns them)` : text);}diff --git a/test/search-saved-facts-match.test.ts b/test/search-saved-facts-match.test.tsnew file mode 100644index 0000000000..6178e129b0--- /dev/null+++ b/test/search-saved-facts-match.test.ts@@ -0,0 +1,138 @@+/**+ * Saved facts in `search` (retrieval.saved_facts): a fact saved with+ * `remember` is found by the next search for its words. Words match whole,+ * after stemming with the brain's full-text language ("renews" and "renewal"+ * share a stem; "count" no longer matches inside "countersigned"), and a fact+ * needs two words of a two- or three-word query and three quarters of a longer+ * one. Among equal matches the newest comes first; at most five facts ride a+ * reply.+ *+ * `search.saved_facts_supersede` (default off): a newer fact carrying a+ * correction cue that shares the query's words with an older returned fact is+ * returned ahead of it, and the older one is labelled `corrected_by`. The+ * fixture is the shape of a harness case where a channel post was corrected+ * in another channel and every search returned the retracted figure.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { importFromContent } from '../src/core/import-file.ts';+import { serializeMarkdown } from '../src/core/markdown.ts';+import { dispatchToolCall } from '../src/mcp/dispatch.ts';+import { SAVED_FACTS_SUPERSEDE_KEY } from '../src/core/ops/search.ts';+import { withEnv } from './helpers/with-env.ts';++const WRITER = { remote: true, transport: 'http' as const, sourceId: 'default' };+const REMOTE = { ...WRITER, auth: { token: 't', clientId: 'sf-client', clientName: 'sf', scopes: ['read'], expiresAt: Date.now() / 1000 + 3600 } };++const FIRST = 'Alice Example wrote in #team-a on 2026-09-02: For your records, no action needed: Orbix Home Goods renews at 52 seats. I will file the signed form once countersigned.';+const CORRECTION = 'Alice Example wrote in #team-b on 2026-09-02: Correction for your records, no action needed: the signed Orbix form says 46 seats, not what I posted in #team-a this morning. Ignore the 52, I was reading the draft.';+const QUESTION = 'Bob Example wrote in a direct message on 2026-09-04: What seat count did Orbix land on for the renewal? I am fixing the seat column in the morning report.';+const WORDINGS = ['Orbix seats renewal', 'Orbix renewal seats', 'Orbix seat count renewal', 'Orbix Home Goods renews seats'];++type Fact = { id: number; fact: string; corrected_by?: number };+let engine: PGLiteEngine;+const ids: Record<string, number> = {};++async function remember(fact: string, validFrom: string): Promise<number> {+ const res = await withEnv({ GBRAIN_BACKUP_CHECK: '0' }, () => dispatchToolCall(engine, 'remember', { fact, provenance: 'fixture', valid_from: validFrom }, { ...WRITER, writeWaitMs: 30_000 }));+ expect(res.isError).toBeFalsy();+ const [row] = await engine.executeRaw<{ id: number }>('SELECT id FROM facts WHERE fact = $1', [fact]);+ return Number(row.id);+}++async function search(query: string): Promise<{ facts: Fact[]; text: string }> {+ const res = await withEnv({ GBRAIN_BACKUP_CHECK: '0' }, () => dispatchToolCall(engine, 'search', { query }, REMOTE));+ const retrieval = (res._meta?.retrieval ?? {}) as { saved_facts?: Fact[] };+ return { facts: (retrieval.saved_facts ?? []).map(f => ({ ...f, id: Number(f.id) })), text: res.content.map(c => c.text).join('\n') };+}++const order = (facts: Fact[], id: number) => facts.findIndex(f => f.id === id);++beforeAll(async () => {+ engine = new PGLiteEngine();+ await engine.connect({});+ await engine.initSchema();+ for (const [slug, title, body] of [+ ['accounts/orbix', 'Orbix Home Goods account', 'Account notes for Orbix Home Goods. Renewal cycle is annual. Seat counts are on the signed form.'],+ ['notes/renewals', 'Renewal tracker', 'Track every renewal: seats, price, signed form.'],+ ['notes/seat-column', 'Seat column', 'The morning report has a seat column per account.'],+ ] as const) {+ await importFromContent(engine, slug, serializeMarkdown({}, body, '', { type: 'note', title, tags: [] }), { noEmbed: true, forceRechunk: true });+ }+ ids.first = await remember(FIRST, '2026-09-02T09:00:00Z');+ ids.correction = await remember(CORRECTION, '2026-09-02T09:30:00Z');+ ids.question = await remember(QUESTION, '2026-09-04T09:00:00Z');+ ids.lease = await remember('Delta Example countersigned the warehouse lease.', '2026-09-01T09:00:00Z');+ ids.olderLease = await remember('The warehouse lease runs five years.', '2026-08-01T09:00:00Z');+ ids.newerLease = await remember('The warehouse lease renews in March.', '2026-08-15T09:00:00Z');+ // A correction the remote caller may not read never comes back and never relabels a fact it can read.+ await engine.executeRaw(`INSERT INTO facts (source_id, fact, kind, visibility, source, valid_from)+ VALUES ('default', 'Correction: Orbix renewal seats are 40, ignore the earlier posts.', 'fact', 'private', 'fixture', '2026-09-03')`);+}, 180_000);+afterAll(async () => { await engine.disconnect(); });++describe('search saved facts: stem-aware whole-word matching', () => {+ test('both posts come back for a three-word query that shares two words with each', async () => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ for (const q of WORDINGS.slice(0, 2)) {+ const { facts } = await search(q);+ expect(facts.map(f => f.id), q).toContain(ids.first);+ expect(facts.map(f => f.id), q).toContain(ids.correction);+ expect(facts.some(f => f.corrected_by !== undefined), q).toBe(false);+ expect(facts.length, q).toBeLessThanOrEqual(5);+ }+ // Four and five words: the first post shares three quarters, the correction only two+ // (the company and the seats), so without the flag only the first post comes back.+ for (const q of WORDINGS.slice(2)) {+ const { facts } = await search(q);+ expect(facts.map(f => f.id), q).toContain(ids.first);+ expect(facts.map(f => f.id), q).not.toContain(ids.correction);+ }+ });++ test('a query word matches whole words only', async () => {+ const { facts } = await search('count lease');+ expect(facts.map(f => f.id)).not.toContain(ids.lease);+ });++ test('newest first among equal matches', async () => {+ const { facts } = await search('warehouse lease');+ expect(order(facts, ids.newerLease)).toBeGreaterThanOrEqual(0);+ expect(order(facts, ids.newerLease)).toBeLessThan(order(facts, ids.olderLease));+ });++ test('a fact the remote caller may not read never comes back', async () => {+ for (const q of WORDINGS) expect((await search(q)).text, q).not.toContain('are 40');+ });+});++describe('search.saved_facts_supersede on', () => {+ test('the correction comes back ahead of the fact it corrects, which is labelled, for every wording', async () => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'true');+ try {+ for (const q of WORDINGS) {+ const { facts, text } = await search(q);+ expect(order(facts, ids.correction), q).toBeGreaterThanOrEqual(0);+ expect(order(facts, ids.correction), q).toBeLessThan(order(facts, ids.first));+ expect(facts.find(f => f.id === ids.first)?.corrected_by, q).toBe(ids.correction);+ expect(facts.find(f => f.id === ids.question)?.corrected_by, q).toBeUndefined();+ expect(facts.find(f => f.id === ids.correction)?.corrected_by, q).toBeUndefined();+ expect(text, q).toContain('corrected by a newer saved fact');+ expect(text, q).not.toContain('are 40');+ expect(facts.length, q).toBeLessThanOrEqual(5);+ }+ } finally {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ }+ });++ test('a newer fact with no correction cue relabels nothing', async () => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'true');+ try {+ const { facts } = await search('warehouse lease');+ expect(facts.some(f => f.corrected_by !== undefined)).toBe(false);+ } finally {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ }+ });+});diff --git a/test/search/saved-facts-probe.test.ts b/test/search/saved-facts-probe.test.tsindex 013b791c25..9af4891ba5 100644--- a/test/search/saved-facts-probe.test.ts+++ b/test/search/saved-facts-probe.test.ts@@ -1,6 +1,6 @@/*** The saved-fact match beside search/query blocks (ops/search.ts- * matchingSavedFacts) gates its LIKE scan on an uncorrelated EXISTS probe so+ * matchingSavedFacts) gates its keyword scan on an uncorrelated EXISTS probe so* a brain with no qualifying fact skips the scan. The probe must read the* facts table alone: with the page-eligibility filters inside it (private* provenance, quarantine, needs-rederive), Postgres planned it as merge anti@@ -59,7 +59,7 @@ beforeAll(async () => {// Unbound and called with the caller's `this`: a transaction-scoped view of the engine must keep its own executor.const run = engine.executeRaw;engine.executeRaw = function (this: PGLiteEngine, ...args: Parameters<typeof run>) {- if (args[0].includes('LIKE ANY($3::text[])') && args[0].includes('FROM facts f')) statements.push(args[0]);+ if (args[0].includes('FROM facts f, k') && args[0].includes('AND EXISTS (SELECT 1 FROM facts f WHERE')) statements.push(args[0]);return run.apply(this, args);} as typeof engine.executeRaw;}, 120_000);