Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(search): saved-fact corrections compare normalised figures and stemmed names; log a failed match read (#7938)
garrytanac7ad285af4a
diff --git a/docs/architecture/key-files/entity-recall.md b/docs/architecture/key-files/entity-recall.mdindex 8828f96ab3..079fb492d0 100644--- a/docs/architecture/key-files/entity-recall.md+++ b/docs/architecture/key-files/entity-recall.md@@ -24,7 +24,7 @@ card and `get_backlinks`. Guide: [entity recall](../../guides/entity-recall.md).- `src/core/verbs/entity-card-identity.ts` — the `entity` card's `aka_sources`, `identity_siblings`, `identity_excerpt` (verbatim lines of the page and its siblings from `sanitizeRemoteBody`, tiered: declarations, label lines with a name-like value, other label lines, lines with another capitalized run or code; 600 characters per page, 1,200 per card, `identity_excerpt_omitted`), `alias_guidance` (surface-aware) and the `identity_siblings_capped` notice. A segmented page is excerpted from its trusted, withdrawal-overlaid lead (`truthLead`, `EXCERPT_LEAD_BYTES` = 64 KiB, cut back to a whole line), never its body getters; the card equals the inline page's when the qualifying lines sit in that lead (`test/large-pages/segmented-read-refusals.test.ts`). `readSiblingIdentity` is its siblings-and-aliases half (one alias read for the page and its siblings; optional preloaded mention policy and pack), used alone by `entity names[]`. Tests: `test/mentions-identity-siblings.test.ts`.- `src/core/search/alias-fanout.ts` — search and query's alias fan-out: `resolveQueryEntity` (query n-grams up to 4 tokens through `resolveAliases` and linkable exact titles, longest first, capitalized or code-shaped unless a multi-word n-gram resolves to one page, case-sensitive aliases as written, a tie only when the pages are identity siblings), the names of the entity and its siblings by origin then length, at most `search.alias_fanout_max` (4) keyword queries of 5 rows (`searchAliasRequired`: the name as a required websearch phrase, ranked by the query's other terms through `SearchOpts.rankQuery` on both engines), at most 8 spliced rows with `matched_alias`, and the `fanout` meta (`alias_fanout` notice when capped). Offset 0 only. `resolving` takes the handler's `resolveQueryEntity` started before the search, so its reads overlap it. Tests: `test/search-alias-fanout.test.ts`, `test/search-other-names.test.ts`, `test/e2e/mentions-parity.test.ts`. `wholeQueryEntityNames`: when the whole query is one entity's name or alias, `match: "keyword"` lists pages matching any of its names (one websearch OR of phrases; `match_names` in the meta and the count line; rows found by another name carry `matched_alias`). `resolveQueryEntities` returns every entity a text names; its alias, linkable-title and case-sensitive alias reads run concurrently (`test/search-entity-resolve-concurrency.test.ts`).- `src/core/search/entity-facts.ts` — `entitySavedFacts`: when a search names an entity, its newest active saved facts (`factEntitySlugs`, 5) lead the model-visible saved-facts block, then up to 3 facts each of up to 2 entities those facts name (`linked_from`), before the word-overlap matches. Same active, audit-row, visibility and read-eligibility rules as recall (`projectionEligibleSql`: quarantined-page and `needs_rederive` facts hidden, the caller's trust floor applied). Tests: `test/memory-sibling-facts.test.ts`, `test/search-entity-facts-eligibility.test.ts` (also the trajectory, entity-card count, standing-question evidence, local CLI recall and consolidate reads, both engines).-- `src/core/search/saved-facts-match.ts` — `matchingSavedFacts`, the word-matched saved facts `search` and `query` report in `retrieval.saved_facts` after the named entity's facts (`ops/search.ts` `buildRetrievalResponseMeta`). Words match whole after stemming with the facts keyword index's expression (`factsFtsDocument`), CJK words as substrings; a fact needs three quarters of the query's words, at most four, question-framing words (`QUESTION_FRAME_WORDS`: count, number, total, amount) not counted; most matched words first, then newest (time, then id), at most five after the #7043 collapse. One statement behind the facts-only EXISTS probe (`test/search/saved-facts-probe.test.ts`). `search.saved_facts_supersede` (default on; `false` turns it off): a newer fact with a correction cue that shares two matched words with a returned fact is returned too, even below the threshold, and the facts it corrects move last with `corrected_by` (the newest correction); every correction of a returned fact comes back; a cue fact that adds no figure to the older fact (`figures`: numbers, with ISO and day-month dates and times left out; `CORRECTION_CUE` leaves out "corrected"/"correcting", which report a correction) corrects nothing when it restates one of its figures or the older fact is itself a correction, nor when it states no figure and withdraws nothing (`RETRACTION`); nor does one saved under another entity, of lower trust, or sharing none of the older fact's matched names (`namedWords`). Correctors rank newest first under the cap and a corrected fact whose corrector is cut is dropped; `buildRetrievalResponseMeta` carries `corrected_by` onto the entity copy. Also owns `FACT_MATCH_STOPWORDS`. Tests: `test/search-saved-facts-match.test.ts`, `test/search-saved-facts.test.ts`.+- `src/core/search/saved-facts-match.ts` — `matchingSavedFacts`, the word-matched saved facts `search` and `query` report in `retrieval.saved_facts` after the named entity's facts (`ops/search.ts` `buildRetrievalResponseMeta`). Words match whole after stemming with the facts keyword index's expression (`factsFtsDocument`), CJK words as substrings; a fact needs three quarters of the query's words, at most four, question-framing words (`QUESTION_FRAME_WORDS`: count, number, total, amount) not counted; most matched words first, then newest (time, then id), at most five after the #7043 collapse. One statement behind the facts-only EXISTS probe (`test/search/saved-facts-probe.test.ts`). `search.saved_facts_supersede` (default on; `false` turns it off): a newer fact with a correction cue that shares two matched words with a returned fact is returned too, even below the threshold, and the facts it corrects move last with `corrected_by` (the newest correction); every correction of a returned fact comes back; a cue fact that adds no figure to the older fact (`figures`: numbers, with ISO and day-month dates and times left out, thousands separators and a zero fraction dropped; `CORRECTION_CUE` leaves out "corrected"/"correcting", which report a correction) corrects nothing when it restates one of its figures or the older fact is itself a correction, nor when it states no figure and withdraws nothing (`RETRACTION`); nor does one saved under another entity, of lower trust, or sharing none of the older fact's matched names (`namedWords`: a name head matches a query word by prefix or, read only when a cue fact is a candidate, by stem). Correctors rank newest first under the cap and a corrected fact whose corrector is cut is dropped; a failed read warns with its SQLSTATE and returns no facts. The result carries every pre-cap decision (`correctedBy`): `buildRetrievalResponseMeta` applies it after merging the entity facts, dropping a corrected fact with no returned corrector (repeated through chains), labelling the rest, and ordering each corrected fact after its corrector. Also owns `FACT_MATCH_STOPWORDS`. Tests: `test/search-saved-facts-match.test.ts`, `test/search-saved-facts.test.ts`.- `src/core/schema-migrations/v206-entity-mention-index.ts` — additive: `page_aliases.origin` (`frontmatter` default, `declared`, `subject`), `case_sensitive`, `alias_text`, unique key `(source_id, alias_norm, slug, origin)`; tables `page_mention_state`, `mention_gazetteer_entries`, `mention_index_status` (RLS on when the role bypasses RLS). `GUARDED_TABLE_COLUMNS` lists the new `page_aliases` columns.Alias readers apply precedence frontmatter > declared > subject per source, and a derived alias never answers for another live page's exact title that the caller can read (the title competitor gets the caller's `pageReadFilter`, so a private title a private-excluding caller can't see never hides a readable alias) (`readAliases` in `src/core/search/read-enrichment.ts`, shared by both engines' `resolveAliases`), so `resolveEntitySlug`, the retrieval reflex, the alias hop, intent weights and `search-diagnose` rank an exact title first. `buildGazetteer` (`src/core/by-mention.ts`) applies the same precedence, drops an alias claimed by two pages at its best origin (unless the claimants are identity siblings, which all keep it), yields to a same-source title, drops a single-word alias equal to the first word of another multi-word entity name, applies the generic-word list to every single-word name, and carries `caseTokens` for case-sensitive entries; `opts.dropped` records every dropped name with its reason.diff --git a/src/core/search/saved-facts-match.ts b/src/core/search/saved-facts-match.tsindex b8a9657b81..015dcb24d8 100644--- a/src/core/search/saved-facts-match.ts+++ b/src/core/search/saved-facts-match.ts@@ -42,17 +42,35 @@ const CORRECTION_CUE = String.raw`\m(correction|${RETRACTION})\M`;const MONTH = String.raw`(?:jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|june?|july?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)\b\.?`;const DATES = new RegExp(String.raw`\b\d{4}-\d{2}-\d{2}(?:[ T]\d{2}:\d{2}(?::\d{2})?)?|\b\d{1,2}:\d{2}\b|\b\d{1,2}(?:st|nd|rd|th)? ${MONTH}|\b${MONTH} \d{1,2}(?:st|nd|rd|th)?\b`, 'gi');-/** The query words a fact writes as the first word of a name ("Orbix" in "Orbix Home Goods", not "Example" in "Alice Example"); term indexes are 1-based. */-function namedWords(text: string, terms: string[], words: Set<number>): Set<number> {- const heads = [...text.matchAll(/[\p{L}\p{N}][\p{L}\p{N}'-]*/gu)]+/** The words a fact writes as the first word of a name ("Orbix" in "Orbix Home Goods", not "Example" in "Alice Example"), lower-cased. */+function nameHeads(text: string): string[] {+ return [...text.matchAll(/[\p{L}\p{N}][\p{L}\p{N}'-]*/gu)].filter(m => m[0][0] !== m[0][0]!.toLowerCase() && !/\p{Lu}[\p{L}\p{N}'-]*\s+$/u.test(text.slice(0, m.index))).map(m => m[0].toLowerCase());- return new Set([...words].filter(i => heads.some(h => h.startsWith(terms[i - 1]!))));}-/** The figures a fact states: its numbers, with dates ("2026-10-14", "14 Oct", "July 1") and clock times left out. */+/** The matched query words a fact writes as a name head, by prefix or by stem (`stems`: head to the 1-based term indexes it matches). */+function namedWords(heads: string[], terms: string[], words: Set<number>, stems: Map<string, Set<number>>): Set<number> {+ return new Set([...words].filter(i => heads.some(h => h.startsWith(terms[i - 1]!) || stems.get(h)?.has(i))));+}++/** Which query terms each name head matches after stemming with the brain's full-text language, as the keyword scan matches words; empty on a read error. */+async function headStems(engine: OperationContext['engine'], lang: string, heads: string[], terms: string[]): Promise<Map<string, Set<number>>> {+ if (heads.length === 0) return new Map();+ const rows = await engine.executeRaw<{ h: string; i: unknown[] }>(+ `SELECT h, ARRAY(SELECT t.i FROM unnest($2::text[]) WITH ORDINALITY AS t(term, i)+ WHERE to_tsvector('${lang}'::regconfig, h) @@ plainto_tsquery('${lang}'::regconfig, t.term)) AS i+ FROM unnest($1::text[]) AS h`, [heads, terms]).catch(() => []);+ return new Map(rows.map(r => [r.h, new Set(r.i.map(Number))]));+}++/**+ * The figures a fact states: its numbers, with dates ("2026-10-14", "14 Oct", "July 1") and clock times left out,+ * thousands separators and a zero fraction dropped, so 52,000, 52000 and 52000.0 are one figure.+ */function figures(text: string): Set<string> {- return new Set(text.replace(DATES, ' ').match(/\d+(?:[.,]\d+)*/g) ?? []);+ return new Set((text.replace(DATES, ' ').match(/\d+(?:[.,]\d+)*/g) ?? [])+ .map(n => (/^\d{1,3}(?:,\d{3})+(?:\.\d+)?$/.test(n) ? n.replace(/,/g, '') : n).replace(/\.0+$/, '')));}async function savedFactsSupersedeEnabled(engine: { getConfig(key: string): Promise<string | null> }): Promise<boolean> {@@ -84,7 +102,8 @@ async function savedFactsSupersedeEnabled(engine: { getConfig(key: string): Prom* fact that adds no figure (a number other than a date or time) to the older* fact, when it restates one of its figures or the older fact is itself a* correction. Nor does a cue fact about another entity, one missing every- * name the older fact starts with a matched word, one of lower trust, or one+ * name the older fact starts with a matched word (prefix or stem; the stems are+ * read only when a cue fact is among the candidates), one of lower trust, or one* stating no figure that withdraws nothing. A corrected fact whose corrector* the five-fact cap leaves out is dropped. `correctedBy` carries the decisions* for facts the cap cut too, so a caller merging other reads of the same facts@@ -149,13 +168,15 @@ export async function matchingSavedFacts(ctx: OperationContext, scope: { sourceI[sources, [...AUDIT_ROW_SOURCES], terms, pats, CORRECTION_CUE, supersede],);const found = rows.map(({ at, matched, need, cue, trust_rank, pool_rank: _p, ...fact }) => ({ fact: { ...fact, id: Number(fact.id) } as SavedFactMatch & CollapseColumns,- at: new Date(at as string).getTime(), words: new Set(matched.map(Number)), need: Number(need), cue, trust: Number(trust_rank), figures: figures(fact.fact) }))- .map(r => ({ ...r, names: namedWords(r.fact.fact, terms, r.words) }));+ at: new Date(at as string).getTime(), words: new Set(matched.map(Number)), need: Number(need), cue, trust: Number(trust_rank), figures: figures(fact.fact) }));const admitted = found.filter(r => r.words.size >= r.need);const correcting = new Set<number>();if (supersede) {const cues = found.filter(n => n.cue);+ const heads = new Map(admitted.map(r => [r, nameHeads(r.fact.fact)]));+ const stems = cues.length ? await headStems(ctx.engine, lang, [...new Set([...heads.values()].flat())], terms) : new Map<string, Set<number>>();for (const older of admitted) {+ const names = namedWords(heads.get(older)!, terms, older.words, stems);// Posts saved in one flush share a time; the later id is the later post.// An acknowledgement is not a correction: a cue fact that adds no figure to the older fact// corrects it only when it restates no figure and the older fact is not a correction itself.@@ -163,7 +184,7 @@ export async function matchingSavedFacts(ctx: OperationContext, scope: { sourceI// higher trust; one stating no figure corrects only by withdrawing ("disregard"), never by a bare "correction".const corrections = cues.filter(n => (n.at > older.at || (n.at === older.at && n.fact.id > older.fact.id))&& (!n.fact.entity_slug || !older.fact.entity_slug || n.fact.entity_slug === older.fact.entity_slug)- && (older.names.size === 0 || [...older.names].some(w => n.words.has(w))) && n.trust >= older.trust+ && (names.size === 0 || [...names].some(w => n.words.has(w))) && n.trust >= older.trust&& [...n.words].filter(w => older.words.has(w)).length >= Math.min(2, n.need)&& (n.figures.size > 0 || RETRACTION_CUE.test(n.fact.fact))&& !([...n.figures].every(x => older.figures.has(x)) && (n.figures.size > 0 || older.cue))).sort((a, b) => b.at - a.at || b.fact.id - a.fact.id);@@ -190,7 +211,9 @@ export async function matchingSavedFacts(ctx: OperationContext, scope: { sourceIreturn corrected_by === undefined ? [r] : [];});return { facts, correctedBy };- } catch {+ } catch (err) {+ const code = (err as { code?: unknown } | null)?.code;+ console.warn(`[search] Saved-facts match failed (SQLSTATE ${typeof code === 'string' ? code : 'unknown'}); this reply carries no saved facts, which does not mean none match.`);return none;}}diff --git a/test/search-saved-facts-match.test.ts b/test/search-saved-facts-match.test.tsindex b544e5032f..a9ea361712 100644--- a/test/search-saved-facts-match.test.ts+++ b/test/search-saved-facts-match.test.ts@@ -12,12 +12,13 @@* fixture is the shape of a harness case where a channel post was corrected* in another channel and every search returned the retracted figure.*/-import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { afterAll, afterEach, beforeAll, describe, expect, spyOn, test } from 'bun:test';import { PGLiteEngine } from '../src/core/pglite-engine.ts';import { importFromContent } from '../src/core/import-file.ts';import { serializeMarkdown } from '../src/core/markdown.ts';import { dispatchToolCall } from '../src/mcp/dispatch.ts';-import { SAVED_FACTS_SUPERSEDE_KEY } from '../src/core/search/saved-facts-match.ts';+import { SAVED_FACTS_SUPERSEDE_KEY, matchingSavedFacts } from '../src/core/search/saved-facts-match.ts';+import type { OperationContext } from '../src/core/operations.ts';import { withEnv } from './helpers/with-env.ts';const WRITER = { remote: true, transport: 'http' as const, sourceId: 'default' };@@ -455,3 +456,65 @@ describe('search.saved_facts_supersede: a corrected fact follows its correction'}});});++// Value: protects=an acknowledgement restating a figure in another format corrects nothing, a figure-free withdrawal still corrects, and a name the query writes in another word form still gates (#7938 items 5, 9, 10); fails_when=figures() compares 52,000 and 52000 (or 52.0 and 52) as different numbers, a withdrawal stating no figure stops correcting, or namedWords compares the literal query word with the fact's name; why_new=every earlier acknowledgement restated the figure verbatim, no withdrawal without a figure was tested, and every name matched literally; seam=none; no_claim=decimal commas+describe('search.saved_facts_supersede: figures, withdrawals and names', () => {+ test.each([+ ['Pemberly', 'Pemberly renews at 52000 seats.', 'Correction noted: Pemberly renews at 52,000 seats.'],+ ['Quintavo', 'Quintavo renews at 52 seats.', 'Correction noted: Quintavo renews at 52.0 seats.'],+ ])('%s: an acknowledgement restating the figure in another format relabels nothing', async (name, fact, ack) => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'true');+ try {+ const older = await remember(fact, '2025-05-01T09:00:00Z');+ const noted = await remember(ack, '2025-05-02T09:00:00Z');+ const { facts } = await search(`${name} renewal seats`);+ expect(facts.map(f => f.id)).toEqual(expect.arrayContaining([older, noted]));+ expect(facts.find(f => f.id === older)?.corrected_by).toBeUndefined();+ } finally {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ }+ });++ test('a withdrawal that states no figure still corrects the fact it withdraws', async () => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'true');+ try {+ const older = await remember('Rostanne renews at 52 seats.', '2025-05-03T09:00:00Z');+ const withdrawal = await remember('Disregard what I said about Rostanne seats.', '2025-05-04T09:00:00Z');+ const { facts } = await search('Rostanne renewal seats');+ expect(facts.find(f => f.id === older)?.corrected_by).toBe(withdrawal);+ expect(order(facts, withdrawal)).toBeLessThan(order(facts, older));+ } finally {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ }+ });++ test('a name the query writes in another word form still keeps out a correction about another name', async () => {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'true');+ try {+ const older = await remember('Agencies Calder renews at 52 seats.', '2025-05-05T09:00:00Z');+ await remember('Correction: Tolvane renews at 46 seats, ignore the 52.', '2025-05-06T09:00:00Z');+ const { facts } = await search('agency renewal seats');+ expect(facts.map(f => f.id)).toContain(older);+ expect(facts.find(f => f.id === older)?.corrected_by).toBeUndefined();+ } finally {+ await engine.setConfig(SAVED_FACTS_SUPERSEDE_KEY, 'false');+ }+ });+});++// Value: protects=a failed saved-facts read is logged, so a statement timeout is not mistaken for "no saved facts" (#7938 item 11); fails_when=the catch swallows the error silently; why_new=no test drove the failure path; seam=none (a stub engine whose read throws); no_claim=the reply's degraded stages+describe('saved facts: a failed match read', () => {+ afterEach(() => { (console.warn as unknown as { mockRestore?: () => void }).mockRestore?.(); });++ test('returns no facts and warns with the SQLSTATE', async () => {+ const warn = spyOn(console, 'warn').mockImplementation(() => {});+ const stub = {+ getConfig: async () => null,+ executeRaw: async () => { throw Object.assign(new Error('canceling statement due to statement timeout'), { code: '57014' }); },+ };+ const ctx = { engine: stub, remote: true, sourceId: 'default', emitResponseMeta: () => {} } as unknown as OperationContext;+ const out = await matchingSavedFacts(ctx, {}, 'Orbix renewal seats');+ expect(out.facts).toEqual([]);+ expect(warn.mock.calls.map(a => String(a[0]))).toEqual([expect.stringContaining('Saved-facts match failed (SQLSTATE 57014)')]);+ });+});