Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(beliefs): journal the mention pass link writes so stale extraction leaves no drift (#7031)
garrytanddd2f1a40a78
diff --git a/docs/architecture/key-files/entity-recall.md b/docs/architecture/key-files/entity-recall.mdindex 64efcf6f6d..e305c073ff 100644--- a/docs/architecture/key-files/entity-recall.md+++ b/docs/architecture/key-files/entity-recall.md@@ -10,7 +10,7 @@ card and `get_backlinks`. Guide: [entity recall](../../guides/entity-recall.md).- `src/core/mentions/siblings.ts` — identity siblings: pages in one source naming the same normalized title subject under pairwise different prefixes, all linkable entities of one canonical type other than `person`, none excluded or `identity: separate`, at most `SIBLING_GROUP_CAP` (3; a larger group is `capped`). `siblingVerdict` is the one rule (card, search fan-out, gazetteer); `readIdentitySiblings` reads a page's group with the private-page predicate for untrusted callers. `siblingCanonical` (the group's lowest slug, for fact attribution by name) and `factEntitySlugs` (the page, its siblings and its bare subject slug, read by the entity card's facts and `recall {entity}`). Nothing is written to `entity_identities`, and aliases and references are never unioned. Tests: `test/mentions-identity-siblings.test.ts`.- `src/core/mentions/aliases.ts` — derived names of an entity page, implementing `docs/designs/ALIAS_CONVENTIONS.md`. Stored aliases and card identity lines read declarations with `anchored: true` (a cue must attach to the page itself; search's per-result `aliasDeclarations` scan reads them all). `titleSubject` (text after the last `: `), `declarationsIn` (each declaration with its line: prose cues, label and value lines and table rows whose label names an alias field (also a label opening a later sentence of a line, alias noun within its first two words, at most 7 words), the quoted-only "calls it" cue, a quoted defined term right after the page's own name; a capture is a quoted name or a run of up to 4 capitalized or code tokens ending at punctuation, a lowercase word or the line end, rejected on a possessive, a run of only common words or a cue after a relational noun; linear in the line length), `declaredNames` (its post-filtered names, shared with search's `aliasDeclarations`), `deriveEntityAliases(page, {multiword, deny})` (subject `origin='subject'`, declarations `origin='declared'` with their lines, read from `publicBody`: private takes and facts fences stripped; a single-token declaration is `case_sensitive`; `mentions.alias_deny` and frontmatter `alias_deny` reject as `denied`). `aliasRejection`: under 4 characters (unless a short code the page declares for itself: a declaration marked `selfCode`), a generic word, or the first word of the page's own multi-word name. Rejections are returned with their line for explain. Tests: `test/mentions-policy-aliases.test.ts`.- `src/core/mentions/short-codes.ts` — `isShortCode` (2-3 letters or digits with a capital, not in `SHORT_CODE_STOPLIST` of common capitals and business acronyms, compared upper-cased). `declarationsIn` marks a short code `selfCode` when an alias label or an `also called` / `a.k.a.` / `known as` / `short for` cue declares it in a sentence naming nothing before the cue but the page (empty, a pronoun or an own name); `aliasRejection` lets it through and `by-mention.ts` keeps a `declared`, `case_sensitive` short code in the gazetteer, matched only as written. Leaf module (aliases.ts imports by-mention.ts). Tests: `test/mentions-policy-aliases.test.ts`, `test/mentions-pass.test.ts`.-- `src/core/mentions/pass.ts` — `runMentionPass(engine, {sourceId?, deadline?, signal?, beforePublish?})`, run by `extractStaleFromDB` (unmanaged) and `extractManagedStaleLinks` (managed, unless `mentions: false`). Per source: policy off → `disableSource` deletes plain mention links (`link_kind` null or `plain`; `typed_ner` stays), derived alias rows, saved entries and page state once and records `disabled`; a changed policy fingerprint (types, cross-source flag, ignore list, `mentions.alias_deny`, `mentions.multiword_aliases`, `mentions.sibling_merge`) marks every page due and recounts `pending`; the status row's `policy_fingerprint` carries the extractor versions its count reflects after the hash (`<hash>;v<mention>.<alias>`, `INDEX_VERSION_STAMP`), and a stamp that differs from the binary's recounts `pending` once (version-behind pages are already due, so nothing is invalidated); `refreshAliases` rewrites derived rows of entity pages whose `page_mention_state.alias_revision`/`alias_version` differ (revision-checked, coordinated write when managed) and removes them from pages that stopped being entities; `buildGazetteer({strict: true})` failing writes nothing and records `failed`; `saveEntries` diffs `mention_gazetteer_entries` under `pg_advisory_xact_lock`, bumps the source generation and marks due only pages linking to a removed target plus pages a mini-gazetteer of added names matches (ASCII names prefiltered with `LIKE` on lower-cased text, others scanned in JS; a first save marks the source due); `reconcileSource` scans due pages (no state row, or `mention_revision`/`mention_version` differ) in batches of 200 and publishes only when the source generations (`FOR SHARE`) and each page's `knowledge_revision` (`FOR SHARE`) are unchanged, writing link diffs (`addLinksBatch`, audit site `extract.mentions`) and page state in one transaction; then the status row gets state, `pending` (`countIndexDuePages`: mention-due pages plus linkable entity pages whose derived aliases are behind their content or `ALIAS_DERIVATION_VERSION`, counted when the source finishes or the deadline stops it) and `last_pass_at`. `writeDerivedAliases` / `writePageAliases` are import's and `reindex --aliases`' alias projection. `MENTION_EXTRACTOR_VERSION` and `ALIAS_DERIVATION_VERSION` are independent of `LINK_EXTRACTOR_VERSION_TS`. The scanned text is title, compiled truth and timeline; frontmatter `mention_ignore` names are skipped per page. Tests: `test/mentions-pass.test.ts`, `test/mentions-coverage-version.test.ts`, `test/mentions-managed.test.ts`, `test/e2e/mentions-parity.test.ts`.+- `src/core/mentions/pass.ts` — `runMentionPass(engine, {sourceId?, deadline?, signal?, beforePublish?})`, run by `extractStaleFromDB` (unmanaged) and `extractManagedStaleLinks` (managed, unless `mentions: false`). #7031: on a belief-covered source a batch's link adds and drops run in an unmanaged `mention_pass` belief point (page keys locked first), so they never read as ledger drift (`test/belief-stale-links-7031.test.ts`). Per source: policy off → `disableSource` deletes plain mention links (`link_kind` null or `plain`; `typed_ner` stays), derived alias rows, saved entries and page state once and records `disabled`; a changed policy fingerprint (types, cross-source flag, ignore list, `mentions.alias_deny`, `mentions.multiword_aliases`, `mentions.sibling_merge`) marks every page due and recounts `pending`; the status row's `policy_fingerprint` carries the extractor versions its count reflects after the hash (`<hash>;v<mention>.<alias>`, `INDEX_VERSION_STAMP`), and a stamp that differs from the binary's recounts `pending` once (version-behind pages are already due, so nothing is invalidated); `refreshAliases` rewrites derived rows of entity pages whose `page_mention_state.alias_revision`/`alias_version` differ (revision-checked, coordinated write when managed) and removes them from pages that stopped being entities; `buildGazetteer({strict: true})` failing writes nothing and records `failed`; `saveEntries` diffs `mention_gazetteer_entries` under `pg_advisory_xact_lock`, bumps the source generation and marks due only pages linking to a removed target plus pages a mini-gazetteer of added names matches (ASCII names prefiltered with `LIKE` on lower-cased text, others scanned in JS; a first save marks the source due); `reconcileSource` scans due pages (no state row, or `mention_revision`/`mention_version` differ) in batches of 200 and publishes only when the source generations (`FOR SHARE`) and each page's `knowledge_revision` (`FOR SHARE`) are unchanged, writing link diffs (`addLinksBatch`, audit site `extract.mentions`) and page state in one transaction; then the status row gets state, `pending` (`countIndexDuePages`: mention-due pages plus linkable entity pages whose derived aliases are behind their content or `ALIAS_DERIVATION_VERSION`, counted when the source finishes or the deadline stops it) and `last_pass_at`. `writeDerivedAliases` / `writePageAliases` are import's and `reindex --aliases`' alias projection. `MENTION_EXTRACTOR_VERSION` and `ALIAS_DERIVATION_VERSION` are independent of `LINK_EXTRACTOR_VERSION_TS`. The scanned text is title, compiled truth and timeline; frontmatter `mention_ignore` names are skipped per page. Tests: `test/mentions-pass.test.ts`, `test/mentions-coverage-version.test.ts`, `test/mentions-managed.test.ts`, `test/e2e/mentions-parity.test.ts`.- `src/core/mentions/stale.ts` — the pass as a sweep phase: `previewMentionPass` (index-due pages per source, as `countIndexDuePages` counts them, and newest `last_pass_at`, for `--dry-run`, doctor `links_extraction_lag` and post-upgrade), `linkPhaseDeadline` (a budgeted sweep gives link work half its budget while mention-due pages exist), `formatMentionSummary`, `mentionJsonFields`.- `src/core/mentions/coverage.ts` — `readMentionCoverage(engine, sourceIds)`: `{state, pending_pages, last_pass_at, degraded?}` from `mention_index_status` plus due pages written at or after the row's `counted_at` (states `complete`, `pending`, `disabled`, `failed`; the card adds `type_not_linkable`). A row whose version stamp differs from the binary's (counted by another release) is recounted once with `countIndexDuePages` and the count and stamp saved on the row (compare-and-set on the stored fingerprint; a failed write still returns the count), so after a version bump every page behind `MENTION_EXTRACTOR_VERSION` or `ALIAS_DERIVATION_VERSION` reads as pending whatever its `updated_at`, and later reads stay one status read plus the recent-pages query. Policy settings changes are detected by the pass, not here. `mentionCoverageNotice` builds the `mention_index` notice (kind `degraded`, a per-call code): the free catch-up sweep for pending/failed, the user's setting for disabled/type-not-linkable, each with a read-only verify.- `src/core/mentions/referrers.ts` — the shared referrer query behind `referenced_by` and `get_backlinks {group:"page"}`: one row per referring page (every link source), semi-join over links to the target (plus identity co-members under `entity_identity.union`), canonical type via bound alias arrays, date `COALESCE(effective_date, updated_at)` with `date_source`, previews (160 characters after an optional `# ` heading, fences stripped, `redactFindings`). `readReferrerGroups`: per-type totals and a bounded top-10 per type, water-filled to 50 rows per card. `readReferrerPage`: keyset paging on `(date DESC, source_id, slug)` with an opaque cursor (microsecond UTC text). Untrusted scopes apply `privatePagesFilterFragment` to referrer and target and `privateLinkOriginFilterFragment` to the link, in rows and counts. #5575: quarantined referrers are left out (rows and counts) unless `includeQuarantined`, the scope's `eligibility` (`pageEligibleSql`) applies before LIMIT, and each row carries `trust_tier`, `origin` and `unconfirmed` from `trustFields`. Tests: `test/mentions-card-backlinks.test.ts`, `test/context-pack-newer-mentions.test.ts`, `test/entity-card-perf.slow.test.ts` (10,000-link hub gate). F1 records: `referrersCteBatched` is the same CTE with a target index (`referrerFilters` shared), `readTargetReferences` reads `referenced_by_count` plus enrolled-type records for many targets in two statements (totals per type; top `RECORD_ROWS_PER_TARGET` per target newest first; `RECORD_ROWS_PER_CALL` allotted in target order before hydration; rows, totals and cursors equal `readReferrerPage {type}`), `readRecordMetadata` adds `page_status`, `amends` (+ `since` from `link_relationships.first_start` in the caller's scope) and `supersedes`, also on `readReferrerPage {recordMetadata}` continuations. F3: `pageStatusValue` (string, trimmed, at most `PAGE_STATUS_MAX_CHARS` 40) is the one `page_status` rule; `readPageStatusByRef` reads it by `(source_id, slug)` in one statement (private pages hidden from untrusted callers, quarantined pages unless allowed) for `readReferrerPage {pageStatus}`, `get_backlinks` link rows and `context_pack` cards and newer-mention rows. Tests: `test/entity-records.test.ts`, `test/e2e/entity-records-postgres.test.ts`.diff --git a/src/core/mentions/pass.ts b/src/core/mentions/pass.tsindex 11f46fcac8..ab0079820e 100644--- a/src/core/mentions/pass.ts+++ b/src/core/mentions/pass.ts@@ -61,6 +61,7 @@ import {import { bodyBlocks, detachedValue } from '../page-body/segment-blocks.ts';import { commitRuns, derivationScope, keyDigest, planRuns, readRuns, SegmentDerivationRaced, type DeriveRun } from '../page-body/segment-derive.ts';import { canonicalJson } from '../belief/keys.ts';+import { beliefCoveredSources, unmanagedBeliefPoint } from '../belief/projection-point.ts';import { classifyItemFailure } from '../item-failure.ts';import { linkableTypesFor, loadSourcePack, parseNameList, readMentionPolicy, type MentionPolicy, type PackTypes } from './policy.ts';import { deriveEntityAliases, type DerivedAlias } from './aliases.ts';@@ -550,7 +551,12 @@ async function reconcileSource(engine: BrainEngine, sourceId: string, gazetteer:}if (!readable.length) continue;await opts.beforePublish?.(readable.map(p => Number(p.id)));+ // #7031: on a belief-covered source the link changes are an unmanaged projection point, like derived links; the+ // page keys are locked first, as every publisher locks them before the page rows.+ let beliefs = null as ReturnType<typeof unmanagedBeliefPoint> | null;const outcome = await engine.transaction(async tx => {+ const journaled = (await beliefCoveredSources(tx, [sourceId])).has(sourceId);+ if (journaled) await tx.lockPageKeys(readable.map(p => ({ sourceId: p.source_id, slug: p.slug })));if (!sameGenerations(generations, await readGenerations(tx, genSources, true))) return null;const current = new Map((await tx.executeRaw<{ id: number; revision: string }>(`SELECT id, knowledge_revision::text AS revision FROM pages WHERE id = ANY($1::int[]) AND deleted_at IS NULL FOR SHARE`,@@ -572,12 +578,18 @@ async function reconcileSource(engine: BrainEngine, sourceId: string, gazetteer:.map(m => `${p.id}\0${m.source_id}\0${m.slug}`)));const have = new Set(existing.map(r => `${r.from_page_id}\0${r.to_source_id}\0${r.to_slug}`));const drop = existing.filter(r => !want.has(`${r.from_page_id}\0${r.to_source_id}\0${r.to_slug}`)).map(r => Number(r.id));- if (drop.length) await tx.executeRaw('DELETE FROM links WHERE id = ANY($1::int[])', [drop]);const add: LinkBatchInput[] = fresh.flatMap(p => (found.get(Number(p.id)) ?? []).filter(m => want.has(`${p.id}\0${m.source_id}\0${m.slug}`) && !have.has(`${p.id}\0${m.source_id}\0${m.slug}`)).map(m => ({ from_slug: p.slug, to_slug: m.slug, link_type: 'mentions', link_source: 'mentions', context: m.name,from_source_id: p.source_id, to_source_id: m.source_id })));- const created = add.length ? await tx.addLinksBatch(add, { auditSite: 'extract.mentions' }) : 0; // gbrain-allow-direct-insert: mention pass — reconciling write of the scan's own plain mention rows+ const write = async () => {+ if (drop.length) await tx.executeRaw('DELETE FROM links WHERE id = ANY($1::int[])', [drop]);+ return add.length ? await tx.addLinksBatch(add, { auditSite: 'extract.mentions' }) : 0; // gbrain-allow-direct-insert: mention pass — reconciling write of the scan's own plain mention rows+ };+ const dropped = new Set(existing.filter(r => drop.includes(Number(r.id))).map(r => Number(r.from_page_id)));+ const changed = fresh.filter(p => dropped.has(Number(p.id))).map(p => p.slug).concat(add.map(link => link.from_slug));+ if (journaled && changed.length) beliefs = unmanagedBeliefPoint(sourceId, changed, 'mention_pass');+ const created = beliefs ? await beliefs.run(tx, write) : await write();await tx.executeRaw(`INSERT INTO page_mention_state (page_id, source_id, mention_revision, mention_version, mention_generation, mentions_scanned_at)SELECT id, $1, rev::uuid, $4, $5::bigint, now() FROM unnest($2::int[], $3::text[]) AS x(id, rev)@@ -586,6 +598,7 @@ async function reconcileSource(engine: BrainEngine, sourceId: string, gazetteer:[sourceId, fresh.map(p => p.id), fresh.map(p => p.revision), MENTION_EXTRACTOR_VERSION, generations.get(sourceId) ?? '0']);return { fresh: fresh.length, created, removed: drop.length };});+ await beliefs?.commit(engine);if (!outcome) { result.skipped += readable.length; return; }result.pages += outcome.fresh;result.skipped += readable.length - outcome.fresh;diff --git a/test/belief-stale-links-7031.test.ts b/test/belief-stale-links-7031.test.tsnew file mode 100644index 0000000000..7c49929492--- /dev/null+++ b/test/belief-stale-links-7031.test.ts@@ -0,0 +1,50 @@+/**+ * #7031: on a belief-covered managed source, every link the managed stale extraction writes is journaled, so the+ * cycle's drift check finds nothing to reconcile and the page interval stays certified.+ *+ * 1. Protects: a page published before the person it names had a page gets its links filled in later by+ * extractManagedStaleLinks: the derived (markdown) link and the mention pass's plain mention link. Each is recorded+ * as an unmanaged, marked publication, and scanBeliefDrift reports no drift for the page.+ * 2. Fails when: a writer in the stale extraction (the mention pass) adds or removes links outside a belief batch:+ * the page reads as drift and the cycle reconciles it as permanently uncertified.+ * 3. Why existing coverage misses it: the projection-point probe in belief-ledger-coverage.test.ts runs the unmanaged+ * extract path with links only; no probe runs the mention pass on a covered source.+ */+import { describe, expect, test } from 'bun:test';+import { scanBeliefDrift } from '../src/core/belief/coverage-report.ts';+import { extractManagedStaleLinks } from '../src/core/persistence/links-maintenance.ts';+import { beliefBrain, beliefEngineCase, beliefPage } from './helpers/belief-ledger.ts';+import { testBackends } from './helpers/test-backends.ts';++for (const engineCase of testBackends().map(beliefEngineCase)) {+ describe('#7031 managed stale link extraction on a belief-covered source (' + engineCase.name + ')', () => {+ test('links filled in by the stale extraction and its mention pass are journaled and leave no drift', async () => {+ await beliefBrain(async b => {+ await b.put('companies/acme-example', beliefPage({ prose: 'Acme Example met Bob Example last week.', links: ['people/alice-example'] }));+ await b.put('people/alice-example', beliefPage({ type: 'person', title: 'Alice Example' }));+ await b.put('people/bob-example', beliefPage({ type: 'person', title: 'Bob Example' }));+ const links = async () => (await b.engine.executeRaw<{ to: string; source: string }>(+ "SELECT t.slug AS to, l.link_source AS source FROM links l JOIN pages a ON a.id = l.from_page_id JOIN pages t ON t.id = l.to_page_id WHERE a.slug = 'companies/acme-example' ORDER BY 1"))+ .map(r => r.to + ':' + r.source);+ expect((await scanBeliefDrift(b.engine, b.sourceId, { deep: true })).drifted).toEqual([]);+ const before = await b.lastId();+ await extractManagedStaleLinks(b.engine, { sourceId: b.sourceId, slugs: ['companies/acme-example'] });+ expect(await links()).toEqual(['people/alice-example:markdown', 'people/bob-example:mentions']);+ const events = await b.events(before);+ const publications = [...new Set(events.map(e => e.publication_id))];+ const marks = await b.engine.executeRaw('SELECT 1 FROM belief_commit_marks WHERE publication_id = ANY($1::uuid[])', [publications]);+ expect(marks.length).toBe(publications.length);+ expect((await scanBeliefDrift(b.engine, b.sourceId, { deep: true })).drifted).toEqual([]);+ expect(events.filter(e => e.subject_kind === 'link').map(e => [e.slug, e.event_type, e.origin, e.cause.op]))+ .toEqual([['companies/acme-example', 'link.asserted', 'unmanaged', 'derived_links'], ['companies/acme-example', 'link.asserted', 'unmanaged', 'mention_pass']]);+ // The mention is edited away: the pass removes its link, journaled the same way.+ await b.put('companies/acme-example', beliefPage({ prose: 'Acme Example met nobody last week.', links: ['people/alice-example'] }));+ const removal = await b.lastId();+ await extractManagedStaleLinks(b.engine, { sourceId: b.sourceId, slugs: ['companies/acme-example'] });+ expect(await links()).toEqual(['people/alice-example:markdown']);+ expect((await b.events(removal)).filter(e => e.subject_kind === 'link').map(e => [e.event_type, e.cause.op])).toEqual([['link.retracted', 'mention_pass']]);+ expect((await scanBeliefDrift(b.engine, b.sourceId, { deep: true })).drifted).toEqual([]);+ }, { managed: true, databaseUrl: engineCase.databaseUrl });+ }, 180_000);+ });+}