Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(search): hide quarantined-page and needs_rederive facts on every entity-fact read
garrytan8c0f5155f062
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4b6906fc4f..8500d93586 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -10,6 +10,24 @@ credits are retained; no result has been reassigned to another provider. Originaidentifiers and attribution are available in the pre-removal Git revision`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).+## [Unreleased]++**A saved fact from a quarantined page, or one waiting to be re-derived after a purge, no longer reaches agents through search, query, trajectories, entity cards, standing questions, local `gbrain recall` or consolidation.**++Before, the facts `search` and `query` put first when a query names an entity skipped the read-eligibility check that recall and `saved_facts` apply, so remote MCP callers too got facts saved from a page the content gate quarantined, and facts a purge had hidden until a deriver regenerates them. The same check was missing from a few other fact reads. Now each one hides those facts the way recall does.++### Itemized changes++- `search` and `query` entity facts (`search/entity-facts.ts`) hide quarantined-page and `needs_rederive` facts, local and remote, and apply the caller's trust floor as `saved_facts` does.+- `find_trajectory`, `gbrain think`'s trajectory read, `gbrain eval trajectory` and `gbrain founder scorecard` hide them too.+- The entity card's `active_fact_count` counts only the facts the card may show.+- A standing question's refresh gathers no fact, take or timeline row awaiting re-derivation as evidence.+- Local `gbrain recall` (every arm, `--supersessions` and `--since-last-run` included) hides them like the `recall` op.+- The consolidate phase never promotes such a fact into a take.+- Tests: `test/search-entity-facts-eligibility.test.ts` (PGLite and Postgres), red with each read's filter removed.++Contributed by @tarush1989 (#7115, rebuilt on current master).+## [0.60.539.0] - 2026-11-01**A remote `put_page` of a page an agent read whole now keeps its frontmatter and the code blocks it added, and timeline sources and ontology values are quoted below "your notes" like the rest of their rows.**diff --git a/docs/architecture/key-files/entity-recall.md b/docs/architecture/key-files/entity-recall.mdindex 64efcf6f6d..8828f96ab3 100644--- a/docs/architecture/key-files/entity-recall.md+++ b/docs/architecture/key-files/entity-recall.md@@ -23,7 +23,7 @@ card and `get_backlinks`. Guide: [entity recall](../../guides/entity-recall.md).- `src/core/verbs/entity-references.ts` — the F1 reference fields on compact rows: `readRowReferences` (config gate per surface `names` / `pack` / `full_card`, `REFERENCES_DEADLINE_MS` 2 s cooperative deadline, `references_unavailable` marker + degraded notice, per-type `next` to `get_backlinks` returning exactly the rest, `record_types_undeclared` notice), `readPackReferences` (context_pack pull verb only: card page + identity siblings), `renderRecordLines` / `RECORDS_HEADER` (packed text, "page date"). Config: `src/core/entity-references-config.ts` (`entity.names_references` default on, `entity.record_types` default empty). Tests: `test/entity-records.test.ts`, `test/e2e/entity-records-postgres.test.ts`.- `src/core/verbs/entity-card-identity.ts` — the `entity` card's `aka_sources`, `identity_siblings`, `identity_excerpt` (verbatim lines of the page and its siblings from `sanitizeRemoteBody`, tiered: declarations, label lines with a name-like value, other label lines, lines with another capitalized run or code; 600 characters per page, 1,200 per card, `identity_excerpt_omitted`), `alias_guidance` (surface-aware) and the `identity_siblings_capped` notice. A segmented page is excerpted from its trusted, withdrawal-overlaid lead (`truthLead`, `EXCERPT_LEAD_BYTES` = 64 KiB, cut back to a whole line), never its body getters; the card equals the inline page's when the qualifying lines sit in that lead (`test/large-pages/segmented-read-refusals.test.ts`). `readSiblingIdentity` is its siblings-and-aliases half (one alias read for the page and its siblings; optional preloaded mention policy and pack), used alone by `entity names[]`. Tests: `test/mentions-identity-siblings.test.ts`.- `src/core/search/alias-fanout.ts` — search and query's alias fan-out: `resolveQueryEntity` (query n-grams up to 4 tokens through `resolveAliases` and linkable exact titles, longest first, capitalized or code-shaped unless a multi-word n-gram resolves to one page, case-sensitive aliases as written, a tie only when the pages are identity siblings), the names of the entity and its siblings by origin then length, at most `search.alias_fanout_max` (4) keyword queries of 5 rows (`searchAliasRequired`: the name as a required websearch phrase, ranked by the query's other terms through `SearchOpts.rankQuery` on both engines), at most 8 spliced rows with `matched_alias`, and the `fanout` meta (`alias_fanout` notice when capped). Offset 0 only. `resolving` takes the handler's `resolveQueryEntity` started before the search, so its reads overlap it. Tests: `test/search-alias-fanout.test.ts`, `test/search-other-names.test.ts`, `test/e2e/mentions-parity.test.ts`. `wholeQueryEntityNames`: when the whole query is one entity's name or alias, `match: "keyword"` lists pages matching any of its names (one websearch OR of phrases; `match_names` in the meta and the count line; rows found by another name carry `matched_alias`). `resolveQueryEntities` returns every entity a text names; its alias, linkable-title and case-sensitive alias reads run concurrently (`test/search-entity-resolve-concurrency.test.ts`).-- `src/core/search/entity-facts.ts` — `entitySavedFacts`: when a search names an entity, its newest active saved facts (`factEntitySlugs`, 5) lead the model-visible saved-facts block, then up to 3 facts each of up to 2 entities those facts name (`linked_from`), before the word-overlap matches. Same active, audit-row and visibility rules as recall. Tests: `test/memory-sibling-facts.test.ts`.+- `src/core/search/entity-facts.ts` — `entitySavedFacts`: when a search names an entity, its newest active saved facts (`factEntitySlugs`, 5) lead the model-visible saved-facts block, then up to 3 facts each of up to 2 entities those facts name (`linked_from`), before the word-overlap matches. Same active, audit-row, visibility and read-eligibility rules as recall (`projectionEligibleSql`: quarantined-page and `needs_rederive` facts hidden, the caller's trust floor applied). Tests: `test/memory-sibling-facts.test.ts`, `test/search-entity-facts-eligibility.test.ts` (also the trajectory, entity-card count, standing-question evidence, local CLI recall and consolidate reads, both engines).- `src/core/search/saved-facts-match.ts` — `matchingSavedFacts`, the word-matched saved facts `search` and `query` report in `retrieval.saved_facts` after the named entity's facts (`ops/search.ts` `buildRetrievalResponseMeta`). Words match whole after stemming with the facts keyword index's expression (`factsFtsDocument`), CJK words as substrings; a fact needs three quarters of the query's words, at most four, question-framing words (`QUESTION_FRAME_WORDS`: count, number, total, amount) not counted; most matched words first, then newest (time, then id), at most five after the #7043 collapse. One statement behind the facts-only EXISTS probe (`test/search/saved-facts-probe.test.ts`). `search.saved_facts_supersede` (default on; `false` turns it off): a newer fact with a correction cue that shares two matched words with a returned fact is returned too, even below the threshold, and the facts it corrects move last with `corrected_by` (the newest correction); every correction of a returned fact comes back; a cue fact that adds no figure to the older fact (`figures`: numbers, with ISO and day-month dates and times left out; `CORRECTION_CUE` leaves out "corrected"/"correcting", which report a correction) corrects nothing when it restates one of its figures or the older fact is itself a correction, nor when it states no figure and withdraws nothing (`RETRACTION`); nor does one saved under another entity, of lower trust, or sharing none of the older fact's matched names (`namedWords`). Correctors rank newest first under the cap and a corrected fact whose corrector is cut is dropped; `buildRetrievalResponseMeta` carries `corrected_by` onto the entity copy. Also owns `FACT_MATCH_STOPWORDS`. Tests: `test/search-saved-facts-match.test.ts`, `test/search-saved-facts.test.ts`.- `src/core/schema-migrations/v206-entity-mention-index.ts` — additive: `page_aliases.origin` (`frontmatter` default, `declared`, `subject`), `case_sensitive`, `alias_text`, unique key `(source_id, alias_norm, slug, origin)`; tables `page_mention_state`, `mention_gazetteer_entries`, `mention_index_status` (RLS on when the role bypasses RLS). `GUARDED_TABLE_COLUMNS` lists the new `page_aliases` columns.diff --git a/scripts/postgres-arm-weights.json b/scripts/postgres-arm-weights.jsonindex a9a773f07d..839d251700 100644--- a/scripts/postgres-arm-weights.json+++ b/scripts/postgres-arm-weights.json@@ -439,6 +439,7 @@"test/schema-missing-tables.test.ts": 5956,"test/schema-replay-deadlock-6143.test.ts": 6758,"test/scripts/test-stress-postgres.test.ts": 12439,+ "test/search-entity-facts-eligibility.test.ts": 7900,"test/search-read-keyword-bound.test.ts": 5120,"test/search-remote-page-cap.test.ts": 7365,"test/search-transcript-quote-8384.test.ts": 16171,diff --git a/src/commands/recall.ts b/src/commands/recall.tsindex 60bab0385a..6cdad8e4da 100644--- a/src/commands/recall.ts+++ b/src/commands/recall.ts@@ -850,6 +850,7 @@ async function fetchCursorPage(engine: BrainEngine, flags: ParsedFlags, sourceIdactiveOnly: !flags.includeExpired,limit: flags.limit,excludeAuditRows: true,+ eligibility: {},grep: flags.grep ?? undefined,sessionId: flags.sessionId || undefined,...narrow,@@ -889,6 +890,7 @@ async function fetchRowsLocal(return engine.listSupersessions(sourceId, {since: resolvedSince ?? undefined,limit: flags.limit,+ eligibility: {},});}// An explicit `--since DURATION` / `--today` cutoff is a "what happened in@@ -910,6 +912,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},}): engine.listFactsByEntity(sourceId, entitySlug, {sessionId: flags.sessionId || undefined,@@ -917,6 +920,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},})));const rows = mergeNewestFirst(lists, flags.limit, resolvedSince && !windowEventTime ? r => r.created_at : r => r.valid_from ?? r.created_at);// #4720: the bare positional is entity-first, but keyless/casual usage@@ -936,6 +940,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.entity.toLowerCase(),excludeAuditRows: true,+ eligibility: {},});if (textRows.length > 0) {noteEntityTextFallback(flags.entity, textRows.length);@@ -953,6 +958,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},});}return engine.listFactsBySession(sourceId, flags.sessionId, {@@ -960,6 +966,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},});}if (resolvedSince) {@@ -978,6 +985,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},});}return engine.listFactsSince(sourceId, new Date(0), {@@ -986,6 +994,7 @@ async function fetchRowsLocal(limit: flags.limit,grep: flags.grep ?? undefined,excludeAuditRows: true,+ eligibility: {},});}diff --git a/src/core/cycle/phases/consolidate.ts b/src/core/cycle/phases/consolidate.tsindex 106e4d7b64..100fce72cc 100644--- a/src/core/cycle/phases/consolidate.ts+++ b/src/core/cycle/phases/consolidate.ts@@ -224,6 +224,7 @@ export async function runPhaseConsolidate(unconsolidatedOnly: true,visibility: managed ? ['world'] : undefined,limit: 100,+ eligibility: {},});// Takes here are the user's own (holder 'self'): a claim the assistant or// a named third party asserted is never promoted into one.diff --git a/src/core/engine-sql/facts.ts b/src/core/engine-sql/facts.tsindex eb099c15cc..d103efec95 100644--- a/src/core/engine-sql/facts.ts+++ b/src/core/engine-sql/facts.ts@@ -703,6 +703,7 @@ export async function findTrajectory(exec: LegacyUnscopedRead, opts: import('../AND entity_slug = ${opts.entitySlug}AND expired_at IS NULL${remoteFilter ? sqlFragment`AND visibility = 'world'` : sqlFragment``}${privacyClause(opts)}+ AND ${trustedSql(projectionEligibleSql('facts', 'facts', undefined))}${metric !== null ? sqlFragment`AND claim_metric = ${metric}` : sqlFragment``}${kind === 'metric' ? sqlFragment`AND claim_metric IS NOT NULL` : sqlFragment``}${kind === 'event' ? sqlFragment`AND event_type IS NOT NULL` : sqlFragment``}diff --git a/src/core/ops/search.ts b/src/core/ops/search.tsindex e064863c4e..5998ae32fa 100644--- a/src/core/ops/search.ts+++ b/src/core/ops/search.ts@@ -420,7 +420,7 @@ async function buildRetrievalResponseMeta(timed('readiness', probeProjectionReadiness(ctx.engine, readinessScope).then(async r =>[r, r.status === 'projection_pending' ? await countPendingProjections(ctx.engine, readinessScope, relevance) : null] as const)),// Facts saved about the entity the query names come first (newest first, one hop through entities they name).- timed('entity_facts', opts.entity && ctx.emitResponseMeta ? entitySavedFacts(ctx.engine, opts.entity, { remote: ctx.remote !== false, excludePrivate }) : []),+ timed('entity_facts', opts.entity && ctx.emitResponseMeta ? entitySavedFacts(ctx.engine, opts.entity, { remote: ctx.remote !== false, excludePrivate, minTrust: scope.minTrust }) : []),timed('saved_facts', matchingSavedFacts(ctx, scope, queryText, aliases.flatMap(a => [a.name, a.alias]))),timed('held', stampHeldHits(ctx.engine, results as SearchResult[], scope, relevance, ctx).catch(() => ({ coverage: [], couldAffect: 0 }))),// GBRA-133 PR 2a.5: per-source freshness from the notifier's cached coverage record (no git call).diff --git a/src/core/questions/refresh.ts b/src/core/questions/refresh.tsindex a8642f6067..364c3668e4 100644--- a/src/core/questions/refresh.ts+++ b/src/core/questions/refresh.ts@@ -31,6 +31,7 @@ import { classifyLlmCallFailure } from '../think/index.ts';import { PINNED_QUESTION_MARKER, normalizeQuestion, sentenceId } from './identity.ts';import { entityAnchoredPages, prefixAnchoredPages } from '../search/entity-anchor.ts';import { quarantinedProvenanceFilterFragment, quarantineFilterFragment } from '../quarantine.ts';+import { projectionEligibleSql } from '../eligibility/sql.ts';import { FACT_HASH_SQL, OWNER_BODY_HASH_SQL, TAKE_HASH_SQL, TIMELINE_HASH_SQL, evaluateAnswer, type EvidenceKind } from './freshness.ts';import { ownerClaims, readQuestionPage } from './pages.ts';import { neutralizeForPrompt } from '../think/sanitize.ts';@@ -225,6 +226,7 @@ export async function retrieveEvidence(engine: BrainEngine, pin: PinRow): Promis`SELECT f.id, f.fact, f.entity_slug, f.valid_from, ${FACT_HASH_SQL('f')} AS hash FROM facts fWHERE f.source_id = $1 AND f.entity_slug = ANY($2::text[]) AND f.expired_at IS NULL AND f.superseded_by IS NULLAND (f.valid_until IS NULL OR f.valid_until > now()) AND ${quarantinedProvenanceFilterFragment('f')}+ AND ${projectionEligibleSql('facts', 'f', undefined)}AND NOT EXISTS (SELECT 1 FROM pages fp WHERE fp.source_id = f.source_id AND fp.slug = f.source_markdown_slugAND (fp.type IN ('question', 'synthesis') OR fp.frontmatter ? '${PINNED_QUESTION_MARKER}'))ORDER BY f.valid_from DESC, f.id DESC LIMIT ${MAX_FACTS}`, [pin.source_id, entitySlugs]);@@ -236,7 +238,7 @@ export async function retrieveEvidence(engine: BrainEngine, pin: PinRow): Promisif (pageIds.length > 0) {const timeline = await engine.executeRaw<{ id: number; date: unknown; summary: string; detail: string; page_id: number; slug: string; generation: string | number; revision: string | null; hash: string }>(`SELECT t.id, t.date, t.summary, t.detail, p.id AS page_id, p.slug, p.generation, p.knowledge_revision::text AS revision, ${TIMELINE_HASH_SQL('t')} AS hash- FROM timeline_entries t JOIN pages p ON p.id = t.page_id WHERE p.id = ANY($1::integer[]) ORDER BY t.date DESC, t.id DESC LIMIT ${MAX_TIMELINE}`, [pageIds]);+ FROM timeline_entries t JOIN pages p ON p.id = t.page_id WHERE p.id = ANY($1::integer[]) AND ${projectionEligibleSql('timeline_entries', 't', undefined)} ORDER BY t.date DESC, t.id DESC LIMIT ${MAX_TIMELINE}`, [pageIds]);for (const t of timeline) {push({ kind: 'timeline', source_id: pin.source_id, page_id: t.page_id, page_slug: t.slug, page_generation: Number(t.generation),page_revision: t.revision, item_id: t.id, content_hash: t.hash, text: `[timeline ${t.slug} ${String(t.date).slice(0, 10)}] ${t.summary}${t.detail ? ` — ${t.detail.slice(0, 300)}` : ''}` });@@ -244,6 +246,7 @@ export async function retrieveEvidence(engine: BrainEngine, pin: PinRow): Promisconst takes = await engine.executeRaw<{ id: string | number; claim: string; holder: string; weight: number | null; page_id: number; slug: string; generation: string | number; revision: string | null; hash: string }>(`SELECT k.id, k.claim, k.holder, k.weight, p.id AS page_id, p.slug, p.generation, p.knowledge_revision::text AS revision, ${TAKE_HASH_SQL('k')} AS hashFROM takes k JOIN pages p ON p.id = k.page_id WHERE p.id = ANY($1::integer[]) AND k.active AND k.superseded_by IS NULL+ AND ${projectionEligibleSql('takes', 'k', undefined)}ORDER BY k.id DESC LIMIT ${MAX_TAKES}`, [pageIds]);for (const k of takes) {push({ kind: 'take', source_id: pin.source_id, page_id: k.page_id, page_slug: k.slug, page_generation: Number(k.generation),diff --git a/src/core/search/entity-facts.ts b/src/core/search/entity-facts.tsindex 06220da541..15e6357f94 100644--- a/src/core/search/entity-facts.ts+++ b/src/core/search/entity-facts.ts@@ -10,7 +10,9 @@* returned first. One hop: an entity a returned fact names ("folded into* Dovalto Energy", "uses Dovalto Energy's discount code") contributes its own* newest facts, tagged `linked_from`, so a redirection reaches the value it- * points to. Same source scope, active-fact and visibility rules as recall,+ * points to. Same source scope, active-fact, visibility and read-eligibility+ * rules as recall (quarantined-page and needs_rederive facts hidden, the+ * caller's trust floor applied),* and recall's collapse (#7043): copies of one claim read as one fact, before* the caps, so they never crowd out a distinct one.*/@@ -22,6 +24,8 @@ import { factEntitySlugs } from '../mentions/siblings.ts';import { resolveQueryEntities, type ResolvedEntity } from './alias-fanout.ts';import { collapseFactReads } from '../facts/capture-dedup.ts';import type { FactAttribution } from '../engine.ts';+import { projectionEligibleSql } from '../eligibility/sql.ts';+import type { TrustTier } from '../trust/tier.ts';export const ENTITY_FACTS_CAP = 5;export const LINKED_FACTS_CAP = 3;@@ -42,8 +46,8 @@ export function stripCollapseColumns<T extends CollapseColumns & { collapsed_idsreturn rest;}-async function newestFacts(engine: BrainEngine, sourceId: string, slugs: string[], remote: boolean, limit: number): Promise<Array<EntityFact & CollapseColumns>> {- const visibility = remote ? `AND f.visibility = 'world' AND ${privateProvenanceFilterFragment('f')}` : '';+async function newestFacts(engine: BrainEngine, sourceId: string, slugs: string[], opts: { remote: boolean; minTrust?: TrustTier }, limit: number): Promise<Array<EntityFact & CollapseColumns>> {+ const visibility = opts.remote ? `AND f.visibility = 'world' AND ${privateProvenanceFilterFragment('f')}` : '';const rows = await engine.executeRaw<EntityFact & CollapseColumns>(`SELECT f.id, f.fact, f.entity_slug, f.kind, f.valid_from::text AS valid_from, f.source, ${SAVED_FACT_COLLAPSE_COLUMNS}FROM facts f@@ -51,6 +55,7 @@ async function newestFacts(engine: BrainEngine, sourceId: string, slugs: string[AND f.expired_at IS NULL AND (f.valid_until IS NULL OR f.valid_until > now())AND f.source != ALL($3::text[])${visibility}+ AND ${projectionEligibleSql('facts', 'f', { floor: opts.minTrust })}ORDER BY f.valid_from DESC, f.id DESCLIMIT ${SAVED_FACTS_COLLAPSE_WINDOW}`,[sourceId, slugs, [...AUDIT_ROW_SOURCES]]);@@ -58,12 +63,12 @@ async function newestFacts(engine: BrainEngine, sourceId: string, slugs: string[}/** The named entity's newest facts, then those of entities its facts name, with the collapse columns. Fail-soft: errors return []. */-export async function entitySavedFacts(engine: BrainEngine, entity: ResolvedEntity, opts: { remote: boolean; excludePrivate: boolean }): Promise<Array<EntityFact & CollapseColumns>> {+export async function entitySavedFacts(engine: BrainEngine, entity: ResolvedEntity, opts: { remote: boolean; excludePrivate: boolean; minTrust?: TrustTier }): Promise<Array<EntityFact & CollapseColumns>> {try {const sourceId = entity.pages[0]!.source_id;const own = new Set<string>();for (const p of entity.pages) for (const s of await factEntitySlugs(engine, sourceId, p.slug, { excludePrivate: opts.excludePrivate })) own.add(s);- const facts = await newestFacts(engine, sourceId, [...own], opts.remote, ENTITY_FACTS_CAP);+ const facts = await newestFacts(engine, sourceId, [...own], opts, ENTITY_FACTS_CAP);const out: Array<EntityFact & CollapseColumns> = [...facts];const seenEntities = new Set(own);let linked = 0;@@ -76,7 +81,7 @@ export async function entitySavedFacts(engine: BrainEngine, entity: ResolvedEntiif ([...slugs].some(s => seenEntities.has(s))) continue;for (const s of slugs) seenEntities.add(s);linked++;- for (const g of await newestFacts(engine, sourceId, [...slugs], opts.remote, LINKED_FACTS_CAP)) {+ for (const g of await newestFacts(engine, sourceId, [...slugs], opts, LINKED_FACTS_CAP)) {if (!out.some(x => x.id === g.id)) out.push({ ...g, linked_from: f.entity_slug ?? entity.pages[0]!.slug });}}diff --git a/src/core/verbs/entity-card.ts b/src/core/verbs/entity-card.tsindex 580e0879b6..1818c80666 100644--- a/src/core/verbs/entity-card.ts+++ b/src/core/verbs/entity-card.ts@@ -27,6 +27,7 @@ import type { ReadEligibility } from '../eligibility/policy.ts';import { trustFields } from '../eligibility/labels.ts';import { isQuarantined } from '../quarantine.ts';import { loadPageTrust } from '../eligibility/stamp.ts';+import { projectionEligibleSql } from '../eligibility/sql.ts';import { loadRelationshipNotes, relationshipNoteKey } from '../link-relationship-notes.ts';import { normalizeAlias } from '../search/alias-normalize.ts';import { isIdentityEntity, slugify } from '../entities/resolve.ts';@@ -654,7 +655,8 @@ async function assembleCard(`SELECT COUNT(*) AS nFROM factsWHERE source_id = $1 AND entity_slug = ANY($2::text[])- AND expired_at IS NULL${remote ? ` AND visibility = 'world'` : ''}${excludePrivate ? ` AND ${privateProvenanceFilterFragment('facts')}` : ''}`,+ AND expired_at IS NULL${remote ? ` AND visibility = 'world'` : ''}${excludePrivate ? ` AND ${privateProvenanceFilterFragment('facts')}` : ''}+ AND ${projectionEligibleSql('facts', 'facts', eligibility)}`,[sourceId, factSlugs],).then(rs => Number(rs[0]?.n ?? 0))diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txtindex 3f918214ad..cce3c47b1f 100644--- a/test/postgres-unit-arms.txt+++ b/test/postgres-unit-arms.txt@@ -457,6 +457,7 @@ test/retrieval-coverage-notices.test.tstest/schema-missing-tables.test.tstest/schema-replay-deadlock-6143.test.tstest/scripts/test-stress-postgres.test.ts+test/search-entity-facts-eligibility.test.tstest/search-read-keyword-bound.test.tstest/search-remote-page-cap.test.tstest/search-transcript-quote-8384.test.tsdiff --git a/test/search-entity-facts-eligibility.test.ts b/test/search-entity-facts-eligibility.test.tsnew file mode 100644index 0000000000..b6cc7dea6c--- /dev/null+++ b/test/search-entity-facts-eligibility.test.ts@@ -0,0 +1,195 @@+/**+ * Saved facts from a quarantined page, and facts awaiting re-derivation+ * (needs_rederive), never surface on the entity-fact readers: search and+ * query's entity facts (search/entity-facts.ts, local and remote), the+ * trajectory read (find_trajectory, think), the entity card's fact count, a+ * pinned question's evidence (facts, takes and timeline), local CLI recall,+ * and the consolidate phase's candidates.+ * Fails when: one of those reads drops projectionEligibleSql (eligibility/sql.ts).+ * PGLite always, Postgres too when DATABASE_URL is set.+ */+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';+import { mkdtempSync, rmSync } from 'node:fs';+import { tmpdir } from 'node:os';+import { join } from 'node:path';+import type { BrainEngine } from '../src/core/engine.ts';+import type { OperationContext } from '../src/core/ops/contract.ts';+import { PGLiteEngine } from '../src/core/pglite-engine.ts';+import { importFromContent } from '../src/core/import-file.ts';+import { registerLocalWriter } from '../src/core/persistence/identity.ts';+import { operations } from '../src/core/operations.ts';+import { QUARANTINE_KEY } from '../src/core/quarantine.ts';+import { buildEntityCard } from '../src/core/verbs/entity-card.ts';+import { retrieveEvidence } from '../src/core/questions/refresh.ts';+import type { PinRow } from '../src/core/questions/store.ts';+import { runRecall } from '../src/commands/recall.ts';+import { runPhaseConsolidate } from '../src/core/cycle/phases/consolidate.ts';+import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';+import { isolatedPersistencePostgres } from './helpers/persistence-postgres.ts';+import { withEnv } from './helpers/with-env.ts';++const home = mkdtempSync(join(tmpdir(), 'gbrain-entity-facts-elig-'));+const env = { GBRAIN_HOME: home };+const engines: BrainEngine[] = [];+let closePostgres: (() => Promise<void>) | undefined;++const ENTITY = 'people/alice-example';+const op = (name: string) => operations.find(o => o.name === name)!;+const page = (title: string, type: string, body: string) => `---\ntitle: ${title}\ntype: ${type}\n---\n\n${body}\n`;++interface Seed { visible: number; quarantined: number; rederive: number; take: number; hiddenTake: number; timeline: number; hiddenTimeline: number }+const seeds = new Map<BrainEngine, Seed>();++async function insertFact(engine: BrainEngine, text: string, sourcePage: string | null): Promise<number> {+ const [row] = await engine.executeRaw<{ id: number }>(+ `INSERT INTO facts (source_id, entity_slug, fact, kind, source, visibility, valid_from, source_markdown_slug)+ VALUES ('default', $1, $2, 'fact', 'test', 'world', now() - interval '1 day', $3) RETURNING id`, [ENTITY, text, sourcePage]);+ return Number(row!.id);+}+const markRederive = (engine: BrainEngine, table: string, id: number) => engine.executeRaw(+ `INSERT INTO needs_rederive (derived_table, derived_id, source_id, reason) VALUES ($1, $2, 'default', 'test')`, [table, String(id)]);++async function seed(engine: BrainEngine): Promise<Seed> {+ await importFromContent(engine, ENTITY, page('Alice Example', 'person', 'Alice Example runs the ledger guild.'), { noEmbed: true, sourceId: 'default' });+ await importFromContent(engine, 'notes/ledger-guild', page('Ledger guild', 'note', 'Minutes of the ledger guild.'), { noEmbed: true, sourceId: 'default' });+ await importFromContent(engine, 'notes/held-note', page('Held note', 'note', 'A note the content gate held.'), { noEmbed: true, sourceId: 'default' });+ await engine.executeRaw(`UPDATE pages SET frontmatter = COALESCE(frontmatter, '{}'::jsonb) || jsonb_build_object($1::text, '{"reason":"junk_pattern","detail":"test","assessed_at":"2026-10-07T00:00:00Z"}'::jsonb) WHERE source_id = 'default' AND slug = 'notes/held-note'`, [QUARANTINE_KEY]);+ const visible = await insertFact(engine, 'Chairs the ledger guild meetings', 'notes/ledger-guild');+ const quarantined = await insertFact(engine, 'Approved the vendor wire transfer', 'notes/held-note');+ const rederive = await insertFact(engine, 'Keeps the vault combination', null);+ await markRederive(engine, 'facts', rederive);+ const [pageRow] = await engine.executeRaw<{ id: number }>(`SELECT id FROM pages WHERE source_id = 'default' AND slug = $1`, [ENTITY]);+ await engine.addTakesBatch([+ { page_id: Number(pageRow!.id), row_num: 1, claim: 'Alice will ship the ledger audit', kind: 'take', holder: 'world', weight: 0.7 },+ { page_id: Number(pageRow!.id), row_num: 2, claim: 'Alice derived a purged estimate', kind: 'take', holder: 'world', weight: 0.7 },+ ]);+ const takes = await engine.executeRaw<{ id: number; row_num: number }>('SELECT id, row_num FROM takes WHERE page_id = $1 ORDER BY row_num', [Number(pageRow!.id)]);+ await markRederive(engine, 'takes', Number(takes[1]!.id));+ await engine.addTimelineEntry(ENTITY, { date: '2026-09-01', summary: 'Alice joined the ledger guild' }, { sourceId: 'default' });+ await engine.addTimelineEntry(ENTITY, { date: '2026-09-02', summary: 'Alice summary derived from a purged input' }, { sourceId: 'default' });+ const timeline = await engine.executeRaw<{ id: number; summary: string }>(+ 'SELECT t.id, t.summary FROM timeline_entries t JOIN pages p ON p.id = t.page_id WHERE p.id = $1 ORDER BY t.date', [Number(pageRow!.id)]);+ await markRederive(engine, 'timeline_entries', Number(timeline[1]!.id));+ return { visible, quarantined, rederive, take: Number(takes[0]!.id), hiddenTake: Number(takes[1]!.id), timeline: Number(timeline[0]!.id), hiddenTimeline: Number(timeline[1]!.id) };+}++beforeAll(async () => withEnv(env, async () => {+ const lite = new PGLiteEngine(); await lite.connect({}); await lite.initSchema(); engines.push(lite);+ if (process.env.DATABASE_URL) {+ const pg = await isolatedPersistencePostgres(process.env.DATABASE_URL);+ engines.push(pg.engine); closePostgres = pg.close;+ }+ for (const engine of engines) { await registerLocalWriter(engine, 'cli'); seeds.set(engine, await seed(engine)); }+}), 120_000);+afterAll(async () => { await engines[0]?.disconnect(); await closePostgres?.(); rmSync(home, { recursive: true, force: true }); });++function ctx(engine: BrainEngine, remote: boolean, sink: { meta?: Record<string, any> }): OperationContext {+ return { engine, config: { engine: engine.kind }, logger: { info() {}, warn() {}, error() {} }, dryRun: false, remote, sourceId: 'default',+ ...(remote ? { auth: { token: 't', clientId: 'c1', scopes: ['read'], sourceId: 'default' } } : {}),+ emitResponseMeta: (key: string, value: unknown) => { if (key === 'retrieval') sink.meta = value as Record<string, any>; },+ } as unknown as OperationContext;+}+const visibleOnly = (s: Seed) => ({ visible: true, quarantined: false, rederive: false, ids: [s.visible] });+const shape = (s: Seed, ids: number[]) => ({ visible: ids.includes(s.visible), quarantined: ids.includes(s.quarantined), rederive: ids.includes(s.rederive),+ ids: ids.filter(id => [s.visible, s.quarantined, s.rederive].includes(id)).sort((a, b) => a - b) });++describe('entity facts hide quarantined-page and needs_rederive facts', () => {+ test('search and query entity facts, local and remote', async () => withEnv(env, async () => {+ const seen: unknown[] = [];+ for (const engine of engines) {+ const s = seeds.get(engine)!;+ for (const remote of [false, true]) {+ for (const name of ['search', 'query'] as const) {+ const sink: { meta?: Record<string, any> } = {};+ const rows = await op(name).handler(ctx(engine, remote, sink), { query: 'Alice Example', limit: 10 }) as Array<{ fact_id?: string }>;+ // query's facts arm may return a fact as a row of its own, and then saved_facts leaves it out.+ const ids = [...((sink.meta?.saved_facts ?? []) as Array<{ id: number }>).map(f => Number(f.id)), ...rows.flatMap(r => r.fact_id ? [Number(r.fact_id)] : [])];+ seen.push({ engine: engine.kind, remote, name, ...shape(s, ids) });+ }+ }+ }+ expect(seen).toEqual(engines.flatMap(engine => [false, true].flatMap(remote => (['search', 'query'] as const).map(name =>+ ({ engine: engine.kind, remote, name, ...visibleOnly(seeds.get(engine)!) })))));+ }), 120_000);++ test('search entity facts apply the caller trust floor, as saved_facts does', async () => withEnv(env, async () => {+ const seen: unknown[] = [];+ for (const engine of engines) {+ const sink: { meta?: Record<string, any> } = {};+ await op('search').handler(ctx(engine, true, sink), { query: 'Alice Example', limit: 10, min_trust: 'operator_curated' });+ const [tier] = await engine.executeRaw<{ trust_tier: string }>('SELECT trust_tier FROM facts WHERE id = $1', [seeds.get(engine)!.visible]);+ seen.push({ engine: engine.kind, tier: tier!.trust_tier, saved_facts: ((sink.meta?.saved_facts ?? []) as Array<{ id: number }>).map(f => Number(f.id)) });+ }+ expect(seen).toEqual(engines.map(engine => ({ engine: engine.kind, tier: 'unknown', saved_facts: [] })));+ }), 120_000);++ test('trajectory, entity card count and local CLI recall', async () => withEnv(env, async () => {+ const seen: unknown[] = [];+ for (const engine of engines) {+ const s = seeds.get(engine)!;+ for (const remote of [false, true]) {+ const points = await engine.findTrajectory({ entitySlug: ENTITY, sourceId: 'default', remote, kind: 'all', limit: 100 });+ seen.push({ engine: engine.kind, remote, read: 'trajectory', ...shape(s, points.map(p => p.fact_id)) });+ const card = await buildEntityCard(engine, 'default', ENTITY, { remote });+ seen.push({ engine: engine.kind, remote, read: 'card', count: card.found ? card.card.active_fact_count : null });+ }+ let out = '';+ const write = process.stdout.write.bind(process.stdout);+ process.stdout.write = ((chunk: string) => { out += chunk; return true; }) as typeof process.stdout.write;+ try { await runRecall(engine, [ENTITY, '--json', '--source', 'default']); } finally { process.stdout.write = write; }+ const rows = JSON.parse(out) as { facts?: Array<{ id: number }> } | Array<{ id: number }>;+ seen.push({ engine: engine.kind, remote: false, read: 'cli_recall', ...shape(s, (Array.isArray(rows) ? rows : rows.facts ?? []).map(f => Number(f.id))) });+ }+ expect(seen).toEqual(engines.flatMap(engine => [+ ...[false, true].flatMap(remote => [+ { engine: engine.kind, remote, read: 'trajectory', ...visibleOnly(seeds.get(engine)!) },+ { engine: engine.kind, remote, read: 'card', count: 1 },+ ]),+ { engine: engine.kind, remote: false, read: 'cli_recall', ...visibleOnly(seeds.get(engine)!) },+ ]));+ }), 120_000);++ test('a pinned question gathers no hidden fact, take or timeline row as evidence', async () => withEnv(env, async () => {+ const seen: unknown[] = [];+ for (const engine of engines) {+ const s = seeds.get(engine)!;+ const pin = { source_id: 'default', slug: 'questions/alice-example', question: 'What does Alice Example do in the ledger guild?', scope_entity: ENTITY, scope_slug_prefix: null } as unknown as PinRow;+ const items = await retrieveEvidence(engine, pin);+ const of = (kind: string) => items.filter(i => i.kind === kind).map(i => Number(i.item_id)).sort((a, b) => a - b);+ seen.push({ engine: engine.kind, facts: shape(s, of('fact')), takes: of('take'), timeline: of('timeline') });+ }+ expect(seen).toEqual(engines.map(engine => {+ const s = seeds.get(engine)!;+ return { engine: engine.kind, facts: visibleOnly(s), takes: [s.take], timeline: [s.timeline] };+ }));+ }), 120_000);+});++describe('consolidate never promotes a hidden fact into a take', () => {+ test('quarantined-page and needs_rederive facts stay unconsolidated (PGLite)', async () => withEnv(env, async () => {+ configureGateway({ embedding_model: 'openai:text-embedding-3-large', embedding_dimensions: 1536, env: { ...process.env } });+ const engine = new PGLiteEngine();+ try {+ await engine.connect({}); await engine.initSchema();+ await engine.executeRaw(`INSERT INTO pages (slug, type, title) VALUES ($1, 'person', 'Alice Example'), ('notes/held-note', 'note', 'Held')`, [ENTITY]);+ await engine.executeRaw(`UPDATE pages SET frontmatter = jsonb_build_object($1::text, '{"reason":"junk_pattern","detail":"test","assessed_at":"2026-10-07T00:00:00Z"}'::jsonb) WHERE slug = 'notes/held-note'`, [QUARANTINE_KEY]);+ const vec = '[' + Array.from({ length: 1536 }, (_, i) => (i === 0 ? 1 : 0)).join(',') + ']';+ const ids: number[] = [];+ for (const [word, sourcePage] of [['alpha', null], ['beta', null], ['gamma', null], ['delta', 'notes/held-note'], ['epsilon', null]] as const) {+ const [row] = await engine.executeRaw<{ id: number }>(+ `INSERT INTO facts (source_id, entity_slug, fact, kind, source, valid_from, confidence, embedding, embedded_at, embedding_model, embedded_text_hash, source_markdown_slug)+ VALUES ('default', $1, $2, 'fact', 'test', now() - interval '30 hours', 0.9, $3::vector, now() - interval '30 hours', 'openai:text-embedding-3-large', md5($2), $4) RETURNING id`,+ [ENTITY, `alice fact ${word}`, vec, sourcePage]);+ ids.push(Number(row!.id));+ }+ await markRederive(engine, 'facts', ids[4]!);+ await runPhaseConsolidate(engine, {});+ const rows = await engine.executeRaw<{ id: number; consolidated_at: Date | null }>('SELECT id, consolidated_at FROM facts ORDER BY id');+ expect(rows.map(r => ({ id: Number(r.id), consolidated: r.consolidated_at !== null })))+ .toEqual(ids.map((id, i) => ({ id, consolidated: i < 3 })));+ } finally {+ await engine.disconnect();+ resetGateway();+ }+ }), 120_000);+});