Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Saved facts show when they were observed, repeated facts collapse to one row, and replaced facts end on one rule (GST-2859) (#9524)

Garry Tan18ab99e870e5
diff --git a/behavior-changes.d/fact_rows_observed_date_gbra574.md b/behavior-changes.d/fact_rows_observed_date_gbra574.md
new file mode 100644
index 0000000000..ee2ab3494f
--- /dev/null
+++ b/behavior-changes.d/fact_rows_observed_date_gbra574.md
@@ -0,0 +1,4 @@
+---
+kind: fix
+---
+Saved facts in query results now say when they were said. The facts block (on by default for a query with a token budget), the temporal fact reserve, the facts arm and recall's date headers fill the observed date in their [observed ...; valid ...] header from the page the fact was taken from, read under the caller's own page access, so a page the caller may not read dates nothing. If that read fails the header shows observed unknown and the response reports the degraded stage fact_observed_date_unavailable (recall: facts_degraded reason fact_observed_date_unavailable). Copies of one claim take one row in the facts block, dated as first stated and followed by (stated N times, last DATE); facts that differ by even one word keep their own rows. Question ranking of saved facts now matches words on their stems, with a word matched only through its stem ranking a little below an exact match (a question naming Roger puts a Roger fact above a Rogers fact), and dates, years and auxiliary verbs such as had or been no longer count toward the match, so a question phrased as of a date admits the facts it names and a fact sharing only a year with the question is no longer admitted. A fact about another person who shares the named person's surname can still be admitted when it also shares a verb with the question; its row names that person.
diff --git a/behavior-changes.d/superseded_fact_valid_until_gbra574.md b/behavior-changes.d/superseded_fact_valid_until_gbra574.md
new file mode 100644
index 0000000000..3e0f5a5121
--- /dev/null
+++ b/behavior-changes.d/superseded_fact_valid_until_gbra574.md
@@ -0,0 +1,4 @@
+---
+kind: fix
+---
+When a newer fact replaces an older one, every writer now ends the older fact's validity by one rule. It ends on the newer fact's valid_from when that date falls after the older fact began and before the older fact's own end (or now, if it had none); otherwise it ends at its own earlier end or now, and never before its own valid_from (a fact dated in the future ends where it begins). Per writer: remember with replaces (managed or not) and the engine supersede path used to leave the older fact open-ended when the newer fact was backdated, and now end it at the time of the write; the unmanaged remember that strikes the row in a page's facts table used to end it on today's date and now ends it at the newer fact's valid_from, which for remember is the time of the write; an accepted gbrain decide proposal used to end it at the time of acceptance and now ends it on the newer fact's valid_from when that falls inside the older fact's window; expireFact takes an optional successor date and applies the same rule. Extraction already used the rule and only gains the backdated case. A backdated replacement is one dated on or before the day the older fact began. For one of those, the older fact's end date is the day gbrain learned of the replacement (the time of the write), not a date taken from the source, unless the older fact already had an earlier end, which it keeps. Only future writes change. No stored row is rewritten, and forgetting or withdrawing a fact works exactly as before.
diff --git a/changelog.d/gbra574-dated-facts-wave-a.md b/changelog.d/gbra574-dated-facts-wave-a.md
new file mode 100644
index 0000000000..acbe5d469e
--- /dev/null
+++ b/changelog.d/gbra574-dated-facts-wave-a.md
@@ -0,0 +1,10 @@
+**Saved facts in query results now say when they were said, collapse their copies, match on word stems, and end at the right date when replaced.**
+
+<!-- behavior-change: fact_rows_observed_date_gbra574 -->
+<!-- behavior-change: superseded_fact_valid_until_gbra574 -->
+
+### Fixed
+- The facts block, temporal fact reserve, facts arm and recall date headers print the observed date of the page each fact came from, read under the caller's page access; a failed read reports `fact_observed_date_unavailable` instead of a silent `observed unknown`.
+- Copies of one claim take one row in the facts block (dated as first stated, `(stated N times, last DATE)`), so restated facts stop crowding out older values; facts that differ by one word keep their rows.
+- Question ranking of saved facts matches on stems under the brain's full-text config (an exact word still ranks above a stem-only match) and ignores dates, years and auxiliary verbs, so an as-of question admits the facts it names and a fact sharing only a year with the question is not admitted.
+- Every supersession writer ends the replaced fact by one rule. It ends on the newer fact's `valid_from` when it falls inside the older fact's window, else at the older fact's own earlier end or now, and never before the older fact began. New writes only.
diff --git a/docs/architecture/key-files/core-search-2-continued.md b/docs/architecture/key-files/core-search-2-continued.md
new file mode 100644
index 0000000000..6b979d8089
--- /dev/null
+++ b/docs/architecture/key-files/core-search-2-continued.md
@@ -0,0 +1,7 @@
+# Key files: Core Search (2/2, continued)
+
+[Subsystem index](../KEY_FILES.md) · [Previous entries](core-search-2.md). Read only the entries relevant to your change.
+Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
+
+- `src/core/search/score-contract.ts` — `scoresFollowRank`: at the search/query op boundary the returned `score` follows the returned order (the rerank score, times any rank prior, in a reranked list; a running minimum otherwise) and the replaced value stays as `fused_score` (#7156).
+- `scripts/bench/ranking-shapes/` + `test/fixtures/ranking-shapes/` — synthetic two-source ranking bench (208 pages, 70 questions): `run.ts` builds a PGLite brain, runs search/query/query --expand through a checkout's CLI behind a caching provider proxy and scores hit@k, MRR and per-shape checks (twins, stubs, families, priors, score order, chunk size).
diff --git a/docs/architecture/key-files/core-search-2.md b/docs/architecture/key-files/core-search-2.md
index d8152ab66f..0d2d02469c 100644
--- a/docs/architecture/key-files/core-search-2.md
+++ b/docs/architecture/key-files/core-search-2.md
@@ -17,7 +17,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- Evidence date headers (`search/evidence-date.ts`) and the tail trim (`search/evidence-tail-trim.ts`): [Core Search (evidence blocks)](core-search-evidence.md).
- `src/core/search/entity-anchor.ts` — entity-anchored retrieval: `entityAnchoredPages` (the entity page plus up to 10 pages that link to it by a ranking link type or name it as a word, newest by `COALESCE(effective_date, updated_at)`; question/synthesis/pinned-question pages excluded) and `prefixAnchoredPages`, shared by pinned-question refresh (`questions/refresh.ts` `anchoredSlugs`) and the `query`/`search` ops; `asksForCurrentState` (deterministic cue regex), `namedEntity` (exactly one `isFactEntityPage` page whose title `isTitleMentionedInQuery` matches, the title match before the read-policy filter; a title inside a longer matched title is the same mention); `applyEntityAnchoring` (anchored rows first, at most half the row count, organic rows of the same page moved up, missing pages read through `getChunkWindows` under the caller's scope, rows stamped `entity_anchored`, token budget re-applied, fail-open); `entityAnchoringEnabled` (`search.entity_anchoring`, default off). Tests: `test/search-entity-anchor.test.ts`. Gates: `docs/eval/decisions/entity-anchoring-query/`.
- The facts arm, temporal fact reserve, facts block and fact source pages (`search/facts-arm.ts`): [Core Search (evidence blocks)](core-search-evidence.md).
-- `src/core/search/fact-relevance.ts` — question relevance for saved facts, shared by recall's `question` (`facts/question-recall.ts`), the temporal fact reserve and (pool only) the facts arm. `collectFactCandidates`: one bounded pool under recall's read policy (source scope, active and not superseded, audit rows excluded, quarantine and rederive hiding through `projectionEligibleSql`, the trust floor, world-only facts when the scope says `worldOnly` (`worldOnlyFacts`), `privateProvenanceFilterFragment` when `excludePrivate`) plus recall's filters inside every arm before its LIMIT (`(source_id, entity_slug)` pairs, `source_session`, event-time since, escaped `ILIKE` grep); arms, run concurrently and merged in a fixed order, are keyword FTS (`factsFtsDocument`, served by `idx_facts_fts`), cosine (cast to the facts column type, filtered on model only so the HNSW index serves it, a stale hash or other dimensions yielding no similarity; `ef_search` sized to the depth and pgvector's iterative scan when `hnswIterativeScan` is set) and the named entity page's facts in its own source; depths are bound parameters; `cosineForAll` scores every candidate's cosine whichever arm found it; `distinctiveTerms` (recall's question, GST-1304) searches a question over `MAX_QUERY_TERMS` (12) content terms on its last 24 and returns in `terms` the 12 of those the fewest candidates hold (`heldTerms`; later word first on a tie), which `scoreFactCandidates` takes as its third argument, so a provider preamble no fact holds stops filling the share's denominator. A missing embedding column or other dimensions skips the cosine arm; any other DB error throws. `scoreFactCandidates`: cosine + whole-word term share + 0.1 when dated, the share's denominator growing past `TERM_STUFFING_RATIO` times the question's terms for a long fact; order score, newest valid_from, id. `admitted`: the `reserve` rule (cosine >= 0.5 or share >= 0.34) or the `facts_arm` rule (cosine >= 0.6, half the terms, or the named entity). `countUncomparableFacts` counts the pool's facts the cosine arm cannot compare, per source: candidates from `idx_facts_unembedded` and `idx_facts_embedding_model` first, the read policy on those rows only. The keyword arm and the count run under `customPlan` (`plan_cache_mode = force_custom_plan` in a transaction): a cached generic plan cannot see the source arrays and nests the policy anti-joins over every match. p95 at 100k facts: Postgres 51 ms local / 65 ms remote, PGLite 152 / 139 ms. Tests: `test/fact-relevance.test.ts`, `test/recall-question.test.ts`.
+- `src/core/search/fact-relevance.ts` — question relevance for saved facts, shared by recall's `question` (`facts/question-recall.ts`), the temporal fact reserve and (pool only) the facts arm. `collectFactCandidates`: one bounded pool under recall's read policy (source scope, active and not superseded, audit rows excluded, quarantine and rederive hiding through `projectionEligibleSql`, the trust floor, world-only facts when the scope says `worldOnly` (`worldOnlyFacts`), `privateProvenanceFilterFragment` when `excludePrivate`) plus recall's filters inside every arm before its LIMIT (`(source_id, entity_slug)` pairs, `source_session`, event-time since, escaped `ILIKE` grep); arms, run concurrently and merged in a fixed order, are keyword FTS (`factsFtsDocument`, served by `idx_facts_fts`), cosine (cast to the facts column type, filtered on model only so the HNSW index serves it, a stale hash or other dimensions yielding no similarity; `ef_search` sized to the depth and pgvector's iterative scan when `hnswIterativeScan` is set) and the named entity page's facts in its own source; depths are bound parameters; `cosineForAll` scores every candidate's cosine whichever arm found it; `distinctiveTerms` (recall's question, GST-1304) searches a question over `MAX_QUERY_TERMS` (12) content terms on its last 24 and returns in `terms` the 12 of those the fewest candidates hold (`heldTerms`; later word first on a tie), which `scoreFactCandidates` takes as its third argument, so a provider preamble no fact holds stops filling the share's denominator. A missing embedding column or other dimensions skips the cosine arm; any other DB error throws. `scoreFactCandidates`: cosine + term share + 0.1 when dated (terms match on their stems under the brain's full-text config, one `to_tsvector` read per pool, whole words when it fails; a stem-only match adds 0.9 of an exact one to the score but counts fully in the share; ISO dates, years and auxiliaries count on neither side), the share's denominator growing past `TERM_STUFFING_RATIO` times the question's terms for a long fact; order score, newest valid_from, id. `admitted`: the `reserve` rule (cosine >= 0.5 or share >= 0.34) or the `facts_arm` rule (cosine >= 0.6, half the terms, or the named entity). `countUncomparableFacts` counts the pool's facts the cosine arm cannot compare, per source: candidates from `idx_facts_unembedded` and `idx_facts_embedding_model` first, the read policy on those rows only. The keyword arm and the count run under `customPlan` (`plan_cache_mode = force_custom_plan` in a transaction): a cached generic plan cannot see the source arrays and nests the policy anti-joins over every match. p95 at 100k facts: Postgres 51 ms local / 65 ms remote, PGLite 152 / 139 ms. Tests: `test/fact-relevance.test.ts`, `test/recall-question.test.ts`.
- `src/core/search/query-arms.ts` — `applyQueryArms`: the `query` op's post-retrieval arms in order (entity anchoring, fact source pages, then the facts arm), each behind its own key (the facts arm default on, the rest off); a query whose caller passed `token_budget` takes the facts block (`search.query_facts_block`, default on; at most `limit` rows, scored at the top page's score; #8411) or, with a temporal cue, the temporal fact reserve (`search.temporal_fact_reserve`) instead of the facts arm; with all off it returns the rows untouched.
- `src/core/search/image-vector-presence.ts` — `brainHasImageVectors` (one `EXISTS embedding_image IS NOT NULL` probe per engine, cached 60 s, fail-open to true). `hybrid/arms.ts` uses it so an inferred image/both modality on a brain with no image embeddings stays text: no multimodal embed, keyword arm kept, no `vector_arm_failed`; the opt-in LLM modality tie-break still runs, and an image or both answer is demoted to text. An explicit `crossModal` is honored. Tests: `test/search-image-arm-text-only.test.ts`, `test/cross-modal-hybrid-integration.serial.test.ts`.
@@ -76,5 +76,5 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL
- `src/core/search/relational-rerank-pin.ts` — relational-arm rows bypass reranker DEMOTION (ranker wave, receipt R1 in `scripts/r1-namedthing-rerank-ab.ts`: on NamedThingBench's 39 graph-relationship questions the shipped `balanced` default lost hit@1 21/39 → 3/39 and hit@3 27/39 → 5/39 with the reranker on — 19 and 22 paired losses — because the cross-encoder scores chunk TEXT and an edge-derived answer's text need not mention the query's entity; with the pin at its default 3, measured with `--autocut on`, the shape that shipped before rule R2 turned autocut off, the same paired run shows 0 hit@1 / 0 hit@3 losses, 21/39 and 27/39, and the 11 non-relational core queries show 0 losses — one hard-negative gain, hit@1 10/11 → 11/11). `pinRelationalRows(reranked, relationalList, {max, fusedOrder?, onPin?})` is a pure PERMUTATION of the post-rerank pool: relational rows (page key `(source_id, slug)` in the arm's list; ONE row per page — its first, highest-ranked occurrence) claim `min(fused_rank, reranked_rank)` (fused rank = position among relational rows in `fusedOrder`, the pre-rerank `deduped` pool; arm order for rows absent from it), claims sort ascending with ties resolved to the FUSED order then the reranked position, the first `max` claimants form the top block in that order, every other row follows in reranked order (an unpinned row lands at its reranked position or up to `pinned.length` lower, never higher). Pinned rows are shallow copies stamped `relational_pinned: true`; no row is added or removed (re-injecting rows dropped upstream stays `ensureRelationalEvidenceSlot`'s job). Every no-op path (`max <= 0`, empty pool, empty arm, no relational page in the pool) returns the input array itself. `normalizeRelationalRerankPin` is the ONE range contract for the knob (non-negative integer `<= 10` or a string parsing to one → that integer; the literals `off` / `false` (any case) or boolean `false` → 0; anything else → `undefined` = fall through), shared by `mode.ts:loadOverridesFromConfig` and both per-call seams in `hybrid.ts`; `DEFAULT_RELATIONAL_RERANK_PIN` (3) and `RELATIONAL_RERANK_PIN_MAX` (10) are exported. `RelationalRerankPinDecision` (`{max, relational_in_pool, pinned:[{slug, source_id, from_rank, to_rank, fused_rank}], moved}`) is surfaced as `HybridSearchMeta.relational_rerank_pin` for `--explain`. Wired in `hybridSearch` immediately after `applyReranker` and before the alias hop, gated on the reranker having actually reordered (`reranked !== deduped` — `applyReranker` returns its input on every fail-open / skip / pass-through path, and the fused order already carries the arm) and on non-image modality; autocut's `scoreOf` ignores pinned rows and its preserve predicate keeps them, so text-row autocut is byte-identical to the pre-pin behavior. Known cost: a false-positive arm (relational shape parsed AND a real seed resolved, but wrong/stale edges) now puts up to `max` edge pages at ranks 1..max instead of one at `limit`; mitigations are the arm's `fallback_slugify` confidence gate, the tier-2 resolution-margin TODO, config `search.relational_rerank_pin off`, and per-call `SearchOpts.relationalRerankPin`. Pinned by `test/search/relational-rerank-pin.test.ts` (pure contract + tie policy) and `test/search/relational-rerank-pin-hybrid.serial.test.ts` (hermetic PGLite on the relational corpus through the REAL gateway rerank path behind `__setRerankTransportForTests` with an inverted-relevance stub: pin 0 reproduces the regression, pin 3 restores hit@1/hit@3 on all 8 who-invested questions, non-relational queries and reranker-off runs are byte-identical).
- `src/core/search/collapse.ts` — `collapseResults`: folds copies of one result into the best-ranked row before reranking and again after the identity tiers (`rank.ts` `collapsePool`, both keyword-only paths): same slug in another source (`twin`), a redirect stub whose readable target it names (`redirect`; a target missing from the pool is read under the alias-hop read policy, an unresolved stub stays stamped `redirect_to`), and copies of one document (`family`: identical bodies, or a derivative or leaf-extension slug in the same source). The kept row lists them in `collapsed`; `meta.collapse` counts them. Knob `search.collapse` (#4612, #7153, #7154).
- `src/core/search/rank-priors.ts` — `applyRankPriors`: multiplicative priors after rerank and pinning (person rows on an organization question, low-coverage image rows stamped `image_page`, undated rows on a when-question, redirect stubs, and archive rows (GST-2768: ARCHIVE_PRIOR under a close non-archive row, ARCHIVE_HITS_PER_DAY per day, off for a conversation or date query)); each changed row carries `rank_prior` and `rank_score`. Knob `search.rank_priors`.
-- `src/core/search/score-contract.ts` — `scoresFollowRank`: at the search/query op boundary the returned `score` follows the returned order (the rerank score, times any rank prior, in a reranked list; a running minimum otherwise) and the replaced value stays as `fused_score` (#7156).
-- `scripts/bench/ranking-shapes/` + `test/fixtures/ranking-shapes/` — synthetic two-source ranking bench (208 pages, 70 questions): `run.ts` builds a PGLite brain, runs search/query/query --expand through a checkout's CLI behind a caching provider proxy and scores hit@k, MRR and per-shape checks (twins, stubs, families, priors, score order, chunk size).
+
+[More entries](core-search-2-continued.md)
diff --git a/docs/architecture/key-files/core-search-evidence.md b/docs/architecture/key-files/core-search-evidence.md
index a1002806fe..25a90f7416 100644
--- a/docs/architecture/key-files/core-search-evidence.md
+++ b/docs/architecture/key-files/core-search-evidence.md
@@ -3,8 +3,8 @@
[Subsystem index](../KEY_FILES.md) · [Core Search (2/2)](core-search-2.md). Read only the entries relevant to your change.
Current behavior and load-bearing invariants; history belongs in Git and CHANGELOG.
-- `src/core/search/evidence-date.ts` — C1 dated evidence blocks: the fixed header grammar (`[observed YYYY-MM-DD|unknown]` for page blocks; `[observed unknown; valid FROM to UNTIL]` for `recall` facts), `pageObservationDate` (effective-date precedence with `event_date` removed and no row-timestamp fallback; mirrors #6020's `resolveObservationDate`, the seam to swap once it lands), `formatBrainDay` from `effective-date.ts` (midnight UTC renders as written, other instants in `brain.timezone`; the one function think's `<page date>` also renders through, while the header's `observed` stays when the text was said, never the content date), `loadPageDateHeaders` (one `pages` read per delivery; only the derived day leaves it) `evidenceDateHeaderEnabled` (`search.evidence_date_header`, default off) and `evidenceTitlesEnabled` (`search.evidence_titles`, GBRA-205, default off: with the header on, `blockDateHeaders` ends a page block's line with its title via `titledPageHeader`, fact headers unchanged; `evidenceTitle` removes bidi overrides, isolates and marks and blanks angle brackets and C0/C1 controls in both sinks; think's `<page>` tag gets a `title` attribute). Tests: `test/evidence-date-header.test.ts`, `test/search-8k-conversion.test.ts`.
+- `src/core/search/evidence-date.ts` — C1 dated evidence blocks: the fixed header grammar (`[observed YYYY-MM-DD|unknown]` for page blocks; `[observed YYYY-MM-DD|unknown; valid FROM to UNTIL]` for facts, `observed` from the source page (`source_markdown_slug`) via `factObservationDates` under the caller's page read policy; a failed read renders `unknown` and the caller reports `fact_observed_date_unavailable`), `pageObservationDate` (effective-date precedence with `event_date` removed and no row-timestamp fallback; mirrors #6020's `resolveObservationDate`, the seam to swap once it lands), `formatBrainDay` from `effective-date.ts` (midnight UTC renders as written, other instants in `brain.timezone`; the one function think's `<page date>` also renders through, while the header's `observed` stays when the text was said, never the content date), `loadPageDateHeaders` (one `pages` read per delivery; only the derived day leaves it) `evidenceDateHeaderEnabled` (`search.evidence_date_header`, default off) and `evidenceTitlesEnabled` (`search.evidence_titles`, GBRA-205, default off: with the header on, `blockDateHeaders` ends a page block's line with its title via `titledPageHeader`, fact headers unchanged; `evidenceTitle` removes bidi overrides, isolates and marks and blanks angle brackets and C0/C1 controls in both sinks; think's `<page>` tag gets a `title` attribute). Tests: `test/evidence-date-header.test.ts`, `test/search-8k-conversion.test.ts`.
- `src/core/search/evidence-tail-trim.ts` — the protected tail trim (`search.evidence_tail_trim`, default off, GBRA-156). `resolveEvidencePlan` sets `EvidencePlan.tailTrim` only for `unit: auto` with the key on; `deliverEvidence` calls `trimEvidenceTail(results, opts.query)` on the rank-order delivered rows after redaction and before the token totals, and only when `DeliverOptions.query` is set (`search`/`query` pass `p.query`; the library `assembleEvidenceForHits` passes `input.query` and frozen hits may carry `score`; the `assemble_evidence` op passes neither, so the trim never runs there). Rows past `TAIL_TRIM_PROTECTED` = 3 that are whole-conversation blocks (`delivered.reason` `conversation_type`/`conversation_slug`) and score under `TAIL_TRIM_SCORE_FRACTION` = 0.5 of rank one are dropped, unless their `chunk_text` holds at least `TAIL_TRIM_TERM_FRACTION` = 0.8 of rank one's share of the query's terms (`queryTerms`: letter/digit runs, stopwords and single characters dropped, cut to six characters). Rank one's score must be a positive finite number and every row's finite, or nothing changes. `delivery.tail_trim` is `{ dropped, kept_by_query_terms }`; `dropped_reasons.tail_trim` counts drops. Pinned by `test/evidence-tail-trim.test.ts` (the knowledge-update regression case).
- `src/core/search/transcript-quote.ts` — quotes third-party text on every read that hands it to a model (#7600, #8042). The signal is stored metadata, in one place: `isQuotedPageType` (every concrete type in `facts/conversation-types.ts` `ALLOWED_TYPE_ALIASES` plus `transcript` and `chat`) for page rows, and for fact rows `isQuotedFactSource` (`facts.source` starting `cli:extract-conversation-facts`, the prefix in `facts/audit-sources.ts`) or `factsFromQuotedPages` (one bounded read, same source: the fact's `source_markdown_slug` page, a `derivation_inputs` page edge the backstop and extract_facts already write, or for a `sync:import` / `mcp:put_page` fact the page its `source_session` names (`sync:<slug>`, `put_page:<slug>`, covering facts saved before edges or rebuilt from git) has a quoted type; it writes nothing, and a lookup failure withholds `saved_facts` (stage `saved_facts_withheld`) and remote recall facts (`facts_degraded.reason`) and fails the delta facts arm; the facts arm carries both on `fact_row.source`/`from_quoted_page`, stripped by `factRowOutput`; marker `fact <id>`); nothing reads the text to decide, so other rows come back byte-for-byte unchanged. `quoteLabel` is the row decision. GST-687: for an untrusted (remote, `ctx.remote !== false`) reader the signal also includes the stored trust tier: `quotesTier` (below `OWNER_TIER_FLOOR`, `operator_curated`; a missing tier reads as `unknown`, so it fails closed) and `quotesStored` (page type or tier) quote agent-written, tool-data, external and unverified-origin text. `quoteLabel(row, untrusted)`, `deliveredQuoteText`, `fenceTranscriptRows` and `quoteFacts(engine, facts, untrusted)` take that flag (`quoteFacts` reads each fact's stored tier from `facts`, never from the caller's row); only third-party types get tool calls summarized. `quotePageText` and `pageTypesById` serve reads whose rows name only a page (`get_timeline`, `get_chunks`). Local reads are unchanged. Beyond search: `recall` (remote callers: neutralize + fence its results before packing, `quoteFacts` its facts), think's pages block (`think/gather.ts renderPagesBlockDetailed`: neutralize before the excerpt, fence what it renders, every run), `saved_facts` in search/query meta, remote `delta` facts and `_meta.brain_hot_memory` (`quoteFacts`, which also quotes a representative when a collapsed member qualifies), remote `get_page known_as_of` and `get_versions` bodies and diffs (`ops/known-as-of.ts intervalView`, `ops/admin.ts`), and remote `get_page`/`fetch` (`ops/get-page-projection.ts projectGetPage` `quote`, which `projectFetchText` reuses: compiled_truth, timeline and `content` fenced without the tool-call summary, so text inside stays byte-exact; windows fenced in `get-page-body.ts envelopeText`; `quoted: "transcript"`; `quotesBody(ctx, page, trust)` also quotes a page below the owner's tier, and `neutralizesBody` summarizes tool calls only on a third-party page's reading shapes, never for the thin client, with `include_content` and `view: markdown` keeping the stored bytes inside the fence). Remote `get_timeline`, `get_chunks`, `search_exact`, think's extractive fallback, `context_pack` and `list_write_requests`/`get_write_request` outcome text quote by tier too, and so (#8377) do the `chronicle_*` reads and `volunteer_chronicle`'s timeline (`ops/chronicle.ts remoteTimeline`, each row by its own page; `quoteTimelineRows` fences a row's free-text `source` with summary and detail, #8384, and an event row's `kind` is fenced by its event page's type and tier), its ontology values, `ontology_get`'s and `ontology_conflicts`' (`remoteOntology`, `quoteFacts` by `fact_id`; remote `ontology_propose` drops a fact fence), `takes_list`/`takes_search` (`ops/takes.ts quoteTakes`), a quoted page's title, frontmatter strings and tags (`quotedPageMeta`, also `get_tags`; remote `add_tag`/`remove_tag` drop a fence naming the page; `include_content` and fetch `text` fence the whole serialized markdown), a quoted search/query/assemble_evidence/recall row's title (`deliveredQuoteTitle`, which `capEvidenceToBudget` counts), a `context_pack` card below the owner's tier rendering its title under the card's trust label, the per-turn push hook's hot facts and the pushed context pack (`untrusted: true`); `context_pack` counts entity facts a failed quote lookup withholds in `suppressed`. Remote `remember` drops a fact fence (`unquoteFactText`) and `add_timeline_entry` a fence naming its page, so a copied-back entry keeps its replay key. Remote `put_page` (`ctx.remote !== false`) runs `unquoteTranscriptText` over `content` (`persistence/page-mutations.ts unquotedPutContent`, after the replay check so the request digest covers what the caller sent; slug lowercased as admission does) when the stored page or the incoming frontmatter has a quoted type or the stored page is below the owner's tier (an owner-tier page keeps a fence naming it only when its stored body already holds one), removing only a marker naming that slug and its own fence (lines may end in CRLF), so get → edit → put writes the body and a note's fence-like text is kept (`unclosed`: a fence opening the content, after blank lines at most, whose closing line the edit replaced is removed too, #8377, #8384; `closingLine` takes the first line that is exactly the read's fence outside a code block the text opens itself, so an agent's added block stays whole; every search of one text shares a budget of eight times its length, past which the rest is kept as written, so many unclosed markers stay linear). Remote `put_page` content holding a tool-call summary that a whole read of the stored conversation page never produces, or fewer of a call's summaries than the page holds (`holdsSummarizedToolCalls`), refuses `invalid_params` and stores nothing; only summaries outside the text's own tool-call blocks count, and a named summary the page never wrote is admitted while every stored call stays (#8377). A thin client paging a quoted page (`cli/page-render.ts spoolWholePage`) unquotes each window before checking size and sha256, then prints the page (and `--json` content) inside one quote fence (`transcriptFence`). Trusted local get_page/fetch/put_page, recall and delta are unchanged. `searchOutput` (`ops/search.ts`) runs `neutralizeTranscriptRows` before the snippet and evidence caps and `fenceTranscriptRows` after them, on copies, after ranking and evidence delivery; `assembleEvidenceForHits` (`evidence-delivery.ts`, so the `assemble_evidence` op and gbrain-evals) does the same around `capEvidenceToBudget`. That cap counts a conversation row as delivered, inside its fence (`deliveredQuoteText`, its `quote` flag; deliverEvidence's own cap, which recall uses, counts the text as returned), so an explicit `auto` token budget still holds, `snippet_chars` included (a rank-one row is kept even when its fence alone exceeds a very small budget); a row it cuts is shortened until the fenced text fits, and other rows are cut exactly as before. The neutralizer replaces each tool-call block with `[tool call: <name>, arguments omitted]` (`[tool calls: a, b, …]` for several): an object typed `tool_use`, `server_tool_use`, `mcp_tool_use`, `function_call`, `custom_tool_call`, `tool_call` or `toolCall` (the enclosing object, found by walking back over candidate `{` with a string-aware scan), the value of a `tool_calls` array or a `function_call` or Gemini `functionCall` object (the message object and its `content` or `parts` stay), and an XML-style `function_calls`, `invoke`, `tool_use` or `tool_call` element with its `parameter` children, namespace prefix optional (the name from `invoke name="…"`, a `<name>` child, or the root `name` of the JSON object a `tool_call` opens with; result elements are never matched, and a tag counts as markup only when it ends its line (LF or CRLF) or the text, runs into the next tag, is an `invoke` with a name, or is a `tool_call` followed by `{`; blocks separated only by whitespace merge). The XML pass runs before the JSON pass. Text between evidence omission lines is read separately; a block cut by the end of the text is replaced to the end or the next message anchor line (an open-ended block never swallows a later message), one whose `{` lies before the text from the text start (or the last message anchor) when the type is followed by `,` or `}`, and prose that only names the shape is left alone. An XML block whose close lies past the text runs to the end or the next message anchor line; a closing tag with no opener (or a `parameter` opener outside a call) starts at the text start or the last message anchor. A trailing evidence cut marker and a leading `[observed …]` date header stay outside any block. Message anchor lines may end in CRLF. Scanning is linear: anchors are indexed once, and past a per-text work budget the rest of a message is summarized (fails closed). Names outside `[A-Za-z0-9_.:/-]{1,64}` are dropped. The fence is a marker line naming the slug (`[`, `]`, `;` and line breaks shown as `?`), then a code fence of at least three backticks, one longer than the longest run in the text; the row gets `quoted: "transcript"` (kept in lean rows). `rowPreview` is the CLI human line: `[quoted transcript]` and the start of the fenced text. Stored pages, chunks, ranking and scores are unchanged; delivered `match_spans` are remapped to the returned text, and `tokens`, `tokens_delivered` and `budget_used` count the fenced text (`fenceTranscriptRows` with a recount). Tests: `test/search/transcript-quote.test.ts`, `test/search-transcript-quote.test.ts` (ChatGPT export through the importer, with a JSON and an XML call; remote MCP `search`/`query`/`assemble_evidence` and CLI human/`--json`, both engines), `test/search-transcript-quote-budget.test.ts` (budget, spans and snippet cap through the ops), `test/search-transcript-quote-coverage.test.ts` (#8042 matrix: slack, imessage, email, meeting and conversation pages plus a conversation-extracted fact against a note and a remembered fact, plus backstop facts linked to a synced slack page by a derivation edge or only by their sync session, and one derived from a note, a failed lookup withholding facts, a retried retype put, over remote search, query, saved_facts, assemble_evidence, recall, get_page body/content/window, local get_page, the get→put round trip (a note's fence-like text, a mixed-case slug), thin-client paging of a 12,000-line slack page, fetch, think's prompt, delta, hot memory, and get_page known_as_of / get_versions bodies and diff, both engines), `test/search-transcript-quote-tiers.test.ts` (GST-687: every placement and read surface by tier, both engines) and `test/search-transcript-quote-legacy-claim.test.ts` (a legacy note is quoted, raw again after claim and backfill, both engines), `test/search-transcript-quote-followups.test.ts` (#8377: the chronicle, takes and page title/frontmatter reads, hook-injected facts, and the copied-back writes, red on master, both engines) and `test/trust-keep-maintenance-reimport-8352.test.ts` (#8352: maintenance re-imports keep an owner note's tier, both engines) and `test/search-transcript-quote-8384.test.ts` (#8384: whole-markdown round-trip edges, timeline source, ontology values, the `quoted_unverified_origin` notice from `get-page-projection.ts unverifiedOriginNotice`, both engines) and `test/search-transcript-quote-8384b.test.ts` (#8384 wave 3: CRLF whole-markdown, chronicle `kind`, search row and card titles, tags, a fenced value next to a naive datetime, both engines).
-- `src/core/search/facts-arm.ts` — the `query` facts arm behind `search.query_facts_arm` (default on since gates 1b and 2 passed; `false` turns it off). `matchQueryFacts` reads the candidate pool from `fact-relevance.ts` (active facts under recall's read policy) and matches them by query-embedding cosine (`FACT_COSINE_MIN` 0.6, the embedding `hybridSearch` already computed, via `HybridSearchOpts.onQueryEmbedding`) or by query-term share (half the terms, substring match) or a named entity (in that page's source), newest `valid_from` first, at most `MAX_FACT_ROWS` (3). `applyFactsArm` adds them as `fact_row` rows in spare capacity only: free slots under the caller's row count, and whole rows that fit in what the page rows leave of the token budget; a page row is never displaced `factRowOutput` strips page fields (slug, id, type, chunk_id) from fact rows on output, leaving `result_type: 'fact'`, `fact_id`, `follow_up` (`recall` by entity, or `grep` without one) and `page_slug` when the caller can read the entity page (`readableEntityPages`); and stamps `superseded_claim` on page rows whose typed `claim_metric` (plus `claim_period` when both carry it) a newer active fact covers. No model call. Tests: `test/search-facts-arm.test.ts`. Gates and verdict: `docs/eval/decisions/query-facts-arm/`. Fact reserves: `applyFactReserve` serves two policies. The temporal fact reserve (`applyTemporalFactReserve`, `search.temporal_fact_reserve`, default off): `hasTemporalCue` (the preregistered word, phrase, ISO date and month-name regex), candidates from `collectFactCandidates` (50 cosine, 50 keyword, 20 entity facts, cosine for every candidate), scored by the shared `scoreFactCandidates` and kept by its `reserve` admission rule (fail-open on any error), packed into 15% of the token budget and 30% of the row count (1-20), rendered as fact rows whose text starts with `factDateHeader` (evidence delivery adds no second header), oldest first after the pages. Tests: `test/search-temporal-fact-reserve.test.ts`. Gates: `docs/eval/decisions/temporal-fact-reserve/`. The facts block (`applyFactsBlock`, `search.query_facts_block`, GBRA-205, default on since the held-out gate passed; `false` or a config read error turns it off): the same facts for every query whose caller passed `token_budget` (the no-budget evidence plan's implicit budget does not run it, #8411), up to 22% of the budget, 40 rows and the caller's `limit`, oldest first ahead of the pages at the top page's score (never above it), adding rows. Fact source pages (`applyFactSourcePages`, `search.fact_source_pages`, GBRA-205, default off): the conversations (`facts.source_markdown_slug`, never the entity page) of the best-matching admitted facts, at most 2 not already present, read through `getChunkWindows` under the caller's read scope and placed after the top 3 rows; the row count never grows. Tests: `test/search-8k-conversion.test.ts`.
+- `src/core/search/facts-arm.ts` — the `query` facts arm behind `search.query_facts_arm` (default on since gates 1b and 2 passed; `false` turns it off). `matchQueryFacts` reads the candidate pool from `fact-relevance.ts` (active facts under recall's read policy) and matches them by query-embedding cosine (`FACT_COSINE_MIN` 0.6, the embedding `hybridSearch` already computed, via `HybridSearchOpts.onQueryEmbedding`) or by query-term share (half the terms, substring match) or a named entity (in that page's source), newest `valid_from` first, at most `MAX_FACT_ROWS` (3). `applyFactsArm` adds them as `fact_row` rows in spare capacity only: free slots under the caller's row count, and whole rows that fit in what the page rows leave of the token budget; a page row is never displaced `factRowOutput` strips page fields (slug, id, type, chunk_id) from fact rows on output, leaving `result_type: 'fact'`, `fact_id`, `follow_up` (`recall` by entity, or `grep` without one) and `page_slug` when the caller can read the entity page (`readableEntityPages`); and stamps `superseded_claim` on page rows whose typed `claim_metric` (plus `claim_period` when both carry it) a newer active fact covers. No model call. Tests: `test/search-facts-arm.test.ts`. Gates and verdict: `docs/eval/decisions/query-facts-arm/`. Fact reserves: `applyFactReserve` serves two policies. The temporal fact reserve (`applyTemporalFactReserve`, `search.temporal_fact_reserve`, default off): `hasTemporalCue` (the preregistered word, phrase, ISO date and month-name regex), candidates from `collectFactCandidates` (50 cosine, 50 keyword, 20 entity facts, cosine for every candidate), scored by the shared `scoreFactCandidates` and kept by its `reserve` admission rule (fail-open on any error), packed into 15% of the token budget and 30% of the row count (1-20), rendered as fact rows whose text starts with `factDateHeader` (evidence delivery adds no second header), oldest first after the pages. Tests: `test/search-temporal-fact-reserve.test.ts`. Gates: `docs/eval/decisions/temporal-fact-reserve/`. The facts block (`applyFactsBlock`, `search.query_facts_block`, GBRA-205, default on since the held-out gate passed; `false` or a config read error turns it off): the same facts for every query whose caller passed `token_budget` (the no-budget evidence plan's implicit budget does not run it, #8411), up to 22% of the budget, 40 rows and the caller's `limit`, oldest first ahead of the pages at the top page's score (never above it), adding rows; copies of one claim (`collapseFactReads`, `facts/capture-dedup.ts`) take one row dated as first stated, followed by `(stated N times, last DATE)`; copies are ordered by when each was said (its observed date, else its write date, then id). Fact source pages (`applyFactSourcePages`, `search.fact_source_pages`, GBRA-205, default off): the conversations (`facts.source_markdown_slug`, never the entity page) of the best-matching admitted facts, at most 2 not already present, read through `getChunkWindows` under the caller's read scope and placed after the top 3 rows; the row count never grows. Tests: `test/search-8k-conversion.test.ts`.
- `src/core/search/remote-page-cap.ts` — GBRA-156: the default distinct-page cap of a remote `search`/`query` that names no `limit` (and no `offset`). `remotePageCap(ctx, p)` reads `search.remote_page_cap` (default `DEFAULT_REMOTE_PAGE_CAP`, `0` = off; null for trusted local callers and `viaSubagent` host tool calls); `capVectorOnlyPages(rows, cap)` keeps every page with lexical evidence (`keyword_hit`, strict or relaxed, `alias_hit`, `qualified_alias_hit`, `exact_lookup`, `relational_pinned`) and admits vector-only pages, in rank order, only while the result holds fewer than `cap` distinct `(source_id, slug)` pages; it reports `{ cap, kept_pages, dropped_pages }` as `_meta.retrieval.page_cap` when it dropped any. `pageCapLine(retrieval, how)` renders that as the model-visible line dispatch's `retrievalNoticeBlocks` appends after the keyword count line (hosts that show only `content[]` never see `_meta`), and the CLI's `formatResult` writes on stderr with `--limit`. Applied to the hybrid output before alias fan-out (search) and to the first pass and an adopted CRAG re-run (query); keyword-only paths carry no vector-only rows and the image branch is uncapped (`test/search-remote-page-cap.test.ts`). Every vector path (the hybrid vector arm, the embed-failure fallback, the `query` image branch) reads the keyword path's source scope (`test/search-vector-scope-parity.test.ts`).
diff --git a/docs/evidence-delivery.md b/docs/evidence-delivery.md
index 435dbc69e9..8d94382c44 100644
--- a/docs/evidence-delivery.md
+++ b/docs/evidence-delivery.md
@@ -173,8 +173,13 @@ and its newline are part of `chunk_text`: they count in `delivered.tokens`,
them first and a cut keeps the header line whole. `recall` facts gain
`date_header: "[observed 2026-02-10; valid 2026-03-01 to unknown]"`. A fact's
`observed` date is the observation date of the page it came from
-(`source_markdown_slug`, such as the conversation it was extracted from), and it
-reads `unknown` for a fact kept only in its entity page's facts fence. `valid`
+(`source_markdown_slug`, such as the conversation it was extracted from), read
+under the caller's own page access, and it reads `unknown` for a fact kept only
+in its entity page's facts fence or taken from a page the caller may not read.
+If that read fails, the header reads `unknown` and the response reports the
+degraded stage `fact_observed_date_unavailable` (`recall`:
+`facts_degraded.reason`). The facts block, the temporal fact reserve and the
+facts arm date their fact rows the same way. `valid`
starts at the fact's `valid_from`: `remember` sets it from its `valid_from`
parameter (when the fact was said or became true). A `valid_from` that only
records the write time (within a minute of `created_at`) reads `unknown`, so a
diff --git a/docs/mcp/README.md b/docs/mcp/README.md
index 64062b1461..3d45902a4a 100644
--- a/docs/mcp/README.md
+++ b/docs/mcp/README.md
@@ -209,7 +209,8 @@ pass it to `get_page`.
```
`search.query_facts_arm=false` turns these rows off. A `query` that passes
-`token_budget` instead gets the facts block (on by default): dated fact rows,
+`token_budget` instead gets the facts block (on by default): dated fact rows
+(copies of one claim take one row, followed by `(stated N times, last DATE)`),
oldest first, ahead of the pages, within 22% of the budget and at most 40 rows
and at most `limit` rows, in place of the facts arm. Block rows carry the top
page's `score`: they are placed first, not ranked above the pages, so read
diff --git a/docs/mcp/TOOL_REFERENCE.md b/docs/mcp/TOOL_REFERENCE.md
index f507a07c67..2982d961b1 100644
--- a/docs/mcp/TOOL_REFERENCE.md
+++ b/docs/mcp/TOOL_REFERENCE.md
@@ -355,7 +355,7 @@ MEMORY VERB (v1): retrieve saved facts/snippets — the protocol read verb. Filt
| Parameter | Type | Guidance |
| --- | --- | --- |
| `entity` | string | Entity slug (canonical). Returns facts about this entity newest first, each labelled with its source_id. Across several granted sources, same-slug entities that no entity-identity group links are different entities: facts comes back empty and ambiguous_entity names each (source_id, entity_slug); pass source_id to read one. |
-| `question` | string | Ranks active facts by relevance to this question (cosine to its embedding plus whole-word term share, +0.1 for a writer-dated fact) instead of newest first. Facts below the admission rule are left out, so an empty facts[] is a valid answer. Each fact carries `relevance` (a ranking score, about 0 to 2.1, not comparable across calls); the response carries `facts_order: "relevance"` and, when ranking ran without full semantic matching, `facts_degraded`. When semantic matching ran on an embedding model whose fact admission floors were never measured, the response also carries the informational `facts_calibration: {status: "unmeasured", model_key}` (one `facts_calibration_unmeasured` notice per session explains it; `gbrain doctor --only fact_admission_floors --json` names the model): treat a short facts list as weaker evidence of "nothing matched". Applies inside `entity`, `session_id`, `since` and `grep`. Does not search pages (`query` does; both may be passed). With `include_expired`, expired and superseded facts are ranked too. A question over 12 content words is searched on its last 24 and scored on the 12 of those the fewest candidates hold. `facts_below_floor: <n>` is present when candidates were found and none cleared the admission rule (absent when nothing matched). Rejected with `supersessions`, and over 2,000 characters (invalid_params reason `question_too_long`, detail `max_chars=2000 received_chars=<n>`, with the corrected call as `fix`; never cut). One query embedding per call, shared with a `query` of the same text. |
+| `question` | string | Ranks active facts by relevance to this question (cosine to its embedding plus term share, with words matched on their stems and dates, years and auxiliary verbs ignored, +0.1 for a writer-dated fact) instead of newest first. Facts below the admission rule are left out, so an empty facts[] is a valid answer. Each fact carries `relevance` (a ranking score, about 0 to 2.1, not comparable across calls); the response carries `facts_order: "relevance"` and, when ranking ran without full semantic matching, `facts_degraded`. When semantic matching ran on an embedding model whose fact admission floors were never measured, the response also carries the informational `facts_calibration: {status: "unmeasured", model_key}` (one `facts_calibration_unmeasured` notice per session explains it; `gbrain doctor --only fact_admission_floors --json` names the model): treat a short facts list as weaker evidence of "nothing matched". Applies inside `entity`, `session_id`, `since` and `grep`. Does not search pages (`query` does; both may be passed). With `include_expired`, expired and superseded facts are ranked too. A question over 12 content words is searched on its last 24 and scored on the 12 of those the fewest candidates hold. `facts_below_floor: <n>` is present when candidates were found and none cleared the admission rule (absent when nothing matched). Rejected with `supersessions`, and over 2,000 characters (invalid_params reason `question_too_long`, detail `max_chars=2000 received_chars=<n>`, with the corrected call as `fix`; never cut). One query embedding per call, shared with a `query` of the same text. |
| `query` | string | MEMORY_VERBS v1: free-text retrieval over pages (hybrid search arm). Response adds results[] (slug, title, chunk, evidence, create_safety, provenance). Combinable with entity (both arms run). Degrades to keyword-only search when no embedding provider is configured (search_degraded notes it; never an error). |
| `budget_tokens` | number | MEMORY_VERBS v1: server-side token budget (char/4 estimate). Facts pack first, then results. Response adds budget_tokens, budget_used, dropped_count. |
| `budget_policy` | string (`facts_first`, `query_first`) | Optional packing order. facts_first (default) packs facts first, then page results. query_first packs the ranked page prefix before facts only with a nonblank query and positive finite budget; a positive budget below one token keeps neither arm. Without a nonblank query and an active budget, packing stays facts_first. Neither policy skips oversized items or truncates. Supplying this option adds budget_packing accounting. Keep fact-focused/entity-filtered questions on facts_first. |
diff --git a/docs/protocol/MCP_META_CHANNELS.md b/docs/protocol/MCP_META_CHANNELS.md
index 8922e6ce2b..d679d29375 100644
--- a/docs/protocol/MCP_META_CHANNELS.md
+++ b/docs/protocol/MCP_META_CHANNELS.md
@@ -188,8 +188,8 @@ The codes in `_meta.degraded.reasons` ([AGENT_OPERATOR_v1.md, Degraded stamp](AG
| Code | Ops | Meaning |
|---|---|---|
-| a retrieval stage: `embed_unavailable`, `embed_timeout`, `expansion_failed`, `expansion_partial`, `rescore_skipped`, `vector_arm_failed`, `keyword_arm_failed`, `title_arm_failed`, `budget_dropped_all`, `budget_truncated`, `keyword_zero`, `cache_prestamp`, `rerank_passthrough`, `rerank_failed`, `keyword_relaxed_carried`, `safe_index_pending`, `vector_candidates_incomplete`, `keyword_candidates_incomplete`, `projection_pending`, `projection_by_design`, `projection_status_unknown`, `keyword_count_unavailable`, `saved_facts_withheld` | search, query, recall, find_experts; think and synthesize (their page search) | the stage in `_meta.retrieval.degraded` (every stage except the ranking-only `reranker_skipped`; `projection_pending` only when a pending page could match). `embed_unavailable` also covers a keyless brain and recall's keyword-only markers, so a keyless brain stamps every search |
-| `no_query_terms`, `facts_embedding_column_missing`, `facts_read_failed`, `saved_facts_withheld` | recall | `facts_degraded.reason` of a question-ranked recall |
+| a retrieval stage: `embed_unavailable`, `embed_timeout`, `expansion_failed`, `expansion_partial`, `rescore_skipped`, `vector_arm_failed`, `keyword_arm_failed`, `title_arm_failed`, `budget_dropped_all`, `budget_truncated`, `keyword_zero`, `cache_prestamp`, `rerank_passthrough`, `rerank_failed`, `keyword_relaxed_carried`, `safe_index_pending`, `vector_candidates_incomplete`, `keyword_candidates_incomplete`, `projection_pending`, `projection_by_design`, `projection_status_unknown`, `keyword_count_unavailable`, `saved_facts_withheld`, `fact_observed_date_unavailable` | search, query, recall, find_experts; think and synthesize (their page search) | the stage in `_meta.retrieval.degraded` (every stage except the ranking-only `reranker_skipped`; `projection_pending` only when a pending page could match). `embed_unavailable` also covers a keyless brain and recall's keyword-only markers, so a keyless brain stamps every search |
+| `no_query_terms`, `facts_embedding_column_missing`, `facts_read_failed`, `saved_facts_withheld`, `fact_observed_date_unavailable` | recall | `facts_degraded.reason` of a question-ranked recall |
| `facts_unembedded` | recall | some facts in scope could not be compared with the question |
| `deadline`, `server_budget`, `budget_trimmed` | context_pack | `degraded_reason`: the pack is partial |
| `delta_incomplete` | delta | delta delivered less than is waiting |
diff --git a/src/core/engine-sql/facts.ts b/src/core/engine-sql/facts.ts
index 9f5ee4c252..f426edaed0 100644
--- a/src/core/engine-sql/facts.ts
+++ b/src/core/engine-sql/facts.ts
@@ -136,7 +136,7 @@ export async function insertFact(
`)).rows;
const id = Number(ins[0].id);
(await tx.run(sqlFragment`UPDATE facts SET expired_at = now(), superseded_by = ${id},
- valid_until = CASE WHEN valid_from < ${validFrom}::timestamptz AND (valid_until IS NULL OR valid_until > ${validFrom}::timestamptz) THEN ${validFrom}::timestamptz ELSE valid_until END
+ valid_until = CASE WHEN valid_from < ${validFrom}::timestamptz AND ${validFrom}::timestamptz < COALESCE(valid_until, now()) THEN ${validFrom}::timestamptz ELSE GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now())) END
WHERE id = ${supersedeId} AND expired_at IS NULL`)).rows;
return id;
});
@@ -171,11 +171,16 @@ export async function insertFact(
return { id, status: 'inserted' };
}
-export async function expireFact(exec: SqlExecutor, id: number, opts?: { supersededBy?: number; at?: Date }): Promise<boolean> {
+/** With `successorFrom` (a supersession) the row's validity ends by supersede-interval.ts's rule; a plain expiry leaves valid_until as stored. */
+export async function expireFact(exec: SqlExecutor, id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date }): Promise<boolean> {
const at = opts?.at ?? new Date();
const supersededBy = opts?.supersededBy ?? null;
+ const successor = opts?.successorFrom ?? null;
const result = await exec.run(sqlFragment`
- UPDATE facts SET expired_at = ${at}, superseded_by = COALESCE(${supersededBy}, superseded_by)
+ UPDATE facts SET expired_at = ${at}, superseded_by = COALESCE(${supersededBy}, superseded_by),
+ valid_until = CASE WHEN ${successor}::timestamptz IS NULL THEN valid_until
+ WHEN valid_from < ${successor}::timestamptz AND ${successor}::timestamptz < COALESCE(valid_until, now()) THEN ${successor}::timestamptz
+ ELSE GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now())) END
WHERE id = ${id} AND expired_at IS NULL
`);
return result.affectedRows > 0;
diff --git a/src/core/engine.ts b/src/core/engine.ts
index 6f6b1f1132..cb772979db 100644
--- a/src/core/engine.ts
+++ b/src/core/engine.ts
@@ -2310,10 +2310,10 @@ export interface BrainEngine {
/**
* Mark a fact expired. Never DELETE. Returns true iff a row was updated.
- * Idempotent-as-false (already expired returns false without changing state).
+ * Idempotent-as-false (already expired returns false without changing state). `successorFrom`: end by facts/supersede-interval.ts's rule.
* @deprecated GBRA-161 PR 2: retire through publishFenceGeneration / publishExtractedFacts (docs/architecture/generations.md).
*/
- expireFact(id: number, opts?: { supersededBy?: number; at?: Date }): Promise<boolean>;
+ expireFact(id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date }): Promise<boolean>;
/** List active facts about an entity within a source, newest first. */
listFactsByEntity(
diff --git a/src/core/facts/forget.ts b/src/core/facts/forget.ts
index d75571e02e..431f3a592c 100644
--- a/src/core/facts/forget.ts
+++ b/src/core/facts/forget.ts
@@ -18,7 +18,7 @@ import { withPageLock } from '../page-lock.ts';
import { assertSourceFilesystemActive, hasSourceFilesystemLock, withSourceFilesystemLock } from '../minions/source-filesystem.ts';
import { isWriteThroughDisabled, resolvePageWriteTarget } from '../write-through.ts';
import { readOnlyCheckout } from '../persistence/mirror-read-only.ts';
-import { parseFactsFence, renderFactsTable, type ParsedFact } from '../facts-fence.ts';
+import { formatFenceDate, parseFactsFence, renderFactsTable, type ParsedFact } from '../facts-fence.ts';
import { contentHash } from '../utils.ts';
import { locateOutsideCode } from '../fence-scan.ts';
import { isSegmentedPage } from '../page-body/page-shape.ts';
@@ -70,10 +70,13 @@ const FENCE_END = '<!--- gbrain:facts:end -->';
/**
* A superseded row is struck like a forgotten one but carries the parser's
* `superseded by #N` reference instead of the withdrawal marker, so the next
- * reconcile links it and it never becomes a durable withdrawal.
+ * reconcile links it and it never becomes a durable withdrawal. With `until`
+ * (supersede-interval.ts's rule as a fence date) the struck row ends there
+ * instead of today.
*/
-export function supersededFact(fact: ParsedFact, today: string, newRowNum: number | null): ParsedFact {
+export function supersededFact(fact: ParsedFact, today: string, newRowNum: number | null, until?: string): ParsedFact {
const context = [newRowNum !== null ? `superseded by #${newRowNum}` : 'superseded', fact.context?.trim()].filter(Boolean).join(' | ');
+ if (until !== undefined) return { ...fact, active: false, validUntil: until, context };
const validUntil = fact.validUntil && /^\d{4}-\d{2}-\d{2}$/.test(fact.validUntil) && fact.validUntil < today ? fact.validUntil : today;
return { ...fact, active: false, validUntil, context };
}
@@ -141,7 +144,8 @@ export async function forgetFactInFence(
* durable withdrawal is recorded, so the old claim can be remembered
* again and no other page is invalidated.
*/
- supersededBy?: { rowNum: number | null };
+ /** `validUntil`: where supersede-interval.ts's rule ends the row; absent (no replacement date) it ends at its own earlier end or now. */
+ supersededBy?: { rowNum: number | null; validUntil?: Date };
/**
* Retirement, not withdrawal: the source item the fact was extracted from
* changed or was deleted upstream. Expires and strikes the row with
@@ -155,7 +159,7 @@ export async function forgetFactInFence(
const today = todayUtc();
const supersession = opts.supersededBy;
const strike = (fact: ParsedFact) => supersession
- ? supersededFact(fact, today, supersession.rowNum)
+ ? supersededFact(fact, today, supersession.rowNum, supersession.validUntil && formatFenceDate(supersession.validUntil))
: opts.retired !== undefined ? retiredFact(fact, today, opts.retired) : withdrawnFact(fact, today, reason);
const rows = await engine.executeRaw<FactDbRow>(
@@ -185,10 +189,10 @@ export async function forgetFactInFence(
// A stale source file or rebuilt index must not silently restore an exact
// withdrawn claim. Intent commits independently of filesystem availability.
- // A supersession or a retirement only expires the row.
+ // A supersession or a retirement only expires the row; a supersession with a date ends it there (supersede-interval.ts).
if (supersession || opts.retired !== undefined) {
- await maintenanceTransaction(engine, tx => tx.executeRaw(`UPDATE facts SET valid_until = LEAST(COALESCE(valid_until, now()), now()), expired_at = now()
- WHERE id = $1 AND expired_at IS NULL`, [factId]));
+ await maintenanceTransaction(engine, tx => tx.executeRaw(`UPDATE facts SET valid_until = COALESCE($2::timestamptz, GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now()))), expired_at = now()
+ WHERE id = $1 AND expired_at IS NULL`, [factId, supersession?.validUntil?.toISOString() ?? null]));
} else {
// W1 #6783 §5.1: the withdrawal's ledger publication is marked committed once its transaction commits.
const publicationId = randomUUID();
diff --git a/src/core/facts/proposal-supersede.ts b/src/core/facts/proposal-supersede.ts
index 7b49603349..9c6057acfc 100644
--- a/src/core/facts/proposal-supersede.ts
+++ b/src/core/facts/proposal-supersede.ts
@@ -30,6 +30,7 @@ import { opError } from '../ops/contract.ts';
import { readFix } from '../ops/op-fix.ts';
import { getProposal, transitionProposal, type ProposalRow } from '../ai/decide/proposals-store.ts';
import { strikeFenceRow, supersededFact } from './forget.ts';
+import { supersededValidUntil } from './supersede-interval.ts';
import { maintenanceTransaction } from '../persistence/attribution.ts';
import { wholeBodySnapshot } from '../persistence/segmented-apply.ts';
import { withTrustKeep } from '../persistence/context.ts';
@@ -48,6 +49,8 @@ export interface PairFact {
visibility: string;
fact: string;
expired_at: string | null;
+ /** Read by the supersession rule (supersede-interval.ts); a pair built without it strikes with today's date. */
+ valid_from?: string | null;
valid_until: string | null;
superseded_by: number | null;
row_num: number | null;
@@ -101,12 +104,12 @@ const sha = (s: string) => createHash('sha256').update(s).digest('hex');
export async function loadPairFact(engine: BrainEngine, id: number, lock = false): Promise<PairFact | null> {
const [r] = await engine.executeRaw<Record<string, unknown>>(
- `SELECT id, source_id, entity_slug, visibility, fact, expired_at, valid_until, superseded_by, row_num, source_markdown_slug
+ `SELECT id, source_id, entity_slug, visibility, fact, expired_at, valid_from, valid_until, superseded_by, row_num, source_markdown_slug
FROM facts WHERE id = $1${lock ? ' FOR UPDATE' : ''}`, [id]);
if (!r) return null;
return {
id: Number(r.id), source_id: String(r.source_id), entity_slug: (r.entity_slug as string | null) ?? null, visibility: String(r.visibility),
- fact: String(r.fact), expired_at: iso(r.expired_at), valid_until: iso(r.valid_until),
+ fact: String(r.fact), expired_at: iso(r.expired_at), valid_from: iso(r.valid_from), valid_until: iso(r.valid_until),
superseded_by: r.superseded_by === null || r.superseded_by === undefined ? null : Number(r.superseded_by),
row_num: r.row_num === null || r.row_num === undefined ? null : Number(r.row_num), source_markdown_slug: (r.source_markdown_slug as string | null) ?? null,
};
@@ -136,13 +139,18 @@ export function staleReason(proposal: Pick<ProposalRow, 'source_id'>, oldF: Pair
return null;
}
-/** The old fact's fence row struck as superseded (with `#N` when the new fact lives on the same page). */
+/** The old fact's validity end on accept (supersede-interval.ts's rule), or undefined when the pair carries no dates. */
+const acceptedUntil = (oldF: PairFact, newF: PairFact): Date | undefined =>
+ oldF.valid_from ? supersededValidUntil({ valid_from: oldF.valid_from, valid_until: oldF.valid_until }, newF.valid_from ? new Date(newF.valid_from) : null) : undefined;
+
+/** The old fact's fence row struck as superseded (with `#N` when the new fact lives on the same page), ending where the rule says. */
export function planAcceptFence(body: string, oldF: PairFact, newF: PairFact, today: string): { body: string; before: ParsedFact; after: ParsedFact } | null {
if (oldF.row_num === null || !oldF.source_markdown_slug) return null;
const before = parseFactsFence(body).facts.find((f) => f.rowNum === oldF.row_num);
if (!before) return null;
const newRow = newF.source_markdown_slug === oldF.source_markdown_slug && newF.row_num !== null ? newF.row_num : null;
- const struck = strikeFenceRow(body, oldF.row_num, (f) => supersededFact(f, today, newRow));
+ const until = acceptedUntil(oldF, newF);
+ const struck = strikeFenceRow(body, oldF.row_num, (f) => supersededFact(f, today, newRow, until && formatFenceDate(until)));
if (struck === null) return null;
const after = parseFactsFence(struck).facts.find((f) => f.rowNum === oldF.row_num);
return after ? { body: struck, before, after } : null;
@@ -169,10 +177,12 @@ export function undoRefusal(state: { before: SupersedeState; after: SupersedeSta
// ---------------------------------------------------------------------------
async function expireOld(tx: BrainEngine, proposal: PairProposal): Promise<void> {
+ const [oldF, newF] = await Promise.all([loadPairFact(tx, proposal.old_fact_id), loadPairFact(tx, proposal.new_fact_id)]);
+ const until = oldF && newF ? acceptedUntil(oldF, newF) ?? null : null;
const rows = await tx.executeRaw<{ id: number }>(
- `UPDATE facts SET expired_at = now(), valid_until = LEAST(COALESCE(valid_until, now()), now()), superseded_by = $3
+ `UPDATE facts SET expired_at = now(), valid_until = COALESCE($4::timestamptz, GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now()))), superseded_by = $3
WHERE id = $1 AND source_id = $2 AND expired_at IS NULL RETURNING id`,
- [proposal.old_fact_id, proposal.source_id, proposal.new_fact_id]);
+ [proposal.old_fact_id, proposal.source_id, proposal.new_fact_id, until?.toISOString() ?? null]);
if (rows.length !== 1) throw new ProposalConflictError('the old fact changed during accept');
noteBelief({ kind: 'superseded', oldId: proposal.old_fact_id, newId: proposal.new_fact_id });
}
diff --git a/src/core/facts/question-recall.ts b/src/core/facts/question-recall.ts
index fdcd79a5a9..b003790bf1 100644
--- a/src/core/facts/question-recall.ts
+++ b/src/core/facts/question-recall.ts
@@ -228,7 +228,7 @@ export async function rankFactsByQuestion(ctx: OperationContext, input: Question
]);
const weighted = switches.weighting;
const grouped = switches.grouping && !input.entity;
- const scored = scoreFactCandidates(question, pool.candidates, pool.terms);
+ const scored = scoreFactCandidates(question, pool.candidates, pool.terms, pool.stems);
const ranked = scored.filter(s => admitted(s, rule));
const unembedded = uncomparable.reduce((n, r) => n + r.n, 0);
const why = reason ?? (embedding && !pool.column ? 'facts_embedding_column_missing' : undefined);
diff --git a/src/core/facts/supersede-interval.ts b/src/core/facts/supersede-interval.ts
index f97e2b85a2..e964801295 100644
--- a/src/core/facts/supersede-interval.ts
+++ b/src/core/facts/supersede-interval.ts
@@ -1,10 +1,16 @@
/**
- * #7214: when a fact replaces another, the retired fact's validity interval ends where the replacement's begins.
- * That holds only when the replacement starts after the retired fact and before any end the retired fact already
- * has; a backdated replacement leaves the interval as it was (null means: keep the stored valid_until).
+ * #7214, GBRA-574 A3: when a fact replaces another, where the retired fact's validity interval ends. It ends where the
+ * replacement begins when that falls inside the retired window (after its valid_from, before its stored end or now);
+ * otherwise it ends at its stored end or now, whichever is earlier, so a backdated or future-dated replacement never
+ * leaves a retired fact open-ended. It never ends before the retired fact's own valid_from (a fact dated after now). A write-time valid_from (no date given) lands at the write time either way.
+ * The one rule every supersession writer applies (engine insertFact and expireFact, managed remember and extraction,
+ * the unmanaged fence remember, proposal accept), at write time only; no stored row is rewritten. engine-sql/facts.ts
+ * states it in SQL for insertFact and expireFact.
*/
-export function supersededValidUntil(retired: { valid_from: Date | string; valid_until?: Date | string | null }, replacementFrom: Date): Date | null {
- const next = replacementFrom.getTime();
- const until = retired.valid_until == null ? null : new Date(retired.valid_until).getTime();
- return next > new Date(retired.valid_from).getTime() && (until === null || until > next) ? replacementFrom : null;
+export function supersededValidUntil(retired: { valid_from: Date | string; valid_until?: Date | string | null }, replacementFrom: Date | null, now = new Date()): Date {
+ const from = new Date(retired.valid_from);
+ const end = retired.valid_until == null ? now : new Date(retired.valid_until);
+ if (replacementFrom && replacementFrom.getTime() > from.getTime() && replacementFrom.getTime() < end.getTime()) return replacementFrom;
+ const earlier = end.getTime() < now.getTime() ? end : now;
+ return earlier.getTime() < from.getTime() ? from : earlier;
}
diff --git a/src/core/facts/write-single.ts b/src/core/facts/write-single.ts
index b58e2fe9d2..2ccf38341b 100644
--- a/src/core/facts/write-single.ts
+++ b/src/core/facts/write-single.ts
@@ -42,6 +42,7 @@ import { maintenanceTransaction } from '../persistence/attribution.ts';
import { recordTaintEdges } from '../trust/taint.ts';
import type { TaintInput, WriteTrust } from '../trust/tier.ts';
import { recordFlaggedRow, type GatedRowDecision } from '../write-gate-store.ts';
+import { supersededValidUntil } from './supersede-interval.ts';
/**
* #4755: null-like entity tokens LLM extractors emit for subjectless
@@ -290,7 +291,7 @@ export async function writeSingleFact(
/**
* Fence-path supersession bookkeeping: strike the old row in the fence with a
- * `superseded by #N` reference (strikethrough + valid_until) and link
+ * `superseded by #N` reference (strikethrough + valid_until by supersede-interval.ts's rule) and link
* `superseded_by` for the audit trail. A supersession is an update, never a
* durable withdrawal: the old claim stays rememberable and no other page is
* invalidated. Both steps best-effort — the new fact is already durably
@@ -303,11 +304,12 @@ async function expireSuperseded(engine: BrainEngine, oldId: number, newId: numbe
);
try {
const { forgetFactInFence } = await import('./forget.ts');
- const [replacement] = await engine.executeRaw<{ row_num: number | null; same_page: boolean }>(
- `SELECT n.row_num, n.source_markdown_slug IS NOT DISTINCT FROM o.source_markdown_slug AS same_page
+ const [replacement] = await engine.executeRaw<{ row_num: number | null; same_page: boolean; valid_from: Date | string; old_from: Date | string; old_until: Date | string | null }>(
+ `SELECT n.row_num, n.source_markdown_slug IS NOT DISTINCT FROM o.source_markdown_slug AS same_page, n.valid_from, o.valid_from AS old_from, o.valid_until AS old_until
FROM facts n, facts o WHERE n.id = $1 AND o.id = $2`, [newId, oldId]);
const rowNum = replacement?.same_page && replacement.row_num !== null ? Number(replacement.row_num) : null;
- await forgetFactInFence(engine, oldId, { supersededBy: { rowNum } });
+ const validUntil = replacement ? supersededValidUntil({ valid_from: replacement.old_from, valid_until: replacement.old_until }, new Date(replacement.valid_from)) : new Date();
+ await forgetFactInFence(engine, oldId, { supersededBy: { rowNum, validUntil } });
} catch (err) {
report('fence strike', err);
}
diff --git a/src/core/interop-notices.ts b/src/core/interop-notices.ts
index fc32a54959..37c51cf09d 100644
--- a/src/core/interop-notices.ts
+++ b/src/core/interop-notices.ts
@@ -59,6 +59,7 @@ export const DEGRADED_STAGE_GUIDANCE: Readonly<Record<RecallStage, StageGuidance
projection_status_unknown: { why: 'whether recent writes are in the search index could not be checked', fix: 'doctor' },
keyword_count_unavailable: { why: 'the keyword match count did not finish, so how many pages match the query is unknown', fix: null },
saved_facts_withheld: { why: 'saved facts could not be checked for third-party text to quote, so they were withheld; the page results are complete', fix: 'doctor' },
+ fact_observed_date_unavailable: { why: 'when saved facts were said could not be read, so their date headers show observed unknown; that is not evidence the date is unknown', fix: 'doctor' },
deadline: { why: 'the pack hit its time limit and is partial', fix: null },
server_budget: { why: 'the server ran out of time budget for this pack', fix: null },
no_query_terms: { why: 'the question has no searchable words and semantic search did not run, so no saved fact could be ranked against it', fix: 'embeddings' },
diff --git a/src/core/ops/facts.ts b/src/core/ops/facts.ts
index b6f9cdca8b..95d3adab4e 100644
--- a/src/core/ops/facts.ts
+++ b/src/core/ops/facts.ts
@@ -592,6 +592,7 @@ const recall: Operation = {
const grouped = ranking?.groupByEntity ? groupFactsByEntity(packedFacts, ranking, p, ctx.emitNotice) : null; if (grouped) packedFacts = grouped.rows; // #2953: reorders delivered facts only
const budgetPacking = recallBudgetPacking(p.budget_policy, queryText, budgetTokens, { rows, packedFacts, searchResults, packedResults }, recallResultCost(delivery)), contested: ContestedLookup = {};
+ let observedUnavailable = false;
const out = {
facts: await withDecideContested(ctx.engine, await withFactDateHeaders(ctx.engine, await stampRowTrust(ctx.engine, 'facts', packedFacts.map(r => ({ // #5575 A6 trust labels; GBRA-216 dp contested
id: r.id,
@@ -625,10 +626,11 @@ const recall: Operation = {
provenance: r.source, ...(r.attributed_to ? { attributed_to: r.attributed_to } : {}), ...(r.quoted ? { quoted: r.quoted } : {}), ...recallClaimColumns(r),
...(r.entity_slugs ? { entity_slugs: r.entity_slugs } : {}), ...(r.collapsed_count ? { collapsed_count: r.collapsed_count } : {}),
...(ranking ? { relevance: ranking.relevance.get(r.id), ...(ranking.weighting ? { rank_score: ranking.weighting.rank.get(r.id), confidence_factor: ranking.weighting.factor.get(r.id) } : {}) } : {}),
- })), f => f.id)), { remote: ctx.remote !== false, excludePrivate, eligibility }, contested),
+ })), f => f.id), { excludePrivate, minTrust: eligibility.floor, suppressFlagged: eligibility.suppressFlagged }, () => { observedUnavailable = true; }), { remote: ctx.remote !== false, excludePrivate, eligibility }, contested),
total: packedFacts.length,
facts_order: ranking ? 'relevance' : 'newest',
- ...(quoted.withheld ? { facts_degraded: { reason: 'saved_facts_withheld' } } : ranking?.degraded || contested.degraded ? { facts_degraded: { ...contested.degraded, ...ranking?.degraded } } : {}),
+ ...(quoted.withheld ? { facts_degraded: { reason: 'saved_facts_withheld' } } : ranking?.degraded || contested.degraded || observedUnavailable
+ ? { facts_degraded: { ...(observedUnavailable ? { reason: 'fact_observed_date_unavailable' } : {}), ...contested.degraded, ...ranking?.degraded } } : {}),
...(ranking?.calibration ? { facts_calibration: ranking.calibration } : {}), ...(ranking?.belowFloor ? { facts_below_floor: ranking.belowFloor } : {}),
...grouped?.fields, ...(ambiguousEntity ? { ambiguous_entity: { candidates: ambiguousEntity, suggestion: AMBIGUOUS_ENTITY_SUGGESTION } } : {}),
...(pending_consolidation_count !== undefined ? { pending_consolidation_count } : {}),
diff --git a/src/core/ops/search.ts b/src/core/ops/search.ts
index 40d46caf99..b5fd36cb69 100644
--- a/src/core/ops/search.ts
+++ b/src/core/ops/search.ts
@@ -496,6 +496,7 @@ async function buildRetrievalResponseMeta(
const degraded = [...(m?.degraded ?? [])];
if (safeIndexPending) degraded.push({ stage: 'safe_index_pending' });
if (quotedFacts === null && visible.length) degraded.push({ stage: 'saved_facts_withheld' });
+ if ((results as SearchResult[]).some(r => r.fact_row?.observed_unavailable)) degraded.push({ stage: 'fact_observed_date_unavailable' });
if (readiness.status === 'by_design') degraded.push({ stage: 'projection_by_design' });
else if (readiness.status !== 'ready') {
degraded.push({ stage: readiness.status === 'projection_pending' ? 'projection_pending' : 'projection_status_unknown' });
diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts
index 9ff2532654..552c764cf0 100644
--- a/src/core/pglite-engine.ts
+++ b/src/core/pglite-engine.ts
@@ -2451,7 +2451,7 @@ export class PGLiteEngine implements BrainEngine {
return factsImpl.insertFact(this.engineSql, undefined, input, ctx);
}
- async expireFact(id: number, opts?: { supersededBy?: number; at?: Date }): Promise<boolean> {
+ async expireFact(id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date }): Promise<boolean> {
return factsImpl.expireFact(this.engineSql, id, opts);
}
diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts
index 436c417ca8..b5c8df36da 100644
--- a/src/core/postgres-engine.ts
+++ b/src/core/postgres-engine.ts
@@ -2310,7 +2310,7 @@ export class PostgresEngine implements BrainEngine {
return factsImpl.insertFact(this.engineSql, () => this.resolveFactsEmbeddingCast(), input, ctx);
}
- async expireFact(id: number, opts?: { supersededBy?: number; at?: Date }): Promise<boolean> {
+ async expireFact(id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date }): Promise<boolean> {
return factsImpl.expireFact(this.engineSql, id, opts);
}
diff --git a/src/core/search/evidence-date.ts b/src/core/search/evidence-date.ts
index f4d2762f97..842fed1e1b 100644
--- a/src/core/search/evidence-date.ts
+++ b/src/core/search/evidence-date.ts
@@ -14,9 +14,13 @@
* frontmatter `date`, `published` or a created key, in effective-date
* precedence with `event_date` removed. An event date is when something
* happened, not when the note was written, and row timestamps are when GBrain
- * stored it, so neither is ever shown as observed. Facts carry no observation
- * date on master, so a fact's `observed` is `unknown` and its validity window
- * (`valid_from` to `valid_until`, an open end shown as `unknown`) follows.
+ * stored it, so neither is ever shown as observed. A fact's `observed` is the
+ * observation date of the page it was taken from (`source_markdown_slug`),
+ * read under the caller's page read policy (GBRA-574 A1); `unknown` when that
+ * page states no date, is its own entity page, or is one the caller may not
+ * read. Its validity window (`valid_from` to `valid_until`, an open end shown
+ * as `unknown`) follows. A failed read also renders `unknown`, and the caller
+ * marks the response degraded.
*
* Seam: `pageObservationDate` mirrors `resolveObservationDate` from
* src/core/ai/date-grounding.ts (#6020); once that lands, call it here
@@ -26,6 +30,8 @@
* said, never the event date think shows as content date.
*/
import { computeEffectiveDate, formatBrainDay } from '../effective-date.ts';
+import type { PageReadScope } from '../types.ts';
+import { pageReadFilter } from './read-policy-sql.ts';
export const EVIDENCE_DATE_HEADER_KEY = 'search.evidence_date_header';
/** GBRA-205: with the date header on, a page block's header line also carries its title (default off). */
@@ -103,39 +109,54 @@ export async function loadPageDateHeaders(
return out;
}
-/** `recall` fact rows plus `date_header` when `search.evidence_date_header` is on; unchanged otherwise. */
+/**
+ * `recall` fact rows plus `date_header` when `search.evidence_date_header` is on; unchanged otherwise. `scope` is
+ * the caller's page read policy for the observation dates; a failed read renders `unknown` and calls `onUnavailable`.
+ */
export async function withFactDateHeaders<T extends { id: number; valid_from: string | null; valid_until: string | null; created_at?: string | null; entity_slug?: string | null; source_id?: string }>(
engine: { getConfig(key: string): Promise<string | null>; executeRaw<R>(sql: string, params?: unknown[]): Promise<R[]> },
facts: T[],
+ scope: PageReadScope | undefined,
+ onUnavailable: () => void,
): Promise<Array<T & { date_header?: string }>> {
if (!await evidenceDateHeaderEnabled(engine)) return facts;
const timeZone = (await engine.getConfig('brain.timezone').catch(() => null))?.trim() || null;
- const observed = await factObservationDates(engine, facts, timeZone).catch(() => new Map<number, string>());
+ const observed = await factObservationDates(engine, facts, timeZone, scope).catch(() => { onUnavailable(); return new Map<number, string>(); });
return facts.map(f => ({ ...f, date_header: factDateHeader(f, timeZone, observed.get(f.id)) }));
}
/**
* When each fact was said: the observation date of the page the fact came
* from (`facts.source_markdown_slug`, e.g. the conversation it was extracted
- * from). A fact stored in its own entity page's facts fence has no such page,
- * so it stays unknown. One facts read and one pages read per call.
+ * from), read under `scope` (the caller's page read policy), so a page the
+ * caller may not read dates nothing. A fact stored in its own entity page's
+ * facts fence has no such page, so it stays unknown. One pages read per call,
+ * plus one facts read for rows that do not carry `source_markdown_slug`.
+ * Throws on a database error; the caller marks the response degraded.
*/
-async function factObservationDates(
+export async function factObservationDates(
engine: { executeRaw<R>(sql: string, params?: unknown[]): Promise<R[]> },
- facts: Array<{ id: number; entity_slug?: string | null }>,
+ facts: Array<{ id: number; entity_slug?: string | null; source_id?: string; source_markdown_slug?: string | null }>,
timeZone: string | null,
+ scope: PageReadScope | undefined,
): Promise<Map<number, string>> {
const out = new Map<number, string>();
- const ids = facts.map(f => f.id).filter(id => Number.isInteger(id) && id > 0);
- if (!ids.length) return out;
- const rows = await engine.executeRaw<{ id: number; source_id: string; slug: string | null; entity_slug: string | null }>(
- 'SELECT id, source_id, source_markdown_slug AS slug, entity_slug FROM facts WHERE id = ANY($1::bigint[])', [ids]);
+ const valid = facts.filter(f => Number.isInteger(f.id) && f.id > 0);
+ if (!valid.length) return out;
+ const carried = valid.filter(f => f.source_markdown_slug !== undefined && f.source_id !== undefined);
+ const missing = valid.filter(f => !carried.includes(f)).map(f => f.id);
+ const rows = [
+ ...carried.map(f => ({ id: f.id, source_id: f.source_id!, slug: f.source_markdown_slug ?? null, entity_slug: f.entity_slug ?? null })),
+ ...(missing.length ? await engine.executeRaw<{ id: number; source_id: string; slug: string | null; entity_slug: string | null }>(
+ 'SELECT id, source_id, source_markdown_slug AS slug, entity_slug FROM facts WHERE id = ANY($1::bigint[])', [missing]) : []),
+ ];
const wanted = rows.filter(r => r.slug && r.slug !== r.entity_slug);
if (!wanted.length) return out;
+ const params: unknown[] = [wanted.map(r => r.source_id), wanted.map(r => r.slug!)];
+ const policy = pageReadFilter('p', scope, params, true);
const pages = await engine.executeRaw<{ source_id: string; slug: string; frontmatter: unknown; import_filename: string | null }>(
`SELECT p.source_id, p.slug, p.frontmatter, p.import_filename FROM pages p
- JOIN unnest($1::text[], $2::text[]) AS w(source_id, slug) ON p.source_id = w.source_id AND p.slug = w.slug WHERE p.deleted_at IS NULL`,
- [wanted.map(r => r.source_id), wanted.map(r => r.slug!)]);
+ JOIN unnest($1::text[], $2::text[]) AS w(source_id, slug) ON p.source_id = w.source_id AND p.slug = w.slug WHERE ${policy}`, params);
const byKey = new Map(pages.map(p => {
const fm = typeof p.frontmatter === 'string' ? JSON.parse(p.frontmatter) : p.frontmatter;
const day = formatBrainDay(pageObservationDate({ slug: p.slug, frontmatter: fm && typeof fm === 'object' ? fm as Record<string, unknown> : null, filename: p.import_filename, timeZone }), timeZone);
@@ -164,7 +185,7 @@ export function titledPageHeader(header: string, title: string | null | undefine
return clean ? `${header} ${clean}` : header;
}
-type HeaderHit = { page_id: number; title?: string | null; chunk_text?: string | null; fact_row?: { valid_from: string; valid_until: string | null } };
+type HeaderHit = { page_id: number; title?: string | null; chunk_text?: string | null; fact_row?: { valid_from: string; valid_until: string | null; observed?: string } };
/**
* The date header line (with its newline) for each delivered block, read for
@@ -178,7 +199,7 @@ export async function blockDateHeaders(engine: Parameters<typeof loadPageDateHea
let headers: Map<number, string>;
try { headers = await loadPageDateHeaders(engine, hits.map(h => h.page_id)); } catch { headers = new Map(); onUnavailable(); }
return hit => {
- if (hit.fact_row) return hit.chunk_text?.startsWith('[observed ') ? '' : `${factDateHeader(hit.fact_row)}\n`;
+ if (hit.fact_row) return hit.chunk_text?.startsWith('[observed ') ? '' : `${factDateHeader(hit.fact_row, undefined, hit.fact_row.observed)}\n`;
const header = headers.get(hit.page_id) ?? pageDateHeader(null);
return `${opts.titles ? titledPageHeader(header, hit.title) : header}\n`;
};
diff --git a/src/core/search/fact-relevance.ts b/src/core/search/fact-relevance.ts
index a3b5397958..fc8797c730 100644
--- a/src/core/search/fact-relevance.ts
+++ b/src/core/search/fact-relevance.ts
@@ -14,9 +14,13 @@
* question names, constrained to that page's source.
*
* Score: cosine + term share + DATED_BONUS when the writer supplied the date.
- * Term share counts whole words, and its denominator grows with a long fact's
- * own term count, so a fact stuffed with common words cannot match most
- * questions. Order: score, then newest valid_from, then id.
+ * Term share matches words on their stems under the brain's full-text config
+ * (one to_tsvector read per pool, as the keyword arm matches them; whole words
+ * when that read fails). ISO dates, years, auxiliaries and the config's own
+ * stopwords count on neither side (GBRA-574 A4), so an as-of phrasing does not
+ * dilute the share. Its denominator grows with a long fact's own term count,
+ * so a fact stuffed with common words cannot match most questions. Order:
+ * score, then newest valid_from, then id.
*
* Errors: a missing embedding column or a dimension mismatch skips the cosine
* arm (expected on old or migrating brains); any other database error throws,
@@ -41,6 +45,10 @@ import type { HnswIterativeScanMode } from './hnsw-iterative-scan.ts';
const STOPWORDS = new Set(['the', 'and', 'for', 'who', 'what', 'when', 'where', 'which', 'with', 'from', 'that', 'this', 'are', 'was', 'were', 'our', 'your', 'their',
'now', 'current', 'currently', 'should', 'does', 'did', 'has', 'have', 'how', 'any', 'all', 'about', 'into', 'its', 'next', 'use', 'uses', 'tell']);
const WORD_RE = /[\p{L}\p{N}][\p{L}\p{N}'-]+/gu;
+/** Words that frame a question's time or tense, never its subject: they count toward neither side of the term share. */
+const DATE_TOKEN = /^\d{4}(-\d{2}(-\d{2})?)?$/;
+const AUXILIARIES = new Set(['had', 'been', 'being', 'having', 'will', 'would', 'could', 'might', 'must', 'shall']);
+const neutralWord = (w: string) => DATE_TOKEN.test(w) || AUXILIARIES.has(w);
/** A question with more content terms than this is scored on the ones its candidates hold (`heldTerms`). */
export const MAX_QUERY_TERMS = 12;
@@ -54,6 +62,8 @@ export function queryTerms(query: string): string[] {
}
export const DATED_BONUS = 0.1;
+/** What a term matched only through its stem (Rogers for Roger, lives for live) adds to the score, as a share of an exact match. */
+const STEM_ONLY_WEIGHT = 0.9;
/** valid_from this far from created_at means the writer supplied the date. */
const DATED_MIN_MS = 24 * 60 * 60 * 1000;
/** A fact with more content terms than this many times the question's has its term share scaled down (term stuffing). */
@@ -117,6 +127,8 @@ export interface FactPoolOpts {
export interface FactCandidate {
id: number; fact: string; kind: string; entity_slug: string | null; source_id: string; source: string;
valid_from: Date | string; valid_until: Date | string | null; created_at?: Date | string | null; claim_metric: string | null; claim_period: string | null;
+ /** The page the fact was taken from (its observation date) and the columns copies collapse on (capture-dedup.ts). */
+ source_markdown_slug?: string | null; visibility?: string; attributed_to?: import('../engine.ts').FactAttribution | null;
similarity?: number;
/** Found by the named-entity arm. */
entity: boolean;
@@ -130,9 +142,30 @@ export interface FactPool {
column: boolean;
/** The content terms the keyword arm searched and `scoreFactCandidates` scores with. */
terms: string[];
+ /** Each question and candidate word's lexemes under the brain's full-text config (empty: a stopword); null when that read failed. */
+ stems: WordStems | null;
}
-const COLS = 'f.id, f.fact, f.kind, f.entity_slug, f.source_id, f.source, f.valid_from, f.valid_until, f.created_at, f.claim_metric, f.claim_period';
+/** Word to its full-text lexemes, for `scoreFactCandidates`. */
+export type WordStems = Map<string, string[]>;
+
+/** One to_tsvector read for every word: its lexemes under the brain's full-text config. Null on any database error (whole words then). */
+async function wordStems(engine: BrainEngine, words: string[]): Promise<WordStems | null> {
+ if (!words.length) return new Map();
+ try {
+ const rows = await engine.executeRaw<{ w: string; lexemes: string[] | string | null }>(
+ `SELECT w, tsvector_to_array(to_tsvector('${getFtsLanguage()}'::regconfig, w)) AS lexemes FROM unnest($1::text[]) AS w`, [words]);
+ return new Map(rows.map(r => {
+ const all = Array.isArray(r.lexemes) ? r.lexemes : typeof r.lexemes === 'string' ? r.lexemes.replace(/^\{|\}$/g, '').split(',').filter(Boolean) : [];
+ const whole = all.filter(l => l.length > 1 || l === r.w);
+ return [r.w, whole.length ? whole : all];
+ }));
+ } catch {
+ return null;
+ }
+}
+
+const COLS = 'f.id, f.fact, f.kind, f.entity_slug, f.source_id, f.source, f.valid_from, f.valid_until, f.created_at, f.claim_metric, f.claim_period, f.source_markdown_slug, f.visibility, f.attributed_to';
/** The facts embedding column's type and declared dimensions, or null when there is none. */
async function factsEmbeddingColumn(engine: BrainEngine): Promise<{ type: string; dims: number } | null> {
@@ -264,7 +297,9 @@ export async function collectFactCandidates(engine: BrainEngine, question: strin
add(byCosine);
add(byEntity, true);
const candidates = [...found.values()];
- return { candidates, cosine, column: !!column, terms: heldTerms(searched, candidates) };
+ const terms = heldTerms(searched, candidates);
+ const stems = await wordStems(engine, [...new Set([...terms, ...candidates.flatMap(c => [...wordsOf(c)])])].filter(w => !neutralWord(w)));
+ return { candidates, cosine, column: !!column, terms, stems };
}
/**
@@ -317,15 +352,26 @@ const wordsOf = (c: FactCandidate) => {
export const isDatedFact = (f: Pick<FactCandidate, 'valid_from' | 'created_at'>) =>
!!f.created_at && Math.abs(new Date(f.valid_from).getTime() - new Date(f.created_at).getTime()) > DATED_MIN_MS;
-/** Score every candidate (cosine + term share + DATED_BONUS when dated), ordered by score, newest valid_from, then id. */
-export function scoreFactCandidates(question: string, candidates: FactCandidate[], terms = queryTerms(question)): ScoredFact[] {
+/**
+ * Score every candidate (cosine + term share + DATED_BONUS when dated), ordered by score, newest valid_from, then id.
+ * With `stems` (the pool's) a term matches a fact word sharing a lexeme, and a word with none is a stopword; dates,
+ * years and auxiliaries count on neither side. A stem-only match adds STEM_ONLY_WEIGHT of an exact one to the score
+ * (never to termShare, which admission reads), so "Roger" ranks the Roger fact above the Rogers one.
+ */
+export function scoreFactCandidates(question: string, candidates: FactCandidate[], terms = queryTerms(question), stems: WordStems | null = null): ScoredFact[] {
+ const lexemes = (w: string) => stems?.get(w) ?? [w];
+ const counted = (w: string) => w.length >= 3 && !STOPWORDS.has(w) && !neutralWord(w) && lexemes(w).length > 0;
+ const asked = terms.filter(counted);
return candidates.map(c => {
- const words = wordsOf(c);
- const matched = terms.filter(t => words.has(t)).length;
- const factTerms = [...words].filter(w => w.length >= 3 && !STOPWORDS.has(w)).length;
- const termShare = terms.length ? matched / Math.max(terms.length, Math.ceil(factTerms / TERM_STUFFING_RATIO)) : 0;
+ const words = [...wordsOf(c)].filter(counted);
+ const held = new Set(words.flatMap(lexemes));
+ const hits = asked.filter(t => lexemes(t).some(l => held.has(l)));
+ const exact = hits.filter(t => words.includes(t)).length;
+ const denominator = Math.max(asked.length, Math.ceil(words.length / TERM_STUFFING_RATIO));
+ const termShare = asked.length ? hits.length / denominator : 0;
+ const scoredShare = asked.length ? (exact + STEM_ONLY_WEIGHT * (hits.length - exact)) / denominator : 0;
const cosine = c.similarity ?? 0;
- return { candidate: c, cosine, matched, terms: terms.length, termShare, score: cosine + termShare + (isDatedFact(c) ? DATED_BONUS : 0) };
+ return { candidate: c, cosine, matched: hits.length, terms: asked.length, termShare, score: cosine + scoredShare + (isDatedFact(c) ? DATED_BONUS : 0) };
}).sort((a, b) => b.score - a.score
|| new Date(b.candidate.valid_from).getTime() - new Date(a.candidate.valid_from).getTime()
|| b.candidate.id - a.candidate.id);
diff --git a/src/core/search/facts-arm.ts b/src/core/search/facts-arm.ts
index 380adbb840..e675333944 100644
--- a/src/core/search/facts-arm.ts
+++ b/src/core/search/facts-arm.ts
@@ -23,7 +23,8 @@ import { resolveExcludePrivatePages, worldOnlyFacts } from './private-visibility
import { admitted, collectFactCandidates, FACTS_ARM_COSINE_MIN, queryTerms, scoreFactCandidates, type FactCandidate as PoolCandidate, type FactPoolScope } from './fact-relevance.ts';
import { pageReadFilter } from './read-policy-sql.ts';
import { enforceTokenBudget, resultTokens } from './token-budget.ts';
-import { configFlagOn, factDateHeader } from './evidence-date.ts';
+import { configFlagOn, factDateHeader, factObservationDates } from './evidence-date.ts';
+import { collapseFactReads } from '../facts/capture-dedup.ts';
import type { TrustTier } from '../trust/tier.ts';
import { conversationSignal } from './evidence-delivery.ts';
import { factsFromQuotedPages } from './transcript-quote.ts';
@@ -92,7 +93,20 @@ export async function matchQueryFacts(engine: BrainEngine, query: string, scope:
.slice(0, MAX_FACT_ROWS);
}
-function factRow(f: FactCandidate, score: number, pageSlug: string | undefined, fromQuotedPage: boolean): SearchResult {
+/**
+ * When each fact was said (evidence-date.ts `factObservationDates`, under the caller's page read policy), or null
+ * when the read failed: its rows then carry `observed_unavailable` and the response is marked degraded.
+ */
+async function observedDates(engine: BrainEngine, facts: FactCandidate[], readScope: PageReadScope | undefined): Promise<{ dates: Map<number, string> | null; timeZone: string | null }> {
+ const timeZone = (await engine.getConfig('brain.timezone').catch(() => null))?.trim() || null;
+ try {
+ return { dates: await factObservationDates(engine, facts, timeZone, readScope), timeZone };
+ } catch {
+ return { dates: null, timeZone };
+ }
+}
+
+function factRow(f: FactCandidate, score: number, pageSlug: string | undefined, fromQuotedPage: boolean, observed: Map<number, string> | null): SearchResult {
const from = day(f.valid_from);
const until = day(f.valid_until);
const args: Record<string, string> = f.entity_slug ? { entity: f.entity_slug } : { grep: f.fact.slice(0, 60) };
@@ -103,7 +117,8 @@ function factRow(f: FactCandidate, score: number, pageSlug: string | undefined,
chunk_text: `Saved fact (${f.kind}; valid from ${from ?? 'unknown'}${until ? ` to ${until}` : ''}; provenance: ${f.source}): ${f.fact}`,
chunk_source: 'compiled_truth', chunk_id: -Number(f.id), chunk_index: 0, score, stale: false, source_id: f.source_id,
fact_row: { id: Number(f.id), valid_from: new Date(f.valid_from).toISOString(), valid_until: f.valid_until ? new Date(f.valid_until).toISOString() : null, source: f.source,
- ...(fromQuotedPage ? { from_quoted_page: true as const } : {}) },
+ ...(fromQuotedPage ? { from_quoted_page: true as const } : {}),
+ ...(observed === null ? { observed_unavailable: true as const } : observed.has(Number(f.id)) ? { observed: observed.get(Number(f.id))! } : {}) },
} as SearchResult;
}
@@ -177,7 +192,8 @@ export async function applyFactsArm(engine: BrainEngine, query: string, results:
const shown = facts.slice(0, free);
const pageSlugs = await readableEntityPages(engine, shown, opts.readScope).catch(() => new Set<string>());
const fromQuoted = await factsFromQuotedPages(engine, shown.map(f => Number(f.id)));
- let rows = shown.map((f, i) => factRow(f, floor - (i + 1) * 1e-6, f.entity_slug && pageSlugs.has(`${f.source_id}\u0000${f.entity_slug}`) ? f.entity_slug : undefined, fromQuoted.has(Number(f.id))));
+ const { dates } = await observedDates(engine, shown, opts.readScope);
+ let rows = shown.map((f, i) => factRow(f, floor - (i + 1) * 1e-6, f.entity_slug && pageSlugs.has(`${f.source_id}\u0000${f.entity_slug}`) ? f.entity_slug : undefined, fromQuoted.has(Number(f.id)), dates));
if (opts.tokenBudget) {
let left = opts.tokenBudget - enforceTokenBudget(pages, opts.tokenBudget).meta.used;
rows = rows.filter(r => { const cost = resultTokens(r); if (cost > left) return false; left -= cost; return true; });
@@ -222,8 +238,10 @@ export function applyTemporalFactReserve(engine: BrainEngine, query: string, res
* temporal reserve, for every query whose caller passed a token budget, up to
* QUERY_FACTS_BLOCK_SHARE of the budget and MAX_FACTS_BLOCK_ROWS rows and
* never more rows than the caller's row count (#8411), oldest first ahead of
- * the pages. Block rows take the top page's score: they are placed, not
- * ranked against the pages, and never outscore them. The facts add rows
+ * the pages. Copies of one claim take one row, dated as first stated with
+ * "(stated N times, last D)" after the fact (GBRA-574 A2), and each row's date
+ * header carries when the fact was said (A1). Block rows take the top page's
+ * score: they are placed, not ranked against the pages, and never outscore them. The facts add rows
* (every page row stays) and pages fill the rest of the budget. No match or
* any error: the results unchanged.
*/
@@ -233,21 +251,31 @@ export function applyFactsBlock(engine: BrainEngine, query: string, results: Sea
async function applyFactReserve(engine: BrainEngine, query: string, results: SearchResult[], opts: FactsArmOpts & { budget: number }, policy: ReservePolicy): Promise<SearchResult[]> {
try {
- const { candidates } = await collectFactCandidates(engine, query, await poolScope(engine, opts), opts.queryEmbedding,
+ const { candidates, stems } = await collectFactCandidates(engine, query, await poolScope(engine, opts), opts.queryEmbedding,
{ depth: { keyword: 50, cosine: 50, entity: policy.maxRows }, cosineForAll: true });
- const scored = scoreFactCandidates(query, candidates).filter(s => admitted(s, 'reserve')).map(s => ({ c: s.candidate }));
+ const scored = scoreFactCandidates(query, candidates, queryTerms(query), stems).filter(s => admitted(s, 'reserve')).map(s => s.candidate);
if (!scored.length) return results;
const callerRows = (await opts.rowCap?.().catch(() => 0)) ?? 0;
const rowCap = Math.max(results.length, callerRows);
const maxRows = policy.rowShare === null ? Math.min(policy.maxRows, callerRows || policy.maxRows) : Math.min(policy.maxRows, Math.max(1, Math.floor(rowCap * policy.rowShare)));
- const pageSlugs = await readableEntityPages(engine, scored.map(x => x.c), opts.readScope).catch(() => new Set<string>());
- const fromQuoted = await factsFromQuotedPages(engine, scored.map(x => Number(x.c.id)));
+ // GBRA-574 A2: copies of one claim (exact fingerprint, capture-dedup.ts) take one row, shown as first stated; a differing value never merges.
+ const byId = new Map(scored.map(c => [Number(c.id), c]));
+ const collapsed = await collapseFactReads(engine, scored.map(c => ({ ...c, id: Number(c.id), created_at: c.created_at ?? c.valid_from, visibility: c.visibility ?? '', attributed_to: c.attributed_to ?? null })));
+ const members = collapsed.map(r => (r.collapsed_ids ?? [r.id]).map(id => byId.get(id)!));
+ const { dates } = await observedDates(engine, members.flat(), opts.readScope);
+ const said = (c: FactCandidate) => dates?.get(Number(c.id)) ?? day(c.created_at ?? c.valid_from) ?? '';
+ const groups = members.map(g => g.sort((a, b) => said(a).localeCompare(said(b)) || Number(a.id) - Number(b.id)));
+ const shown = groups.map(g => g[0]!);
+ const pageSlugs = await readableEntityPages(engine, shown, opts.readScope).catch(() => new Set<string>());
+ const fromQuoted = await factsFromQuotedPages(engine, shown.map(c => Number(c.id)));
let left = Math.floor(opts.budget * policy.share);
const chosen: Array<{ c: FactCandidate; row: SearchResult }> = [];
- for (const { c } of scored) {
+ for (const group of groups) {
if (chosen.length >= maxRows) break;
- const row = factRow(c, 0, c.entity_slug && pageSlugs.has(`${c.source_id}\u0000${c.entity_slug}`) ? c.entity_slug : undefined, fromQuoted.has(Number(c.id)));
- row.chunk_text = `${factDateHeader(c)}\n${c.fact}`;
+ const c = group[0]!;
+ const restated = group.length > 1 ? ` (stated ${group.length} times, last ${said(group[group.length - 1]!)})` : '';
+ const row = factRow(c, 0, c.entity_slug && pageSlugs.has(`${c.source_id}\u0000${c.entity_slug}`) ? c.entity_slug : undefined, fromQuoted.has(Number(c.id)), dates);
+ row.chunk_text = `${factDateHeader(c, undefined, dates?.get(Number(c.id)))}\n${c.fact}${restated}`;
const cost = resultTokens(row);
if (cost > left) continue;
left -= cost;
@@ -282,9 +310,9 @@ async function applyFactReserve(engine: BrainEngine, query: string, results: Sea
*/
export async function applyFactSourcePages(engine: BrainEngine, query: string, results: SearchResult[], opts: FactsArmOpts): Promise<SearchResult[]> {
try {
- const { candidates } = await collectFactCandidates(engine, query, await poolScope(engine, opts), opts.queryEmbedding,
+ const { candidates, stems } = await collectFactCandidates(engine, query, await poolScope(engine, opts), opts.queryEmbedding,
{ depth: { keyword: 50, cosine: 50, entity: MAX_RESERVE_ROWS }, cosineForAll: true });
- const ranked = scoreFactCandidates(query, candidates).filter(s => admitted(s, 'reserve')).map(s => Number(s.candidate.id));
+ const ranked = scoreFactCandidates(query, candidates, queryTerms(query), stems).filter(s => admitted(s, 'reserve')).map(s => Number(s.candidate.id));
if (!ranked.length) return results;
const sources = await engine.executeRaw<{ id: number; page_id: number; slug: string; title: string | null; type: string | null }>(
`SELECT f.id, p.id AS page_id, p.slug, p.title, p.type FROM facts f
diff --git a/src/core/types.ts b/src/core/types.ts
index 9fce9b0b10..6138f075c1 100644
--- a/src/core/types.ts
+++ b/src/core/types.ts
@@ -1131,8 +1131,8 @@ export interface SearchResult {
* entity's page, or a page that links to it or names it (newest first).
*/
entity_anchored?: 'entity' | 'linked';
- /** Set on a saved-fact row the `search.query_facts_arm` arm added (search/facts-arm.ts); page_id is 0. */
- fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true };
+ /** Set on a saved-fact row the `search.query_facts_arm` arm added (search/facts-arm.ts); page_id is 0. `observed`: when it was said; `observed_unavailable`: that read failed. */
+ fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true; observed?: string; observed_unavailable?: true };
/** Fact rows only, and what callers see of them: never a page (no slug or id); follow with `follow_up`, read `page_slug` when the entity page exists. */
result_type?: 'fact'; fact_id?: string; page_slug?: string; follow_up?: { op: 'recall'; args: Record<string, string> };
/** #7600, #8042: a row of third-party text (a chat page or a fact saved from one), returned fenced as quoted source material with its tool-call blocks summarized (search/transcript-quote.ts). */
@@ -2105,7 +2105,7 @@ export const DEGRADED_STAGES = [
'keyword_candidates_incomplete',
'projection_pending', 'projection_by_design',
'projection_status_unknown',
- 'keyword_count_unavailable', 'saved_facts_withheld',
+ 'keyword_count_unavailable', 'saved_facts_withheld', 'fact_observed_date_unavailable',
] as const;
export type DegradedStage = (typeof DEGRADED_STAGES)[number];
diff --git a/src/core/verbs.ts b/src/core/verbs.ts
index c6d0691005..1e809b34d5 100644
--- a/src/core/verbs.ts
+++ b/src/core/verbs.ts
@@ -629,9 +629,9 @@ export const RESPONSE_SCHEMAS: Record<VerbName, Record<string, unknown>> = {
facts_order: { type: 'string', enum: ['relevance', 'newest'], description: 'How facts[] is ordered: relevance (question passed) or newest. A server without question ranking omits it.' },
facts_degraded: {
type: 'object',
- description: 'Present when question ranking ran without full semantic matching.',
+ description: 'Present when question ranking ran without full semantic matching, or date headers could not read when facts were said.',
properties: {
- reason: { type: 'string', description: 'Why the cosine arm did not run: keyword_only_no_embedding_provider, keyword_only_embedding_disabled, embed_timeout, embed_unavailable, no_query_terms or facts_embedding_column_missing.' },
+ reason: { type: 'string', description: 'Why the cosine arm did not run: keyword_only_no_embedding_provider, keyword_only_embedding_disabled, embed_timeout, embed_unavailable, no_query_terms or facts_embedding_column_missing; or fact_observed_date_unavailable: date headers show observed unknown because the source-page read failed.' },
unembedded: { type: 'integer', description: 'Facts in scope without a comparable embedding (ranked by word overlap only).' },
},
},
diff --git a/test/dated-facts-wave-a.test.ts b/test/dated-facts-wave-a.test.ts
new file mode 100644
index 0000000000..d41c7f177a
--- /dev/null
+++ b/test/dated-facts-wave-a.test.ts
@@ -0,0 +1,154 @@
+/**
+ * GBRA-574 wave A: dated facts reach the reader of a time-indexed question. PGLite, keyless, no network.
+ *
+ * Fixture shape mirrors what extract-conversation-facts writes for a chat history: one conversation page per
+ * session (frontmatter date = when it was said, about 400 tokens each), one fact row per extracted claim, source
+ * cli:extract-conversation-facts:..., source_markdown_slug = the session page, valid_from = the event date the
+ * extractor stated, else the session date. Retrieval is query with limit 50 and token_budget 5,500: the facts
+ * block on (default), no date headers on page blocks (default).
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { handleToolCall } from '../src/mcp/server.ts';
+import { collectFactCandidates, scoreFactCandidates } from '../src/core/search/fact-relevance.ts';
+import { newSource, page, putPage } from './helpers/pinned-questions-fixture.ts';
+
+type Row = { result_type?: string; chunk_text?: string; slug?: string };
+const SRC = 'cli:extract-conversation-facts:v1';
+const ENTITY = 'people/alice-example';
+let engine: PGLiteEngine;
+let sourceId: string;
+
+const day = (i: number) => new Date(Date.UTC(2025, 0, 1) + i * 3 * 86_400_000).toISOString().slice(0, 10);
+const session = (i: number) => 'conversations/chat-' + String(i).padStart(3, '0');
+const FILLER = (i: number) => Array.from({ length: 22 }, (_, k) => 'Agenda item ' + (k + 1) + ' of meeting ' + i
+ + ' covered budget forecasts, vendor timelines, quarterly hiring plans and the office move checklist.').join(' ');
+
+async function fact(text: string, sessionIdx: number, validFrom = day(sessionIdx), entity = ENTITY) {
+ await engine.executeRaw(
+ "INSERT INTO facts (source_id, entity_slug, fact, kind, source, visibility, confidence, valid_from, source_markdown_slug) VALUES ($1, $2, $3, 'event', $4, 'world', 1, $5::timestamptz, $6)",
+ [sourceId, entity, text, SRC, validFrom + 'T00:00:00Z', session(sessionIdx)]);
+}
+
+const ask = async (query: string) =>
+ await handleToolCall(engine, 'query', { query, expand: false, use_cache: false, source_id: sourceId, limit: 50, token_budget: 5500 }) as Row[];
+const context = (rows: Row[]) => rows.map(r => r.chunk_text ?? '').join('\n---\n');
+
+const CITIES: Array<[string, number, string]> = [
+ ['Lisbon', 5, 'Alice Example just signed a lease on a flat in Lisbon.'],
+ ['Porto', 30, 'Alice Example relocated to Porto for the new studio.'],
+ ['Bergen', 70, 'Alice Example packed up again; her new place is in Bergen.'],
+ ['Oslo', 110, 'Alice Example lives in Oslo after another move.'],
+];
+const S8Q = 'In order from earliest, where has Alice Example lived?';
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ sourceId = await newSource(engine);
+ await putPage(engine, sourceId, ENTITY, page('person', 'Alice Example', 'Alice Example is a designer.'));
+ const said: Record<number, string> = Object.fromEntries(CITIES.map(([, i, s]) => [i, s]));
+ said[10] = 'Bob Example works at Beta Corp as a designer.';
+ said[60] = 'I just found out Bob Example left Beta Corp back in February and joined Gamma Labs.';
+ for (let i = 1; i <= 120; i++) {
+ const body = said[i] ?? (i >= 40 ? 'We caught up with Alice Example about the design review for release ' + i + '.' : 'Weekly planning, session ' + i + '.');
+ await putPage(engine, sourceId, session(i), page('conversation', 'Chat ' + i, '**User:** ' + body + ' ' + FILLER(i) + '\n\n**Assistant:** Noted.', 'date: ' + day(i) + '\n'));
+ }
+ for (const [city, i] of CITIES) await fact('Alice Example lives in ' + city + '.', i);
+ for (let i = 111; i <= 120; i++) await fact('Alice Example still lives in Oslo.', i);
+ for (let i = 40; i <= 120; i += 2) await fact('Alice Example led the design review for release ' + i + '.', i);
+ await putPage(engine, sourceId, 'people/bob-example', page('person', 'Bob Example', 'Bob Example is an engineer.'));
+ await fact('Bob Example works at Beta Corp.', 10, day(10), 'people/bob-example');
+ await fact('Bob Example left Beta Corp and joined Gamma Labs.', 60, '2025-02-15', 'people/bob-example');
+ await fact('Dana Example had been promoted in 2025.', 80, day(80), 'people/dana-example');
+ for (const [slug, name] of [['people/sam-roger', 'Sam Roger'], ['people/sam-rogers', 'Sam Rogers']]) await putPage(engine, sourceId, slug, page('person', name, name + ' is a client.'));
+ await fact('Sam Roger lives in Porto.', 20, day(20), 'people/sam-roger');
+ await fact('Sam Rogers lives in Lisbon.', 21, day(21), 'people/sam-rogers');
+ await fact('Carol Example moved to Madrid.', 7, '2020-01-01', 'people/carol-example');
+ await fact('Carol Example moved to Madrid.', 9, '2019-01-01', 'people/carol-example');
+ await fact("Kim O'Brien plays the cello.", 23, day(23), 'people/kim-obrien');
+ await fact("Lee O'Neil plays the violin.", 24, day(24), 'people/lee-oneil');
+ await fact('Pat Example built an AI-powered search tool.', 25, day(25), 'people/pat-example');
+}, 300_000);
+afterAll(async () => { await engine?.disconnect(); }, 60_000);
+
+describe('S8 history: every dated value of the slot reaches the reader', () => {
+ test('S8a extractor rows (all active): every city reaches the reader in a dated row', async () => {
+ const rows = await ask(S8Q);
+ const dated = rows.filter(r => /\[observed \d{4}-\d\d-\d\d|valid \d{4}-\d\d-\d\d/.test(r.chunk_text ?? '')).map(r => r.chunk_text).join('\n');
+ for (const [c] of CITIES) expect(dated).toContain(c);
+ });
+ // Ended values reach the reader only with B1 (history in the facts block, the next PR); it removes `.failing`.
+ test.failing('S8b after consolidate stamps valid_until on the older values: all four cities are in the context', async () => {
+ const close = (city: string, until: number) => engine.executeRaw(
+ 'UPDATE facts SET valid_until = $2::timestamptz WHERE source_id = $1 AND fact = $3', [sourceId, day(until) + 'T00:00:00Z', 'Alice Example lives in ' + city + '.']);
+ await close('Lisbon', 30);
+ await close('Porto', 70);
+ await close('Bergen', 110);
+ const ctx = context(await ask(S8Q));
+ for (const [c] of CITIES) expect(ctx).toContain(c);
+ });
+});
+
+describe('A2 copies of one claim take one row; a differing value keeps its own', () => {
+ test('ten restatements collapse to one row dated as first stated; the value one word apart stays', async () => {
+ const facts = (await ask(S8Q)).filter(r => r.result_type === 'fact').map(r => r.chunk_text ?? '');
+ const still = facts.filter(t => t.includes('Alice Example still lives in Oslo.'));
+ expect(still).toHaveLength(1);
+ expect(still[0]!.split('\n')[0]).toBe('[observed ' + day(111) + '; valid ' + day(111) + ' to unknown]');
+ expect(still[0]).toContain('Alice Example still lives in Oslo. (stated 10 times, last ' + day(120) + ')');
+ expect(facts.filter(t => /(^|\n)Alice Example lives in Oslo\.(\n|$)/.test(t))).toHaveLength(1);
+ });
+ test('copies are ordered by when they were said, not by their event date', async () => {
+ const facts = (await ask('Where did Carol Example move to?')).filter(r => r.result_type === 'fact').map(r => r.chunk_text ?? '');
+ const moved = facts.filter(t => t.includes('Carol Example moved to Madrid.'));
+ expect(moved).toHaveLength(1);
+ expect(moved[0]!.split('\n')[0]).toBe('[observed ' + day(7) + '; valid 2020-01-01 to unknown]');
+ expect(moved[0]).toContain('Carol Example moved to Madrid. (stated 2 times, last ' + day(9) + ')');
+ });
+});
+
+describe('S7 keyless admission', () => {
+ test('an as-known phrasing still admits the employer facts (date token and auxiliaries do not dilute the term share)', async () => {
+ const rows = await ask('As of 2025-04-01, what had I been told about where Bob Example works?');
+ expect(rows.filter(r => r.result_type === 'fact' && /Gamma Labs/.test(r.chunk_text ?? '')).length).toBe(1);
+ });
+ test('precision: a fact that shares only a year and auxiliaries with the question is not admitted', async () => {
+ const rows = await ask('What had been decided in 2025 about the vendor contract?');
+ expect(rows.filter(r => r.result_type === 'fact' && /Dana Example/.test(r.chunk_text ?? ''))).toEqual([]);
+ });
+});
+
+describe('A4 near names: an exact word outranks a stem-only match', () => {
+ const Q = 'Where did Sam Roger live in 2025?';
+ test('the named person\'s fact scores above the plural-shaped name', async () => {
+ const pool = await collectFactCandidates(engine, Q, { sourceIds: [sourceId], worldOnly: true, excludePrivate: false }, null, { depth: { keyword: 50, cosine: 50, entity: 40 } });
+ const order = scoreFactCandidates(Q, pool.candidates, pool.terms, pool.stems).map(s => s.candidate.fact).filter(f => f.startsWith('Sam Roger'));
+ expect(order).toEqual(['Sam Roger lives in Porto.', 'Sam Rogers lives in Lisbon.']);
+ });
+ test('a one-letter name fragment is no match: a question about O\'Brien does not admit an O\'Neil fact', async () => {
+ const facts = (await ask("What does Kim O'Brien play?")).filter(r => r.result_type === 'fact').map(r => r.chunk_text ?? '');
+ expect(facts.some(t => t.includes("Kim O'Brien plays the cello."))).toBe(true);
+ expect(facts.filter(t => t.includes("O'Neil"))).toEqual([]);
+ });
+ test('a two-letter acronym still matches the hyphenated word it starts', async () => {
+ const pool = await collectFactCandidates(engine, 'Which AI tool did Pat build?', { sourceIds: [sourceId], worldOnly: true, excludePrivate: false }, null, { depth: { keyword: 50, cosine: 50, entity: 40 } });
+ const pat = scoreFactCandidates('Which AI tool did Pat build?', pool.candidates, pool.terms, pool.stems).find(s => s.candidate.fact.startsWith('Pat Example'));
+ expect(pool.stems?.get('ai-powered')).toContain('ai');
+ expect(pat).toBeDefined();
+ });
+ test('the facts block puts it first', async () => {
+ const facts = (await ask(Q)).filter(r => r.result_type === 'fact').map(r => r.chunk_text ?? '').filter(t => t.includes('Sam Roger'));
+ expect(facts[0]).toContain('Sam Roger lives in Porto.');
+ });
+});
+
+describe('S7 as-known: the context says when each value was learned', () => {
+ test('the late-learned change carries the date it was said (session 60), not only its event date', async () => {
+ const rows = await ask('As of 2025-04-01, where did Bob Example work?');
+ const gamma = rows.find(r => r.result_type === 'fact' && r.chunk_text?.includes('Gamma Labs'));
+ expect(gamma).toBeDefined();
+ expect(gamma!.chunk_text).toContain('observed ' + day(60));
+ });
+});
diff --git a/test/fact-observed-date-read-policy.test.ts b/test/fact-observed-date-read-policy.test.ts
new file mode 100644
index 0000000000..e9e6b515e6
--- /dev/null
+++ b/test/fact-observed-date-read-policy.test.ts
@@ -0,0 +1,101 @@
+/**
+ * GBRA-574 A1: a fact row's `observed` date (when it was said: the observation date of the page it was taken from) is
+ * read under the caller's own page read policy on every path that prints it, and a failed read marks the response
+ * degraded instead of passing `observed unknown` off as a fact. PGLite, keyword-only, no network.
+ */
+import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { dispatchToolCall } from '../src/mcp/dispatch.ts';
+import { EVIDENCE_DATE_HEADER_KEY, factObservationDates } from '../src/core/search/evidence-date.ts';
+import { withCallerAccess } from '../src/core/search/private-visibility.ts';
+import { newSource, page, putPage } from './helpers/pinned-questions-fixture.ts';
+
+let engine: PGLiteEngine;
+let sourceId: string;
+const Q = 'Where is the Forge offsite venue booked?';
+const facts = () => [
+ { id: 1, source_id: sourceId, entity_slug: 'forge', source_markdown_slug: 'conversations/open-chat' },
+ { id: 2, source_id: sourceId, entity_slug: 'forge', source_markdown_slug: 'conversations/team-chat' },
+ { id: 3, source_id: sourceId, entity_slug: 'forge', source_markdown_slug: 'conversations/secret-chat' },
+ { id: 4, source_id: sourceId, entity_slug: 'conversations/open-chat', source_markdown_slug: 'conversations/open-chat' },
+];
+const read = (access: Parameters<typeof withCallerAccess>[0] | null) => {
+ const run = () => factObservationDates(engine, facts(), null, access ? { sourceIds: [sourceId], excludePrivate: true } : { sourceIds: [sourceId] });
+ return access ? withCallerAccess(access, run) : run();
+};
+const call = async (name: string, params: Record<string, unknown>, remote: boolean) => {
+ const auth = { token: 't', clientId: 'observed-date', scopes: ['read'], sourceId, allowedSources: [sourceId] };
+ const r = await dispatchToolCall(engine, name, params,
+ { remote, transport: 'stdio', sourceId, ...(remote ? { auth } : {}), config: { engine: engine.kind } as never, logger: { info() {}, warn() {}, error() {} } });
+ return r.content.map(c => (c as { text?: string }).text ?? '').join('\n') + '\n' + JSON.stringify(r._meta ?? {});
+};
+/** Runs `fn` with every observation-date page read failing (`this` kept: a transaction scope inherits the method). */
+async function observationReadFails<T>(fn: () => Promise<T>): Promise<T> {
+ const original = PGLiteEngine.prototype.executeRaw;
+ engine.executeRaw = (function (this: PGLiteEngine, sql: string, ...rest: unknown[]) {
+ if (sql.includes('p.import_filename FROM pages p') && sql.includes('unnest($1::text[], $2::text[])')) return Promise.reject(new Error('observation read failed'));
+ return (original as (...a: unknown[]) => unknown).call(this, sql, ...rest);
+ }) as typeof engine.executeRaw;
+ try { return await fn(); } finally { delete (engine as { executeRaw?: unknown }).executeRaw; }
+}
+
+beforeAll(async () => {
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ sourceId = await newSource(engine);
+ await putPage(engine, sourceId, 'conversations/open-chat', page('conversation', 'Open chat', '**User:** The Forge offsite venue changed.', 'date: 2025-02-01\n'));
+ await putPage(engine, sourceId, 'conversations/team-chat', page('conversation', 'Team chat', '**User:** The Forge offsite venue changed again.', 'date: 2025-03-01\nreadable_by: [alice-example]\n'));
+ await putPage(engine, sourceId, 'conversations/secret-chat', page('conversation', 'Secret chat', '**User:** Another venue.', 'date: 2025-04-01\nvisibility: private\n'));
+ const fact = (text: string, slug: string) => engine.executeRaw(
+ "INSERT INTO facts (source_id, entity_slug, fact, kind, source, visibility, confidence, valid_from, source_markdown_slug) VALUES ($1, 'forge', $2, 'event', 'conversation', 'world', 1, '2025-06-02T00:00:00Z', $3)",
+ [sourceId, text, slug]);
+ await fact('The Forge offsite venue is booked in Missoula.', 'conversations/open-chat');
+ await fact('The Forge offsite venue is booked in Taos.', 'conversations/team-chat');
+}, 120_000);
+afterAll(async () => { await engine?.disconnect(); }, 60_000);
+
+describe('factObservationDates reads under the caller policy', () => {
+ test('a local caller reads every source page date; a fact on its own entity page has none', async () => {
+ expect([...(await read(null)).entries()]).toEqual([[1, '2025-02-01'], [2, '2025-03-01'], [3, '2025-04-01']]);
+ });
+
+ test('a remote caller gets no date from a private page or a readable_by page that does not name it', async () => {
+ expect([...(await read({ identity: [] })).entries()]).toEqual([[1, '2025-02-01']]);
+ });
+
+ test('a member-scoped caller reads the date of a readable_by page that names it, and only that', async () => {
+ expect([...(await read({ identity: ['member:alice-example'] })).entries()]).toEqual([[1, '2025-02-01'], [2, '2025-03-01']]);
+ expect([...(await read({ identity: ['member:bob-example'] })).entries()]).toEqual([[1, '2025-02-01']]);
+ });
+});
+
+describe('query facts block', () => {
+ test('a remote caller sees when the readable fact was said, and nothing of the restricted page', async () => {
+ const text = await call('query', { query: Q, expand: false, limit: 6, token_budget: 3000 }, true);
+ expect(text).toContain('[observed 2025-02-01; valid 2025-06-02 to unknown]');
+ expect(text).not.toContain('2025-03-01');
+ expect(text).not.toContain('Taos');
+ expect(text).not.toContain('fact_observed_date_unavailable');
+ });
+
+ test('a failed observation read renders observed unknown and marks the response degraded', async () => {
+ const text = await observationReadFails(() => call('query', { query: Q, expand: false, limit: 6, token_budget: 3000 }, true));
+ expect(text).toContain('[observed unknown; valid 2025-06-02 to unknown]');
+ expect(text).toContain('fact_observed_date_unavailable');
+ }, 60_000);
+});
+
+describe('recall date headers', () => {
+ test('a failed observation read reports facts_degraded instead of a silent unknown', async () => {
+ await engine.setConfig(EVIDENCE_DATE_HEADER_KEY, 'true');
+ try {
+ const ok = await call('recall', { entity: 'forge' }, false);
+ expect(ok).toContain('[observed 2025-02-01; valid 2025-06-02 to unknown]');
+ expect(ok).not.toContain('fact_observed_date_unavailable');
+ const failed = await observationReadFails(() => call('recall', { entity: 'forge' }, false));
+ expect(failed).toContain('[observed unknown; valid 2025-06-02 to unknown]');
+ expect(failed).toContain('fact_observed_date_unavailable');
+ } finally { await engine.setConfig(EVIDENCE_DATE_HEADER_KEY, 'false'); }
+ }, 60_000);
+});
diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.json
index ca0faa10fe..380abba5e1 100644
--- a/test/fixtures/goldens/exports/types.json
+++ b/test/fixtures/goldens/exports/types.json
@@ -86,7 +86,7 @@
"discoverOntologyDimensions: (opts?: { sourceId?: string; sourceIds?: string[]; }) => Promise<OntologyDimensionStat[]>",
"executeRaw: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; timeoutMs?: number; deadlineAt?: number; jitOff?: boolean; prepare?: boolean; }) => Promise<T[]>",
"executeRawDirect: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; }) => Promise<T[]>",
- "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; }) => Promise<boolean>",
+ "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date; }) => Promise<boolean>",
"findAnomalies: (opts: AnomaliesOpts) => Promise<AnomalyResult[]>",
"findByTitleFuzzy: (name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) => Promise<{ slug: string; similarity: number; } | null>",
"findCandidateDuplicates: (source_id: string, entitySlug: string, factText: string, opts?: { k?: number; embedding?: Float32Array; embeddingModel?: string | null; arm?: \"keyword\"; attributedTo?: FactAttribution | null; excludeMemberPages?: boolean; onlyPage?: string; }) => Promise<FactRow[]>",
@@ -496,7 +496,7 @@
"kind": "variable",
"name": "DEGRADED_STAGES",
"signature": [
- "readonly [\"embed_unavailable\", \"embed_timeout\", \"expansion_failed\", \"expansion_partial\", \"rescore_skipped\", \"vector_arm_failed\", \"keyword_arm_failed\", \"title_arm_failed\", \"budget_dropped_all\", \"budget_truncated\", \"keyword_zero\", \"cache_prestamp\", \"reranker_skipped\", \"rerank_passthrough\", \"rerank_failed\", \"keyword_relaxed_carried\", \"safe_index_pending\", \"vector_candidates_incomplete\", \"keyword_candidates_incomplete\", \"projection_pending\", \"projection_by_design\", \"projection_status_unknown\", \"keyword_count_unavailable\", \"saved_facts_withheld\"]"
+ "readonly [\"embed_unavailable\", \"embed_timeout\", \"expansion_failed\", \"expansion_partial\", \"rescore_skipped\", \"vector_arm_failed\", \"keyword_arm_failed\", \"title_arm_failed\", \"budget_dropped_all\", \"budget_truncated\", \"keyword_zero\", \"cache_prestamp\", \"reranker_skipped\", \"rerank_passthrough\", \"rerank_failed\", \"keyword_relaxed_carried\", \"safe_index_pending\", \"vector_candidates_incomplete\", \"keyword_candidates_incomplete\", \"projection_pending\", \"projection_by_design\", \"projection_status_unknown\", \"keyword_count_unavailable\", \"saved_facts_withheld\", \"fact_observed_date_unavailable\"]"
]
},
{
@@ -510,14 +510,14 @@
"kind": "type",
"name": "DegradedStage",
"signature": [
- "\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\""
+ "\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\""
]
},
{
"kind": "interface",
"members": [
"reason?: \"candidate_budget\" | \"iterative_scan_unavailable\" | \"timeout\" | \"provider_error\" | \"no_provider\" | \"variant_embed_failed\" | \"original_embed_failed\" | \"first_result_truncated\" | \"no_key\" | \"empty_result_set\" | \"malformed_shape\" | \"budget\" | \"rate_limited\" | \"unreachable\" | \"auth\" | \"egress_denied\" | \"embedding_disabled\" | \"skipped_after_timeout\" | undefined",
- "stage: \"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\""
+ "stage: \"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\""
],
"name": "DegradedStageEntry"
},
@@ -1102,7 +1102,7 @@
"discoverOntologyDimensions: (opts?: { sourceId?: string; sourceIds?: string[]; }) => Promise<OntologyDimensionStat[]>",
"executeRaw: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; timeoutMs?: number; deadlineAt?: number; jitOff?: boolean; prepare?: boolean; }) => Promise<T[]>",
"executeRawDirect: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; }) => Promise<T[]>",
- "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; }) => Promise<boolean>",
+ "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date; }) => Promise<boolean>",
"explainVectorSearch: (embedding: Float32Array, opts?: SearchOpts) => Promise<Record<string, unknown>>",
"findAnomalies: (opts: AnomaliesOpts) => Promise<AnomalyResult[]>",
"findByTitleFuzzy: (name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) => Promise<{ slug: string; similarity: number; } | null>",
@@ -1325,7 +1325,7 @@
"kind": "variable",
"name": "RANKING_ONLY_DEGRADED_STAGES",
"signature": [
- "ReadonlySet<\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\">"
+ "ReadonlySet<\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\">"
]
},
{
@@ -1525,7 +1525,7 @@
"exact_lookup?: \"title\" | \"slug\" | \"identifier\" | undefined",
"exact_match_boost?: number | undefined",
"fact_id?: string | undefined",
- "fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true; } | undefined",
+ "fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true; observed?: string; observed_unavailable?: true; } | undefined",
"feedback_boost?: number | undefined",
"follow_up?: { op: \"recall\"; args: Record<string, string>; } | undefined",
"fused_score?: number | undefined",
@@ -3659,7 +3659,7 @@
"discoverOntologyDimensions: (opts?: { sourceId?: string; sourceIds?: string[]; }) => Promise<OntologyDimensionStat[]>",
"executeRaw: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; timeoutMs?: number; deadlineAt?: number; jitOff?: boolean; prepare?: boolean; }) => Promise<T[]>",
"executeRawDirect: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; }) => Promise<T[]>",
- "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; }) => Promise<boolean>",
+ "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date; }) => Promise<boolean>",
"findAnomalies: (opts: AnomaliesOpts) => Promise<AnomalyResult[]>",
"findByTitleFuzzy: (name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) => Promise<{ slug: string; similarity: number; } | null>",
"findCandidateDuplicates: (source_id: string, entitySlug: string, factText: string, opts?: { k?: number; embedding?: Float32Array; embeddingModel?: string | null; arm?: \"keyword\"; attributedTo?: FactAttribution | null; excludeMemberPages?: boolean; onlyPage?: string; }) => Promise<FactRow[]>",
@@ -7124,7 +7124,7 @@
"discoverOntologyDimensions: (opts?: { sourceId?: string; sourceIds?: string[]; }) => Promise<OntologyDimensionStat[]>",
"executeRaw: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; }) => Promise<T[]>",
"executeRawDirect: <T = Record<string, unknown>>(sql: string, params?: unknown[], opts?: { signal?: AbortSignal; }) => Promise<T[]>",
- "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; }) => Promise<boolean>",
+ "expireFact: (id: number, opts?: { supersededBy?: number; at?: Date; successorFrom?: Date; }) => Promise<boolean>",
"findAnomalies: (opts: AnomaliesOpts) => Promise<AnomalyResult[]>",
"findByTitleFuzzy: (name: string, dirPrefix?: string, minSimilarity?: number, sourceId?: string) => Promise<{ slug: string; similarity: number; } | null>",
"findCandidateDuplicates: (source_id: string, entitySlug: string, factText: string, opts?: { k?: number; embedding?: Float32Array; embeddingModel?: string | null; arm?: \"keyword\"; attributedTo?: FactAttribution | null; excludeMemberPages?: boolean; onlyPage?: string; }) => Promise<FactRow[]>",
@@ -9606,7 +9606,7 @@
"kind": "variable",
"name": "DEGRADED_STAGES",
"signature": [
- "readonly [\"embed_unavailable\", \"embed_timeout\", \"expansion_failed\", \"expansion_partial\", \"rescore_skipped\", \"vector_arm_failed\", \"keyword_arm_failed\", \"title_arm_failed\", \"budget_dropped_all\", \"budget_truncated\", \"keyword_zero\", \"cache_prestamp\", \"reranker_skipped\", \"rerank_passthrough\", \"rerank_failed\", \"keyword_relaxed_carried\", \"safe_index_pending\", \"vector_candidates_incomplete\", \"keyword_candidates_incomplete\", \"projection_pending\", \"projection_by_design\", \"projection_status_unknown\", \"keyword_count_unavailable\", \"saved_facts_withheld\"]"
+ "readonly [\"embed_unavailable\", \"embed_timeout\", \"expansion_failed\", \"expansion_partial\", \"rescore_skipped\", \"vector_arm_failed\", \"keyword_arm_failed\", \"title_arm_failed\", \"budget_dropped_all\", \"budget_truncated\", \"keyword_zero\", \"cache_prestamp\", \"reranker_skipped\", \"rerank_passthrough\", \"rerank_failed\", \"keyword_relaxed_carried\", \"safe_index_pending\", \"vector_candidates_incomplete\", \"keyword_candidates_incomplete\", \"projection_pending\", \"projection_by_design\", \"projection_status_unknown\", \"keyword_count_unavailable\", \"saved_facts_withheld\", \"fact_observed_date_unavailable\"]"
]
},
{
@@ -9620,14 +9620,14 @@
"kind": "type",
"name": "DegradedStage",
"signature": [
- "\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\""
+ "\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\""
]
},
{
"kind": "interface",
"members": [
"reason?: \"candidate_budget\" | \"iterative_scan_unavailable\" | \"timeout\" | \"provider_error\" | \"no_provider\" | \"variant_embed_failed\" | \"original_embed_failed\" | \"first_result_truncated\" | \"no_key\" | \"empty_result_set\" | \"malformed_shape\" | \"budget\" | \"rate_limited\" | \"unreachable\" | \"auth\" | \"egress_denied\" | \"embedding_disabled\" | \"skipped_after_timeout\" | undefined",
- "stage: \"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\""
+ "stage: \"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\""
],
"name": "DegradedStageEntry"
},
@@ -10179,7 +10179,7 @@
"kind": "variable",
"name": "RANKING_ONLY_DEGRADED_STAGES",
"signature": [
- "ReadonlySet<\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\">"
+ "ReadonlySet<\"embed_unavailable\" | \"embed_timeout\" | \"expansion_failed\" | \"expansion_partial\" | \"rescore_skipped\" | \"vector_arm_failed\" | \"keyword_arm_failed\" | \"title_arm_failed\" | \"budget_dropped_all\" | \"budget_truncated\" | \"keyword_zero\" | \"cache_prestamp\" | \"reranker_skipped\" | \"rerank_passthrough\" | \"rerank_failed\" | \"keyword_relaxed_carried\" | \"safe_index_pending\" | \"vector_candidates_incomplete\" | \"keyword_candidates_incomplete\" | \"projection_pending\" | \"projection_by_design\" | \"projection_status_unknown\" | \"keyword_count_unavailable\" | \"saved_facts_withheld\" | \"fact_observed_date_unavailable\">"
]
},
{
@@ -10379,7 +10379,7 @@
"exact_lookup?: \"title\" | \"slug\" | \"identifier\" | undefined",
"exact_match_boost?: number | undefined",
"fact_id?: string | undefined",
- "fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true; } | undefined",
+ "fact_row?: { id: number; valid_from: string; valid_until: string | null; source?: string; from_quoted_page?: true; observed?: string; observed_unavailable?: true; } | undefined",
"feedback_boost?: number | undefined",
"follow_up?: { op: \"recall\"; args: Record<string, string>; } | undefined",
"fused_score?: number | undefined",
diff --git a/test/fixtures/goldens/sql-text/_driver.json b/test/fixtures/goldens/sql-text/_driver.json
index 092fa3cc74..253452addf 100644
--- a/test/fixtures/goldens/sql-text/_driver.json
+++ b/test/fixtures/goldens/sql-text/_driver.json
@@ -1711,7 +1711,7 @@
"expireFact": {
"default": [
{
- "paramCount": 3,
+ "paramCount": 7,
"unsafeOptions": {
"cancelFence": false,
"prepare": true,
@@ -1722,7 +1722,7 @@
],
"supersededBy": [
{
- "paramCount": 3,
+ "paramCount": 7,
"unsafeOptions": {
"cancelFence": false,
"prepare": true,
diff --git a/test/fixtures/goldens/sql-text/facts.json b/test/fixtures/goldens/sql-text/facts.json
index 7865450253..0625f8e1ed 100644
--- a/test/fixtures/goldens/sql-text/facts.json
+++ b/test/fixtures/goldens/sql-text/facts.json
@@ -83,10 +83,14 @@
"params": [
"Date",
"null",
+ "null",
+ "null",
+ "null",
+ "null",
"number"
],
- "sql": "UPDATE facts SET expired_at = $1, superseded_by = COALESCE($2, superseded_by) WHERE id = $3 AND expired_at IS NULL",
- "textSha256": "38221fd054ba68ba27987a96d101fc7b5b74a790b8b7c01dca3f2a8dd32c4629"
+ "sql": "UPDATE facts SET expired_at = $1, superseded_by = COALESCE($2, superseded_by), valid_until = CASE WHEN $3::timestamptz IS NULL THEN valid_until WHEN valid_from < $4::timestamptz AND $5::timestamptz < COALESCE(valid_until, now()) THEN $6::timestamptz ELSE GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now())) END WHERE id = $7 AND expired_at IS NULL",
+ "textSha256": "62c68fe82f53ef0e4cc71c1150b5ef1f7d1eff1f23df4631d923ccb512cfb75f"
}
],
"supersededBy": [
@@ -95,10 +99,14 @@
"params": [
"Date",
"number",
+ "null",
+ "null",
+ "null",
+ "null",
"number"
],
- "sql": "UPDATE facts SET expired_at = $1, superseded_by = COALESCE($2, superseded_by) WHERE id = $3 AND expired_at IS NULL",
- "textSha256": "38221fd054ba68ba27987a96d101fc7b5b74a790b8b7c01dca3f2a8dd32c4629"
+ "sql": "UPDATE facts SET expired_at = $1, superseded_by = COALESCE($2, superseded_by), valid_until = CASE WHEN $3::timestamptz IS NULL THEN valid_until WHEN valid_from < $4::timestamptz AND $5::timestamptz < COALESCE(valid_until, now()) THEN $6::timestamptz ELSE GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now())) END WHERE id = $7 AND expired_at IS NULL",
+ "textSha256": "62c68fe82f53ef0e4cc71c1150b5ef1f7d1eff1f23df4631d923ccb512cfb75f"
}
]
},
@@ -404,8 +412,8 @@
"Date",
"number"
],
- "sql": "UPDATE facts SET expired_at = now(), superseded_by = $1, valid_until = CASE WHEN valid_from < $2::timestamptz AND (valid_until IS NULL OR valid_until > $3::timestamptz) THEN $4::timestamptz ELSE valid_until END WHERE id = $5 AND expired_at IS NULL",
- "textSha256": "aa8e803c695469877ce7546bddfbd038658079b3d7dba01bdbc3acd2f5cfdeaa"
+ "sql": "UPDATE facts SET expired_at = now(), superseded_by = $1, valid_until = CASE WHEN valid_from < $2::timestamptz AND $3::timestamptz < COALESCE(valid_until, now()) THEN $4::timestamptz ELSE GREATEST(valid_from, LEAST(COALESCE(valid_until, now()), now())) END WHERE id = $5 AND expired_at IS NULL",
+ "textSha256": "96b77af3aec9877244bf76b54ad21c6c3f16617ffa3ea8ec42a037b7e9a77b2a"
},
{
"event": "commit",
diff --git a/test/remember-replaces-valid-until-7214.test.ts b/test/remember-replaces-valid-until-7214.test.ts
index 20eb767e97..8f05405fd3 100644
--- a/test/remember-replaces-valid-until-7214.test.ts
+++ b/test/remember-replaces-valid-until-7214.test.ts
@@ -3,7 +3,7 @@
* replacement starts after the retired fact, the retired row's valid_until is
* the replacement's valid_from, in the facts table and in the struck fence row,
* and a re-extraction of the page keeps it. A backdated replacement (starting
- * on or before the retired fact) leaves the interval as it was.
+ * on or before the retired fact) ends the interval now (GBRA-574 A3).
*/
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
@@ -65,23 +65,23 @@ for (const backend of testBackends()) {
expect(await validUntil(res.body.id)).toBeNull();
});
- test('a backdated replacement leaves the retired interval open', async () => {
+ test('a backdated replacement ends the retired interval now', async () => {
const old = await remember('The test charge is 100 units.', { valid_from: '2026-03-01' });
const res = await remember('The test charge is 200 units.', { valid_from: '2026-01-01', replaces: old.body.id });
expect(res.body.status).toBe('superseded');
- expect(await validUntil(old.body.id)).toBeNull();
+ expect(await validUntil(old.body.id)).toBe(new Date().toISOString().slice(0, 10));
});
});
describe('#7214 insertFact supersede closes the retired interval', () => {
- test('later replacement closes at its valid_from; backdated leaves it open', async () => {
+ test('later replacement closes at its valid_from; backdated ends it now', async () => {
const a = await engine.insertFact({ fact: 'The test charge is 100 units.', entity_slug: 'people/alice-example', source: 'test', valid_from: new Date('2026-01-01') }, { source_id: 'default' }); // gbrain-allow-direct-insert: #7214 probe of the engine supersede path
const b = await engine.insertFact({ fact: 'The test charge is 200 units.', entity_slug: 'people/alice-example', source: 'test', valid_from: new Date('2026-03-01') }, { source_id: 'default', supersedeId: a.id }); // gbrain-allow-direct-insert: #7214 probe of the engine supersede path
expect(b.status).toBe('superseded');
expect(await validUntil(String(a.id))).toBe('2026-03-01');
const c = await engine.insertFact({ fact: 'The test charge is 50 units.', entity_slug: 'people/alice-example', source: 'test', valid_from: new Date('2025-06-01') }, { source_id: 'default', supersedeId: b.id }); // gbrain-allow-direct-insert: #7214 probe of the engine supersede path
expect(c.status).toBe('superseded');
- expect(await validUntil(String(b.id))).toBeNull();
+ expect(await validUntil(String(b.id))).toBe(new Date().toISOString().slice(0, 10));
});
});
});
diff --git a/test/search-8k-conversion.test.ts b/test/search-8k-conversion.test.ts
index bec1a5a88e..419a1b6e9b 100644
--- a/test/search-8k-conversion.test.ts
+++ b/test/search-8k-conversion.test.ts
@@ -142,7 +142,7 @@ describe('search.query_facts_block', () => {
expect(facts.length).toBeGreaterThanOrEqual(2);
expect(facts.length).toBeLessThanOrEqual(MAX_FACTS_BLOCK_ROWS);
expect(rows.slice(0, facts.length).every(r => r.result_type === 'fact')).toBe(true);
- expect(facts[0]!.chunk_text.split('\n')[0]).toBe('[observed unknown; valid 2025-06-02 to unknown]');
+ expect(facts[0]!.chunk_text.split('\n')[0]).toBe('[observed 2025-05-03; valid 2025-06-02 to unknown]');
expect(facts.reduce((n, r) => n + resultTokens(r), 0)).toBeLessThanOrEqual(Math.floor(budget * QUERY_FACTS_BLOCK_SHARE));
});
diff --git a/test/supersession-valid-until-gbra574.test.ts b/test/supersession-valid-until-gbra574.test.ts
new file mode 100644
index 0000000000..a9f98dcb0a
--- /dev/null
+++ b/test/supersession-valid-until-gbra574.test.ts
@@ -0,0 +1,129 @@
+/**
+ * GBRA-574 A3: one rule ends a superseded fact's validity, whichever writer supersedes it (facts/supersede-interval.ts):
+ * at the replacement's valid_from when that falls inside the retired window, else at the retired fact's own earlier end
+ * or now. Future writes only: rows already stored keep their valid_until, and a withdrawn or forgotten fact's state is
+ * untouched. PGLite, no network.
+ */
+import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
+import { PGLiteEngine } from '../src/core/pglite-engine.ts';
+import { resetPgliteState } from './helpers/reset-pglite.ts';
+import { supersededValidUntil } from '../src/core/facts/supersede-interval.ts';
+import { insertProposal } from '../src/core/ai/decide/proposals-store.ts';
+import { applyProposalAction, planAcceptFence, type PairFact } from '../src/core/facts/proposal-supersede.ts';
+import { forgetFactInFence } from '../src/core/facts/forget.ts';
+import { parseFactsFence } from '../src/core/facts-fence.ts';
+
+let engine: PGLiteEngine;
+const NOW = new Date('2026-06-01T00:00:00Z');
+const at = (d: string) => new Date(`${d}T00:00:00Z`);
+const day = (v: Date | string | null) => (v === null ? null : new Date(v).toISOString().slice(0, 10));
+const add = async (fact: string, validFrom: string, opts: { supersedeId?: number; validUntil?: string } = {}) =>
+ (await engine.insertFact({ fact, entity_slug: 'people/alice-example', source: 'test', visibility: 'world', valid_from: at(validFrom), // gbrain-allow-direct-insert: GBRA-574 A3 probe of the engine supersede rule
+ ...(opts.validUntil ? { valid_until: at(opts.validUntil) } : {}) }, { source_id: 'default', ...(opts.supersedeId ? { supersedeId: opts.supersedeId } : {}) })).id;
+const row = async (id: number) => (await engine.executeRaw<{ valid_until: Date | string | null; expired_at: Date | string | null; superseded_by: number | null }>(
+ 'SELECT valid_until, expired_at, superseded_by FROM facts WHERE id = $1', [id]))[0]!;
+
+beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 120_000);
+afterAll(async () => { await engine?.disconnect(); }, 60_000);
+beforeEach(async () => { await resetPgliteState(engine); });
+
+describe('the rule', () => {
+ const retired = { valid_from: at('2025-01-10'), valid_until: null };
+ test('a replacement dated inside the retired window ends it there', () => {
+ expect(supersededValidUntil(retired, at('2025-02-15'), NOW)).toEqual(at('2025-02-15'));
+ });
+ test('a backdated replacement ends it now, never leaves it open', () => {
+ expect(supersededValidUntil(retired, at('2025-01-01'), NOW)).toEqual(NOW);
+ });
+ test('a replacement dated after now ends it now', () => {
+ expect(supersededValidUntil(retired, at('2026-09-01'), NOW)).toEqual(NOW);
+ });
+ test('a retired fact that already ended earlier keeps its end', () => {
+ expect(supersededValidUntil({ valid_from: at('2025-01-10'), valid_until: at('2025-01-20') }, at('2025-02-15'), NOW)).toEqual(at('2025-01-20'));
+ expect(supersededValidUntil({ valid_from: at('2025-01-10'), valid_until: at('2025-03-01') }, at('2025-02-15'), NOW)).toEqual(at('2025-02-15'));
+ });
+ test('no replacement date: its own earlier end or now', () => {
+ expect(supersededValidUntil(retired, null, NOW)).toEqual(NOW);
+ });
+ test('a retired fact dated after now never ends before it began', () => {
+ const future = { valid_from: at('2026-09-01'), valid_until: null };
+ expect(supersededValidUntil(future, at('2026-10-01'), NOW)).toEqual(at('2026-09-01'));
+ expect(supersededValidUntil(future, at('2025-01-01'), NOW)).toEqual(at('2026-09-01'));
+ expect(supersededValidUntil(future, null, NOW)).toEqual(at('2026-09-01'));
+ });
+});
+
+describe('engine insertFact with supersedeId', () => {
+ test('ends the retired fact at the replacement date inside its window, else now', async () => {
+ const old = await add('Bob Example works at Beta Corp.', '2025-01-10');
+ await add('Bob Example works at Gamma Labs.', '2025-02-15', { supersedeId: old });
+ expect(day((await row(old)).valid_until)).toBe('2025-02-15');
+ const later = await add('Carol Example lives in Porto.', '2025-03-01');
+ const before = Date.now();
+ await add('Carol Example lives in Lisbon.', '2025-01-01', { supersedeId: later });
+ const until = (await row(later)).valid_until;
+ expect(until).not.toBeNull();
+ expect(new Date(until!).getTime()).toBeGreaterThanOrEqual(before - 1000);
+ });
+});
+
+describe('a retired fact whose valid_from is in the future', () => {
+ test('insertFact and expireFact end it no earlier than its own valid_from', async () => {
+ const begins = (id: number) => engine.executeRaw<{ valid_from: Date | string }>('SELECT valid_from FROM facts WHERE id = $1', [id]).then(r => new Date(r[0]!.valid_from).getTime());
+ const a = await add('Dana Example starts at Beta Corp.', '2099-01-01');
+ await add('Dana Example starts at Gamma Labs.', '2025-01-01', { supersedeId: a });
+ expect(new Date((await row(a)).valid_until!).getTime()).toBeGreaterThanOrEqual(await begins(a));
+ const b = await add('Dana Example moves to Porto.', '2099-01-01');
+ expect(await engine.expireFact(b, { successorFrom: at('2025-01-01') })).toBe(true);
+ expect(new Date((await row(b)).valid_until!).getTime()).toBeGreaterThanOrEqual(await begins(b));
+ });
+});
+
+describe('engine expireFact', () => {
+ test('with the successor date it ends the row by the rule; a plain expiry keeps valid_until as stored', async () => {
+ const a = await add('Bob Example works at Beta Corp.', '2025-01-10');
+ expect(await engine.expireFact(a, { successorFrom: at('2025-02-15') })).toBe(true);
+ expect(day((await row(a)).valid_until)).toBe('2025-02-15');
+ const b = await add('Bob Example works at Delta Inc.', '2025-01-10');
+ expect(await engine.expireFact(b)).toBe(true);
+ expect((await row(b)).valid_until).toBeNull();
+ });
+});
+
+describe('proposal accept', () => {
+ test('ends the old fact at the newer fact\'s valid_from, in the row and in the struck fence row', async () => {
+ const old = await add('Bob Example works at Beta Corp.', '2025-01-10');
+ const fresh = await add('Bob Example works at Gamma Labs.', '2025-02-15');
+ const id = (await insertProposal(engine, { source_id: 'default', sweep_id: 's', pair_index: 0, new_fact_id: fresh, old_fact_id: old,
+ direction: 'new_supersedes_old', p_supersede: 0.8, threshold: 0.8, proposal_floor: 0.5, model_resolved: 'jev-test' }))!;
+ expect((await applyProposalAction(engine, id, 'accept')).status).toBe('accepted');
+ expect(await row(old)).toMatchObject({ superseded_by: fresh });
+ expect(day((await row(old)).valid_until)).toBe('2025-02-15');
+ const pair = (f: Partial<PairFact>): PairFact => ({ id: 1, source_id: 'default', entity_slug: 'people/alice-example', visibility: 'world', fact: 'x', expired_at: null,
+ valid_until: null, superseded_by: null, row_num: 1, source_markdown_slug: 'people/alice-example', ...f });
+ const body = '<!--- gbrain:facts:begin -->\n| # | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |\n|---|-------|------|------------|------------|------------|------------|-------------|--------|---------|\n| 1 | Bob Example works at Beta Corp. | fact | 1.0 | world | medium | 2025-01-10 | | test | |\n<!--- gbrain:facts:end -->\n';
+ const plan = planAcceptFence(body, pair({ valid_from: '2025-01-10T00:00:00.000Z' }), pair({ id: 2, row_num: null, valid_from: '2025-02-15T00:00:00.000Z' }), '2026-06-01');
+ expect(parseFactsFence(plan!.body).facts[0]).toMatchObject({ active: false, validUntil: '2025-02-15' });
+ });
+});
+
+describe('future writes only; withdrawals untouched', () => {
+ test('a row superseded before this change keeps its stored valid_until', async () => {
+ const old = await add('Bob Example works at Beta Corp.', '2025-01-10');
+ const fresh = await add('Bob Example works at Gamma Labs.', '2025-02-15');
+ await engine.executeRaw('UPDATE facts SET expired_at = now(), superseded_by = $2, valid_until = NULL WHERE id = $1', [old, fresh]);
+ await add('Bob Example works at Delta Inc.', '2025-03-01', { supersedeId: fresh });
+ expect((await row(old)).valid_until).toBeNull();
+ });
+ test('a forgotten fact is withdrawn as before: expired, no successor, ended at the withdrawal', async () => {
+ const id = await add('Bob Example works at Beta Corp.', '2025-01-10', { validUntil: '2030-01-01' });
+ const before = Date.now();
+ await forgetFactInFence(engine, id);
+ const r = await row(id);
+ expect(r.expired_at).not.toBeNull();
+ expect(r.superseded_by).toBeNull();
+ expect(new Date(r.valid_until!).getTime()).toBeGreaterThanOrEqual(before - 1000);
+ expect(new Date(r.valid_until!).getTime()).toBeLessThanOrEqual(Date.now() + 1000);
+ expect((await engine.executeRaw('SELECT 1 FROM fact_withdrawals')).length).toBe(1);
+ });
+});