Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
26.0 KB
Raw Blame

Answer trust: verified answer-time quotes, statement-level claim attribution, durable raw-source archives, and explicit no-session identity semantics

Status: draft Author: design orchestrator (feature-release-playbook, phases 0-1) Audience: implementation

Problem statement

An operator reading an Open Second Brain answer today cannot tell whether the quotation marks in it were earned. Recall injection puts vault notes into agent context (hooks/recall-inject.ts:6, opt-in via recall_inject_enabled, resolved at src/core/config.ts:1185-1188), the host agent answers with quoted spans drawn from those notes, and nothing checks that a quoted string actually occurs in the note it is cited against. The comparison machinery already exists and is pure: the distill lane has verified every quotation on a distillation page since v1.67.0 (src/core/brain/distill/quote-spans.ts:173 findQuoteSpans, :278 unquoteSpans, :410 normalizeForQuoteComparison, :466 quoteHaystack, :486 spanOccursIn, with the verdict vocabulary in src/core/brain/distill/quote-verdict.ts). The gap is the answer boundary, and Open Second Brain owns both of its ends: what enters context (the injection ledger hooks/lib/injection-ledger.ts:23-36 records injected notes per phase, 24 h TTL, 2000-entry set cap) and turn-end observation (the Stop hook family at hooks/hooks.json:112, the additionalContext precedent in hooks/stop-log-guardrail.ts:10, and transcript_path on hook stdin at hooks/lib/stdin.ts:33).

Stated claims carry no pointer to the recall they drew on. The substrate for such a pointer is complete: context receipts name every injected item (src/core/brain/context-receipts.ts:120-135), keep only a hash of the final answer (:139), and a durable receiptId rides the context-pack report (src/core/brain/context-pack.ts:929-955, surfaced as receipt_id at src/mcp/brain/pack-tools.ts:403); outcome rows are posted per pack under a sample_id key with first-pass-versus-repair verdicts (src/core/brain/context-pack-outcome.ts:164-259). But the claim ledger's ClaimEvent (src/core/brain/truth/types.ts:50-71) has neither field, brain_truth state (src/mcp/brain/knowledge-tools.ts:751-812, persisted through appendStatedClaims at src/core/brain/truth/stated-claims.ts:158-235) freezes claims with no linkage, and receipts being hash-only means Open Second Brain never sees answer text - so any attribution must be threaded at write time from the caller that does, and joined at read time.

Provenance rots because intake throws the bytes away. Intake commits stated-level provenance - source references only (src/mcp/brain/intake-args.ts:184, the rebuilt form at src/core/brain/ingest/ingest.ts:265) - through a classifier that reads a source and keeps only trust plus hash (classifySourceOrigin, src/core/brain/intake/source-trust.ts:418-420), while the byte-keeping form readSourceOrigin (:430-458) has exactly one caller, in distill intake (src/core/brain/intake/extract-intake.ts:322). The artifact store is 24 h-TTL scratch (src/mcp/artifact-store.ts:110), and the distill capture scope keeps a 64 KiB excerpt plus digest on the page (src/core/brain/provenance/capture-scope.ts:83, :165) - so a full-local source whose file decays falls back to url-only at the next sweep with no bytes behind it, which is precisely the condition the capture-scope hygiene detector names and nothing can repair. A trusted source whose digest is refused to remote readers (src/core/brain/intake/extract-intake.ts:332-336) shows the rule the archive must inherit: bytes persist only under the visibility rules, never past them.

Session identity is a coercion bug with a grouping symptom. Writers stamp session_id conditionally - lanes with no session in scope write records with the key absent (src/core/brain/recall-telemetry.ts:453, src/core/brain/gate-telemetry.ts:76,127, src/core/brain/observed-use.ts:120, src/core/brain/recent-turns.ts:96-97, src/core/brain/token-impact.ts:279,321, src/core/brain/context-pack-evidence.ts:282) - and five readers coerce the absence to the empty string with String(payload["session_id"] ?? "") (src/core/brain/session-recall.ts:577, src/core/brain/session-summary.ts:243,339,375, src/core/brain/idea-lineage.ts:120,198). The effect cuts both ways: scoped recall silently drops unattributed records (the lineage ledger already omits blank declarations, src/core/brain/session-recall.ts:619-627, so the scope set can never contain ""), while unscoped grouping lumps every unattributed record from every unrelated lane into one shared "" bucket. The repo has already answered the semantic question once: session:unknown "is a statement that there was no session, not a pointer to a missing one" (src/core/brain/doctor/orphan-session-check.ts:37-40).

Scope

Four cards and one repair lane, one PR on feat/answer-trust:

  • Answer-time quote verification (t_245378f9) - a new Stop-hook step that walks the last assistant messages of the turn, finds quoted spans with the existing distill helpers (imported, never forked), resolves each span's cited note (a wikilink or path in the span's context, else membership in the injection ledger), and checks it with a new pure module src/core/brain/answer-quotes/check.ts over verdicts verified | not-in-note | source-truncated | note-unavailable | not-text. One additionalContext annotation per turn lists unverified spans. Never blocks, never edits transcripts, fail-open exit 0. Config: quote_check.recall|active|subagent, each off|shadow|enforce, default shadow (annotate-only), scoped to phases whose injection is enabled.
  • Statement-level claim attribution (t_2ffffd28) - an optional sample_id on the claim path: brain_truth state accepts it, the ClaimEvent freezes it (optional field, schema version unchanged, omit-don't-invent), and a new read-only join module src/core/brain/truth/claim-provenance.ts resolves a claim's sample_id to its receipt items and outcome row at read time. Zero new answer-time writes; claims whose outcome never arrived render "receipt known, outcome unknown" and never an invented verdict.
  • Durable raw-source archive (t_9b835c8f) - a new content-addressed store src/core/brain/provenance/source-archive.ts at Brain/.archives/<sha256>.txt: the ingest trusted-local branch reads the source once (readSourceOrigin-style), redacts, refuses past 1 MiB rather than truncating, persists, and stamps source_archive_hash (plus source_archive_bytes) on the summary page on success only. No TTL - evidence, not scratch. The capture-scope hygiene detector gains a missing/mismatched-archive finding, delete-by-source cleanup gains an archive-entry kind, and an archive_get read on the existing brain_hygiene tool (modeled on brain_artifact_get, local reach only) fetches bytes.
  • Blank session identity (t_22d58fa7) - absence stays absence: the five coercion sites treat a missing or non-string session_id as "no session" explicitly, scoped recall excludes such records, and grouping sites skip or emit undefined rather than lumping under "". A new count-only, fail-soft doctor check blank-session-identity reports one issue per affected kind with a count. No payload rewrites, no repair verb.
  • Repair lane (post-merge external review, verified) - M1: a freshness gate on stale-index attribute narrowing so a document whose indexed snapshot no longer matches its file falls through to the live post-rank filter (src/core/search/property-pushdown.ts:303-321, applied :386, against the advertised "judged live" semantics at src/mcp/search-tools.ts:293). L12: CHANGELOG whitespace damage at CHANGELOG.md:264 and CHANGELOG.md:2450, docs-only.

Out of scope

  • Distill-lane changes. The distill quote check (src/core/brain/distill/quote-check.ts) and its capture scope stay untouched; checkClaimQuotes is not extended and quote-spans.ts is consumed in place by import.
  • Paraphrase and semantic checks. The verifier is literal-only: a span occurs in the note's normalized haystack or it does not. Semantic entailment is a different, larger card.
  • Transcript rewriting. The hook annotates; it never strips quotation marks from an answer the host wrote, unlike the distill lane which strips on its own pages.
  • Blocking refusals. No Stop hook in this repo blocks (tests/hooks/hooks-json-shape.test.ts pins the exit 0 shape), and enforce in this wave means annotate plus one durable Brain-log record, never a blocked answer.
  • Per-statement provenance rows (rejected). One continuity row per statement duplicates the claim ledger, grows the append-only continuity store without bound, and is unfillable anyway - receipts keep the answer hash-only, so Open Second Brain never sees the text a row would quote.
  • Remote URL fetch and archival. Url-only sources keep being served by distill's 64 KiB excerpt; archiving remote content is a fetch-trust design of its own.
  • Session payload rewrites and a repair verb. The continuity store is append-only per-device shards; in-place edits and a backup-gated repair posture are t_6cc80627's territory. The doctor check this wave is informational and count-only, with no fix field.
  • A unified wave-wide phase ladder (rejected). One off|shadow|enforce dial across all four cards conflates independent failure domains against the per-feature resolver house rule, excludes D on day one (its decision is semantic and fixed, not a posture), and would demand the largest config and test surface before any card ships value.
  • Review items L3 (unused taxonomy indexes) and L15 (no hallways.json consumer). Both are measurement and design questions, not repairs; they wait for a wave that measures first.

Chosen approach

Variant 1, independent lanes at natural seams. Each card lands exactly where its verified premises point: A as a Stop-hook step beside stop-log-guardrail plus a pure check module; B as an additive field on the claim path plus a read-only join; C as a new provenance module wired into the one branch of ingest that already reads trusted local files; D as reader-site semantics plus one registered doctor check; the repair lane as two independent fixes. No shared machinery is introduced beyond the vault layout itself. The hot path - the answer the host produces - is touched by exactly one component, the A hook step, which is bounded (named constants for characters and wall clock) and fail-open. Config stays per-feature (quote_check.* resolvers beside resolveRecallInjectEnabled, src/core/config.ts:1185), doctor checks register individually in DOCTOR_CHECKS (src/core/brain/doctor.ts:174-201), and each card carries its own census test. Wave coherence is editorial, not structural, and that is accepted deliberately: both of the wave's additive schema fields - B's optional sample_id and C's source_archive_hash stamp - are exactly the shape a future provenance home would adopt without rework.

The rejected alternatives are rejected for cause, not taste. Variant 2 (a provenance spine fusing B and C under one integrity contract) buys a literal claim-to-bytes chain but couples two lanes that the parallel process needs disjoint, forces C's format to land before B's join consumers, and widens blast radius for a chain whose only consumer this wave is doctor and CLI reporting. Variant 3 (one wave-wide answer_trust phase ladder) trades the per-feature resolver house rule for one operational dial, conflates independent failure domains, and needs an exception for D immediately.

Premise verification

Every card premise was re-verified against the branch point a00fa3f5 (v1.80.0). The table records what changed and what held; anchors below are live as of this design.

Card Premise as mined Verdict Live evidence
t_245378f9 Validator anchor src/core/brain/host-facts.ts:710 Refuted No such file exists; the live file is src/core/runtime/host-facts.ts, which attaches citations to built-in host facts and does no quote verification. The gap statement survives; the anchor was never real.
t_245378f9 Distill quote-check exists and its helpers are pure-reusable Confirmed src/core/brain/distill/quote-spans.ts:173,278,410,466,486; verdict vocabulary src/core/brain/distill/quote-verdict.ts; shipped v1.67.0.
t_245378f9 "On by default" Corrected Recall injection itself is opt-in (hooks/recall-inject.ts:6, src/core/config.ts:1188), so default-on can only scope to phases whose injection the operator enabled. The card's intent lands as per-phase shadow default.
t_2ffffd28 Receipts and outcomes are complete; the claim ledger has no linkage Confirmed src/core/brain/context-receipts.ts:120-135,139; src/core/brain/context-pack.ts:929-955; src/mcp/brain/pack-tools.ts:403,459; src/core/brain/context-pack-outcome.ts:164-259; src/core/brain/truth/types.ts:50-71 (no session_id/sample_id).
t_2ffffd28 Card's eval anchors (retrieval-trail.ts:227,346, search-tools.ts:1786) Stale Live anchors are src/core/search/retrieval-trail.ts:323,504 and brain_eval at src/mcp/search-tools.ts:2003. The gap the card names survives unchanged.
t_9b835c8f Intake commits stated-level provenance and discards bytes; the artifact store is ephemeral Confirmed src/mcp/brain/intake-args.ts:184; src/core/brain/ingest/ingest.ts:265; byte-discarding classifySourceOrigin at src/core/brain/intake/source-trust.ts:418-420 vs byte-keeping readSourceOrigin at :430-458 (only distill calls it, src/core/brain/intake/extract-intake.ts:322); TTL prune at src/mcp/artifact-store.ts:110; 64 KiB excerpt at src/core/brain/provenance/capture-scope.ts:83.
t_22d58fa7 Blank coercion lives at src/core/brain/session-recall.ts:470 Stale The coercion moved: it now lives at src/core/brain/session-recall.ts:577, in the read-path scope filter. The defect is unchanged.
t_22d58fa7 resolveSessionScope throws on blank; the lineage ledger omits blank ids; no session table exists Confirmed src/core/brain/session-scope.ts:35-39; src/core/brain/session-recall.ts:619-627; continuity is an append-only JSONL store, so there is no session row to rewrite - the card's migration subject never existed.

Design decisions

  1. The quote check is per-phase, shadow by default, and scoped to phases whose injection is enabled. Three keys - quote_check.recall, quote_check.active, quote_check.subagent in the quote_check block of Brain/_brain.yaml, each off|shadow|enforce, default shadow - each with its own resolver in src/core/config.ts (resolveAnswerQuoteCheckRecall|Active|Subagent), following both the per-feature resolver convention (resolveRecallInjectEnabled, src/core/config.ts:1185) and the decision-model per-use off|shadow|enforce switch (src/core/decision-model/config-types.ts:62-85, answerable.ts:12). The card's "on by default" becomes shadow-by-default for a reason the premise verification surfaced: injection itself is opt-in (hooks/recall-inject.ts:6), so a default that changed answer behavior for vaults that never enabled any injection would be a default-changing flag wearing a default-preserving name. shadow annotates only; enforce annotates and appends one Brain-log event per turn with unverified spans (via appendLogEvent, src/core/brain/log.ts:289) so findings survive the session - and never blocks.
  2. The injection ledger is ground truth for what was available, and degradation never upgrades a verdict. A span is checked only against evidence the turn actually had: the note named in the span's context (wikilink or vault-relative path), else membership in the ledger's phase keys (LEDGER_KEY_RECALL|ACTIVE|REGROUND|SUBAGENT_INJECT, hooks/lib/injection-ledger.ts:23-26). A note that cannot be read yields note-unavailable with the annotation kept - "we could not check" is a finding, not an all-clear. A note whose body reached context truncated (ContextPackItem.trimmed, src/core/brain/context-pack.ts:156; receipt truncated flags at src/core/brain/context-receipts.ts:336,447) is matched against the available prefix, and a miss is source-truncated, never not-in-note - a truncated source can under-deliver evidence but never acquits a span.
  3. sample_id is an optional ClaimEvent field, frozen at write, joined at read. truth/store.ts extends AppendClaimInput/ClaimEvent with optional sample_id - omitted when absent, never invented - leaving the schema version unchanged exactly as the optional extractor tag did (src/core/brain/truth/types.ts:64-69); truth/stated-claims.ts accepts an optional sampleId on StatedClaimsPayload that must be trimmed and non-empty when present. The new truth/claim-provenance.ts is read-only and composes existing readers: receipt items by receipt id (listContextReceipts, src/mcp/brain/pack-tools.ts:459) and outcome rows by sample_id (listContextPackOutcomes, src/core/brain/context-pack-outcome.ts:293-305). The correlation guard is omit-don't-invent both ways: a claim with a sample_id and no outcome row renders "receipt known, outcome unknown"; a claim timestamped before its claimed receipt is flagged as an inversion rather than silently joined.
  4. The source archive is content-addressed, redacted before persist, refused not truncated, and evidence rather than scratch. put(vault, bytes) -> sha256 writes Brain/.archives/<sha256>.txt - a dot-directory excluded from the walker exactly like .snapshots and .artifacts (src/core/brain/path-constants.ts:109-116 precedent), which is also the dedup: identical bytes persist once. ARCHIVE_MAX_BYTES is 1 MiB, reusing the PRE_EXTRACT_MAX_SOURCE_BYTES ceiling (src/core/brain/ingest/ingest.ts:437) with the capture-scope refusal posture (src/core/brain/provenance/capture-scope.ts:243-248) - an oversized source is not captured, and absence on the page still means exactly "not captured". Redaction runs redactRawOutput (src/core/redactor.ts:971) before persist so credentials captured in v1.78-v1.79 terms never land in a second file, and visibility tokens hold: a source whose digest is refused to remote readers (src/core/brain/intake/extract-intake.ts:332-336) is never archived, and remote reach never reads archived bytes. No TTL - this is evidence, the direct contrast to the artifact store's prune (src/mcp/artifact-store.ts:110); retention is explicit operator pruning, and delete-by-source cleanup gains an archive-entry kind so orphaned archives do not accumulate (src/mcp/brain/ingest-tools.ts:218-260).
  5. Absence stays absence for session identity. The five coercion sites (src/core/brain/session-recall.ts:577, src/core/brain/session-summary.ts:243,339,375, src/core/brain/idea-lineage.ts:120,198) treat a missing or non-string session_id as no-session explicitly - no String(... ?? ""), no fabricated bucket - so scoped recall excludes the record (matching what the lineage ledger already does, src/core/brain/session-recall.ts:619-627) and grouping sites skip rather than lump unrelated lanes into one shared "" group. This is the orphan-session precedent applied at the reader (src/core/brain/doctor/orphan-session-check.ts:37-40): mapping absence to "" manufactured a phantom session that no writer ever created. The public-ish digest shape changes deliberately: SessionSummaryDigest.sessionId becomes presence-gated (key omitted when absent), and its consumers are swept in the same task.
  6. Doctor checks this wave are informational and count-only. blank-session-identity is a fail-soft check in src/core/brain/doctor/session-identity-check.ts: one pass over listContinuityRecords, one issue per affected record kind with a count, no fix field - repair belongs to t_6cc80627's backup-gated posture, and an in-place rewrite of append-only per-device shards is out of scope by the store's own contract. It registers in DOCTOR_CHECKS adjacent to orphanSessionCheck (src/core/brain/doctor.ts:174-201). The same informational posture governs C's surfaces: the hygiene finding names the page, and the doctor CLI prints counts.
  7. Stale-index narrowing gets a freshness gate instead of weaker narrowing. The EXISTS ... OR NOT EXISTS fragment (src/core/search/property-pushdown.ts:303-321, applied at :386) narrows on the attribute snapshot while the contract says values are "narrow on the attribute snapshot, judged live" (src/mcp/search-tools.ts:293). The gate compares the candidate's indexed snapshot against the live document and lets stale candidates fall through to the existing live post-rank filter, so a draft-then-done document stops vanishing from status=done queries. No new store machinery: the check rides the index rows store.ts already maintains.

File changes

New core: src/core/brain/answer-quotes/check.ts (pure quote verdicts over distill helpers), src/core/brain/provenance/source-archive.ts (content-addressed put/verify/stats, redaction, 1 MiB refusal), src/core/brain/truth/claim-provenance.ts (read-only claim-to-receipt-to-outcome join), src/core/brain/doctor/session-identity-check.ts (count-only blank-session check), hooks/answer-quote-check.ts (bounded, fail-open Stop-hook step). Extended core: src/core/brain/truth/store.ts + truth/stated-claims.ts (optional sample_id, trim-and-refuse validation), src/core/brain/ingest/ingest.ts (trusted-branch archive capture, page stamp on success only), src/core/brain/session-recall.ts, session-summary.ts, idea-lineage.ts (explicit no-session semantics at the five coercion sites), src/core/brain/doctor.ts (blank-session-identity registration), src/core/brain/path-constants.ts (BRAIN_ARCHIVES_DIR/BRAIN_ARCHIVES_REL), src/core/state/surfaces.ts (.archives row), src/core/config.ts (quote_check.* resolvers), src/core/search/property-pushdown.ts + src/core/search/store.ts (freshness gate), src/core/brain/hygiene/detectors/capture-scope.ts (missing/mismatched-archive finding). MCP: src/mcp/brain/knowledge-tools.ts (brain_truth state accepts and echoes sample_id; read op hydrates provenance), src/mcp/brain/ingest-tools.ts (source_archive_hash on the ingest result; archive-entry kind in delete-by-source cleanup), src/mcp/brain/hygiene-tools.ts (archive_get op, local reach). Hooks + CLI: hooks/hooks.json (one Stop entry after stop-log-guardrail), src/cli/brain/verbs/truth.ts (state --sample-id, claim-read hydration), src/cli/brain/verbs/doctor.ts (informational archive section). Tests: tests/core/brain/answer-quotes/check.test.ts, tests/core/config-answer-quote-check.test.ts, tests/hooks/answer-quote-check.test.ts, tests/core/brain/truth/claim-provenance.test.ts plus the store/stated-claims suite extensions, tests/core/brain/provenance/source-archive.test.ts, tests/core/brain/session-identity-check.test.ts, tests/core/architecture/claim-attribution-census.test.ts, and the extensions the plan names per task (tests/mcp/ingest-tool.test.ts, tests/mcp/hygiene-capture-scope-reach.test.ts, tests/cli/brain-hygiene.test.ts, tests/core/search/kind-attributes-pushdown.test.ts, tests/hooks/hooks-json-shape.test.ts). Docs: docs/observability.md (quote-check annotation and the blank-session count), CHANGELOG.md under the release heading plus the L12 whitespace repair.

Risks and open questions

  • Stop-hook latency bounds. The check runs inside the turn-end Stop pipeline (hooks/hooks.json:112) where timeout: 10 already governs; the step reads only the last assistant messages under named character and wall-clock constants and gives up by returning verified-nothing rather than by running long. The risk to watch in review is a transcript reader that scans the whole file - the bounded-window read is the acceptance criterion, not an implementation detail.
  • False-positive quote normalization residuals. The distill helpers' normalization (normalizeForQuoteComparison, src/core/brain/distill/quote-spans.ts:410) is tuned for distillation pages; answers may paraphrase whitespace, ligatures or punctuation differently. The upstream card itself reports non-zero false-positive rates, which is exactly why the default is shadow: the annotation wording says "unverified", not "fabricated", and the mirror suite pins that supported quotes pass at least as well as the distill lane's own suite (tests/core/brain/distill/quote-check.test.ts is the bar).
  • Vault bloat and secrets in raw captures. Brain/.archives/ is the first durable byte store with no TTL; a chatty ingest lane grows it without bound, and a redaction gap would duplicate a secret into a second file after v1.78-v1.79 taught the rest of the tree to hold credentials in one custody store. The 1 MiB refuse-not-truncate ceiling and the redactRawOutput-before-persist ordering are the mitigations; the doctor's informational archive section is the operator's visibility into growth.
  • The session-summary digest shape has consumers. SessionSummaryDigest.sessionId: "" (src/core/brain/session-summary.ts:375) reaches src/cli/brain/verbs/session-summary.ts and src/mcp/brain/synthesis-tools.ts; flipping to presence-gated is swept in the same task, and a consumer that cannot tolerate the absent key stops the lane for a design check rather than papering over it with a fallback.
  • Correlation quality when outcomes never arrive. sample_id is caller-supplied and optional; a caller that mints ids without ever posting the outcome row (postContextPackOutcome, src/core/brain/context-pack-outcome.ts:164-259) leaves every claim at "receipt known, outcome unknown" forever. The join renders that honestly and flags timestamp inversions, but the wave does not add a write-time check that the id corresponds to a real sample - the caller's contract, documented on the tool.
  • Freshness-gate performance. The gate adds a per-candidate liveness comparison inside attribute narrowing (src/core/search/property-pushdown.ts:386); on a large index with narrow filters this must not turn a SQL-side filter into a per-row file read. The gate rides the snapshot metadata the store already indexes, and the extended suite pins both the correctness case (draft-to-done flips the result) and the no-regression case (fresh candidates never touch the filesystem).