Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat(brain): retrieval precision and quality loop (v1.22.0) (#118)

Sol Aitkena98bed1dab01
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 567c145e..e9c7b95a 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.21.0",
+ "version": "1.22.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index f2783853..504d17ef 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.21.0",
+ "version": "1.22.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"skills": "./skills",
"hooks": "./hooks/hooks.json",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 71f494bd..23a7423c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,60 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.22.0] - 2026-07-05
+
+A retrieval-precision and quality-loop pass. Four new surfaces close the
+loop between what recall surfaces, how honestly it reports its own
+grounding, and how observed failures turn into a prioritized backlog.
+Each new field is optional and absent by default, so every call site
+whose caller does not opt in produces byte-identical output, and the
+kernel still calls no LLM.
+
+### Added
+
+- **Epistemic provenance markers on packed and recalled items.** Every
+ item emitted by `brain_context_pack`, `brain_context_receipts`, and
+ recall results now carries a structural `epistemic` status
+ (`observed` | `derived` | `hypothesis` | `plan` | `unknown`) plus
+ `evidence_refs`, derived from existing graph metadata with no manual
+ tagging and no natural-language word lists. A consuming model can now
+ tell a source-backed fact from a conjecture or a plan. The status is
+ language-agnostic by construction; fields are absent when the status
+ is `unknown`, so the off-path is byte-identical to today.
+- **Recall adequacy verdict with explicit low-adequacy actions.** A thin
+ verdict layer classifies each recall attempt as `sufficient`, `weak`,
+ or `insufficient` and emits a recommended action (`proceed`,
+ `re_recall`, `abstain`) with an optional `escalate` flag, reusing the
+ existing gate-telemetry relevance scores plus the new epistemic mix.
+ The verdict appears in `brain_recall_gate` output and the
+ context-receipts summary when the caller passes recall scores;
+ otherwise the field is absent and output is unchanged. Thresholds are
+ configurable (`recall_adequacy_sufficient`, `recall_adequacy_weak`,
+ `recall_adequacy_min_results`).
+- **Cross-query demand log surfacing recurring poorly-answered queries.**
+ A rolling, size-capped demand log under `Brain/log/query-demand.jsonl`
+ records each recall (normalized terms plus timestamp, result count,
+ and the existing coverage score) and an aggregation reader ranks
+ recurring queries the vault answers poorly, ranked by frequency
+ times (1 minus IDF-weighted coverage). Surfaced read-only as the
+ `brain_knowledge_gaps` MCP tool and `o2b brain knowledge-gaps` CLI.
+ The log is written only when recall gate telemetry is opt-in (default
+ off) and queries are normalized before append; the reader is always
+ available and never writes.
+- **Unified outcome-tagged lessons loop.** A `Brain/lessons.md` digest
+ folds positive knowledge (preferences and their applied evidence) and
+ negative knowledge (`violated` / `outdated` rows and `Brain/dead-ends/`)
+ into one outcome-tagged corpus scored by a signed, recency-decayed
+ weight (shared 30-day half-life from working-memory continuity). Each
+ lesson lands in a tier: `preferred` (net-positive and corroborated by
+ at least N distinct results), `tentative` (net-positive but under the
+ corroboration floor), `contested` (both positive and negative recent
+ evidence, recency-wins), or `avoid` (dead-ends and net-negative). The
+ digest is regenerated at the tail of every `dream` pass alongside
+ `active.md`, exposed as the `osb://lessons` MCP resource, and injected
+ on SessionStart and PostCompact. Tunable via `lessons.limit`,
+ `lessons.half_life_days`, and `lessons.corroboration_min`.
+
## [1.21.0] - 2026-07-04
A safety-hardening pass around the surfaces that write to or trust the
@@ -6118,6 +6172,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[1.22.0]: https://github.com/itechmeat/open-second-brain/compare/v1.21.0...v1.22.0
[1.21.0]: https://github.com/itechmeat/open-second-brain/compare/v1.20.0...v1.21.0
[1.20.0]: https://github.com/itechmeat/open-second-brain/compare/v1.19.1...v1.20.0
[1.19.1]: https://github.com/itechmeat/open-second-brain/compare/v1.19.0...v1.19.1
diff --git a/README.md b/README.md
index 4dd89985..662d3074 100644
--- a/README.md
+++ b/README.md
@@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm
## What is new
-Open Second Brain 1.21.0 is a safety-hardening pass around the surfaces that write to or trust the vault and the codegraph. The secret redactor now enforces a fail-closed 1 MiB scan window instead of silently truncating oversized input (always on), with opt-in infrastructure-topology detectors that extend coverage beyond secrets when enabled. Before the partner surface mutates the codegraph, a read-only health check refuses the write when the index is stale or inconsistent rather than importing into a graph it cannot trust (always on). A report-only hardcoded-path check surfaces path leakage in shipped source without failing on pre-existing hits (always on), and the recall-telemetry cost meter now separates memory write volume from read volume (always on). Two quiet loss and collision gaps close as well: punctuation-only note titles get a deterministic `unnamed-<hash>` slug instead of collapsing into one shared note (always on), and session provenance is recorded as a stable logical key so a vault moved between machines keeps its lookups (always on). The opt-in pieces (the extra infra-topology detectors) are off by default and byte-identical when the new path is unused, and the kernel still calls no LLM.
+Open Second Brain 1.22.0 closes the retrieval quality loop on four surfaces. Packed and recalled items now carry a structural epistemic status (observed, derived, hypothesis, plan, or unknown) plus evidence refs, derived from existing graph metadata so a consuming model can tell a source-backed fact from a conjecture (always on, fields absent when unknown). A thin recall adequacy verdict classifies each recall as sufficient, weak, or insufficient with a recommended action (proceed, re-recall, abstain) and an optional escalate flag, surfaced in recall gate output and context receipts when the caller passes scores (thresholds configurable). A persisted cross-query demand log records each recall and an aggregation reader ranks recurring queries the vault answers poorly, exposed as the brain_knowledge_gaps tool and o2b brain knowledge-gaps (the log is written only when recall gate telemetry is opt-in, queries normalized before append). A unified lessons digest folds positive knowledge and dead-ends into one signed, recency-scored, corroboration-tiered corpus under Brain/lessons.md, regenerated every dream pass and loaded on session start. Every new field is optional and absent by default, so callers that do not opt in see byte-identical output, and the kernel still calls no LLM.
## Why
diff --git a/docs/brainstorm/retrieval-precision-quality-loop/cli-output/claude.md b/docs/brainstorm/retrieval-precision-quality-loop/cli-output/claude.md
new file mode 100644
index 00000000..938ede38
--- /dev/null
+++ b/docs/brainstorm/retrieval-precision-quality-loop/cli-output/claude.md
@@ -0,0 +1,32 @@
+## Variant 1
+Approach: Introduce one shared per-item annotation envelope early on the branch — a `RetrievalAnnotations` extension threaded through `context-pack.ts`/`context-receipts.ts` and the reader tail — and have every card attach its field to that single struct (A: rerank score, C: epistemic status + `evidence_refs`, D: adequacy verdict, E: satisfaction signal). Cards land in sequence, each extending the same envelope rather than opening a new output seam. B folds its outcome tag onto the same annotation vocabulary.
+Trade-offs:
+- Cleanest composition: C→D and A→enrich share one growth point, so the C/D output-shape collision and the A/rerank-tail collision collapse into "extend the struct."
+- Broad blast radius on the shared type; a defect in the envelope touches all five read paths at once.
+- Per-card bit-identity is harder to verify: each card mutates a struct other cards also read, so the off-branch proof is no longer a single guarded call site but "this field defaults absent and nothing consumes it."
+- Front-loads an abstraction before the second consumer exists, risking speculative shape.
+Complexity: large
+Risk: medium
+
+## Variant 2
+Approach: Keep each card a self-contained stage with its own module, its own default-off config key, and one guarded call site, minimizing shared types. Resolve the real collisions purely by ordering so the later card extends an already-merged seam: A → C → D → E → B. A appends after the existing heuristic reranks in the reader tail; C adds the epistemic field to context items; D extends C's now-merged item shape with the verdict and reads gate-telemetry/coverage; E instruments the recall path D already touched; B lands last as the capstone assembly over a fully-merged branch.
+Trade-offs:
+- Maximal per-card bit-identity: every off-branch is a single guarded call site whose unchanged output is trivially verifiable.
+- Smallest blast radius per card; a regression is isolated to one stage.
+- Directly mirrors invariant #6 and the demonstrated one-PR-many-cards cadence — no up-front shared abstraction to design or defend.
+- Mild duplication: C's status and D's verdict are emitted as adjacent-but-separate fields; B and E each own their persistence (`Brain/lessons/*` vs `Brain/demand-log/*`) with no shared writer.
+Complexity: medium
+Risk: low
+
+## Variant 3
+Approach: Land the two persistence substrates and a shared recall-event record first — B's outcome-tagged lessons corpus and E's demand log both emitted as continuity/`gate_telemetry`-style records — then rebuild the read-side cards (A, C, D) as consumers of that common event stream. D and E share one recall-event emission; B and E share the record substrate; the dream pass folds everything.
+Trade-offs:
+- Resolves the B/E persistence collision and the D/E recall-instrumentation collision via one shared emission point.
+- Inverts the natural read→write flow, forcing the persistence/event layer to be designed before its consumers exist.
+- Centralized emission on the hot path stresses the fail-open telemetry contract and Syncthing determinism, and makes the off-branch bit-identity proof depend on the whole bus being inert, not a local call site.
+- Largest blast radius and the weakest fit with driving cards one at a time — the first card carries most of the risk for cards that land later.
+Complexity: large
+Risk: high
+
+## Recommended: Variant 2
+Variant 2 is the only option whose off-branch state is provable the way invariant #2 asks for it — a single guarded call site per card, each off-branch byte-identical to today — because it refuses a shared mutable struct or a shared emission bus that would make bit-identity a whole-subsystem property. The ordering A → C → D → E → B is exactly invariant #6 in practice: A is pure search-side with no vault writes and lowest blast radius, so it seeds the cross-encoder-provider pattern first (mirroring `embeddings/registry.ts`) without touching any other card; C then establishes the epistemic field on context items so D extends an already-merged shape rather than colliding on `context-pack.ts`/`context-receipts.ts`; D reuses C's status plus the existing `gate-telemetry`/`coverage` scores as a thin verdict+action layer (honoring its "do not build a separate attention organ" guardrail); E appends its demand log at the recall seam D has already opened, reusing `coverage.ts`'s IDF score as the satisfaction axis; and B lands last as the heaviest assembly, unifying `dead-ends.ts`/`preference.ts` under the `usage-signal.ts` decay scorer and the dream pass over a fully-merged branch. This maximizes reuse of existing scoring/telemetry/coverage primitives, keeps every decision in-kernel deterministic (the cross-encoder stays an opt-in external call, graceful no-op when unset), and matches the repository's proven multi-card-suite cadence. Variant 1's shared envelope and Variant 3's event bus both trade that per-card verifiability and small blast radius for a composition elegance the suite does not need — and both raise, rather than lower, the risk to the load-bearing bit-identity and fail-open invariants.
diff --git a/docs/brainstorm/retrieval-precision-quality-loop/cli-output/prompt.md b/docs/brainstorm/retrieval-precision-quality-loop/cli-output/prompt.md
new file mode 100644
index 00000000..e25a14aa
--- /dev/null
+++ b/docs/brainstorm/retrieval-precision-quality-loop/cli-output/prompt.md
@@ -0,0 +1,222 @@
+# Architecture consultation - retrieval-precision-quality-loop
+
+You are a senior software architect advising on the design of a new feature suite for an
+existing TypeScript project. Produce **exactly 3 distinct architectural variants** and then
+**exactly one recommendation**. Variants and recommendation only - no code, nothing outside
+those sections.
+
+## Output format (follow EXACTLY)
+
+```
+## Variant 1
+Approach: <2-3 sentences>
+Trade-offs:
+- <bullet>
+Complexity: small | medium | large
+Risk: low | medium | high
+
+## Variant 2
+Approach: <2-3 sentences>
+Trade-offs:
+- <bullet>
+Complexity: small | medium | large
+Risk: low | medium | high
+
+## Variant 3
+Approach: <2-3 sentences>
+Trade-offs:
+- <bullet>
+Complexity: small | medium | large
+Risk: low | medium | high
+
+## Recommended: Variant N
+<rationale - why this variant over the others, in the context of the project conventions below>
+```
+
+Do not include code, pseudo-code, or file listings. No preamble, no closing remarks.
+
+## Project
+
+- **Name**: Open Second Brain (package `open-second-brain`, v1.21.0).
+- **Language/runtime**: TypeScript (strict), runs on Bun. Obsidian-native second-brain memory
+ layer for AI agents. Plain Markdown vault under `Brain/`. Ships a deterministic CLI (`o2b`),
+ an MCP server, and native plugins (Hermes/Claude Code/Codex/OpenClaw/opencode/Grok).
+- **Invariants you MUST honor** (load-bearing conventions; violating any breaks the release):
+ 1. **The kernel calls no LLM.** Every decision in search/recall/ranking/lessons logic is
+ deterministic. An optional cross-encoder endpoint is an external service call, not an
+ in-kernel model - it must be opt-in and a graceful no-op when unconfigured.
+ 2. **Default-off switches are bit-identical to today.** Any ranking/score/cost change must
+ produce byte-identical results when the new path is off. The easiest property to verify is
+ a single guarded call site whose off-branch is unchanged.
+ 3. **Telemetry is fail-open** via `emitGatedTelemetry` - never throws into the hot path.
+ 4. **Language-agnostic.** No hardcoded natural-language word lists (stopwords, keywords,
+ negations) in any language. Use structural signals, frontmatter fields, corpus document
+ frequency / IDF, or explicit agent/LLM extraction.
+ 5. **Syncthing-peer determinism.** Anything written to the vault is deterministic across
+ machines (stable sort keys, stable slugs, no wall-clock-only tiebreaks).
+ 6. **One release = one PR = many cards, driven ONE AT A TIME on a shared branch.** Each card
+ builds on commits the previous cards already landed. Non-conflict is achieved by ordering
+ the real collision pairs so the later card extends an already-merged seam.
+ 7. **Markdown-first persistence.** New knowledge (lessons, demand logs) is plain `.md` /
+ JSON sidecar under `Brain/` so FTS indexes it and the operator can grep/edit it.
+ 8. No TypeScript cast crutches (`as unknown as T`). Build correct shapes with conditional
+ spreads / narrowing. English-only strings in code; abstract multi-language.
+
+## Release scope: retrieval-precision-quality-loop
+
+Closing the full recall/search quality loop: rank (cross-encoder) -> verify (epistemic
+provenance) -> observe (adequacy verdict + cross-query demand log) -> learn (outcome-tagged
+lessons loop). Five cards ship together as one PR on branch
+`feat/retrieval-precision-quality-loop`, driven one at a time:
+
+### Card A (t_110867f5, p4) - Optional pluggable cross-encoder rerank stage for the search reader
+OSB has hybrid search (rank-fusion of semantic + keyword) and several *heuristic* rerank stages
+(relevance rerank, entity-match boost, MMR diversity, usage/activation decay) but no learned
+cross-encoder re-scoring pass. Add an optional, opt-in cross-encoder rerank stage
+(OpenAI-compatible / pluggable endpoint) that re-scores the top-K fused candidates as a final
+reader step. Mirror the existing embedding-provider config pattern (key + endpoint, graceful
+no-op when unset) so the stage is zero-cost when disabled. Default behavior unchanged when no
+rerank endpoint is configured.
+
+### Card B (t_62363378, p4) - Outcome-tagged, recency-scored lessons loop unifying dead-ends and preferences with corroboration tiering
+A closed-loop outcome-driven feedback system over one artifact class (saved results/Q&A): an
+agent stamps each saved result with an outcome (`useful | dead_end | corrected`), and a
+deterministic aggregation pass folds those outcomes into a single `LESSONS` file using a SIGNED,
+recency-decayed score (configurable half-life). A node is promoted to PREFERRED only once
+corroborated by >=N distinct results, otherwise TENTATIVE, with mixed-signal nodes rendered as
+CONTESTED (recency-wins). The lessons file is auto-loaded at session start and auto-refreshed.
+**Assembly/integration gap, not from-scratch**: OSB already owns negative knowledge
+(`src/core/brain/dead-ends.ts`, bounded active set N=100), positive knowledge (preferences with
+confidence band + `retired` state via the dream pass), AND a signed recency-decay scorer
+(`src/core/brain/continuity/usage-signal.ts` `decayWeight()`/`rankByUsageDecay()`, exponential
+half-life default 30 days). The delta is (1) collapsing the two registries into one
+outcome-tagged corpus, (2) reusing usage-signal-style scoring as the promotion signal with an
+explicit corroboration threshold, (3) a single auto-loaded/auto-refreshed lessons digest. Watch
+for overlap with the existing dream pass (repeat signals -> rules, retire stale) - the
+improvement should extend that machinery (signed + corroboration-gated tiers) rather than
+introduce a parallel loop. Auto-load maps onto OSB's existing `active.md`/SessionStart-hook
+surface and the Hermes cron (dream runs on cron), not git hooks.
+
+### Card C (t_e163feb2, p3) - Epistemic provenance markers on recalled/packed items
+Attach an epistemic status (`Observed | Derived | Hypothesis | Plan | Unknown`) + `evidence_refs`
+to each item emitted by `brain_context_pack` (and recall results). Today `brain_truth` /
+`brain_derive_fact` exist but "derived" means preference keys (`normalizeDerivedKeys`), not an
+epistemic status on context items. `evidence_ref` = 0 hits, `epistemic` = 0 hits in src.
+`brain_context_pack` / `brain_context_receipts` emit structured context without a per-item
+epistemic status. Derive the status from existing graph metadata (source-backed vs inferred vs
+preference/hypothesis tiers) - no manual tagging. Lets downstream reasoning (and the
+recall-adequacy gate, Card D) distinguish grounded facts from conjecture, cutting confident
+fabrication. Pairs with Card D.
+
+### Card D (t_b8f66fec, p3) - Recall adequacy verdict with explicit low-adequacy actions (abstain / re-recall / escalate)
+On top of the existing gate telemetry (`brain_recall_gate` + `src/core/brain/gate-telemetry.ts`
+emit gate/relevance scores; `brain_recall_telemetry` / `brain_recall_feedback` exist), classify
+each recall result (`sufficient | weak | insufficient`) and wire explicit outcomes:
+- `sufficient` -> proceed
+- `weak` -> automatic re-recall via an alternate strategy (broaden scope / different path)
+ before answering
+- `insufficient` -> **abstain** (return an explicit "insufficient grounding" signal rather than a
+ low-confidence answer) and/or **escalate** (flag for review / surface to operator)
+Expose the verdict in `brain_recall_gate` output and `brain_context_receipts` so callers can
+branch on it. **Scope guardrail**: reuse existing telemetry scores; thresholds configurable. Do
+NOT build a separate "attention organ" - this is a thin verdict + action layer over what already
+exists. Complements Card C (that feeds this gate).
+
+### Card E (t_97091fff, p3) - Cross-query demand log surfacing queries that are asked often but recall poorly
+Log every recall query against the vault together with a deterministic satisfaction signal
+(result count + the IDF-weighted coverage score already computed per query in
+`src/core/search/coverage.ts`), then aggregate over time to surface queries that recur
+frequently but consistently return weak/empty results. This is a cross-query *demand* signal -
+"what do operators keep asking that the Brain can't answer" - distinct from OSB's existing
+structural gaps (dangling wikilink targets, `deep-synthesis.ts:407` iterates `report.gaps`) and
+from its in-the-moment per-query coverage check (`coverage.ts:141` `planTargetedRetry`, scoped to
+the current query, not persisted/aggregated). **Keep it LLM-free and deterministic**: append each
+recall to a rolling log under `Brain/` (query text or normalized terms + timestamp + result count
++ coverage score), then provide an aggregation (a `knowledge_gaps`-style read tool / CLI) that
+buckets by normalized query and ranks by frequency x low-satisfaction. **Watch privacy/size**:
+queries may contain sensitive text - normalize/redact and cap/rotate the log. Reuse the existing
+coverage score as the "answered poorly" axis rather than inventing a new metric.
+
+## Key existing files / surfaces (read-only references)
+
+- `src/core/search/fusion.ts` - rank-fusion (`linear` weighted-sum + `rrf` reciprocal-rank),
+ min-max-normalised fused relevance on [0,1] composing with downstream boosts.
+- `src/core/search/ranker.ts` - primary scorer (BM25 + cosine + intent multipliers + boosts).
+- `src/core/search/enrich.ts` - heuristic reranks: `rerankByRelevance` (core textual relevance =
+ keyword + semantic lane contributions, ignoring recency/usage/link boosts); `deriveTrust`
+ (age_days, superseded, conflict).
+- `src/core/search/coverage.ts` - single source of truth for query-term verification:
+ `significantTerms` (no stopword list, language-agnostic), `idfForTerm` (smoothed IDF:
+ `ln(1 + N/(1+df))`), `isRareTerm`, `buildCoverageReport`, `planTargetedRetry` (deterministic
+ follow-up over uncovered RARE terms, pure verdict over an already-built report).
+- `src/core/search/embeddings/` - provider pattern to mirror for the cross-encoder:
+ - `registry.ts`: named provider profiles persisted to `Brain/search/embedding-providers.json`
+ (CLI-added, fail-soft load), stores only env-key NAMES never secret values, safe to sync.
+ - `openai-compat.ts`: OpenAI-compatible `/v1/embeddings` provider (base_url + model +
+ env-resolved api key, fail-closed validation when semantic enabled, HTTP client with caching).
+ - `provider.ts` / `registry.ts`: `expandRegisteredProvider` resolves a registered name to
+ `openai-compat` config at config-resolution time, after built-ins.
+- `src/core/search/profiles.ts` - search profiles ("thorough" etc.); cross-encoder most valuable
+ for the thorough profile where recall quality matters most.
+- `src/core/brain/dead-ends.ts` - negative-knowledge class: one `.md` note under
+ `Brain/dead-ends/` (approach + why-failed + context), bounded active set N=100, overflow
+ archives oldest to `Brain/dead-ends/archive/`. `sanitiseTextField`, `writeFrontmatterAtomic`.
+- `src/core/brain/preference.ts` + `src/core/brain/apply-evidence.ts` - positive knowledge:
+ signals -> rules via the dream pass, confidence/confidence_value band, `last_evidence_at`,
+ `retired` state (`Brain/retired/`).
+- `src/core/brain/continuity/usage-signal.ts` - SIGNED recency-decay scorer: `decayWeight()`
+ (exponential half-life, `DEFAULT_HALF_LIFE_DAYS = 30`, configurable), frequency gain,
+ `rankByUsageDecay()`. Currently over continuity records ranked by recall telemetry, NOT over
+ an outcome-tagged corpus.
+- `src/core/brain/context-pack.ts` - `packContext()` emits `ContextPackItem[]` with `tier`
+ (core/supporting/peripheral), bounded-token vault slice. Each item extends
+ `ContextTransformAnnotations`.
+- `src/core/brain/context-receipts.ts` - `emitContextReceipt` / `listContextReceipts` /
+ `summarizeContextReceipt` over continuity records.
+- `src/core/brain/gate-telemetry.ts` - `emitGateTelemetry` (records every automatic-recall gate
+ decision as a `gate_telemetry` continuity record; never stores raw prompt, only SHA-256 prefix
+ + length; default OFF behind `recall_gate_telemetry` config key).
+- `src/core/brain/deep-synthesis.ts` - `report.gaps` iterates dangling-wikilink targets
+ (structural gaps only); corroboration concept appears only narratively at line 171.
+- `src/core/brain/active.ts` - `Brain/active.md` auto-loaded at session start via SessionStart
+ hook; auto-generated by `dream`. (Reusable surface for a lessons digest auto-load.)
+- `src/core/brain/dream.ts` / `dream-refresh.ts` / `dream-stage.ts` - the deterministic dream
+ pass (repeat signals -> rules, retire stale) that runs on the Hermes cron.
+
+## Git log (last 20, most recent first) - shows the one-PR-themed-suite cadence
+
+```
+4281605 feat(brain): integrity & safety hardening suite (1.21.0) (#115)
+313d061 feat: configurable skills_dir + trigger-keyword auto-attach scoring (#114)
+a3ea315 fix: v1.19.1 - cross-vault cards, event-trace exit codes, registry-guard hygiene (#113)
+bb5f320 feat(brain): session-boundary capture durability and post-compaction pinned-anchor survival audit (v1.19.0) (#112)
+c5e30b8 fix: cross-vault chain-stop reads the max normalized score (v1.18.1) (#111)
+33b4fba feat: recall precision, coverage, and provenance hardening (v1.18.0) (#110)
+254b580 feat: codegraph link-graph depth and MCP exposure (v1.17.0) (#108)
+da2e3cc feat: memory subsystem alignment - honest pinned budgets, atomic batch writes, on_memory_write host bridge (v1.16.0) (#107)
+4db7862 fix(hermes): pass --repo so bridge skill discovery resolves repoRoot (#103) (#106)
+0a4b6da feat: calendar obligations, agenda synthesis, OKF portability, Obsidian Bases and steelman synthesis (v1.15.0) (#105)
+f8b4abf feat(brain): add feedback default scope and vault write containment (#104)
+20ea7ef feat: per-handoff LLM generation tracing and prompt-prefix stability metric (#102)
+9c1d48f feat: CodeGraph and MCP operational readability (v1.12.0) (#101)
+c2c3ff4 feat: Session Knowledge Synthesis Suite - structured session summaries, idea-lineage, episodic note history (v1.11.0) (#100)
+56dd3dd fix(hermes): bridge EOF - byte streams, stderr drain, retry loop (#92)
+35b824e feat: Recall & Working-Memory Quality Suite - selectable profiles, usage decay, co-occurrence, file-context (v1.10.0) (#99)
+929d54c feat: Brain Portability & Interop Suite - bank export/import, page contract, brain_create_note, in-process SDK (v1.9.0) (#98)
+7cdbfc0 feat: Indexer Durability & Resilience Suite - cooperative abort, graceful watch shutdown, resumable reindex (v1.8.0) (#97)
+8b679fe feat: Knowledge Provenance Suite - ingest, research, NER, derived facts, owner-scope, standing-query (v1.7.0) (#96)
+6e59a42 feat: Vault Integrity & Trust Suite - untrusted-source containment, NFC identity, watch-sync, O(1) graph, agent-scope (v1.6.0) (#95)
+```
+
+## What to design
+
+An architecture that lands all five cards as one themed suite on the shared branch, each behind
+its own explicit default-off switch whose off-branch is bit-identical to today, driven one at a
+time in a non-conflicting order, honoring every invariant above. Identify the real cross-card
+collision points (e.g. Card C and Card D both touch `context-pack.ts`/`context-receipts.ts`
+output shape; Card A and the existing heuristic reranks both touch the reader tail;
+Cards B/E both add new persistence under `Brain/`; Card D re-uses Card C's status and
+`gate-telemetry`/`coverage` scores) and say how each variant resolves them. Weigh: blast radius,
+per-card bit-identity verifiability, LLM-free-kernel preservation, reuse of existing
+scoring/telemetry/coverage primitives, and fit with the demonstrated multi-card-suite release
+cadence.
diff --git a/docs/brainstorm/retrieval-precision-quality-loop/design.md b/docs/brainstorm/retrieval-precision-quality-loop/design.md
new file mode 100644
index 00000000..52cd27d7
--- /dev/null
+++ b/docs/brainstorm/retrieval-precision-quality-loop/design.md
@@ -0,0 +1,198 @@
+# Design - retrieval-precision-quality-loop
+
+Branch: `feat/retrieval-precision-quality-loop`
+Slug: `retrieval-precision-quality-loop`
+Cards: A `t_110867f5`, B `t_62363378`, C `t_e163feb2`, D `t_b8f66fec`, E `t_97091fff`
+
+## Problem
+
+Open Second Brain's recall/search pipeline is rich (hybrid rank-fusion, heuristic reranks,
+IDF-weighted coverage, gate telemetry, continuity records, dream-promoted preferences,
+dead-ends) but the full quality loop is not yet closed. Five concrete gaps leave recall less
+precise, less honest about its own grounding, and less able to turn observed failures into
+improvements:
+
+1. **No learned re-scoring pass.** Heuristic reranks (relevance, entity boost, MMR, usage
+ decay) order the top-K, but no cross-encoder jointly re-scores query/document pairs, so
+ precision-at-top stays bounded by what lexical + cosine heuristics can express.
+2. **Context items have no epistemic status.** `brain_context_pack` / `brain_context_receipts`
+ emit structured context but the consuming model cannot tell a source-backed fact from a
+ conjecture or a plan, so it reasons over all of it with the same confidence.
+3. **No adequacy verdict that drives behavior.** Gate telemetry emits scores, but there is no
+ classification that says "this recall is too weak to answer from" and then abstains or
+ re-recalls; the system hands top-K to the model regardless of fitness.
+4. **No cross-query demand signal.** Per-query coverage and structural dangling-link gaps
+ exist, but nothing persists or aggregates which queries recur yet consistently answer
+ poorly, so the operator never sees the prioritized backlog of "what to write next".
+5. **Lessons are split across two registries.** Negative knowledge (dead-ends) and positive
+ knowledge (dream-promoted preferences) are separate registries with a coarse confidence
+ band and no signed recency-decayed scoring or corroboration gate over a single outcome
+ corpus, so a fresh dead-end does not outweigh a stale "useful", and a one-off outcome can
+ become a rule.
+
+## Scope
+
+All five cards ship in this release, driven one at a time on the shared branch in blast-radius
+order so each later card extends an already-merged seam rather than colliding:
+
+- **A** `t_110867f5` - optional, opt-in cross-encoder rerank stage (OpenAI-compatible / pluggable
+ endpoint) re-scoring the top-K fused candidates as a final reader step.
+- **C** `t_e163feb2` - epistemic provenance markers (`Observed | Derived | Hypothesis | Plan |
+ Unknown` + `evidence_refs`) on each item emitted by `brain_context_pack` and recall results.
+- **D** `t_b8f66fec` - recall adequacy verdict (`sufficient | weak | insufficient`) with explicit
+ low-adequacy actions (proceed / re-recall / abstain / escalate), a thin layer over existing
+ telemetry + coverage scores plus C's epistemic mix.
+- **E** `t_97091fff` - persisted cross-query demand log + aggregation reader surfacing queries
+ asked often but answered poorly, reusing the existing coverage score as the satisfaction axis.
+- **B** `t_62363378` - outcome-tagged, recency-scored lessons loop unifying dead-ends and
+ preferences with corroboration tiering, reusing the `usage-signal.ts` decay scorer and the
+ dream pass.
+
+**Drive order:** A → C → D → E → B.
+
+## Out of scope
+
+- A learned model living inside the kernel. The cross-encoder is an opt-in external service
+ call, graceful no-op when unconfigured; every other decision (adequacy verdict, corroboration
+ tier, demand aggregation, epistemic derivation) stays deterministic and LLM-free.
+- Any change to default ranking/score/output when a feature is off. Each card is a single
+ guarded call site whose off-branch is byte-identical to today.
+- A web dashboard. Every new surface ships as CLI / MCP reader or as plain Markdown under
+ `Brain/`.
+- Manual epistemic tagging. C derives status from existing graph metadata only.
+- A separate "attention organ" for adequacy. D is a thin verdict + action layer over what
+ already exists, per its own scope guardrail.
+- Migrating or rewriting the existing dead-ends or preference registries. B collapses them into
+ one outcome-tagged view and extends the dream pass; it does not delete or rewrite the source
+ notes.
+- HyDE-style generative query rewriting or learned query expansion. D's re-recall uses the
+ existing deterministic expansion / alternate-strategy machinery.
+
+## Chosen approach
+
+**Variant 2 - each card a self-contained stage behind its own default-off switch, ordered so the
+later card extends an already-merged seam (A → C → D → E → B).**
+
+Every card lands as its own module with its own config key and exactly one guarded call site
+whose off-branch is provably bit-identical to today. This is the only variant whose off-branch
+state is verifiable the way the project demands (a single guarded call site per card, not a
+shared mutable struct or a shared emission bus that turns bit-identity into a whole-subsystem
+property). Cross-card collisions are resolved by the cheapest sufficient mechanism - ordering -
+rather than by a speculative up-front abstraction:
+
+- **A → heuristic reranks** (both touch the reader tail): A appends after the existing heuristic
+ reranks as an isolated final stage; it does not alter them.
+- **C → D** (both touch `context-pack.ts` / `context-receipts.ts` output shape): C establishes
+ the epistemic field on context items; D extends C's now-merged item shape with the verdict
+ instead of colliding on the same files.
+- **D → E** (both touch the recall path): D opens the recall seam with its verdict + action
+ layer; E appends its demand-log instrumentation at that seam, reusing the coverage score D
+ already surfaces.
+- **B ↔ E** (both add persistence under `Brain/`): independent directories
+ (`Brain/lessons/*` vs `Brain/recall/demand-log*`), no shared writer; B lands last over a
+ fully-merged branch so a later shift to the recall/telemetry surfaces never forces a rebase.
+
+Telemetry on every card routes through `emitGatedTelemetry` and stays fail-open. The kernel
+calls no LLM.
+
+## Design decisions
+
+### A - cross-encoder rerank (search-side, no vault writes)
+
+Mirror the existing embedding-provider pattern (`src/core/search/embeddings/registry.ts` +
+`openai-compat.ts`): a `rerank/` module with a named-provider registry persisted to
+`Brain/search/rerank-providers.json` (env-key NAMES only, fail-soft load) and an
+OpenAI-compatible rerank provider (base_url + model + env-resolved api key, fail-closed
+validation when enabled-but-unconfigured). The stage re-scores the top-K fused candidates as a
+final reader step appended after the heuristic reranks. Config surface (`search_rerank_*`
+keys: enabled, model, base_url, env_key, top_k, min_score) resolves through
+`resolveSearchConfig` (config + env, with validation), and is most valuable on the "thorough"
+search profile. When disabled (the default), the stage returns its input unchanged - bit
+identical ordering, zero HTTP cost. On endpoint error it degrades gracefully to the
+heuristic-ordered input and logs via `emitGatedTelemetry` (fail-open), never throwing into the
+hot path.
+
+### C - epistemic provenance (read-time derivation, optional fields)
+
+A pure `epistemic-status.ts` under `src/core/brain/provenance/` derives an `EpistemicStatus`
+(`observed | derived | hypothesis | plan | unknown`) plus `evidence_refs` from existing graph
+metadata: a source-backed note/wikilink -> `observed` (carry the source as evidence_ref); an
+inferred/derived fact -> `derived` (link the basis); a preference/hypothesis tier ->
+`hypothesis`; an intended/not-yet-true item -> `plan`; an acknowledged gap -> `unknown`. No
+manual tagging and no natural-language word lists (language-agnostic by construction). The
+fields are added to `ContextPackItem` and recall results as **optional** absent-by-default
+fields, so the off-branch (feature flag off) produces byte-identical pack/receipt output. D
+reads this status.
+
+### D - adequacy verdict (thin layer over existing scores)
+
+A pure `adequacy.ts` under `src/core/brain/recall/` classifies each recall into
+`sufficient | weak | insufficient` and emits a recommended action
+(`proceed | re_recall | abstain | escalate`), reusing the existing `gate-telemetry` relevance
+scores, the `coverage.ts` IDF-weighted coverage, and C's epistemic mix (a recall that is mostly
+`hypothesis`/`unknown` scores lower fitness than one grounded in `observed` facts). Thresholds
+are configurable. The verdict + action are exposed in `brain_recall_gate` output and the
+`context-receipts` summary so callers can branch on them. `weak` triggers an alternate-strategy
+re-recall (broaden scope / different path) via existing recall machinery; `insufficient`
+returns an explicit "insufficient grounding" abstain signal and/or an escalate flag instead of
+a low-confidence answer. This honors the card's guardrail: reuse existing telemetry, do NOT
+build a separate attention organ.
+
+### E - cross-query demand log (LLM-free, privacy-capped persistence)
+
+A `demand-log.ts` under `src/core/brain/recall/` appends each recall (normalized terms or a
+redacted query form + timestamp + result count + the coverage score already computed per query
+in `coverage.ts`) to a rolling, size-capped log under `Brain/recall/demand-log.jsonl`.
+Privacy/size discipline: queries may contain sensitive text, so the query is normalized/redacted
+before append and the log is capped/rotated. An aggregation reader (a `knowledge_gaps`-style
+read tool / CLI) buckets by normalized query and ranks by frequency x low-satisfaction,
+reusing the existing coverage score as the "answered poorly" axis rather than inventing a new
+metric. Appends at the recall seam D has already opened. Deterministic across Syncthing peers.
+
+### B - outcome-tagged lessons loop (assembly over existing primitives)
+
+A `lessons/` module under `src/core/brain/` unifies `dead-ends.ts` (negative outcomes) and
+`preference.ts`/`apply-evidence.ts` (positive outcomes) into one outcome-tagged corpus
+(`useful | dead_end | corrected`), reusing `usage-signal.ts` `decayWeight()` (configurable
+half-life, signed) as the promotion signal with an explicit corroboration threshold: a node is
+`PREFERRED` only once corroborated by >=N distinct results, otherwise `TENTATIVE`, with
+mixed-signal nodes rendered `CONTESTED` (recency-wins). A deterministic aggregation pass
+folds the corpus into a single `Brain/lessons/LESSONS.md` digest that is auto-loaded at session
+start (extending the existing `active.ts` / SessionStart-hook surface) and auto-refreshed by
+the dream cron (extending `dream-refresh.ts`). This extends the dream pass with signed +
+corroboration-gated tiers rather than introducing a parallel loop. It lands last over a
+fully-merged branch because it is the heaviest assembly and depends on the recall/telemetry
+surfaces not shifting underneath it.
+
+## File changes (per card, indicative)
+
+- **A**: new `src/core/search/rerank/{cross-encoder,registry,provider}.ts`; config keys via
+ `resolveSearchConfig`; one guarded call site in the reader tail; MCP/CLI exposure for provider
+ registration (mirror embedding registry).
+- **C**: new `src/core/brain/provenance/epistemic-status.ts`; optional `epistemic_status` +
+ `evidence_refs` fields on `ContextPackItem` and recall result type; derivation wired into
+ `context-pack.ts` / recall rendering behind a feature flag.
+- **D**: new `src/core/brain/recall/adequacy.ts`; verdict + action added to `brain_recall_gate`
+ output and `context-receipts` summary; re-recall trigger and abstain/escalate paths behind a
+ feature flag.
+- **E**: new `src/core/brain/recall/demand-log.ts` (append + rotate/cap + redact) and an
+ aggregation reader (CLI/MCP); append wired at the recall seam behind a feature flag.
+- **B**: new `src/core/brain/lessons/{outcome-corpus,score,digest}.ts`; `active.ts` /
+ `dream-refresh.ts` extension for auto-load + auto-refresh; corroboration-tiered dream pass
+ extension behind a feature flag.
+
+## Risks
+
+- **Per-card bit-identity under shared output shapes (C/D).** Mitigated by landing C before D so
+ D extends a merged shape, and by making every new field optional/absent-by-default so the
+ off-branch is byte-identical. Acceptance tests assert off-branch output equality.
+- **Cross-encoder latency/availability leaking into the hot path (A).** Mitigated by default-off,
+ a strict timeout, and fail-open degradation to heuristic-ordered input on endpoint error; zero
+ HTTP cost when unconfigured.
+- **Demand-log privacy growth (E).** Mitigated by query normalization/redaction before append
+ and size cap/rotation; the log is operator-inspectable Markdown/JSONL so it can be audited.
+- **Lessons loop double-counting the dream pass (B).** Mitigated by extending the dream pass
+ (signed + corroboration-gated) rather than running a parallel aggregation; the corroboration
+ threshold (>=N distinct results) prevents one-off outcomes from promoting.
+- **Syncthing-peer determinism (B, E).** Mitigated by stable sort keys, stable slugs, and
+ deterministic aggregation (no wall-clock-only tiebreaks) in every vault write.
diff --git a/docs/brainstorm/retrieval-precision-quality-loop/plan.md b/docs/brainstorm/retrieval-precision-quality-loop/plan.md
new file mode 100644
index 00000000..00fe893e
--- /dev/null
+++ b/docs/brainstorm/retrieval-precision-quality-loop/plan.md
@@ -0,0 +1,190 @@
+# Plan - retrieval-precision-quality-loop
+
+Branch: `feat/retrieval-precision-quality-loop`
+Slug: `retrieval-precision-quality-loop`
+Combined design: `docs/brainstorm/retrieval-precision-quality-loop/design.md`
+Drive order: **A → C → D → E → B** (blast-radius order; each later card extends an already-merged
+seam). Cards are driven ONE AT A TIME on the shared branch. Each worker builds on the commits the
+previously-driven in-scope cards already landed and must not duplicate or conflict with sibling
+tasks.
+
+Global conventions for every card:
+
+- TDD: write the failing test first, then implement until it passes. Run `bun run validate`
+ (typecheck + lint + test) before claiming the card done.
+- Default-off switch per card (one config key), off-branch byte-identical to today.
+- Telemetry via `emitGatedTelemetry`, fail-open.
+- Kernel calls no LLM. The cross-encoder (A) is an opt-in external service call.
+- No TypeScript cast crutches; build correct shapes with conditional spreads / narrowing.
+- English-only strings in code; logic is language-agnostic (no hardcoded NL word lists).
+
+In-scope cards shipping together in this one release:
+
+| id | card | one-line title |
+|---|---|---|
+| `t_110867f5` | A | Optional pluggable cross-encoder rerank stage for the search reader |
+| `t_e163feb2` | C | Epistemic provenance markers on recalled/packed items |
+| `t_b8f66fec` | D | Recall adequacy verdict with explicit low-adequacy actions |
+| `t_97091fff` | E | Cross-query demand log surfacing queries asked often but recall poorly |
+| `t_62363378` | B | Outcome-tagged, recency-scored lessons loop unifying dead-ends and preferences with corroboration tiering |
+
+---
+
+## A (t_110867f5) - Optional pluggable cross-encoder rerank stage
+
+### Files
+- `src/core/search/rerank/cross-encoder.ts` - the rerank provider (OpenAI-compatible endpoint:
+ base_url + model + env-resolved api key, re-scores top-K query/doc pairs).
+- `src/core/search/rerank/registry.ts` - named-provider registry persisted to
+ `Brain/search/rerank-providers.json` (env-key NAMES only, fail-soft load), mirroring
+ `embeddings/registry.ts`.
+- `src/core/search/rerank/provider.ts` - provider interface + built-in/default resolution.
+- `src/core/search/rerank/index.ts` - `applyCrossEncoderRerank(results, query, config)` final
+ reader step; returns input unchanged when disabled or on endpoint error.
+- config keys threaded through `resolveSearchConfig`: `search_rerank_enabled` (default false),
+ `search_rerank_model`, `search_rerank_base_url`, `search_rerank_env_key`, `search_rerank_top_k`,
+ `search_rerank_min_score`, with env parity + validation.
+- one guarded call site in the reader tail (after `enrich.ts` heuristic reranks).
+- MCP/CLI exposure for provider registration (mirror the embedding registry CLI verbs).
+- tests under the module's sibling `.test.ts` (follow repo convention).
+
+### Acceptance (a passing test)
+- Disabled (default): the reader tail output is byte-identical to the pre-feature baseline for
+ the same query/results (snapshot/diff equality asserted in a test).
+- Enabled + endpoint unconfigured: validation fails closed with a clear error (mirrors
+ `openai-compat.ts` when semantic is enabled without base_url/model).
+- Enabled + endpoint returns scores: top-K are re-ordered by the cross-encoder scores; a doc the
+ heuristic ranker placed 3rd but the cross-encoder scored highest lands 1st.
+- Enabled + endpoint errors: stage degrades to the heuristic-ordered input (no throw into the
+ hot path) and emits one fail-open telemetry record.
+
+### Depends on
+- None. Lands first (pure search-side, no vault writes, lowest blast radius, seeds the
+ provider-registry pattern). Must not touch `context-pack.ts`, `context-receipts.ts`, or any
+ brain/ persistence.
+
+---
+
+## C (t_e163feb2) - Epistemic provenance markers
+
+### Files
+- `src/core/brain/provenance/epistemic-status.ts` - pure derivation:
+ `deriveEpistemicStatus(meta)` -> `EpistemicStatus`
+ (`observed | derived | hypothesis | plan | unknown`) + `evidence_refs`, from existing graph
+ metadata (source-backed vs inferred vs preference/hypothesis vs intended vs gap). Language
+ agnostic, no manual tagging.
+- optional fields `epistemic_status` and `evidence_refs` added to `ContextPackItem` and the
+ recall result type (absent by default).
+- derivation wired into `context-pack.ts` and recall result rendering behind a feature flag.
+- `src/mcp/brain/*` (or the pack/recall surface) so the field surfaces in
+ `brain_context_pack` / `brain_context_receipts` output when enabled.
+
+### Acceptance (a passing test)
+- Flag off: `brain_context_pack` and recall output is byte-identical to the pre-feature baseline
+ (the new fields are absent, not present-with-empty).
+- Flag on: a source-backed note item is tagged `observed` with the source wikilink as
+ `evidence_refs`; an inferred/derived-fact item is `derived` with its basis linked; a
+ preference-tier item is `hypothesis`; an intended/not-yet-true item is `plan`; an
+ acknowledged gap is `unknown`.
+- Derivation is deterministic across two runs on the same input (Syncthing-peer safe).
+
+### Depends on
+- A landed (so the branch is stable at the reader tail). Establishes the item-shape seam D
+ extends. Must not collide with `context-receipts.ts` summary shape beyond adding optional
+ fields.
+
+---
+
+## D (t_b8f66fec) - Recall adequacy verdict with explicit low-adequacy actions
+
+### Files
+- `src/core/brain/recall/adequacy.ts` - pure verdict:
+ `classifyAdequacy(gateScores, coverageReport, epistemicMix, thresholds)` ->
+ `{ verdict: sufficient|weak|insufficient, action: proceed|re_recall|abstain|escalate }`.
+ Reuses `gate-telemetry` relevance scores, `coverage.ts` IDF-weighted coverage, and C's
+ epistemic mix; thresholds configurable.
+- verdict + action exposed in `brain_recall_gate` output and `context-receipts` summary.
+- `weak` action: trigger an alternate-strategy re-recall (broaden scope / different path) via
+ existing recall machinery, behind the feature flag.
+- `insufficient` action: abstain signal ("insufficient grounding") and/or escalate flag, behind
+ the feature flag.
+
+### Acceptance (a passing test)
+- Flag off: `brain_recall_gate` output and `context-receipts` summary are byte-identical to the
+ pre-feature baseline.
+- Flag on, high-coverage + mostly-`observed` recall: verdict `sufficient`, action `proceed`.
+- Flag on, partial coverage / mixed `hypothesis`-heavy recall: verdict `weak`, action `re_recall`
+ (one alternate-strategy re-recall fires, deterministic).
+- Flag on, near-zero coverage / mostly `unknown`: verdict `insufficient`, action `abstain`
+ (explicit "insufficient grounding" signal returned, not a low-confidence answer) and/or
+ `escalate`.
+- Verdict is deterministic over the same scores.
+
+### Depends on
+- C landed (D extends C's merged item shape with the verdict; D reads C's epistemic mix). Reuses
+ existing `gate-telemetry.ts` and `coverage.ts` scores - must NOT introduce a new score or a
+ separate attention organ (card guardrail).
+
+---
+
+## E (t_97091fff) - Cross-query demand log surfacing weak-recall queries
+
+### Files
+- `src/core/brain/recall/demand-log.ts` - append + rotate/cap + redact: each recall appends
+ (normalized terms or redacted query form + timestamp + result count + the coverage score
+ already computed per query in `coverage.ts`) to a rolling, size-capped
+ `Brain/recall/demand-log.jsonl`. Query normalized/redacted before append; log capped/rotated.
+- aggregation reader (a `knowledge_gaps`-style read tool / CLI) bucketing by normalized query
+ and ranking by frequency x low-satisfaction, reusing the existing coverage score as the
+ "answered poorly" axis.
+- append wired at the recall seam D opened, behind a feature flag.
+
+### Acceptance (a passing test)
+- Flag off: no file is written under `Brain/`, recall path byte-identical to baseline.
+- Flag on: three recalls of the same normalized query that each return low coverage produce one
+ bucket ranked by frequency x low-satisfaction; a different query asked once with good coverage
+ ranks far lower.
+- Privacy: the on-disk log contains the normalized/redacted form, never the raw sensitive query
+ (asserted by inspecting the written file in the test).
+- Size cap: after N appends the oldest entries are rotated out; the file never grows unbounded.
+- Aggregation is deterministic across two runs on the same log (Syncthing-peer safe).
+
+### Depends on
+- D landed (E appends at the recall seam D opened and reuses the coverage score D surfaces).
+ Independent persistence directory `Brain/recall/` - must not share a writer with B's
+ `Brain/lessons/`.
+
+---
+
+## B (t_62363378) - Outcome-tagged, recency-scored lessons loop with corroboration tiering
+
+### Files
+- `src/core/brain/lessons/outcome-corpus.ts` - unify `dead-ends.ts` (negative) and
+ `preference.ts`/`apply-evidence.ts` (positive) into one outcome-tagged corpus
+ (`useful | dead_end | corrected`).
+- `src/core/brain/lessons/score.ts` - signed recency-decayed score reusing
+ `usage-signal.ts` `decayWeight()` (configurable half-life) as the promotion signal.
+- `src/core/brain/lessons/digest.ts` - corroboration-gated aggregation: a node is `PREFERRED`
+ only once corroborated by >=N distinct results (configurable N), else `TENTATIVE`;
+ mixed-signal nodes render `CONTESTED` (recency-wins). Folds the corpus into a single
+ `Brain/lessons/LESSONS.md`.
+- `active.ts` / `dream-refresh.ts` extension for auto-load (SessionStart) + auto-refresh (dream
+ cron) of the digest.
+- dream-pass extension (`dream.ts`/`dream-stage.ts`) adding signed + corroboration-gated tiers,
+ behind a feature flag.
+
+### Acceptance (a passing test)
+- Flag off: dream pass and `active.md` byte-identical to baseline; no `Brain/lessons/` written.
+- Flag on: a node corroborated by <N distinct results is `TENTATIVE`; once >=N distinct results
+ corroborate it, it promotes to `PREFERRED`; a fresh `dead_end` outcome outweighs a stale
+ `useful` on the same node (recency + sign), rendering it `CONTESTED` when mixed.
+- A one-off outcome does NOT promote to `PREFERRED` (corroboration threshold prevents it).
+- The `LESSONS.md` digest is deterministic across two runs on the same corpus (stable sort keys,
+ stable slugs; Syncthing-peer safe).
+- Auto-load: the digest surface is reachable via the existing `active.ts`/SessionStart path; no
+ git-hook dependency.
+
+### Depends on
+- A, C, D, E landed (B is the capstone assembly over a fully-merged branch; it must not force a
+ rebase of the recall/telemetry surfaces). Extends the dream pass rather than running a parallel
+ aggregation; does not rewrite or delete the source dead-ends/preference notes.
diff --git a/docs/brainstorm/retrieval-precision-quality-loop/variants.md b/docs/brainstorm/retrieval-precision-quality-loop/variants.md
new file mode 100644
index 00000000..c1474bab
--- /dev/null
+++ b/docs/brainstorm/retrieval-precision-quality-loop/variants.md
@@ -0,0 +1,108 @@
+# Architecture variants - retrieval-precision-quality-loop
+
+Branch: `feat/retrieval-precision-quality-loop`. Consultant output is preserved verbatim in
+`cli-output/claude.md`; this file restates the three variants verbatim and records the final
+decision and rationale.
+
+## Variant 1
+Approach: Introduce one shared per-item annotation envelope early on the branch - a
+`RetrievalAnnotations` extension threaded through `context-pack.ts`/`context-receipts.ts` and
+the reader tail - and have every card attach its field to that single struct (A: rerank score,
+C: epistemic status + `evidence_refs`, D: adequacy verdict, E: satisfaction signal). Cards land
+in sequence, each extending the same envelope rather than opening a new output seam. B folds its
+outcome tag onto the same annotation vocabulary.
+Trade-offs:
+- Cleanest composition: C->D and A->enrich share one growth point, so the C/D output-shape
+ collision and the A/rerank-tail collision collapse into "extend the struct."
+- Broad blast radius on the shared type; a defect in the envelope touches all five read paths
+ at once.
+- Per-card bit-identity is harder to verify: each card mutates a struct other cards also read,
+ so the off-branch proof is no longer a single guarded call site but "this field defaults
+ absent and nothing consumes it."
+- Front-loads an abstraction before the second consumer exists, risking speculative shape.
+Complexity: large
+Risk: medium
+
+## Variant 2
+Approach: Keep each card a self-contained stage with its own module, its own default-off config
+key, and one guarded call site, minimizing shared types. Resolve the real collisions purely by
+ordering so the later card extends an already-merged seam: A -> C -> D -> E -> B. A appends
+after the existing heuristic reranks in the reader tail; C adds the epistemic field to context
+items; D extends C's now-merged item shape with the verdict and reads gate-telemetry/coverage; E
+instruments the recall path D already touched; B lands last as the capstone assembly over a
+fully-merged branch.
+Trade-offs:
+- Maximal per-card bit-identity: every off-branch is a single guarded call site whose unchanged
+ output is trivially verifiable.
+- Smallest blast radius per card; a regression is isolated to one stage.
+- Directly mirrors invariant #6 and the demonstrated one-PR-many-cards cadence - no up-front
+ shared abstraction to design or defend.
+- Mild duplication: C's status and D's verdict are emitted as adjacent-but-separate fields; B
+ and E each own their persistence (`Brain/lessons/*` vs `Brain/demand-log/*`) with no shared
+ writer.
+Complexity: medium
+Risk: low
+
+## Variant 3
+Approach: Land the two persistence substrates and a shared recall-event record first - B's
+outcome-tagged lessons corpus and E's demand log both emitted as continuity/`gate_telemetry`-
+style records - then rebuild the read-side cards (A, C, D) as consumers of that common event
+stream. D and E share one recall-event emission; B and E share the record substrate; the dream
+pass folds everything.
+Trade-offs:
+- Resolves the B/E persistence collision and the D/E recall-instrumentation collision via one
+ shared emission point.
+- Inverts the natural read->write flow, forcing the persistence/event layer to be designed
+ before its consumers exist.
+- Centralized emission on the hot path stresses the fail-open telemetry contract and Syncthing
+ determinism, and makes the off-branch bit-identity proof depend on the whole bus being inert,
+ not a local call site.
+- Largest blast radius and the weakest fit with driving cards one at a time - the first card
+ carries most of the risk for cards that land later.
+Complexity: large
+Risk: high
+
+## Recommended: Variant 2
+Variant 2 is the only option whose off-branch state is provable the way invariant #2 asks for
+it - a single guarded call site per card, each off-branch byte-identical to today - because it
+refuses a shared mutable struct or a shared emission bus that would make bit-identity a
+whole-subsystem property. The ordering A -> C -> D -> E -> B is exactly invariant #6 in
+practice: A is pure search-side with no vault writes and lowest blast radius, so it seeds the
+cross-encoder-provider pattern first (mirroring `embeddings/registry.ts`) without touching any
+other card; C then establishes the epistemic field on context items so D extends an
+already-merged shape rather than colliding on `context-pack.ts`/`context-receipts.ts`; D reuses
+C's status plus the existing `gate-telemetry`/`coverage` scores as a thin verdict+action layer
+(honoring its "do not build a separate attention organ" guardrail); E appends its demand log at
+the recall seam D has already opened, reusing `coverage.ts`'s IDF score as the satisfaction
+axis; and B lands last as the heaviest assembly, unifying `dead-ends.ts`/`preference.ts` under
+the `usage-signal.ts` decay scorer and the dream pass over a fully-merged branch. This maximizes
+reuse of existing scoring/telemetry/coverage primitives, keeps every decision in-kernel
+deterministic (the cross-encoder stays an opt-in external call, graceful no-op when unset), and
+matches the repository's proven multi-card-suite cadence. Variant 1's shared envelope and
+Variant 3's event bus both trade that per-card verifiability and small blast radius for a
+composition elegance the suite does not need - and both raise, rather than lower, the risk to
+the load-bearing bit-identity and fail-open invariants.
+
+## Final decision: Variant 2 (accepted)
+
+Agreed with the consultant's recommendation. Variant 2 is the only variant whose off-branch
+state is provable per-card as a single guarded call site - the precondition the repo demands for
+any ranking/score/output change (invariant #2) and the easiest property to verify. It preserves
+the load-bearing conventions: an LLM-free kernel (the cross-encoder is an opt-in external call,
+graceful no-op when unconfigured), fail-open telemetry, language-agnostic logic, Syncthing-peer
+determinism, and the demonstrated one-PR-themed-suite release cadence. The drive order
+A -> C -> D -> E -> B resolves every real cross-card collision by the cheapest sufficient
+mechanism (ordering) rather than a speculative up-front abstraction: A seeds the provider-registry
+pattern at the reader tail with no vault writes; C establishes the item-shape seam so D extends
+a merged shape rather than colliding on `context-pack.ts`/`context-receipts.ts`; D reuses
+existing gate-telemetry + coverage scores as a thin verdict+action layer (honoring the card's
+"no separate attention organ" guardrail); E appends at the recall seam D opened, reusing the
+coverage score as the satisfaction axis; B lands last as the capstone assembly over a
+fully-merged branch, extending the dream pass rather than running a parallel loop.
+
+Variant 1's shared annotation envelope turns bit-identity into a whole-subsystem property and
+front-loads an abstraction before its consumers exist. Variant 3's shared emission bus
+inverts the read->write flow and stresses the fail-open + Syncthing-determinism invariants on
+the hot path. Neither is warranted for this suite.
+
+Drive order: **A -> C -> D -> E -> B**.
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index d6d5c227..8661c415 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -426,6 +426,21 @@ flowchart LR
The writer is idempotent — if the rendered body matches the file
on disk, no I/O happens. The same window / limit drive a mirrored
`Most-applied (Nd)` section in `brain_digest` output.
+- **`Brain/lessons.md`** is a second derived digest that folds the
+ positive-knowledge (preferences + their `apply-evidence applied`
+ rows) and negative-knowledge (`violated` / `outdated` rows and
+ `Brain/dead-ends/`) into ONE outcome-tagged corpus scored by a
+ **signed, recency-decayed** weight (shared 30-day half-life from
+ working-memory continuity; tunable via `lessons.half_life_days`).
+ Each lesson lands in a tier: **preferred** (net-positive and
+ corroborated by ≥N distinct results — `lessons.corroboration_min`,
+ default 2), **tentative** (net-positive but under the corroboration
+ floor), **contested** (both positive and negative recent evidence —
+ the shown stance follows the sign of the decayed score, so recency
+ wins), or **avoid** (dead-ends and rules whose recent evidence is
+ net-negative). Rendered top-N (`lessons.limit`, default 20), ranked
+ by decayed salience. Regenerated at the tail of every `dream` pass
+ alongside `active.md`, with the same idempotent write.
- **SessionStart hook** (`startup | resume | clear | compact`) injects
the body as `additionalContext` so the agent sees current rules at
the start of every session and again after `/compact` - the
@@ -434,7 +449,9 @@ flowchart LR
budgeted (`active.inject_budget_chars`, default 8,000 chars):
sections drop deterministically (recently retired first, then
quarantine, then most-applied) and a one-line notice points the
- agent at `brain_context` for the full set. Fails closed - any error
+ agent at `brain_context` for the full set. When `Brain/lessons.md`
+ exists, its (separately budgeted) body is appended so the unified
+ lessons corpus loads on the same surface. Fails closed - any error
path exits 0 with no output so the runtime proceeds unaffected.
- **MCP Resources** expose the same content for hosts that prefer
pull access (`osb://preferences/active` and friends in the table
diff --git a/docs/mcp.md b/docs/mcp.md
index 1eaca976..16c4b8a3 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -46,10 +46,11 @@ flags for a narrower per-process full server.
| `brain_analytics` | Read-only Brain analytics for any lens: `view: timeline \| attention_flows \| belief_evolution \| concept_synthesis`. | `view` |
| `brain_search` | Read-only vault search with optional structured query lanes, explicit focus hints, time ranges, evidence-pack diagnostics, and a selectable recall `profile` (`fast \| balanced \| thorough`). | `query` |
| `brain_recall_feedback` | Record explicit up/down recall feedback for one search result; feeds the deterministic learned-weight fold. | `query`, `result_path`, `verdict` |
-| `brain_recall_gate` | Read-only classifier for whether an automatic recall attempt should run; returns `retrieve` plus a stable reason. | `prompt` |
-| `brain_context_pack` | Budgeted context slice; pass `lanes: true` to return directives, constraints, and consider lanes. Filtered items include `safety.reasons`. | `max_tokens` |
+| `brain_recall_gate` | Read-only classifier for whether an automatic recall attempt should run; returns `retrieve` plus a stable reason. When the caller passes `scores`, also attaches an adequacy verdict (`sufficient` \| `weak` \| `insufficient`), a recommended action (`proceed` \| `re_recall` \| `abstain`), and an optional `escalate` flag over the gate-telemetry relevance scores plus the epistemic mix; thresholds via `recall_adequacy_sufficient` / `recall_adequacy_weak` / `recall_adequacy_min_results`. | `prompt` |
+| `brain_context_pack` | Budgeted context slice; pass `lanes: true` to return directives, constraints, and consider lanes. Filtered items include `safety.reasons`. Each item carries a structural `epistemic` status (`observed` \| `derived` \| `hypothesis` \| `plan` \| `unknown`) plus `evidence_refs` derived from existing graph metadata; fields are absent when the status is `unknown`. | `max_tokens` |
| `brain_context_receipts` | List or show opt-in prompt context receipt continuity records with budgets, hashes, source refs, safety/redaction metadata, and item IDs. | `operation` |
| `brain_recall_telemetry` | List or summarise opt-in recall telemetry records for search, context-pack, and pre-compress calls. | `operation` |
+| `brain_knowledge_gaps` | Aggregate the persisted cross-query demand log into recurring queries the vault answers poorly, ranked by frequency × (1 − IDF-weighted coverage). Read-only; the log is written only by opt-in recall telemetry. | — |
| `brain_generation_reports` | Inbound, opt-in LLM generation tracing: `record` posts a generation's usage for a handoff (gated, default off; stores prompt hash + token counts only); `list`/`summary` read records and join them to memory paths. Kernel never calls an LLM. | `action` |
| `brain_obligation` | Recurring obligations under `Brain/obligations/` with a deterministic cadence-driven next-due date: `add`, `done` (advances next_due by one cadence interval), `list` (optionally overdue-only), `show`, `remove`. Cadences: daily/weekly/biweekly/monthly/quarterly/yearly/every-<N>-days. | `operation` |
| `brain_agenda` | Stateless agenda synthesis over caller-provided calendar events (the host fetches them; the Brain never calls a calendar API): overlap conflicts, free focus blocks (optionally clipped to a workday window), and events organised outside the operator's own email domain(s). No vault writes. | `events` |
@@ -148,6 +149,12 @@ come back from `resources/list`:
generated digest of confirmed + quarantined preferences plus the
last three retired entries. Auto-regenerated on first read if the
file does not exist yet.
+- `osb://lessons` — body of `Brain/lessons.md`, the auto-generated,
+ signed and recency-scored lessons corpus that unifies preferences and
+ dead-ends into corroboration-tiered lessons (`preferred` / `tentative`
+ / `contested` / `avoid`). Auto-regenerated on first read if the file
+ does not exist yet. The SessionStart / PostCompact hook injects it
+ alongside `active.md`.
- `osb://digest/latest` — same body as `brain_brief` `view="digest"`
in its default (24h) Markdown window.
- `osb://status` — Brain operational snapshot: counts (inbox /
diff --git a/hooks/active-inject.ts b/hooks/active-inject.ts
index fc40283a..96fa363c 100644
--- a/hooks/active-inject.ts
+++ b/hooks/active-inject.ts
@@ -31,7 +31,7 @@ import { existsSync, readFileSync } from "node:fs";
import { resolveVault } from "../src/core/config.ts";
import { parseFrontmatterText } from "../src/core/vault.ts";
-import { brainActivePath } from "../src/core/brain/paths.ts";
+import { brainActivePath, brainLessonsPath } from "../src/core/brain/paths.ts";
import { budgetActiveBody } from "../src/core/brain/active-budget.ts";
import { INJECT_BUDGET_CHARS_DEFAULT, loadBrainConfig } from "../src/core/brain/policy.ts";
import { healCliSymlinks } from "../src/cli/install-cli.ts";
@@ -121,15 +121,45 @@ async function main(): Promise<void> {
// intentional fallback - a corrupted _brain.yaml is doctor's job
}
+ // Auto-load the lessons digest alongside active.md so the agent gets
+ // the unified, signed, recency-scored corpus (preferences + dead-ends)
+ // on the same SessionStart surface. Fail-soft and budgeted separately:
+ // a missing / unreadable / oversized lessons file must never disturb
+ // the active-preferences injection above.
+ const lessonsBody = readLessonsBody(brainLessonsPath(vault), budget);
+
+ const additionalContext =
+ lessonsBody === null
+ ? budgetActiveBody(trimmed, budget)
+ : `${budgetActiveBody(trimmed, budget)}\n\n${lessonsBody}`;
+
const out = {
hookSpecificOutput: {
hookEventName,
- additionalContext: budgetActiveBody(trimmed, budget),
+ additionalContext,
},
};
process.stdout.write(JSON.stringify(out) + "\n");
}
+/**
+ * Read and budget the `Brain/lessons.md` body for injection. Returns
+ * `null` on any failure mode (missing file, unreadable, empty body) so
+ * the caller falls back to injecting active.md alone.
+ */
+function readLessonsBody(lessonsPath: string, budget: number): string | null {
+ if (!existsSync(lessonsPath)) return null;
+ try {
+ const raw = readFileSync(lessonsPath, "utf8");
+ const [, body] = parseFrontmatterText(raw);
+ const trimmed = body.trim();
+ if (trimmed.length === 0) return null;
+ return budgetActiveBody(trimmed, budget);
+ } catch {
+ return null;
+ }
+}
+
main().catch(() => {
// Never crash the runtime; the session start should proceed
// regardless of any hook misbehaviour.
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 67d20cbe..eb80dddc 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.21.0",
+ "version": "1.22.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index 92c8623a..a7f13e59 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.21.0",
+ "version": "1.22.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index 821e81fb..1f5c1458 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.21.0"
+version: "1.22.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index 75b37097..1237650e 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.21.0",
+ "version": "1.22.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 821e81fb..1f5c1458 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.21.0"
+version: "1.22.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index d193117f..2c5a61ab 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "1.21.0"
+version = "1.22.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/brain.ts b/src/cli/brain.ts
index 77f6f62d..d1b3a93c 100644
--- a/src/cli/brain.ts
+++ b/src/cli/brain.ts
@@ -103,6 +103,7 @@ import {
cmdBrainPreCompactExtract,
cmdBrainPostCompactAudit,
cmdBrainRecallTelemetry,
+ cmdBrainKnowledgeGaps,
cmdBrainGenerationReports,
cmdBrainSkillProposals,
cmdBrainProceduralMemory,
@@ -335,6 +336,8 @@ export async function handleBrainSubcommand(argv: ReadonlyArray<string>): Promis
return await cmdBrainPostCompactAudit(rest);
case "recall-telemetry":
return await cmdBrainRecallTelemetry(rest);
+ case "knowledge-gaps":
+ return await cmdBrainKnowledgeGaps(rest);
case "generation-reports":
return await cmdBrainGenerationReports(rest);
case "skill-proposals":
diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.ts
index d38c09d6..25c90a7d 100644
--- a/src/cli/brain/help-text.ts
+++ b/src/cli/brain/help-text.ts
@@ -86,6 +86,7 @@ Brain verbs (observing memory):
pre-compact-extract Extract typed continuity records from bounded text
post-compact-audit Audit pinned-anchor survival after a compaction; re-assert drifted anchors
recall-telemetry List/summarize opt-in recall telemetry records
+ knowledge-gaps Rank recurring queries the vault answers poorly (unmet demand)
generation-reports Record/list/summarize opt-in LLM generation traces
skill-proposals Learn/list/review deterministic skill proposals
procedural-memory Reconcile/list procedural memory index and usage
@@ -589,6 +590,9 @@ export const VERB_HELP: Record<string, string> = {
" o2b brain recall-telemetry summary [same filters] [--vault <path>] [--json]\n" +
" o2b brain recall-telemetry cost [--since <iso>] [--until <iso>] [--write-cost <n>] [--read-cost <n>] [--write-heavy-ratio <n>] [--vault <path>] [--json]\n" +
"Read opt-in recall telemetry continuity records and aggregate coverage gaps. cost folds write volume (feedback/apply-evidence/note/host writes) against reads into a write-vs-read ratio, a write-heavy flag, and a rough weighted cost signal per period.\n",
+ "knowledge-gaps":
+ "usage: o2b brain knowledge-gaps [--min-occurrences <n>] [--max-satisfaction <0..1>] [--since <iso>] [--until <iso>] [--limit <n>] [--vault <path>] [--json]\n" +
+ "Aggregate the persisted cross-query demand log (Brain/log/query-demand.jsonl) into recurring queries the vault answers poorly. Buckets by normalized query terms and ranks by frequency x (1 - satisfaction), where satisfaction is the reused IDF-weighted coverage (or the non-empty-result fraction when coverage was not recorded). Turns repeated recall failures into a prioritized backlog of what to write next. Read-only; the log is written only by opt-in recall telemetry.\n",
"generation-reports":
"usage: o2b brain generation-reports record <write_session|context_pack|dream_stage> --ref <id> --agent <name> --prompt <text> [--enable] [--provider <p>] [--model <m>] [--finish-reason <r>] [--latency-ms <n>] [--input-tokens <n>] [--output-tokens <n>] [--cached-tokens <n>] [--total-tokens <n>] [--scope <s>] [--source <id[=path]>...] [--created-at <iso>] [--vault <path>] [--json]\n" +
" o2b brain generation-reports list [--handoff <kind>] [--agent <name>] [--since <iso>] [--until <iso>] [--limit <n>] [--vault <path>] [--json]\n" +
diff --git a/src/cli/brain/verbs/context-pack.ts b/src/cli/brain/verbs/context-pack.ts
index 27c62326..628b483b 100644
--- a/src/cli/brain/verbs/context-pack.ts
+++ b/src/cli/brain/verbs/context-pack.ts
@@ -88,6 +88,8 @@ export async function cmdBrainContextPack(argv: string[]): Promise<number> {
path: i.path,
tier: i.tier,
tokens: i.tokens,
+ epistemic: i.epistemic,
+ ...(i.evidenceRefs.length > 0 ? { evidence_refs: i.evidenceRefs } : {}),
...(i.originalRank !== undefined ? { original_rank: i.originalRank } : {}),
...(i.stableRank !== undefined ? { stable_rank: i.stableRank } : {}),
...(i.dedupedFrom !== undefined ? { deduped_from: i.dedupedFrom } : {}),
@@ -106,7 +108,7 @@ export async function cmdBrainContextPack(argv: string[]): Promise<number> {
process.stdout.write(`pages skipped: ${report.skipped.length}\n\n`);
if (report.receiptId) process.stdout.write(`receipt: ${report.receiptId}\n`);
for (const i of report.items) {
- process.stdout.write(`[${i.tier}] ${i.id} (${i.tokens} tokens)\n`);
+ process.stdout.write(`[${i.tier}] ${i.epistemic} ${i.id} (${i.tokens} tokens)\n`);
}
return 0;
}
diff --git a/src/cli/brain/verbs/context-receipts.ts b/src/cli/brain/verbs/context-receipts.ts
index fe3cb7b6..f1f670db 100644
--- a/src/cli/brain/verbs/context-receipts.ts
+++ b/src/cli/brain/verbs/context-receipts.ts
@@ -82,6 +82,10 @@ function showReceipt(argv: string[]): number {
if (summary.turn_id) process.stdout.write(`turn_id: ${summary.turn_id}\n`);
process.stdout.write(`item_count: ${summary.item_count ?? "?"}\n`);
process.stdout.write(`source_count: ${summary.source_count}\n`);
+ if (summary.adequacy) {
+ const { level, action, escalate } = summary.adequacy;
+ process.stdout.write(`adequacy: ${level} -> ${action}${escalate ? " (escalate)" : ""}\n`);
+ }
return 0;
}
diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.ts
index 1bc20fd4..fb90913b 100644
--- a/src/cli/brain/verbs/index.ts
+++ b/src/cli/brain/verbs/index.ts
@@ -87,6 +87,7 @@ export { cmdBrainContextPresets } from "./context-presets.ts";
export { cmdBrainPreCompactExtract } from "./pre-compact-extract.ts";
export { cmdBrainPostCompactAudit } from "./post-compact-audit.ts";
export { cmdBrainRecallTelemetry } from "./recall-telemetry.ts";
+export { cmdBrainKnowledgeGaps } from "./knowledge-gaps.ts";
export { cmdBrainGenerationReports } from "./generation-reports.ts";
export { cmdBrainSkillProposals } from "./skill-proposals.ts";
export { cmdBrainProceduralMemory } from "./procedural-memory.ts";
diff --git a/src/cli/brain/verbs/knowledge-gaps.ts b/src/cli/brain/verbs/knowledge-gaps.ts
new file mode 100644
index 00000000..dedcb821
--- /dev/null
+++ b/src/cli/brain/verbs/knowledge-gaps.ts
@@ -0,0 +1,86 @@
+import {
+ aggregateQueryDemand,
+ serializeQueryDemandReport,
+} from "../../../core/brain/query-demand.ts";
+import { CliError, brainVerbContext, parse } from "../helpers.ts";
+
+/**
+ * `o2b brain knowledge-gaps` — surface recurring queries the vault
+ * answers poorly from the persisted cross-query demand log
+ * (t_97091fff). Read-only aggregation; no LLM.
+ */
+export async function cmdBrainKnowledgeGaps(argv: string[]): Promise<number> {
+ const { flags } = parse(argv, {
+ vault: { type: "string" },
+ json: { type: "boolean" },
+ since: { type: "string" },
+ until: { type: "string" },
+ "min-occurrences": { type: "string" },
+ "max-satisfaction": { type: "string" },
+ limit: { type: "string" },
+ });
+ const vault = brainVerbContext(flags).vault;
+ const since = trimOrUndefined(flags["since"]);
+ const until = trimOrUndefined(flags["until"]);
+ const minOccurrences = parsePositiveInteger(
+ trimOrUndefined(flags["min-occurrences"]),
+ "--min-occurrences",
+ );
+ const limit = parsePositiveInteger(trimOrUndefined(flags["limit"]), "--limit");
+ const maxSatisfaction = parseUnitInterval(
+ trimOrUndefined(flags["max-satisfaction"]),
+ "--max-satisfaction",
+ );
+
+ const report = aggregateQueryDemand(vault, {
+ ...(since !== undefined ? { since } : {}),
+ ...(until !== undefined ? { until } : {}),
+ ...(minOccurrences !== undefined ? { minOccurrences } : {}),
+ ...(maxSatisfaction !== undefined ? { maxSatisfaction } : {}),
+ ...(limit !== undefined ? { limit } : {}),
+ });
+
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(serializeQueryDemandReport(report), null, 2) + "\n");
+ return 0;
+ }
+
+ process.stdout.write(
+ `${report.totalRecords} demand record(s), ${report.distinctQueries} distinct quer(y/ies), ` +
+ `${report.gaps.length} gap(s):\n`,
+ );
+ for (const gap of report.gaps) {
+ const coverage = gap.meanCoverage === null ? "n/a" : gap.meanCoverage.toFixed(2);
+ process.stdout.write(
+ ` [${gap.demandScore.toFixed(2)}] ${gap.verdict} x${gap.occurrences} ` +
+ `coverage=${coverage} empty=${gap.emptyCount}/${gap.occurrences} ` +
+ `${gap.terms.join(" ")}\n`,
+ );
+ }
+ return 0;
+}
+
+function trimOrUndefined(value: string | boolean | string[] | undefined): string | undefined {
+ if (typeof value !== "string") return undefined;
+ const trimmed = value.trim();
+ return trimmed.length > 0 ? trimmed : undefined;
+}
+
+function parsePositiveInteger(value: string | undefined, flag: string): number | undefined {
+ if (value === undefined) return undefined;
+ if (!/^[0-9]+$/.test(value)) {
+ throw new CliError(`brain knowledge-gaps: ${flag} must be a positive integer`);
+ }
+ const parsed = Number.parseInt(value, 10);
+ if (parsed < 1) throw new CliError(`brain knowledge-gaps: ${flag} must be a positive integer`);
+ return parsed;
+}
+
+function parseUnitInterval(value: string | undefined, flag: string): number | undefined {
+ if (value === undefined) return undefined;
+ const parsed = Number(value);
+ if (!Number.isFinite(parsed) || parsed < 0 || parsed > 1) {
+ throw new CliError(`brain knowledge-gaps: ${flag} must be a number in [0, 1]`);
+ }
+ return parsed;
+}
diff --git a/src/core/brain/context-pack.ts b/src/core/brain/context-pack.ts
index 66738ed5..e1e03c88 100644
--- a/src/core/brain/context-pack.ts
+++ b/src/core/brain/context-pack.ts
@@ -26,10 +26,16 @@ import {
} from "./safety/context-guard.ts";
import { brainDirs } from "./paths.ts";
import { PAGE_TIER, readTier, type PageTier } from "./page-meta/tier.ts";
+import {
+ deriveEpistemicStatus,
+ EPISTEMIC_STATUS,
+ type EpistemicStatus,
+} from "./provenance/epistemic.ts";
import { estimateTokens } from "./text/tokenizer.ts";
import { normalizeForDedup } from "./text/normalize.ts";
import { applyCharBudget, type CharBudgetDegradationMode } from "./recall-budget.ts";
import { emitContextReceipt, type ContextReceiptOptions } from "./context-receipts.ts";
+import type { RecallAdequacyVerdict } from "./recall-adequacy.ts";
import { emitGatedTelemetry } from "./continuity/emit.ts";
import {
canonicalSegment,
@@ -73,6 +79,15 @@ export interface ContextPackItem extends ContextTransformAnnotations {
readonly contextLane: ContextLaneName | null;
/** True when `body` was truncated by `maxCharsPerMemory` (v0.20.0). */
readonly trimmed: boolean;
+ /**
+ * Epistemic grounding of this item (ACM), derived from the page's
+ * provenance / status / lifecycle metadata: `observed | derived |
+ * hypothesis | plan | unknown`. Lets downstream reasoning distinguish a
+ * grounded fact from a conjecture.
+ */
+ readonly epistemic: EpistemicStatus;
+ /** `evidenced_by` wikilinks grounding this item; empty when it cites none. */
+ readonly evidenceRefs: ReadonlyArray<string>;
/** Present when the surfaced body was filtered or explicitly trusted. */
readonly safety?: ContextSafetyReport;
}
@@ -120,6 +135,13 @@ export interface ContextPackOptions {
readonly includeLanes?: boolean;
/** Opt-in audit receipt for the final emitted context. */
readonly receipt?: ContextReceiptOptions;
+ /**
+ * Recall adequacy verdict (t_b8f66fec) for the recall that produced
+ * this pack. When present alongside `receipt`, it is persisted in the
+ * receipt so the audit trail records the sufficient/weak/insufficient
+ * verdict and the action the caller was told to take.
+ */
+ readonly recallAdequacy?: RecallAdequacyVerdict;
/** Opt-in telemetry for recall coverage and gap diagnostics. */
readonly telemetry?: RecallTelemetryOptions;
/**
@@ -153,6 +175,8 @@ interface Candidate {
readonly contextLane: ContextLaneName | null;
readonly body: string;
readonly tokens: number;
+ readonly epistemic: EpistemicStatus;
+ readonly evidenceRefs: ReadonlyArray<string>;
readonly safety?: ContextSafetyReport;
}
@@ -207,6 +231,7 @@ function collectCandidates(vault: string, delimitUntrusted: boolean): Candidate[
const topic = typeof meta["topic"] === "string" ? meta["topic"] : "";
const principle = typeof meta["principle"] === "string" ? meta["principle"] : "";
const contextLane = normalizeContextLane(meta["context_lane"]);
+ const epistemic = deriveEpistemicStatus(meta);
const guarded = guardBrainContextSnippet(body, {
source: { id, path: full, metadata: meta },
...(meta["context_safety"] === "trusted-instruction"
@@ -231,6 +256,8 @@ function collectCandidates(vault: string, delimitUntrusted: boolean): Candidate[
contextLane,
body: guarded.safeText,
tokens: estimateTokens(guarded.safeText),
+ epistemic: epistemic.status,
+ evidenceRefs: epistemic.evidenceRefs,
...(contextSafetyReport(guarded) ? { safety: contextSafetyReport(guarded) } : {}),
});
}
@@ -323,6 +350,8 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPac
principle: c.principle,
contextLane: c.contextLane,
trimmed,
+ epistemic: c.epistemic,
+ evidenceRefs: c.evidenceRefs,
...(c.safety ? { safety: c.safety } : {}),
});
used += tokens;
@@ -354,6 +383,10 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPac
principle: "Declarative attention flow output",
contextLane: "directives",
trimmed: false,
+ // Synthesized from the vault's attention-flow graph, not a
+ // source-backed page, so it is `derived` with no evidence refs.
+ epistemic: EPISTEMIC_STATUS.derived,
+ evidenceRefs: [],
...(safetyReport ? { safety: safetyReport } : {}),
});
used += tokens;
@@ -445,9 +478,12 @@ function finalizeContextPackReport(
tier: item.tier,
trimmed: item.trimmed,
safetyFiltered: item.safety?.filtered,
+ epistemic: item.epistemic,
+ evidenceRefs: item.evidenceRefs,
})),
finalText: report.items.map((item) => item.body).join("\n\n"),
budget: contextPackBudgetMetadata(opts, report),
+ ...(opts.recallAdequacy ? { adequacy: opts.recallAdequacy } : {}),
extra: {
skipped_count: report.skipped.length,
lanes: opts.includeLanes === true,
diff --git a/src/core/brain/context-receipts.ts b/src/core/brain/context-receipts.ts
index 8a79cbc0..8c103bf8 100644
--- a/src/core/brain/context-receipts.ts
+++ b/src/core/brain/context-receipts.ts
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
import { appendContinuityRecord, listContinuityRecords } from "./continuity/store.ts";
import type { ContinuityRecord } from "./continuity/types.ts";
+import type { RecallAdequacyVerdict } from "./recall-adequacy.ts";
export type ContextReceiptTrigger = "context_pack" | "pre_compress";
@@ -21,6 +22,10 @@ export interface ContextReceiptItemInput {
readonly tier?: string;
readonly trimmed?: boolean;
readonly safetyFiltered?: boolean;
+ /** Epistemic grounding of the item (ACM): observed | derived | ... */
+ readonly epistemic?: string;
+ /** `evidenced_by` wikilinks grounding the item. */
+ readonly evidenceRefs?: ReadonlyArray<string>;
}
export interface EmitContextReceiptInput {
@@ -29,6 +34,26 @@ export interface EmitContextReceiptInput {
readonly finalText: string;
readonly budget?: Readonly<Record<string, unknown>>;
readonly extra?: Readonly<Record<string, unknown>>;
+ /**
+ * Recall adequacy verdict (t_b8f66fec) for the recall that produced
+ * this context. When present it is persisted so the audit trail records
+ * whether the surfaced grounding was judged sufficient / weak /
+ * insufficient and what action the caller was told to take.
+ */
+ readonly adequacy?: RecallAdequacyVerdict;
+}
+
+/** Flatten a verdict into the snake_cased receipt payload shape. */
+export function serializeAdequacy(verdict: RecallAdequacyVerdict): Record<string, unknown> {
+ return {
+ level: verdict.level,
+ action: verdict.action,
+ escalate: verdict.escalate,
+ result_count: verdict.resultCount,
+ top_score: verdict.topScore,
+ mean_score: verdict.meanScore,
+ reason: verdict.reason,
+ };
}
export interface ContextReceiptFilter {
@@ -49,6 +74,12 @@ export interface ContextReceiptSummary {
readonly source_count: number;
readonly private: boolean;
readonly redacted: boolean;
+ /** Recall adequacy verdict summary, when the receipt recorded one. */
+ readonly adequacy?: {
+ readonly level: string;
+ readonly action: string;
+ readonly escalate: boolean;
+ };
}
export function emitContextReceipt(
@@ -64,6 +95,10 @@ export function emitContextReceipt(
...(item.tier ? { tier: item.tier } : {}),
...(item.trimmed !== undefined ? { trimmed: item.trimmed } : {}),
...(item.safetyFiltered !== undefined ? { safety_filtered: item.safetyFiltered } : {}),
+ ...(item.epistemic !== undefined ? { epistemic: item.epistemic } : {}),
+ ...(item.evidenceRefs && item.evidenceRefs.length > 0
+ ? { evidence_refs: [...item.evidenceRefs] }
+ : {}),
...(item.text ? { text_hash: sha256(item.text) } : {}),
}));
const payload: Record<string, unknown> = {
@@ -76,6 +111,7 @@ export function emitContextReceipt(
final_text_chars: [...input.finalText].length,
items: itemPayloads,
...(input.budget ? { budget: input.budget } : {}),
+ ...(input.adequacy ? { adequacy: serializeAdequacy(input.adequacy) } : {}),
};
mergeReceiptExtra(payload, input.extra);
return appendContinuityRecord(vault, {
@@ -136,9 +172,21 @@ export function summarizeContextReceipt(record: ContinuityRecord): ContextReceip
source_count: record.sourceRefs.length,
private: record.private,
redacted: record.redacted,
+ ...adequacySummary(payload["adequacy"]),
};
}
+function adequacySummary(
+ raw: unknown,
+): { adequacy: { level: string; action: string; escalate: boolean } } | Record<string, never> {
+ if (typeof raw !== "object" || raw === null) return {};
+ const record = raw as Record<string, unknown>;
+ const level = record["level"];
+ const action = record["action"];
+ if (typeof level !== "string" || typeof action !== "string") return {};
+ return { adequacy: { level, action, escalate: record["escalate"] === true } };
+}
+
export function isContextReceiptTrigger(value: unknown): value is ContextReceiptTrigger {
return value === "context_pack" || value === "pre_compress";
}
diff --git a/src/core/brain/dream.ts b/src/core/brain/dream.ts
index 479284e8..32392155 100644
--- a/src/core/brain/dream.ts
+++ b/src/core/brain/dream.ts
@@ -39,6 +39,7 @@ import { basename, join } from "node:path";
import { parseFrontmatter } from "../vault.ts";
import { regenerateActiveQuiet } from "./active.ts";
+import { regenerateLessonsQuiet } from "./lessons.ts";
import { openWorkrun, WORKRUN_PHASE, type WorkrunHandle } from "./dream-workrun.ts";
import { DREAM_PHASE, type DreamPhase, type DreamPhaseSummary } from "./dream-phases.ts";
import { extractTemporalConstraints } from "./temporal-extract.ts";
@@ -345,7 +346,10 @@ export function dream(vault: string, opts: DreamOptions = {}): DreamRunSummary {
scan.corrupted.length > 0;
if (!changed) {
- if (!dryRun) regenerateActiveQuiet(vault, { now });
+ if (!dryRun) {
+ regenerateActiveQuiet(vault, { now });
+ regenerateLessonsQuiet(vault, { now });
+ }
return Object.freeze({
run_id: runId,
changed: false,
@@ -772,6 +776,7 @@ export function dream(vault: string, opts: DreamOptions = {}): DreamRunSummary {
process.stderr.write(`warning: prune snapshots failed: ${(err as Error).message}\n`);
}
regenerateActiveQuiet(vault, { now });
+ regenerateLessonsQuiet(vault, { now });
}
// v0.12.0 Brain Integrity Suite: finalise the durable workrun
diff --git a/src/core/brain/index.ts b/src/core/brain/index.ts
index 15adb286..dabf244f 100644
--- a/src/core/brain/index.ts
+++ b/src/core/brain/index.ts
@@ -73,6 +73,23 @@ export {
export { regenerateActive, regenerateActiveQuiet } from "./active.ts";
export type { RegenerateActiveOptions, RegenerateActiveResult } from "./active.ts";
+// ----- Lessons digest (signed, recency-scored, corroboration-tiered) --------
+export {
+ computeLessons,
+ regenerateLessons,
+ regenerateLessonsQuiet,
+ LESSON_TIER,
+ LESSON_STANCE,
+} from "./lessons.ts";
+export type {
+ LessonEntry,
+ LessonTier,
+ LessonStance,
+ ComputeLessonsOptions,
+ RegenerateLessonsOptions,
+ RegenerateLessonsResult,
+} from "./lessons.ts";
+
// ----- Backlink index -------------------------------------------------------
export { buildBacklinkIndex, backlinkCount } from "./backlinks.ts";
export type { BacklinkIndex, BacklinkRef, BacklinkSourceKind } from "./backlinks.ts";
diff --git a/src/core/brain/lessons.ts b/src/core/brain/lessons.ts
new file mode 100644
index 00000000..7866949b
--- /dev/null
+++ b/src/core/brain/lessons.ts
@@ -0,0 +1,555 @@
+/**
+ * `Brain/lessons.md` — the unified, signed, recency-scored lessons
+ * digest (Outcome-tagged lessons loop, t_62363378).
+ *
+ * OSB already owns the two halves of a lessons corpus as separate
+ * registries:
+ *
+ * - NEGATIVE knowledge — `Brain/dead-ends/` ("avoid X"), and the
+ * `violated` / `outdated` apply-evidence rows against a preference;
+ * - POSITIVE knowledge — `Brain/preferences/` confirmed rules and the
+ * `applied` apply-evidence rows that corroborate them.
+ *
+ * This module folds both classes into ONE outcome-tagged corpus scored
+ * by a SIGNED, recency-decayed weight so that a fresh dead-end
+ * outweighs a stale "useful", and promotes a lesson to a corroborated
+ * tier only once ≥N distinct results back it — otherwise it stays
+ * tentative, and a rule with both positive and negative recent
+ * evidence renders as CONTESTED (the displayed stance follows the sign
+ * of the decayed score, so recency wins).
+ *
+ * It is NOT a parallel loop: the corpus is derived from the existing
+ * dead-end notes and apply-evidence log, the decay curve is the shared
+ * `decayWeight` from continuity's usage-signal, and the digest is
+ * regenerated at the tail of every `dream` pass exactly like
+ * `Brain/active.md`. The SessionStart / PostCompact hook injects it
+ * alongside `active.md`, so the agent gets a single auto-loaded lessons
+ * surface without querying dead-ends and preferences independently.
+ *
+ * Pure derivation: no LLM, no network, no clock-dependent content
+ * beyond the `generated_at` stamp. Identical inputs produce identical
+ * bodies; the write is idempotent and atomic (mirrors `active.ts`).
+ */
+
+import { existsSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+
+import { atomicWriteFileSync } from "../fs-atomic.ts";
+import { parseFrontmatter } from "../vault.ts";
+import { decayWeight } from "./continuity/usage-signal.ts";
+import { listDeadEnds, type DeadEndEntry } from "./dead-ends.ts";
+import { listLogDates, readLogDay } from "./log-jsonl.ts";
+import { brainDirs, brainLessonsPath } from "./paths.ts";
+import { parsePreference } from "./preference.ts";
+import {
+ LESSONS_CORROBORATION_MIN_DEFAULT,
+ LESSONS_HALF_LIFE_DAYS_DEFAULT,
+ LESSONS_LIMIT_DEFAULT,
+ loadBrainConfig,
+} from "./policy.ts";
+import { isoSecond } from "./time.ts";
+import { normaliseWikilinkTarget } from "./wikilink.ts";
+import {
+ BRAIN_APPLY_RESULT,
+ BRAIN_LOG_EVENT_KIND,
+ BRAIN_PREFERENCE_STATUS,
+ type BrainPreference,
+} from "./types.ts";
+
+const FRONTMATTER_KIND = "brain-lessons";
+
+/**
+ * A net-positive lesson below the corroboration floor is `tentative`;
+ * once ≥N distinct results back it, `preferred`. A rule carrying BOTH
+ * positive and negative recent evidence is `contested` (recency-wins:
+ * the displayed stance follows the sign of the decayed score). A
+ * purely-negative node (a dead-end, or a rule only ever violated) is
+ * `avoid`.
+ */
+export const LESSON_TIER = {
+ preferred: "preferred",
+ tentative: "tentative",
+ contested: "contested",
+ avoid: "avoid",
+} as const;
+export type LessonTier = (typeof LESSON_TIER)[keyof typeof LESSON_TIER];
+
+/** The stance a `contested` lesson currently leans toward. */
+export const LESSON_STANCE = {
+ positive: "positive",
+ negative: "negative",
+} as const;
+export type LessonStance = (typeof LESSON_STANCE)[keyof typeof LESSON_STANCE];
+
+export interface LessonEntry {
+ /** `pref-<slug>` or `de-<date>-<slug>`. */
+ readonly id: string;
+ readonly kind: "preference" | "dead-end";
+ /** Preference principle or dead-end approach. */
+ readonly title: string;
+ readonly scope: string | null;
+ readonly tier: LessonTier;
+ /** Positive decayed mass minus negative decayed mass, rounded to 4dp. */
+ readonly signedScore: number;
+ readonly positiveMass: number;
+ readonly negativeMass: number;
+ /** Distinct positive-outcome artifacts corroborating the lesson. */
+ readonly corroboration: number;
+ /** For `contested` lessons, the recency-decided leaning; else null. */
+ readonly stance: LessonStance | null;
+ /** ISO timestamp of the most recent contributing outcome, or null. */
+ readonly lastOutcomeAt: string | null;
+}
+
+export interface ComputeLessonsOptions {
+ /** Scoring anchor. Defaults to `new Date()`. */
+ readonly now?: Date;
+ /** Exponential half-life of the recency decay, in days. Default 30. */
+ readonly halfLifeDays?: number;
+ /** Distinct-result count required for the `preferred` tier. Default 2. */
+ readonly corroborationMin?: number;
+ /** Max lessons returned. Default 20. */
+ readonly limit?: number;
+}
+
+/** Mutable per-node accumulator during the scoring pass. */
+interface Accumulator {
+ positiveMass: number;
+ negativeMass: number;
+ readonly positiveArtifacts: Set<string>;
+ lastOutcomeMs: number;
+}
+
+const EPSILON = 1e-9;
+
+/**
+ * Single-event recency weight from the shared continuity decay curve.
+ * Modelled as a record with zero access count and no last-access, so
+ * `decayWeight` reduces to the pure `0.5^(age/halfLife)` exponential
+ * (floored at its `minWeight`) — the same half-life OSB uses for
+ * working-memory continuity.
+ */
+function eventWeight(eventMs: number, nowMs: number, halfLifeDays: number): number {
+ return decayWeight({ createdAtMs: eventMs, accessCount: 0, lastAccessAtMs: null }, nowMs, {
+ halfLifeDays,
+ });
+}
+
+/**
+ * Fold the preference corpus and the dead-end corpus into one ranked,
+ * signed, recency-scored, corroboration-tiered lesson list.
+ *
+ * Pure read. The caller passes the candidate preferences (already
+ * filtered to `confirmed | quarantine`) and the active dead-ends;
+ * apply-evidence outcomes are harvested from `Brain/log/`.
+ */
+export function computeLessons(
+ vault: string,
+ preferences: ReadonlyArray<BrainPreference>,
+ deadEnds: ReadonlyArray<DeadEndEntry>,
+ opts: ComputeLessonsOptions = {},
+): ReadonlyArray<LessonEntry> {
+ const now = opts.now ?? new Date();
+ const nowMs = now.getTime();
+ const halfLifeDays = opts.halfLifeDays ?? LESSONS_HALF_LIFE_DAYS_DEFAULT;
+ const corroborationMin = opts.corroborationMin ?? LESSONS_CORROBORATION_MIN_DEFAULT;
+ const limit = opts.limit ?? LESSONS_LIMIT_DEFAULT;
+
+ // Index the candidate preferences by their normalised id so an
+ // apply-evidence row spelled `[[pref-x|principle]]` resolves to one
+ // node. Retired rules are excluded upstream and never scored here.
+ const prefByKey = new Map<string, BrainPreference>();
+ for (const p of preferences) {
+ if (
+ p.status !== BRAIN_PREFERENCE_STATUS.confirmed &&
+ p.status !== BRAIN_PREFERENCE_STATUS.quarantine
+ ) {
+ continue;
+ }
+ const key = normaliseWikilinkTarget(p.id);
+ if (key !== null) prefByKey.set(key, p);
+ }
+
+ const accByKey = new Map<string, Accumulator>();
+ const accFor = (key: string): Accumulator => {
+ let acc = accByKey.get(key);
+ if (acc === undefined) {
+ acc = { positiveMass: 0, negativeMass: 0, positiveArtifacts: new Set(), lastOutcomeMs: 0 };
+ accByKey.set(key, acc);
+ }
+ return acc;
+ };
+
+ // One pass over every log day, bucketing signed outcomes by pref.
+ if (prefByKey.size > 0) scoreApplyEvidence(vault, prefByKey, accFor, nowMs, halfLifeDays);
+
+ const entries: LessonEntry[] = [];
+
+ // Preference-derived lessons.
+ for (const [key, pref] of prefByKey) {
+ const acc = accByKey.get(key);
+ if (acc === undefined) continue; // no outcome evidence → not a lesson yet
+ entries.push(
+ buildEntry({
+ id: pref.id,
+ kind: "preference",
+ title: pref.principle,
+ scope: pref.scope ?? null,
+ acc,
+ corroborationMin,
+ }),
+ );
+ }
+
+ // Dead-end lessons: one negative outcome stamped at creation. They
+ // are inherently `avoid` — a recorded failed approach.
+ for (const de of deadEnds) {
+ const createdMs = Date.parse(de.created_at);
+ if (Number.isNaN(createdMs)) continue;
+ const weight = eventWeight(createdMs, nowMs, halfLifeDays);
+ const acc: Accumulator = {
+ positiveMass: 0,
+ negativeMass: weight,
+ positiveArtifacts: new Set(),
+ lastOutcomeMs: createdMs,
+ };
+ entries.push(
+ buildEntry({
+ id: de.id,
+ kind: "dead-end",
+ title: de.approach,
+ scope: null,
+ acc,
+ corroborationMin,
+ }),
+ );
+ }
+
+ // Rank by salience — the dominant decayed mass — so the strongest
+ // signal (positive, negative, or contested) surfaces first. Ties
+ // break to the most recent outcome, then to a stable id order.
+ entries.sort((a, b) => {
+ const sa = Math.max(a.positiveMass, a.negativeMass);
+ const sb = Math.max(b.positiveMass, b.negativeMass);
+ if (sb !== sa) return sb - sa;
+ const ta = a.lastOutcomeAt ? Date.parse(a.lastOutcomeAt) : 0;
+ const tb = b.lastOutcomeAt ? Date.parse(b.lastOutcomeAt) : 0;
+ if (tb !== ta) return tb - ta;
+ return a.id.localeCompare(b.id);
+ });
+
+ return Object.freeze(entries.slice(0, limit).map((e) => Object.freeze(e)));
+}
+
+/** Walk the log once, accumulating signed apply-evidence per pref. */
+function scoreApplyEvidence(
+ vault: string,
+ prefByKey: ReadonlyMap<string, BrainPreference>,
+ accFor: (key: string) => Accumulator,
+ nowMs: number,
+ halfLifeDays: number,
+): void {
+ const dirs = brainDirs(vault);
+ if (!existsSync(dirs.log)) return;
+
+ for (const datePrefix of listLogDates(vault)) {
+ let parsed;
+ try {
+ parsed = readLogDay(vault, datePrefix);
+ } catch (err) {
+ process.stderr.write(
+ `warning: lessons: failed to read ${datePrefix}: ${(err as Error).message}\n`,
+ );
+ continue;
+ }
+ for (const e of parsed.entries) {
+ if (e.eventType !== BRAIN_LOG_EVENT_KIND.applyEvidence) continue;
+ const prefField = e.body["preference"];
+ if (typeof prefField !== "string") continue;
+ const key = normaliseWikilinkTarget(prefField);
+ if (key === null) continue;
+ const pref = prefByKey.get(key);
+ if (pref === undefined) continue;
+ // Never harvest rows that predate the pref's existence — that would
+ // be a different rule under a reused slug history.
+ if (e.timestamp < pref.created_at) continue;
+ const eventMs = Date.parse(e.timestamp);
+ if (Number.isNaN(eventMs)) continue;
+ const result = e.body["result"];
+ if (typeof result !== "string") continue;
+
+ const weight = eventWeight(eventMs, nowMs, halfLifeDays);
+ const acc = accFor(key);
+ acc.lastOutcomeMs = Math.max(acc.lastOutcomeMs, eventMs);
+ if (result === BRAIN_APPLY_RESULT.applied) {
+ acc.positiveMass += weight;
+ const artifact = e.body["artifact"];
+ if (typeof artifact === "string" && artifact.trim().length > 0) {
+ acc.positiveArtifacts.add(normaliseWikilinkTarget(artifact) ?? artifact.trim());
+ }
+ } else if (result === BRAIN_APPLY_RESULT.violated || result === BRAIN_APPLY_RESULT.outdated) {
+ acc.negativeMass += weight;
+ }
+ }
+ }
+}
+
+interface BuildEntryInput {
+ readonly id: string;
+ readonly kind: "preference" | "dead-end";
+ readonly title: string;
+ readonly scope: string | null;
+ readonly acc: Accumulator;
+ readonly corroborationMin: number;
+}
+
+function buildEntry(input: BuildEntryInput): LessonEntry {
+ const { acc } = input;
+ const signed = acc.positiveMass - acc.negativeMass;
+ const corroboration = acc.positiveArtifacts.size;
+
+ let tier: LessonTier;
+ let stance: LessonStance | null = null;
+ if (acc.positiveMass > EPSILON && acc.negativeMass > EPSILON) {
+ tier = LESSON_TIER.contested;
+ // Recency-wins: the decayed sum already leans toward whichever
+ // outcome class is fresher / stronger. Treat a dead-even score as
+ // negative — an unresolved conflict is not an endorsement.
+ stance = signed > EPSILON ? LESSON_STANCE.positive : LESSON_STANCE.negative;
+ } else if (signed > EPSILON) {
+ tier = corroboration >= input.corroborationMin ? LESSON_TIER.preferred : LESSON_TIER.tentative;
+ } else {
+ tier = LESSON_TIER.avoid;
+ }
+
+ return {
+ id: input.id,
+ kind: input.kind,
+ title: input.title,
+ scope: input.scope,
+ tier,
+ signedScore: round4(signed),
+ positiveMass: round4(acc.positiveMass),
+ negativeMass: round4(acc.negativeMass),
+ corroboration,
+ stance,
+ lastOutcomeAt: acc.lastOutcomeMs > 0 ? new Date(acc.lastOutcomeMs).toISOString() : null,
+ };
+}
+
+function round4(n: number): number {
+ return Math.round(n * 10000) / 10000;
+}
+
+// ----- Regeneration (mirrors active.ts) ------------------------------------
+
+export interface RegenerateLessonsOptions {
+ /** Wall clock for `generated_at`. Defaults to `new Date()`. */
+ readonly now?: Date;
+}
+
+export interface RegenerateLessonsResult {
+ readonly path: string;
+ readonly changed: boolean;
+ readonly counts: {
+ readonly preferred: number;
+ readonly tentative: number;
+ readonly contested: number;
+ readonly avoid: number;
+ readonly total: number;
+ };
+}
+
+/**
+ * Regenerate `<vault>/Brain/lessons.md`. Idempotent: the rendered body
+ * is compared byte-for-byte against the existing file before writing so
+ * a no-op `dream` rerun leaves the file (and its mtime) untouched.
+ *
+ * Parser failures on individual preference / dead-end files are
+ * swallowed and the offending file is omitted, exactly like
+ * `regenerateActive` — a single corrupted frontmatter must not blank
+ * the whole lessons view. `brain_doctor` is the surface for corruption.
+ */
+export function regenerateLessons(
+ vault: string,
+ opts: RegenerateLessonsOptions = {},
+): RegenerateLessonsResult {
+ const now = opts.now ?? new Date();
+ const path = brainLessonsPath(vault);
+
+ const preferences = readActivePreferences(vault);
+ const { entries: deadEnds } = listDeadEnds(vault);
+
+ // Tunables from `_brain.yaml:lessons.*`. A malformed config never
+ // blocks the digest — fall back to the defaults (doctor's job).
+ let halfLifeDays = LESSONS_HALF_LIFE_DAYS_DEFAULT;
+ let corroborationMin = LESSONS_CORROBORATION_MIN_DEFAULT;
+ let limit = LESSONS_LIMIT_DEFAULT;
+ try {
+ const cfg = loadBrainConfig(vault);
+ if (cfg.lessons) {
+ halfLifeDays = cfg.lessons.half_life_days ?? halfLifeDays;
+ corroborationMin = cfg.lessons.corroboration_min ?? corroborationMin;
+ limit = cfg.lessons.limit ?? limit;
+ }
+ } catch {
+ // intentional fallback — config error is doctor's job to surface
+ }
+
+ const lessons = computeLessons(vault, preferences, deadEnds, {
+ now,
+ halfLifeDays,
+ corroborationMin,
+ limit,
+ });
+
+ const body = renderBody(lessons, { corroborationMin, halfLifeDays });
+ const existingBody = readExistingBody(path);
+ const changed = existingBody === null || existingBody !== body.trim();
+ if (changed) {
+ atomicWriteFileSync(path, renderDocument(body, isoSecond(now)));
+ }
+
+ const counts = {
+ preferred: lessons.filter((l) => l.tier === LESSON_TIER.preferred).length,
+ tentative: lessons.filter((l) => l.tier === LESSON_TIER.tentative).length,
+ contested: lessons.filter((l) => l.tier === LESSON_TIER.contested).length,
+ avoid: lessons.filter((l) => l.tier === LESSON_TIER.avoid).length,
+ total: lessons.length,
+ };
+ return { path, changed, counts };
+}
+
+/**
+ * Fire-and-warn wrapper around {@link regenerateLessons}, matching
+ * `regenerateActiveQuiet`. The lessons digest is a derived view: a
+ * failed write (disk full, permissions) must not mask the success of
+ * the `dream` pass that triggered it. The next pass retries.
+ */
+export function regenerateLessonsQuiet(vault: string, opts: RegenerateLessonsOptions = {}): void {
+ try {
+ regenerateLessons(vault, opts);
+ } catch (err) {
+ process.stderr.write(`warning: regenerate lessons.md failed: ${(err as Error).message}\n`);
+ }
+}
+
+// ----- Scan helpers --------------------------------------------------------
+
+function readActivePreferences(vault: string): BrainPreference[] {
+ const dirs = brainDirs(vault);
+ if (!existsSync(dirs.preferences)) return [];
+ const out: BrainPreference[] = [];
+ for (const name of readdirSync(dirs.preferences)) {
+ if (!name.endsWith(".md")) continue;
+ try {
+ out.push(parsePreference(join(dirs.preferences, name)));
+ } catch {
+ // Corrupted / status-folder-mismatched — omit; doctor flags it.
+ }
+ }
+ return out;
+}
+
+// ----- Render helpers ------------------------------------------------------
+
+interface RenderMeta {
+ readonly corroborationMin: number;
+ readonly halfLifeDays: number;
+}
+
+function renderBody(lessons: ReadonlyArray<LessonEntry>, meta: RenderMeta): string {
+ const out: string[] = [];
+ out.push("# Lessons");
+ out.push("");
+ out.push("Auto-generated by `dream`. Do not edit — changes will be overwritten.");
+ out.push("");
+ out.push(
+ `Signed, recency-decayed outcome corpus (half-life ${meta.halfLifeDays}d). ` +
+ `A rule reaches **preferred** once ≥${meta.corroborationMin} distinct results corroborate it.`,
+ );
+ out.push("");
+
+ if (lessons.length === 0) {
+ out.push("_No lessons yet — record apply-evidence outcomes and dead-ends to seed the corpus._");
+ while (out.length > 0 && out[out.length - 1] === "") out.pop();
+ return out.join("\n") + "\n";
+ }
+
+ const sections: ReadonlyArray<{ tier: LessonTier; heading: string; blurb: string }> = [
+ {
+ tier: LESSON_TIER.preferred,
+ heading: "Preferred",
+ blurb: "Net-positive and corroborated by enough distinct results.",
+ },
+ {
+ tier: LESSON_TIER.tentative,
+ heading: "Tentative",
+ blurb: "Net-positive but not yet corroborated by enough distinct results.",
+ },
+ {
+ tier: LESSON_TIER.contested,
+ heading: "Contested",
+ blurb: "Mixed recent evidence — the stance shown is the one recency currently favours.",
+ },
+ {
+ tier: LESSON_TIER.avoid,
+ heading: "Avoid",
+ blurb: "Dead-ends and rules whose recent evidence is net-negative.",
+ },
+ ];
+
+ for (const section of sections) {
+ const group = lessons.filter((l) => l.tier === section.tier);
+ if (group.length === 0) continue;
+ out.push(`## ${section.heading} (${group.length})`);
+ out.push("");
+ out.push(`_${section.blurb}_`);
+ out.push("");
+ for (const l of group) out.push(renderLine(l));
+ out.push("");
+ }
+
+ while (out.length > 0 && out[out.length - 1] === "") out.pop();
+ return out.join("\n") + "\n";
+}
+
+function renderLine(l: LessonEntry): string {
+ const tags: string[] = [];
+ if (l.scope) tags.push(`scope: ${l.scope}`);
+ tags.push(`score: ${formatSigned(l.signedScore)}`);
+ if (l.tier === LESSON_TIER.preferred || l.tier === LESSON_TIER.tentative) {
+ tags.push(`corroboration: ${l.corroboration}`);
+ }
+ if (l.tier === LESSON_TIER.contested && l.stance) {
+ tags.push(`leaning: ${l.stance}`);
+ }
+ return `- \`${l.id}\` (${tags.join(", ")}) — ${l.title}`;
+}
+
+function formatSigned(n: number): string {
+ const fixed = n.toFixed(2);
+ return n > 0 ? `+${fixed}` : fixed;
+}
+
+function renderDocument(body: string, generatedAt: string): string {
+ return [
+ "---",
+ `kind: ${FRONTMATTER_KIND}`,
+ `generated_at: ${generatedAt}`,
+ "---",
+ "",
+ body.trimEnd(),
+ "",
+ ].join("\n");
+}
+
+function readExistingBody(path: string): string | null {
+ if (!existsSync(path)) return null;
+ try {
+ const [, body] = parseFrontmatter(path);
+ return body;
+ } catch {
+ return null;
+ }
+}
diff --git a/src/core/brain/paths.ts b/src/core/brain/paths.ts
index d7633c26..ee51ac4f 100644
--- a/src/core/brain/paths.ts
+++ b/src/core/brain/paths.ts
@@ -72,6 +72,7 @@ export const BRAIN_ARTIFACTS_REL = posix.join(BRAIN_ROOT_REL, ".artifacts");
export const BRAIN_CONFIG_FILE = "_brain.yaml";
export const BRAIN_MANUAL_FILE = "_BRAIN.md";
export const BRAIN_ACTIVE_FILE = "active.md";
+export const BRAIN_LESSONS_FILE = "lessons.md";
export const BRAIN_PINNED_FILE = "pinned.md";
export const BRAIN_INDEX_FILE = "_INDEX.md";
@@ -143,6 +144,17 @@ export function brainActivePath(vault: string): string {
return ensureInsideVault(join(brainDirs(vault).brain, BRAIN_ACTIVE_FILE), vault);
}
+/**
+ * Path of the auto-generated lessons digest (`Brain/lessons.md`) written
+ * by `dream`: the unified, signed, recency-scored corpus over
+ * preferences and dead-ends. Read by the `SessionStart` / `PostCompact`
+ * hook alongside `active.md` and exposed as the MCP resource
+ * `osb://lessons`.
+ */
+export function brainLessonsPath(vault: string): string {
+ return ensureInsideVault(join(brainDirs(vault).brain, BRAIN_LESSONS_FILE), vault);
+}
+
/** Path of the transient current-task scratchpad read by `brain_context`. */
export function brainPinnedPath(vault: string): string {
return ensureInsideVault(join(brainDirs(vault).brain, BRAIN_PINNED_FILE), vault);
@@ -274,6 +286,16 @@ export function proceduralRecurrencePath(vault: string): string {
return ensureInsideVault(join(vault, BRAIN_LOG_REL, "recurrence-support.jsonl"), vault);
}
+/**
+ * Cross-query demand ledger path: `Brain/log/query-demand.jsonl`. A
+ * rolling, byte-budget-capped append-only log of normalized recall
+ * queries with their result count and IDF-weighted coverage, aggregated
+ * to surface recurring queries the vault answers poorly (unmet demand).
+ */
+export function queryDemandLogPath(vault: string): string {
+ return ensureInsideVault(join(vault, BRAIN_LOG_REL, "query-demand.jsonl"), vault);
+}
+
/** Log file for the given UTC date: `Brain/log/<YYYY-MM-DD>.md`. */
export function logPath(vault: string, date: string): string {
const d = validateIsoDate(date);
diff --git a/src/core/brain/policy.ts b/src/core/brain/policy.ts
index 670a88fc..ceea5337 100644
--- a/src/core/brain/policy.ts
+++ b/src/core/brain/policy.ts
@@ -30,6 +30,7 @@ import type {
BrainGuardrailConfig,
BrainHealthConfig,
BrainHygieneConfig,
+ BrainLessonsConfig,
BrainRecallConfig,
BrainLinkGraphConfig,
BrainNotesConfig,
@@ -87,6 +88,25 @@ export const INJECT_BUDGET_CHARS_MAX = 200_000;
/** Default top-N limit when `_brain.yaml` lacks `active.most_applied.limit`. */
export const MOST_APPLIED_LIMIT_DEFAULT = 10;
+/**
+ * Bounds and defaults for the `lessons:` block (t_62363378) that tunes
+ * the signed, recency-scored lessons digest (`Brain/lessons.md`).
+ * Out-of-range values are hard errors — an operator-tunable knob must
+ * never silently clamp. The half-life default matches the working-memory
+ * continuity decay (30 days); the corroboration floor (2 distinct
+ * results) keeps a one-off application from promoting straight to
+ * `preferred`.
+ */
+export const LESSONS_HALF_LIFE_DAYS_DEFAULT = 30;
+export const LESSONS_HALF_LIFE_DAYS_MIN = 1;
+export const LESSONS_HALF_LIFE_DAYS_MAX = 365;
+export const LESSONS_CORROBORATION_MIN_DEFAULT = 2;
+export const LESSONS_CORROBORATION_MIN_MIN = 1;
+export const LESSONS_CORROBORATION_MIN_MAX = 100;
+export const LESSONS_LIMIT_DEFAULT = 20;
+export const LESSONS_LIMIT_MIN = 1;
+export const LESSONS_LIMIT_MAX = 200;
+
/**
* Hard upper bound on `guardrails.instruction_file_max_lines`. Any
* value above this is a misconfiguration - vault-root instruction
@@ -913,6 +933,56 @@ export function validateBrainConfigDetailed(
};
}
+ // Optional `lessons:` block (t_62363378). Tunes the signed,
+ // recency-scored lessons digest. Hard-error on out-of-range values —
+ // operator-tunable knobs must never silently fall back to defaults.
+ let lessons: BrainLessonsConfig | undefined;
+ if ("lessons" in obj) {
+ const rawLessons = obj["lessons"];
+ if (typeof rawLessons !== "object" || rawLessons === null || Array.isArray(rawLessons)) {
+ throw new BrainConfigError(
+ `block must be a map of keys; got ${describe(rawLessons)}`,
+ "lessons",
+ source,
+ );
+ }
+ const lessonsMap = rawLessons as Record<string, unknown>;
+ const halfLife = readBoundedInt(
+ lessonsMap,
+ "half_life_days",
+ LESSONS_HALF_LIFE_DAYS_MIN,
+ LESSONS_HALF_LIFE_DAYS_MAX,
+ source,
+ );
+ const corroborationMin = readBoundedInt(
+ lessonsMap,
+ "corroboration_min",
+ LESSONS_CORROBORATION_MIN_MIN,
+ LESSONS_CORROBORATION_MIN_MAX,
+ source,
+ );
+ const lessonsLimit = readBoundedInt(
+ lessonsMap,
+ "limit",
+ LESSONS_LIMIT_MIN,
+ LESSONS_LIMIT_MAX,
+ source,
+ );
+ for (const k of Object.keys(lessonsMap)) {
+ if (k !== "half_life_days" && k !== "corroboration_min" && k !== "limit") {
+ warnings.push({
+ path: source ?? "<config>",
+ message: `lessons.${k}: unknown field ignored (forward-compat)`,
+ });
+ }
+ }
+ lessons = {
+ ...(halfLife !== undefined ? { half_life_days: halfLife } : {}),
+ ...(corroborationMin !== undefined ? { corroboration_min: corroborationMin } : {}),
+ ...(lessonsLimit !== undefined ? { limit: lessonsLimit } : {}),
+ };
+ }
+
// Optional `discipline_report` section. On any type mismatch, emit a
// warning and drop the section (return undefined) rather than throwing —
// the rest of the CLI surface must keep working.
@@ -1563,6 +1633,7 @@ export function validateBrainConfigDetailed(
"snapshots",
"vault",
"active",
+ "lessons",
"discipline_report",
"guardrails",
"link_graph",
@@ -1757,6 +1828,7 @@ export function validateBrainConfigDetailed(
},
...(vault !== undefined ? { vault } : {}),
...(active !== undefined ? { active } : {}),
+ ...(lessons !== undefined ? { lessons } : {}),
...(disciplineReport !== undefined ? { discipline_report: disciplineReport } : {}),
...(guardrails !== undefined ? { guardrails } : {}),
...(linkGraph !== undefined ? { link_graph: linkGraph } : {}),
@@ -1827,6 +1899,32 @@ function requireUnitInterval(field: string, value: unknown, source: string | nul
}
}
+/**
+ * Read an optional integer field bounded to `[min, max]` from a config
+ * sub-map. Returns `undefined` when the key is absent; throws a
+ * {@link BrainConfigError} (never clamps) when present but out of range
+ * or non-integer. Shared by the `lessons:` block loader.
+ */
+function readBoundedInt(
+ map: Record<string, unknown>,
+ key: string,
+ min: number,
+ max: number,
+ source: string | null,
+ field: string = `lessons.${key}`,
+): number | undefined {
+ if (!(key in map)) return undefined;
+ const raw = map[key];
+ if (typeof raw !== "number" || !Number.isInteger(raw) || raw < min || raw > max) {
+ throw new BrainConfigError(
+ `must be an integer between ${min} and ${max}; got ${describe(raw)}`,
+ field,
+ source,
+ );
+ }
+ return raw;
+}
+
function describe(value: unknown): string {
if (value === null) return "null";
if (value === undefined) return "undefined";
diff --git a/src/core/brain/provenance/epistemic.ts b/src/core/brain/provenance/epistemic.ts
new file mode 100644
index 00000000..655fca7c
--- /dev/null
+++ b/src/core/brain/provenance/epistemic.ts
@@ -0,0 +1,120 @@
+/**
+ * Epistemic provenance markers (ACM - Artifact-Constrained Metacognition).
+ *
+ * Every item placed into a model's context should say how grounded it is, so
+ * the consuming model can tell a source-backed fact from a conjecture and stop
+ * treating both as equally true. This module owns the fixed status vocabulary
+ * and the deterministic derivation of a status from a page's existing
+ * frontmatter - no manual tagging required.
+ *
+ * The five statuses, most-grounded first:
+ * - `observed` - an operator-stated, evidence-backed fact.
+ * - `derived` - inferred/deduced from cited premises (a derived fact).
+ * - `hypothesis` - a conjecture on trial (unconfirmed / quarantined).
+ * - `plan` - intended, not-yet-true (explicit tag only).
+ * - `unknown` - an acknowledged gap or contested/disputed page.
+ *
+ * Each marker carries `evidenceRefs`: the page's `evidenced_by` wikilinks
+ * (origin signals for a stated rule, premise facts for a derived one), so the
+ * consumer can follow the grounding.
+ *
+ * Derivation reuses the existing graph metadata (provenance trust level,
+ * preference status, lifecycle) rather than a new hand-maintained field. A
+ * page MAY still override the result with an explicit `epistemic:` frontmatter
+ * value - the only path that can yield `plan`, which no automatic signal maps
+ * to. This mirrors provenance.ts: the primitive owns the vocabulary and the
+ * mapping, never a specific record's frontmatter schema.
+ *
+ * Language-agnostic: the status is a fixed structural token set, never derived
+ * from natural-language vocabulary.
+ */
+
+import { asProvenanceLevel } from "./provenance.ts";
+import { readLifecycle, PAGE_LIFECYCLE } from "../page-meta/lifecycle.ts";
+
+/** Epistemic grounding of a context item, most-grounded first. */
+export const EPISTEMIC_STATUS = Object.freeze({
+ observed: "observed",
+ derived: "derived",
+ hypothesis: "hypothesis",
+ plan: "plan",
+ unknown: "unknown",
+} as const);
+
+export type EpistemicStatus = (typeof EPISTEMIC_STATUS)[keyof typeof EPISTEMIC_STATUS];
+
+const ALL: ReadonlySet<string> = new Set(Object.values(EPISTEMIC_STATUS));
+
+/**
+ * Narrow an arbitrary value to an {@link EpistemicStatus}, or null when it is
+ * not one. Case- and whitespace-insensitive, so an explicit `epistemic:`
+ * frontmatter value round-trips without an `as` cast.
+ */
+export function asEpistemicStatus(value: unknown): EpistemicStatus | null {
+ if (typeof value !== "string") return null;
+ const token = value.trim().toLowerCase();
+ return ALL.has(token) ? (token as EpistemicStatus) : null;
+}
+
+/** An epistemic marker: the status plus the evidence links that ground it. */
+export interface EpistemicMarker {
+ readonly status: EpistemicStatus;
+ /** `evidenced_by` wikilinks - origin signals or premise facts. */
+ readonly evidenceRefs: readonly string[];
+}
+
+/**
+ * Read `evidenced_by` into a clean string list. Accepts an array (the normal
+ * shape) or a lone string, drops blanks, and trims incidental whitespace so a
+ * marker's refs are stable regardless of how the frontmatter serialized them.
+ *
+ * Reads the derived `_evidenced_by` key or its `normalizeDerivedKeys`-renamed
+ * `evidenced_by` form, so it works on both raw and normalized frontmatter -
+ * the same dual-read the `_lifecycle`/`lifecycle` readers use.
+ */
+export function readEvidenceRefs(meta: Readonly<Record<string, unknown>>): string[] {
+ const raw = meta["_evidenced_by"] ?? meta["evidenced_by"];
+ const values = Array.isArray(raw) ? raw : raw === undefined || raw === null ? [] : [raw];
+ const out: string[] = [];
+ for (const v of values) {
+ if (typeof v !== "string") continue;
+ const trimmed = v.trim();
+ if (trimmed) out.push(trimmed);
+ }
+ return out;
+}
+
+/**
+ * Derive an {@link EpistemicMarker} from a page's frontmatter.
+ *
+ * Precedence:
+ * 1. an explicit, valid `epistemic:` value always wins (the only `plan` path);
+ * 2. a `disputed` lifecycle is `unknown` (its truth is contested);
+ * 3. a `deduced`/`inferred` provenance level is `derived`;
+ * 4. otherwise (a `stated`/absent level) an `unconfirmed`/`quarantine`
+ * preference status is `hypothesis`, and anything else is `observed`.
+ *
+ * The evidence refs are the page's `evidenced_by` links regardless of status.
+ */
+export function deriveEpistemicStatus(meta: Readonly<Record<string, unknown>>): EpistemicMarker {
+ const evidenceRefs = readEvidenceRefs(meta);
+
+ const explicit = asEpistemicStatus(meta["epistemic"]);
+ if (explicit !== null) return { status: explicit, evidenceRefs };
+
+ if (readLifecycle(meta) === PAGE_LIFECYCLE.disputed) {
+ return { status: EPISTEMIC_STATUS.unknown, evidenceRefs };
+ }
+
+ const level = asProvenanceLevel(meta["provenance"]) ?? "stated";
+ if (level === "deduced" || level === "inferred") {
+ return { status: EPISTEMIC_STATUS.derived, evidenceRefs };
+ }
+
+ const statusRaw = meta["_status"] ?? meta["status"];
+ const status = typeof statusRaw === "string" ? statusRaw.trim() : "";
+ if (status === "unconfirmed" || status === "quarantine") {
+ return { status: EPISTEMIC_STATUS.hypothesis, evidenceRefs };
+ }
+ return { status: EPISTEMIC_STATUS.observed, evidenceRefs };
+}
diff --git a/src/core/brain/query-demand.ts b/src/core/brain/query-demand.ts
new file mode 100644
index 00000000..2234afd7
--- /dev/null
+++ b/src/core/brain/query-demand.ts
@@ -0,0 +1,462 @@
+/**
+ * Cross-query demand log (t_97091fff).
+ *
+ * A persisted, deterministic record of every recall query against the
+ * vault, together with two satisfaction signals already computed
+ * elsewhere: the result count and the IDF-weighted coverage score from
+ * {@link ../search/coverage.ts}. Aggregated over time, the log surfaces
+ * the *demand* gap — queries operators keep asking that the vault
+ * answers poorly (weak or empty) — as a prioritized backlog of what to
+ * write into `Brain/` next.
+ *
+ * This is distinct from the existing structural gaps (dangling wikilink
+ * targets in deep-synthesis) and from the in-the-moment per-query
+ * coverage guard (coverage.ts / evidence-pack.ts): those look at one
+ * query or one link at a time and are never persisted. Here the signal
+ * is cross-query and durable.
+ *
+ * No LLM, no ranking model — pure counting and the reused coverage
+ * score. Privacy: only the normalized significant terms are stored
+ * (never the raw query), each run through the secret redactor, so a
+ * query carrying a token or path cannot leak into the log.
+ *
+ * Storage: `Brain/log/query-demand.jsonl`, one JSON record per line,
+ * rolling and byte-budget-capped (see {@link DEMAND_LOG_MAX_BYTES}).
+ */
+
+import {
+ appendFileSync,
+ existsSync,
+ mkdirSync,
+ readFileSync,
+ statSync,
+ writeFileSync,
+} from "node:fs";
+import { dirname } from "node:path";
+
+import { ensureInsideVault } from "../path-safety.ts";
+import { redactRawOutput } from "../redactor.ts";
+import { queryDemandLogPath } from "./paths.ts";
+import { acquireLockSync } from "./sync-lockfile.ts";
+import {
+ COMPLETENESS_COMPLETE_THRESHOLD,
+ COMPLETENESS_PARTIAL_THRESHOLD,
+ significantTerms,
+ type CompletenessVerdict,
+} from "../search/coverage.ts";
+
+/** Longest single normalized term kept; longer tokens are dropped as a
+ * privacy guard (a 40+ char alnum run is far likelier a secret/id than a
+ * word). */
+export const DEMAND_TERM_MAX_LEN = 40;
+
+/** Most terms retained per record — bounds one pathological query's line
+ * size while keeping the bucket key stable for realistic queries. */
+export const DEMAND_MAX_TERMS_PER_RECORD = 24;
+
+/** Compaction trigger: once the log file exceeds this many bytes, the
+ * oldest lines are dropped so the tail fits in {@link DEMAND_LOG_COMPACT_BYTES}.
+ * Bounds unbounded growth without reading the whole file on every append
+ * (the common path only stats the file). */
+export const DEMAND_LOG_MAX_BYTES = 1_000_000;
+
+/** Target size after compaction — keep the newest lines whose combined
+ * size is at most this. Kept below {@link DEMAND_LOG_MAX_BYTES} so
+ * compaction is amortized, not run on every subsequent append. */
+export const DEMAND_LOG_COMPACT_BYTES = 750_000;
+
+/** A query must recur at least this many times to count as demand. */
+export const DEMAND_DEFAULT_MIN_OCCURRENCES = 2;
+
+/**
+ * One persisted recall observation. `coverage` is absent when the caller
+ * did not compute the evidence-pack coverage (the common non-verified
+ * search path); the aggregator falls back to the result-presence signal
+ * for those buckets.
+ */
+export interface QueryDemandRecord {
+ readonly ts: string;
+ readonly terms: ReadonlyArray<string>;
+ readonly results: number;
+ readonly coverage?: number;
+}
+
+export interface RecordQueryDemandInput {
+ /** Raw query text — normalized/redacted to terms before it is stored. */
+ readonly query?: string;
+ /** Pre-normalized terms; used verbatim (after redaction) if given. */
+ readonly terms?: ReadonlyArray<string>;
+ readonly resultCount: number;
+ /** IDF-weighted coverage (0..1) from coverage.ts, when available. */
+ readonly coverage?: number | null;
+ /** ISO timestamp; defaults to now. */
+ readonly at?: string;
+}
+
+export interface QueryDemandFilter {
+ readonly since?: string;
+ readonly until?: string;
+}
+
+export interface AggregateQueryDemandOptions extends QueryDemandFilter {
+ /** Minimum occurrences to surface a bucket (default 2). */
+ readonly minOccurrences?: number;
+ /**
+ * Surface only buckets whose mean satisfaction is at or below this
+ * (default {@link COMPLETENESS_COMPLETE_THRESHOLD}) — i.e. queries not
+ * already answered well. Ranking by demand pushes the worst to the top.
+ */
+ readonly maxSatisfaction?: number;
+ /** Cap on returned gaps. */
+ readonly limit?: number;
+}
+
+export interface QueryDemandGap {
+ /** Sorted normalized terms — the bucket key, split for display. */
+ readonly terms: ReadonlyArray<string>;
+ readonly occurrences: number;
+ /** Mean IDF-weighted coverage over records that carried one, else null. */
+ readonly meanCoverage: number | null;
+ readonly meanResultCount: number;
+ /** Records with zero results. */
+ readonly emptyCount: number;
+ readonly emptyRate: number;
+ /**
+ * Satisfaction in [0,1]: the coverage mean when any record carried
+ * coverage, otherwise the fraction of runs that returned any result.
+ */
+ readonly satisfaction: number;
+ /** Whether satisfaction came from coverage (true) or the result-count
+ * fallback (false). */
+ readonly coverageKnown: boolean;
+ /** occurrences × (1 − satisfaction): recurring + unmet ranks highest. */
+ readonly demandScore: number;
+ /** Completeness verdict over the satisfaction score. */
+ readonly verdict: CompletenessVerdict;
+ readonly firstSeen: string;
+ readonly lastSeen: string;
+}
+
+export interface QueryDemandReport {
+ readonly totalRecords: number;
+ readonly distinctQueries: number;
+ readonly gaps: ReadonlyArray<QueryDemandGap>;
+}
+
+/** Length at/above which a mixed letter+digit token is treated as a
+ * high-entropy secret/id (API key, UUID, hash) rather than a query word
+ * and dropped. Real words this long are essentially never both alpha and
+ * numeric. */
+export const DEMAND_SECRET_TOKEN_MIN_LEN = 20;
+
+/**
+ * Normalize a raw query into the terms actually stored: the significant
+ * terms (length ≥ 3, deduped) sorted for a stable bucket key, with the
+ * privacy filters below applied. Capped at
+ * {@link DEMAND_MAX_TERMS_PER_RECORD}.
+ */
+export function normalizeQueryTerms(query: string): string[] {
+ return sanitizeTerms(significantTerms(query));
+}
+
+function sanitizeTerms(terms: ReadonlyArray<string>): string[] {
+ const out = new Set<string>();
+ for (const raw of terms) {
+ const term = raw.toLocaleLowerCase().trim();
+ if (term.length < 3 || term.length > DEMAND_TERM_MAX_LEN) continue;
+ if (isSecretShapedTerm(term)) continue;
+ // Belt-and-suspenders: a term the key/value redactor still rewrites
+ // is secret-shaped — never a legitimate query word, so drop it rather
+ // than store a `***REDACTED***` bucket key.
+ if (redactRawOutput(term) !== term) continue;
+ out.add(term);
+ }
+ return [...out].toSorted().slice(0, DEMAND_MAX_TERMS_PER_RECORD);
+}
+
+/**
+ * A bare token is secret-shaped when it is long AND mixes letters with
+ * digits — the signature of an API key, UUID, or hash. The redactor's
+ * passes are context-based (`key: value`, bearer headers, env), so a
+ * single high-entropy word carries no shape they can match; this catches
+ * it before it becomes a persisted bucket key. Kept deliberately narrow
+ * (length gate + both classes) so ordinary words and short alnum terms
+ * (`oauth2`, `sha256`, `utf8`) are never dropped.
+ */
+function isSecretShapedTerm(term: string): boolean {
+ if (term.length < DEMAND_SECRET_TOKEN_MIN_LEN) return false;
+ return /[a-z]/.test(term) && /[0-9]/.test(term);
+}
+
+/**
+ * Append one recall observation to the demand log. Terms are derived
+ * from `terms` (redacted) or `query` (normalized+redacted); a query with
+ * no significant terms is not recorded (returns null). Best-effort by
+ * contract — callers gate this behind opt-in telemetry and must not let
+ * a log write fail the search.
+ */
+export function recordQueryDemand(
+ vault: string,
+ input: RecordQueryDemandInput,
+): QueryDemandRecord | null {
+ const terms =
+ input.terms !== undefined ? sanitizeTerms(input.terms) : normalizeQueryTerms(input.query ?? "");
+ if (terms.length === 0) return null;
+ const record: QueryDemandRecord = {
+ ts: normalizeDemandTimestamp(input.at),
+ terms,
+ results: Number.isFinite(input.resultCount) ? Math.max(0, Math.floor(input.resultCount)) : 0,
+ ...(typeof input.coverage === "number" && Number.isFinite(input.coverage)
+ ? { coverage: clamp01(input.coverage) }
+ : {}),
+ };
+ const path = queryDemandLogPath(vault);
+ mkdirSync(ensureInsideVault(dirname(path), vault), { recursive: true });
+ const handle = acquireLockSync(path);
+ try {
+ appendFileSync(path, `${JSON.stringify(record)}\n`, { encoding: "utf8" });
+ compactIfNeeded(path);
+ } finally {
+ handle.release();
+ }
+ return record;
+}
+
+/** Read all demand records, filtered by [since, until] and sorted by ts. */
+export function readQueryDemand(
+ vault: string,
+ filter: QueryDemandFilter = {},
+): ReadonlyArray<QueryDemandRecord> {
+ const path = queryDemandLogPath(vault);
+ if (!existsSync(path)) return Object.freeze([]);
+ const out: QueryDemandRecord[] = [];
+ for (const line of readFileSync(path, "utf8").split("\n")) {
+ if (!line.trim()) continue;
+ const record = coerceRecord(line);
+ if (record === null) continue;
+ if (filter.since !== undefined && record.ts < filter.since) continue;
+ if (filter.until !== undefined && record.ts > filter.until) continue;
+ out.push(record);
+ }
+ out.sort((a, b) => (a.ts < b.ts ? -1 : a.ts > b.ts ? 1 : 0));
+ return Object.freeze(out);
+}
+
+/**
+ * Aggregate the demand log into ranked knowledge gaps: recurring queries
+ * the vault answers poorly. Buckets by the sorted-terms key, scores each
+ * by occurrences × (1 − satisfaction), and returns the worst first.
+ */
+export function aggregateQueryDemand(
+ vault: string,
+ options: AggregateQueryDemandOptions = {},
+): QueryDemandReport {
+ const records = readQueryDemand(vault, {
+ ...(options.since !== undefined ? { since: options.since } : {}),
+ ...(options.until !== undefined ? { until: options.until } : {}),
+ });
+ const minOccurrences = Math.max(
+ 1,
+ Math.floor(options.minOccurrences ?? DEMAND_DEFAULT_MIN_OCCURRENCES),
+ );
+ const maxSatisfaction = clamp01(options.maxSatisfaction ?? COMPLETENESS_COMPLETE_THRESHOLD);
+
+ interface Bucket {
+ terms: ReadonlyArray<string>;
+ occurrences: number;
+ resultSum: number;
+ emptyCount: number;
+ coverageSum: number;
+ coverageSamples: number;
+ firstSeen: string;
+ lastSeen: string;
+ }
+ const buckets = new Map<string, Bucket>();
+ for (const record of records) {
+ const key = record.terms.join(" ");
+ let bucket = buckets.get(key);
+ if (!bucket) {
+ bucket = {
+ terms: record.terms,
+ occurrences: 0,
+ resultSum: 0,
+ emptyCount: 0,
+ coverageSum: 0,
+ coverageSamples: 0,
+ firstSeen: record.ts,
+ lastSeen: record.ts,
+ };
+ buckets.set(key, bucket);
+ }
+ bucket.occurrences += 1;
+ bucket.resultSum += record.results;
+ if (record.results === 0) bucket.emptyCount += 1;
+ if (typeof record.coverage === "number") {
+ bucket.coverageSum += record.coverage;
+ bucket.coverageSamples += 1;
+ }
+ if (record.ts < bucket.firstSeen) bucket.firstSeen = record.ts;
+ if (record.ts > bucket.lastSeen) bucket.lastSeen = record.ts;
+ }
+
+ const gaps: QueryDemandGap[] = [];
+ for (const bucket of buckets.values()) {
+ if (bucket.occurrences < minOccurrences) continue;
+ const coverageKnown = bucket.coverageSamples > 0;
+ const meanCoverage = coverageKnown ? bucket.coverageSum / bucket.coverageSamples : null;
+ // Prefer the reused coverage score; only when no record in this
+ // bucket carried one do we fall back to "did it return anything".
+ const satisfaction = coverageKnown
+ ? meanCoverage!
+ : (bucket.occurrences - bucket.emptyCount) / bucket.occurrences;
+ if (satisfaction > maxSatisfaction) continue;
+ gaps.push(
+ Object.freeze({
+ terms: bucket.terms,
+ occurrences: bucket.occurrences,
+ meanCoverage: meanCoverage === null ? null : round4(meanCoverage),
+ meanResultCount: round4(bucket.resultSum / bucket.occurrences),
+ emptyCount: bucket.emptyCount,
+ emptyRate: round4(bucket.emptyCount / bucket.occurrences),
+ satisfaction: round4(satisfaction),
+ coverageKnown,
+ demandScore: round4(bucket.occurrences * (1 - satisfaction)),
+ verdict: verdictFor(satisfaction),
+ firstSeen: bucket.firstSeen,
+ lastSeen: bucket.lastSeen,
+ }),
+ );
+ }
+
+ gaps.sort(
+ (a, b) =>
+ b.demandScore - a.demandScore ||
+ b.occurrences - a.occurrences ||
+ a.terms.join(" ").localeCompare(b.terms.join(" ")),
+ );
+ const limited =
+ options.limit !== undefined ? gaps.slice(0, Math.max(0, Math.floor(options.limit))) : gaps;
+
+ return Object.freeze({
+ totalRecords: records.length,
+ distinctQueries: buckets.size,
+ gaps: Object.freeze(limited),
+ });
+}
+
+/**
+ * Snake_case JSON projection of a report for the MCP tool and CLI
+ * `--json` surfaces, matching the sibling recall-telemetry convention.
+ * Kept here so both surfaces serialize identically.
+ */
+export function serializeQueryDemandReport(report: QueryDemandReport): Record<string, unknown> {
+ return {
+ total_records: report.totalRecords,
+ distinct_queries: report.distinctQueries,
+ gaps: report.gaps.map((gap) => ({
+ terms: gap.terms,
+ occurrences: gap.occurrences,
+ mean_coverage: gap.meanCoverage,
+ mean_result_count: gap.meanResultCount,
+ empty_count: gap.emptyCount,
+ empty_rate: gap.emptyRate,
+ satisfaction: gap.satisfaction,
+ coverage_known: gap.coverageKnown,
+ demand_score: gap.demandScore,
+ verdict: gap.verdict,
+ first_seen: gap.firstSeen,
+ last_seen: gap.lastSeen,
+ })),
+ };
+}
+
+function verdictFor(satisfaction: number): CompletenessVerdict {
+ if (satisfaction >= COMPLETENESS_COMPLETE_THRESHOLD) return "complete";
+ if (satisfaction >= COMPLETENESS_PARTIAL_THRESHOLD) return "partial";
+ return "sparse";
+}
+
+function coerceRecord(line: string): QueryDemandRecord | null {
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(line);
+ } catch {
+ return null;
+ }
+ if (parsed === null || typeof parsed !== "object") return null;
+ const obj = parsed as Record<string, unknown>;
+ const ts = obj["ts"];
+ const terms = obj["terms"];
+ const results = obj["results"];
+ if (typeof ts !== "string") return null;
+ if (!Array.isArray(terms) || !terms.every((t) => typeof t === "string")) return null;
+ if (typeof results !== "number" || !Number.isFinite(results)) return null;
+ const coverage = obj["coverage"];
+ return {
+ ts,
+ terms: Object.freeze(terms as string[]),
+ results: Math.max(0, Math.floor(results)),
+ ...(typeof coverage === "number" && Number.isFinite(coverage)
+ ? { coverage: clamp01(coverage) }
+ : {}),
+ };
+}
+
+/**
+ * Drop the oldest lines once the file grows past
+ * {@link DEMAND_LOG_MAX_BYTES}, keeping the newest lines that fit in
+ * {@link DEMAND_LOG_COMPACT_BYTES}. Called under the append lock. The
+ * common path only stats the file — no read — so per-append cost stays
+ * O(1) until the rare compaction.
+ */
+function compactIfNeeded(path: string): void {
+ let size: number;
+ try {
+ size = statSync(path).size;
+ } catch {
+ return;
+ }
+ if (size <= DEMAND_LOG_MAX_BYTES) return;
+ const lines = readFileSync(path, "utf8")
+ .split("\n")
+ .filter((line) => line.trim().length > 0);
+ const kept: string[] = [];
+ let bytes = 0;
+ for (let i = lines.length - 1; i >= 0; i--) {
+ const line = lines[i]!;
+ const lineBytes = Buffer.byteLength(line, "utf8") + 1;
+ if (bytes + lineBytes > DEMAND_LOG_COMPACT_BYTES && kept.length > 0) break;
+ kept.push(line);
+ bytes += lineBytes;
+ }
+ kept.reverse();
+ writeFileSync(path, kept.length > 0 ? `${kept.join("\n")}\n` : "", { encoding: "utf8" });
+}
+
+function clamp01(value: number): number {
+ if (value < 0) return 0;
+ if (value > 1) return 1;
+ return value;
+}
+
+/**
+ * Normalize a caller-supplied (or absent) timestamp into a canonical
+ * millisecond-precision ISO string. `readQueryDemand` and
+ * `aggregateQueryDemand` compare `ts` as raw strings (sort + range
+ * filter), which is only stable when every record uses the same precision
+ * — `2026-07-01T00:00:00.000Z` must sort after `2026-07-01T00:00:00Z`,
+ * but lexically `.` < `Z` flips them. Parsing-and-reformatting through
+ * `Date` collapses the difference and also rejects non-ISO input by
+ * falling back to now, so a malformed `at` can't silently misorder the
+ * log. Non-finite `Date` (invalid input) resolves to the current time.
+ */
+function normalizeDemandTimestamp(at: string | undefined): string {
+ if (at === undefined) return new Date().toISOString();
+ const parsed = new Date(at);
+ const ms = parsed.getTime();
+ return Number.isFinite(ms) ? parsed.toISOString() : new Date().toISOString();
+}
+
+function round4(value: number): number {
+ return Math.round(value * 10_000) / 10_000;
+}
diff --git a/src/core/brain/recall-adequacy.ts b/src/core/brain/recall-adequacy.ts
new file mode 100644
index 00000000..db8d2d5c
--- /dev/null
+++ b/src/core/brain/recall-adequacy.ts
@@ -0,0 +1,169 @@
+/**
+ * Recall adequacy verdict (retrieval-precision-quality-loop, t_b8f66fec).
+ *
+ * A thin verdict + action layer over the relevance scores the recall
+ * stack already produces (search / recall-telemetry `top_artifacts`
+ * scores). It does NOT search or re-rank; given the top-k scores of a
+ * recall attempt it classifies grounding fitness and names the explicit
+ * low-adequacy action so callers can branch instead of always feeding
+ * top-k to the LLM:
+ *
+ * sufficient -> proceed (grounding is strong enough to answer)
+ * weak -> re_recall (broaden scope / try an alternate path first)
+ * insufficient -> abstain (return an explicit 'insufficient grounding'
+ * signal, and escalate for review)
+ *
+ * Language-agnostic and deterministic: it reads only numeric scores, so
+ * it behaves identically for any prompt language. Complements the
+ * epistemic-provenance card (which feeds grounded scores into this gate).
+ */
+
+export const RECALL_ADEQUACY_LEVELS = ["sufficient", "weak", "insufficient"] as const;
+export type RecallAdequacyLevel = (typeof RECALL_ADEQUACY_LEVELS)[number];
+
+export const RECALL_ADEQUACY_ACTIONS = ["proceed", "re_recall", "abstain"] as const;
+export type RecallAdequacyAction = (typeof RECALL_ADEQUACY_ACTIONS)[number];
+
+export interface RecallAdequacyThresholds {
+ /** Top score at/above which recall is sufficient. */
+ readonly sufficient: number;
+ /** Top score at/above which recall is at least weak (below => insufficient). */
+ readonly weak: number;
+ /** Minimum usable-result count below which recall cannot be sufficient. */
+ readonly minResults: number;
+}
+
+export const DEFAULT_RECALL_ADEQUACY_THRESHOLDS: RecallAdequacyThresholds = Object.freeze({
+ sufficient: 0.6,
+ weak: 0.3,
+ minResults: 1,
+});
+
+export interface RecallAdequacyVerdict {
+ readonly level: RecallAdequacyLevel;
+ readonly action: RecallAdequacyAction;
+ /** True when the result should be flagged for review / surfaced to an operator. */
+ readonly escalate: boolean;
+ readonly resultCount: number;
+ /** Highest usable score, or 0 when there are no usable results. */
+ readonly topScore: number;
+ /** Mean of usable scores, or 0 when there are no usable results. */
+ readonly meanScore: number;
+ readonly reason: string;
+}
+
+const ACTION_BY_LEVEL: Readonly<Record<RecallAdequacyLevel, RecallAdequacyAction>> = Object.freeze({
+ sufficient: "proceed",
+ weak: "re_recall",
+ insufficient: "abstain",
+});
+
+function resolveThresholds(
+ overrides?: Partial<RecallAdequacyThresholds>,
+): RecallAdequacyThresholds {
+ const merged = { ...DEFAULT_RECALL_ADEQUACY_THRESHOLDS, ...overrides };
+ const { sufficient, weak, minResults } = merged;
+ for (const [name, value] of [
+ ["sufficient", sufficient],
+ ["weak", weak],
+ ] as const) {
+ if (!Number.isFinite(value) || value < 0 || value > 1) {
+ throw new Error(`recall adequacy: ${name} threshold must be in [0,1]; got ${value}`);
+ }
+ }
+ if (weak > sufficient) {
+ throw new Error(
+ `recall adequacy: weak threshold (${weak}) must not exceed sufficient threshold (${sufficient})`,
+ );
+ }
+ if (!Number.isInteger(minResults) || minResults < 1) {
+ throw new Error(`recall adequacy: minResults must be a positive integer; got ${minResults}`);
+ }
+ return Object.freeze({ sufficient, weak, minResults });
+}
+
+/** Round to 4 decimals so verdict payloads stay stable and compact. */
+function round4(value: number): number {
+ return Math.round(value * 1e4) / 1e4;
+}
+
+/**
+ * Classify recall fitness from the relevance scores of a recall attempt
+ * and name the explicit action. Non-finite scores are dropped; negative
+ * scores clamp to 0 (search scores are normalized to [0,1]).
+ */
+export function assessRecallAdequacy(
+ scores: ReadonlyArray<number>,
+ overrides?: Partial<RecallAdequacyThresholds>,
+): RecallAdequacyVerdict {
+ const thresholds = resolveThresholds(overrides);
+ const usable = scores.filter((s) => Number.isFinite(s)).map((s) => Math.max(0, s));
+ const resultCount = usable.length;
+
+ if (resultCount === 0) {
+ return finalize("insufficient", {
+ resultCount: 0,
+ topScore: 0,
+ meanScore: 0,
+ reason: "no recall results — insufficient grounding",
+ });
+ }
+
+ const topScore = Math.max(...usable);
+ const meanScore = usable.reduce((a, b) => a + b, 0) / resultCount;
+
+ if (topScore >= thresholds.sufficient) {
+ if (resultCount >= thresholds.minResults) {
+ return finalize("sufficient", {
+ resultCount,
+ topScore,
+ meanScore,
+ reason: `top score ${round4(topScore)} >= sufficient ${thresholds.sufficient} across ${resultCount} result(s) — sufficient grounding`,
+ });
+ }
+ // Strong single hit but too few corroborating results: re-recall to
+ // broaden before answering rather than proceed on a lone signal.
+ return finalize("weak", {
+ resultCount,
+ topScore,
+ meanScore,
+ reason: `strong top score ${round4(topScore)} but only ${resultCount} result(s) < min_results ${thresholds.minResults} — re-recall to broaden`,
+ });
+ }
+
+ if (topScore >= thresholds.weak) {
+ return finalize("weak", {
+ resultCount,
+ topScore,
+ meanScore,
+ reason: `top score ${round4(topScore)} in [${thresholds.weak}, ${thresholds.sufficient}) — weak grounding, re-recall via alternate strategy`,
+ });
+ }
+
+ return finalize("insufficient", {
+ resultCount,
+ topScore,
+ meanScore,
+ reason: `top score ${round4(topScore)} < weak ${thresholds.weak} — insufficient grounding, abstain`,
+ });
+}
+
+function finalize(
+ level: RecallAdequacyLevel,
+ parts: {
+ readonly resultCount: number;
+ readonly topScore: number;
+ readonly meanScore: number;
+ readonly reason: string;
+ },
+): RecallAdequacyVerdict {
+ return Object.freeze({
+ level,
+ action: ACTION_BY_LEVEL[level],
+ escalate: level === "insufficient",
+ resultCount: parts.resultCount,
+ topScore: round4(parts.topScore),
+ meanScore: round4(parts.meanScore),
+ reason: parts.reason,
+ });
+}
diff --git a/src/core/brain/types.ts b/src/core/brain/types.ts
index 7bc4e2f8..a2f60f2f 100644
--- a/src/core/brain/types.ts
+++ b/src/core/brain/types.ts
@@ -905,6 +905,21 @@ export interface BrainActiveConfig {
readonly inject_budget_chars?: number;
}
+/**
+ * Container for the `lessons:` block of `_brain.yaml` (t_62363378).
+ * Tunes the signed, recency-scored lessons digest (`Brain/lessons.md`).
+ * Every field is optional; absence falls back to the `LESSONS_*`
+ * defaults in policy.ts.
+ */
+export interface BrainLessonsConfig {
+ /** Exponential half-life of the recency decay, in days. */
+ readonly half_life_days?: number;
+ /** Distinct-result count required to promote a lesson to `preferred`. */
+ readonly corroboration_min?: number;
+ /** Max lessons rendered into the digest. */
+ readonly limit?: number;
+}
+
/**
* Optional configuration for the daily discipline report (§D of the
* agent-discipline-tail design). Absent on vaults that have not opted
@@ -1059,6 +1074,13 @@ export interface BrainConfig {
* mirrored `most_applied` block in `brain_digest`.
*/
readonly active?: BrainActiveConfig;
+ /**
+ * Optional `lessons:` block (t_62363378). Tunes the signed,
+ * recency-scored lessons digest (`Brain/lessons.md`): decay
+ * half-life, corroboration threshold, and rendered limit. Absent:
+ * callers fall back to the `LESSONS_*` defaults in policy.ts.
+ */
+ readonly lessons?: BrainLessonsConfig;
/** Optional daily discipline-report configuration (§D). Absent when not configured. */
readonly discipline_report?: DisciplineReportConfig;
/**
diff --git a/src/core/config.ts b/src/core/config.ts
index a4043337..a7dfd11d 100644
--- a/src/core/config.ts
+++ b/src/core/config.ts
@@ -411,6 +411,68 @@ export function resolveRecallGateTelemetry(configPath?: string): boolean {
return raw === "true" || raw === "1";
}
+/**
+ * Recall adequacy thresholds (retrieval-precision-quality-loop,
+ * t_b8f66fec). Configurable floors that drive the sufficient / weak /
+ * insufficient verdict over existing recall relevance scores. Defaults
+ * mirror DEFAULT_RECALL_ADEQUACY_THRESHOLDS (0.6 / 0.3 / 1). Invalid
+ * values fail fast rather than silently reverting to a default.
+ */
+export function resolveRecallAdequacyThresholds(configPath?: string): {
+ sufficient: number;
+ weak: number;
+ minResults: number;
+} {
+ const data = discoverConfig(configPath).data;
+ const sufficient = resolveAdequacyFloor(
+ "recall_adequacy_sufficient",
+ process.env["OPEN_SECOND_BRAIN_RECALL_ADEQUACY_SUFFICIENT"],
+ data["recall_adequacy_sufficient"],
+ 0.6,
+ );
+ const weak = resolveAdequacyFloor(
+ "recall_adequacy_weak",
+ process.env["OPEN_SECOND_BRAIN_RECALL_ADEQUACY_WEAK"],
+ data["recall_adequacy_weak"],
+ 0.3,
+ );
+ if (weak > sufficient) {
+ throw new Error(
+ `recall_adequacy_weak (${weak}) must not exceed recall_adequacy_sufficient (${sufficient})`,
+ );
+ }
+ const minResults = resolveAdequacyMinResults(
+ process.env["OPEN_SECOND_BRAIN_RECALL_ADEQUACY_MIN_RESULTS"],
+ data["recall_adequacy_min_results"],
+ );
+ return { sufficient, weak, minResults };
+}
+
+function resolveAdequacyFloor(
+ key: string,
+ env: string | undefined,
+ fileValue: string | undefined,
+ fallback: number,
+): number {
+ const raw = (env?.trim() || fileValue?.trim()) ?? "";
+ if (raw === "") return fallback;
+ const value = Number(raw);
+ if (!Number.isFinite(value) || value < 0 || value > 1) {
+ throw new Error(`${key} must be a number in [0,1]; got '${raw}'`);
+ }
+ return value;
+}
+
+function resolveAdequacyMinResults(env: string | undefined, fileValue: string | undefined): number {
+ const raw = (env?.trim() || fileValue?.trim()) ?? "";
+ if (raw === "") return 1;
+ const value = Number(raw);
+ if (!Number.isInteger(value) || value < 1) {
+ throw new Error(`recall_adequacy_min_results must be a positive integer; got '${raw}'`);
+ }
+ return value;
+}
+
/**
* Generation-report tracing gate (Hindsight brain-loop ops, t_281c3edc).
* Default OFF: the inbound `generation_report` continuity path stays
diff --git a/src/mcp/brain/pack-tools.ts b/src/mcp/brain/pack-tools.ts
index afaacfaf..8cc2ab45 100644
--- a/src/mcp/brain/pack-tools.ts
+++ b/src/mcp/brain/pack-tools.ts
@@ -8,10 +8,14 @@
* BRAIN_TOOLS surface.
*/
-import { resolveSearchFocusContextPack } from "../../core/config.ts";
+import {
+ resolveRecallAdequacyThresholds,
+ resolveSearchFocusContextPack,
+} from "../../core/config.ts";
import { resolveSearchConfig } from "../../core/search/index.ts";
import { readActiveSessionFocus } from "../../core/search/session-focus.ts";
import { packContext } from "../../core/brain/context-pack.ts";
+import { assessRecallAdequacy } from "../../core/brain/recall-adequacy.ts";
import {
readAnticipatoryContext,
refreshAnticipatoryCache,
@@ -74,6 +78,17 @@ async function toolBrainContextPack(
const maxTotalChars = optionalPositiveInt(args, "max_total_chars", "brain_context_pack");
const receipt = receiptOptionsFromArgs("brain_context_pack", args, "context_pack", "mcp");
const telemetry = telemetryOptionsFromArgs("brain_context_pack", args, "mcp");
+ // Adequacy verdict (t_b8f66fec): when the caller passes the recall
+ // scores that produced this material, classify grounding fitness and
+ // name the action; also persist it in the receipt for the audit trail.
+ const recallScores = parseRecallScores(args["recall_scores"]);
+ const adequacy =
+ recallScores !== undefined
+ ? assessRecallAdequacy(
+ recallScores,
+ resolveRecallAdequacyThresholds(ctx.configPath ?? undefined),
+ )
+ : undefined;
// Focus wiring (Agent Surface Suite, t_5b478e47): gated on the
// search_focus_context_pack config key (default off) so the default
// pack stays byte-identical. Fail-soft - a broken search config
@@ -100,6 +115,7 @@ async function toolBrainContextPack(
...(query ? { query } : {}),
...(includeLanes ? { includeLanes: true } : {}),
...(receipt !== undefined ? { receipt } : {}),
+ ...(adequacy !== undefined ? { recallAdequacy: adequacy } : {}),
...(cacheStable || dedupRepeated
? {
transforms: {
@@ -127,6 +143,8 @@ async function toolBrainContextPack(
tokens: i.tokens,
body: i.body,
trimmed: i.trimmed,
+ epistemic: i.epistemic,
+ ...(i.evidenceRefs.length > 0 ? { evidence_refs: i.evidenceRefs } : {}),
...(i.originalRank !== undefined ? { original_rank: i.originalRank } : {}),
...(i.stableRank !== undefined ? { stable_rank: i.stableRank } : {}),
...(i.dedupedFrom !== undefined ? { deduped_from: i.dedupedFrom } : {}),
@@ -141,9 +159,52 @@ async function toolBrainContextPack(
...(report.receiptId ? { receipt_id: report.receiptId } : {}),
...(report.telemetryId ? { telemetry_id: report.telemetryId } : {}),
...(report.lanes ? { lanes: report.lanes } : {}),
+ ...(adequacy !== undefined
+ ? {
+ adequacy: {
+ level: adequacy.level,
+ action: adequacy.action,
+ escalate: adequacy.escalate,
+ result_count: adequacy.resultCount,
+ top_score: adequacy.topScore,
+ mean_score: adequacy.meanScore,
+ reason: adequacy.reason,
+ },
+ }
+ : {}),
};
}
+/**
+ * Parse the optional `recall_scores` argument. Returns `undefined` when
+ * absent (verdict skipped) and throws INVALID_PARAMS on a malformed
+ * shape. An empty array is a valid "no results" signal.
+ */
+function parseRecallScores(raw: unknown): ReadonlyArray<number> | undefined {
+ if (raw === undefined || raw === null) return undefined;
+ if (!Array.isArray(raw)) {
+ throw new MCPError(
+ INVALID_PARAMS,
+ "brain_context_pack: recall_scores must be an array of numbers",
+ );
+ }
+ if (raw.length > 200) {
+ throw new MCPError(
+ INVALID_PARAMS,
+ "brain_context_pack: recall_scores must not exceed 200 items",
+ );
+ }
+ for (const item of raw) {
+ if (typeof item !== "number") {
+ throw new MCPError(
+ INVALID_PARAMS,
+ "brain_context_pack: recall_scores must contain only numbers",
+ );
+ }
+ }
+ return raw as ReadonlyArray<number>;
+}
+
// ----- brain_context_receipts ---------------------------------------------
async function toolBrainContextReceipts(
@@ -514,6 +575,13 @@ export const PACK_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
type: "string",
description: "Optional host/runtime name for emitted receipts; defaults to `mcp`.",
},
+ recall_scores: {
+ type: "array",
+ maxItems: 200,
+ items: { type: "number" },
+ description:
+ "Optional relevance scores of the recall behind this material. When given, the response adds an adequacy verdict (level + action) and persists it in the receipt.",
+ },
telemetry: {
type: "boolean",
description:
diff --git a/src/mcp/brain/recall-tools.ts b/src/mcp/brain/recall-tools.ts
index 5424ed3c..6808ff1b 100644
--- a/src/mcp/brain/recall-tools.ts
+++ b/src/mcp/brain/recall-tools.ts
@@ -26,6 +26,7 @@ import {
expandSessionRecall,
searchSessionRecall,
} from "../../core/brain/session-recall.ts";
+import { aggregateQueryDemand, serializeQueryDemandReport } from "../../core/brain/query-demand.ts";
import { isoSecond } from "../../core/brain/time.ts";
import { INVALID_PARAMS, MCPError } from "../protocol.ts";
import type { ServerContext, ToolDefinition } from "../tools.ts";
@@ -272,6 +273,39 @@ function coerceRecallTelemetryStatus(raw: unknown): RecallTelemetryStatus | unde
return trimmed;
}
+// ----- brain_knowledge_gaps (t_97091fff) -----------------------------------
+
+async function toolBrainKnowledgeGaps(
+ ctx: ServerContext,
+ args: Record<string, unknown>,
+): Promise<Record<string, unknown>> {
+ const since = optionalStringArg("brain_knowledge_gaps", args, "since");
+ const until = optionalStringArg("brain_knowledge_gaps", args, "until");
+ const minOccurrences = coercePositiveInteger(
+ "brain_knowledge_gaps",
+ "min_occurrences",
+ args["min_occurrences"],
+ );
+ const limit = coercePositiveInteger("brain_knowledge_gaps", "limit", args["limit"]);
+ const maxSatisfaction = coerceUnitInterval("max_satisfaction", args["max_satisfaction"]);
+ const report = aggregateQueryDemand(ctx.vault, {
+ ...(since !== undefined ? { since } : {}),
+ ...(until !== undefined ? { until } : {}),
+ ...(minOccurrences !== undefined ? { minOccurrences } : {}),
+ ...(maxSatisfaction !== undefined ? { maxSatisfaction } : {}),
+ ...(limit !== undefined ? { limit } : {}),
+ });
+ return { vault_path: ctx.vault, ...serializeQueryDemandReport(report) };
+}
+
+function coerceUnitInterval(name: string, raw: unknown): number | undefined {
+ if (raw === undefined || raw === null) return undefined;
+ if (typeof raw !== "number" || !Number.isFinite(raw) || raw < 0 || raw > 1) {
+ throw new MCPError(INVALID_PARAMS, `brain_knowledge_gaps: ${name} must be a number in [0, 1]`);
+ }
+ return raw;
+}
+
// ----- brain_context_presets ----------------------------------------------
async function toolBrainSessionGrep(
@@ -433,6 +467,34 @@ export const RECALL_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
},
handler: toolBrainRecallTelemetry,
},
+ {
+ name: "brain_knowledge_gaps",
+ previewBudget: MCP_PREVIEW_BUDGET,
+ description:
+ "Cross-query demand gaps: aggregate the persisted recall demand log into recurring queries the vault answers poorly, ranked by frequency × (1 − IDF-weighted coverage). Read-only; log written only by opt-in recall telemetry.",
+ inputSchema: {
+ type: "object",
+ properties: {
+ min_occurrences: {
+ type: "integer",
+ minimum: 1,
+ description: "Minimum times a query must recur to surface (default 2).",
+ },
+ max_satisfaction: {
+ type: "number",
+ minimum: 0,
+ maximum: 1,
+ description:
+ "Surface only queries at/below this mean satisfaction (default 0.8) — i.e. not already answered well.",
+ },
+ since: { type: "string", description: "Optional inclusive lower timestamp bound." },
+ until: { type: "string", description: "Optional inclusive upper timestamp bound." },
+ limit: { type: "integer", minimum: 1, description: "Optional maximum gaps to return." },
+ },
+ additionalProperties: false,
+ },
+ handler: toolBrainKnowledgeGaps,
+ },
{
name: "brain_session_grep",
previewBudget: MCP_PREVIEW_BUDGET,
diff --git a/src/mcp/resources.ts b/src/mcp/resources.ts
index ae92d6b4..abdee416 100644
--- a/src/mcp/resources.ts
+++ b/src/mcp/resources.ts
@@ -37,12 +37,14 @@ import { existsSync, readFileSync } from "node:fs";
import { listLogMarkdownFiles } from "../core/brain/log-jsonl.ts";
import { regenerateActive } from "../core/brain/active.ts";
+import { regenerateLessons } from "../core/brain/lessons.ts";
import { buildBacklinkIndex } from "../core/brain/backlinks.ts";
import type { BacklinkRef } from "../core/brain/backlinks.ts";
import { renderDigest } from "../core/brain/digest.ts";
import { computeBrainStatus, type BrainStatusSnapshot } from "../core/brain/status.ts";
import {
brainActivePath,
+ brainLessonsPath,
preferencePath,
retiredPath,
validateIsoDate,
@@ -87,6 +89,13 @@ const CONCRETE: ReadonlyArray<ResourceDescriptor> = [
"Auto-generated digest of confirmed and quarantined preferences plus the three most recently retired entries. Regenerated at the tail of every dream run.",
mimeType: MIME_MARKDOWN,
},
+ {
+ uri: "osb://lessons",
+ name: "Brain lessons digest",
+ description:
+ "Auto-generated, signed and recency-scored lessons corpus unifying preferences and dead-ends into corroboration-tiered lessons (preferred / tentative / contested / avoid). Regenerated at the tail of every dream run.",
+ mimeType: MIME_MARKDOWN,
+ },
{
uri: "osb://digest/latest",
name: "Latest Brain digest",
@@ -145,6 +154,8 @@ export function readResource(ctx: ResourceContext, uri: string): ResourceContent
switch (parsed.kind) {
case "active":
return readActive(ctx, uri);
+ case "lessons":
+ return readLessons(ctx, uri);
case "digestLatest":
return readDigestLatest(ctx, uri);
case "status":
@@ -164,6 +175,7 @@ export function readResource(ctx: ResourceContext, uri: string): ResourceContent
type Parsed =
| { readonly kind: "active" }
+ | { readonly kind: "lessons" }
| { readonly kind: "digestLatest" }
| { readonly kind: "status" }
| { readonly kind: "preference"; readonly id: string }
@@ -183,6 +195,9 @@ function parseUri(uri: string): Parsed {
if (head === "preferences" && parts[1] === "active" && parts.length === 2) {
return { kind: "active" };
}
+ if (head === "lessons" && parts.length === 1) {
+ return { kind: "lessons" };
+ }
if (head === "digest" && parts[1] === "latest" && parts.length === 2) {
return { kind: "digestLatest" };
}
@@ -227,6 +242,24 @@ function readActive(ctx: ResourceContext, uri: string): ResourceContent {
return readMarkdown(uri, path);
}
+function readLessons(ctx: ResourceContext, uri: string): ResourceContent {
+ const path = brainLessonsPath(ctx.vault);
+ // Same on-demand-on-first-miss policy as readActive: a fresh vault
+ // that has never been dreamed has no lessons.md yet — generate it
+ // once so the resource is always meaningful, but never on every read.
+ if (!existsSync(path)) {
+ try {
+ regenerateLessons(ctx.vault);
+ } catch (err) {
+ throw new MCPError(
+ INTERNAL_ERROR,
+ `lessons.md not present and on-demand regeneration failed: ${(err as Error).message}`,
+ );
+ }
+ }
+ return readMarkdown(uri, path);
+}
+
function readDigestLatest(ctx: ResourceContext, uri: string): ResourceContent {
const rendered = renderDigest(ctx.vault, { format: "markdown" });
return {
diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.ts
index 598d8a22..48588e27 100644
--- a/src/mcp/search-tools.ts
+++ b/src/mcp/search-tools.ts
@@ -25,7 +25,12 @@ import { searchAcrossVaults } from "../core/search/cross-vault.ts";
import { RECALL_PROFILE_NAMES } from "../core/search/profiles.ts";
import { fileContextRecall } from "../core/brain/file-recall.ts";
import { withTimeout } from "../core/search/with-timeout.ts";
-import { defaultConfigPath, resolveRecallGateTelemetry } from "../core/config.ts";
+import {
+ defaultConfigPath,
+ resolveRecallAdequacyThresholds,
+ resolveRecallGateTelemetry,
+} from "../core/config.ts";
+import { assessRecallAdequacy } from "../core/brain/recall-adequacy.ts";
import { INTERNAL_ERROR, INVALID_PARAMS, MCPError } from "./protocol.ts";
import type { ServerContext, ToolDefinition } from "./tools.ts";
import { coerceBoolOptional, coerceStr, coerceStringOptional } from "./coerce.ts";
@@ -37,6 +42,7 @@ import { parseRecallBenchmarkDataset, runRecallBenchmark } from "../core/search/
import { emitRecallTelemetry } from "../core/brain/recall-telemetry.ts";
import { emitGateTelemetry } from "../core/brain/gate-telemetry.ts";
import { emitGatedTelemetry } from "../core/brain/continuity/emit.ts";
+import { recordQueryDemand } from "../core/brain/query-demand.ts";
const MCP_LIMIT_MAX = 50;
const MCP_CONTENT_MAX = 600;
@@ -271,6 +277,13 @@ const RECALL_GATE_INPUT_SCHEMA: Record<string, unknown> = {
explicit: { type: "boolean" },
telemetry_host: { type: "string", maxLength: 200 },
session_id: { type: "string", maxLength: 512 },
+ scores: {
+ type: "array",
+ maxItems: 200,
+ items: { type: "number" },
+ description:
+ "Optional top-k recall relevance scores. When given, the gate adds an adequacy verdict: sufficient/proceed, weak/re_recall, or insufficient/abstain.",
+ },
},
required: ["prompt"],
additionalProperties: false,
@@ -282,6 +295,27 @@ const RECALL_GATE_OUTPUT_SCHEMA: NonNullable<ToolDefinition["outputSchema"]> = {
properties: {
retrieve: { type: "boolean" },
reason: { type: "string" },
+ adequacy: {
+ type: "object",
+ required: [
+ "level",
+ "action",
+ "escalate",
+ "result_count",
+ "top_score",
+ "mean_score",
+ "reason",
+ ],
+ properties: {
+ level: { type: "string", enum: ["sufficient", "weak", "insufficient"] },
+ action: { type: "string", enum: ["proceed", "re_recall", "abstain"] },
+ escalate: { type: "boolean" },
+ result_count: { type: "integer" },
+ top_score: { type: "number" },
+ mean_score: { type: "number" },
+ reason: { type: "string" },
+ },
+ },
},
};
@@ -572,6 +606,18 @@ async function toolBrainSearch(
},
}),
);
+ // Cross-query demand log (t_97091fff): persist the normalized query
+ // terms, result count, and (when the evidence pack computed it) the
+ // IDF-weighted coverage so recurring poorly-answered queries can be
+ // surfaced as unmet-demand knowledge gaps. Gated behind the same
+ // telemetry opt-in and fail-open — a log write never breaks search.
+ emitGatedTelemetry(telemetry || undefined, () =>
+ recordQueryDemand(ctx.vault, {
+ query,
+ resultCount: surfaced.length,
+ coverage: outcome.evidencePack?.idfWeightedCoverage ?? null,
+ }),
+ );
return {
results: outcome.results.map((r: BrainSearchResult) => ({
path: r.path,
@@ -706,7 +752,47 @@ async function toolBrainRecallGate(
...(sessionId !== undefined ? { sessionId } : {}),
});
});
- return { ...decision };
+ // Adequacy verdict (t_b8f66fec): thin verdict + action layer over the
+ // relevance scores of a recall attempt. Only computed when the caller
+ // passes `scores`, keeping the pure structural-gate contract otherwise.
+ const scores = parseRecallScores(args["scores"]);
+ if (scores === undefined) return { ...decision };
+ const thresholds = resolveRecallAdequacyThresholds(ctx.configPath ?? undefined);
+ const verdict = assessRecallAdequacy(scores, thresholds);
+ return {
+ ...decision,
+ adequacy: {
+ level: verdict.level,
+ action: verdict.action,
+ escalate: verdict.escalate,
+ result_count: verdict.resultCount,
+ top_score: verdict.topScore,
+ mean_score: verdict.meanScore,
+ reason: verdict.reason,
+ },
+ };
+}
+
+/**
+ * Parse the optional `scores` argument for the recall gate. Returns
+ * `undefined` when absent (verdict skipped) and throws INVALID_PARAMS on
+ * a malformed shape so callers get a clear error rather than a silently
+ * dropped verdict. An empty array is a valid "no results" signal.
+ */
+function parseRecallScores(raw: unknown): ReadonlyArray<number> | undefined {
+ if (raw === undefined || raw === null) return undefined;
+ if (!Array.isArray(raw)) {
+ throw new MCPError(INVALID_PARAMS, "argument 'scores' must be an array of numbers");
+ }
+ if (raw.length > 200) {
+ throw new MCPError(INVALID_PARAMS, "argument 'scores' must not exceed 200 items");
+ }
+ for (const item of raw) {
+ if (typeof item !== "number") {
+ throw new MCPError(INVALID_PARAMS, "argument 'scores' must contain only numbers");
+ }
+ }
+ return raw as ReadonlyArray<number>;
}
const RECALL_FEEDBACK_INPUT_SCHEMA: Record<string, unknown> = {
@@ -1140,7 +1226,7 @@ export const SEARCH_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
{
name: "brain_recall_gate",
description:
- "Classify whether an automatic recall/surfacing attempt should run. Diagnostics only; does not search.",
+ "Classify whether an automatic recall/surfacing attempt should run. Diagnostics only; does not search. Pass `scores` (a recall attempt's top-k relevance scores) to also get an adequacy verdict — sufficient (proceed) / weak (re_recall) / insufficient (abstain + escalate).",
inputSchema: RECALL_GATE_INPUT_SCHEMA,
outputSchema: RECALL_GATE_OUTPUT_SCHEMA,
handler: toolBrainRecallGate,
diff --git a/tests/cli/brain-knowledge-gaps.test.ts b/tests/cli/brain-knowledge-gaps.test.ts
new file mode 100644
index 00000000..f038c498
--- /dev/null
+++ b/tests/cli/brain-knowledge-gaps.test.ts
@@ -0,0 +1,81 @@
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { recordQueryDemand } from "../../src/core/brain/query-demand.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+let vault: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-knowledge-gaps-cli-"));
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+});
+
+test("brain knowledge-gaps ranks recurring poorly-answered queries as JSON", async () => {
+ for (let i = 0; i < 3; i++) {
+ recordQueryDemand(vault, {
+ query: "incident response runbook",
+ resultCount: 0,
+ coverage: 0.0,
+ at: `2026-07-0${i + 1}T00:00:00.000Z`,
+ });
+ }
+ recordQueryDemand(vault, {
+ query: "deployment pipeline overview",
+ resultCount: 4,
+ coverage: 0.9,
+ at: "2026-07-01T02:00:00.000Z",
+ });
+ recordQueryDemand(vault, {
+ query: "deployment pipeline overview",
+ resultCount: 4,
+ coverage: 0.9,
+ at: "2026-07-02T02:00:00.000Z",
+ });
+
+ const result = await runCli(["brain", "knowledge-gaps", "--vault", vault, "--json"]);
+ expect(result.returncode).toBe(0);
+ const report = JSON.parse(result.stdout);
+ expect(report.total_records).toBe(5);
+ expect(report.gaps).toHaveLength(1);
+ expect(report.gaps[0].terms).toEqual(["incident", "response", "runbook"]);
+ expect(report.gaps[0].occurrences).toBe(3);
+ expect(report.gaps[0].empty_count).toBe(3);
+});
+
+test("brain knowledge-gaps renders a human summary", async () => {
+ recordQueryDemand(vault, {
+ query: "cron secret rotation",
+ resultCount: 0,
+ at: "2026-07-01T00:00:00Z",
+ });
+ recordQueryDemand(vault, {
+ query: "cron secret rotation",
+ resultCount: 0,
+ at: "2026-07-02T00:00:00Z",
+ });
+
+ const result = await runCli(["brain", "knowledge-gaps", "--vault", vault]);
+ expect(result.returncode).toBe(0);
+ expect(result.stdout).toContain("gap(s)");
+ expect(result.stdout).toContain("cron rotation secret");
+});
+
+test("brain knowledge-gaps rejects a bad --max-satisfaction", async () => {
+ const result = await runCli([
+ "brain",
+ "knowledge-gaps",
+ "--vault",
+ vault,
+ "--max-satisfaction",
+ "5",
+ ]);
+ expect(result.returncode).not.toBe(0);
+ expect(result.stderr).toContain("--max-satisfaction");
+});
diff --git a/tests/core/brain/context-pack.test.ts b/tests/core/brain/context-pack.test.ts
index 47cb5c8e..ffeec845 100644
--- a/tests/core/brain/context-pack.test.ts
+++ b/tests/core/brain/context-pack.test.ts
@@ -370,3 +370,40 @@ describe("packContext - untrusted_source_delimiting flag (Unit 1)", () => {
expect(body).toContain(HOSTILE); // content preserved, not blanked
});
});
+
+describe("packContext - epistemic provenance markers (ACM)", () => {
+ function writeRaw(slug: string, frontLines: string[], body = "x") {
+ writeFileSync(
+ join(vault, "Brain", "preferences", `pref-${slug}.md`),
+ ["---", `id: pref-${slug}`, "topic: t", "principle: p", ...frontLines, "---", "", body].join(
+ "\n",
+ ),
+ );
+ }
+
+ test("a plain preference defaults to observed with no evidence refs", () => {
+ writePref("plain", { topic: "t", principle: "p", tier: "core" });
+ const item = packContext(vault, { maxTokens: 10_000 }).items[0]!;
+ expect(item.epistemic).toBe("observed");
+ expect(item.evidenceRefs).toEqual([]);
+ });
+
+ test("a confirmed stated rule is observed and surfaces its evidence refs", () => {
+ // Real preferences store derived fields `_`-prefixed with inline arrays.
+ writeRaw("obs", ["provenance: stated", "_status: confirmed", '_evidenced_by: ["[[sig-1]]"]']);
+ const item = packContext(vault, { maxTokens: 10_000 }).items[0]!;
+ expect(item.epistemic).toBe("observed");
+ expect(item.evidenceRefs).toEqual(["[[sig-1]]"]);
+ });
+
+ test("a deduced/inferred fact is derived; unconfirmed is hypothesis; disputed is unknown", () => {
+ writeRaw("der", ["provenance: deduced", '_evidenced_by: ["[[pref-obs]]"]']);
+ writeRaw("hyp", ["_status: unconfirmed"]);
+ writeRaw("unk", ["_lifecycle: disputed", "_status: confirmed"]);
+ const byId = new Map(packContext(vault, { maxTokens: 10_000 }).items.map((i) => [i.id, i]));
+ expect(byId.get("pref-der")!.epistemic).toBe("derived");
+ expect(byId.get("pref-der")!.evidenceRefs).toEqual(["[[pref-obs]]"]);
+ expect(byId.get("pref-hyp")!.epistemic).toBe("hypothesis");
+ expect(byId.get("pref-unk")!.epistemic).toBe("unknown");
+ });
+});
diff --git a/tests/core/brain/lessons.test.ts b/tests/core/brain/lessons.test.ts
new file mode 100644
index 00000000..6f8ad3a3
--- /dev/null
+++ b/tests/core/brain/lessons.test.ts
@@ -0,0 +1,284 @@
+/**
+ * Tests for `src/core/brain/lessons.ts` (t_62363378).
+ *
+ * Drives the unified, signed, recency-scored lessons corpus:
+ * - positive (apply-evidence `applied`) vs negative (`violated` /
+ * `outdated`, plus dead-end notes) outcomes folded into one score;
+ * - a signed, exponentially-decayed weight so a fresh dead-end
+ * outweighs a stale "useful";
+ * - a ≥N-distinct-results corroboration gate promoting `tentative`
+ * to `preferred`, with mixed-signal nodes rendered `contested`
+ * (recency-wins) and purely-negative nodes `avoid`.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, mkdirSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ computeLessons,
+ regenerateLessons,
+ LESSON_TIER,
+ LESSON_STANCE,
+} from "../../../src/core/brain/lessons.ts";
+import { appendLogEvent } from "../../../src/core/brain/log.ts";
+import { recordDeadEnd } from "../../../src/core/brain/dead-ends.ts";
+import type { BrainPreference } from "../../../src/core/brain/types.ts";
+import {
+ BRAIN_APPLY_RESULT,
+ BRAIN_LOG_EVENT_KIND,
+ BRAIN_PREFERENCE_STATUS,
+} from "../../../src/core/brain/types.ts";
+import type { DeadEndEntry } from "../../../src/core/brain/dead-ends.ts";
+
+const NOW = new Date("2026-05-20T00:00:00Z");
+
+function buildPref(id: string, overrides: Partial<BrainPreference> = {}): BrainPreference {
+ const base: BrainPreference = {
+ kind: "brain-preference",
+ id,
+ created_at: "2026-01-01T00:00:00Z",
+ confirmed_at: null,
+ unconfirmed_until: "2026-01-08T00:00:00Z",
+ tags: [],
+ topic: id.replace(/^pref-/, ""),
+ status: BRAIN_PREFERENCE_STATUS.confirmed,
+ principle: `principle for ${id}`,
+ evidenced_by: [],
+ applied_count: 0,
+ violated_count: 0,
+ last_evidence_at: null,
+ confidence: "medium",
+ confidence_value: 0.5,
+ pinned: false,
+ };
+ return { ...base, ...overrides };
+}
+
+function seed(
+ vault: string,
+ timestamp: string,
+ prefWikilink: string,
+ result: string,
+ artifact = "[[src/foo.ts]]",
+): void {
+ appendLogEvent(vault, {
+ timestamp,
+ eventType: BRAIN_LOG_EVENT_KIND.applyEvidence,
+ body: { preference: prefWikilink, artifact, agent: "tester", result },
+ });
+}
+
+/** A lightweight dead-end entry (only the fields computeLessons reads). */
+function deadEnd(id: string, approach: string, created_at: string): DeadEndEntry {
+ return { id, path: `${id}.md`, approach, reason: "why", context: null, agent: "t", created_at };
+}
+
+describe("computeLessons", () => {
+ let vault: string;
+ beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "osb-lessons-"));
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ });
+ afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+ });
+
+ test("empty corpus yields no lessons", () => {
+ expect(computeLessons(vault, [], [], { now: NOW })).toEqual([]);
+ });
+
+ test("a preference with no outcome evidence is not yet a lesson", () => {
+ const result = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(result).toEqual([]);
+ });
+
+ test("two distinct applied artifacts promote to preferred", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ seed(vault, "2026-05-19T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[b.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.preferred);
+ expect(lesson!.corroboration).toBe(2);
+ expect(lesson!.signedScore).toBeGreaterThan(0);
+ });
+
+ test("a single applied artifact stays tentative (below corroboration floor)", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ seed(vault, "2026-05-19T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.tentative);
+ expect(lesson!.corroboration).toBe(1); // same artifact twice = one distinct result
+ });
+
+ test("corroborationMin option gates promotion", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ seed(vault, "2026-05-19T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[b.ts]]");
+ const [strict] = computeLessons(vault, [buildPref("pref-a")], [], {
+ now: NOW,
+ corroborationMin: 3,
+ });
+ expect(strict!.tier).toBe(LESSON_TIER.tentative);
+ });
+
+ test("mixed applied + violated evidence renders contested", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.violated, "[[b.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.contested);
+ expect(lesson!.stance).not.toBeNull();
+ });
+
+ test("contested stance follows recency: a fresh violation outweighs a stale applied", () => {
+ // Stale positive (~4 months old), fresh negative (2 days old).
+ seed(vault, "2026-01-20T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.violated, "[[b.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.contested);
+ expect(lesson!.stance).toBe(LESSON_STANCE.negative);
+ expect(lesson!.signedScore).toBeLessThan(0);
+ });
+
+ test("a rule only ever violated is avoid", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.violated, "[[a.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.avoid);
+ expect(lesson!.signedScore).toBeLessThan(0);
+ });
+
+ test("dead-ends are avoid lessons with a negative score", () => {
+ const de = deadEnd("de-2026-05-18-x", "tried the flaky retry loop", "2026-05-18T00:00:00Z");
+ const [lesson] = computeLessons(vault, [], [de], { now: NOW });
+ expect(lesson!.kind).toBe("dead-end");
+ expect(lesson!.tier).toBe(LESSON_TIER.avoid);
+ expect(lesson!.negativeMass).toBeGreaterThan(0);
+ expect(lesson!.positiveMass).toBe(0);
+ expect(lesson!.title).toBe("tried the flaky retry loop");
+ });
+
+ test("recency: a fresh dead-end outscores a stale preference in salience ranking", () => {
+ // Stale applied preference (~4 months old) vs a 1-day-old dead-end.
+ seed(vault, "2026-01-20T00:00:00Z", "[[pref-old]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const de = deadEnd("de-2026-05-19-x", "fresh failed approach", "2026-05-19T00:00:00Z");
+ const result = computeLessons(vault, [buildPref("pref-old")], [de], { now: NOW });
+ // The fresh dead-end has larger decayed mass, so it ranks first.
+ expect(result[0]!.id).toBe("de-2026-05-19-x");
+ });
+
+ test("outdated counts as a negative outcome", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.outdated, "[[a.ts]]");
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.avoid);
+ });
+
+ test("evidence predating the preference's created_at is ignored", () => {
+ const pref = buildPref("pref-a", { created_at: "2026-05-01T00:00:00Z" });
+ seed(vault, "2026-04-01T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const result = computeLessons(vault, [pref], [], { now: NOW });
+ expect(result).toEqual([]);
+ });
+
+ test("retired-status preferences are excluded from scoring", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const pref = buildPref("pref-a", { status: "retired" as BrainPreference["status"] });
+ expect(computeLessons(vault, [pref], [], { now: NOW })).toEqual([]);
+ });
+
+ test("wikilink aliases collapse to the bare id", () => {
+ seed(
+ vault,
+ "2026-05-18T00:00:00Z",
+ "[[pref-a|principle text]]",
+ BRAIN_APPLY_RESULT.applied,
+ "[[a.ts]]",
+ );
+ seed(
+ vault,
+ "2026-05-19T00:00:00Z",
+ "[[pref-a|principle text]]",
+ BRAIN_APPLY_RESULT.applied,
+ "[[b.ts]]",
+ );
+ const [lesson] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(lesson!.tier).toBe(LESSON_TIER.preferred);
+ });
+
+ test("halfLifeDays option sharpens the decay", () => {
+ // 30 days old applied event. A short half-life shrinks its weight.
+ seed(vault, "2026-04-20T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const [gentle] = computeLessons(vault, [buildPref("pref-a")], [], {
+ now: NOW,
+ halfLifeDays: 60,
+ });
+ const [sharp] = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW, halfLifeDays: 5 });
+ expect(gentle!.signedScore).toBeGreaterThan(sharp!.signedScore);
+ });
+
+ test("limit truncates the ranked output", () => {
+ const des = [
+ deadEnd("de-1", "a", "2026-05-18T00:00:00Z"),
+ deadEnd("de-2", "b", "2026-05-17T00:00:00Z"),
+ deadEnd("de-3", "c", "2026-05-16T00:00:00Z"),
+ ];
+ const result = computeLessons(vault, [], des, { now: NOW, limit: 2 });
+ expect(result.length).toBe(2);
+ });
+
+ test("output is deterministic and frozen", () => {
+ seed(vault, "2026-05-18T00:00:00Z", "[[pref-a]]", BRAIN_APPLY_RESULT.applied, "[[a.ts]]");
+ const first = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ const second = computeLessons(vault, [buildPref("pref-a")], [], { now: NOW });
+ expect(first).toEqual(second);
+ expect(Object.isFrozen(first)).toBe(true);
+ expect(Object.isFrozen(first[0])).toBe(true);
+ });
+});
+
+describe("regenerateLessons", () => {
+ let vault: string;
+ beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "osb-lessons-regen-"));
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ });
+ afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+ });
+
+ test("writes a lessons.md digest and reports counts", () => {
+ recordDeadEnd(vault, {
+ approach: "polling loop that never converges",
+ reason: "burned the token budget",
+ agent: "tester",
+ now: new Date("2026-05-18T00:00:00Z"),
+ });
+ const result = regenerateLessons(vault, { now: NOW });
+ expect(result.changed).toBe(true);
+ expect(result.counts.avoid).toBe(1);
+ expect(result.counts.total).toBe(1);
+
+ const body = require("node:fs").readFileSync(result.path, "utf8") as string;
+ expect(body).toContain("kind: brain-lessons");
+ expect(body).toContain("## Avoid (1)");
+ expect(body).toContain("polling loop that never converges");
+ });
+
+ test("idempotent: a second regeneration with the same clock does not rewrite", () => {
+ recordDeadEnd(vault, {
+ approach: "x",
+ reason: "y",
+ agent: "tester",
+ now: new Date("2026-05-18T00:00:00Z"),
+ });
+ const first = regenerateLessons(vault, { now: NOW });
+ expect(first.changed).toBe(true);
+ const second = regenerateLessons(vault, { now: NOW });
+ expect(second.changed).toBe(false);
+ });
+
+ test("empty corpus renders a placeholder body", () => {
+ const result = regenerateLessons(vault, { now: NOW });
+ expect(result.counts.total).toBe(0);
+ const body = require("node:fs").readFileSync(result.path, "utf8") as string;
+ expect(body).toContain("No lessons yet");
+ });
+});
diff --git a/tests/core/brain/policy-lessons.test.ts b/tests/core/brain/policy-lessons.test.ts
new file mode 100644
index 00000000..7e60d2d6
--- /dev/null
+++ b/tests/core/brain/policy-lessons.test.ts
@@ -0,0 +1,77 @@
+/**
+ * `lessons:` config block (t_62363378) — tunables for the signed,
+ * recency-scored lessons digest (`Brain/lessons.md`).
+ */
+
+import { describe, expect, test } from "bun:test";
+import { parseBrainYaml } from "../../../src/core/brain/yaml-parse.ts";
+import { BrainConfigError, validateBrainConfigDetailed } from "../../../src/core/brain/policy.ts";
+
+function validate(yaml: string) {
+ return validateBrainConfigDetailed(parseBrainYaml(yaml), "<test>");
+}
+
+const HEAD = `schema_version: 1\n`;
+
+describe("lessons config block", () => {
+ test("absent block → cfg.lessons is undefined; consumers use defaults", () => {
+ const { config } = validate(HEAD);
+ expect(config.lessons).toBeUndefined();
+ });
+
+ test("present values load into BrainLessonsConfig", () => {
+ const { config } = validate(
+ HEAD + `lessons:\n half_life_days: 14\n corroboration_min: 3\n limit: 50\n`,
+ );
+ expect(config.lessons?.half_life_days).toBe(14);
+ expect(config.lessons?.corroboration_min).toBe(3);
+ expect(config.lessons?.limit).toBe(50);
+ });
+
+ test("partial block keeps other fields undefined (defaults apply downstream)", () => {
+ const { config } = validate(HEAD + `lessons:\n half_life_days: 45\n`);
+ expect(config.lessons?.half_life_days).toBe(45);
+ expect(config.lessons?.corroboration_min).toBeUndefined();
+ expect(config.lessons?.limit).toBeUndefined();
+ });
+
+ test("half_life_days: 0 rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n half_life_days: 0\n`)).toThrow(BrainConfigError);
+ });
+
+ test("half_life_days: 366 rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n half_life_days: 366\n`)).toThrow(BrainConfigError);
+ });
+
+ test("corroboration_min: 0 rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n corroboration_min: 0\n`)).toThrow(BrainConfigError);
+ });
+
+ test("limit: 201 rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n limit: 201\n`)).toThrow(BrainConfigError);
+ });
+
+ test("non-integer half_life_days rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n half_life_days: 30.5\n`)).toThrow(BrainConfigError);
+ });
+
+ test("string value rejected", () => {
+ expect(() => validate(HEAD + `lessons:\n half_life_days: "thirty"\n`)).toThrow(
+ BrainConfigError,
+ );
+ });
+
+ test("unknown sub-key warns, doesn't throw", () => {
+ const { warnings } = validate(HEAD + `lessons:\n half_life_days: 30\n future_thing: 1\n`);
+ expect(warnings.find((w) => w.message.includes("future_thing"))).toBeDefined();
+ });
+
+ test("non-object lessons value is a hard error", () => {
+ expect(() => validate(HEAD + `lessons: not-a-map\n`)).toThrow(BrainConfigError);
+ });
+
+ test("empty lessons block is permitted", () => {
+ const { config } = validate(HEAD + `lessons:\n __reserved: 1\n`);
+ expect(config.lessons).toEqual({});
+ });
+});
diff --git a/tests/core/brain/provenance/epistemic.test.ts b/tests/core/brain/provenance/epistemic.test.ts
new file mode 100644
index 00000000..4d1a639f
--- /dev/null
+++ b/tests/core/brain/provenance/epistemic.test.ts
@@ -0,0 +1,98 @@
+import { describe, expect, test } from "bun:test";
+
+import {
+ asEpistemicStatus,
+ deriveEpistemicStatus,
+ EPISTEMIC_STATUS,
+ readEvidenceRefs,
+} from "../../../../src/core/brain/provenance/epistemic.ts";
+
+describe("asEpistemicStatus", () => {
+ test("narrows a valid token case/space-insensitively", () => {
+ expect(asEpistemicStatus("observed")).toBe("observed");
+ expect(asEpistemicStatus(" Derived ")).toBe("derived");
+ expect(asEpistemicStatus("PLAN")).toBe("plan");
+ });
+
+ test("rejects unknown strings and non-strings", () => {
+ expect(asEpistemicStatus("guess")).toBeNull();
+ expect(asEpistemicStatus(42)).toBeNull();
+ expect(asEpistemicStatus(undefined)).toBeNull();
+ });
+});
+
+describe("readEvidenceRefs", () => {
+ test("reads an array, dropping blanks and trimming", () => {
+ expect(readEvidenceRefs({ evidenced_by: ["[[sig-a]]", " [[sig-b]] ", "", 7] })).toEqual([
+ "[[sig-a]]",
+ "[[sig-b]]",
+ ]);
+ });
+
+ test("accepts a lone string and tolerates absence", () => {
+ expect(readEvidenceRefs({ evidenced_by: "[[pref-x]]" })).toEqual(["[[pref-x]]"]);
+ expect(readEvidenceRefs({})).toEqual([]);
+ expect(readEvidenceRefs({ evidenced_by: null })).toEqual([]);
+ });
+});
+
+describe("deriveEpistemicStatus", () => {
+ test("stated + confirmed is observed and carries evidence refs", () => {
+ const m = deriveEpistemicStatus({
+ provenance: "stated",
+ status: "confirmed",
+ evidenced_by: ["[[sig-1]]"],
+ });
+ expect(m.status).toBe(EPISTEMIC_STATUS.observed);
+ expect(m.evidenceRefs).toEqual(["[[sig-1]]"]);
+ });
+
+ test("absent provenance defaults to stated -> observed", () => {
+ expect(deriveEpistemicStatus({ status: "confirmed" }).status).toBe(EPISTEMIC_STATUS.observed);
+ // a page with no status at all is still an authored (observed) statement
+ expect(deriveEpistemicStatus({}).status).toBe(EPISTEMIC_STATUS.observed);
+ });
+
+ test("deduced / inferred provenance is derived", () => {
+ expect(deriveEpistemicStatus({ provenance: "deduced", evidenced_by: ["[[pref-a]]"] })).toEqual({
+ status: EPISTEMIC_STATUS.derived,
+ evidenceRefs: ["[[pref-a]]"],
+ });
+ expect(deriveEpistemicStatus({ provenance: "inferred" }).status).toBe(EPISTEMIC_STATUS.derived);
+ });
+
+ test("unconfirmed / quarantine stated preference is hypothesis", () => {
+ expect(deriveEpistemicStatus({ status: "unconfirmed" }).status).toBe(
+ EPISTEMIC_STATUS.hypothesis,
+ );
+ expect(deriveEpistemicStatus({ status: "quarantine" }).status).toBe(
+ EPISTEMIC_STATUS.hypothesis,
+ );
+ });
+
+ test("a disputed lifecycle is unknown regardless of provenance", () => {
+ expect(
+ deriveEpistemicStatus({ _lifecycle: "disputed", provenance: "stated", status: "confirmed" })
+ .status,
+ ).toBe(EPISTEMIC_STATUS.unknown);
+ expect(deriveEpistemicStatus({ lifecycle: "disputed" }).status).toBe(EPISTEMIC_STATUS.unknown);
+ });
+
+ test("an explicit epistemic value overrides derivation - the only plan path", () => {
+ const m = deriveEpistemicStatus({
+ epistemic: "plan",
+ provenance: "stated",
+ status: "confirmed",
+ evidenced_by: ["[[sig-2]]"],
+ });
+ expect(m.status).toBe(EPISTEMIC_STATUS.plan);
+ // evidence refs still surface under an override
+ expect(m.evidenceRefs).toEqual(["[[sig-2]]"]);
+ });
+
+ test("an invalid explicit value falls through to derivation", () => {
+ expect(deriveEpistemicStatus({ epistemic: "bogus", status: "unconfirmed" }).status).toBe(
+ EPISTEMIC_STATUS.hypothesis,
+ );
+ });
+});
diff --git a/tests/core/brain/query-demand.test.ts b/tests/core/brain/query-demand.test.ts
new file mode 100644
index 00000000..0c373e23
--- /dev/null
+++ b/tests/core/brain/query-demand.test.ts
@@ -0,0 +1,227 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, mkdtempSync, readFileSync, rmSync, statSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { queryDemandLogPath } from "../../../src/core/brain/paths.ts";
+import {
+ DEMAND_LOG_MAX_BYTES,
+ aggregateQueryDemand,
+ normalizeQueryTerms,
+ readQueryDemand,
+ recordQueryDemand,
+} from "../../../src/core/brain/query-demand.ts";
+
+let vault: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-query-demand-"));
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+});
+
+describe("normalizeQueryTerms", () => {
+ test("keeps significant terms, sorts, dedupes, drops short tokens", () => {
+ expect(normalizeQueryTerms("How does the AUTH flow work flow")).toEqual([
+ "auth",
+ "does",
+ "flow",
+ "how",
+ "the",
+ "work",
+ ]);
+ });
+
+ test("drops secret-shaped tokens the redactor rewrites", () => {
+ const terms = normalizeQueryTerms("api key sk-abcdef0123456789abcdef0123456789 rotation");
+ expect(terms).toContain("api");
+ expect(terms).toContain("key");
+ expect(terms).toContain("rotation");
+ expect(terms.some((t) => t.includes("sk-abcdef"))).toBe(false);
+ expect(terms.some((t) => t.includes("REDACTED"))).toBe(false);
+ });
+
+ test("empty when no significant terms", () => {
+ expect(normalizeQueryTerms("a it")).toEqual([]);
+ });
+});
+
+describe("recordQueryDemand", () => {
+ test("appends a normalized record and never stores the raw query", () => {
+ const record = recordQueryDemand(vault, {
+ query: "deployment rollback procedure",
+ resultCount: 0,
+ coverage: 0.1,
+ at: "2026-07-01T10:00:00.000Z",
+ });
+ expect(record).not.toBeNull();
+ expect(record!.terms).toEqual(["deployment", "procedure", "rollback"]);
+ expect(record!.results).toBe(0);
+ expect(record!.coverage).toBe(0.1);
+
+ const raw = readFileSync(queryDemandLogPath(vault), "utf8");
+ expect(raw).not.toContain("deployment rollback procedure");
+ expect(raw.trim().split("\n")).toHaveLength(1);
+ });
+
+ test("omits coverage when not provided", () => {
+ const record = recordQueryDemand(vault, { query: "orphaned notes cleanup", resultCount: 3 });
+ expect(record!.coverage).toBeUndefined();
+ });
+
+ test("does not record a query with no significant terms", () => {
+ const record = recordQueryDemand(vault, { query: "a of to", resultCount: 5 });
+ expect(record).toBeNull();
+ expect(existsSync(queryDemandLogPath(vault))).toBe(false);
+ });
+
+ test("clamps out-of-range coverage and negative result counts", () => {
+ const r = recordQueryDemand(vault, {
+ terms: ["alpha"],
+ resultCount: -4,
+ coverage: 1.7,
+ at: "2026-07-01T10:00:00.000Z",
+ });
+ expect(r!.results).toBe(0);
+ expect(r!.coverage).toBe(1);
+ });
+});
+
+describe("readQueryDemand", () => {
+ test("filters by since/until and sorts by timestamp", () => {
+ recordQueryDemand(vault, { terms: ["aaa"], resultCount: 1, at: "2026-07-03T00:00:00.000Z" });
+ recordQueryDemand(vault, { terms: ["bbb"], resultCount: 1, at: "2026-07-01T00:00:00.000Z" });
+ recordQueryDemand(vault, { terms: ["ccc"], resultCount: 1, at: "2026-07-02T00:00:00.000Z" });
+
+ const all = readQueryDemand(vault);
+ expect(all.map((r) => r.ts)).toEqual([
+ "2026-07-01T00:00:00.000Z",
+ "2026-07-02T00:00:00.000Z",
+ "2026-07-03T00:00:00.000Z",
+ ]);
+
+ const windowed = readQueryDemand(vault, {
+ since: "2026-07-02T00:00:00.000Z",
+ until: "2026-07-02T23:59:59.000Z",
+ });
+ expect(windowed.map((r) => r.terms[0])).toEqual(["ccc"]);
+ });
+
+ test("empty when the log does not exist", () => {
+ expect(readQueryDemand(vault)).toEqual([]);
+ });
+});
+
+describe("aggregateQueryDemand", () => {
+ test("buckets recurring poorly-answered queries and ranks by demand", () => {
+ // "vault backups" asked 3x, always weak coverage -> high demand.
+ for (let i = 0; i < 3; i++) {
+ recordQueryDemand(vault, {
+ query: "vault backups schedule",
+ resultCount: 0,
+ coverage: 0.05,
+ at: `2026-07-0${i + 1}T00:00:00.000Z`,
+ });
+ }
+ // "release process" asked 2x, well covered -> excluded by default.
+ for (let i = 0; i < 2; i++) {
+ recordQueryDemand(vault, {
+ query: "release process steps",
+ resultCount: 5,
+ coverage: 0.95,
+ at: `2026-07-0${i + 1}T01:00:00.000Z`,
+ });
+ }
+ // "token budget" asked once -> below minOccurrences.
+ recordQueryDemand(vault, {
+ query: "token budget tuning",
+ resultCount: 1,
+ coverage: 0.2,
+ at: "2026-07-01T02:00:00.000Z",
+ });
+
+ const report = aggregateQueryDemand(vault);
+ expect(report.totalRecords).toBe(6);
+ expect(report.distinctQueries).toBe(3);
+ expect(report.gaps).toHaveLength(1);
+
+ const gap = report.gaps[0]!;
+ expect(gap.terms).toEqual(["backups", "schedule", "vault"]);
+ expect(gap.occurrences).toBe(3);
+ expect(gap.meanCoverage).toBeCloseTo(0.05, 5);
+ expect(gap.emptyCount).toBe(3);
+ expect(gap.emptyRate).toBe(1);
+ expect(gap.coverageKnown).toBe(true);
+ expect(gap.verdict).toBe("sparse");
+ expect(gap.demandScore).toBeCloseTo(3 * (1 - 0.05), 4);
+ expect(gap.firstSeen).toBe("2026-07-01T00:00:00.000Z");
+ expect(gap.lastSeen).toBe("2026-07-03T00:00:00.000Z");
+ });
+
+ test("falls back to result-presence when no coverage was recorded", () => {
+ recordQueryDemand(vault, { query: "cron secrets", resultCount: 0, at: "2026-07-01T00:00:00Z" });
+ recordQueryDemand(vault, { query: "cron secrets", resultCount: 0, at: "2026-07-02T00:00:00Z" });
+
+ const report = aggregateQueryDemand(vault);
+ expect(report.gaps).toHaveLength(1);
+ const gap = report.gaps[0]!;
+ expect(gap.coverageKnown).toBe(false);
+ expect(gap.meanCoverage).toBeNull();
+ expect(gap.satisfaction).toBe(0);
+ expect(gap.demandScore).toBe(2);
+ });
+
+ test("honors minOccurrences, maxSatisfaction, and limit", () => {
+ for (let i = 0; i < 4; i++) {
+ recordQueryDemand(vault, {
+ query: "graph traversal depth",
+ resultCount: 2,
+ coverage: 0.5,
+ at: `2026-07-0${i + 1}T00:00:00.000Z`,
+ });
+ }
+ // Excluded when maxSatisfaction is stricter than 0.5.
+ expect(aggregateQueryDemand(vault, { maxSatisfaction: 0.4 }).gaps).toHaveLength(0);
+ // Included at the default 0.8 ceiling.
+ expect(aggregateQueryDemand(vault, { maxSatisfaction: 0.8 }).gaps).toHaveLength(1);
+ // Excluded when minOccurrences exceeds the count.
+ expect(aggregateQueryDemand(vault, { minOccurrences: 5 }).gaps).toHaveLength(0);
+
+ for (let i = 0; i < 3; i++) {
+ recordQueryDemand(vault, {
+ query: "another weak query",
+ resultCount: 0,
+ coverage: 0.0,
+ at: `2026-07-0${i + 1}T03:00:00.000Z`,
+ });
+ }
+ const limited = aggregateQueryDemand(vault, { limit: 1 });
+ expect(limited.gaps).toHaveLength(1);
+ // The zero-coverage query outranks the 0.5-coverage one.
+ expect(limited.gaps[0]!.terms).toEqual(["another", "query", "weak"]);
+ });
+});
+
+describe("rolling cap", () => {
+ test("compacts the log once it exceeds the byte budget", () => {
+ const path = queryDemandLogPath(vault);
+ // Each record ~ small; write enough to blow past the 1MB cap.
+ const terms = ["compaction", "stress", "budget"];
+ for (let i = 0; i < 20_000; i++) {
+ recordQueryDemand(vault, {
+ terms,
+ resultCount: 0,
+ coverage: 0.1,
+ at: "2026-07-01T00:00:00.000Z",
+ });
+ }
+ const size = statSync(path).size;
+ expect(size).toBeLessThanOrEqual(DEMAND_LOG_MAX_BYTES);
+ // Compaction keeps recent lines readable and parseable.
+ const records = readQueryDemand(vault);
+ expect(records.length).toBeGreaterThan(0);
+ expect(records.every((r) => r.terms.join(" ") === "budget compaction stress")).toBe(true);
+ });
+});
diff --git a/tests/core/brain/recall-adequacy.test.ts b/tests/core/brain/recall-adequacy.test.ts
new file mode 100644
index 00000000..1e25a263
--- /dev/null
+++ b/tests/core/brain/recall-adequacy.test.ts
@@ -0,0 +1,84 @@
+import { expect, test } from "bun:test";
+
+import {
+ assessRecallAdequacy,
+ DEFAULT_RECALL_ADEQUACY_THRESHOLDS,
+} from "../../../src/core/brain/recall-adequacy.ts";
+
+test("a strong top hit is sufficient and proceeds", () => {
+ const verdict = assessRecallAdequacy([0.82, 0.5, 0.31]);
+ expect(verdict.level).toBe("sufficient");
+ expect(verdict.action).toBe("proceed");
+ expect(verdict.escalate).toBe(false);
+ expect(verdict.topScore).toBeCloseTo(0.82);
+ expect(verdict.resultCount).toBe(3);
+});
+
+test("a middling top hit is weak and triggers re-recall", () => {
+ const verdict = assessRecallAdequacy([0.42, 0.3]);
+ expect(verdict.level).toBe("weak");
+ expect(verdict.action).toBe("re_recall");
+ expect(verdict.escalate).toBe(false);
+});
+
+test("a poor top hit is insufficient, abstains, and escalates", () => {
+ const verdict = assessRecallAdequacy([0.12, 0.08]);
+ expect(verdict.level).toBe("insufficient");
+ expect(verdict.action).toBe("abstain");
+ expect(verdict.escalate).toBe(true);
+});
+
+test("no results is insufficient with zero scores and escalates", () => {
+ const verdict = assessRecallAdequacy([]);
+ expect(verdict.level).toBe("insufficient");
+ expect(verdict.action).toBe("abstain");
+ expect(verdict.escalate).toBe(true);
+ expect(verdict.topScore).toBe(0);
+ expect(verdict.meanScore).toBe(0);
+ expect(verdict.resultCount).toBe(0);
+});
+
+test("min_results downgrades a strong-but-lonely hit to weak/re-recall", () => {
+ const verdict = assessRecallAdequacy([0.9], { minResults: 2 });
+ expect(verdict.level).toBe("weak");
+ expect(verdict.action).toBe("re_recall");
+ expect(verdict.escalate).toBe(false);
+});
+
+test("custom thresholds move the boundaries", () => {
+ // Raise the sufficient floor so 0.7 is no longer sufficient.
+ const verdict = assessRecallAdequacy([0.7, 0.4], { sufficient: 0.8, weak: 0.5 });
+ expect(verdict.level).toBe("weak");
+});
+
+test("non-finite scores are ignored; the rest still classify", () => {
+ const verdict = assessRecallAdequacy([Number.NaN, Infinity, 0.75]);
+ expect(verdict.resultCount).toBe(1);
+ expect(verdict.level).toBe("sufficient");
+ expect(verdict.topScore).toBeCloseTo(0.75);
+});
+
+test("negative scores clamp to zero", () => {
+ const verdict = assessRecallAdequacy([-0.3, 0.05]);
+ expect(verdict.topScore).toBeCloseTo(0.05);
+ expect(verdict.level).toBe("insufficient");
+});
+
+test("defaults are the documented 0.6 / 0.3 / 1", () => {
+ expect(DEFAULT_RECALL_ADEQUACY_THRESHOLDS).toEqual({
+ sufficient: 0.6,
+ weak: 0.3,
+ minResults: 1,
+ });
+});
+
+test("invalid thresholds throw", () => {
+ expect(() => assessRecallAdequacy([0.5], { sufficient: 0.2, weak: 0.5 })).toThrow();
+ expect(() => assessRecallAdequacy([0.5], { sufficient: 1.5 })).toThrow();
+ expect(() => assessRecallAdequacy([0.5], { minResults: 0 })).toThrow();
+});
+
+test("verdict carries a human-readable reason", () => {
+ expect(assessRecallAdequacy([0.9]).reason).toContain("sufficient");
+ expect(assessRecallAdequacy([]).reason.length).toBeGreaterThan(0);
+});
diff --git a/tests/core/config.test.ts b/tests/core/config.test.ts
index 71216c01..b1749336 100644
--- a/tests/core/config.test.ts
+++ b/tests/core/config.test.ts
@@ -10,6 +10,7 @@ import {
redactMapping,
resolveAgentName,
resolveLinkOutputFormat,
+ resolveRecallAdequacyThresholds,
resolveSkillsAttachTriggers,
resolveSkillsDir,
resolveTimezone,
@@ -32,6 +33,9 @@ beforeEach(() => {
"OBSIDIAN_LINK_FORMAT",
"OPEN_SECOND_BRAIN_SKILLS_DIR",
"OPEN_SECOND_BRAIN_SKILLS_ATTACH_TRIGGERS",
+ "OPEN_SECOND_BRAIN_RECALL_ADEQUACY_SUFFICIENT",
+ "OPEN_SECOND_BRAIN_RECALL_ADEQUACY_WEAK",
+ "OPEN_SECOND_BRAIN_RECALL_ADEQUACY_MIN_RESULTS",
]) {
savedEnv[k] = process.env[k];
delete process.env[k];
@@ -357,3 +361,53 @@ describe("resolveSkillsAttachTriggers", () => {
expect(resolveSkillsAttachTriggers(cfg)).toBe(true);
});
});
+
+describe("resolveRecallAdequacyThresholds", () => {
+ test("defaults to 0.6 / 0.3 / 1 when unset", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(cfg, "");
+ expect(resolveRecallAdequacyThresholds(cfg)).toEqual({
+ sufficient: 0.6,
+ weak: 0.3,
+ minResults: 1,
+ });
+ });
+
+ test("reads configured floors and min_results", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(
+ cfg,
+ 'recall_adequacy_sufficient: "0.75"\nrecall_adequacy_weak: "0.4"\nrecall_adequacy_min_results: "2"\n',
+ );
+ expect(resolveRecallAdequacyThresholds(cfg)).toEqual({
+ sufficient: 0.75,
+ weak: 0.4,
+ minResults: 2,
+ });
+ });
+
+ test("env overrides the config file", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(cfg, 'recall_adequacy_sufficient: "0.6"\n');
+ process.env["OPEN_SECOND_BRAIN_RECALL_ADEQUACY_SUFFICIENT"] = "0.9";
+ expect(resolveRecallAdequacyThresholds(cfg).sufficient).toBe(0.9);
+ });
+
+ test("rejects out-of-range floors", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(cfg, 'recall_adequacy_sufficient: "1.5"\n');
+ expect(() => resolveRecallAdequacyThresholds(cfg)).toThrow();
+ });
+
+ test("rejects weak above sufficient", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(cfg, 'recall_adequacy_sufficient: "0.4"\nrecall_adequacy_weak: "0.6"\n');
+ expect(() => resolveRecallAdequacyThresholds(cfg)).toThrow();
+ });
+
+ test("rejects a non-positive min_results", () => {
+ const cfg = join(tmp, "config.yaml");
+ writeFileSync(cfg, 'recall_adequacy_min_results: "0"\n');
+ expect(() => resolveRecallAdequacyThresholds(cfg)).toThrow();
+ });
+});
diff --git a/tests/hooks/active-inject.test.ts b/tests/hooks/active-inject.test.ts
index 5537217f..5f84a2f0 100644
--- a/tests/hooks/active-inject.test.ts
+++ b/tests/hooks/active-inject.test.ts
@@ -60,6 +60,10 @@ function writeActive(body: string): void {
writeFileSync(join(vault, "Brain", "active.md"), body, "utf8");
}
+function writeLessons(body: string): void {
+ writeFileSync(join(vault, "Brain", "lessons.md"), body, "utf8");
+}
+
describe("active-inject hook", () => {
test("injects active.md content as SessionStart additionalContext", async () => {
writeActive(
@@ -206,6 +210,36 @@ describe("active-inject hook", () => {
expect(injected).toContain("brain_context");
});
+ test("appends lessons.md body alongside active.md when present", async () => {
+ writeActive(
+ "---\nkind: brain-active\ngenerated_at: 2026-05-15T10:00:00Z\n---\n\n# Active Brain Preferences\n\n## Confirmed (1)\n\n- `pref-foo` — Rule body\n",
+ );
+ writeLessons(
+ "---\nkind: brain-lessons\ngenerated_at: 2026-05-15T10:00:00Z\n---\n\n# Lessons\n\n## Avoid (1)\n\n- `de-2026-05-18-x` (score: -0.95) — the flaky retry loop\n",
+ );
+
+ const r = await runHook({ hook_event_name: "SessionStart" }, { VAULT_DIR: vault });
+ expect(r.exit).toBe(0);
+ const out = JSON.parse(r.stdout);
+ const injected: string = out.hookSpecificOutput.additionalContext;
+ expect(injected).toContain("# Active Brain Preferences");
+ expect(injected).toContain("pref-foo");
+ expect(injected).toContain("# Lessons");
+ expect(injected).toContain("the flaky retry loop");
+ });
+
+ test("injects active.md alone when lessons.md is absent", async () => {
+ writeActive(
+ "---\nkind: brain-active\ngenerated_at: 2026-05-15T10:00:00Z\n---\n\n# Active Brain Preferences\n\n## Confirmed (1)\n\n- `pref-foo` — Rule body\n",
+ );
+ const r = await runHook({ hook_event_name: "SessionStart" }, { VAULT_DIR: vault });
+ expect(r.exit).toBe(0);
+ const out = JSON.parse(r.stdout);
+ const injected: string = out.hookSpecificOutput.additionalContext;
+ expect(injected).toContain("pref-foo");
+ expect(injected).not.toContain("# Lessons");
+ });
+
test("stays silent when active.md is empty whitespace only", async () => {
writeActive(" \n \n");
const r = await runHook({ hook_event_name: "SessionStart" }, { VAULT_DIR: vault });
diff --git a/tests/mcp/brain-tools-parity.test.ts b/tests/mcp/brain-tools-parity.test.ts
index 6833d117..2294da0b 100644
--- a/tests/mcp/brain-tools-parity.test.ts
+++ b/tests/mcp/brain-tools-parity.test.ts
@@ -66,6 +66,7 @@ const FROZEN_BRAIN_TOOL_NAMES = [
"brain_intake_entities",
"brain_intent_review",
"brain_intention",
+ "brain_knowledge_gaps",
"brain_labels",
"brain_maintenance",
"brain_mcp_landscape",
diff --git a/tests/mcp/context-pack-tool.test.ts b/tests/mcp/context-pack-tool.test.ts
index 7c8a4e5b..2514e2b5 100644
--- a/tests/mcp/context-pack-tool.test.ts
+++ b/tests/mcp/context-pack-tool.test.ts
@@ -100,9 +100,24 @@ describe("brain_context_pack tool — round trip", () => {
const out = await callPack(server, { max_tokens: 10_000 });
expect(out["max_tokens"]).toBe(10000);
expect(typeof out["tokens_used"]).toBe("number");
- const items = out["items"] as Array<{ id: string; tier: string }>;
+ const items = out["items"] as Array<{ id: string; tier: string; epistemic: string }>;
expect(items[0]!.id).toBe("pref-core");
expect(items[1]!.id).toBe("pref-supp");
+ // Every emitted item carries an epistemic provenance marker (ACM).
+ expect(items[0]!.epistemic).toBe("observed");
+ });
+
+ test("emits derived epistemic status and evidence_refs from provenance metadata", async () => {
+ writeFileSync(
+ join(vault, "Brain", "preferences", "pref-derived.md"),
+ '---\nid: pref-derived\ntopic: x\nprinciple: derived principle\ntier: core\nprovenance: inferred\n_evidenced_by: ["[[pref-a]]"]\n---\n',
+ );
+ const server = new MCPServer({ vault, configPath });
+ await initialize(server);
+ const out = await callPack(server, { max_tokens: 10_000 });
+ const items = out["items"] as Array<{ epistemic: string; evidence_refs?: string[] }>;
+ expect(items[0]!.epistemic).toBe("derived");
+ expect(items[0]!.evidence_refs).toEqual(["[[pref-a]]"]);
});
test("returns polarity lanes when requested", async () => {
diff --git a/tests/mcp/knowledge-gaps-tool.test.ts b/tests/mcp/knowledge-gaps-tool.test.ts
new file mode 100644
index 00000000..6473128c
--- /dev/null
+++ b/tests/mcp/knowledge-gaps-tool.test.ts
@@ -0,0 +1,115 @@
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { recordQueryDemand } from "../../src/core/brain/query-demand.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";
+import { buildToolTable } from "../../src/mcp/tools.ts";
+
+let tmp: string;
+let vault: string;
+let configHome: string;
+let configPath: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-mcp-knowledge-gaps-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ configHome = mkdtempSync(join(tmpdir(), "o2b-mcp-knowledge-gaps-cfg-"));
+ configPath = join(configHome, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ rmSync(configHome, { recursive: true, force: true });
+});
+
+async function initialize(server: MCPServer): Promise<void> {
+ await server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ id: 1,
+ method: "initialize",
+ params: {
+ protocolVersion: PROTOCOL_VERSION,
+ capabilities: {},
+ clientInfo: { name: "knowledge-gaps-test", version: "0" },
+ },
+ });
+ await server.handleRequest({ jsonrpc: JSONRPC_VERSION, method: "notifications/initialized" });
+}
+
+async function callTool(
+ server: MCPServer,
+ name: string,
+ args: Record<string, unknown>,
+): Promise<Record<string, unknown>> {
+ const response = (await server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ id: 9,
+ method: "tools/call",
+ params: { name, arguments: args },
+ })) as
+ | { result: { content: ReadonlyArray<{ type: string; text: string }> } }
+ | { error: { message: string } };
+ if ("error" in response) throw new Error(response.error.message);
+ return JSON.parse(response.result.content[0]!.text);
+}
+
+describe("brain_knowledge_gaps tool registration", () => {
+ test("registered in the full tool table only", () => {
+ expect(
+ buildToolTable("full").find((tool) => tool.name === "brain_knowledge_gaps"),
+ ).toBeDefined();
+ expect(
+ buildToolTable("writer").find((tool) => tool.name === "brain_knowledge_gaps"),
+ ).toBeUndefined();
+ });
+});
+
+describe("brain_knowledge_gaps tool", () => {
+ test("ranks recurring poorly-answered queries and honors filters", async () => {
+ for (let i = 0; i < 3; i++) {
+ recordQueryDemand(vault, {
+ query: "vault backup schedule",
+ resultCount: 0,
+ coverage: 0.05,
+ at: `2026-07-0${i + 1}T00:00:00.000Z`,
+ });
+ }
+ recordQueryDemand(vault, {
+ query: "release checklist steps",
+ resultCount: 5,
+ coverage: 0.95,
+ at: "2026-07-01T01:00:00.000Z",
+ });
+ recordQueryDemand(vault, {
+ query: "release checklist steps",
+ resultCount: 5,
+ coverage: 0.95,
+ at: "2026-07-02T01:00:00.000Z",
+ });
+
+ const server = new MCPServer({ vault, configPath });
+ await initialize(server);
+
+ const report = await callTool(server, "brain_knowledge_gaps", {});
+ expect(report["vault_path"]).toBe(vault);
+ expect(report["total_records"]).toBe(5);
+ const gaps = report["gaps"] as ReadonlyArray<Record<string, unknown>>;
+ expect(gaps).toHaveLength(1);
+ expect(gaps[0]!["terms"]).toEqual(["backup", "schedule", "vault"]);
+ expect(gaps[0]!["occurrences"]).toBe(3);
+ expect(gaps[0]!["verdict"]).toBe("sparse");
+ });
+
+ test("rejects an out-of-range max_satisfaction", async () => {
+ const server = new MCPServer({ vault, configPath });
+ await initialize(server);
+ await expect(callTool(server, "brain_knowledge_gaps", { max_satisfaction: 2 })).rejects.toThrow(
+ /max_satisfaction/,
+ );
+ });
+});
diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.ts
index cf4c66fe..21cc0780 100644
--- a/tests/mcp/mcp.test.ts
+++ b/tests/mcp/mcp.test.ts
@@ -221,6 +221,7 @@ describe("tool listing", () => {
"brain_context_receipts",
"brain_event_trace",
"brain_recall_telemetry",
+ "brain_knowledge_gaps",
"brain_context_presets",
"brain_pre_compact_extract",
"brain_session_grep",
@@ -565,7 +566,8 @@ describe("stdio loop", () => {
// memory-subsystem-alignment) = 86.
// + brain_event_trace (dashboard-context-trace: event→trace join) = 87.
// + brain_search_expand (progressive disclosure: search→expand→transcript) = 88.
- expect(list.result.tools.length).toBe(88);
+ // + brain_knowledge_gaps (cross-query demand log, t_97091fff) = 89.
+ expect(list.result.tools.length).toBe(89);
});
test("returns parse error for invalid JSON", async () => {
diff --git a/tests/mcp/recall-adequacy-tool.test.ts b/tests/mcp/recall-adequacy-tool.test.ts
new file mode 100644
index 00000000..2b477d84
--- /dev/null
+++ b/tests/mcp/recall-adequacy-tool.test.ts
@@ -0,0 +1,124 @@
+/**
+ * Recall adequacy verdict wiring (retrieval-precision-quality-loop,
+ * t_b8f66fec): `brain_recall_gate` returns a verdict when `scores` is
+ * supplied, `brain_context_pack` persists it in the receipt, and
+ * `brain_context_receipts` surfaces it.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { buildToolTable, findTool, type ServerContext } from "../../src/mcp/tools.ts";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-adequacy-mcp-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ writeFileSync(configPath, `vault: "${vault}"\n`);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function ctx(): ServerContext {
+ return { vault, configPath, repoRoot: null };
+}
+
+function tool(name: string) {
+ return findTool(buildToolTable("full"), name);
+}
+
+test("gate omits adequacy when no scores are supplied", async () => {
+ const out = (await tool("brain_recall_gate").handler(ctx(), {
+ prompt: "what did we decide?",
+ })) as Record<string, unknown>;
+ expect(out["retrieve"]).toBe(true);
+ expect(out["adequacy"]).toBeUndefined();
+});
+
+test("gate returns a sufficient/proceed verdict for strong scores", async () => {
+ const out = (await tool("brain_recall_gate").handler(ctx(), {
+ prompt: "what did we decide?",
+ scores: [0.83, 0.4],
+ })) as { adequacy: Record<string, unknown> };
+ expect(out.adequacy["level"]).toBe("sufficient");
+ expect(out.adequacy["action"]).toBe("proceed");
+ expect(out.adequacy["escalate"]).toBe(false);
+ expect(out.adequacy["result_count"]).toBe(2);
+});
+
+test("gate returns weak/re_recall and insufficient/abstain per scores", async () => {
+ const weak = (await tool("brain_recall_gate").handler(ctx(), {
+ prompt: "x",
+ scores: [0.4],
+ })) as { adequacy: Record<string, unknown> };
+ expect(weak.adequacy["level"]).toBe("weak");
+ expect(weak.adequacy["action"]).toBe("re_recall");
+
+ const insufficient = (await tool("brain_recall_gate").handler(ctx(), {
+ prompt: "x",
+ scores: [],
+ })) as { adequacy: Record<string, unknown> };
+ expect(insufficient.adequacy["level"]).toBe("insufficient");
+ expect(insufficient.adequacy["action"]).toBe("abstain");
+ expect(insufficient.adequacy["escalate"]).toBe(true);
+});
+
+test("gate honours configurable thresholds", async () => {
+ writeFileSync(configPath, `vault: "${vault}"\nrecall_adequacy_sufficient: "0.9"\n`);
+ const out = (await tool("brain_recall_gate").handler(ctx(), {
+ prompt: "x",
+ scores: [0.7],
+ })) as { adequacy: Record<string, unknown> };
+ // 0.7 would be sufficient at the default 0.6 floor, but not at 0.9.
+ expect(out.adequacy["level"]).toBe("weak");
+});
+
+test("gate rejects a malformed scores argument", async () => {
+ await expect(
+ tool("brain_recall_gate").handler(ctx(), { prompt: "x", scores: ["nope"] }),
+ ).rejects.toThrow();
+});
+
+test("context_pack persists the verdict into the receipt and returns it", async () => {
+ const packed = (await tool("brain_context_pack").handler(ctx(), {
+ max_tokens: 1000,
+ receipt: true,
+ recall_scores: [0.2, 0.1],
+ })) as { adequacy?: Record<string, unknown>; receipt_id?: string };
+ expect(packed.adequacy).toBeDefined();
+ expect(packed.adequacy!["level"]).toBe("insufficient");
+ expect(packed.receipt_id).toBeDefined();
+
+ const summaries = (await tool("brain_context_receipts").handler(ctx(), {
+ operation: "list",
+ })) as { receipts: Array<{ adequacy?: Record<string, unknown> }> };
+ expect(summaries.receipts.length).toBeGreaterThan(0);
+ const withVerdict = summaries.receipts.find((r) => r.adequacy);
+ expect(withVerdict?.adequacy?.["level"]).toBe("insufficient");
+ expect(withVerdict?.adequacy?.["action"]).toBe("abstain");
+
+ const shown = (await tool("brain_context_receipts").handler(ctx(), {
+ operation: "show",
+ id: packed.receipt_id,
+ })) as { payload: Record<string, unknown> };
+ const adequacy = shown.payload["adequacy"] as Record<string, unknown>;
+ expect(adequacy["level"]).toBe("insufficient");
+ expect(adequacy["top_score"]).toBeCloseTo(0.2);
+});
+
+test("context_pack omits adequacy when recall_scores is absent", async () => {
+ const packed = (await tool("brain_context_pack").handler(ctx(), {
+ max_tokens: 1000,
+ receipt: true,
+ })) as { adequacy?: unknown };
+ expect(packed.adequacy).toBeUndefined();
+});
diff --git a/tests/mcp/removed-tools.test.ts b/tests/mcp/removed-tools.test.ts
index a4767b8d..914ed406 100644
--- a/tests/mcp/removed-tools.test.ts
+++ b/tests/mcp/removed-tools.test.ts
@@ -148,5 +148,6 @@ test("the shadow surface is gone: no hidden tools, removed names unlisted", asyn
// + brain_memory_bridge (Hermes on_memory_write host bridge) = 86.
// + brain_event_trace (dashboard-context-trace: event→trace join) = 87.
// + brain_search_expand (progressive disclosure: search→expand→transcript) = 88.
- expect(list.result.tools.length).toBe(88);
+ // + brain_knowledge_gaps (cross-query demand log, t_97091fff) = 89.
+ expect(list.result.tools.length).toBe(89);
});
diff --git a/tests/mcp/resources.test.ts b/tests/mcp/resources.test.ts
index a938eadd..b53aa0d6 100644
--- a/tests/mcp/resources.test.ts
+++ b/tests/mcp/resources.test.ts
@@ -111,7 +111,12 @@ describe("MCP resources — capabilities and discovery", () => {
await initialize(server);
const r = await listResources(server);
const uris = r.result.resources.map((x: any) => x.uri).toSorted();
- expect(uris).toEqual(["osb://digest/latest", "osb://preferences/active", "osb://status"]);
+ expect(uris).toEqual([
+ "osb://digest/latest",
+ "osb://lessons",
+ "osb://preferences/active",
+ "osb://status",
+ ]);
for (const desc of r.result.resources) {
expect(desc.mimeType).toBe("text/markdown");
expect(typeof desc.name).toBe("string");
@@ -187,6 +192,19 @@ describe("MCP resources — read osb://preferences/active", () => {
});
});
+describe("MCP resources — read osb://lessons", () => {
+ test("regenerates lessons.md on demand when missing and returns its body", async () => {
+ const server = makeServer();
+ await initialize(server);
+ const r = await read(server, "osb://lessons");
+ expect(r.result.contents).toHaveLength(1);
+ const c = r.result.contents[0];
+ expect(c.uri).toBe("osb://lessons");
+ expect(c.mimeType).toBe("text/markdown");
+ expect(c.text).toContain("# Lessons");
+ });
+});
+
describe("MCP resources — read osb://digest/latest", () => {
test("returns rendered digest markdown (empty window is non-empty markdown)", async () => {
const server = makeServer();
diff --git a/tests/mcp/search.test.ts b/tests/mcp/search.test.ts
index b89de650..5f2d532d 100644
--- a/tests/mcp/search.test.ts
+++ b/tests/mcp/search.test.ts
@@ -18,6 +18,7 @@ import { resolveSearchConfig } from "../../src/core/search/index.ts";
import { LATEST_SCHEMA_VERSION } from "../../src/core/search/schema.ts";
import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
import { listRecallTelemetry } from "../../src/core/brain/recall-telemetry.ts";
+import { readQueryDemand } from "../../src/core/brain/query-demand.ts";
let tmp: string;
let vault: string;
@@ -153,6 +154,24 @@ test("brain_search can opt in to recall telemetry", async () => {
});
expect(records).toHaveLength(1);
expect(records[0]!.payload).toMatchObject({ status: "ok", result_count: 1 });
+
+ // The same opt-in also persists a cross-query demand record with the
+ // normalized query terms and result count (t_97091fff).
+ const demand = readQueryDemand(vault);
+ expect(demand).toHaveLength(1);
+ expect(demand[0]!.terms).toEqual(["telemetry"]);
+ expect(demand[0]!.results).toBe(1);
+});
+
+test("brain_search does not persist demand records without the telemetry opt-in", async () => {
+ writeMd("notes/quiet.md", "# Quiet\n\nsilent recall leaves no demand trace.");
+ const cfg = resolveSearchConfig({ vault, configPath });
+ await indexVault(cfg);
+
+ const server = makeServer();
+ await initialize(server);
+ await call(server, "brain_search", { query: "silent", limit: 5 });
+ expect(readQueryDemand(vault)).toHaveLength(0);
});
test("brain_search accepts structured query_document", async () => {