Local-first ๐ง memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly ๐ด dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: recall and ranking quality - Weibull recency, query intent, synonym expansion, recall budgets, query cache, pre-compress pack (#47)
Sol Aitkenbd49cd5e36db
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 3f223062..fab2aa69 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.19.0",+ "version": "0.20.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 49eeaa97..ab12e4c1 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.19.0",+ "version": "0.20.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3cbb986e..ace3bc1f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,72 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [0.20.0] - 2026-05-29++Recall and ranking quality. The retrieval layer gains a tunable recency+curve, query-shape awareness, and language-agnostic recall broadening;+the context-injection surface gains character budgets and a+pre-compression bundle; and repeated searches can be served from a+self-invalidating cache.++### Added++- Configurable Weibull recency decay (`src/core/search/recency.ts`).+ The previous stepwise recency boost becomes a continuous Weibull+ survival curve with `search_recency_shape`, `search_recency_scale`+ (days), and `search_recency_amplitude` config keys. Defaults+ approximate the prior curve; a sub-display epsilon floors effectively+ stale content to exactly zero. Pure and deterministic.+- Query-intent classification via a pure `QueryPlan`+ (`src/core/search/query-plan.ts`). One structural pass classifies a+ query (neutral / exact / entity / broad) and emits a bounded ranking+ weight profile that re-weights the keyword / semantic / entity /+ recency layers. Intent derives only from query shape - quoted phrases,+ FTS wildcards, wikilinks, entity-token share, token count - never from+ any natural-language word list. On by default+ (`search_intent_enabled`); the neutral profile is bit-identical to+ prior ranking.+- Language-agnostic synonym / query expansion+ (`src/core/search/synonyms.ts`). Local co-occurrence (pseudo-relevance+ feedback) over the top candidates' own content surfaces related terms+ that are OR'd onto the FTS query to broaden recall, with no synonym+ dictionary or per-language list. Off by default+ (`search_synonym_enabled`), capped by `search_synonym_max_terms`, and+ always suppressed for exact-intent queries.+- Per-memory and total character caps for `brain_context_pack`, backed+ by a shared `recall-budget` primitive (`src/core/brain/recall-budget.ts`).+ `max_chars_per_memory` trims an oversized page before it consumes the+ token budget; `max_total_chars` bounds the cumulative size and drops+ the lowest-priority overflow. Caps are measured in Unicode code points.+- Persistent query cache with corpus-generation invalidation (schema+ migration v4, `query_cache` table). A search result is served for an+ identical request while the corpus generation - embedding model ++ dimension + schema version + a monotonic index revision bumped on every+ content reindex - is unchanged and the row is within the TTL. Off by+ default (`search_cache_enabled`, `search_cache_ttl_seconds`); cache+ reads and writes are best-effort and never fail a search.+- `brain_pre_compress_pack` MCP tool: a read-only, budgeted+ system-prompt addendum of the top confirmed preferences (by confidence,+ then recency) plus the head of `active.md`, for a host runtime to+ inject just before a context-compression event.++### Changed++- Search ranking now reflects the resolved recency curve and, when query+ intent is enabled, the per-query weight profile. Vaults that opt out of+ intent and keep the default recency parameters see ranking consistent+ with prior behaviour aside from the intentional recency-curve smoothing.++### Notes++- The schema migration to v4 is additive and reindex-safe; existing+ rows are preserved and a newer-than-supported index still raises+ `SCHEMA_MISMATCH`. As with prior schema bumps, run `o2b search reindex`+ after upgrading.+- Synonym expansion and the query cache are opt-in, so a default install+ keeps byte-identical search results (apart from the recency-curve+ change) until they are enabled.+## [0.19.0] - 2026-05-28Typed graph semantics. The vault's link graph and page frontmatter gaindiff --git a/README.md b/README.mdindex 402ceaab..afaad607 100644--- a/README.md+++ b/README.md@@ -112,7 +112,7 @@ The capabilities you actually feel day to day:| 9 | Hybrid search that explains itself | `o2b search "<query>" --property type=decision --property status=open` โ SQLite + FTS5 and an optional semantic layer, fused and then sharpened by a recall-quality suite: MMR diversity so near-duplicates do not crowd the top, link-graph traversal that surfaces notes one hop from a hit, entity-boosting for proper-noun precision, and header-anchored chunking so a mid-document chunk keeps its section context. Every result carries a `why_retrieved` breakdown of the layers that ranked it. || 10 | Pay Memory โ audit every paid action | When the agent spends money (Solana-Foundation `pay`, third-party APIs) every receipt lands in `Brain/payments/<date>/`. Spending policy, approval gate, daily Telegram report. The agent never holds wallet keys. |-These are the headline capabilities. The full surface also includes: importing Claude Code memory directories, daily logging-discipline cron with a complexity-to-thinking warning, cross-project pointers for shared vaults, transient pinned session context through `brain_pinned_context` / `Brain/pinned.md`, source-agent query and comparison (`brain_agent_query`, `brain_agent_diff`, `o2b brain agent-query`, `o2b brain agent-diff`), the codegraph partner skill, vault hygiene lints, per-MOC coverage audit, concept synthesis, an operator dashboard, configurable Markdown-link presentation output, MCP output-contract checks for core agent-facing tools, private-region stripping with `<private>...</private>`, the v0.12.0 Brain Integrity Suite (content-hash drift detection, durable workrun checkpoints, destructive-from-confirmed retire gate, and the `brain_review_candidates` MCP tool that previews what the next dream pass would do without writing anything), the v0.13.0 Hybrid Search and Recall Quality suite (explainable recall, MMR diversity, link-graph traversal, entity-boosted retrieval, header-anchored chunking), the v0.14.0 Semantic Brain Health and Self-Maintenance suite (cross-preference contradiction detection, concept-gap and stale-claim surfacing, per-preference edit-history, a clean/watch/investigate reconciliation verdict via `brain_health`, and dependency-ordered `doctor --remediate`), the v0.17.0 Brain Lifecycle Review Suite (`intent-review`, `retention`, `monthly`, lifecycle schemas, and discipline complexity ratio), and the v0.18.0 MCP context economy (a per-tool preview budget that returns a bounded inline preview and parks the full payload in a vault-local artifact the agent fetches on demand with `brain_artifact_get`, plus a `recall_hint` one-liner on search results), and the v0.19.0 typed graph semantics (typed frontmatter relationships - `related` / `extends` / `contradicts` / `superseded_by` - that surface inline on search results, a `visibility:` scoping axis for `brain_search`, and a `brain_mcp_landscape` view of the Model Context Protocol servers configured across the vault). Browse [`docs/cli-reference.md`](docs/cli-reference.md) for every verb and [`docs/how-it-works.md`](docs/how-it-works.md) for the mental model.+These are the headline capabilities. The full surface also includes: importing Claude Code memory directories, daily logging-discipline cron with a complexity-to-thinking warning, cross-project pointers for shared vaults, transient pinned session context through `brain_pinned_context` / `Brain/pinned.md`, source-agent query and comparison (`brain_agent_query`, `brain_agent_diff`, `o2b brain agent-query`, `o2b brain agent-diff`), the codegraph partner skill, vault hygiene lints, per-MOC coverage audit, concept synthesis, an operator dashboard, configurable Markdown-link presentation output, MCP output-contract checks for core agent-facing tools, private-region stripping with `<private>...</private>`, the v0.12.0 Brain Integrity Suite (content-hash drift detection, durable workrun checkpoints, destructive-from-confirmed retire gate, and the `brain_review_candidates` MCP tool that previews what the next dream pass would do without writing anything), the v0.13.0 Hybrid Search and Recall Quality suite (explainable recall, MMR diversity, link-graph traversal, entity-boosted retrieval, header-anchored chunking), the v0.14.0 Semantic Brain Health and Self-Maintenance suite (cross-preference contradiction detection, concept-gap and stale-claim surfacing, per-preference edit-history, a clean/watch/investigate reconciliation verdict via `brain_health`, and dependency-ordered `doctor --remediate`), the v0.17.0 Brain Lifecycle Review Suite (`intent-review`, `retention`, `monthly`, lifecycle schemas, and discipline complexity ratio), and the v0.18.0 MCP context economy (a per-tool preview budget that returns a bounded inline preview and parks the full payload in a vault-local artifact the agent fetches on demand with `brain_artifact_get`, plus a `recall_hint` one-liner on search results), and the v0.19.0 typed graph semantics (typed frontmatter relationships - `related` / `extends` / `contradicts` / `superseded_by` - that surface inline on search results, a `visibility:` scoping axis for `brain_search`, and a `brain_mcp_landscape` view of the Model Context Protocol servers configured across the vault), and the v0.20.0 Recall and Ranking Quality suite (a configurable Weibull recency-decay curve, structural query-intent classification that re-weights ranking by query shape, language-agnostic synonym expansion via local co-occurrence, per-memory and total recall character budgets for `brain_context_pack`, a persistent query cache that self-invalidates on a corpus-generation change, and a `brain_pre_compress_pack` tool that returns a budgeted top-preferences addendum for a host runtime to inject before context compression). Browse [`docs/cli-reference.md`](docs/cli-reference.md) for every verb and [`docs/how-it-works.md`](docs/how-it-works.md) for the mental model.## CLIdiff --git a/docs/brainstorm/recall-ranking-suite/cli-output/claude.md b/docs/brainstorm/recall-ranking-suite/cli-output/claude.mdnew file mode 100644index 00000000..7ff51cdf--- /dev/null+++ b/docs/brainstorm/recall-ranking-suite/cli-output/claude.md@@ -0,0 +1,35 @@+### Variant 1: Localized pure-function extensions + cache wrapper+- **Approach**: Each feature lands as a self-contained change in the module that already owns its concern โ Weibull replaces `recencyBoost` in `ranker.ts`, intent classification is a small pure function whose output is a set of weight multipliers consumed inside `rankResults`, synonym expansion augments the FTS candidate set (extra terms โ extra candidates merged before hydrate), and the cache is a thin memoizing wrapper around `search()`. Budgeting (per-memory cap, total cap) is extracted once into a shared `budget` helper reused by `context-pack.ts` and the new `brain_pre_compress_pack`. Corpus generation is read from `index_state` and folded into the cache key.+- **Trade-offs**:+ - Pro: smallest diff, each feature maps to one module โ clean per-commit TDD and trivial no-op gating.+ - Pro: lowest blast radius on the determinism contract; ranker stays pure, only its inputs change.+ - Pro: cache + generation gate is one wrapper, easy to disable wholesale.+ - Con: intent classification and synonym expansion both re-derive the same query signals (entities, quoted phrases, wildcards) independently โ duplicated structural parsing.+ - Con: weight modulation is spread between the classifier and `rankResults`, so "how intent changes ranking" is not in one readable place.+- **Complexity**: small+- **Risk**: low++### Variant 2: Dedicated query-analysis layer (`QueryPlan`)+- **Approach**: Introduce one upstream pure stage that runs before candidate retrieval and produces a `QueryPlan` { intent, expandedTerms, weightProfile } from a single structural pass over the query (reusing `extractEntities`, quote/wildcard/wikilink/digit detection). Features 2 and 6 both read from this shared analysis; the `weightProfile` is threaded into `rankResults` (which keeps Weibull recency as a pure config-driven curve), and `expandedTerms` feed candidate augmentation. The cache sits in front of `search()` keyed by `(normalizedQuery, scope, QueryPlan-hash, corpusGeneration)`; budgeting is a shared primitive consumed by `context-pack` and `pre_compress_pack`.+- **Trade-offs**:+ - Pro: the two language-agnostic features share one signal-extraction seam, so the structural-only constraint is enforced in exactly one auditable place.+ - Pro: "how intent reshapes ranking" lives in one `weightProfile`, and the plan is a clean injectable test fixture (deterministic, no I/O).+ - Pro: cache key naturally incorporates the plan + generation, making invalidation coherent rather than bolted-on.+ - Con: introduces a new orchestration object threaded through `search()` โ more upfront design than Variant 1.+ - Con: per-feature commits must land the `QueryPlan` skeleton first, so the first commit is slightly larger / shared across later features.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Composable pipeline-stage middleware+- **Approach**: Refactor `search.ts` into an ordered list of stages implementing a common `Stage` interface (retrieve โ expand โ rank โ traverse โ diversify โ filter โ scope โ slice), with intent, expansion, cache, and budgeting each registered as pluggable, individually toggleable stages. Generation tracking becomes a cache-stage dependency; budgeting becomes a terminal stage shared with the pack tools via the same stage contract.+- **Trade-offs**:+ - Pro: maximally extensible; future recall layers slot in without touching the orchestrator.+ - Pro: uniform enable/disable and ordering semantics across all stages.+ - Con: large refactor of an already-shipped, determinism-critical pipeline โ high regression risk against the byte-identical Syncthing contract.+ - Con: over-abstracts for 8 features; the stage interface must accommodate very different shapes (pure rank vs. I/O cache vs. budget trim), leaking abstractions.+ - Con: hardest to land incrementally under per-feature TDD; the framework must precede any feature.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 2+**Rationale**: The hard language-agnostic constraint couples features 2 and 6 to the same structural signals (`extractEntities`, quoted phrases, wildcards, wikilinks, digits), so computing them once in a `QueryPlan` both removes duplication and gives a single place to audit that no language tables crept in. It keeps the ranker pure and Weibull config-driven while centralizing weight modulation, and the plan-hash + corpus-generation cache key makes feature 5/7 invalidation coherent โ all without the regression risk of rewriting the shipped, determinism-bound pipeline that Variant 3 demands.diff --git a/docs/brainstorm/recall-ranking-suite/cli-output/prompt.md b/docs/brainstorm/recall-ranking-suite/cli-output/prompt.mdnew file mode 100644index 00000000..25051f48--- /dev/null+++ b/docs/brainstorm/recall-ranking-suite/cli-output/prompt.md@@ -0,0 +1,83 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++Ship one coherent "Recall & Ranking Quality" pull request for Open Second Brain (an agent-owned second brain over an Obsidian-compatible Markdown vault; TypeScript + Bun; exposes an `o2b` CLI and MCP servers). The PR bundles 8 features drawn from 5 upstream-inspired kanban tasks, all touching the retrieval / recall / context-injection layer. Implementation will be TDD, one feature at a time, on a single feature branch.++The 8 features:++1. **Weibull recency decay** (replace the current stepwise `recencyBoost`). Today `src/core/search/ranker.ts:recencyBoost` is a hardcoded step function (<=7d -> 0.05, <=30d -> 0.025, <=90d -> 0.01, else 0). Generalise to a configurable Weibull decay curve (shape + scale parameters) so different vaults can tune how fast old content loses relevance. Must default to a curve that is close to today's behaviour and remain bounded and deterministic.++2. **Query-intent classification**. Classify the incoming query into an intent (e.g. fact-seeking, decision-seeking, pattern-seeking, instruction-seeking, exact-phrase) and adjust ranking strategy accordingly (e.g. fact queries lean keyword/entity, pattern queries lean recency/semantic). HARD CONSTRAINT: the classifier MUST be language-agnostic and structural/statistical only - NO per-language keyword lists, NO hardcoded natural-language phrases. Signals available without language tables: presence of quoted exact phrases, share of entity-like tokens (the codebase already has structural, language-agnostic `extractEntities`), query length, presence of wildcard/boolean FTS operators, wikilink-shaped tokens, digits/dates.++3. **Per-memory character cap** in `brain_context_pack` (`src/core/brain/context-pack.ts`): trim individual oversized entries by score order before they consume the bundle budget. Zero/unset = no trimming (today's behaviour).++4. **Total recall character cap**: a second budget dimension alongside the existing `max_tokens`, capping total characters across the returned bundle, discarding lowest-scored overflow.++5. **Query result cache**: cache search results to avoid recomputing FTS5 + semantic phases for repeated/identical queries within a scope. Decide session-scoped (ephemeral) vs vault-scoped (persistent with TTL).++6. **Synonym / query expansion**. Broaden recall when the query wording differs from stored wording. HARD CONSTRAINT: language-agnostic. NO WordNet, NO bundled per-language synonym tables. Acceptable approaches: vault-derived co-occurrence/statistical expansion, morphological/affix folding that is not language-specific, or embedding-neighbour expansion reusing the existing optional embedding provider. Must be a no-op (bit-identical results) when the signal is absent.++7. **Corpus-generation tracking + query-cache invalidation gate**. Track a "corpus generation" fingerprint (composed of embedding model + embedding dimension + relevant index/schema version) so the query cache (feature 5) can be invalidated when the underlying embedding state changes, never serving stale results. Smaller-first-cut acceptable: generation/dimension-level invalidation rather than per-page hashing.++8. **`brain_pre_compress_pack` MCP tool**: a read-only tool that returns a tight, budgeted bundle of the top-K highest-confidence preferences plus the head of `active.md`, formatted as a system-prompt addendum, so an external runtime can inject it just before a context-compression event. OSB ships only the tool; any host-runtime (Hermes) wiring is an out-of-scope integration recipe. Reuses the budgeting/ranking primitives from features 3/4.++# Project context++- Language/runtime: TypeScript, Bun. Search layer is pure-function-heavy with I/O confined to a `Store` (SQLite, FTS5, optional sqlite-vec).+- Current search pipeline (`src/core/search/search.ts`): fts candidates -> semantic candidates -> hydrate -> `rankResults` (keyword + semantic + link + recency + tier + entity) -> link-graph traversal -> MMR diversify -> property filter -> visibility scope -> slice. NOTE: MMR diversity rerank (`src/core/search/mmr.ts`), link traversal (`traversal.ts`), entity boost (`entities.ts`), explainable-recall `reasons`, and tier weighting ALREADY SHIPPED in v0.13.0 - do NOT propose re-implementing them.+- Config: `resolveSearchConfig` in `src/core/search/index.ts` resolves a frozen `ResolvedSearchConfig` from env-or-config keys (`OPEN_SECOND_BRAIN_SEARCH_*`) with documented `DEFAULTS` and a `validateResolvedConfig` gate. Recall tunables live in a nested `ResolvedRecallConfig` (`recall.mmrLambda`, `recall.maxHops`, `recall.hopDecay`, `recall.maxExpansionPerHit`). Per-query overrides ride on `SearchOptions`.+- Persistence: `src/core/search/schema.ts` has `LATEST_SCHEMA_VERSION = 3` with an additive, reindex-safe migration ladder. A key-value `index_state` table already persists `embedding_model` and `embedding_dimension` (see `store.ts:683`). `chunk_entities` table exists (v2).+- Ranking is pure and deterministic (a hard "Syncthing determinism" contract: same vault -> byte-identical results across peers). `extractEntities` is explicitly structural with "no language word lists".+- `brain_context_pack` (`src/mcp/brain-tools.ts:1588`, core `src/core/brain/context-pack.ts`) returns the highest-tier, newest vault slice under `max_tokens`, tier-ordered core->supporting->peripheral.++Recent commits (git log --oneline):+- cbbe18f feat: typed graph semantics - typed relations, visibility scoping, MCP landscape (#46)+- 5fa7eb0 feat: MCP context economy - preview budget, artifact fetch, recall hint (#45)+- 2bd3f48 v0.17.0 - Brain Lifecycle Review Suite+- 2147640 v0.13.0 - Hybrid Search and Recall Quality: explainable recall, MMR, link traversal, entity boost, header anchoring++Related files:+- src/core/search/search.ts (orchestrator)+- src/core/search/ranker.ts (pure ranker; recencyBoost step function lives here)+- src/core/search/mmr.ts, traversal.ts, entities.ts (shipped recall layers)+- src/core/search/types.ts (BrainSearchResult, SearchOptions, ResolvedRecallConfig, ResolvedSearchConfig)+- src/core/search/index.ts (resolveSearchConfig, DEFAULTS, validation)+- src/core/search/store.ts (Store: FTS, semantic, hydrate, index_state KV at :683)+- src/core/search/schema.ts (migration ladder, LATEST_SCHEMA_VERSION=3)+- src/core/search/fts.ts (runFtsQuery)+- src/core/brain/context-pack.ts (budgeted context pack)+- src/mcp/brain-tools.ts (brain_search, brain_context_pack, MCP tool registration)+- src/mcp/instructions.ts (agent-facing tool docs)++Conventions:+- Pure deterministic modules; I/O confined to Store. Same vault -> identical output across Syncthing peers.+- Per-feature graceful degradation: every new layer must be a documented no-op when its input is absent/disabled, so untagged/pre-reindex vaults stay byte-identical.+- Config via env-or-config with DEFAULTS + validation; per-query opt-outs on SearchOptions.+- Additive, reindex-safe schema migrations; bump LATEST_SCHEMA_VERSION only when a column/table is genuinely needed.+- TDD: each atomic unit is a separate conventional commit with failing tests first.+- One PR = one CHANGELOG version.++Constraints:+- LANGUAGE-AGNOSTIC IS NON-NEGOTIABLE: no per-language keyword/synonym/stopword lists, no hardcoded natural-language phrases anywhere. Intent classification and synonym expansion must use structural/statistical/vault-derived/embedding signals only. This is both a project invariant and an explicit operator rule.+- Preserve determinism (no Date.now() baked into pure modules without an injectable nowMs; no Math.random()).+- Do not change the meaning of existing BrainSearchResult fields; additive only.+- Avoid new heavy external dependencies; prefer reusing the existing optional embedding provider and SQLite.+- Keep each feature individually disable-able and bounded.++# Required output format++Produce exactly 3 distinct architectural variants for how to STRUCTURE and INTEGRATE these 8 features into the existing pipeline (e.g. where intent classification plugs in and how it adjusts weights; where the cache + invalidation layer sits relative to search(); whether budgeting is a shared primitive reused by context_pack and pre_compress_pack; whether synonym expansion is query-rewrite-before-FTS vs candidate-augmentation). For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/recall-ranking-suite/design.md b/docs/brainstorm/recall-ranking-suite/design.mdnew file mode 100644index 00000000..8d6e8879--- /dev/null+++ b/docs/brainstorm/recall-ranking-suite/design.md@@ -0,0 +1,186 @@+# Recall & Ranking Suite - sharpen the retrieval, recall-budget, and pre-injection layers++**Status:** draft+**Author:** claude-vps-agent (via feature-release-playbook)+**Audience:** implementation++## Problem statement++Open Second Brain's retrieval layer fuses FTS5, semantic vectors, MMR diversity,+link traversal, entity boost, and tier weighting (v0.13.0), but recency is a+hardcoded step function, every query is treated identically regardless of what it+is asking for, identical queries are recomputed from scratch, and a query whose+wording differs from the stored wording under-recalls. Separately, the+context-injection surface (`brain_context_pack`) budgets only by token count - a+single oversized preference can still crowd out the rest - and there is no+read-only bundle an external runtime can inject just before a compression event.+Each gap is something a memory product is expected to address; together they are+the recall-quality and context-economy core of the product.++## Scope++Eight features across seven atomic units, one pull request, separate commits, in+dependency order. All are individually disable-able and bounded.++- **F1 - Weibull recency decay.** Replace the stepwise `recencyBoost` in+ `ranker.ts` with a configurable Weibull curve (shape + scale). Default+ parameters approximate today's curve; bounded and deterministic (injectable+ `nowMs`).+- **F2 - Query-intent classification.** A pure `QueryPlan` stage classifies the+ query into a structural intent and emits a `weightProfile` that nudges the+ keyword/semantic/entity/recency mix. Structural signals only - no language+ tables.+- **F3 - Synonym / query expansion.** `QueryPlan.expandedTerms` augment the FTS+ candidate set, language-agnostically (vault-derived term co-occurrence; optional+ embedding-neighbour expansion when the embedding provider is configured).+ No-op (bit-identical) when no expansion signal exists.+- **F4 - Per-memory character cap.** A shared `recall-budget` primitive trims+ individual oversized entries by score order before they consume the bundle.+ Consumed by `context-pack.ts`.+- **F5 - Total recall character cap.** A second budget axis alongside `max_tokens`,+ discarding lowest-scored overflow. Same shared primitive.+- **F6 - Persistent query cache + corpus-generation invalidation.** A SQLite-backed+ `query_cache` table (additive migration v4) keyed by+ `(normalizedQuery, scope, planHash, corpusGeneration)` with a TTL. The+ `corpusGeneration` fingerprint (embedding model + dimension + schema version ++ monotonic index revision) is derived from `index_state`; rows whose generation+ differs from current are never served and are swept. A cache hit is byte-identical+ to a fresh compute. The index-revision component is what makes a content reindex+ (which changes the index DB `search()` reads from, but not the embedding+ model/dimension/schema) invalidate the cache.+- **F7 - `brain_pre_compress_pack` MCP tool.** Read-only tool returning a budgeted+ bundle of top-K highest-confidence preferences plus the head of `active.md`,+ formatted as a system-prompt addendum. Reuses the F4/F5 budget primitive. OSB+ ships only the tool; host-runtime wiring is an out-of-scope recipe.++(F6 bundles the cache and its corpus-generation invalidation gate into one atomic+unit because the gate is meaningless without the cache it guards - the "smaller+first cut" the source task calls for.)++## Out of scope++- In-memory-only caching (rejected: the CLI runs a fresh process per call).+- Per-page corpus-generation hashing (the gbrain full model); generation is+ tracked at embedding-model/dimension/schema granularity - the documented+ smaller first cut.+- LLM-based intent classification or LLM query rewriting (non-deterministic,+ language-coupled).+- Any per-language keyword / synonym / stopword list (forbidden by the+ language-agnostic invariant).+- Host-runtime (Hermes) pre-compress wiring - shipped separately as a recipe.+- Changing the meaning of existing `BrainSearchResult` fields.++## Chosen approach++Variant 2 (dedicated query-analysis layer), per `variants.md`. A new pure module+`query-plan.ts` runs one structural pass over the query and returns+`QueryPlan { intent, weightProfile, expandedTerms, planHash }`. `search.ts`+consumes it: `expandedTerms` augment FTS candidates; `weightProfile` is threaded+into `rankResults` as per-query weight multipliers. Weibull recency stays a pure+config-driven curve inside `ranker.ts` (not routed through the plan). A persistent,+generation-gated `query_cache` wraps `search()`. A shared `recall-budget` primitive+backs both `context-pack.ts` and the new `brain_pre_compress_pack` tool.++Pipeline after the change (`search.ts`):++```text+buildQueryPlan(query) [F2 intent, F3 terms, planHash]+ -> query-cache lookup (normQuery, scope, planHash, generation) -- hit? return [F6]+ -> fts candidates (+ expandedTerms augmentation) [F3]+ -> semantic candidates -> hydrate+ -> rankResults(... weightProfile, Weibull recency ...) [F1, F2]+ -> traversal -> MMR -> property filter -> visibility -> slice (unchanged v0.13.0)+ -> query-cache store (with current generation, TTL) [F6]+```++Graceful degradation, per feature:+- F1: defaults approximate the prior curve; a config flag can restore exact prior steps if needed. Pure, deterministic.+- F2: `weightProfile` defaults to neutral (all multipliers 1.0) for an unclassifiable query -> bit-identical ranking.+- F3: empty `expandedTerms` (no co-occurrence signal, no embeddings) -> identical candidate set.+- F4/F5: zero/unset caps -> no trimming (today's behaviour).+- F6: cache disabled by config, or empty on first query, or generation mismatch -> always falls through to a fresh compute that equals the uncached path.+- F7: additive new tool; no existing surface changes.++## Design decisions++- **`QueryPlan` is the single audit seam for the language-agnostic invariant.**+ Both intent classification and synonym expansion derive exclusively from+ structural signals computed once here: `extractEntities` (already structural),+ quoted-phrase spans, FTS wildcard/boolean operators, wikilink-shaped tokens,+ digit/date tokens, token count. No natural-language word list appears anywhere.+- **Intent is structural, not semantic.** Intents are derived from query shape+ (e.g. high entity share -> entity-leaning profile; a quoted exact phrase ->+ keyword-leaning, expansion suppressed; long natural-language run with no+ entities -> semantic/recency-leaning). The mapping is a small fixed table from+ structural features to weight multipliers - never from words to meaning.+- **Weibull stays pure and config-driven, outside the plan.** `recencyBoost(age)`+ becomes `weibullDecay(ageDays, shape, scale, amplitude)`; recency does not depend+ on the query, so threading it through `QueryPlan` would be artificial coupling.+- **Synonym expansion is vault-derived, never a bundled dictionary.** Primary+ source: co-occurrence statistics already implicit in the index (terms frequently+ sharing chunks). Optional: nearest-neighbour terms via the existing embedding+ provider when configured. Both are language-agnostic by construction and degrade+ to no-op without data.+- **The query cache is persistent and generation-gated.** New `query_cache` table+ (migration v4, additive, reindex-safe), columns include the serialized result,+ `corpus_generation`, and `created_at` for TTL. Lookups filter on the current+ generation, so an embedding-model/dimension/schema change silently invalidates+ the cache. A cache hit is asserted byte-identical to a fresh compute.+ `corpusGeneration` is a pure function of `(embedding_model, embedding_dimension,+ LATEST_SCHEMA_VERSION, index_revision)` read from `index_state`. `index_revision`+ is a monotonic counter the indexer bumps whenever it actually mutates the index+ (added + updated + deleted > 0), so a content reindex invalidates the cache even+ though the embedding model/dimension/schema are unchanged - `search()` reads only+ from the index DB, so the cache stays exactly as fresh as that DB.+- **One shared budget primitive.** `recall-budget.ts` exposes a pure+ `applyBudget(entries, { maxTokens?, maxCharsPerEntry?, maxTotalChars? })` that+ trims per-entry then enforces the total, discarding lowest-scored overflow.+ `context-pack.ts` and `brain_pre_compress_pack` both consume it - no duplicated+ trimming logic.+- **`brain_pre_compress_pack` reuses ranking + budget, adds no new storage.** It+ selects top-K confirmed preferences by confidence/recency, prepends the head of+ `active.md`, and renders a compact system-prompt addendum under the same budget+ primitive. Read-only; ToolScope consistent with other read tools.++## File changes++New:+- `src/core/search/query-plan.ts` - pure `buildQueryPlan(query)` -> `{ intent, weightProfile, expandedTerms, planHash }`.+- `src/core/search/recency.ts` - pure `weibullDecay(ageDays, opts)` (or folded into `ranker.ts` if cleaner).+- `src/core/search/synonyms.ts` - pure expansion given a co-occurrence/embedding-neighbour source.+- `src/core/search/query-cache.ts` - persistent cache get/put + generation gate + TTL sweep (I/O via Store).+- `src/core/search/corpus-generation.ts` - pure generation fingerprint from index_state values.+- `src/core/brain/recall-budget.ts` - shared pure budget primitive.+- `src/core/brain/pre-compress-pack.ts` - core builder for the pre-compress bundle.+- Test files under `tests/core/search/` and `tests/core/brain/` for each new module, plus integration tests in the search/context-pack suites.++Modified:+- `src/core/search/ranker.ts` - Weibull recency; accept per-query `weightProfile`.+- `src/core/search/types.ts` - `QueryPlan`, `weightProfile`, recall-config fields (weibull shape/scale/amplitude, intent toggle, expansion toggle, cache TTL/enable); `SearchOptions` opt-outs.+- `src/core/search/index.ts` - resolve + validate the new config fields and `DEFAULTS`.+- `src/core/search/search.ts` - build plan, augment candidates, thread weightProfile, wrap with cache.+- `src/core/search/store.ts` - `query_cache` read/write/sweep; `corpusGeneration` accessor; co-occurrence accessor for synonyms.+- `src/core/search/schema.ts` - migration v4 (`query_cache` table); bump `LATEST_SCHEMA_VERSION` to 4.+- `src/core/brain/context-pack.ts` - consume `recall-budget` (per-memory + total char caps).+- `src/mcp/brain-tools.ts` - `brain_context_pack` new cap args; register `brain_pre_compress_pack`.+- `src/mcp/instructions.ts` - document the new tool + cap args.+- `src/cli/brain/verbs/*` - surface new flags where a CLI verb maps (context-pack / search), if applicable.+- `README.md`, `CHANGELOG.md`, `docs/mcp.md`, `docs/cli-reference.md`, `docs/how-it-works.md`.++## Risks and open questions++- **Migration v4 correctness.** Adding `query_cache` is additive, but the migration+ must be reindex-safe and a newer-than-supported DB must still raise+ `SCHEMA_MISMATCH`. Mitigation: a migration test that opens a v3 fixture, migrates,+ and asserts the table exists and old data is intact.+- **Cache transparency.** A hit must equal a fresh compute. Mitigation: a test that+ runs a query twice and asserts deep-equality of results, and a test that mutates+ generation and asserts the stale row is not served.+- **Intent over-steering.** A misclassified intent must never inject irrelevant+ docs - only re-weight an already-relevant set. Mitigation: bound weight+ multipliers to a narrow band and keep the neutral profile bit-identical.+- **Synonym precision.** Vault-derived expansion can add noise. Mitigation: cap the+ number of expansion terms, require a minimum co-occurrence threshold, and keep+ expansion off for quoted exact-phrase queries.+- **Determinism.** All new ranking/plan/budget math is pure with injectable time;+ no `Date.now()`/`Math.random()` inside pure modules.diff --git a/docs/brainstorm/recall-ranking-suite/plan.md b/docs/brainstorm/recall-ranking-suite/plan.mdnew file mode 100644index 00000000..9cd71b93--- /dev/null+++ b/docs/brainstorm/recall-ranking-suite/plan.md@@ -0,0 +1,51 @@+# Recall & Ranking Suite - implementation plan++Seven atomic units (eight features), one feature branch (`feat/recall-ranking-suite`),+separate conventional commits, TDD throughout. Order respects dependencies: pure+isolated change first, then the shared `QueryPlan` seam, then features that build on it,+then the budget primitive and its consumers.++## Tasks++### Task 1: Weibull recency decay (F1)+- **Files**: `src/core/search/recency.ts` (new, pure `weibullDecay`), `tests/core/search/recency.test.ts` (new), `src/core/search/ranker.ts` (swap `recencyBoost`), `src/core/search/types.ts` + `index.ts` (config: `recencyShape`, `recencyScale`, `recencyAmplitude` + DEFAULTS + validation), affected ranker test fixtures.+- **Acceptance**: `weibullDecay` unit tests cover the curve shape and bounds; default params keep ranker output within tolerance of the prior step function on fixtures; injectable `nowMs` keeps it deterministic. `bun test tests/core/search/recency.test.ts` green.+- **Depends on**: none.++### Task 2: QueryPlan + query-intent classification (F2)+- **Files**: `src/core/search/query-plan.ts` (new: `buildQueryPlan` -> `{ intent, weightProfile, expandedTerms: [], planHash }`), `tests/core/search/query-plan.test.ts` (new), `src/core/search/ranker.ts` (accept optional `weightProfile` multipliers; neutral default = bit-identical), `src/core/search/search.ts` (build plan, thread weightProfile), `types.ts`/`index.ts` (intent enable flag).+- **Acceptance**: structural inputs (quoted phrase, entity share, wildcard, wikilink, digits, length) map to documented intents/profiles with zero language tables; neutral profile asserts bit-identical ranking vs pre-change fixtures; plan is pure (no I/O). Tests green.+- **Depends on**: Task 1 (ranker already touched; keep changes additive).++### Task 3: Synonym / query expansion (F3)+- **Files**: `src/core/search/synonyms.ts` (new, pure expansion given a term-source), `tests/core/search/synonyms.test.ts` (new), `src/core/search/store.ts` (co-occurrence accessor; optional embedding-neighbour terms), `src/core/search/query-plan.ts` (populate `expandedTerms`), `src/core/search/search.ts` (augment FTS candidates with expanded terms).+- **Acceptance**: expansion is language-agnostic (vault co-occurrence / embedding neighbours only); empty source -> identical candidate set (no-op test); quoted exact-phrase query suppresses expansion; expansion-term count capped. Tests green.+- **Depends on**: Task 2 (QueryPlan).++### Task 4: Shared recall-budget primitive + per-memory char cap (F4)+- **Files**: `src/core/brain/recall-budget.ts` (new, pure `applyBudget`), `tests/core/brain/recall-budget.test.ts` (new), `src/core/brain/context-pack.ts` (consume primitive; `maxCharsPerEntry`), `src/mcp/brain-tools.ts` (`brain_context_pack` arg `max_chars_per_memory`).+- **Acceptance**: per-entry trimming is score-ordered; zero/unset cap = today's output (no-op test); oversized entry trimmed without dropping lower-scored entries below budget. Tests green.+- **Depends on**: none (can run after Task 1; independent of the QueryPlan chain).++### Task 5: Total recall char cap (F5)+- **Files**: `src/core/brain/recall-budget.ts` (extend with `maxTotalChars`), `tests/core/brain/recall-budget.test.ts` (extend), `src/core/brain/context-pack.ts` (wire total cap), `src/mcp/brain-tools.ts` (`brain_context_pack` arg `max_total_chars`).+- **Acceptance**: total cap discards lowest-scored overflow after per-entry trim; interaction of both caps + existing `max_tokens` covered; unset = no-op. Tests green.+- **Depends on**: Task 4.++### Task 6: Persistent query cache + corpus-generation invalidation (F6)+- **Files**: `src/core/search/corpus-generation.ts` (new, pure fingerprint), `tests/core/search/corpus-generation.test.ts` (new), `src/core/search/query-cache.ts` (new: get/put/sweep via Store), `tests/core/search/query-cache.test.ts` (new), `src/core/search/schema.ts` (migration v4 `query_cache`; bump `LATEST_SCHEMA_VERSION` to 4), `tests/core/search/schema-migration.test.ts` (v3->v4), `src/core/search/store.ts` (cache table I/O + generation accessor), `src/core/search/search.ts` (wrap with cache), `types.ts`/`index.ts` (cache enable + TTL config).+- **Acceptance**: cache hit deep-equals fresh compute; embedding model/dimension change AND a content reindex (bumped `index_revision`) both produce a generation mismatch that is never served and is swept; TTL expiry covered; migration test opens a v3 fixture, migrates, asserts `query_cache` present and prior data intact; newer-than-latest DB still raises `SCHEMA_MISMATCH`; cache disabled = uncached path identical. Tests green.+- **Note**: `index_revision` is a monotonic counter bumped by the indexer on real mutations (added+updated+deleted > 0); folded into the corpus-generation fingerprint so content reindex invalidates the cache.+- **Depends on**: Task 2 (planHash is part of the cache key).++### Task 7: brain_pre_compress_pack MCP tool (F7)+- **Files**: `src/core/brain/pre-compress-pack.ts` (new builder), `tests/core/brain/pre-compress-pack.test.ts` (new), `src/mcp/brain-tools.ts` (register `brain_pre_compress_pack`), `src/mcp/instructions.ts` (document tool), `tests/mcp/*` (tool registration + budget round-trip).+- **Acceptance**: returns top-K confirmed preferences by confidence/recency + head of `active.md`, rendered as a system-prompt addendum under the shared budget; read-only; honours ToolScope; empty brain -> empty-but-valid bundle. Tests green.+- **Depends on**: Task 4, Task 5 (budget primitive).++## End state++All seven units' tests pass; `bun run validate` (typecheck + lint + test) green;+`bun run sync-version:check` consistent after the Phase 9 version bump. Diff is+feature-complete per `design.md`. Each feature individually disable-able and a+documented no-op when its input is absent.diff --git a/docs/brainstorm/recall-ranking-suite/variants.md b/docs/brainstorm/recall-ranking-suite/variants.mdnew file mode 100644index 00000000..4615a317--- /dev/null+++ b/docs/brainstorm/recall-ranking-suite/variants.md@@ -0,0 +1,64 @@+# Recall & Ranking Suite - brainstorm variants (audit trail)++Primary consultant: Claude Code CLI (`claude -p`), exit 0, 3 parseable variants.+Fallback (Codex) not run - primary succeeded.+Prompt: `cli-output/prompt.md`. Raw output: `cli-output/claude.md`.++## Variants produced by the consultant (verbatim)++### Variant 1: Localized pure-function extensions + cache wrapper+- **Approach**: Each feature lands as a self-contained change in the module that already owns its concern - Weibull replaces `recencyBoost` in `ranker.ts`, intent classification is a small pure function whose output is a set of weight multipliers consumed inside `rankResults`, synonym expansion augments the FTS candidate set (extra terms -> extra candidates merged before hydrate), and the cache is a thin memoizing wrapper around `search()`. Budgeting (per-memory cap, total cap) is extracted once into a shared `budget` helper reused by `context-pack.ts` and the new `brain_pre_compress_pack`. Corpus generation is read from `index_state` and folded into the cache key.+- **Trade-offs**:+ - Pro: smallest diff, each feature maps to one module -> clean per-commit TDD and trivial no-op gating.+ - Pro: lowest blast radius on the determinism contract; ranker stays pure, only its inputs change.+ - Pro: cache + generation gate is one wrapper, easy to disable wholesale.+ - Con: intent classification and synonym expansion both re-derive the same query signals (entities, quoted phrases, wildcards) independently -> duplicated structural parsing.+ - Con: weight modulation is spread between the classifier and `rankResults`, so "how intent changes ranking" is not in one readable place.+- **Complexity**: small+- **Risk**: low++### Variant 2: Dedicated query-analysis layer (`QueryPlan`)+- **Approach**: Introduce one upstream pure stage that runs before candidate retrieval and produces a `QueryPlan` { intent, expandedTerms, weightProfile } from a single structural pass over the query (reusing `extractEntities`, quote/wildcard/wikilink/digit detection). Features 2 and 6 both read from this shared analysis; the `weightProfile` is threaded into `rankResults` (which keeps Weibull recency as a pure config-driven curve), and `expandedTerms` feed candidate augmentation. The cache sits in front of `search()` keyed by `(normalizedQuery, scope, QueryPlan-hash, corpusGeneration)`; budgeting is a shared primitive consumed by `context-pack` and `pre_compress_pack`.+- **Trade-offs**:+ - Pro: the two language-agnostic features share one signal-extraction seam, so the structural-only constraint is enforced in exactly one auditable place.+ - Pro: "how intent reshapes ranking" lives in one `weightProfile`, and the plan is a clean injectable test fixture (deterministic, no I/O).+ - Pro: cache key naturally incorporates the plan + generation, making invalidation coherent rather than bolted-on.+ - Con: introduces a new orchestration object threaded through `search()` - more upfront design than Variant 1.+ - Con: per-feature commits must land the `QueryPlan` skeleton first, so the first commit is slightly larger / shared across later features.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Composable pipeline-stage middleware+- **Approach**: Refactor `search.ts` into an ordered list of stages implementing a common `Stage` interface (retrieve -> expand -> rank -> traverse -> diversify -> filter -> scope -> slice), with intent, expansion, cache, and budgeting each registered as pluggable, individually toggleable stages. Generation tracking becomes a cache-stage dependency; budgeting becomes a terminal stage shared with the pack tools via the same stage contract.+- **Trade-offs**:+ - Pro: maximally extensible; future recall layers slot in without touching the orchestrator.+ - Pro: uniform enable/disable and ordering semantics across all stages.+ - Con: large refactor of an already-shipped, determinism-critical pipeline -> high regression risk against the byte-identical Syncthing contract.+ - Con: over-abstracts for 8 features; the stage interface must accommodate very different shapes (pure rank vs. I/O cache vs. budget trim), leaking abstractions.+ - Con: hardest to land incrementally under per-feature TDD; the framework must precede any feature.+- **Complexity**: large+- **Risk**: high++### Consultant recommendation: Variant 2+**Rationale**: The hard language-agnostic constraint couples features 2 and 6 to the same structural signals, so computing them once in a `QueryPlan` removes duplication and gives a single place to audit that no language tables crept in. It keeps the ranker pure and Weibull config-driven while centralizing weight modulation, and the plan-hash + corpus-generation cache key makes feature 5/7 invalidation coherent - without the regression risk of rewriting the shipped, determinism-bound pipeline that Variant 3 demands.++## Orchestrator decision: Variant 2, adopted (with one pragmatic graft from Variant 1)++I agree with the consultant. Variant 2 is the right backbone: a single pure `QueryPlan`+seam is the only honest way to satisfy the non-negotiable language-agnostic constraint for+both intent classification and synonym expansion while keeping the audit surface to one file.+It preserves the determinism-bound, already-shipped pipeline (rejecting Variant 3's high-risk+rewrite) and avoids the duplicated structural parsing of Variant 1.++One graft from Variant 1, which the consultant's own text already endorses: **Weibull recency+decay stays a localized pure replacement inside `ranker.ts`/config and is NOT routed through+`QueryPlan`** - recency does not depend on query analysis, so threading it through the plan+would be artificial coupling.++One concrete decision the consultant left open: **the query cache is persistent+(SQLite-backed, new `query_cache` table, additive migration v4) with a TTL and a+`corpus_generation` column**, not in-memory. Rationale: the `o2b` CLI runs a fresh process per+invocation, so an in-memory cache would never hit there; a persistent cache also makes feature+7 (corpus-generation invalidation) load-bearing rather than decorative. The cache is transparent+(a hit must be byte-identical to a fresh compute - asserted by test) and lives only in the local+`.db` index, which is not Syncthing-replicated, so it does not touch the determinism contract.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 0944f04c..3afe0569 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -140,6 +140,25 @@ layer config-tunable and bounded:| `search_hop_decay` | `OPEN_SECOND_BRAIN_SEARCH_HOP_DECAY` | `0.5` | Per-hop score multiplier for traversal-surfaced docs || `search_max_expansion_per_hit` | `OPEN_SECOND_BRAIN_SEARCH_MAX_EXPANSION_PER_HIT` | `3` | Cap on outbound links followed per node |+Recall and ranking quality (v0.20.0), each tunable and bounded:++| Config key | Env var | Default | Effect |+| ------------------------------ | ------------------------------------------------ | ------- | ----------------------------------------------------------------- |+| `search_recency_shape` | `OPEN_SECOND_BRAIN_SEARCH_RECENCY_SHAPE` | `0.8` | Weibull recency curve shape (k) |+| `search_recency_scale` | `OPEN_SECOND_BRAIN_SEARCH_RECENCY_SCALE` | `30` | Weibull characteristic lifetime in days |+| `search_recency_amplitude` | `OPEN_SECOND_BRAIN_SEARCH_RECENCY_AMPLITUDE` | `0.05` | Max recency boost at age 0; `0` disables the recency layer |+| `search_intent_enabled` | `OPEN_SECOND_BRAIN_SEARCH_INTENT_ENABLED` | `true` | Re-weight ranking by structural query intent; `false` is neutral |+| `search_synonym_enabled` | `OPEN_SECOND_BRAIN_SEARCH_SYNONYM_ENABLED` | `false` | Opt-in co-occurrence query expansion (language-agnostic) |+| `search_synonym_max_terms` | `OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS` | `3` | Cap on expansion terms OR'd onto the query |+| `search_cache_enabled` | `OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED` | `false` | Opt-in persistent query cache, gated by corpus generation |+| `search_cache_ttl_seconds` | `OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL` | `300` | Cache row time-to-live in seconds |++`brain_context_pack` also accepts `max_chars_per_memory` and+`max_total_chars` (code-point caps), and the read-only+`brain_pre_compress_pack` MCP tool returns a budgeted+top-preferences-plus-`active.md` addendum for a host runtime to inject+before a context-compression event.+Entity-boosted retrieval and header-anchored chunking populate on thenext reindex and need no configuration. Every result carries a`why_retrieved` list naming the scoring layers that ranked it.diff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex ba751bc5..0d43aeae 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -669,9 +669,13 @@ semantic_weightยทcosine + link_boost + recency_boost + entity_boost)`.`1 - L2ยฒ / 2` on unit-normalised vectors; link boost rewardscandidates that other candidates reference via `[[wikilink]]` /markdown link (capped at 0.03) or share a tag with (capped at- 0.02); recency is a step function on `mtime`; entity boost (capped- at 0.04) rewards proper-noun overlap between query and chunk. Every- result carries a `why_retrieved` list of the layers that fired.+ 0.02); recency is a configurable Weibull decay curve on `mtime`+ (`search_recency_shape` / `search_recency_scale` /+ `search_recency_amplitude`, v0.20.0), defaulting close to the prior+ step function and flooring effectively-stale content to zero; entity+ boost (capped at 0.04) rewards proper-noun overlap between query and+ chunk. Every result carries a `why_retrieved` list of the layers that+ fired.- **Recall layers (v0.13.0).** After ranking, link-graph traversalwalks outbound wikilinks from the top hits and merges in relateddocuments scored `parentยทhop_decay^hop` (bounded by `search_max_hops`@@ -681,6 +685,22 @@ semantic_weightยทcosine + link_boost + recency_boost + entity_boost)`.indexes each chunk's heading breadcrumb in a dedicated FTS column(weighted below content) so a mid-document chunk keeps its topicalanchor. Entity and heading layers populate on the next reindex.+- **Query plan + recall economy (v0.20.0).** A pure structural pass+ classifies each query's intent (neutral / exact / entity / broad) from+ its shape - quoted phrases, FTS wildcards, wikilinks, entity-token+ share, token count, no language word lists - and emits a bounded weight+ profile that re-weights the ranking layers (on by default,+ `search_intent_enabled`; neutral is bit-identical). Opt-in synonym+ expansion (`search_synonym_enabled`) broadens recall by OR-ing in terms+ that co-occur across the top candidates' own content - language-agnostic,+ suppressed for exact intent. An opt-in persistent query cache+ (`search_cache_enabled`, `search_cache_ttl_seconds`) serves an identical+ request while the corpus generation (embedding model + dimension ++ schema + a content-reindex revision) is unchanged; cache reads and+ writes are best-effort. `brain_context_pack` additionally accepts+ per-memory and total character caps, and `brain_pre_compress_pack`+ returns a budgeted top-preferences-plus-active.md addendum for+ pre-compression injection.- **Semantic policy.** Implicit semantic (config default) warns andfalls back to keyword-only when sqlite-vec is unavailable, the keyis missing, the provider is down, or no embeddings exist yet.diff --git a/docs/mcp.md b/docs/mcp.mdindex 880cf1c8..16723f75 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -259,7 +259,7 @@ server to your Codex MCP config the same way as Hermes.The plugin's `.mcp.json` ships **two** MCP-server entries:-- `open-second-brain` - the full surface (41 tools, including `brain_health`, `brain_mcp_landscape`, `brain_agent_query`, `brain_agent_diff`, and `brain_pinned_context`); subject to Claude Code's `MCPSearch` tool-search deferral when MCP definitions push the system prompt past 10% of the context window.+- `open-second-brain` - the full surface (42 tools, including `brain_health`, `brain_mcp_landscape`, `brain_agent_query`, `brain_agent_diff`, `brain_pinned_context`, and `brain_pre_compress_pack`); subject to Claude Code's `MCPSearch` tool-search deferral when MCP definitions push the system prompt past 10% of the context window.- `open-second-brain-writer` - a minimal always-loaded surface of five tools: `brain_feedback`, `brain_apply_evidence`, `brain_note`, `brain_pinned_context` (writers) and `brain_context` (read-only pull-bootstrap of `Brain/active.md` plus pinned context, v0.16.0). The agent records taste signals, evidence events, milestone notes, and current-task pinned facts - and fetches the active rule digest at session start in runtimes without a SessionStart hook - without a ToolSearch round-trip on every session boot.Both servers reuse the same backing CLI (`o2b mcp --scope writer` vs the default `--scope full`). Handlers are byte-identical; the writer-mode instructions text explicitly tells the agent to prefer the writer copy over any duplicate the full server still exposes (both call the same code path).diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 51f02225..cb790c42 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "0.19.0",+ "version": "0.20.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex a60249fb..24004a22 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.19.0",+ "version": "0.20.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 92b89fc2..434acd94 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.19.0"+version: "0.20.0"description: "Hermes plugin entrypoint for OpenSecondBrain vault health checks and the optional MCP tool server."author: "OpenSecondBrain contributors"kind: standalonediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 2b79be5b..e6a95493 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.19.0",+ "version": "0.20.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 13d9c57a..b195fae8 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.19.0"+version: "0.20.0"description: "Hermes adapter for OpenSecondBrain vault health checks and optional MCP tool server."author: "OpenSecondBrain contributors"kind: standalonediff --git a/pyproject.toml b/pyproject.tomlindex ac53ac33..8d41e1cc 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "0.19.0"+version = "0.20.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/core/brain/context-pack.ts b/src/core/brain/context-pack.tsindex 8d4df590..b8a6e571 100644--- a/src/core/brain/context-pack.ts+++ b/src/core/brain/context-pack.ts@@ -21,6 +21,7 @@ import { brainDirs } from "./paths.ts";import { PAGE_TIER, readTier, type PageTier } from "./page-meta/tier.ts";import { estimateTokens } from "./text/tokenizer.ts";import { normalizeForDedup } from "./text/normalize.ts";+import { applyCharBudget } from "./recall-budget.ts";const TIER_ORDER: ReadonlyArray<PageTier> = [PAGE_TIER.core,@@ -34,12 +35,14 @@ export interface ContextPackItem {readonly tier: PageTier;readonly tokens: number;readonly body: string;+ /** True when `body` was truncated by `maxCharsPerMemory` (v0.20.0). */+ readonly trimmed: boolean;}export interface ContextPackSkipped {readonly id: string;readonly tokens: number;- readonly reason: "over-budget" | "filter-miss";+ readonly reason: "over-budget" | "filter-miss" | "over-char-budget";}export interface ContextPackReport {@@ -53,6 +56,19 @@ export interface ContextPackOptions {readonly maxTokens: number;/** Optional case-insensitive substring filter on topic + principle. */readonly query?: string;+ /**+ * Per-memory character cap (v0.20.0): trim any single page's body to+ * this many code points before it consumes the token budget, so one+ * oversized page cannot crowd out the rest. <= 0 / undefined disables.+ */+ readonly maxCharsPerMemory?: number;+ /**+ * Total recall character cap (v0.20.0): a second ceiling alongside+ * `maxTokens`, bounding the cumulative code points across the emitted+ * pages. Lowest-priority overflow is dropped with an+ * `over-char-budget` skip reason. <= 0 / undefined disables.+ */+ readonly maxTotalChars?: number;}interface Candidate {@@ -134,10 +150,19 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPacreturn a.id < b.id ? -1 : a.id > b.id ? 1 : 0;});+ // Per-memory character cap (v0.20.0): trim oversized bodies in priority+ // order via the shared budget primitive before the token budget runs,+ // so one huge page cannot starve the rest. A trimmed body is re-tokenised+ // so the token budget charges the emitted text, not the original.+ const budgeted = applyCharBudget(+ candidates.map((c) => ({ item: c, text: c.body })),+ { maxCharsPerEntry: opts.maxCharsPerMemory },+ );+const items: ContextPackItem[] = [];const skipped: ContextPackSkipped[] = [];let used = 0;- for (const c of candidates) {+ for (const { item: c, text: body, trimmed } of budgeted.kept) {if (query !== null) {const haystack = normalizeForDedup(`${c.topic} ${c.principle}`);if (!haystack.includes(query)) {@@ -145,18 +170,48 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPaccontinue;}}- if (used + c.tokens > opts.maxTokens) {- skipped.push({ id: c.id, tokens: c.tokens, reason: "over-budget" });+ const tokens = trimmed ? estimateTokens(body) : c.tokens;+ if (used + tokens > opts.maxTokens) {+ skipped.push({ id: c.id, tokens, reason: "over-budget" });continue;}items.push({id: c.id,path: c.path,tier: c.tier,- tokens: c.tokens,- body: c.body,+ tokens,+ body,+ trimmed,});- used += c.tokens;+ used += tokens;+ }++ // Total recall character cap (v0.20.0): a second ceiling over the+ // token-budgeted set. Applied via the shared primitive on the emitted+ // items only (so query-missed pages never count), dropping the+ // lowest-priority overflow.+ if (opts.maxTotalChars && opts.maxTotalChars > 0) {+ const capped = applyCharBudget(+ items.map((i) => ({ item: i, text: i.body })),+ { maxTotalChars: opts.maxTotalChars },+ );+ if (capped.dropped.length > 0) {+ const keptItems = capped.kept.map((k) => k.item);+ const droppedSet = new Set(capped.dropped);+ let recomputed = 0;+ for (const i of keptItems) recomputed += i.tokens;+ for (const d of items) {+ if (droppedSet.has(d)) {+ skipped.push({ id: d.id, tokens: d.tokens, reason: "over-char-budget" });+ }+ }+ return Object.freeze({+ maxTokens: opts.maxTokens,+ tokensUsed: recomputed,+ items: Object.freeze(keptItems),+ skipped: Object.freeze(skipped),+ });+ }}return Object.freeze({diff --git a/src/core/brain/pre-compress-pack.ts b/src/core/brain/pre-compress-pack.tsnew file mode 100644index 00000000..2d72bc6b--- /dev/null+++ b/src/core/brain/pre-compress-pack.ts@@ -0,0 +1,140 @@+/**+ * Pre-compress injection pack (v0.20.0).+ *+ * A read-only bundle of the highest-confidence confirmed preferences plus+ * the head of `active.md`, rendered as a compact system-prompt addendum.+ * An external runtime (e.g. a host agent's pre-compression hook) can+ * inject it just before a context-compression event so the brain's+ * highest-salience constraints survive context rotation without the agent+ * having to remember to query. OSB ships only this builder and the MCP+ * tool around it; the host-runtime wiring is an out-of-scope recipe.+ *+ * It reuses the shared recall-budget primitive (the same per-entry and+ * total character caps as `brain_context_pack`) so one oversized+ * preference cannot dominate the addendum. Deterministic given the vault:+ * the only ordering inputs are confidence, creation time, and id.+ */++import { existsSync, readFileSync, readdirSync } from "node:fs";+import { join } from "node:path";++import { brainActivePath, brainDirs } from "./paths.ts";+import { parsePreference } from "./preference.ts";+import { applyCharBudget } from "./recall-budget.ts";+import { BRAIN_PREFERENCE_STATUS } from "./types.ts";++/** Sentinel item id for the active-head entry inside the budget pass. */+const ACTIVE_ID = "__active__";++export interface PreCompressItem {+ readonly id: string;+ /** Preference principle text after any per-entry trim. */+ readonly principle: string;+ /** True when `principle` was truncated by `maxCharsPerMemory`. */+ readonly trimmed: boolean;+}++export interface PreCompressPack {+ /** Rendered system-prompt addendum (empty string when nothing fits). */+ readonly text: string;+ readonly items: ReadonlyArray<PreCompressItem>;+ readonly activeHeadIncluded: boolean;+ readonly totalChars: number;+}++export interface PreCompressOptions {+ /** Maximum number of preferences to consider (highest-confidence first). */+ readonly topK: number;+ /** Per-entry character cap (code points); <= 0 / undefined disables. */+ readonly maxCharsPerMemory?: number;+ /** Total character cap across the bundle; <= 0 / undefined disables. */+ readonly maxTotalChars?: number;+}++interface ConfirmedPref {+ readonly id: string;+ readonly principle: string;+ readonly confidence: number;+ readonly createdAt: string;+}++function collectConfirmed(vault: string): ConfirmedPref[] {+ const dir = brainDirs(vault).preferences;+ if (!existsSync(dir)) return [];+ const out: ConfirmedPref[] = [];+ for (const name of readdirSync(dir)) {+ if (!name.endsWith(".md")) continue;+ let pref;+ try {+ pref = parsePreference(join(dir, name));+ } catch {+ continue;+ }+ if (pref.status !== BRAIN_PREFERENCE_STATUS.confirmed) continue;+ out.push({+ id: pref.id,+ principle: pref.principle,+ confidence: pref.confidence_value ?? Number.NEGATIVE_INFINITY,+ createdAt: pref.created_at,+ });+ }+ return out;+}++function readActiveHead(vault: string): string | null {+ const path = brainActivePath(vault);+ if (!existsSync(path)) return null;+ try {+ const text = readFileSync(path, "utf8").trim();+ return text.length > 0 ? text : null;+ } catch {+ return null;+ }+}++/**+ * Build the pre-compress addendum for a vault. Confirmed preferences are+ * ranked by confidence (desc), then recency (desc), then id; the top+ * `topK` plus the active head are bounded by the shared char budget.+ */+export function buildPreCompressPack(vault: string, opts: PreCompressOptions): PreCompressPack {+ const ranked = collectConfirmed(vault).toSorted((a, b) => {+ if (b.confidence !== a.confidence) return b.confidence - a.confidence;+ if (a.createdAt !== b.createdAt) return a.createdAt < b.createdAt ? 1 : -1;+ return a.id < b.id ? -1 : a.id > b.id ? 1 : 0;+ });+ const top = ranked.slice(0, Math.max(0, opts.topK));++ const activeHead = readActiveHead(vault);+ const entries: Array<{ item: string; text: string }> = [];+ if (activeHead !== null) entries.push({ item: ACTIVE_ID, text: activeHead });+ for (const p of top) entries.push({ item: p.id, text: p.principle });++ const budgeted = applyCharBudget(entries, {+ maxCharsPerEntry: opts.maxCharsPerMemory,+ maxTotalChars: opts.maxTotalChars,+ });++ let activeText: string | null = null;+ const items: PreCompressItem[] = [];+ for (const kept of budgeted.kept) {+ if (kept.item === ACTIVE_ID) {+ activeText = kept.text;+ continue;+ }+ items.push({ id: kept.item, principle: kept.text, trimmed: kept.trimmed });+ }++ const sections: string[] = [];+ if (activeText !== null) sections.push(`# Active brain context\n\n${activeText}`);+ if (items.length > 0) {+ sections.push(["Preferences:", ...items.map((i) => `- ${i.principle}`)].join("\n"));+ }++ return Object.freeze({+ text: sections.join("\n\n"),+ items: Object.freeze(items),+ activeHeadIncluded: activeText !== null,+ totalChars: budgeted.totalChars,+ });+}diff --git a/src/core/brain/recall-budget.ts b/src/core/brain/recall-budget.tsnew file mode 100644index 00000000..34faf6f4--- /dev/null+++ b/src/core/brain/recall-budget.ts@@ -0,0 +1,90 @@+/**+ * Shared, pure recall-budget primitive (v0.20.0).+ *+ * A single place to enforce character budgets on an ordered list of+ * recall entries, reused by `context-pack.ts` and+ * `brain_pre_compress_pack` so neither reimplements trimming. Two+ * independent dimensions:+ *+ * - `maxCharsPerEntry` trims any single oversized entry's text (so one+ * huge memory cannot crowd out the rest);+ * - `maxTotalChars` caps the cumulative characters across the kept+ * entries, discarding the lowest-priority overflow.+ *+ * Entries MUST arrive pre-sorted by priority (highest first); the+ * primitive preserves that order and trims/drops from the tail. Lengths+ * are measured in Unicode code points, so multi-byte scripts and astral+ * characters are counted and truncated without splitting a surrogate+ * pair. Pure and deterministic: no I/O, clock, or randomness.+ */++export interface CharBudgetOptions {+ /** Max code points per entry; <= 0 or undefined disables per-entry trimming. */+ readonly maxCharsPerEntry?: number;+ /** Max cumulative code points across kept entries; <= 0 or undefined disables. */+ readonly maxTotalChars?: number;+}++export interface BudgetedEntry<T> {+ readonly item: T;+ /** Entry text after any per-entry trim. */+ readonly text: string;+ /** True when `text` was truncated by `maxCharsPerEntry`. */+ readonly trimmed: boolean;+}++export interface CharBudgetResult<T> {+ readonly kept: ReadonlyArray<BudgetedEntry<T>>;+ /** Items dropped because the total-character cap was reached. */+ readonly dropped: ReadonlyArray<T>;+ /** Total code points across kept entries. */+ readonly totalChars: number;+}++function codePoints(s: string): string[] {+ return [...s];+}++/**+ * Apply per-entry and total character caps to an ordered entry list.+ * With no caps set this is an identity pass (every entry kept, untrimmed,+ * nothing dropped).+ */+export function applyCharBudget<T>(+ entries: ReadonlyArray<{ readonly item: T; readonly text: string }>,+ opts: CharBudgetOptions,+): CharBudgetResult<T> {+ const perEntry = opts.maxCharsPerEntry && opts.maxCharsPerEntry > 0 ? opts.maxCharsPerEntry : 0;+ const total = opts.maxTotalChars && opts.maxTotalChars > 0 ? opts.maxTotalChars : 0;++ const kept: BudgetedEntry<T>[] = [];+ const dropped: T[] = [];+ let used = 0;++ for (const { item, text } of entries) {+ let outText = text;+ let trimmed = false;+ if (perEntry > 0) {+ const cps = codePoints(text);+ if (cps.length > perEntry) {+ outText = cps.slice(0, perEntry).join("");+ trimmed = true;+ }+ }+ const len = codePoints(outText).length;+ if (total > 0 && used + len > total) {+ // Lowest-priority overflow: drop and keep trying smaller tail+ // entries (matches the existing context-pack token-budget policy).+ dropped.push(item);+ continue;+ }+ used += len;+ kept.push({ item, text: outText, trimmed });+ }++ return Object.freeze({+ kept: Object.freeze(kept),+ dropped: Object.freeze(dropped),+ totalChars: used,+ });+}diff --git a/src/core/search/corpus-generation.ts b/src/core/search/corpus-generation.tsnew file mode 100644index 00000000..68652183--- /dev/null+++ b/src/core/search/corpus-generation.ts@@ -0,0 +1,33 @@+/**+ * Corpus-generation fingerprint (v0.20.0).+ *+ * A short, deterministic string identifying the state of the index that+ * `search()` reads from. The query cache stores it alongside each cached+ * result and refuses to serve a row whose generation differs from the+ * current one, so the cache is invalidated the moment any input that can+ * change results changes:+ *+ * - the embedding model or dimension (semantic ranking changes);+ * - the schema version (index layout changes);+ * - the monotonic index revision, bumped by the indexer whenever it+ * mutates the index - so a content reindex invalidates the cache even+ * though the embedding model/dimension/schema are unchanged.+ *+ * This is the documented "smaller first cut": generation-level+ * invalidation, not per-page corpus hashing.+ *+ * Pure and deterministic: no I/O, clock, or randomness.+ */++export interface CorpusGenerationInputs {+ readonly embeddingModel: string | null;+ readonly embeddingDimension: number | null;+ readonly schemaVersion: number;+ readonly indexRevision: number;+}++export function computeCorpusGeneration(inputs: CorpusGenerationInputs): string {+ const model = inputs.embeddingModel ?? "none";+ const dim = inputs.embeddingDimension ?? 0;+ return `${model}|${dim}|${inputs.schemaVersion}|${inputs.indexRevision}`;+}diff --git a/src/core/search/fts.ts b/src/core/search/fts.tsindex 66ca79e0..099271fe 100644--- a/src/core/search/fts.ts+++ b/src/core/search/fts.ts@@ -12,22 +12,56 @@import type { KeywordHit, Store } from "./store.ts";+function quoteToken(t: string): string {+ return `"${t.replace(/"/g, '""')}"`;+}+export function buildFtsMatch(rawQuery: string): string {const tokens = rawQuery.split(/\s+/).map((t) => t.trim()).filter((t) => t.length > 0);if (tokens.length === 0) return "";- return tokens.map((t) => `"${t.replace(/"/g, '""')}"`).join(" ");+ return tokens.map(quoteToken).join(" ");+}++/**+ * Query-expansion match (v0.20.0): keep the original tokens as an+ * implicit-AND group (the strong signal, still scored highest by bm25+ * because it matches more terms) and OR in each expansion term as an+ * optional broadening alternative:+ *+ * ("orig1" "orig2") OR "exp1" OR "exp2"+ *+ * With no expansion terms this is byte-identical to {@link buildFtsMatch},+ * so expansion is a true no-op when disabled or empty.+ */+export function buildExpandedFtsMatch(+ rawQuery: string,+ expandedTerms: ReadonlyArray<string>,+): string {+ const base = buildFtsMatch(rawQuery);+ if (base === "" || expandedTerms.length === 0) return base;+ const ors = expandedTerms.map(quoteToken).join(" OR ");+ return `(${base}) OR ${ors}`;}export interface RunFtsOptions {readonly limit: number;readonly pathPrefix?: string | null;+ /**+ * Optional synonym-expansion terms (v0.20.0). When present and+ * non-empty they are OR'd onto the original AND group to broaden+ * recall; absent/empty leaves the query byte-identical.+ */+ readonly expandedTerms?: ReadonlyArray<string>;}export function runFtsQuery(store: Store, rawQuery: string, opts: RunFtsOptions): KeywordHit[] {- const match = buildFtsMatch(rawQuery);+ const match =+ opts.expandedTerms && opts.expandedTerms.length > 0+ ? buildExpandedFtsMatch(rawQuery, opts.expandedTerms)+ : buildFtsMatch(rawQuery);if (match === "") return [];return store.keywordTopK(match, opts);}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex e55edbe7..e22814fe 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -64,6 +64,11 @@ const DEFAULTS = {maxHops: 1,hopDecay: 0.5,maxExpansionPerHit: 3,+ recencyShape: 0.8,+ recencyScale: 30,+ recencyAmplitude: 0.05,+ synonymMaxTerms: 3,+ cacheTtlSeconds: 300,};type IntegerRange = { readonly min?: number; readonly max?: number };@@ -97,6 +102,16 @@ function parseBool(raw: string | null, fallback: boolean, fieldName: string): bo}}+/** Parse a strictly-positive finite float (e.g. Weibull shape / scale). */+function parsePositiveFloat(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || n <= 0) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number > 0, got '${raw}'`);+ }+ return n;+}+function validateIntegerRange(n: number, fieldName: string, range?: IntegerRange): void {if (range?.min !== undefined && n < range.min) {throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${range.min}, got ${n}`);@@ -267,11 +282,71 @@ export function resolveSearchConfig(opts: {"search_max_expansion_per_hit",{ min: 0 },);+ const recencyShape = parsePositiveFloat(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RECENCY_SHAPE", "search_recency_shape"),+ DEFAULTS.recencyShape,+ "search_recency_shape",+ );+ const recencyScale = parsePositiveFloat(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RECENCY_SCALE", "search_recency_scale"),+ DEFAULTS.recencyScale,+ "search_recency_scale",+ );+ const recencyAmplitude = parseFloat01(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_RECENCY_AMPLITUDE",+ "search_recency_amplitude",+ ),+ DEFAULTS.recencyAmplitude,+ "search_recency_amplitude",+ );+ const intentEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_INTENT_ENABLED", "search_intent_enabled"),+ true,+ "search_intent_enabled",+ );+ const synonymEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SYNONYM_ENABLED", "search_synonym_enabled"),+ false,+ "search_synonym_enabled",+ );+ const synonymMaxTerms = parseInteger(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS",+ "search_synonym_max_terms",+ ),+ DEFAULTS.synonymMaxTerms,+ "search_synonym_max_terms",+ { min: 0 },+ );+ const cacheEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED", "search_cache_enabled"),+ false,+ "search_cache_enabled",+ );+ const cacheTtlSeconds = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL", "search_cache_ttl_seconds"),+ DEFAULTS.cacheTtlSeconds,+ "search_cache_ttl_seconds",+ { min: 1 },+ );const recall: ResolvedRecallConfig = Object.freeze({mmrLambda,maxHops,hopDecay,maxExpansionPerHit,+ recencyShape,+ recencyScale,+ recencyAmplitude,+ intentEnabled,+ synonymEnabled,+ synonymMaxTerms,+ cacheEnabled,+ cacheTtlSeconds,});const base: ResolvedSearchConfig = Object.freeze({diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex 72a6729c..9d2244ec 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -269,6 +269,14 @@ async function indexInto(store.setState("last_indexed_at", now);if (opts?.force) store.setState("last_full_index_at", now);+ // Bump the corpus-generation revision whenever the index actually+ // changed, so the persistent query cache (v0.20.0) is invalidated+ // after a content reindex even though the embedding model/dimension+ // and schema are unchanged.+ if (stats.added + stats.updated + stats.deleted > 0) {+ store.bumpIndexRevision();+ }+return freezeStats(stats, Date.now() - t0);} finally {if (ownsStore) await store.close();diff --git a/src/core/search/query-cache.ts b/src/core/search/query-cache.tsnew file mode 100644index 00000000..91ead65f--- /dev/null+++ b/src/core/search/query-cache.ts@@ -0,0 +1,108 @@+/**+ * Persistent query-cache helpers (v0.20.0).+ *+ * The cache stores a serialized {@link SearchOutcome} keyed by a hash of+ * the result-affecting request (the search options + the query plan hash+ * + a fingerprint of the resolved config) and tagged with the corpus+ * generation it was computed under. A row is served only when its+ * generation matches the current one and it is within the TTL; otherwise+ * the caller recomputes. This module owns the key construction and the+ * (de)serialization; the SQLite I/O lives on `Store`.+ *+ * The expansion terms are deliberately NOT part of the key: they are a+ * deterministic function of (query, index content), and any index+ * content change bumps the corpus generation, so the generation gate+ * already covers them.+ */++import type { Store } from "./store.ts";+import type { SearchOptions, SearchOutcome } from "./types.ts";++/** Deterministic FNV-1a 32-bit hash, hex-encoded. */+function fnv1a(input: string): string {+ let h = 0x811c9dc5;+ for (let i = 0; i < input.length; i++) {+ h ^= input.charCodeAt(i);+ h = (h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24))) >>> 0;+ }+ return h.toString(16).padStart(8, "0");+}++/** Canonicalise a property filter map: sort values per key, then keys. */+function canonicalProperties(+ props: ReadonlyMap<string, ReadonlyArray<string>> | undefined,+): Array<[string, string[]]> | null {+ if (!props || props.size === 0) return null;+ return [...props.entries()]+ .map(([k, v]) => [k, [...v].toSorted()] as [string, string[]])+ .toSorted((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));+}++/**+ * Build a stable cache key from the result-affecting request. Every+ * option that can change the result set is folded in, in a canonical+ * (order-independent) form, alongside the plan hash and a fingerprint of+ * the resolved config.+ */+export function buildCacheKey(+ opts: SearchOptions,+ planHash: string,+ configFingerprint: string,+): string {+ const canonical = JSON.stringify({+ q: (opts.query ?? "").trim(),+ limit: opts.limit ?? null,+ semantic: opts.semantic ?? null,+ keywordOnly: opts.keywordOnly ?? false,+ pathPrefix: opts.pathPrefix ?? null,+ keywordWeight: opts.keywordWeight ?? null,+ semanticWeight: opts.semanticWeight ?? null,+ mmrLambda: opts.mmrLambda ?? null,+ maxHops: opts.maxHops ?? null,+ properties: canonicalProperties(opts.properties),+ visibility: opts.visibility ? [...opts.visibility].toSorted() : null,+ plan: planHash,+ cfg: configFingerprint,+ });+ return fnv1a(canonical);+}++/**+ * Return the cached outcome for `key` when it is present, computed under+ * the current `generation`, and within `ttlMs` of `nowMs`. Otherwise+ * null (the caller recomputes). A corrupt payload also returns null.+ */+export function getCachedOutcome(+ store: Store,+ key: string,+ generation: string,+ ttlMs: number,+ nowMs: number,+): SearchOutcome | null {+ const row = store.queryCacheGet(key);+ if (!row) return null;+ if (row.generation !== generation) return null;+ if (nowMs - row.createdAt > ttlMs) return null;+ try {+ const parsed = JSON.parse(row.payload) as SearchOutcome;+ // Match the fresh-compute path's immutability: freeze the results+ // array and each result so a hit is indistinguishable from a miss.+ for (const r of parsed.results) Object.freeze(r);+ Object.freeze(parsed.results);+ Object.freeze(parsed.warnings);+ return Object.freeze(parsed);+ } catch {+ return null;+ }+}++/** Serialize and store an outcome under the current generation. */+export function putCachedOutcome(+ store: Store,+ key: string,+ generation: string,+ outcome: SearchOutcome,+ nowMs: number,+): void {+ store.queryCachePut(key, generation, JSON.stringify(outcome), nowMs);+}diff --git a/src/core/search/query-plan.ts b/src/core/search/query-plan.tsnew file mode 100644index 00000000..14a0c3dc--- /dev/null+++ b/src/core/search/query-plan.ts@@ -0,0 +1,112 @@+/**+ * Pure query analysis (v0.20.0): the single seam where a query is+ * inspected before retrieval. It produces a {@link QueryPlan} carrying a+ * structural intent, a bounded ranking {@link WeightProfile}, the+ * synonym-expansion terms (populated by `synonyms.ts`), and a stable+ * `planHash` used in the query-cache key.+ *+ * LANGUAGE-AGNOSTIC INVARIANT (audited here, in one place): intent and+ * expansion derive ONLY from structural signals - quoted spans, FTS+ * wildcards, wikilink shapes, the share of entity-like tokens (via the+ * already-structural `extractEntities`), and token count. No+ * natural-language word, synonym, or stopword list appears anywhere. The+ * classifier behaves identically across scripts and locales.+ *+ * The module is pure and deterministic: same query string in, same plan+ * out, with no I/O and no clock/random source.+ */++import { extractEntities } from "./entities.ts";+import type { QueryIntent, QueryPlan, WeightProfile } from "./types.ts";++/** No-effect profile: every layer keeps its configured weight. */+export const NEUTRAL_PROFILE: WeightProfile = Object.freeze({+ keywordMul: 1,+ semanticMul: 1,+ entityMul: 1,+ recencyMul: 1,+});++/**+ * Fixed structural-feature -> profile table. Every multiplier stays+ * within [0.7, 1.4], so a (mis)classification can only re-weight an+ * already-relevant set, never float an unrelated document.+ */+const PROFILES: Record<QueryIntent, WeightProfile> = Object.freeze({+ neutral: NEUTRAL_PROFILE,+ // Literal lookup: trust the keyword/FTS layer, discount fuzzy semantic.+ exact: Object.freeze({ keywordMul: 1.3, semanticMul: 0.7, entityMul: 1, recencyMul: 1 }),+ // Proper-noun lookup: amplify the entity layer, nudge keyword.+ entity: Object.freeze({ keywordMul: 1.15, semanticMul: 0.9, entityMul: 1.4, recencyMul: 1 }),+ // Open-ended exploration: lean on semantic similarity and recency.+ broad: Object.freeze({ keywordMul: 0.9, semanticMul: 1.2, entityMul: 1, recencyMul: 1.1 }),+});++const QUOTED_PHRASE_RE = /"[^"\n]{2,}"/u;+const WILDCARD_RE = /\*/u;+const WIKILINK_RE = /\[\[[^\]\n]+\]\]/u;++/** Lowercase + trim + collapse internal whitespace. No word lists. */+function normalize(query: string): string {+ return query.trim().replace(/\s+/gu, " ").toLowerCase();+}++/** Deterministic FNV-1a 32-bit hash, hex-encoded. No crypto/I/O. */+function fnv1a(input: string): string {+ let h = 0x811c9dc5;+ for (let i = 0; i < input.length; i++) {+ h ^= input.charCodeAt(i);+ // 32-bit FNV prime multiply via shifts, kept in uint32.+ h = (h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24))) >>> 0;+ }+ return h.toString(16).padStart(8, "0");+}++/** Token count over normalized whitespace. Empty string -> 0. */+function tokenCount(normalized: string): number {+ if (normalized === "") return 0;+ return normalized.split(" ").length;+}++function classify(query: string, normalized: string): QueryIntent {+ if (normalized === "") return "neutral";++ // Rule 1: a literal phrase or prefix wildcard means the caller wants+ // an exact textual hit - favour the keyword layer.+ if (QUOTED_PHRASE_RE.test(query) || WILDCARD_RE.test(query)) return "exact";++ const entities = extractEntities(query);+ const tokens = tokenCount(normalized);+ const entityShare = tokens === 0 ? 0 : entities.length / tokens;++ // Rule 2: explicit wikilinks or a query dominated by entity-like+ // tokens is a proper-noun lookup.+ if (WIKILINK_RE.test(query) || entityShare >= 0.5) return "entity";++ // Rule 3: a long query with few entities reads as open-ended.+ if (tokens >= 6 && entityShare < 0.2) return "broad";++ return "neutral";+}++/**+ * Analyse a query into a deterministic plan. `expandedTerms` is empty+ * here; synonym expansion (a later layer) fills it before the hash is+ * meaningful for caching. The hash folds in the normalized query, the+ * intent, and any expanded terms - everything that changes results.+ */+export function buildQueryPlan(+ query: string,+ expandedTerms: ReadonlyArray<string> = [],+): QueryPlan {+ const normalized = normalize(query);+ const intent = classify(query, normalized);+ const terms = Object.freeze([...expandedTerms]);+ const planHash = fnv1a(`${normalized}|${intent}|${terms.join(",")}`);+ return Object.freeze({+ intent,+ weightProfile: PROFILES[intent],+ expandedTerms: terms,+ planHash,+ });+}diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.tsindex f159a3f6..0dc0a713 100644--- a/src/core/search/ranker.ts+++ b/src/core/search/ranker.ts@@ -10,8 +10,9 @@*/import { PAGE_TIER_DEFAULT, tierWeight, type PageTier } from "../brain/page-meta/tier.ts";+import { weibullDecay, DEFAULT_RECENCY, type WeibullRecencyOptions } from "./recency.ts";import type { KeywordHit, SemanticHit, HydratedChunk } from "./store.ts";-import type { BrainSearchResult } from "./types.ts";+import type { BrainSearchResult, WeightProfile } from "./types.ts";export interface RankerInputs {readonly keyword: ReadonlyArray<KeywordHit>;@@ -45,6 +46,18 @@ export interface RankerOptions {readonly nowMs?: number;/** When false, semantic_score is ignored regardless of inputs. */readonly semanticEnabled?: boolean;+ /**+ * Weibull recency curve parameters. Absent uses {@link DEFAULT_RECENCY},+ * which approximates the legacy step function. Callers (search.ts)+ * thread the resolved config here.+ */+ readonly recency?: WeibullRecencyOptions;+ /**+ * Per-query ranking multipliers from the query plan (v0.20.0). Absent+ * (or an all-1.0 neutral profile) leaves every layer at its configured+ * weight, so ranking is bit-identical to pre-intent behaviour.+ */+ readonly weightProfile?: WeightProfile;}const DAY_MS = 24 * 60 * 60 * 1000;@@ -81,13 +94,10 @@ function semanticFromDistance(distance: number): number {return clamp01(sim);}-function recencyBoost(mtime: number, nowMs: number): number {+function recencyBoost(mtime: number, nowMs: number, opts: WeibullRecencyOptions): number {const ageMs = Math.max(0, nowMs - mtime * 1000);const ageDays = ageMs / DAY_MS;- if (ageDays <= 7) return 0.05;- if (ageDays <= 30) return 0.025;- if (ageDays <= 90) return 0.01;- return 0;+ return weibullDecay(ageDays, opts);}interface Candidate {@@ -134,6 +144,13 @@ function buildReasons(parts: {export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSearchResult[] {const nowMs = opts.nowMs ?? Date.now();const semanticEnabled = opts.semanticEnabled !== false;+ const recencyOpts = opts.recency ?? DEFAULT_RECENCY;+ // Per-query intent multipliers. Absent or neutral (all 1.0) leaves the+ // score bit-identical to pre-intent behaviour.+ const kwMul = opts.weightProfile?.keywordMul ?? 1;+ const semMul = opts.weightProfile?.semanticMul ?? 1;+ const entMul = opts.weightProfile?.entityMul ?? 1;+ const recMul = opts.weightProfile?.recencyMul ?? 1;const kwNorm = normalizeBm25(inputs.keyword);@@ -224,10 +241,10 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSeaconst link = linkBoostFor(c);const tag = tagBoostFor(c);const linkBoost = Math.min(0.05, link + tag);- const recency = recencyBoost(c.mtime, nowMs);+ const recency = recencyBoost(c.mtime, nowMs, recencyOpts) * recMul;const weighted =- opts.keywordWeight * c.keywordScore +- (semanticEnabled ? opts.semanticWeight : 0) * c.semanticScore;+ opts.keywordWeight * kwMul * c.keywordScore ++ (semanticEnabled ? opts.semanticWeight * semMul : 0) * c.semanticScore;// Tier multiplier applied to the relevance portion only so the// tag / link / recency boosts stay tier-neutral. Default// `supporting` โ 1.0 keeps untagged vaults bit-identical.@@ -237,7 +254,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea// Per-match 0.02, capped at 0.04 so it only re-ranks an already// relevant set - never enough to float an irrelevant chunk.const entityMatches = inputs.entityMatchByChunk?.get(c.chunkId) ?? 0;- const entityBoost = Math.min(0.04, entityMatches * 0.02);+ const entityBoost = Math.min(0.04, entityMatches * 0.02 * entMul);const score = clamp01(weighted * tierMul + linkBoost + recency + entityBoost);ranked.push(diff --git a/src/core/search/recency.ts b/src/core/search/recency.tsnew file mode 100644index 00000000..72225b30--- /dev/null+++ b/src/core/search/recency.ts@@ -0,0 +1,72 @@+/**+ * Pure Weibull recency decay (v0.20.0).+ *+ * Replaces the prior hardcoded step function (<=7d -> 0.05, <=30d ->+ * 0.025, <=90d -> 0.01, else 0) with a continuous, configurable curve so+ * a vault can tune how fast older content loses its recency boost.+ *+ * The boost is the Weibull survival function scaled by an amplitude:+ *+ * boost(age) = amplitude * exp( -(age / scale)^shape )+ *+ * At age 0 the survival term is 1, so the boost equals `amplitude` (the+ * maximum). As age grows the boost decays monotonically toward 0. `shape`+ * controls the curvature (k); `scale` is the characteristic lifetime in+ * days (lambda). The defaults approximate the old step function while+ * smoothing the cliffs at the 7/30/90-day boundaries.+ *+ * The function is pure and deterministic: no Date.now(), no Math.random().+ * Callers pass an already-computed age in days, so the time source stays+ * injectable at the ranker boundary.+ */++export interface WeibullRecencyOptions {+ /** Weibull shape parameter k (> 0). Lower values decay faster early. */+ readonly shape: number;+ /** Weibull scale parameter lambda in days (> 0): the characteristic lifetime. */+ readonly scale: number;+ /** Maximum boost at age 0, clamped into [0, 1]. */+ readonly amplitude: number;+}++/**+ * Default curve. Approximates the legacy step function: ~0.043 at 3 days,+ * ~0.024 at 20 days, ~0.009 at 60 days, and decays below the display+ * epsilon by roughly half a year.+ */+export const DEFAULT_RECENCY: WeibullRecencyOptions = Object.freeze({+ shape: 0.8,+ scale: 30,+ amplitude: 0.05,+});++/**+ * Below this the boost rounds to 0.000 at the 3-decimal precision the+ * explainable-recall `reasons` use, so we floor it to exactly 0. This+ * keeps an effectively-stale page from carrying a noise-level recency+ * layer and preserves the "old content has no recency reason" contract.+ */+const EPSILON = 0.0005;++function clamp01(x: number): number {+ if (!Number.isFinite(x)) return 0;+ if (x < 0) return 0;+ if (x > 1) return 1;+ return x;+}++/**+ * Recency boost for a content age in days under a Weibull curve. Ages at+ * or below 0 (including future timestamps) yield the full amplitude.+ * Non-positive shape/scale/amplitude disable the boost (returns 0), the+ * documented off switch. Results below the display epsilon floor to 0.+ */+export function weibullDecay(ageDays: number, opts: WeibullRecencyOptions): number {+ const amplitude = clamp01(opts.amplitude);+ if (amplitude === 0 || !(opts.shape > 0) || !(opts.scale > 0)) return 0;+ if (!Number.isFinite(ageDays) || ageDays <= 0) return amplitude;++ const survival = Math.exp(-Math.pow(ageDays / opts.scale, opts.shape));+ const boost = amplitude * survival;+ return boost < EPSILON ? 0 : boost;+}diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsindex 8fdc8114..50ae1dc6 100644--- a/src/core/search/schema.ts+++ b/src/core/search/schema.ts@@ -16,7 +16,7 @@ import { SearchError } from "./types.ts";* this raises `SCHEMA_MISMATCH` on open โ the operator must reindex* with a newer binary.*/-export const LATEST_SCHEMA_VERSION = 3;+export const LATEST_SCHEMA_VERSION = 4;const DDL_V1 = `CREATE TABLE IF NOT EXISTS documents (@@ -200,6 +200,27 @@ export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([}},},+ {+ // v4 (v0.20.0) - persistent query cache. `query_cache` stores a+ // serialized search result keyed by a hash of the result-affecting+ // request, tagged with the corpus generation it was computed under+ // and a creation timestamp for TTL. The generation tag is what makes+ // the cache self-invalidating: a row whose generation no longer+ // matches the current one is never served and is swept. Additive and+ // reindex-safe; existing data is untouched.+ version: 4,+ up(db) {+ db.exec(`+ CREATE TABLE IF NOT EXISTS query_cache (+ cache_key TEXT PRIMARY KEY,+ generation TEXT NOT NULL,+ payload TEXT NOT NULL,+ created_at INTEGER NOT NULL+ );+ CREATE INDEX IF NOT EXISTS idx_query_cache_generation ON query_cache(generation);+ `);+ },+ },]);/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 339c1955..8385b6d3 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -17,6 +17,9 @@ import { makeProvider } from "./embeddings/provider.ts";import { extractEntities } from "./entities.ts";import { runFtsQuery } from "./fts.ts";import { mmrRerank } from "./mmr.ts";+import { buildQueryPlan } from "./query-plan.ts";+import { buildCacheKey, getCachedOutcome, putCachedOutcome } from "./query-cache.ts";+import { deriveExpansionTerms, tokenizeForExpansion, DEFAULT_EXPANSION } from "./synonyms.ts";import { filterByProperties } from "./property-filter.ts";import { rankResults } from "./ranker.ts";import { expandByTraversal, type TraversalOptions } from "./traversal.ts";@@ -45,6 +48,32 @@ function resolveSemanticPolicy(config: ResolvedSearchConfig, opts: SearchOptionsreturn { explicit: false, wantSemantic: config.semantic.enabled };}+/**+ * A compact fingerprint of the resolved-config fields that change search+ * results, folded into the cache key so a config change (weights,+ * semantic toggle, recall tunables) invalidates cached rows alongside the+ * corpus generation. Cache-only knobs (enable/TTL) are excluded - they do+ * not change result content.+ */+function configFingerprint(config: ResolvedSearchConfig): string {+ const r = config.recall;+ return JSON.stringify({+ kw: config.keywordWeight,+ sw: config.semanticWeight,+ sem: config.semantic.enabled,+ mmr: r.mmrLambda,+ hops: r.maxHops,+ hopDecay: r.hopDecay,+ maxExp: r.maxExpansionPerHit,+ rShape: r.recencyShape,+ rScale: r.recencyScale,+ rAmp: r.recencyAmplitude,+ intent: r.intentEnabled,+ syn: r.synonymEnabled,+ synMax: r.synonymMaxTerms,+ });+}+function assertSafePathPrefix(prefix: string | undefined): string | undefined {if (!prefix) return undefined;if (prefix.includes("..") || prefix.startsWith("/")) {@@ -66,14 +95,81 @@ export async function search(const policy = resolveSemanticPolicy(config, opts);const warnings: string[] = [];+ // Query plan (v0.20.0): one structural pass yields the intent weight+ // profile and the cache key. Pure; no I/O. Expanded terms (if any) are+ // folded in once they have been derived from the store below.+ const basePlan = buildQueryPlan(query);+const store = await Store.open(config, { mode: "read" });try {+ // Persistent query cache (v0.20.0): opt-in. Keyed by the request ++ // base plan hash + a config fingerprint, gated by the corpus+ // generation and a TTL. A hit returns the previously computed+ // outcome; generation changes (embedding change or content reindex)+ // and TTL expiry invalidate it. Expansion terms are not in the key:+ // they are determined by (query, index content) and any content+ // change bumps the generation. The cache write is best-effort.+ const cacheEnabled = config.recall.cacheEnabled;+ const ttlMs = config.recall.cacheTtlSeconds * 1000;+ let cacheKey: string | null = null;+ let generation = "";+ if (cacheEnabled) {+ // The whole cache lookup is best-effort: any failure (e.g. a+ // SQLITE_BUSY past the busy_timeout under a concurrent reindex)+ // falls through to a normal fresh compute rather than breaking the+ // search. Key on the EFFECTIVE request (clamped limit, resolved+ // semantic decision) so equivalent calls share a cache entry.+ try {+ generation = store.corpusGeneration();+ const keyOpts = { ...opts, limit, semantic: policy.wantSemantic, keywordOnly: false };+ cacheKey = buildCacheKey(keyOpts, basePlan.planHash, configFingerprint(config));+ const hit = getCachedOutcome(store, cacheKey, generation, ttlMs, Date.now());+ if (hit) return hit;+ } catch {+ cacheKey = null;+ }+ }+ const finalize = (outcome: SearchOutcome): SearchOutcome => {+ if (cacheEnabled && cacheKey) {+ try {+ store.queryCacheSweep(generation, Date.now() - ttlMs);+ putCachedOutcome(store, cacheKey, generation, outcome, Date.now());+ } catch {+ // Cache persistence is best-effort; never fail a search on it.+ }+ }+ return outcome;+ };+// Keyword candidates.- const kwHits = runFtsQuery(store, query, {+ let kwHits = runFtsQuery(store, query, {limit: limit * 3,pathPrefix,});+ // Synonym / query expansion (v0.20.0): opt-in and never for an+ // exact-intent (quoted/wildcard) query. Derive related terms from+ // the top candidates' own content (local co-occurrence) and re-run+ // FTS with them OR'd onto the original query to broaden recall. A+ // no-op - byte-identical kwHits - when disabled or no term qualifies.+ let plan = basePlan;+ if (config.recall.synonymEnabled && basePlan.intent !== "exact" && kwHits.length > 0) {+ const topIds = kwHits.slice(0, 10).map((h) => h.chunkId);+ const ctx = store.hydrateChunks(topIds);+ const texts = topIds+ .map((id) => ctx.get(id)?.content ?? "")+ .filter((t) => t.length > 0);+ const expandedTerms = deriveExpansionTerms(tokenizeForExpansion(query), texts, {+ ...DEFAULT_EXPANSION,+ maxTerms: config.recall.synonymMaxTerms,+ });+ if (expandedTerms.length > 0) {+ plan = buildQueryPlan(query, expandedTerms);+ kwHits = runFtsQuery(store, query, { limit: limit * 3, pathPrefix, expandedTerms });+ }+ }+ const weightProfile = config.recall.intentEnabled ? plan.weightProfile : undefined;+// Semantic candidates (may be skipped).let semHits: ReturnType<Store["semanticTopK"]> = [];let semanticAttempted = false;@@ -94,11 +190,13 @@ export async function search(for (const h of semHits) allChunkIds.add(h.chunkId);const idsList = Array.from(allChunkIds);if (idsList.length === 0) {- return Object.freeze({- results: Object.freeze([] as ReadonlyArray<BrainSearchResult>),- warnings: Object.freeze(warnings),- total: 0,- });+ return finalize(+ Object.freeze({+ results: Object.freeze([] as ReadonlyArray<BrainSearchResult>),+ warnings: Object.freeze(warnings),+ total: 0,+ }),+ );}const hydrated = store.hydrateChunks(idsList);@@ -163,6 +261,12 @@ export async function search(semanticWeight: opts.semanticWeight ?? config.semanticWeight,limit: rankCap,semanticEnabled: policy.wantSemantic && semanticAttempted,+ recency: {+ shape: config.recall.recencyShape,+ scale: config.recall.recencyScale,+ amplitude: config.recall.recencyAmplitude,+ },+ ...(weightProfile !== undefined ? { weightProfile } : {}),},);const capHit = ranked.length >= rankCap;@@ -221,11 +325,13 @@ export async function search(return rels && rels.length > 0 ? { ...r, relations: Object.freeze(rels) } : r;});- return Object.freeze({- results: Object.freeze(withRelations),- warnings: Object.freeze(warnings),- total: withRelations.length,- });+ return finalize(+ Object.freeze({+ results: Object.freeze(withRelations),+ warnings: Object.freeze(warnings),+ total: withRelations.length,+ }),+ );} finally {await store.close();}diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex cf9c5c5f..82ed2214 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -18,6 +18,7 @@ import { existsSync, mkdirSync, renameSync } from "node:fs";import { dirname } from "node:path";import lockfile from "proper-lockfile";+import { computeCorpusGeneration } from "./corpus-generation.ts";import { SearchError } from "./types.ts";import type { ResolvedSearchConfig } from "./types.ts";import {@@ -143,6 +144,11 @@ function applyPragmas(db: Database): void {db.exec("PRAGMA journal_mode = WAL");db.exec("PRAGMA foreign_keys = ON");db.exec("PRAGMA synchronous = NORMAL");+ // Wait briefly for a concurrent writer (e.g. an indexer holding the WAL+ // write lock) instead of failing immediately with SQLITE_BUSY. Matters+ // for the opportunistic query-cache writes a read-mode connection makes+ // during search (v0.20.0); search itself also degrades gracefully.+ db.exec("PRAGMA busy_timeout = 5000");}function ensureFts5(db: Database): void {@@ -379,6 +385,67 @@ export class Store {this.db.run("DELETE FROM index_state WHERE key = ?", [key]);}+ // โโ index revision + corpus generation (v0.20.0) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++ /** Monotonic counter bumped on every index mutation; 0 if never set. */+ indexRevision(): number {+ const raw = this.getState("index_revision");+ const n = raw === null ? 0 : Number(raw);+ return Number.isFinite(n) && n >= 0 ? Math.floor(n) : 0;+ }++ /** Increment the index revision. Called after a mutating index run. */+ bumpIndexRevision(): void {+ this.setState("index_revision", String(this.indexRevision() + 1));+ }++ /**+ * Current corpus-generation fingerprint: embedding model + dimension ++ * schema version + index revision. The query cache gates on this so any+ * embedding change or content reindex invalidates cached results.+ */+ corpusGeneration(): string {+ const dimRaw = this.getState("embedding_dimension");+ const dim = dimRaw === null ? null : Number(dimRaw);+ return computeCorpusGeneration({+ embeddingModel: this.getState("embedding_model"),+ embeddingDimension: dim !== null && Number.isFinite(dim) ? dim : null,+ schemaVersion: LATEST_SCHEMA_VERSION,+ indexRevision: this.indexRevision(),+ });+ }++ // โโ query cache (v0.20.0) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ++ queryCacheGet(+ key: string,+ ): { generation: string; payload: string; createdAt: number } | null {+ const row = this.db+ .query<{ generation: string; payload: string; created_at: number }, [string]>(+ "SELECT generation, payload, created_at FROM query_cache WHERE cache_key = ?",+ )+ .get(key);+ if (!row) return null;+ return { generation: row.generation, payload: row.payload, createdAt: row.created_at };+ }++ queryCachePut(key: string, generation: string, payload: string, createdAtMs: number): void {+ this.db.run(+ "INSERT INTO query_cache(cache_key, generation, payload, created_at) VALUES (?, ?, ?, ?) " ++ "ON CONFLICT(cache_key) DO UPDATE SET generation = excluded.generation, " ++ "payload = excluded.payload, created_at = excluded.created_at",+ [key, generation, payload, createdAtMs],+ );+ }++ /** Delete rows from a stale generation or created before the cutoff. */+ queryCacheSweep(currentGeneration: string, expiredBeforeMs: number): void {+ this.db.run("DELETE FROM query_cache WHERE generation <> ? OR created_at < ?", [+ currentGeneration,+ expiredBeforeMs,+ ]);+ }+// โโ documents โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโlistDocuments(): Map<string, DocumentSummary> {diff --git a/src/core/search/synonyms.ts b/src/core/search/synonyms.tsnew file mode 100644index 00000000..7df4e71a--- /dev/null+++ b/src/core/search/synonyms.ts@@ -0,0 +1,91 @@+/**+ * Pure, language-agnostic query expansion via local co-occurrence+ * (pseudo-relevance feedback), v0.20.0.+ *+ * Goal: broaden recall when the caller's wording differs from the stored+ * wording, WITHOUT any per-language synonym dictionary. The signal is+ * derived entirely from the vault's own content: given the top candidate+ * chunks for the original query, the terms that co-occur across several+ * of those chunks (but are not themselves in the query) are likely+ * related and worth adding as optional FTS alternatives.+ *+ * Tokenization is purely structural - maximal Unicode letter/digit runs,+ * lowercased. There is deliberately NO language-specific segmentation+ * (a continuous CJK run stays one token, exactly as FTS5 indexes it), so+ * the behaviour is identical across scripts and bit-stable on every+ * Syncthing peer.+ *+ * The module is pure and deterministic: no I/O, no clock, no randomness.+ * Ties break alphabetically so the output never depends on input order+ * beyond the documented frequency ranking.+ */++export interface ExpansionOptions {+ /** Maximum number of expansion terms to return. */+ readonly maxTerms: number;+ /** Minimum token length (in code points) to be eligible. */+ readonly minLength: number;+ /** Minimum number of distinct candidate docs a term must appear in. */+ readonly minDocFreq: number;+}++export const DEFAULT_EXPANSION: ExpansionOptions = Object.freeze({+ maxTerms: 3,+ minLength: 3,+ minDocFreq: 2,+});++const TOKEN_RE = /[\p{L}\p{N}]+/gu;++/** Maximal Unicode letter/digit runs, lowercased. No language rules. */+export function tokenizeForExpansion(text: string): string[] {+ const out: string[] = [];+ for (const m of text.matchAll(TOKEN_RE)) {+ out.push(m[0].toLowerCase());+ }+ return out;+}++/** Code-point length, so multi-byte scripts are measured fairly. */+function codePointLength(s: string): number {+ let n = 0;+ for (const _ of s) n++;+ return n;+}++/**+ * Derive expansion terms from the texts of the top candidate chunks for+ * a query. Returns up to `maxTerms` terms that (a) are not query tokens,+ * (b) are at least `minLength` code points, and (c) appear in at least+ * `minDocFreq` distinct candidate docs - ranked by document frequency+ * descending, then alphabetically. Empty input or no qualifying term+ * yields `[]` (a no-op for the caller).+ */+export function deriveExpansionTerms(+ queryTokens: ReadonlyArray<string>,+ candidateTexts: ReadonlyArray<string>,+ opts: ExpansionOptions,+): string[] {+ if (candidateTexts.length === 0 || opts.maxTerms <= 0) return [];++ const queryset = new Set(queryTokens.map((t) => t.toLowerCase()));+ const docFreq = new Map<string, number>();++ for (const text of candidateTexts) {+ // Count each term once per document (document frequency).+ const seen = new Set<string>();+ for (const tok of tokenizeForExpansion(text)) {+ if (seen.has(tok)) continue;+ seen.add(tok);+ if (queryset.has(tok)) continue;+ if (codePointLength(tok) < opts.minLength) continue;+ docFreq.set(tok, (docFreq.get(tok) ?? 0) + 1);+ }+ }++ return [...docFreq.entries()]+ .filter(([, df]) => df >= opts.minDocFreq)+ .toSorted((a, b) => (b[1] !== a[1] ? b[1] - a[1] : a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0))+ .slice(0, opts.maxTerms)+ .map(([term]) => term);+}diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex b2e31fc6..211367d4 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -63,6 +63,40 @@ export interface BrainSearchResult {readonly relations?: ReadonlyArray<{ readonly relation: string; readonly target: string }>;}+/**+ * Structural query intent (v0.20.0). Derived purely from query shape -+ * quoted phrases, FTS wildcards, wikilinks, entity-token share, token+ * count - never from a natural-language word list. `neutral` trips no+ * rule and keeps ranking bit-identical.+ */+export type QueryIntent = "neutral" | "exact" | "entity" | "broad";++/**+ * Per-query ranking multipliers emitted by the query plan. Each is a+ * bounded multiplier applied to the corresponding ranking layer; the+ * neutral profile is all 1.0 (no effect).+ */+export interface WeightProfile {+ readonly keywordMul: number;+ readonly semanticMul: number;+ readonly entityMul: number;+ readonly recencyMul: number;+}++/**+ * Pure analysis of an incoming query (v0.20.0). Computed once before+ * retrieval and shared by intent-aware ranking (the `weightProfile`) and+ * candidate augmentation (`expandedTerms`, populated by synonym+ * expansion). `planHash` is a stable fingerprint of everything in the+ * plan that affects results - used as part of the query-cache key.+ */+export interface QueryPlan {+ readonly intent: QueryIntent;+ readonly weightProfile: WeightProfile;+ readonly expandedTerms: ReadonlyArray<string>;+ readonly planHash: string;+}+export interface IndexStats {readonly added: number;readonly updated: number;@@ -190,6 +224,41 @@ export interface ResolvedRecallConfig {readonly hopDecay: number;/** Cap on outbound links followed per node. */readonly maxExpansionPerHit: number;+ /**+ * Weibull recency decay curve (v0.20.0). `recencyShape` is the Weibull+ * shape k (> 0); `recencyScale` is the characteristic lifetime in days+ * (> 0); `recencyAmplitude` is the maximum boost at age 0, in [0, 1].+ * Amplitude 0 disables the recency layer. See `recency.ts`.+ */+ readonly recencyShape: number;+ readonly recencyScale: number;+ readonly recencyAmplitude: number;+ /**+ * Query-intent classification (v0.20.0). When true (default) the query+ * plan's weight profile re-weights ranking per detected intent; when+ * false the neutral profile is used and ranking is bit-identical to+ * pre-intent behaviour.+ */+ readonly intentEnabled: boolean;+ /**+ * Synonym / query expansion (v0.20.0). Off by default: expansion+ * broadens the candidate set via local co-occurrence, so it changes+ * results and is opt-in. `synonymMaxTerms` caps how many expansion+ * terms are OR'd onto the query. Always suppressed for exact-intent+ * (quoted/wildcard) queries. See `synonyms.ts`.+ */+ readonly synonymEnabled: boolean;+ readonly synonymMaxTerms: number;+ /**+ * Persistent query cache (v0.20.0). Off by default: when enabled,+ * `search()` serves a previously computed result for an identical+ * request as long as the corpus generation is unchanged and the row is+ * within `cacheTtlSeconds`. A cache hit is the result that was+ * computed and stored; generation changes (embedding change or content+ * reindex) and TTL expiry invalidate it.+ */+ readonly cacheEnabled: boolean;+ readonly cacheTtlSeconds: number;}export interface ResolvedSearchConfig {diff --git a/src/mcp/brain-tools.ts b/src/mcp/brain-tools.tsindex 044869db..991f67d0 100644--- a/src/mcp/brain-tools.ts+++ b/src/mcp/brain-tools.ts@@ -63,6 +63,7 @@ import { buildWeeklySynthesis } from "../core/brain/temporal/weekly-brief.ts";import { loadTemporalConfigSafe } from "../core/brain/policy.ts";import { isBrainLogEventKind, type BrainLogEventKind } from "../core/brain/types.ts";import { packContext } from "../core/brain/context-pack.ts";+import { buildPreCompressPack } from "../core/brain/pre-compress-pack.ts";import { collectMaintenanceActions } from "../core/brain/maintenance/collect.ts";import { normaliseWikilinkTarget } from "../core/brain/wikilink.ts";import { renderDigest, type DigestFormat } from "../core/brain/digest.ts";@@ -1606,9 +1607,13 @@ async function toolBrainContextPack(throw new MCPError(INVALID_PARAMS, "brain_context_pack: max_tokens must be a positive integer");}const query = typeof args["query"] === "string" ? (args["query"] as string) : undefined;+ const maxCharsPerMemory = optionalPositiveInt(args, "max_chars_per_memory", "brain_context_pack");+ const maxTotalChars = optionalPositiveInt(args, "max_total_chars", "brain_context_pack");const report = packContext(ctx.vault, {maxTokens,...(query ? { query } : {}),+ ...(maxCharsPerMemory !== undefined ? { maxCharsPerMemory } : {}),+ ...(maxTotalChars !== undefined ? { maxTotalChars } : {}),});return {vault_path: ctx.vault,@@ -1620,6 +1625,7 @@ async function toolBrainContextPack(tier: i.tier,tokens: i.tokens,body: i.body,+ trimmed: i.trimmed,})),skipped: report.skipped.map((s) => ({id: s.id,@@ -1629,6 +1635,54 @@ async function toolBrainContextPack(};}+// ----- brain_pre_compress_pack (v0.20.0) -----------------------------------++/** Parse an optional positive-integer arg, throwing INVALID_PARAMS otherwise. */+function optionalPositiveInt(+ args: Record<string, unknown>,+ key: string,+ tool: string,+): number | undefined {+ const raw = args[key];+ if (raw === undefined) return undefined;+ const n =+ typeof raw === "number"+ ? raw+ : typeof raw === "string" && /^[0-9]+$/.test(raw.trim())+ ? Number.parseInt(raw.trim(), 10)+ : Number.NaN;+ if (!Number.isInteger(n) || n <= 0) {+ throw new MCPError(INVALID_PARAMS, `${tool}: ${key} must be a positive integer`);+ }+ return n;+}++/**+ * Read-only bundle of the highest-confidence confirmed preferences plus+ * the head of active.md, rendered as a system-prompt addendum for a host+ * runtime to inject just before a context-compression event.+ */+async function toolBrainPreCompressPack(+ ctx: ServerContext,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const topK = optionalPositiveInt(args, "top_k", "brain_pre_compress_pack") ?? 10;+ const maxCharsPerMemory = optionalPositiveInt(args, "max_chars_per_memory", "brain_pre_compress_pack");+ const maxTotalChars = optionalPositiveInt(args, "max_total_chars", "brain_pre_compress_pack");+ const pack = buildPreCompressPack(ctx.vault, {+ topK,+ ...(maxCharsPerMemory !== undefined ? { maxCharsPerMemory } : {}),+ ...(maxTotalChars !== undefined ? { maxTotalChars } : {}),+ });+ return {+ vault_path: ctx.vault,+ text: pack.text,+ active_head_included: pack.activeHeadIncluded,+ total_chars: pack.totalChars,+ items: pack.items.map((i) => ({ id: i.id, principle: i.principle, trimmed: i.trimmed })),+ };+}+async function toolMcpLandscape(ctx: ServerContext): Promise<Record<string, unknown>> {const landscape = buildMcpLandscape(ctx.vault);return {@@ -2123,12 +2177,53 @@ export const BRAIN_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([type: "string",description: "Optional case/Unicode-insensitive substring filter on topic + principle.",},+ max_chars_per_memory: {+ type: "integer",+ minimum: 1,+ description:+ "Optional per-page character cap (code points): trim any single oversized page's body before it consumes the token budget, so one huge page cannot crowd out the rest. Trimmed pages carry `trimmed: true`.",+ },+ max_total_chars: {+ type: "integer",+ minimum: 1,+ description:+ "Optional second ceiling (code points) on the cumulative size of the returned slice. Lowest-priority overflow is dropped with an `over-char-budget` skip reason.",+ },},required: ["max_tokens"],additionalProperties: false,},handler: toolBrainContextPack,},+ {+ // No preview budget: the addendum is meant to be injected whole and+ // is already bounded by its own per-entry / total character caps.+ name: "brain_pre_compress_pack",+ description:+ "Return a compact system-prompt addendum of the top-K highest-confidence confirmed preferences plus the head of active.md, so a host runtime can inject it just before a context-compression event and keep high-salience constraints from rotating out. Bounded by optional per-memory and total character caps. Read-only.",+ inputSchema: {+ type: "object",+ properties: {+ top_k: {+ type: "integer",+ minimum: 1,+ description: "Maximum number of preferences to include, highest-confidence first (default 10).",+ },+ max_chars_per_memory: {+ type: "integer",+ minimum: 1,+ description: "Optional per-entry character cap (code points); trimmed entries carry `trimmed: true`.",+ },+ max_total_chars: {+ type: "integer",+ minimum: 1,+ description: "Optional total character cap (code points) across the addendum; lowest-priority overflow is dropped.",+ },+ },+ additionalProperties: false,+ },+ handler: toolBrainPreCompressPack,+ },{name: "brain_unlinked_mentions",description:diff --git a/src/mcp/instructions.ts b/src/mcp/instructions.tsindex bd3821b7..4024ed7c 100644--- a/src/mcp/instructions.ts+++ b/src/mcp/instructions.ts@@ -110,7 +110,10 @@ export function buildInstructions(opts: BuildInstructionsOpts | string): string"coverage.\n\n" +"Other tools: second_brain_status (config status), " +"vault_health (verify vault), second_brain_query (list vault pages " +- "by title - read-only).\n\n" ++ "by title - read-only). brain_pre_compress_pack returns a compact " ++ "system-prompt addendum (top confirmed preferences + the head of " ++ "active.md) for a host runtime to inject just before a " ++ "context-compression event - read-only.\n\n" +"Preview budget: large tool results may come back as a preview " +"envelope - a JSON object with `preview_truncated: true`, " +"`bytes_preview` (a head slice), `full_chars`, and an `artifact_id`. " +diff --git a/tests/core/brain/context-pack.test.ts b/tests/core/brain/context-pack.test.tsindex 79753b73..9119d7b1 100644--- a/tests/core/brain/context-pack.test.ts+++ b/tests/core/brain/context-pack.test.ts@@ -40,6 +40,39 @@ describe("packContext", () => {expect(r.skipped.length).toBe(0);});+ test("maxCharsPerMemory trims an oversized page body and flags it trimmed", () => {+ const body = "x".repeat(500);+ writeFileSync(+ join(vault, "Brain", "preferences", "pref-big.md"),+ ["---", "id: pref-big", "topic: t", "principle: p", "tier: core", "---", "", body].join("\n"),+ );++ const full = packContext(vault, { maxTokens: 100_000 });+ expect(full.items[0]!.trimmed).toBe(false);+ expect([...full.items[0]!.body].length).toBeGreaterThanOrEqual(500);++ const capped = packContext(vault, { maxTokens: 100_000, maxCharsPerMemory: 100 });+ expect([...capped.items[0]!.body].length).toBe(100);+ expect(capped.items[0]!.trimmed).toBe(true);+ });++ test("maxTotalChars drops lowest-priority overflow with an over-char-budget reason", () => {+ const writeBody = (slug: string, tier: string, body: string) =>+ writeFileSync(+ join(vault, "Brain", "preferences", `pref-${slug}.md`),+ ["---", `id: pref-${slug}`, "topic: t", "principle: p", `tier: ${tier}`, "---", "", body].join(+ "\n",+ ),+ );+ // Core (highest priority) emitted first, then peripheral.+ writeBody("a", "core", "a".repeat(60));+ writeBody("z", "peripheral", "z".repeat(60));++ const capped = packContext(vault, { maxTokens: 100_000, maxTotalChars: 60 });+ expect(capped.items.map((i) => i.id)).toEqual(["pref-a"]);+ expect(capped.skipped.find((s) => s.id === "pref-z")?.reason).toBe("over-char-budget");+ });+test("orders core โ supporting โ peripheral", () => {writePref("p", { topic: "x", principle: "peripheral one", tier: "peripheral" });writePref("s", { topic: "x", principle: "supporting one", tier: "supporting" });diff --git a/tests/core/brain/pre-compress-pack.test.ts b/tests/core/brain/pre-compress-pack.test.tsnew file mode 100644index 00000000..ab32cd6c--- /dev/null+++ b/tests/core/brain/pre-compress-pack.test.ts@@ -0,0 +1,106 @@+import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { writePreference } from "../../../src/core/brain/preference.ts";+import { BRAIN_PREFERENCE_STATUS, BRAIN_CONFIDENCE } from "../../../src/core/brain/types.ts";+import { brainActivePath } from "../../../src/core/brain/paths.ts";+import { buildPreCompressPack } from "../../../src/core/brain/pre-compress-pack.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-precompress-"));+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });+ mkdirSync(join(vault, "Brain", "inbox"), { recursive: true });+});+afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++function makePref(opts: {+ slug: string;+ principle: string;+ status?: (typeof BRAIN_PREFERENCE_STATUS)[keyof typeof BRAIN_PREFERENCE_STATUS];+ confidence_value?: number;+}): void {+ const status = opts.status ?? BRAIN_PREFERENCE_STATUS.confirmed;+ writePreference(vault, {+ slug: opts.slug,+ topic: opts.slug,+ principle: opts.principle,+ created_at: "2026-05-01T00:00:00Z",+ unconfirmed_until: "2026-05-08T00:00:00Z",+ status,+ evidenced_by: [`[[sig-2026-05-01-${opts.slug}]]`],+ confirmed_at: status === BRAIN_PREFERENCE_STATUS.unconfirmed ? null : "2026-05-02T00:00:00Z",+ applied_count: 1,+ violated_count: 0,+ last_evidence_at: "2026-05-02T00:00:00Z",+ confidence: BRAIN_CONFIDENCE.high,+ confidence_value: opts.confidence_value ?? 0.8,+ });+}++test("an empty brain yields no items and no active head", () => {+ const pack = buildPreCompressPack(vault, { topK: 10 });+ expect(pack.items).toEqual([]);+ expect(pack.activeHeadIncluded).toBe(false);+});++test("confirmed preferences are ranked by confidence then capped to topK", () => {+ makePref({ slug: "low", principle: "low one", confidence_value: 0.3 });+ makePref({ slug: "high", principle: "high one", confidence_value: 0.95 });+ makePref({ slug: "mid", principle: "mid one", confidence_value: 0.6 });+ const pack = buildPreCompressPack(vault, { topK: 2 });+ expect(pack.items.map((i) => i.id)).toEqual(["pref-high", "pref-mid"]);+});++test("unconfirmed and quarantine preferences are excluded", () => {+ makePref({ slug: "ok", principle: "confirmed one", status: BRAIN_PREFERENCE_STATUS.confirmed });+ makePref({+ slug: "trial",+ principle: "unconfirmed one",+ status: BRAIN_PREFERENCE_STATUS.unconfirmed,+ });+ makePref({+ slug: "quar",+ principle: "quarantine one",+ status: BRAIN_PREFERENCE_STATUS.quarantine,+ });+ const pack = buildPreCompressPack(vault, { topK: 10 });+ expect(pack.items.map((i) => i.id)).toEqual(["pref-ok"]);+});++test("the head of active.md is included in the rendered text when present", () => {+ writeFileSync(brainActivePath(vault), "# Active\n\nA pinned active line.\n");+ makePref({ slug: "p", principle: "a principle" });+ const pack = buildPreCompressPack(vault, { topK: 10 });+ expect(pack.activeHeadIncluded).toBe(true);+ expect(pack.text).toContain("pinned active line");+ expect(pack.text).toContain("a principle");+});++test("per-memory and total caps trim and drop, flagging trimmed items", () => {+ makePref({ slug: "a", principle: "x".repeat(200), confidence_value: 0.9 });+ makePref({ slug: "b", principle: "y".repeat(200), confidence_value: 0.5 });+ const pack = buildPreCompressPack(vault, {+ topK: 10,+ maxCharsPerMemory: 50,+ maxTotalChars: 60,+ });+ // a trimmed to 50; b would overflow the 60-char total and is dropped.+ expect(pack.items.map((i) => i.id)).toEqual(["pref-a"]);+ expect(pack.items[0]!.trimmed).toBe(true);+ expect([...pack.items[0]!.principle].length).toBe(50);+});++test("is deterministic for identical inputs", () => {+ makePref({ slug: "a", principle: "one", confidence_value: 0.9 });+ makePref({ slug: "b", principle: "two", confidence_value: 0.5 });+ expect(buildPreCompressPack(vault, { topK: 10 })).toEqual(+ buildPreCompressPack(vault, { topK: 10 }),+ );+});diff --git a/tests/core/brain/recall-budget.test.ts b/tests/core/brain/recall-budget.test.tsnew file mode 100644index 00000000..14d03540--- /dev/null+++ b/tests/core/brain/recall-budget.test.ts@@ -0,0 +1,79 @@+import { test, expect } from "bun:test";+import { applyCharBudget } from "../../../src/core/brain/recall-budget.ts";++const e = (item: string, text: string) => ({ item, text });++test("with no options every entry passes through untrimmed and nothing is dropped", () => {+ const r = applyCharBudget([e("a", "hello"), e("b", "world")], {});+ expect(r.kept.map((k) => k.text)).toEqual(["hello", "world"]);+ expect(r.kept.every((k) => !k.trimmed)).toBe(true);+ expect(r.dropped).toEqual([]);+ expect(r.totalChars).toBe(10);+});++test("maxCharsPerEntry truncates an oversized entry and flags it trimmed", () => {+ const r = applyCharBudget([e("a", "abcdefghij")], { maxCharsPerEntry: 4 });+ expect(r.kept[0]!.text).toBe("abcd");+ expect(r.kept[0]!.trimmed).toBe(true);+});++test("an entry at or under the per-entry cap is untouched", () => {+ const r = applyCharBudget([e("a", "abcd"), e("b", "ab")], { maxCharsPerEntry: 4 });+ expect(r.kept.map((k) => k.text)).toEqual(["abcd", "ab"]);+ expect(r.kept.every((k) => !k.trimmed)).toBe(true);+});++test("per-entry cap is measured in code points, not UTF-16 units", () => {+ // 5 astral code points; cap 3 keeps exactly 3 code points intact.+ const r = applyCharBudget([e("a", "๐๐๐๐๐")], { maxCharsPerEntry: 3 });+ expect([...r.kept[0]!.text].length).toBe(3);+ expect(r.kept[0]!.text).toBe("๐๐๐");+});++test("a non-positive per-entry cap disables trimming", () => {+ const r = applyCharBudget([e("a", "abcdef")], { maxCharsPerEntry: 0 });+ expect(r.kept[0]!.text).toBe("abcdef");+ expect(r.kept[0]!.trimmed).toBe(false);+});++test("entry order is preserved", () => {+ const r = applyCharBudget([e("a", "x"), e("b", "y"), e("c", "z")], { maxCharsPerEntry: 10 });+ expect(r.kept.map((k) => k.item)).toEqual(["a", "b", "c"]);+});++test("maxTotalChars keeps entries until the cap and drops lowest-priority overflow", () => {+ const r = applyCharBudget([e("a", "aaaa"), e("b", "bbbb"), e("c", "cccc")], { maxTotalChars: 8 });+ expect(r.kept.map((k) => k.item)).toEqual(["a", "b"]);+ expect(r.dropped).toEqual(["c"]);+ expect(r.totalChars).toBe(8);+});++test("total cap is measured in code points", () => {+ const r = applyCharBudget([e("a", "๐๐"), e("b", "๐๐")], { maxTotalChars: 3 });+ expect(r.kept.map((k) => k.item)).toEqual(["a"]);+ expect(r.dropped).toEqual(["b"]);+});++test("per-entry trim and total cap compose (trim first, then total)", () => {+ const r = applyCharBudget([e("a", "aaaaaa"), e("b", "bbbbbb")], {+ maxCharsPerEntry: 3,+ maxTotalChars: 5,+ });+ expect(r.kept.map((k) => k.text)).toEqual(["aaa"]);+ expect(r.dropped).toEqual(["b"]);+});++test("a smaller tail entry still fits after a larger overflow is dropped", () => {+ const r = applyCharBudget([e("a", "aaaa"), e("big", "bbbbbbbb"), e("c", "cc")], {+ maxTotalChars: 6,+ });+ expect(r.kept.map((k) => k.item)).toEqual(["a", "c"]);+ expect(r.dropped).toEqual(["big"]);+});++test("is deterministic for identical inputs", () => {+ const input = [e("a", "abcdef"), e("b", "ghijkl")];+ expect(applyCharBudget(input, { maxCharsPerEntry: 3 })).toEqual(+ applyCharBudget(input, { maxCharsPerEntry: 3 }),+ );+});diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.tsindex 3b3a9616..16cc1ef7 100644--- a/tests/core/search/config.test.ts+++ b/tests/core/search/config.test.ts@@ -25,6 +25,14 @@ const ENV_KEYS = ["OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT","OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY","OPEN_SECOND_BRAIN_EMBEDDING_BATCH",+ "OPEN_SECOND_BRAIN_SEARCH_RECENCY_SHAPE",+ "OPEN_SECOND_BRAIN_SEARCH_RECENCY_SCALE",+ "OPEN_SECOND_BRAIN_SEARCH_RECENCY_AMPLITUDE",+ "OPEN_SECOND_BRAIN_SEARCH_INTENT_ENABLED",+ "OPEN_SECOND_BRAIN_SEARCH_SYNONYM_ENABLED",+ "OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS",+ "OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED",+ "OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL",];beforeEach(() => {@@ -67,6 +75,63 @@ test("defaults are returned when config and env are empty", () => {expect(ignoreRaws).toContain("Brain/.snapshots");});+test("recall recency defaults to the Weibull curve params", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.recall.recencyShape).toBe(0.8);+ expect(cfg.recall.recencyScale).toBe(30);+ expect(cfg.recall.recencyAmplitude).toBe(0.05);+});++test("recency curve params are overridable via env and config", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_recency_scale: "45"\n`);+ process.env["OPEN_SECOND_BRAIN_SEARCH_RECENCY_SHAPE"] = "1.2";+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.recall.recencyShape).toBe(1.2);+ expect(cfg.recall.recencyScale).toBe(45);+});++test("query-intent classification defaults on and is toggleable", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.intentEnabled).toBe(true);+ process.env["OPEN_SECOND_BRAIN_SEARCH_INTENT_ENABLED"] = "false";+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.intentEnabled).toBe(false);+});++test("synonym expansion defaults off and is toggleable with a term cap", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const def = resolveSearchConfig({ vault: tmp, configPath }).recall;+ expect(def.synonymEnabled).toBe(false);+ expect(def.synonymMaxTerms).toBe(3);+ process.env["OPEN_SECOND_BRAIN_SEARCH_SYNONYM_ENABLED"] = "true";+ process.env["OPEN_SECOND_BRAIN_SEARCH_SYNONYM_MAX_TERMS"] = "5";+ const on = resolveSearchConfig({ vault: tmp, configPath }).recall;+ expect(on.synonymEnabled).toBe(true);+ expect(on.synonymMaxTerms).toBe(5);+});++test("query cache defaults off with a 300s TTL and is toggleable", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const def = resolveSearchConfig({ vault: tmp, configPath }).recall;+ expect(def.cacheEnabled).toBe(false);+ expect(def.cacheTtlSeconds).toBe(300);+ process.env["OPEN_SECOND_BRAIN_SEARCH_CACHE_ENABLED"] = "true";+ process.env["OPEN_SECOND_BRAIN_SEARCH_CACHE_TTL"] = "60";+ const on = resolveSearchConfig({ vault: tmp, configPath }).recall;+ expect(on.cacheEnabled).toBe(true);+ expect(on.cacheTtlSeconds).toBe(60);+});++test("non-positive recency scale is rejected", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_recency_scale: "0"\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/search_recency_scale/);+});++test("recency amplitude outside [0,1] is rejected", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_recency_amplitude: "2"\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/search_recency_amplitude/);+});+test("OPEN_SECOND_BRAIN_SEARCH_IGNORE has no effect (removed in v0.10.9)", () => {writeFileSync(configPath, `vault: "${tmp}"\n`);process.env["OPEN_SECOND_BRAIN_SEARCH_IGNORE"] = "from-env-1,from-env-2";diff --git a/tests/core/search/corpus-generation.test.ts b/tests/core/search/corpus-generation.test.tsnew file mode 100644index 00000000..86999f70--- /dev/null+++ b/tests/core/search/corpus-generation.test.ts@@ -0,0 +1,43 @@+import { test, expect } from "bun:test";+import { computeCorpusGeneration } from "../../../src/core/search/corpus-generation.ts";++const base = {+ embeddingModel: "text-embed-3",+ embeddingDimension: 1536,+ schemaVersion: 4,+ indexRevision: 7,+};++test("identical inputs produce an identical fingerprint", () => {+ expect(computeCorpusGeneration(base)).toBe(computeCorpusGeneration({ ...base }));+});++test("a different embedding model changes the fingerprint", () => {+ expect(computeCorpusGeneration({ ...base, embeddingModel: "other" })).not.toBe(+ computeCorpusGeneration(base),+ );+});++test("a different embedding dimension changes the fingerprint", () => {+ expect(computeCorpusGeneration({ ...base, embeddingDimension: 768 })).not.toBe(+ computeCorpusGeneration(base),+ );+});++test("a different schema version changes the fingerprint", () => {+ expect(computeCorpusGeneration({ ...base, schemaVersion: 5 })).not.toBe(+ computeCorpusGeneration(base),+ );+});++test("a bumped index revision (content reindex) changes the fingerprint", () => {+ expect(computeCorpusGeneration({ ...base, indexRevision: 8 })).not.toBe(+ computeCorpusGeneration(base),+ );+});++test("a null model / dimension (keyword-only vault) is stable and distinct from a set one", () => {+ const keywordOnly = { embeddingModel: null, embeddingDimension: null, schemaVersion: 4, indexRevision: 1 };+ expect(computeCorpusGeneration(keywordOnly)).toBe(computeCorpusGeneration({ ...keywordOnly }));+ expect(computeCorpusGeneration(keywordOnly)).not.toBe(computeCorpusGeneration(base));+});diff --git a/tests/core/search/query-cache.test.ts b/tests/core/search/query-cache.test.tsnew file mode 100644index 00000000..a1e939bb--- /dev/null+++ b/tests/core/search/query-cache.test.ts@@ -0,0 +1,35 @@+import { test, expect } from "bun:test";+import { buildCacheKey } from "../../../src/core/search/query-cache.ts";+import type { SearchOptions } from "../../../src/core/search/types.ts";++const opts = (o: Partial<SearchOptions>): SearchOptions => ({ query: "alpha", ...o });++test("identical request, plan, and config fingerprint produce the same key", () => {+ expect(buildCacheKey(opts({}), "plan1", "fp1")).toBe(buildCacheKey(opts({}), "plan1", "fp1"));+});++test("a different query changes the key", () => {+ expect(buildCacheKey(opts({ query: "alpha" }), "p", "fp")).not.toBe(+ buildCacheKey(opts({ query: "beta" }), "p", "fp"),+ );+});++test("a different limit changes the key", () => {+ expect(buildCacheKey(opts({ limit: 10 }), "p", "fp")).not.toBe(+ buildCacheKey(opts({ limit: 20 }), "p", "fp"),+ );+});++test("a different plan hash changes the key", () => {+ expect(buildCacheKey(opts({}), "planA", "fp")).not.toBe(buildCacheKey(opts({}), "planB", "fp"));+});++test("a different config fingerprint changes the key", () => {+ expect(buildCacheKey(opts({}), "p", "fpA")).not.toBe(buildCacheKey(opts({}), "p", "fpB"));+});++test("property-filter order does not change the key (canonical)", () => {+ const a = buildCacheKey(opts({ properties: new Map([["k", ["1", "2"]]]) }), "p", "fp");+ const b = buildCacheKey(opts({ properties: new Map([["k", ["2", "1"]]]) }), "p", "fp");+ expect(a).toBe(b);+});diff --git a/tests/core/search/query-plan.test.ts b/tests/core/search/query-plan.test.tsnew file mode 100644index 00000000..c31ffa04--- /dev/null+++ b/tests/core/search/query-plan.test.ts@@ -0,0 +1,73 @@+import { test, expect } from "bun:test";+import { buildQueryPlan, NEUTRAL_PROFILE } from "../../../src/core/search/query-plan.ts";++test("an empty query is neutral with the neutral weight profile", () => {+ const plan = buildQueryPlan(" ");+ expect(plan.intent).toBe("neutral");+ expect(plan.weightProfile).toEqual(NEUTRAL_PROFILE);+ expect(plan.expandedTerms).toEqual([]);+ expect(typeof plan.planHash).toBe("string");+ expect(plan.planHash.length).toBeGreaterThan(0);+});++test("a quoted phrase is classified exact and leans keyword over semantic", () => {+ const plan = buildQueryPlan(`the "exact phrase" here`);+ expect(plan.intent).toBe("exact");+ expect(plan.weightProfile.keywordMul).toBeGreaterThan(1);+ expect(plan.weightProfile.semanticMul).toBeLessThan(1);+});++test("a wildcard query is classified exact", () => {+ expect(buildQueryPlan("deploy*").intent).toBe("exact");+});++test("a wikilink-heavy query is classified entity and boosts the entity layer", () => {+ const plan = buildQueryPlan("[[Postgres]] [[Backups]]");+ expect(plan.intent).toBe("entity");+ expect(plan.weightProfile.entityMul).toBeGreaterThan(1);+});++test("a long low-entity natural query is classified broad and leans semantic", () => {+ const plan = buildQueryPlan("how do i configure the deployment process for staging here");+ expect(plan.intent).toBe("broad");+ expect(plan.weightProfile.semanticMul).toBeGreaterThanOrEqual(1);+});++test("a plain short query trips no rule and stays neutral", () => {+ const plan = buildQueryPlan("backup schedule");+ expect(plan.intent).toBe("neutral");+ expect(plan.weightProfile).toEqual(NEUTRAL_PROFILE);+});++test("classification is structural, not lexical: any-script tokens behave the same as latin", () => {+ // Two CJK tokens carry no latin words yet must classify by shape+ // (token count / entity structure), never by a language word list.+ const plan = buildQueryPlan("้ ็ฝฎ ้จ็ฝฒ");+ expect(["neutral", "broad", "entity"]).toContain(plan.intent);+ // bounded profile regardless of script+ for (const m of Object.values(plan.weightProfile)) {+ expect(m).toBeGreaterThanOrEqual(0.5);+ expect(m).toBeLessThanOrEqual(1.5);+ }+});++test("is deterministic: identical query yields a deep-equal plan", () => {+ expect(buildQueryPlan("deploy the [[Gateway]]")).toEqual(buildQueryPlan("deploy the [[Gateway]]"));+});++test("planHash collapses surrounding/internal whitespace but distinguishes content", () => {+ expect(buildQueryPlan(" backup schedule ").planHash).toBe(+ buildQueryPlan("backup schedule").planHash,+ );+ expect(buildQueryPlan("backup schedule").planHash).not.toBe(+ buildQueryPlan("restore schedule").planHash,+ );+});++test("planHash distinguishes queries whose case changes the structural intent", () => {+ // "Backup Schedule" is an entity run; lowercase is not - different+ // intent must yield a different cache key.+ expect(buildQueryPlan("Backup Schedule").planHash).not.toBe(+ buildQueryPlan("backup schedule").planHash,+ );+});diff --git a/tests/core/search/ranker.test.ts b/tests/core/search/ranker.test.tsindex 27bf5fe3..e57890cd 100644--- a/tests/core/search/ranker.test.ts+++ b/tests/core/search/ranker.test.ts@@ -1,5 +1,7 @@import { test, expect } from "bun:test";import { rankResults } from "../../../src/core/search/ranker.ts";+import { weibullDecay, DEFAULT_RECENCY } from "../../../src/core/search/recency.ts";+import { NEUTRAL_PROFILE } from "../../../src/core/search/query-plan.ts";import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";function hyd(chunkId: number, docId: number, mtime: number): HydratedChunk {@@ -175,7 +177,7 @@ test("tag_boost adds +0.01 when candidates share a tag", () => {expect(ranked[0]?.linkBoost).toBeCloseTo(0.01, 6);});-test("recency_boost is 0.05 for โค7d, 0.025 for โค30d, 0.01 for โค90d, 0 older", () => {+test("recency_boost follows the default Weibull curve (smooth, monotonic)", () => {const make = (mt: number) =>rankResults({@@ -189,10 +191,64 @@ test("recency_boost is 0.05 for โค7d, 0.025 for โค30d, 0.01 for โค90d, 0 olde)[0]!.recencyBoost;const baseSec = NOW / 1000;- expect(make(baseSec - 3 * 86400)).toBe(0.05);- expect(make(baseSec - 20 * 86400)).toBe(0.025);- expect(make(baseSec - 60 * 86400)).toBe(0.01);- expect(make(baseSec - 200 * 86400)).toBe(0);+ // The age the ranker derives from mtime is exact, so the boost equals+ // the pure curve evaluated at that age in days.+ for (const days of [3, 20, 60, 90]) {+ expect(make(baseSec - days * 86400)).toBeCloseTo(weibullDecay(days, DEFAULT_RECENCY), 6);+ }+ // Monotonic non-increasing with age.+ expect(make(baseSec - 3 * 86400)).toBeGreaterThan(make(baseSec - 60 * 86400));+ // Effectively stale content floors to exactly 0 (no recency layer).+ expect(make(baseSec - 365 * 86400)).toBe(0);+});++test("ranker honours a custom recency curve passed via options", () => {+ const custom = { shape: 0.8, scale: 120, amplitude: 0.05 };+ const boost = rankResults(+ {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [],+ hydrated: new Map([[1, hyd(1, 10, NOW / 1000 - 60 * 86400)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ },+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW, recency: custom },+ )[0]!.recencyBoost;+ // A larger scale than the default decays slower, so a 60-day-old hit+ // keeps a higher boost than it would under the default curve.+ expect(boost).toBeCloseTo(weibullDecay(60, custom), 6);+ expect(boost).toBeGreaterThan(weibullDecay(60, DEFAULT_RECENCY));+});++test("an absent weightProfile ranks identically to the neutral profile", () => {+ const inputs = {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [{ chunkId: 1, documentId: 10, distance: 0.3 }],+ hydrated: new Map([[1, hyd(1, 10, RECENT_MTIME)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ };+ const opts = { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW };+ const without = rankResults(inputs, opts)[0]!;+ const neutral = rankResults(inputs, { ...opts, weightProfile: NEUTRAL_PROFILE })[0]!;+ expect(neutral.score).toBe(without.score);+});++test("a keyword-leaning weightProfile raises a keyword hit's score", () => {+ const inputs = {+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],+ semantic: [],+ hydrated: new Map([[1, hyd(1, 10, OLD_MTIME)]]),+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ };+ const opts = { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW };+ const base = rankResults(inputs, opts)[0]!.score;+ const boosted = rankResults(inputs, {+ ...opts,+ weightProfile: { keywordMul: 1.3, semanticMul: 0.7, entityMul: 1, recencyMul: 1 },+ })[0]!.score;+ expect(boosted).toBeGreaterThan(base);});test("tie-break: equal final_score โ higher keywordScore wins; then mtime; then chunkId", () => {diff --git a/tests/core/search/recency.test.ts b/tests/core/search/recency.test.tsnew file mode 100644index 00000000..0bdebefd--- /dev/null+++ b/tests/core/search/recency.test.ts@@ -0,0 +1,54 @@+import { test, expect } from "bun:test";+import { weibullDecay, DEFAULT_RECENCY } from "../../../src/core/search/recency.ts";++test("age 0 yields the full amplitude", () => {+ expect(weibullDecay(0, DEFAULT_RECENCY)).toBeCloseTo(DEFAULT_RECENCY.amplitude, 10);+});++test("a future item (negative age) is clamped to the full amplitude", () => {+ expect(weibullDecay(-5, DEFAULT_RECENCY)).toBeCloseTo(DEFAULT_RECENCY.amplitude, 10);+});++test("boost is monotonically non-increasing as age grows", () => {+ const ages = [1, 3, 7, 20, 45, 90, 150];+ let prev = Infinity;+ for (const a of ages) {+ const b = weibullDecay(a, DEFAULT_RECENCY);+ expect(b).toBeLessThanOrEqual(prev);+ prev = b;+ }+});++test("boost stays within [0, amplitude] across the curve", () => {+ for (const a of [0, 1, 10, 50, 100, 500, 5000]) {+ const b = weibullDecay(a, DEFAULT_RECENCY);+ expect(b).toBeGreaterThanOrEqual(0);+ expect(b).toBeLessThanOrEqual(DEFAULT_RECENCY.amplitude + 1e-12);+ }+});++test("very old content decays to exactly zero via the epsilon floor", () => {+ expect(weibullDecay(365, DEFAULT_RECENCY)).toBe(0);+});++test("a larger scale decays more slowly (higher boost at the same age)", () => {+ const fast = weibullDecay(60, { ...DEFAULT_RECENCY, scale: 15 });+ const slow = weibullDecay(60, { ...DEFAULT_RECENCY, scale: 120 });+ expect(slow).toBeGreaterThan(fast);+});++test("non-positive shape, scale, or amplitude disable the boost (returns 0)", () => {+ expect(weibullDecay(3, { shape: 0, scale: 30, amplitude: 0.05 })).toBe(0);+ expect(weibullDecay(3, { shape: 0.8, scale: 0, amplitude: 0.05 })).toBe(0);+ expect(weibullDecay(3, { shape: 0.8, scale: 30, amplitude: 0 })).toBe(0);+});++test("amplitude above 1 is clamped into [0, 1]", () => {+ expect(weibullDecay(0, { shape: 0.8, scale: 30, amplitude: 5 })).toBe(1);+});++test("is deterministic for identical inputs", () => {+ const a = weibullDecay(12.5, DEFAULT_RECENCY);+ const b = weibullDecay(12.5, DEFAULT_RECENCY);+ expect(a).toBe(b);+});diff --git a/tests/core/search/schema-migration-v3.test.ts b/tests/core/search/schema-migration-v3.test.tsindex 63dad1b8..43fd2136 100644--- a/tests/core/search/schema-migration-v3.test.ts+++ b/tests/core/search/schema-migration-v3.test.ts@@ -38,17 +38,17 @@ function linksColumns(db: Database): string[] {test("fresh migration reaches v3 and links carries a relation column", () => {const db = new Database(dbPath);- expect(applyMigrations(db)).toBe(3);- expect(LATEST_SCHEMA_VERSION).toBe(3);+ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);+ // The relation column lands at v3 and persists through later versions.expect(linksColumns(db)).toContain("relation");db.close();});test("applyMigrations is idempotent at the latest version", () => {const db = new Database(dbPath);- expect(applyMigrations(db)).toBe(3);+ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);// Re-running must not throw and must not double-add the column.- expect(applyMigrations(db)).toBe(3);+ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);expect(linksColumns(db).filter((c) => c === "relation")).toHaveLength(1);db.close();});@@ -70,7 +70,7 @@ test("a v2 index upgrades to v3, adding relation and preserving link rows", () =expect(readSchemaVersion(db)).toBe(2);expect(linksColumns(db)).not.toContain("relation");- expect(applyMigrations(db)).toBe(3);+ expect(applyMigrations(db)).toBe(LATEST_SCHEMA_VERSION);expect(linksColumns(db)).toContain("relation");// The pre-existing link row survives, with a NULL relation.const row = dbdiff --git a/tests/core/search/schema-migration-v4.test.ts b/tests/core/search/schema-migration-v4.test.tsnew file mode 100644index 00000000..11053387--- /dev/null+++ b/tests/core/search/schema-migration-v4.test.ts@@ -0,0 +1,82 @@+/**+ * Schema migration v3 -> v4 (query cache). v4 adds a `query_cache` table+ * backing the persistent, corpus-generation-gated query result cache. A+ * fresh index lands at v4; an existing v3 index upgrades without losing+ * any prior data.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ applyMigrations,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+} from "../../../src/core/search/schema.ts";++let tmp: string;+let dbPath: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-mig-v4-"));+ dbPath = join(tmp, "test.sqlite");+});+afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function hasTable(db: Database, name: string): boolean {+ return (+ db+ .query<{ name: string }, [string]>(+ "SELECT name FROM sqlite_master WHERE type='table' AND name = ?",+ )+ .get(name) !== null+ );+}++test("fresh migration reaches v4 and creates query_cache", () => {+ const db = new Database(dbPath);+ expect(applyMigrations(db)).toBe(4);+ expect(LATEST_SCHEMA_VERSION).toBe(4);+ expect(hasTable(db, "query_cache")).toBe(true);+ db.close();+});++test("applyMigrations is idempotent at v4", () => {+ const db = new Database(dbPath);+ expect(applyMigrations(db)).toBe(4);+ expect(applyMigrations(db)).toBe(4);+ db.close();+});++test("a v3 index upgrades to v4 preserving prior data", () => {+ const db = new Database(dbPath);+ applyMigrations(db); // -> v4+ db.run(+ "INSERT INTO documents(id, path, title, content_hash, mtime, size, created_at, updated_at, indexed_at) " ++ "VALUES (1, 'a.md', 'A', 'h', 1, 1, 't', 't', 't')",+ );+ // Simulate an older v3 index: drop query_cache and rewind the version.+ db.run("DROP TABLE query_cache");+ db.run("UPDATE index_state SET value = '3' WHERE key = 'schema_version'");+ expect(readSchemaVersion(db)).toBe(3);+ expect(hasTable(db, "query_cache")).toBe(false);++ expect(applyMigrations(db)).toBe(4);+ expect(hasTable(db, "query_cache")).toBe(true);+ const row = db.query<{ c: number }, []>("SELECT count(*) AS c FROM documents").get();+ expect(row?.c).toBe(1);+ db.close();+});++test("a newer-than-latest index still raises SCHEMA_MISMATCH", () => {+ const db = new Database(dbPath);+ applyMigrations(db);+ db.run("UPDATE index_state SET value = '999' WHERE key = 'schema_version'");+ expect(() => applyMigrations(db)).toThrow(/newer than this binary/);+ db.close();+});diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.tsindex d0dc0da1..ba9000e8 100644--- a/tests/core/search/search.test.ts+++ b/tests/core/search/search.test.ts@@ -1,4 +1,5 @@import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";import { indexVault } from "../../../src/core/search/indexer.ts";import { search } from "../../../src/core/search/search.ts";@@ -54,6 +55,83 @@ async function seedKeyword() {return makeConfig({ vault, dbPath });}+function cacheRowCount(): number {+ const db = new Database(dbPath);+ try {+ return db.query<{ c: number }, []>("SELECT count(*) AS c FROM query_cache").get()?.c ?? 0;+ } finally {+ db.close();+ }+}++test("query cache (opt-in) stores a row and serves an identical request", async () => {+ await seedKeyword();+ const cfg = makeConfig({ vault, dbPath, cacheEnabled: true });+ await indexVault(cfg);+ expect(cacheRowCount()).toBe(0);++ const r1 = await search(cfg, { query: "fox", limit: 5 });+ expect(cacheRowCount()).toBeGreaterThan(0);++ const r2 = await search(cfg, { query: "fox", limit: 5 });+ expect(r2.results.map((x) => x.path)).toEqual(r1.results.map((x) => x.path));+});++test("query cache is invalidated by a reindex (corpus-generation bump)", async () => {+ await seedKeyword();+ const cfg = makeConfig({ vault, dbPath, cacheEnabled: true });+ await indexVault(cfg);++ const r1 = await search(cfg, { query: "fox", limit: 10 });+ expect(r1.results.map((x) => x.path)).not.toContain("Notes/fox2.md");++ // A new matching doc + reindex bumps the index revision -> generation+ // changes -> the cached row is not served and the result reflects it.+ writeMd(vault, "Notes/fox2.md", "# Fox2\n\nAnother fox appears here.");+ await indexVault(cfg);+ const r2 = await search(cfg, { query: "fox", limit: 10 });+ expect(r2.results.map((x) => x.path)).toContain("Notes/fox2.md");+});++test("no query_cache rows are written when the cache is disabled", async () => {+ await seedKeyword();+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ await search(cfg, { query: "fox", limit: 5 });+ expect(cacheRowCount()).toBe(0);+});++async function seedExpansion() {+ // d1 + d2 both match "alpha" and co-mention "beta" -> "beta" is a+ // co-occurrence expansion term. d3 has "beta" but not "alpha".+ writeMd(vault, "Notes/d1.md", "# D1\n\nalpha beta gamma notes.");+ writeMd(vault, "Notes/d2.md", "# D2\n\nalpha beta delta notes.");+ writeMd(vault, "Notes/d3.md", "# D3\n\nbeta epsilon unrelated content.");+}++test("synonym expansion (opt-in) broadens recall to a co-occurring-term doc", async () => {+ await seedExpansion();+ const off = makeConfig({ vault, dbPath });+ await indexVault(off);++ const baseline = await search(off, { query: "alpha", limit: 10 });+ expect(baseline.results.map((r) => r.path)).not.toContain("Notes/d3.md");++ const on = makeConfig({ vault, dbPath, synonymEnabled: true });+ const expanded = await search(on, { query: "alpha", limit: 10 });+ // "beta" co-occurs in d1+d2, so expansion pulls in d3 via the OR.+ expect(expanded.results.map((r) => r.path)).toContain("Notes/d3.md");+});++test("synonym expansion is suppressed for an exact-intent (quoted) query", async () => {+ await seedExpansion();+ const on = makeConfig({ vault, dbPath, synonymEnabled: true });+ await indexVault(on);+ const res = await search(on, { query: '"alpha"', limit: 10 });+ // Quoted -> exact intent -> no expansion -> d3 (no "alpha") stays out.+ expect(res.results.map((r) => r.path)).not.toContain("Notes/d3.md");+});+test("search returns keyword-only results when semantic is disabled", async () => {const cfg = await seedKeyword();await indexVault(cfg);diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsindex dec37d42..f944ee1d 100644--- a/tests/core/search/store.test.ts+++ b/tests/core/search/store.test.ts@@ -35,7 +35,20 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCokeywordWeight: 0.6,semanticWeight: 0.4,semantic,- recall: Object.freeze({ mmrLambda: 0.7, maxHops: 1, hopDecay: 0.5, maxExpansionPerHit: 3 }),+ recall: Object.freeze({+ mmrLambda: 0.7,+ maxHops: 1,+ hopDecay: 0.5,+ maxExpansionPerHit: 3,+ recencyShape: 0.8,+ recencyScale: 30,+ recencyAmplitude: 0.05,+ intentEnabled: true,+ synonymEnabled: false,+ synonymMaxTerms: 3,+ cacheEnabled: false,+ cacheTtlSeconds: 300,+ }),...overrides,});}diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsindex daf798fe..549964cb 100644--- a/tests/core/search/store.vec.test.ts+++ b/tests/core/search/store.vec.test.ts@@ -43,6 +43,14 @@ function semanticConfig(maxHops: 1,hopDecay: 0.5,maxExpansionPerHit: 3,+ recencyShape: 0.8,+ recencyScale: 30,+ recencyAmplitude: 0.05,+ intentEnabled: true,+ synonymEnabled: false,+ synonymMaxTerms: 3,+ cacheEnabled: false,+ cacheTtlSeconds: 300,}),...overrides,});diff --git a/tests/core/search/synonyms.test.ts b/tests/core/search/synonyms.test.tsnew file mode 100644index 00000000..12df77e4--- /dev/null+++ b/tests/core/search/synonyms.test.ts@@ -0,0 +1,88 @@+import { test, expect } from "bun:test";+import {+ tokenizeForExpansion,+ deriveExpansionTerms,+ DEFAULT_EXPANSION,+} from "../../../src/core/search/synonyms.ts";++test("tokenizeForExpansion splits on non-letter/digit and lowercases", () => {+ expect(tokenizeForExpansion("Deploy the Gateway-v2, now!")).toEqual([+ "deploy",+ "the",+ "gateway",+ "v2",+ "now",+ ]);+});++test("tokenizeForExpansion treats a CJK run as one structural token (no segmentation)", () => {+ // No language-specific segmenter: a continuous letter run is one token.+ expect(tokenizeForExpansion("้ ็ฝฎ้จ็ฝฒ staging")).toEqual(["้ ็ฝฎ้จ็ฝฒ", "staging"]);+});++test("no candidate texts yields no expansion (no-op)", () => {+ expect(deriveExpansionTerms(["backup"], [], DEFAULT_EXPANSION)).toEqual([]);+});++test("query tokens are never returned as expansion terms", () => {+ const out = deriveExpansionTerms(+ ["backup"],+ ["backup schedule schedule", "backup schedule cron"],+ { maxTerms: 5, minLength: 3, minDocFreq: 2 },+ );+ expect(out).not.toContain("backup");+});++test("docFreq counts distinct candidate docs, not raw term frequency", () => {+ // "schedule" appears many times but in a single doc -> docFreq 1.+ const out = deriveExpansionTerms(["backup"], ["backup schedule schedule schedule"], {+ maxTerms: 5,+ minLength: 3,+ minDocFreq: 2,+ });+ expect(out).toEqual([]);+});++test("terms below the doc-frequency threshold are dropped", () => {+ const out = deriveExpansionTerms(+ ["backup"],+ ["backup nightly", "backup nightly", "backup weekly"],+ { maxTerms: 5, minLength: 3, minDocFreq: 2 },+ );+ expect(out).toContain("nightly"); // 2 docs+ expect(out).not.toContain("weekly"); // 1 doc+});++test("short tokens below minLength are excluded", () => {+ const out = deriveExpansionTerms(["backup"], ["backup to s3", "backup to s3"], {+ maxTerms: 5,+ minLength: 3,+ minDocFreq: 2,+ });+ expect(out).not.toContain("to");+ expect(out).not.toContain("s3");+});++test("returns at most maxTerms, ranked by docFreq then alphabetically", () => {+ const docs = [+ "backup alpha bravo charlie",+ "backup alpha bravo",+ "backup alpha",+ "backup bravo charlie",+ ];+ // docFreq: alpha=3, bravo=3, charlie=2. maxTerms=2 -> top two by freq,+ // alpha before bravo on the alphabetical tie-break.+ const out = deriveExpansionTerms(["backup"], docs, {+ maxTerms: 2,+ minLength: 3,+ minDocFreq: 2,+ });+ expect(out).toEqual(["alpha", "bravo"]);+});++test("is deterministic for identical inputs", () => {+ const docs = ["backup alpha bravo", "backup alpha bravo"];+ expect(deriveExpansionTerms(["backup"], docs, DEFAULT_EXPANSION)).toEqual(+ deriveExpansionTerms(["backup"], docs, DEFAULT_EXPANSION),+ );+});diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex 73b728b2..85f7cb11 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -55,6 +55,12 @@ export function makeConfig(opts: {mmrLambda?: number;/** Link-graph traversal depth; defaults to 1. Pass 0 to disable. */maxHops?: number;+ /** Query-intent reweighting; defaults to true. */+ intentEnabled?: boolean;+ /** Synonym / query expansion; defaults to false (opt-in). */+ synonymEnabled?: boolean;+ /** Persistent query cache; defaults to false (opt-in). */+ cacheEnabled?: boolean;}): ResolvedSearchConfig {const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({enabled: false,@@ -89,6 +95,14 @@ export function makeConfig(opts: {maxHops: opts.maxHops ?? 1,hopDecay: 0.5,maxExpansionPerHit: 3,+ recencyShape: 0.8,+ recencyScale: 30,+ recencyAmplitude: 0.05,+ intentEnabled: opts.intentEnabled ?? true,+ synonymEnabled: opts.synonymEnabled ?? false,+ synonymMaxTerms: 3,+ cacheEnabled: opts.cacheEnabled ?? false,+ cacheTtlSeconds: 300,}),});}diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.tsindex ff095692..6d4ae632 100644--- a/tests/mcp/mcp.test.ts+++ b/tests/mcp/mcp.test.ts@@ -217,6 +217,8 @@ describe("tool listing", () => {"brain_daily_brief","brain_weekly_synthesis","brain_operator_summary",+ // Pre-compress injection pack (added in v0.20.0).+ "brain_pre_compress_pack",// Pay Memory (unchanged)."payment_memory_init","payment_receipt_append",@@ -466,7 +468,8 @@ describe("stdio loop", () => {// tools added in v0.17.0) + 8 Pay Memory + 1 Search// + brain_artifact_get (v0.18.0) = 40.// + brain_mcp_landscape (typed graph semantics) = 41.- expect(list.result.tools.length).toBe(41);+ // + brain_pre_compress_pack (v0.20.0) = 42.+ expect(list.result.tools.length).toBe(42);});test("returns parse error for invalid JSON", async () => {