Skip to content

Local-first ๐Ÿง  memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly ๐Ÿ˜ด dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: embedding-window chunk fit, sharded retirement links, decision-model provider chain and local thresholds, self-healing launchers (v1.93.0) (#274)

Serg Eroshe3064defd7f3c
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index d8666348..da433b5d 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.92.0",
+ "version": "1.93.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 71bb044d..7ac8a732 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.92.0",
+ "version": "1.93.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e446837e..6a437d8c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,25 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.93.0] - 2026-10-23
+
+Open Second Brain 1.93.0 closes the gap between what a configuration promises and what the model, the filesystem and the package manager actually do. The chunker now charges a long whitespace-free run such as a URL at a conservative token rate and splits it across chunks, so the declared budget bounds what the embedding model really receives. Retirement provenance lands in the log shard the device actually writes, so a fresh retirement no longer reads as a broken link. The decision model tries providers in a configured order with automatic fallback inside the existing budgets, and per-model enforce thresholds move out of the source tree into the installation, keyed by the model id the server reports and kept for every model ever tuned. The launchers repair a checkout that a Codex marketplace add or upgrade strips, so the documented install path starts the CLI. The release also closes all twenty-seven findings of the previous release's external review, among them two paths that let marker-private content leave the vault, a quadratic regex on the synchronous write path, a hold rule that withheld fitting chunks from embedding, and a ranking rule that let one entity match outrank higher-scoring results.
+
+### Fixed
+
+- **Chunks fit the model window when a page is one long URL.** The token estimator counted any whitespace-free run as one word, so a page holding a single long link produced a chunk declared at 4 tokens that the embedding model reads as more than 1,700 - past its 512-token input window, where the serving stack silently truncated it. A whitespace-free run of 16 characters or more is now charged one token per 3 characters, and the splitter cuts inside such a run, so the text is distributed across chunks that each fit the budget and every character stays indexed. Reducing `search_chunk_size` therefore works for link-heavy pages, and the chunker version moves to 3, which schedules a one-time re-chunk of the index.
+- **Retirement provenance names the log shard the device writes.** A merge retirement, a dream retirement and `brain reject` stamped `retired_by: [[Brain/log/<date>]]` while the log appender - for a vault with a configured `device_id` - writes `Brain/log/<date>.<device>.md`, so the doctor reported every fresh retirement as a broken link. All three now build the link through one helper that resolves the same shard id the appender resolves: sharded when a device id is configured, byte-identical to the historical shape when not. Historical records are untouched.
+- **A Codex marketplace add or upgrade leaves a launcher that starts.** The marketplace flow replaces the checkout wholesale and takes `node_modules` with it, so the first launcher command died importing `proper-lockfile` before `install-cli` could repair anything. The bun launchers - `o2b`, `o2b-mcp`, `o2b-aider`, `vault-log` and their Windows twins - now probe for the dependency tree with one file stat and run `bun install --frozen-lockfile` once when it is missing, refusing with the manual command named if that install fails; a warm checkout pays nothing. A cold-checkout regression drives the real entry point end to end.
+- **A page carrying an informal privacy marker is now withheld at every egress, not only at the remote read roots.** The decision-model gates - rerank, recall injection, label suggestions and advisories - and the shared knowledge-pack export treat a marker page exactly like a formally reserved one, a marker-private source's ingest summary is stamped reserved before its derived sections can copy its rows, and marker keys match case-insensitively, so an imported `Access: private` reserves like `access: private`. Existing search indexes re-measure the indexed visibility column of marker pages on upgrade (schema v16). The `update_note` guard refuses a marker-key write only when it would clear or forge the page's deny status - `access: read-only` and a `public` to `team` move are writable again - and `brain privacy-marker-restamp` now says that a stamped page also leaves default-scope local search and how to reach it again.
+- **Chunks are held out of embedding only when even a conservative lower-bound estimate puts them over the declared window.** The hold trigger was an average-rate estimate, so ordinary English chunks that fit the model's window were held and never embedded. Chunks merely estimated over the window are embedded again and reported in the undecided band, `search check` no longer recommends a vector backfill for chunks that are held, a declined embedding provider is reported as decided instead of not-configured, decline refusals name only the keys that recorded the decline, and `o2b doctor --readiness` reports an unreadable consent record as unknown and lists registered-provider and rerank credential variables by name. The decline this feature covers is the embedding provider's; rerank and decision-model endpoints keep their own settings' declines, recorded as follow-up work.
+- **Entity ranking can no longer let a single entity match outrank higher-scoring results.** The declared-over-derived rule now breaks only exact score ties, so a keyword-strong chunk stays first and a weak entity hit does not jump ahead. The usual spellings `A. Lovelace` and `Lovelace, Ada` resolve, short operator nicknames like `al` resolve (with the doctor naming any entry it still skips), a derived form that reaches two people is reported as ambiguous with its own remedy instead of silently expanding to both, and the nickname-derived index is built once per vault change, which turns the doctor's entity scan from seconds to milliseconds at five thousand people.
+- **The undated-quantity detector no longer stalls the write path and no longer fires on non-claims.** A comma-joined digit run used to backtrack quadratically - about forty-five seconds on a 256 KiB body, on the synchronous write-receipt path; the scan is now linear. Numbers inside URLs, markdown link destinations and wikilink targets, dates inside code regions, and dimension spellings like `3D` are no longer reported as undated claims, a date silences a page's quantities only where claims can stand, and the finding's offset now states that it is counted after frontmatter.
+- **The default OKF export no longer contradicts its own omissions.** The export's log no longer carries entries the agent wrote over the MCP channel or entries naming the pages the class split left out, and kept pages' structured links, relations and citations to a left-out page - in the OKF manifest, the graph export, the bank bundle and the knowledge pack - are dropped and counted, with `--include-prompt-derived` carrying all of it back. Source summaries ingested over MCP are now stamped `content_class: prompt-derived` themselves.
+
+### Added
+
+- **Ordered decision-model providers with fallback, and per-model thresholds in the installation.** `decision_model_providers` accepts an ordered list, with the single `decision_model_provider` kept as the one-entry form: the first provider that answers serves the call, a connection error or timeout falls through to the next entry within the same budget, and every `decision_model_call` record names the provider and model that actually answered, so reports never pool two models. `o2b decision-model check` shows the order and which entries answer. Per-model enforce thresholds move from the compiled `DECISION_THRESHOLD_PROFILES` table - now the hosted baseline family only - into `decision-thresholds.json` beside `config.yaml`, keyed by the model id the server reports rather than the provider preset, with the entries of every model ever tuned kept across switches and an optional per-vault override. A use with no stored thresholds for the active model runs as `shadow` and `check`/`report` say so explicitly, naming the tuning command; `o2b decision-model thresholds fit --use <use>` derives the threshold from the use's labelled report data against a false-action target and writes it for the answering model, and `thresholds list` / `thresholds set` round the flow out. Docs: `docs/decision-models.md` and `docs/decision-models/providers.md`.
+
## [1.92.0] - 2026-10-22
Open Second Brain 1.92.0 makes the write path and the egress path keep promises the rest of the system already made. Informal privacy markers in frontmatter - `access: private`, `confidentiality: secret`, `sensitivity: confidential` - now resolve to the reserved remote-deny token at every remote read root, and a preview-first verb stamps the formal token onto the pages that only ever carried the informal spelling. Chunks the declared embedding window cannot hold are held out of embedding under a persistent record instead of being truncated and indexed as something shorter, with the hold reported and released when its premise moves. People resolve through derived name forms - first initial plus surname, word order, and operator-supplied nicknames - while declared spellings keep precedence and ambiguity is reported rather than guessed. A fifth page-lint detector asks a count or an amount for its date, prompt-driven writes carry a server-stamped content class that shared exports honor by default, and a provider decline is a persisted decision honored before any network call. Review of the release closed four defects, among them an OKF bundle that left out its own reserved root files.
diff --git a/README.md b/README.md
index 18ad7d87..cee77854 100644
--- a/README.md
+++ b/README.md
@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi
## What is new
-Open Second Brain 1.92.0 makes the write path and the egress path keep the promises the rest of the system already made. Informal privacy markers in frontmatter now resolve to the reserved remote-deny token at every remote read root, and `o2b brain privacy-marker-restamp` previews and then stamps the formal reservation onto pages that only carried the informal spelling. A chunk over the embedding model's declared input window is held out of embedding under a record naming the model, the window and the signature, reported on `search status` and released when its premise moves, instead of being silently truncated. Person entities resolve through initials, word order and operator-supplied nicknames, with declared spellings always outranking derived ones and ambiguity reported rather than guessed. A fifth page-lint detector asks a count or an amount for its date, prompt-driven writes carry a server-stamped content class that shared exports leave out by default, and a provider decline is honored before any network call. Every release is described in the [CHANGELOG](CHANGELOG.md).
+Open Second Brain 1.93.0 makes the configured budget, the provenance and the install path hold at the edges where the model, the filesystem and the package manager move under them. Chunks charge long whitespace-free runs like URLs at a conservative token rate and split them across chunks, so the declared budget bounds what the embedding model actually receives. Preference retirement names the log shard the device actually writes, so a fresh retirement is never a broken link. The decision model accepts an ordered provider list with automatic fallback inside the existing budgets, records name the model that answered, and per-model enforce thresholds live in the installation - keyed by the model id the server reports, kept for every model ever tuned, and settable with `o2b decision-model thresholds fit`. The launchers re-install a dependency tree that a Codex marketplace add or upgrade removes, so the documented install path starts. The release also closes every finding of the previous release's external review, including two paths that let marker-private content leave the vault. Every release is described in the [CHANGELOG](CHANGELOG.md).
## Documentation
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index b6146d69..73fae6bb 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -701,6 +701,8 @@ o2b brain doctor gains the `symlink-escape` error (vault-internal s
Extracted facts pass a deterministic durability gate before persisting: structural signals only (temp paths, progress counters, run-id and timestamp shapes, measurement-token dominance, exit-status shapes), extendable with `durability.denylist` regexes; rejections log `durability-skip` events and are surfaced by count in the route result. Entity labels are sanitized (`**Foo**` becomes `Foo`) and structurally validated at every intake boundary, with `entities.label_denylist` as the only vocabulary source. Since v1.80.0 the structural gate also learns from the store: entity creation consults the derived lexicon (`Brain/entities/lexicon.json`, dataset-hashed over the vault's own pages - no word list ships with the binary) and two more verdicts feed the existing heal triangle (`entity-label-malformed` findings plus `o2b brain entity prune`): a multi-token all-caps label with no prior stored occurrence (`shouted-heading`) and a caseful label whose every token is a learned common word (`function-word-run`) are refused at creation, while code-shaped labels (containing `_`, `.`, `/`, a digit, or internal uppercase) bypass both case rules, and labels in caseless scripts admit on the structural gate alone. The verdicts are advisory after admission: existing records are heal candidates, never auto-removed. Rebuild the lexicon with `o2b brain entity lexicon rebuild` after corpus edits; a stale, torn or expired artifact reads as absent, so admission degrades to today's structural behavior and never mis-rejects. Feedback writes that closely resemble a confirmed same-scope preference return a conflict advisory (and log `write-conflict-advisory`) while the write proceeds. Embedding-side: `embedding_prefix_query`/`embedding_prefix_passage` (env twins, e5 preset defaults) control instruction prefixes, invalid vectors are rejected with `EMBEDDING_INVALID_VECTOR`, and quota exhaustion surfaces as non-retriable `EMBEDDING_QUOTA_EXHAUSTED` with Retry-After-aware backoff for plain rate limits.
+Beside the denylist, person lookups and query expansion read one more operator-supplied vocabulary, `Brain/entities/person-nicknames.json`: a JSON object mapping each nickname to the array of full person names it stands for (for example `{"bob": ["Robert Smith"]}`), with keys and values compared after the same normalization as entity names - case, whitespace and quote folding. Like the denylist it is operator data only: nothing is derived into it and no vocabulary ships with the binary. A missing file means no nickname data; a person also answers to the derived structural forms of its declared names (first-initial-plus-surname and reversed word order, plus the dotted-initial and `Last, First` spellings). A present but malformed file is reported by `o2b brain doctor` as `person-nicknames-unreadable` and nickname lookups degrade to the declared and structural forms; an entry the parser folds away - a nickname that normalizes to nothing, a value list with no usable name, a nickname equal to a name it stands for - surfaces as a `person-nicknames-entry-skipped` warning naming the entry's position.
+
### Belief lifecycle and decision memory (since v1.33.0)
```text
@@ -2288,6 +2290,7 @@ so there is no separate lane field that could drift from it:
| `semantic-embedding-abi-drift` | the index carries embeddings written by another build; `detail.fields` counts the contradicted ABI fields |
| `hybrid-degraded` | hybrid recall was asked for and the semantic lane did not run, so this answer is keyword-only |
| `hybrid-deadline-exceeded` | the composite hybrid path (embed, semantic top-k, rerank, second pass) outlived `search_hybrid_deadline_ms`, so the phases past the budget were cut; `detail.budgetMs` is the deadline, `detail.elapsedMs` the moment it fired |
+| `person-derived-ambiguity` | a person form in the query reached several people only through derivation (an initial, a word order or an operator nickname), so nothing was expanded for it rather than guess one person; `detail` maps that form to the competing person ids, dot-joined |
| `rank-cap-truncated-pool` | the rank cap truncated the candidate pool; `detail.cap` is the cap that bit |
| `rerank-provider-unavailable` | the configured cross-encoder reranker (remote endpoint or local model) could not answer, so the answer keeps the heuristic order; `detail.category` is one of `auth`, `quota`, `gone`, `rejected`, `transient`, `timeout`, `network`, `malformed`, `unclassified` |
| `rerank-model-sunset` | the configured rerank model's announced decommission date has passed, so no rerank request was sent and the answer keeps the heuristic order |
@@ -3020,24 +3023,44 @@ names it (`docs/decision-models/providers.md`).
## Decision model (optional)
```text
-o2b decision-model check Config state: enabled, provider, base URL, pinned model, the key
- variable's NAME and whether it is set (never the value), per-use
- modes, vault opt-out, cost gate and today's spend, processor terms,
- adapter and calibration, threshold profile (with one warning for
- uses whose enforce runs as shadow), choice option limit, licence
+o2b decision-model check Config state: enabled, the provider chain in order
+ (decision_model_providers), each entry's base URL, pinned model,
+ the key variable's NAME and whether it is set (never the value),
+ per-use modes, vault opt-out, cost gate and today's spend,
+ processor terms, adapter and calibration, threshold tuning state
+ (with one warning per use whose enforce runs as shadow, naming
+ the model and the tuning command), choice option limit, licence
note for open weights with a commercial-use restriction.
- --ping sends one request over a synthetic state (no vault content)
- and records it (use ping, counted toward the cost gate).
- Exit 1 only for an invalid config or a failed ping of an active
- provider; a missing key is exit 0 with a hint. --vault --config --json
+ --ping sends one request per configured provider over a
+ synthetic state (no vault content) and records each (use ping,
+ counted toward the cost gate). Exit 1 only for an invalid config
+ or when no configured provider answers the ping; a missing key
+ is exit 0 with a hint. --vault --config --json
o2b decision-model report decision_model_call records per use: calls, outcome mix, p50/p95
latency, input tokens, cost, rerank shadow agreement (top-1, top-5
- overlap, ordinary shadow records only). --since DATE --use USE
- --vault --config --json
+ overlap, ordinary shadow records only), and a shadow section
+ naming the uses that run as shadow for want of stored thresholds.
+ --since DATE --use USE --vault --config --json
+o2b decision-model eval run --use USE grades the preregistered per-use eval over the
+ configured adapter (real spend; a promote writes the machine-side
+ tuning mark); list prints cached decision_model_eval_grade
+ records, oldest first. --use --since --vault --config --json
+o2b decision-model thresholds fit --use USE derives the use's threshold from its labelled
+ report data against a false-action target (--target, default
+ 0.05) and writes it for the answering model in
+ decision-thresholds.json next to config.yaml; writes nothing
+ when the data cannot decide. list prints the store by model and
+ use with per-vault overrides and kept history. set --model ID
+ --use USE --value N [--name NAME] [--vault PATH] writes one by
+ hand; nothing is ever deleted. --config --json
```
-Off by default; active only when `decision_model_enabled: "true"` AND the
-environment variable named by `decision_model_env_key` is set. Config keys,
+Off by default; a provider entry is active only when `decision_model_enabled:
+"true"` AND the environment variable its key setting names is set. The
+entries answer in the configured order (`decision_model_providers`, with
+`decision_model_provider` as the one-entry form): a connection error or
+timeout falls through to the next entry within the same budget, and every
+record names the provider and model that actually answered. Config keys,
presets, privacy and the vault opt-out: [`docs/decision-models.md`](decision-models.md).
## Helpers
diff --git a/docs/decision-models.md b/docs/decision-models.md
index 959227b3..6cab6fdb 100644
--- a/docs/decision-models.md
+++ b/docs/decision-models.md
@@ -109,12 +109,16 @@ All keys live in the machine config file outside the vault. Each has an
| Key | Meaning | Default |
|---|---|---|
| `decision_model_enabled` | master switch | `false` |
-| `decision_model_provider` | preset name (table below) | none |
-| `decision_model_base_url` | overrides the preset's base URL | from preset |
+| `decision_model_provider` | one preset name (the one-element form) | none |
+| `decision_model_providers` | ordered preset names, tried in order (see "Ordered providers" below) | none |
+| `decision_model_providers_<preset>_base_url` | base URL override for one chain entry | from preset |
+| `decision_model_providers_<preset>_id` | pinned model id override for one chain entry | from preset |
+| `decision_model_providers_<preset>_env_key` | key variable name override for one chain entry | from preset |
+| `decision_model_base_url` | overrides the preset's base URL (the first provider's; a chain entry overrides with its own key above) | from preset |
| `decision_model_id` | pinned model id; overrides the preset. A moving alias such as `jev-latest` is refused. It plays the part `search_rerank_model` and `embedding_model` play for those features; the key is named `_id` because the value must pin a version | from preset |
| `decision_model_env_key` | **name** of the env var holding the key (`A-Z`, `0-9`, `_`); anything else, such as the key itself, makes the config invalid and is never printed | from preset |
| `decision_model_allow_insecure_http` | allow plain `http://` to a non-loopback host; config or env only | `false` |
-| `decision_model_timeout_ms` | per-request timeout for MCP and CLI paths | `3000` |
+| `decision_model_timeout_ms` | per-request timeout for MCP and CLI paths, shared by the whole provider chain | `3000` |
| `decision_model_hook_budget_ms` | sub-budget for the `recall_inject` hook use: the request gets `min(this, remaining retrieval budget)` and never extends the hook's 2,500 ms total (see [the recall-inject hook filter](decision-models/recall-inject.md)) | `700` |
| `decision_model_max_state_tokens` | client-side state cap, clamped to the preset maximum | preset, at most `32000` |
| `decision_model_uses` | `use:mode` comma list, mode `off`, `shadow` or `enforce` | every use `off` |
@@ -122,7 +126,7 @@ All keys live in the machine config file outside the vault. Each has an
| `decision_model_input_price_usd_per_mtok` | price for routes that do not report cost | from preset |
| `decision_model_output_price_usd_per_mtok` | output price, read only for the chat-shaped routes (`llm-emulation`, `converse`), whose cost is `unknown` unless both prices are set | none |
| `decision_model_allow_uncalibrated` | allow an uncalibrated provider in `enforce` | `false` |
-| `decision_model_threshold_profile` | the model family a `compatible` server serves (`jev-1.13`, `laya`, `openjev`, `llm-emulation`, `converse`), so its uses may enforce; ignored for every other preset | from preset (none for `compatible`) |
+| `decision_model_threshold_profile` | the built-in model family a `compatible` server serves (`jev-1.13` is the only one), so its uses may enforce on the shipped thresholds; every other model is tuned through the local thresholds store instead (see "Thresholds") | from preset |
Uses: `rerank`, `answerable`, `skills`, `extract_prefilter`, `dedup`,
`tension`, `labels`, `recall_inject`. Every use is implemented; each one is
@@ -157,12 +161,59 @@ licence and data handling in [providers](decision-models/providers.md)):
`decision_model_base_url` must not carry `user:password@` credentials: the key
travels in a header, and a URL with credentials is an invalid config.
-Each preset names a threshold profile. Every threshold is tuned against the
-Jev 1.13 family (`jev-1.13`), which the hosted presets use; on any other
-profile, and on a `compatible` server without
-`decision_model_threshold_profile`, a use set to `enforce` runs as `shadow`,
-and `check` says so. Measure a `compatible` model with the eval gate before
-naming its family.
+## Ordered providers
+
+`decision_model_providers` names an ordered chain, tried per call from the
+front:
+
+```yaml
+decision_model_enabled: "true"
+decision_model_providers: compatible,typesafe
+decision_model_base_url: http://127.0.0.1:8080
+decision_model_id: my-local-decision-model
+decision_model_env_key: LOCAL_DECISION_KEY
+decision_model_uses: "rerank:shadow"
+```
+
+With the server up, every call goes local; with the server down, the call
+falls through to the hosted preset with no failed hook. The rules:
+
+- The first entry that is **configured** (its key variable set, or its
+ preset needs none) **and answers** wins. An entry without its key is
+ skipped, and the feature is off only when no entry is configured.
+- A connection error or timeout falls through to the next entry. Any other
+ failure (an HTTP status, an invalid reply) is the endpoint's answer about
+ this request and ends the chain; retrying it elsewhere would multiply the
+ same refusal.
+- The whole chain shares the one budget: `decision_model_timeout_ms`, or
+ `decision_model_hook_budget_ms` for the hook use. Each attempt gets an
+ even slice of what remains, so a dead first server can never stretch a
+ call past the budget.
+- Every attempt writes its own `decision_model_call` record naming the
+ provider and the model that attempted or answered it, so reports never
+ pool two models.
+- `enforce` is held to thresholds per entry (see "Thresholds"): a use may
+ enforce on the answering model only when that model is tuned, whatever
+ the other entries can do.
+
+`decision_model_provider` stays as the one-element form and keeps working
+unchanged; when both keys are set the list decides the order and a note in
+`check` says so. Each entry runs on its preset's defaults. The first entry
+additionally reads the global endpoint keys (`decision_model_base_url`,
+`decision_model_id`, `decision_model_env_key`); any entry can override its
+own endpoint with `decision_model_providers_<preset>_base_url`,
+`_id` and `_env_key` (config only, no environment override):
+
+```yaml
+decision_model_providers: laya,typesafe
+decision_model_providers_laya_base_url: http://127.0.0.1:8100
+```
+
+`o2b decision-model check` lists the order and each entry's key state;
+`check --ping` sends one synthetic request per configured entry and shows
+which answer. It exits 1 only when no configured entry answers - a dead
+first entry with a live second is exactly the situation the chain exists
+for.
Example, a hosted route with the reranker in shadow:
@@ -190,6 +241,87 @@ decision_model_max_state_tokens: "4000"
decision_model_uses: "rerank:shadow"
```
+## Thresholds: the local store
+
+Thresholds are a property of the installation, not of the code. Every
+threshold Open Second Brain ships was tuned against the hosted Jev 1.13
+family, and those built-in values stay as the fallback for that family
+only. Every other model takes its thresholds from a local store,
+`decision-thresholds.json`, next to `config.yaml` in the user config
+directory:
+
+```json
+{
+ "version": 1,
+ "models": {
+ "my-local-decision-model": {
+ "dedup": {
+ "values": { "low_priority_min": 0.9 },
+ "source": "fit",
+ "target": 0.05,
+ "samples": 96,
+ "updated_at": "2026-10-22T09:00:00.000Z",
+ "history": [{ "value": 0.8, "source": "set", "updated_at": "2026-10-21T09:00:00.000Z" }]
+ }
+ }
+ },
+ "vaults": {}
+}
+```
+
+- Entries are keyed by `(model id, use)`, and the model id is the one the
+ server reports, never the provider preset: a self-hosted server that
+ swaps models inherits nothing, and the store file can tell the two
+ apart.
+- A use may run `enforce` on a model exactly when the store carries an
+ entry for `(model, use)` - or when the model answers on the built-in
+ `jev-1.13` baseline, or carries a valid machine-side eval mark. A model
+ with no entry for a use runs that use as `shadow`, and `check` and
+ `report` say so explicitly and name the tuning command. Before this
+ store existed, such a use simply ran as shadow with no way to tune it
+ short of a release.
+- Entries are never deleted. Switching to another model leaves every
+ existing entry in place, so switching back restores `enforce` without
+ re-tuning; rewriting an entry keeps the previous value in the entry's
+ `history`.
+- A `vaults` section overrides the global entry for one vault, keyed by
+ the vault's absolute path.
+- The threshold `values` are named per use (`low_priority_min` for `dedup`
+ and `tension`, `drop_below` for `extract_prefilter`, `suggest_min` for
+ `labels`, and so on; `o2b decision-model thresholds set` names the
+ valid ones). They are what a later tune converge against; the enforce
+ gate reads the entry's presence.
+
+### Tuning a model
+
+```sh
+o2b decision-model thresholds fit --use dedup
+```
+
+`fit` reads the use's labelled report data from the vault's records -
+`dedup` and `tension` verdicts joined to the operator's later status,
+`label` suggestions joined to later acceptance, `extract_prefilter` plans
+joined to later citations - sweeps the threshold, and keeps the value that
+acts on the most samples while holding the share of false actions at or
+under `--target` (default 0.05). It writes the entry for the model that
+answered, refuses to pool several models into one threshold, and writes
+nothing when too few samples decide or no threshold meets the target -
+the refusal names the closest the data came. Only the four uses with
+labelled reports fit: `dedup`, `tension`, `labels`, `extract_prefilter`.
+
+`thresholds set` writes a value by hand for a model and use; `thresholds
+list` prints everything stored, per model and per vault, with the kept
+history. All three are local operations: no request is sent.
+
+```sh
+o2b decision-model thresholds list
+o2b decision-model thresholds set --model my-local-decision-model --use dedup --value 0.9
+```
+
+The other tuning path stays: the preregistered eval gate (below) measures
+a use and, on a promote verdict, writes a machine-side mark that arms
+`enforce` for the measured endpoint.
+
## Uses
Each use has its own mode in `decision_model_uses`, its own failure policy
@@ -212,8 +344,9 @@ blocks a write or an answer, and no use writes to the vault.
In `shadow` every use sends and records its request and returns today's
result; for `labels` the suggestions are null and for `dedup` and `tension`
-the listings are unchanged. `enforce` on a threshold profile without tuned
-thresholds for the use runs as `shadow` (see [providers](decision-models/providers.md#threshold-profiles)).
+the listings are unchanged. `enforce` on a model without tuned thresholds
+for the use runs as `shadow` (see "Thresholds" above and
+[providers](decision-models/providers.md#thresholds)).
Moving a use to `enforce` needs evidence from that use's report: a gain
against the deterministic baseline without a regression, and for
@@ -407,16 +540,19 @@ timeout. A reply body larger than 1 MiB is an `invalid_reply`.
## Operator commands
-- `o2b decision-model check [--ping] [--json]`: enabled state, provider, base
- URL, pinned model, the key variable's name and whether it is set (never the
- value), per-use modes as configured (with a warning when the feature is
- disabled, since every use then runs `off`), vault opt-out and any ignored
- `decision_model:` field in the vault's `_brain.yaml`, cost gate and today's
- spend, the processor. The hint names only what is missing. `--ping` sends
- one request over a synthetic state with no vault content, prints the
- answering model and latency, and records the request. Exit 1 only for a
- broken configured provider (invalid config, or a failed ping of an active
- one).
+- `o2b decision-model check [--ping] [--json]`: enabled state, the provider
+ chain in order (with `--ping`, each configured entry's live answer), base
+ URL, pinned model, the key variable's name and whether it is set (never
+ the value), per-use modes as configured (with a warning when the feature
+ is disabled, since every use then runs `off`), vault opt-out and any
+ ignored `decision_model:` field in the vault's `_brain.yaml`, cost gate
+ and today's spend, the processor. An `enforce` use that runs as `shadow`
+ for want of tuned thresholds is warned about once, naming the model and
+ the tuning command. The hint names only what is missing. `--ping` sends
+ one request per configured provider over a synthetic state with no vault
+ content, prints the answering model and latency per entry, and records
+ each request. Exit 1 only for a broken configured provider (invalid
+ config, or no configured entry answering the ping).
- `o2b decision-model report [--since <date>] [--use <use>] [--json]`: per
use, calls, outcome mix, p50/p95 latency, input tokens, cost, and for the
reranker the shadow agreement (top-1 agreement and top-5 overlap between the
@@ -426,7 +562,19 @@ timeout. A reply body larger than 1 MiB is an `invalid_reply`.
shares for `extract_prefilter`, verdict bands against the operator's later
decisions for `dedup` and `tension`, acceptance for `labels`, level against
advisory band and outcomes for `answerable`, and added latency and hook
- timeouts for `recall_inject`. Each use's page describes its metric.
+ timeouts for `recall_inject`. Each use's page describes its metric. When a
+ use configured `enforce` runs as `shadow` for want of tuned thresholds,
+ the report says so and names the tuning command.
+- `o2b decision-model thresholds fit --use <use> [--target <n>] [--json]`:
+ derive the use's threshold from its labelled report data against the
+ false-action target and write it for the answering model (see
+ "Thresholds"). Writes nothing and says why when the data cannot decide.
+- `o2b decision-model thresholds list [--json]`: the stored thresholds by
+ model and use, with per-vault overrides and kept history.
+- `o2b decision-model thresholds set --model <id> --use <use> --value <n>
+ [--name <name>] [--vault <path>] [--json]`: write one threshold by hand;
+ a `--vault` scopes it to that vault as an override. Nothing is ever
+ deleted by these commands.
- `o2b doctor --readiness` has a `decision_model` line for every install:
`skipped` when off, without a key, or opted out (with how to proceed),
`fail` for an invalid config, `pass` when active. It also names an ignored
diff --git a/docs/decision-models/providers.md b/docs/decision-models/providers.md
index ea91ccb7..0cd7329e 100644
--- a/docs/decision-models/providers.md
+++ b/docs/decision-models/providers.md
@@ -2,12 +2,16 @@
Every route Open Second Brain can send a decision request to, how each one is
configured, and what each one means for your data. The feature itself, its
-config keys and the privacy rules that apply to every route are in
+config keys, the provider order, the local thresholds store and the privacy
+rules that apply to every route are in
[`docs/decision-models.md`](../decision-models.md).
Nothing here is used unless you set `decision_model_enabled: "true"` and name
-one of these presets in `decision_model_provider`. No preset is ever picked
-for you, and no route is ever a fallback for another one.
+one of these presets in `decision_model_provider` (or several, in order, in
+`decision_model_providers`). No preset is ever picked for you. With a single
+`decision_model_provider` no route is ever a fallback for another one; an
+explicit `decision_model_providers` list is the one way to name an order,
+and then the first entry that is configured and answers wins.
## Presets
@@ -34,11 +38,11 @@ Other per-route limits:
| Preset | Key | Most `choice` options | Threshold profile | Input price (USD / M tokens) |
| --------------------------------------------------------------------- | -------------------- | --------------------- | --------------------------------------------------------- | -------------------------------------- |
| `typesafe`, `openrouter`, `vercel`, `vercel-evaluate`, `opencode-zen` | required | 255 | `jev-1.13` | 0.042 |
-| `laya` | optional on loopback | 100 | `laya` | 0 |
-| `openjev` | optional on loopback | 52 | `openjev` | 0 |
-| `compatible` | required | 255 | none, unless `decision_model_threshold_profile` names one | unknown |
-| `llm-emulation` | required | 255 | `llm-emulation` | unknown unless both price keys are set |
-| `converse` | required | 255 | `converse` | unknown unless both price keys are set |
+| `laya` | optional on loopback | 100 | the server's model id, via the local store | 0 |
+| `openjev` | optional on loopback | 52 | the server's model id, via the local store | 0 |
+| `compatible` | required | 255 | the server's model id, via the local store, or `jev-1.13` if named | unknown |
+| `llm-emulation` | required | 255 | the endpoint's model id, via the local store | unknown unless both price keys are set |
+| `converse` | required | 255 | the gateway's model id, via the local store | unknown unless both price keys are set |
A `choice` question above the route's option limit is never sent: a use that
can split it does, and otherwise that request degrades with `budget`, like an
@@ -101,8 +105,10 @@ route. Open Second Brain does not download, install or start these servers.
permission, and `o2b decision-model check` prints that note.
A self-hosted model has its own calibration: its probabilities are not Jev's.
-Each has its own threshold profile, and no use is tuned for either yet, so an
-`enforce` setting runs as `shadow` there (see "Threshold profiles").
+Its thresholds come from the local store, keyed by the model id the server
+reports; until `o2b decision-model thresholds fit` (or `set`) writes an
+entry for a use, an `enforce` setting runs as `shadow` there (see
+"Thresholds").
```yaml
decision_model_enabled: "true"
@@ -120,15 +126,17 @@ key is required again.
Any other server that speaks `POST /v1/systemone`. The base URL, the model id
and the key variable must be set. Open Second Brain cannot know which model
-family answers, so a `compatible` route has no threshold profile and never
-enforces, unless you name the family it serves:
+family answers, so a `compatible` route has no built-in thresholds and its
+uses run as `shadow` until the store carries entries for its model id
+(`o2b decision-model thresholds fit --use <use>`); if the server serves the
+hosted Jev 1.13 family, naming it arms the shipped thresholds:
```yaml
decision_model_provider: compatible
decision_model_base_url: https://decisions.example.com
decision_model_id: jev-1.13.0
decision_model_env_key: EXAMPLE_DECISION_KEY
-decision_model_threshold_profile: jev-1.13
+decision_model_threshold_profile: jev-1.13 # the one built-in family
```
## `llm-emulation` (uncalibrated)
@@ -156,8 +164,8 @@ compare routes in `shadow` when only a chat endpoint is available.
`calibrated: false`, so reports keep these runs apart.
- `enforce` is refused as a config error unless
`decision_model_allow_uncalibrated: "true"` is set; `shadow` is allowed.
- Even with the override, no use has tuned thresholds on the `llm-emulation`
- profile yet, so `enforce` runs as `shadow`.
+ Even with the override, the model needs tuned thresholds in the local
+ store before a use enforces; until then `enforce` runs as `shadow`.
- A chat model bills output tokens and costs far more than a decision model.
The cost is recorded as `unknown` unless both
`decision_model_input_price_usd_per_mtok` and
@@ -199,9 +207,10 @@ validated item by item by the same rules as every route.
fails that validation is invalid while the rest of the batch stands.
Records carry `calibrated: false`, and an `enforce` setting additionally
needs `decision_model_allow_uncalibrated: "true"`.
-- It carries its own threshold profile, `converse`, and no use is tuned for
- it yet: an `enforce` setting runs as `shadow` until the profile is tuned
- from this route's own shadow data. It never borrows `jev-1.13`.
+- Its thresholds are its own: an `enforce` setting runs as `shadow` until
+ the store carries entries for the gateway model's id, tuned from this
+ route's own shadow data (`o2b decision-model thresholds fit`). It never
+ borrows `jev-1.13`.
- The state is vault content travelling inside a generated-text request, so
it is fenced as untrusted content and passes the same redaction as every
route (registry id `decision-model-converse`).
@@ -224,21 +233,37 @@ decision_model_env_key: GATEWAY_DECISION_KEY # the variable's name, not the key
decision_model_uses: "rerank:shadow"
```
-## Threshold profiles
+## Thresholds
Probabilities from different model families are not comparable, so every
-threshold a use applies is held to the family it was tuned against. Each
-preset names a threshold profile:
-
-- `jev-1.13`: every threshold Open Second Brain ships was set against this
- family, so every use may enforce on it;
-- `laya`, `openjev`, `llm-emulation`, `converse`: no use is tuned yet;
-- none (`compatible` without `decision_model_threshold_profile`).
-
-A use set to `enforce` on a profile without tuned thresholds for it runs as
+threshold a use applies is held to the model it was tuned against. What is
+tuned comes from three places, in this order:
+
+1. The built-in baseline: the hosted Jev 1.13 family (`jev-1.13`). Every
+ threshold Open Second Brain ships was set against this family, so every
+ use may enforce on it. It is the only profile the code carries.
+2. The local thresholds store (`decision-thresholds.json` next to
+ `config.yaml`), keyed by `(model id, use)` - the model id the server
+ reports, never the preset. An entry for a pair lets that use enforce on
+ that model; a `vaults` section overrides an entry for one vault; entries
+ are never deleted, so switching models and switching back needs no
+ re-tuning. `o2b decision-model thresholds fit` derives an entry from the
+ use's labelled report data against a false-action target, and
+ `thresholds set` writes one by hand.
+3. A valid machine-side eval mark (below), for the endpoint that earned it.
+
+A use set to `enforce` whose model has no tuned thresholds for it runs as
`shadow`: the request is sent and recorded, and the deterministic result is
returned. Nothing on a hot path warns about it; `o2b decision-model check`
-prints one warning naming the uses concerned.
+prints one warning naming the model and the tuning command, and
+`o2b decision-model report` says the same. On an ordered chain the hold is
+per entry: the primary model and the fallback model are each judged by
+their own tuning, and the record names whichever answered.
+
+The `decision_model_threshold_profile` key names the built-in family a
+`compatible` server serves; `jev-1.13` is the only valid value. Every other
+model is tuned through the store, keyed by model id, so the key has nothing
+to say about them.
## Measuring a profile: the preregistered eval
@@ -277,6 +302,9 @@ fail-closed direction. Deleting the mark does the same; nothing breaks.
`o2b decision-model check` names the measured state (untuned, baseline,
measured under a spec hash, or measured under a spec hash on another
endpoint - provider, model or base-url mismatch) and lists the cached
-grades, and the doctor readiness line carries the same summary. Tuning a
-profile by hand-editing the source table stays possible and still wins
-over a mark; the mark only ever carries pairs the table leaves untuned.
+grades, and the doctor readiness line carries the same summary. The local
+thresholds store ("Thresholds" above) is the other tuning path: it needs
+no eval run and no mark, only the use's labelled report data, and it keys
+on the model id instead of the profile. Tuning a profile by hand-editing
+the source table stays possible and still wins over a mark; the mark only
+ever carries pairs the table leaves untuned.
diff --git a/docs/updating.md b/docs/updating.md
index 1d7796df..2dc5868b 100644
--- a/docs/updating.md
+++ b/docs/updating.md
@@ -1053,6 +1053,15 @@ codex plugin add open-second-brain@open-second-brain # re-stage local marketpl
hermes plugins update open-second-brain
```
+A Codex marketplace add or upgrade replaces the clone wholesale, so the
+clone arrives without `node_modules`. No step follows: the first `o2b`
+call notices the missing dependency tree and runs
+`bun install --frozen-lockfile` inside the clone once, announced by a
+one-line notice on stderr. If that install cannot run - no network, or a
+clone the launcher cannot write - the launcher exits nonzero and names the
+manual command; running `bun install --frozen-lockfile` inside the clone
+yourself is the fallback. See [`install/codex.md`](../install/codex.md).
+
You do **not** need to delete or re-create any `~/.local/bin` symlink by hand.
The hooks resolve the current version on their own, and the global
`o2b`/`o2b-hook`/`vault-log` symlinks self-heal on the next session start. If
@@ -1168,6 +1177,10 @@ These properties are locked by tests; do not weaken them:
- `tests/hooks/o2b-hook.test.ts` โ€” fail-soft + `$CLAUDE_PLUGIN_ROOT`-first resolution.
- `tests/hooks/hooks-json-shape.test.ts` โ€” every command is version-current, has a PATH fallback, and ends with `exit 0`.
- `tests/cli/install-cli.test.ts` โ€” idempotent reclaim and conservative self-heal.
+- `tests/scripts/launcher-bootstrap.test.ts` โ€” the launchers re-install the
+ dependency tree a marketplace replacement removed, degrade to a named
+ manual command when the install cannot run, and add nothing when the tree
+ is present.
- `tests/core/search/self-heal.test.ts` โ€” search rebuilds a stale/missing index on read.
- `tests/core/maintenance/ensure-current.test.ts` โ€” state-driven migration, idempotent, never throws.
- `tests/core/maintenance/self-heal-reindex.test.ts` โ€” the herd is thinned against the writer lock, and a detached rebuild's outcome is readable.
diff --git a/install/codex.md b/install/codex.md
index d954d9c6..c4e80031 100644
--- a/install/codex.md
+++ b/install/codex.md
@@ -33,6 +33,16 @@ O2B_SCRIPT="$(find "${CODEX_HOME:-$HOME/.codex}/.tmp/marketplaces" -path '*open-
"$O2B_SCRIPT" install-cli
```
+The clone arrives without `node_modules`, and the launcher heals that
+itself: the first time it runs, it notices the missing dependency tree and
+executes `bun install --frozen-lockfile` inside the clone once, announced
+by a one-line notice on stderr. Every later run finds the tree present and
+boots straight into the CLI. If that install cannot run - no network, or a
+clone the launcher cannot write - the launcher exits nonzero and names the
+manual command instead of dying with a module-not-found trace. Running
+`bun install --frozen-lockfile` inside the clone yourself repairs the same
+state.
+
## 3. Initialize the vault
```bash
@@ -127,6 +137,12 @@ what was added; `unprotect` removes the same.
codex plugin marketplace upgrade open-second-brain
```
+An upgrade replaces the clone, so its `node_modules` is gone afterwards.
+No extra step is needed: the next launcher run notices the missing
+dependency tree and reinstalls it, exactly as in step 2 (the same one-line
+notice on stderr, and the same manual `bun install --frozen-lockfile`
+fallback if the automatic install cannot run).
+
## Uninstall
```bash
diff --git a/link-ratchet.json b/link-ratchet.json
index 5228196e..ab4ec38e 100644
--- a/link-ratchet.json
+++ b/link-ratchet.json
@@ -4,7 +4,7 @@
"subjects": [
{
"path": "templates/brain-starter",
- "dangling": 22
+ "dangling": 23
}
]
}
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 3c654445..5e554bae 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.92.0",
+ "version": "1.93.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/openclaw/index.js b/openclaw/index.js
index 20a0cdbb..b816a609 100644
--- a/openclaw/index.js
+++ b/openclaw/index.js
@@ -5844,8 +5844,11 @@ function declaredValues(raw) {
return [];
return raw.filter((v) => typeof v === "string");
}
+var PRIVACY_MARKER_KEY_TOKENS = Object.freeze(new Set(PRIVACY_MARKER_KEYS.map((key) => normToken(key))));
function hasPrivacyDenyMarker(meta) {
- for (const key of PRIVACY_MARKER_KEYS) {
+ for (const key of Object.keys(meta)) {
+ if (!PRIVACY_MARKER_KEY_TOKENS.has(normToken(key)))
+ continue;
for (const value of declaredValues(meta[key])) {
if (PRIVACY_DENY_MARKER_VALUES.includes(normToken(value)))
return true;
diff --git a/package.json b/package.json
index 98a1798e..4e6eed70 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.92.0",
+ "version": "1.93.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index 99779597..3aaa6dce 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.92.0"
+version: "1.93.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index 61dd8e99..4bf1ae35 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.92.0",
+ "version": "1.93.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/codex/README.md b/plugins/codex/README.md
index 825f2f82..a0e31e11 100644
--- a/plugins/codex/README.md
+++ b/plugins/codex/README.md
@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite
## What is new
-Open Second Brain 1.92.0 makes the write path and the egress path keep the promises the rest of the system already made. Informal privacy markers in frontmatter now resolve to the reserved remote-deny token at every remote read root, and `o2b brain privacy-marker-restamp` previews and then stamps the formal reservation onto pages that only carried the informal spelling. A chunk over the embedding model's declared input window is held out of embedding under a record naming the model, the window and the signature, reported on `search status` and released when its premise moves, instead of being silently truncated. Person entities resolve through initials, word order and operator-supplied nicknames, with declared spellings always outranking derived ones and ambiguity reported rather than guessed. A fifth page-lint detector asks a count or an amount for its date, prompt-driven writes carry a server-stamped content class that shared exports leave out by default, and a provider decline is honored before any network call. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
+Open Second Brain 1.93.0 makes the configured budget, the provenance and the install path hold at the edges where the model, the filesystem and the package manager move under them. Chunks charge long whitespace-free runs like URLs at a conservative token rate and split them across chunks, so the declared budget bounds what the embedding model actually receives. Preference retirement names the log shard the device actually writes, so a fresh retirement is never a broken link. The decision model accepts an ordered provider list with automatic fallback inside the existing budgets, records name the model that answered, and per-model enforce thresholds live in the installation - keyed by the model id the server reports, kept for every model ever tuned, and settable with `o2b decision-model thresholds fit`. The launchers re-install a dependency tree that a Codex marketplace add or upgrade removes, so the documented install path starts. The release also closes every finding of the previous release's external review, including two paths that let marker-private content leave the vault. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
## Documentation
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 99779597..3aaa6dce 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.92.0"
+version: "1.93.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index a48e2ae3..b8ee3731 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta"
# `hermes_agent.memory_providers` group, not a CLI script.
[project]
name = "open-second-brain"
-version = "1.92.0"
+version = "1.93.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/scripts/_bun-precheck.sh b/scripts/_bun-precheck.sh
index 870fea6d..d8553fee 100755
--- a/scripts/_bun-precheck.sh
+++ b/scripts/_bun-precheck.sh
@@ -1,11 +1,15 @@
#!/usr/bin/env bash
-# Sourced by scripts/o2b and scripts/vault-log to verify Bun is present
-# and meets the declared minimum version (>=1.1.0, mirrored against
-# `engines.bun` in package.json).
+# Sourced by scripts/o2b, scripts/o2b-mcp, scripts/o2b-aider, scripts/vault-log
+# and scripts/test to verify Bun is present and meets the declared minimum
+# version (>=1.1.0, mirrored against `engines.bun` in package.json), and to
+# bootstrap the checkout's dependency tree when a marketplace replacement
+# removed it (see the block at the bottom).
#
# Side effects: writes a clear error to stderr and exits the calling
# script if Bun is missing, unparseable, or older than the minimum.
-# No-op on success.
+# If the checkout's node_modules is missing, runs one
+# `bun install --frozen-lockfile` before returning. No-op on success
+# when the dependency tree is present.
# shellcheck shell=bash
set -euo pipefail
@@ -57,3 +61,43 @@ Upgrade with:
EOS
exit 1
fi
+
+# โ”€โ”€ Dependency bootstrap โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+# A marketplace add/upgrade (`codex plugin marketplace add|upgrade`) replaces
+# this checkout wholesale, so node_modules can arrive missing even though
+# package.json and bun.lock ship with the tree. The CLI then cannot start
+# (`Cannot find package 'proper-lockfile'`, the one runtime dependency), and
+# `o2b install-cli` cannot repair anything because it runs through the same
+# import. Probe for that package with one file stat - no Bun startup - and
+# run the documented install once when it is missing. The common case
+# (node_modules present) pays the stat and nothing more.
+#
+# The root is the checkout the launcher belongs to. Launchers export
+# REPO_ROOT before sourcing this file; anything else (scripts/test) falls
+# back to the parent of this script's own scripts/ directory, derived with
+# parameter expansion so no external utility is needed - the isolated-PATH
+# environments this precheck is tested in ship none beyond cat, tr and head.
+# Pure-bash suffix strip: <root>/scripts/_bun-precheck.sh -> <root>.
+#
+# Test/CI seam (undocumented in user-facing help), following
+# _macos-sqlite.sh: O2B_SKIP_DEP_BOOTSTRAP=1 skips the install for callers
+# that manage node_modules themselves.
+_BUN_PRECHECK_DEPS_ROOT=${REPO_ROOT:-}
+if [[ -z "$_BUN_PRECHECK_DEPS_ROOT" ]]; then
+ _BUN_PRECHECK_DEPS_ROOT=$(cd -- "${BASH_SOURCE[0]%/*}/.." 2>/dev/null && pwd || true)
+fi
+if [[ -n "$_BUN_PRECHECK_DEPS_ROOT" ]] && [[ -z "${O2B_SKIP_DEP_BOOTSTRAP:-}" ]] \
+ && [[ ! -f "${_BUN_PRECHECK_DEPS_ROOT}/node_modules/proper-lockfile/package.json" ]]; then
+ echo "o2b: dependency tree missing in this checkout; running 'bun install --frozen-lockfile'..." >&2
+ # stdout goes to stderr: the MCP launchers serve JSON-RPC on stdout, and
+ # bun's install report must never interleave with it.
+ if ! (cd -- "$_BUN_PRECHECK_DEPS_ROOT" && bun install --frozen-lockfile 1>&2); then
+ cat >&2 <<EOS
+error: the Open Second Brain checkout at ${_BUN_PRECHECK_DEPS_ROOT} has no dependency tree and the automatic
+'bun install --frozen-lockfile' failed. Run that command inside the checkout yourself (plain
+'bun install' if the lockfile is out of sync), then retry. See install/codex.md and docs/updating.md.
+EOS
+ exit 1
+ fi
+fi
+unset _BUN_PRECHECK_DEPS_ROOT _BUN_PRECHECK_VERSION _BUN_MAJOR _BUN_MINOR _BUN_PATCH
diff --git a/scripts/o2b-aider.cmd b/scripts/o2b-aider.cmd
index 4c311293..32264a92 100644
--- a/scripts/o2b-aider.cmd
+++ b/scripts/o2b-aider.cmd
@@ -21,5 +21,24 @@ if errorlevel 1 (
exit /b 127
)
)
+rem Dependency bootstrap. A marketplace add/upgrade replaces this checkout
+rem wholesale, so node_modules can arrive missing even though package.json and
+rem bun.lock ship with the tree, and the CLI then cannot start. One file stat
+rem when the tree is present; one frozen install when it is not. Mirrors the
+rem block in scripts/o2b; O2B_SKIP_DEP_BOOTSTRAP opts out.
+if not exist "%O2B_ROOT%\node_modules\proper-lockfile\package.json" (
+ if not defined O2B_SKIP_DEP_BOOTSTRAP (
+ >&2 echo o2b-aider: dependency tree missing in this checkout; running 'bun install --frozen-lockfile'...
+ pushd "%O2B_ROOT%"
+ "%O2B_BUN%" install --frozen-lockfile 1>&2
+ if errorlevel 1 (
+ >&2 echo error: the automatic 'bun install --frozen-lockfile' failed in "%O2B_ROOT%".
+ >&2 echo Run that command inside the checkout yourself, then retry. See install/codex.md and docs/updating.md.
+ popd
+ exit /b 1
+ )
+ popd
+ )
+)
"%O2B_BUN%" run "%O2B_ROOT%\src\cli\main.ts" aider wrap %*
exit /b %ERRORLEVEL%
diff --git a/scripts/o2b-mcp.cmd b/scripts/o2b-mcp.cmd
index 193b472f..3c7a82ce 100644
--- a/scripts/o2b-mcp.cmd
+++ b/scripts/o2b-mcp.cmd
@@ -21,5 +21,26 @@ if errorlevel 1 (
exit /b 127
)
)
+rem Dependency bootstrap. A marketplace add/upgrade replaces this checkout
+rem wholesale, so node_modules can arrive missing even though package.json and
+rem bun.lock ship with the tree, and the CLI then cannot start. One file stat
+rem when the tree is present; one frozen install when it is not. Mirrors the
+rem block at the bottom of _bun-precheck.sh; O2B_SKIP_DEP_BOOTSTRAP opts out.
+rem stdout joins stderr: this launcher serves the MCP stdio transport, and
+rem bun's install report must never interleave with the JSON-RPC stream.
+if not exist "%O2B_ROOT%\node_modules\proper-lockfile\package.json" (
+ if not defined O2B_SKIP_DEP_BOOTSTRAP (
+ >&2 echo o2b: dependency tree missing in this checkout; running 'bun install --frozen-lockfile'...
+ pushd "%O2B_ROOT%"
+ "%O2B_BUN%" install --frozen-lockfile 1>&2
+ if errorlevel 1 (
+ >&2 echo error: the automatic 'bun install --frozen-lockfile' failed in "%O2B_ROOT%".
+ >&2 echo Run that command inside the checkout yourself, then retry. See install/codex.md and docs/updating.md.
+ popd
+ exit /b 1
+ )
+ popd
+ )
+)
"%O2B_BUN%" run "%O2B_ROOT%\src\cli\main.ts" mcp %*
exit /b %ERRORLEVEL%
diff --git a/scripts/o2b.cmd b/scripts/o2b.cmd
index 9669e92a..769e5827 100644
--- a/scripts/o2b.cmd
+++ b/scripts/o2b.cmd
@@ -21,5 +21,26 @@ if errorlevel 1 (
exit /b 127
)
)
+rem Dependency bootstrap. A marketplace add/upgrade replaces this checkout
+rem wholesale, so node_modules can arrive missing even though package.json and
+rem bun.lock ship with the tree, and the CLI then cannot start. One file stat
+rem when the tree is present; one frozen install when it is not. Mirrors the
+rem block at the bottom of _bun-precheck.sh; O2B_SKIP_DEP_BOOTSTRAP opts out.
+if not exist "%O2B_ROOT%\node_modules\proper-lockfile\package.json" (
+ if not defined O2B_SKIP_DEP_BOOTSTRAP (
+ >&2 echo o2b: dependency tree missing in this checkout; running 'bun install --frozen-lockfile'...
+ pushd "%O2B_ROOT%"
+ rem stdout joins stderr: this launcher also serves the MCP stdio transport,
+ rem and bun's install report must never interleave with the JSON-RPC stream.
+ "%O2B_BUN%" install --frozen-lockfile 1>&2
+ if errorlevel 1 (
+ >&2 echo error: the automatic 'bun install --frozen-lockfile' failed in "%O2B_ROOT%".
+ >&2 echo Run that command inside the checkout yourself, then retry. See install/codex.md and docs/updating.md.
+ popd
+ exit /b 1
+ )
+ popd
+ )
+)
"%O2B_BUN%" run "%O2B_ROOT%\src\cli\main.ts" %*
exit /b %ERRORLEVEL%
diff --git a/scripts/vault-log.cmd b/scripts/vault-log.cmd
index 703be2bc..c8f13bba 100644
--- a/scripts/vault-log.cmd
+++ b/scripts/vault-log.cmd
@@ -21,5 +21,24 @@ if errorlevel 1 (
exit /b 127
)
)
+rem Dependency bootstrap. A marketplace add/upgrade replaces this checkout
+rem wholesale, so node_modules can arrive missing even though package.json and
+rem bun.lock ship with the tree, and the CLI then cannot start. One file stat
+rem when the tree is present; one frozen install when it is not. Mirrors the
+rem block in scripts/o2b; O2B_SKIP_DEP_BOOTSTRAP opts out.
+if not exist "%O2B_ROOT%\node_modules\proper-lockfile\package.json" (
+ if not defined O2B_SKIP_DEP_BOOTSTRAP (
+ >&2 echo vault-log: dependency tree missing in this checkout; running 'bun install --frozen-lockfile'...
+ pushd "%O2B_ROOT%"
+ "%O2B_BUN%" install --frozen-lockfile 1>&2
+ if errorlevel 1 (
+ >&2 echo error: the automatic 'bun install --frozen-lockfile' failed in "%O2B_ROOT%".
+ >&2 echo Run that command inside the checkout yourself, then retry. See install/codex.md and docs/updating.md.
+ popd
+ exit /b 1
+ )
+ popd
+ )
+)
"%O2B_BUN%" run "%O2B_ROOT%\src\cli\main.ts" append-event %*
exit /b %ERRORLEVEL%
diff --git a/src/cli/brain/verbs/bank-export.ts b/src/cli/brain/verbs/bank-export.ts
index 0f9366db..a8a77d94 100644
--- a/src/cli/brain/verbs/bank-export.ts
+++ b/src/cli/brain/verbs/bank-export.ts
@@ -1,6 +1,9 @@
import { atomicWriteFileSync } from "../../../core/fs-atomic.ts";
import { exportBankBundle } from "../../../core/brain/portability/bundle.ts";
-import { isPromptDerivedPage } from "../../../core/brain/provenance/content-class.ts";
+import {
+ isPromptDerivedPage,
+ omittedPageReferenceMatcher,
+} from "../../../core/brain/provenance/content-class.ts";
import {
EGRESS_OUTCOME,
EGRESS_CONTENT_CLASS_INCLUDE_NOTICE,
@@ -11,6 +14,20 @@ import {
} from "../../../core/egress/guard.ts";
import { brainVerbContext, fail, parse } from "../helpers.ts";
+/**
+ * The stderr line counting the kept pages' structured references (graph
+ * links and relation targets, page-contract citations) that named a
+ * left-out page and were dropped. A bundle the policy shortened without
+ * a word is the same silently smaller export the page omission notice
+ * exists to prevent.
+ */
+function referenceOmissionNotice(dropped: number): string {
+ return (
+ `note: ${dropped} link/relation/citation target(s) to left-out page(s) dropped from ` +
+ "kept pages' structured references\n"
+ );
+}
+
/**
* `o2b brain bank-export [--out <file>] [--include-prompt-derived]` -
* serialise a whole-vault bank bundle (preferences + page graph + page
@@ -29,6 +46,13 @@ import { brainVerbContext, fail, parse } from "../helpers.ts";
* carries the pages back in and says so. The sources dashboard is an
* aggregate over signal records, not pages, and carries no page text, so
* the class policy does not reach it.
+ *
+ * A kept page can still name a left-out one in its structured
+ * references - graph links and relation targets, contract citations -
+ * and such a reference is the omitted page's name leaving through a page
+ * the policy kept, so those targets drop from the bundle (counted on
+ * stderr). Everything else about the kept records, their remaining
+ * targets included, stays exactly as the exporters derived it.
*/
export async function cmdBrainBankExport(argv: string[]): Promise<number> {
const { flags } = parse(argv, {
@@ -42,6 +66,7 @@ export async function cmdBrainBankExport(argv: string[]): Promise<number> {
let json: string;
let redacted: boolean;
let omitted: ReadonlyArray<string> = [];
+ let droppedReferences = 0;
try {
const bundle = exportBankBundle(vault);
const nodeSplit = egressContentClassSplit(bundle.graph.nodes, classPolicy, {
@@ -54,10 +79,32 @@ export async function cmdBrainBankExport(argv: string[]): Promise<number> {
});
// One page can appear in both sections; an omission is named once.
omitted = [...new Set([...nodeSplit.omitted, ...pageSplit.omitted])];
+ // The structured references of the kept records: a graph link, a
+ // relation target or a contract citation naming a left-out page drops,
+ // counted; a relation left with no targets drops with it, matching the
+ // graph exporter's own shape of carrying only non-empty fields.
+ const namesOmitted = omittedPageReferenceMatcher(omitted);
+ const keptNodes = nodeSplit.kept.map((node) => {
+ const links = node.links.filter((target) => !namesOmitted(target));
+ const relations: Record<string, ReadonlyArray<string>> = {};
+ let dropped = node.links.length - links.length;
+ for (const [relation, targets] of Object.entries(node.relations)) {
+ const kept = targets.filter((target) => !namesOmitted(target));
+ dropped += targets.length - kept.length;
+ if (kept.length > 0) relations[relation] = kept;
+ }
+ droppedReferences += dropped;
+ return { ...node, links, relations };
+ });
+ const keptPages = pageSplit.kept.map((page) => {
+ const citations = page.citations.filter((target) => !namesOmitted(target));
+ droppedReferences += page.citations.length - citations.length;
+ return { ...page, citations };
+ });
const verdict = redactForEgress("brain-bank-export", {
...bundle,
- graph: { ...bundle.graph, nodes: nodeSplit.kept },
- pages: pageSplit.kept,
+ graph: { ...bundle.graph, nodes: keptNodes },
+ pages: keptPages,
});
if (verdict.outcome !== EGRESS_OUTCOME.released) return fail(verdict.detail);
json = JSON.stringify(verdict.payload, null, 2) + "\n";
@@ -78,8 +125,9 @@ export async function cmdBrainBankExport(argv: string[]): Promise<number> {
}
if (flags["include-prompt-derived"] === true) {
process.stderr.write(EGRESS_CONTENT_CLASS_INCLUDE_NOTICE);
- } else if (omitted.length > 0) {
- process.stderr.write(contentClassOmissionNotice(omitted));
+ } else {
+ if (omitted.length > 0) process.stderr.write(contentClassOmissionNotice(omitted));
+ if (droppedReferences > 0) process.stderr.write(referenceOmissionNotice(droppedReferences));
}
if (redacted) process.stderr.write(EGRESS_REDACTION_NOTICE);
return 0;
diff --git a/src/cli/brain/verbs/graph-export.ts b/src/cli/brain/verbs/graph-export.ts
index 9d709c89..a30c486c 100644
--- a/src/cli/brain/verbs/graph-export.ts
+++ b/src/cli/brain/verbs/graph-export.ts
@@ -1,6 +1,9 @@
import { atomicWriteFileSync } from "../../../core/fs-atomic.ts";
import { exportVaultGraph } from "../../../core/brain/portability/graph.ts";
-import { isPromptDerivedPage } from "../../../core/brain/provenance/content-class.ts";
+import {
+ isPromptDerivedPage,
+ omittedPageReferenceMatcher,
+} from "../../../core/brain/provenance/content-class.ts";
import {
EGRESS_OUTCOME,
EGRESS_CONTENT_CLASS_INCLUDE_NOTICE,
@@ -11,6 +14,16 @@ import {
} from "../../../core/egress/guard.ts";
import { brainVerbContext, fail, parse } from "../helpers.ts";
+/**
+ * The stderr line counting the kept nodes' structured link and relation
+ * targets that named a left-out page and were dropped. A graph the
+ * policy shortened without a word is the same silently smaller export
+ * the page omission notice exists to prevent.
+ */
+function referenceOmissionNotice(dropped: number): string {
+ return `note: ${dropped} link/relation target(s) to left-out page(s) dropped from kept nodes\n`;
+}
+
/**
* `o2b brain graph-export [--out <file>] [--include-prompt-derived]` -
* serialise the user's vault pages (links + typed relations) to a stable
@@ -25,10 +38,14 @@ import { brainVerbContext, fail, parse } from "../helpers.ts";
* `prompt-derived` value is left out of the graph: its node - title,
* path, links and typed relations - is absent from the JSON, and the
* omission is named on stderr. `--include-prompt-derived` is the operator
- * override; it carries the nodes back in and says so. Links and relation
- * targets INSIDE a kept node are that page's own content and are not
- * rewritten: the graph serialises what the kept pages declare, it does
- * not editorialise their citations.
+ * override; it carries the nodes back in and says so.
+ *
+ * A kept node's links and relation targets are that page's own content,
+ * but they are structured references, and one naming a left-out page is
+ * the omitted page's name leaving through a node the policy kept - so
+ * such targets are dropped from the kept node (counted on stderr), while
+ * every other byte of the node, its title and its remaining targets
+ * stays exactly as the vault declares them.
*/
export async function cmdBrainGraphExport(argv: string[]): Promise<number> {
const { flags } = parse(argv, {
@@ -42,6 +59,7 @@ export async function cmdBrainGraphExport(argv: string[]): Promise<number> {
let json: string;
let redacted: boolean;
let omitted: ReadonlyArray<string> = [];
+ let droppedReferences = 0;
try {
const graph = exportVaultGraph(vault);
const split = egressContentClassSplit(graph.nodes, classPolicy, {
@@ -49,7 +67,23 @@ export async function cmdBrainGraphExport(argv: string[]): Promise<number> {
isPromptDerived: (node) => isPromptDerivedPage(vault, node.path),
});
omitted = split.omitted;
- const verdict = redactForEgress("brain-graph-export", { ...graph, nodes: split.kept });
+ // A kept node's structured targets naming a left-out page drop;
+ // a relation left with no targets drops with them, matching the
+ // exporter's own shape of carrying only non-empty relation fields.
+ const namesOmitted = omittedPageReferenceMatcher(omitted);
+ const keptNodes = split.kept.map((node) => {
+ const links = node.links.filter((target) => !namesOmitted(target));
+ const relations: Record<string, ReadonlyArray<string>> = {};
+ let dropped = node.links.length - links.length;
+ for (const [relation, targets] of Object.entries(node.relations)) {
+ const kept = targets.filter((target) => !namesOmitted(target));
+ dropped += targets.length - kept.length;
+ if (kept.length > 0) relations[relation] = kept;
+ }
+ droppedReferences += dropped;
+ return { ...node, links, relations };
+ });
+ const verdict = redactForEgress("brain-graph-export", { ...graph, nodes: keptNodes });
if (verdict.outcome !== EGRESS_OUTCOME.released) return fail(verdict.detail);
json = JSON.stringify(verdict.payload, null, 2) + "\n";
redacted = verdict.redacted;
@@ -69,8 +103,9 @@ export async function cmdBrainGraphExport(argv: string[]): Promise<number> {
}
if (flags["include-prompt-derived"] === true) {
process.stderr.write(EGRESS_CONTENT_CLASS_INCLUDE_NOTICE);
- } else if (omitted.length > 0) {
- process.stderr.write(contentClassOmissionNotice(omitted));
+ } else {
+ if (omitted.length > 0) process.stderr.write(contentClassOmissionNotice(omitted));
+ if (droppedReferences > 0) process.stderr.write(referenceOmissionNotice(droppedReferences));
}
if (redacted) process.stderr.write(EGRESS_REDACTION_NOTICE);
return 0;
diff --git a/src/cli/brain/verbs/okf-export.ts b/src/cli/brain/verbs/okf-export.ts
index b4ddb54a..bce682ba 100644
--- a/src/cli/brain/verbs/okf-export.ts
+++ b/src/cli/brain/verbs/okf-export.ts
@@ -1,4 +1,5 @@
import {
+ buildLog,
buildOkfBundleFromPages,
collectOkfPages,
OKF_INDEX_FILENAME,
@@ -9,7 +10,10 @@ import {
type OkfBundle,
type OkfBundleFile,
} from "../../../core/brain/portability/okf.ts";
-import { readContentClass } from "../../../core/brain/provenance/content-class.ts";
+import {
+ omittedPageReferenceMatcher,
+ readContentClass,
+} from "../../../core/brain/provenance/content-class.ts";
import {
EGRESS_OUTCOME,
EGRESS_CONTENT_CLASS_INCLUDE_NOTICE,
@@ -20,6 +24,32 @@ import {
} from "../../../core/egress/guard.ts";
import { brainVerbContext, fail, parse } from "../helpers.ts";
+/**
+ * The stderr line counting the change-log entries the default content-
+ * class policy dropped from `log.md`, beside the page omission notice.
+ * A log the policy shortened without a word is the same silently smaller
+ * export the page notice exists to prevent.
+ */
+function logOmissionNotice(dropped: number): string {
+ return (
+ `note: ${dropped} log line(s) left out by the same policy (agent-channel lines and ` +
+ "lines naming a left-out page); pass --include-prompt-derived to keep the log whole\n"
+ );
+}
+
+/**
+ * The stderr line counting the kept pages' structured citations that
+ * named a left-out page and were dropped from the manifest. The page
+ * bodies are untouched - a dropped citation is a metadata omission, and
+ * the notice says so.
+ */
+function citationOmissionNotice(dropped: number): string {
+ return (
+ `note: ${dropped} citation(s) to left-out page(s) dropped from kept pages' manifest ` +
+ "metadata; page text is untouched\n"
+ );
+}
+
/**
* `o2b brain okf-export --out <dir> [--force] [--include-prompt-derived]`
* โ€” write a portable Open Knowledge Format bundle (concepts / queries /
@@ -48,10 +78,21 @@ import { brainVerbContext, fail, parse } from "../helpers.ts";
* re-rendered from the REDACTED manifest by the same renderer the
* builder uses - the pre-redaction index named the excluded pages, and
* an index that lists pages or values the shipped `okf.json` does not
- * carry would make the bundle disagree with itself - and `log.md` is
- * kept whole, because its entries are day-keyed events (a timestamp, a
- * kind, body bullets), not page content; it still rides the same
- * redaction scan as the page files.
+ * carry would make the bundle disagree with itself - and `log.md` gets
+ * the content-class policy too. Its entries are day-keyed events, not
+ * page content, but the shards hold agent-written material: an entry the
+ * MCP transport appended is agent-dictated text, and a write entry whose
+ * `target` is a page the split left out would name the very omission
+ * back into the bundle. Under the default, both kinds of entry leave
+ * (counted on stderr beside the page notice) and every kept entry rides
+ * byte-exactly; `--include-prompt-derived` keeps the log whole and says
+ * so. The log still rides the same redaction scan as the page files.
+ *
+ * A kept page's own prose is never rewritten, but the manifest's
+ * structured `citations` are not prose: a citation naming a left-out
+ * page is the omitted page's name leaving through a kept page, so such
+ * citations are dropped from the manifest (counted on stderr) while the
+ * body text that carries the link stays exactly as written.
*
* A redacted bundle is no longer a lossless round-trip of the vault. The
* verb says so on stderr rather than leaving the interchange contract to
@@ -84,14 +125,36 @@ export async function cmdBrainOkfExport(argv: string[]): Promise<number> {
isPromptDerived: (page) => readContentClass(page.frontmatter) !== null,
});
- const built = buildOkfBundleFromPages(vault, split.kept);
+ // A kept page's structured citations naming a left-out page are
+ // metadata, not prose: they leave the manifest, counted, while the
+ // page body that carries the link is untouched. The bundle's
+ // `sources` need no second pass - they are resolved against the kept
+ // pages alone, so a citation to an omitted page already resolves to
+ // nothing there.
+ const namesOmitted = omittedPageReferenceMatcher(split.omitted);
+ let droppedCitations = 0;
+ const keptPages = split.kept.map((page) => {
+ const citations = page.citations.filter((target) => !namesOmitted(target));
+ droppedCitations += page.citations.length - citations.length;
+ return { ...page, citations };
+ });
+
+ // The log gets the same policy the pages got: agent-channel entries
+ // and entries naming a left-out page leave under the default, and the
+ // prebuilt result is handed to the builder so the vault is read once.
+ const log = buildLog(vault, {
+ includePromptDerived: flags["include-prompt-derived"] === true,
+ omittedPaths: new Set(split.omitted),
+ });
+
+ const built = buildOkfBundleFromPages(vault, keptPages, log);
const verdict = redactForEgress("brain-okf-export", {
manifest: built.manifest,
// The two reserved files rendered FROM the manifest are re-rendered
// after redaction below, so only the per-page files and the log are
- // scanned as text here. The log stays whole - its entries name days
- // and events, not page bodies - but it is scanned like any other
- // markdown the bundle carries.
+ // scanned as text here. The log arrives already carrying the class
+ // policy's filter, and is scanned like any other markdown the
+ // bundle carries.
files: built.files.filter(
(file) => file.path !== OKF_MANIFEST_FILENAME && file.path !== OKF_INDEX_FILENAME,
),
@@ -119,8 +182,14 @@ export async function cmdBrainOkfExport(argv: string[]): Promise<number> {
);
if (flags["include-prompt-derived"] === true) {
process.stderr.write(EGRESS_CONTENT_CLASS_INCLUDE_NOTICE);
- } else if (split.omitted.length > 0) {
- process.stderr.write(contentClassOmissionNotice(split.omitted));
+ } else {
+ if (split.omitted.length > 0) {
+ process.stderr.write(contentClassOmissionNotice(split.omitted));
+ }
+ if (log.droppedEntries > 0) process.stderr.write(logOmissionNotice(log.droppedEntries));
+ if (droppedCitations > 0) {
+ process.stderr.write(citationOmissionNotice(droppedCitations));
+ }
}
if (verdict.redacted) process.stderr.write(EGRESS_REDACTION_NOTICE);
} catch (exc) {
diff --git a/src/cli/brain/verbs/privacy-marker-restamp.ts b/src/cli/brain/verbs/privacy-marker-restamp.ts
index e92ecdea..1bed33d4 100644
--- a/src/cli/brain/verbs/privacy-marker-restamp.ts
+++ b/src/cli/brain/verbs/privacy-marker-restamp.ts
@@ -1,4 +1,5 @@
import { planPrivacyMarkerRestamp } from "../../../core/brain/privacy-marker-restamp.ts";
+import { emitNextStep } from "../../advisory-rail.ts";
import { brainVerbContext, info, ok, okJson, parse } from "../helpers.ts";
/**
@@ -18,6 +19,7 @@ export async function cmdBrainPrivacyMarkerRestamp(argv: string[]): Promise<numb
});
const { vault } = brainVerbContext(flags);
const apply = flags["apply"] === true;
+ const jsonRequested = flags["json"] === true;
const result = planPrivacyMarkerRestamp(vault, { apply });
@@ -52,6 +54,20 @@ export async function cmdBrainPrivacyMarkerRestamp(argv: string[]): Promise<numb
);
info("re-run with --apply to union the reserved token into visibility:");
}
+ // The consequence of the stamp, for the pages this run would touch: the
+ // formal token dominates the scope rule, so a page found by default-scope
+ // local search before the stamp is found by neither scope after it. The
+ // sentence states what happens; the how-to travels through the advisory
+ // rail (`restamp-local-visibility`), the one mechanism that names a
+ // forward command.
+ if (result.candidates.length > 0) {
+ info(
+ apply
+ ? "note: each stamped page is now also excluded from default-scope local search"
+ : "note: after --apply each stamped page is also excluded from default-scope local search",
+ );
+ emitNextStep("restamp-local-visibility", { command: "brain", argv, jsonRequested });
+ }
for (const c of result.candidates) info(` ${c.path}: ${c.visibilityLine}`);
return 0;
}
diff --git a/src/cli/brain/verbs/reject.ts b/src/cli/brain/verbs/reject.ts
index efa3956e..a13d9b32 100644
--- a/src/cli/brain/verbs/reject.ts
+++ b/src/cli/brain/verbs/reject.ts
@@ -14,7 +14,7 @@ import {
import { preferencePath } from "../../../core/brain/paths.ts";
import { isoDate, isoSecond } from "../../../core/brain/time.ts";
import { renderPrefLink } from "../../../core/brain/wikilink.ts";
-import { appendLogEvent } from "../../../core/brain/log.ts";
+import { appendLogEvent, logShardWikilink } from "../../../core/brain/log.ts";
import { BRAIN_LOG_EVENT_KIND, BRAIN_RETIRED_REASON } from "../../../core/brain/types.ts";
import { brainVerbContext, fail, ok, okJson, parse } from "../helpers.ts";
@@ -58,7 +58,7 @@ export async function cmdBrainReject(argv: string[]): Promise<number> {
const now = new Date();
const todayDate = isoDate(now);
- const retiredBy = `[[Brain/log/${todayDate}]]`;
+ const retiredBy = logShardWikilink(todayDate);
const reasonText = String(flags["reason"]).trim();
try {
diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts
index 52aad2b4..d5178015 100644
--- a/src/cli/command-manifest.ts
+++ b/src/cli/command-manifest.ts
@@ -939,6 +939,41 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({
),
],
),
+ command(
+ "thresholds",
+ "The local per-model thresholds store next to config.yaml (issue #264)",
+ [],
+ [
+ command(
+ "fit",
+ "Derive a use's threshold from its labelled report data against a false-action target and write it for the answering model",
+ [
+ flag("use", "string"),
+ flag("target", "string"),
+ flag("since", "string"),
+ flag("vault", "string"),
+ flag("config", "string"),
+ ],
+ ),
+ command(
+ "list",
+ "List the stored thresholds by model and use, with per-vault overrides and kept history",
+ [flag("vault", "string"), flag("config", "string")],
+ ),
+ command(
+ "set",
+ "Write one threshold by hand for a model and use; existing entries keep their history",
+ [
+ flag("model", "string"),
+ flag("use", "string"),
+ flag("value", "string"),
+ flag("name", "string"),
+ flag("vault", "string"),
+ flag("config", "string"),
+ ],
+ ),
+ ],
+ ),
],
),
command(
diff --git a/src/cli/decision-model.ts b/src/cli/decision-model.ts
index 93adb33b..7238d907 100644
--- a/src/cli/decision-model.ts
+++ b/src/cli/decision-model.ts
@@ -1,15 +1,17 @@
/**
* `o2b decision-model` subcommand dispatcher (issue #213, Part 1).
*
- * check [--ping] the configured state: enabled or not, provider, base
- * URL, pinned model, the key variable's NAME and whether
- * it is set (never the value), per-use modes, vault
- * opt-out, cost gate and today's spend, processor terms,
- * the measured tuning state and the cached eval grades.
- * `--ping` sends one request over a synthetic state that
- * carries no vault content. Exit 1 only for a broken
- * configured provider (invalid config, or a failed ping
- * of an active one); a missing key is exit 0.
+ * check [--ping] the configured state: enabled or not, the provider
+ * chain in order, base URL, pinned model, the key
+ * variable's NAME and whether it is set (never the
+ * value), per-use modes, vault opt-out, cost gate and
+ * today's spend, processor terms, the measured tuning
+ * state and the cached eval grades. `--ping` sends one
+ * request per configured provider over a synthetic
+ * state that carries no vault content and reports each
+ * one. Exit 1 only for a broken configured provider
+ * (invalid config, or no configured provider answering
+ * the ping); a missing key is exit 0.
* report per use: calls, outcome mix, latency, tokens, cost,
* rerank shadow agreement and the cached eval grades,
* from `decision_model_call` and
@@ -25,12 +27,22 @@
* eval list the cached grade records, oldest first, in prose or
* `--json`; malformed records are skipped with a named
* note, never silently.
+ * thresholds fit derive one use's threshold from the use's labelled
+ * report data against a false-action target
+ * (`--target`, default 0.05) and write it for the
+ * answering model, in the local thresholds store next
+ * to config.yaml.
+ * thresholds list the stored thresholds, global and per-vault.
+ * thresholds set write one threshold by hand for a model.
*
* check and list are read-only. eval run writes only the grade record in
* the vault's continuity log and, on a promote, the mark under the
- * platform state root.
+ * platform state root. thresholds fit and set write only the thresholds
+ * store; nothing here deletes an entry.
*/
+import { readFileSync } from "node:fs";
+
import { CliError, parseFlags } from "./argparse.ts";
import { defaultConfigPath, discoverConfig, resolveVault } from "../core/config.ts";
import {
@@ -44,6 +56,16 @@ import {
} from "../core/decision-model/diagnostics.ts";
import { isDecisionModelUse, type DecisionModelUse } from "../core/decision-model/contract.ts";
import { resolveDecisionModelConfig } from "../core/decision-model/config.ts";
+import {
+ BUILTIN_USE_THRESHOLDS,
+ PRIMARY_THRESHOLD_NAME,
+} from "../core/decision-model/questions.ts";
+import {
+ decisionThresholdsPath,
+ parseThresholdsStore,
+ writeThresholdEntry,
+} from "../core/decision-model/thresholds.ts";
+import { runThresholdFit } from "../core/decision-model/thresholds-fit.ts";
import { decisionEvalSpecFor } from "../core/decision-model/eval-spec.ts";
import { runDecisionModelEval } from "../core/decision-model/eval/grade.ts";
import { listDecisionModelEvalGrades } from "../core/decision-model/record.ts";
@@ -55,16 +77,22 @@ const USAGE =
"[--config <path>] [--json]\n" +
" o2b decision-model eval run --use <use> [--vault <path>] [--config <path>] [--json]\n" +
" o2b decision-model eval list [--use <use>] [--since <date>] [--vault <path>] " +
- "[--config <path>] [--json]\n";
+ "[--config <path>] [--json]\n" +
+ " o2b decision-model thresholds fit --use <use> [--target <n>] [--since <date>] " +
+ "[--vault <path>] [--config <path>] [--json]\n" +
+ " o2b decision-model thresholds list [--vault <path>] [--config <path>] [--json]\n" +
+ " o2b decision-model thresholds set --model <id> --use <use> --value <n> " +
+ "[--name <name>] [--vault <path>] [--config <path>] [--json]\n";
function resolveTargets(flags: Record<string, unknown>): {
+ readonly configPath: string;
readonly config: Readonly<Record<string, string>>;
readonly vault: string | null;
} {
const configPath = (flags["config"] as string | undefined) ?? defaultConfigPath();
const config = discoverConfig(configPath).data;
const vault = (flags["vault"] as string | undefined) ?? resolveVault(configPath) ?? null;
- return { config, vault };
+ return { configPath, config, vault };
}
async function checkVerb(argv: ReadonlyArray<string>): Promise<number> {
@@ -115,13 +143,18 @@ function reportVerb(argv: ReadonlyArray<string>): number {
if (use !== undefined && !isDecisionModelUse(use)) {
throw new CliError(`--use must be a decision-model use, got '${use}'`);
}
- const { vault } = resolveTargets(flags);
+ const { config, vault } = resolveTargets(flags);
if (vault === null) {
throw new CliError("no vault configured. Pass --vault <path>.");
}
const report = buildDecisionModelReport(vault, {
...(since !== undefined ? { since } : {}),
...(use !== undefined ? { use } : {}),
+ config: resolveDecisionModelConfig({
+ config,
+ vault,
+ secretsVault: vault ?? undefined,
+ }),
});
if (flags["json"] === true) {
process.stdout.write(JSON.stringify(report, null, 2) + "\n");
@@ -296,6 +329,248 @@ function evalVerb(argv: ReadonlyArray<string>): number | Promise<number> {
}
}
+// ----- thresholds (the local per-model store, issue #264) ----------------
+
+function useFlagRequired(flags: Record<string, unknown>): DecisionModelUse {
+ const use = useFlag(flags);
+ if (use === undefined) throw new CliError("decision-model thresholds needs --use <use>");
+ return use;
+}
+
+function thresholdsFitVerb(argv: ReadonlyArray<string>): number {
+ const { flags, positional } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ use: { type: "string" },
+ target: { type: "string" },
+ since: { type: "string" },
+ });
+ if (positional.length > 0) {
+ throw new CliError(
+ `decision-model thresholds fit takes no positional arguments: ${positional.join(" ")}`,
+ );
+ }
+ const use = useFlagRequired(flags);
+ const { configPath, vault } = resolveTargets(flags);
+ if (vault === null) {
+ throw new CliError("no vault configured. Pass --vault <path>.");
+ }
+ const targetRaw = flags["target"] as string | undefined;
+ let target: number | undefined;
+ if (targetRaw !== undefined) {
+ target = Number(targetRaw);
+ if (!Number.isFinite(target) || target <= 0 || target >= 1) {
+ throw new CliError(`--target must be a share between 0 and 1, got '${targetRaw}'`);
+ }
+ }
+ const since = sinceFlag(flags);
+ const result = runThresholdFit(decisionThresholdsPath(configPath), {
+ vault,
+ use,
+ ...(target !== undefined ? { target } : {}),
+ ...(since !== undefined ? { since } : {}),
+ });
+ if (!result.ok) {
+ throw new CliError(`decision-model thresholds fit: ${result.note}; nothing was written`);
+ }
+ if (flags["json"] === true) {
+ process.stdout.write(
+ JSON.stringify(
+ {
+ model: result.model,
+ use: result.use,
+ name: result.name,
+ value: result.entry.values[result.name],
+ target: result.entry.target ?? null,
+ samples: result.entry.samples ?? null,
+ updated_at: result.entry.updated_at,
+ already_tuned: result.alreadyTuned,
+ store_path: decisionThresholdsPath(configPath),
+ },
+ null,
+ 2,
+ ) + "\n",
+ );
+ return 0;
+ }
+ const share =
+ result.entry.samples !== undefined && result.entry.samples > 0
+ ? `over ${result.entry.samples} action(s)`
+ : "over no actions";
+ const lines = [
+ `thresholds fit ${result.use}: wrote ${result.name} = ` +
+ `${result.entry.values[result.name]} for model '${result.model}'`,
+ ` false actions at most ${((result.entry.target ?? 0) * 100).toFixed(1)}% (${share})`,
+ ` store: ${decisionThresholdsPath(configPath)}` +
+ (result.alreadyTuned
+ ? " (entry updated; the previous value is kept in its history)"
+ : " (new entry; switching models never deletes one)"),
+ ];
+ process.stdout.write(lines.join("\n") + "\n");
+ return 0;
+}
+
+function renderThresholdEntry(
+ model: string,
+ use: string,
+ entry: {
+ values: Readonly<Record<string, number>>;
+ source: string;
+ updated_at: string;
+ target?: number;
+ samples?: number;
+ history?: ReadonlyArray<{ value: number; source: string; updated_at: string }>;
+ },
+): string[] {
+ const values = Object.entries(entry.values)
+ .map(([name, value]) => `${name}=${value}`)
+ .join(", ");
+ const facts = [
+ entry.source,
+ entry.target !== undefined ? `target ${entry.target}` : null,
+ entry.samples !== undefined ? `${entry.samples} action(s)` : null,
+ entry.updated_at,
+ ]
+ .filter((v) => v !== null)
+ .join(", ");
+ const lines = [` ${model} / ${use}: ${values} (${facts})`];
+ for (const h of entry.history ?? []) {
+ lines.push(` was ${h.value} (${h.source}, ${h.updated_at})`);
+ }
+ return lines;
+}
+
+function thresholdsListVerb(argv: ReadonlyArray<string>): number {
+ const { flags, positional } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ });
+ if (positional.length > 0) {
+ throw new CliError(
+ `decision-model thresholds list takes no positional arguments: ${positional.join(" ")}`,
+ );
+ }
+ const { configPath } = resolveTargets(flags);
+ const storePath = decisionThresholdsPath(configPath);
+ let store;
+ try {
+ store = parseThresholdsStore(readFileSync(storePath, "utf8"));
+ } catch {
+ store = parseThresholdsStore("");
+ }
+ if (flags["json"] === true) {
+ process.stdout.write(JSON.stringify({ path: storePath, store }, null, 2) + "\n");
+ return 0;
+ }
+ const lines: string[] = [`thresholds: ${storePath}`];
+ const models = Object.entries(store.models).toSorted((a, b) => a[0].localeCompare(b[0]));
+ if (models.length === 0) lines.push("models: none stored");
+ else {
+ lines.push("models:");
+ for (const [model, uses] of models) {
+ for (const use of Object.keys(uses).toSorted()) {
+ lines.push(...renderThresholdEntry(model, use, uses[use as never]!));
+ }
+ }
+ }
+ const vaults = Object.entries(store.vaults).toSorted((a, b) => a[0].localeCompare(b[0]));
+ if (vaults.length > 0) {
+ lines.push("vault overrides:");
+ for (const [vaultPath, section] of vaults) {
+ lines.push(` ${vaultPath}`);
+ for (const [model, uses] of Object.entries(section).toSorted((a, b) =>
+ a[0].localeCompare(b[0]),
+ )) {
+ for (const use of Object.keys(uses).toSorted()) {
+ lines.push(...renderThresholdEntry(model, use, uses[use as never]!));
+ }
+ }
+ }
+ }
+ process.stdout.write(lines.join("\n") + "\n");
+ return 0;
+}
+
+function thresholdsSetVerb(argv: ReadonlyArray<string>): number {
+ const { flags, positional } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ model: { type: "string" },
+ use: { type: "string" },
+ value: { type: "string" },
+ name: { type: "string" },
+ });
+ if (positional.length > 0) {
+ throw new CliError(
+ `decision-model thresholds set takes no positional arguments: ${positional.join(" ")}`,
+ );
+ }
+ const model = flags["model"] as string | undefined;
+ if (model === undefined || model.trim() === "") {
+ throw new CliError("decision-model thresholds set needs --model <id>");
+ }
+ const use = useFlagRequired(flags);
+ const valueRaw = flags["value"] as string | undefined;
+ const value = Number(valueRaw);
+ if (valueRaw === undefined || !Number.isFinite(value) || value < 0 || value > 1) {
+ throw new CliError(`--value must be a probability between 0 and 1, got '${valueRaw}'`);
+ }
+ const names = Object.keys(BUILTIN_USE_THRESHOLDS[use]!);
+ const name = (flags["name"] as string | undefined) ?? PRIMARY_THRESHOLD_NAME[use];
+ if (!names.includes(name)) {
+ throw new CliError(`--name must be one of ${names.join(", ")} for use '${use}', got '${name}'`);
+ }
+ const { configPath } = resolveTargets(flags);
+ const storePath = decisionThresholdsPath(configPath);
+ const vaultOverride = flags["vault"] as string | undefined;
+ const entry = writeThresholdEntry(
+ storePath,
+ model.trim(),
+ use,
+ { values: { [name]: value }, source: "set" },
+ vaultOverride ?? null,
+ );
+ if (flags["json"] === true) {
+ process.stdout.write(
+ JSON.stringify(
+ {
+ model: model.trim(),
+ use,
+ name,
+ value: entry.values[name],
+ updated_at: entry.updated_at,
+ vault: vaultOverride ?? null,
+ store_path: storePath,
+ },
+ null,
+ 2,
+ ) + "\n",
+ );
+ return 0;
+ }
+ const lines = [
+ `thresholds set: wrote ${name} = ${entry.values[name]} for model '${model.trim()}' (use ${use})`,
+ ` store: ${storePath}` +
+ (vaultOverride !== undefined ? ` (override for vault ${vaultOverride})` : ""),
+ ];
+ process.stdout.write(lines.join("\n") + "\n");
+ return 0;
+}
+
+function thresholdsVerb(argv: ReadonlyArray<string>): number {
+ switch (argv[0]) {
+ case "fit":
+ return thresholdsFitVerb(argv.slice(1));
+ case "list":
+ return thresholdsListVerb(argv.slice(1));
+ case "set":
+ return thresholdsSetVerb(argv.slice(1));
+ default:
+ process.stderr.write(USAGE);
+ return 2;
+ }
+}
+
export async function handleDecisionModelSubcommand(argv: ReadonlyArray<string>): Promise<number> {
const verb = argv[0];
const rest = argv.slice(1);
@@ -306,6 +581,8 @@ export async function handleDecisionModelSubcommand(argv: ReadonlyArray<string>)
return reportVerb(rest);
case "eval":
return await evalVerb(rest);
+ case "thresholds":
+ return thresholdsVerb(rest);
default:
process.stderr.write(USAGE);
return 2;
diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.ts
index a36c7715..71224dbb 100644
--- a/src/core/brain/diagnostics.ts
+++ b/src/core/brain/diagnostics.ts
@@ -826,6 +826,19 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new Map
nextCommand: "o2b brain tiers accept <path>",
autoRepairable: false,
},
+ // The restamp's local-search consequence, named once: unioning the
+ // reserved token into `visibility:` moves a marker page out of
+ // default-scope local search too, because the formal token dominates
+ // the scope rule. The command is fully structural - the reserved deny
+ // token is a fixed vocabulary identifier, not a runtime parameter -
+ // so the rail carries what a hand-written pointer beside it could
+ // only retype.
+ {
+ code: "restamp-local-visibility",
+ issueClass: "a restamped page is also reserved against default-scope local search",
+ nextCommand: "o2b search --visibility=private",
+ autoRepairable: false,
+ },
// --- Doctor classes the exit census found unregistered ---
// The census (`tests/core/brain/doctor-exit-census.test.ts`) forced
// every doctor code into one of two buckets. These three turned out
diff --git a/src/core/brain/doctor-exits.ts b/src/core/brain/doctor-exits.ts
index dfd4598c..54bbd7f5 100644
--- a/src/core/brain/doctor-exits.ts
+++ b/src/core/brain/doctor-exits.ts
@@ -225,6 +225,15 @@ const EXCLUSIONS: ReadonlyArray<DoctorExitExclusion> = [
"was meant to hold is not derivable from the error, and minting nickname data would put " +
"words in the operator's mouth about who knows whom",
},
+ {
+ code: "person-nicknames-entry-skipped",
+ reason:
+ "one entry of an otherwise usable person nickname map resolves nobody - the nickname " +
+ "normalizes to nothing, every value it lists normalizes to nothing, or the nickname " +
+ "equals a name it stands for. The repair is an edit to that entry by whoever wrote it: " +
+ "which real-world nickname was meant, and for whom, is their knowledge and is not " +
+ "derivable from the finding, so the doctor names the entry's position and stops there",
+ },
{
code: "iso-invalid",
reason:
diff --git a/src/core/brain/doctor/chunk-holds-check.ts b/src/core/brain/doctor/chunk-holds-check.ts
index ba04d028..8a9b8d2e 100644
--- a/src/core/brain/doctor/chunk-holds-check.ts
+++ b/src/core/brain/doctor/chunk-holds-check.ts
@@ -71,12 +71,14 @@ function indexProbe(dbPath: string, hold: ChunkHoldExclusion): HeldChunkDocRow[]
function holdMessage(row: HeldChunkDocRow, hold: ChunkHoldExclusion): string {
return (
- `${row.heldChunks} chunk(s) in this document are held back from embedding as provably ` +
- `over the declared input window of ${hold.model}: the largest single-chunk estimate is ` +
- `${row.maxEstimatedTokens} tokens against a ${hold.windowTokens}-token window. The page ` +
- "still searches lexically and is never sent to the embedding provider; narrowing the " +
- "chunking (search_chunk_size) or leaving the page unembedded are the ways to change the " +
- "outcome, both per-document decisions. `o2b search status` lists the held chunks."
+ `${row.heldChunks} chunk(s) in this document are held back from embedding: even a ` +
+ `conservative lower-bound estimate puts them over the declared input window of ` +
+ `${hold.model} - the largest single-chunk bound is ${row.maxEstimatedTokens} tokens ` +
+ `against a ${hold.windowTokens}-token window, and chunks merely estimated over that ` +
+ "window are still sent. The page still searches lexically and is never sent to the " +
+ "embedding provider; narrowing the chunking (search_chunk_size) or leaving the page " +
+ "unembedded are the ways to change the outcome, both per-document decisions. " +
+ "`o2b search status` lists the held chunks."
);
}
diff --git a/src/core/brain/doctor/embeddings-health-check.ts b/src/core/brain/doctor/embeddings-health-check.ts
index 9d69a2a8..679034a2 100644
--- a/src/core/brain/doctor/embeddings-health-check.ts
+++ b/src/core/brain/doctor/embeddings-health-check.ts
@@ -53,10 +53,12 @@
*/
import { resolveSearchConfig } from "../../search/index.ts";
+import { chunkHoldExclusion } from "../../search/embedding-spend.ts";
import { readPendingVectorCensus } from "../../search/indexer.ts";
import { staleBaseline } from "../../search/store/counts.ts";
import { LAST_INDEXED_AT_STATE_KEY, withReadonlyIndex } from "../../search/store/state.ts";
import { staleEmbeddings } from "../../search/store/vectors.ts";
+import type { ChunkHoldExclusion } from "../../search/store/chunks.ts";
import type { PendingVectorCensus, ResolvedEmbeddingConfig } from "../../search/types.ts";
import type { DoctorIssue } from "../types.ts";
import type { DoctorCheck, DoctorCheckContext, DoctorFindings } from "./check.ts";
@@ -98,15 +100,19 @@ export type EmbeddingsHealthSemantic = Pick<
/**
* The seam the check reads its facts through.
*
- * Receives the resolved index path and the embedding configuration so
- * the default probe below can resolve the pair stored vectors are
- * compared against (`staleBaseline`: an unset dimension falls back to
- * the stored one, and with none the comparison is by model alone); a
- * test injects any snapshot it wants without building an index.
+ * Receives the resolved index path, the embedding configuration, and the
+ * configuration's chunk-hold exclusion key (`null` when this config can
+ * hold nothing), so the default probe below can resolve the pair stored
+ * vectors are compared against (`staleBaseline`: an unset dimension falls
+ * back to the stored one, and with none the comparison is by model alone)
+ * and keep held chunks out of the pending count - the backfill they would
+ * otherwise recommend finds nothing to do and never clears. A test
+ * injects any snapshot it wants without building an index.
*/
export type EmbeddingsHealthProbe = (
dbPath: string,
semantic: EmbeddingsHealthSemantic,
+ hold: ChunkHoldExclusion | null,
) => EmbeddingsHealthProbeSnapshot;
/**
@@ -115,15 +121,19 @@ export type EmbeddingsHealthProbe = (
*
* Two peeks over one file, not one, because the census's unrecorded
* contract is folded in `readPendingVectorCensus` and re-deriving it
- * here is exactly the drift the reuse exists to prevent. A file that
- * vanishes between the two peeks is a half read, and the check answers
- * that with uncertainty rather than a health claim.
+ * here is exactly the drift the reuse exists to prevent. The hold
+ * exclusion threads into that census, so a chunk held out of embedding
+ * under the active key stops reading as backlog here just as it stops
+ * being priced everywhere else. A file that vanishes between the two
+ * peeks is a half read, and the check answers that with uncertainty
+ * rather than a health claim.
*/
function indexProbe(
dbPath: string,
semantic: EmbeddingsHealthSemantic,
+ hold: ChunkHoldExclusion | null,
): EmbeddingsHealthProbeSnapshot {
- const census = readPendingVectorCensus(dbPath);
+ const census = readPendingVectorCensus(dbPath, hold ?? undefined);
if (census.verdict !== "measured") {
return { census, staleEmbeddings: null, lastIndexedAt: null };
}
@@ -242,7 +252,7 @@ export function makeEmbeddingsHealthCheck(probe: EmbeddingsHealthProbe = indexPr
}
let snap;
try {
- snap = probe(dbPath, search.semantic);
+ snap = probe(dbPath, search.semantic, chunkHoldExclusion(search));
} catch (err) {
pushUncertain(out.uncertain, {
code: EMBEDDINGS_HEALTH_UNMEASURED_CODE,
diff --git a/src/core/brain/doctor/entity-checks.ts b/src/core/brain/doctor/entity-checks.ts
index 62742f2f..df7dc4a5 100644
--- a/src/core/brain/doctor/entity-checks.ts
+++ b/src/core/brain/doctor/entity-checks.ts
@@ -5,6 +5,9 @@
* through hand edits or sync merges - observable, never auto-deleted.
*/
+import { join } from "node:path";
+import { readFileSync } from "node:fs";
+
import {
entityLexicalAliasCandidates,
resolveEntitySemanticDedupConfig,
@@ -22,18 +25,10 @@ import {
ENTITY_CONFLICT_KIND,
type EntityIndex,
} from "../entities/index-builder.ts";
-import {
- PersonNicknameConfigError,
- resolvePersonNicknameMap,
-} from "../entities/person-nicknames.ts";
-import {
- derivePersonNameVariants,
- EMPTY_PERSON_NICKNAME_MAP,
- isPersonCategory,
- type PersonNicknameMap,
-} from "../entities/person-variants.ts";
+import { PERSON_NICKNAMES_ARTIFACT_REL } from "../entities/person-nicknames.ts";
+import { isPersonCategory } from "../entities/person-variants.ts";
+import { resolvePersonDerivedIndex, type PersonDerivedIndex } from "../entities/registry.ts";
import { findMalformedEntityLabels } from "../entities/label-hygiene.ts";
-import { ENTITY_STATUS_SCOPE, entityStatusInScope } from "../entities/status-scope.ts";
import type { BrainEntity, EntityConflict } from "../entities/types.ts";
import type { DoctorCheck } from "./check.ts";
@@ -129,36 +124,92 @@ export function personVariantCollisionMessage(a: BrainEntity, b: BrainEntity): s
* neither record declared it, so the collision scan has nothing to offer
* the merge that the ambiguity error at lookup time does not already
* say better.
+ *
+ * The join runs over the shared derived person-form index: one pass over
+ * its per-form claimant lists, linear in forms times claimants, instead
+ * of a variants-per-person rebuild inside an all-pairs loop.
*/
-function derivedVariantCollisionMessages(
- index: EntityIndex,
- nicknames: PersonNicknameMap,
-): string[] {
- const persons = index.entities.filter(
- (e) =>
- entityStatusInScope(e.status, ENTITY_STATUS_SCOPE.canonical) && isPersonCategory(e.category),
- );
- const declared = new Map<string, ReadonlySet<string>>(
- persons.map((p) => [p.id, new Set([p.name, ...p.aliases].map((f) => normalizeEntityName(f)))]),
- );
+function derivedVariantCollisionMessages(derivedIndex: PersonDerivedIndex): string[] {
const seenPairs = new Set<string>();
const messages: string[] = [];
- for (const deriver of persons) {
- const variants = new Set<string>();
- for (const form of [deriver.name, ...deriver.aliases]) {
- for (const v of derivePersonNameVariants(normalizeEntityName(form), nicknames)) {
- variants.add(v);
+ for (const claimants of derivedIndex.byForm.values()) {
+ if (claimants.derived.length === 0) continue;
+ // The collision is person-to-person: the declared claimant lane also
+ // carries non-person records (every canonical entity claims its
+ // declared forms there), and a person's nickname matching an org's
+ // name is not the alias-merge this scan nominates.
+ for (const deriver of claimants.derived) {
+ for (const claimant of claimants.declared) {
+ if (deriver.id === claimant.id) continue;
+ if (!isPersonCategory(claimant.category)) continue;
+ const pairKey = [deriver.id, claimant.id].toSorted().join("\u0000");
+ if (seenPairs.has(pairKey)) continue;
+ seenPairs.add(pairKey);
+ messages.push(personVariantCollisionMessage(deriver, claimant));
}
}
- if (variants.size === 0) continue;
- for (const claimant of persons) {
- if (claimant.id === deriver.id) continue;
- const pairKey = [deriver.id, claimant.id].toSorted().join("\u0000");
- if (seenPairs.has(pairKey)) continue;
- const claimantForms = declared.get(claimant.id)!;
- if (![...variants].some((v) => claimantForms.has(v))) continue;
- seenPairs.add(pairKey);
- messages.push(personVariantCollisionMessage(deriver, claimant));
+ }
+ return messages;
+}
+
+/** Doctor code of a silently-skipped `person-nicknames.json` entry. */
+export const PERSON_NICKNAMES_ENTRY_SKIPPED_CODE = "person-nicknames-entry-skipped";
+
+/**
+ * The entries of `person-nicknames.json` the parser keeps the file's
+ * other entries running without - a nickname that normalizes to an
+ * empty form, a value list with no usable name, a nickname equal to the
+ * full name it stands for - as one warning per skipped entry, naming
+ * the entry's POSITION (the same numbering the parser's shape errors
+ * use). A nickname below the derived-form floor is NOT a skip: declared
+ * nicknames are exempt from that floor by design.
+ *
+ * The doctor is the surfacing point for operator data faults, and a
+ * silently dropped entry is one: the file resolves, the read lanes
+ * report no degradation, and without this pass the entry would vanish.
+ * The raw document is read here because the shared reader deliberately
+ * folds the skipped entries away. A file that is missing, unreadable or
+ * unparseable reports nothing here - the unreadable warning or the
+ * quiet missing-file posture already covers it.
+ */
+function personNicknameSkippedEntryWarnings(vault: string): string[] {
+ const path = join(vault, PERSON_NICKNAMES_ARTIFACT_REL);
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(readFileSync(path, "utf8"));
+ } catch {
+ return [];
+ }
+ if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return [];
+ const messages: string[] = [];
+ let entryIndex = 0;
+ for (const [rawKey, rawValue] of Object.entries(parsed as Record<string, unknown>)) {
+ entryIndex += 1;
+ const nick = normalizeEntityName(rawKey);
+ if (!nick) {
+ messages.push(
+ `person-nicknames entry #${entryIndex} is skipped: the nickname normalizes to an ` +
+ "empty form and can never resolve.",
+ );
+ continue;
+ }
+ if (!Array.isArray(rawValue)) continue; // the parser raises an unreadable-config fault for this
+ const values = rawValue
+ .filter((v): v is string => typeof v === "string")
+ .map((v) => normalizeEntityName(v))
+ .filter((v) => v !== "");
+ if (values.length === 0) {
+ messages.push(
+ `person-nicknames entry #${entryIndex} is skipped: none of its values normalizes ` +
+ "to a name, so the nickname stands for no one.",
+ );
+ continue;
+ }
+ if (values.includes(nick)) {
+ messages.push(
+ `person-nicknames entry #${entryIndex} never resolves: the nickname is one of the ` +
+ "full names it stands for, so it adds no form.",
+ );
}
}
return messages;
@@ -264,23 +315,28 @@ export const entityRegistryCheck: DoctorCheck = {
// their documented degradations too - but the doctor is the
// surfacing point: a present-yet-unusable operator file is one
// warning, naming the path, with the error's own short
- // (position-only) detail.
- let nicknames = EMPTY_PERSON_NICKNAME_MAP;
- try {
- nicknames = resolvePersonNicknameMap(vault);
- } catch (err) {
- if (!(err instanceof PersonNicknameConfigError)) throw err;
- nicknames = EMPTY_PERSON_NICKNAME_MAP;
+ // (position-only) detail. Entries the parser folds away silently
+ // surface beside it, one warning each, by position.
+ const derivedIndex = resolvePersonDerivedIndex(vault);
+ if (derivedIndex.nicknameError !== null) {
issues.push({
severity: "warning",
code: PERSON_NICKNAMES_UNREADABLE_CODE,
- path: err.path,
+ path: derivedIndex.nicknameError.path,
message:
- `the person nickname map at ${err.path} is unusable and nickname-derived ` +
- `lookups run without it: ${err.detail}`,
+ `the person nickname map at ${derivedIndex.nicknameError.path} is unusable and ` +
+ `nickname-derived lookups run without it: ${derivedIndex.nicknameError.detail}`,
+ });
+ }
+ for (const message of personNicknameSkippedEntryWarnings(vault)) {
+ issues.push({
+ severity: "warning",
+ code: PERSON_NICKNAMES_ENTRY_SKIPPED_CODE,
+ path: join(vault, PERSON_NICKNAMES_ARTIFACT_REL),
+ message,
});
}
- for (const message of derivedVariantCollisionMessages(index, nicknames)) {
+ for (const message of derivedVariantCollisionMessages(derivedIndex)) {
issues.push({
severity: "warning",
code: ENTITY_ALIAS_CANDIDATE_CODE,
diff --git a/src/core/brain/doctor/privacy-marker-check.ts b/src/core/brain/doctor/privacy-marker-check.ts
index 82371446..41f704d7 100644
--- a/src/core/brain/doctor/privacy-marker-check.ts
+++ b/src/core/brain/doctor/privacy-marker-check.ts
@@ -53,7 +53,9 @@ export const privacyMarkerCheck: DoctorCheck = {
"(access / confidentiality / sensitivity) rather than a visibility: declaration: " +
`${named.join(", ")}${remainder}. Remote reads are already denied, but tools that ` +
"read visibility: cannot see the reservation. The preview-first restamp reports each " +
- "would-be visibility line and writes only on request, additively",
+ "would-be visibility line and writes only on request, additively. After an apply a " +
+ "stamped page is also excluded from default-scope local search (the CLI reaches it " +
+ "again with search --visibility private)",
} satisfies DoctorIssue);
},
};
diff --git a/src/core/brain/dream.ts b/src/core/brain/dream.ts
index c7c919a0..d76a40f4 100644
--- a/src/core/brain/dream.ts
+++ b/src/core/brain/dream.ts
@@ -84,6 +84,7 @@ import type {
} from "./dream-types.ts";
import { openWorkrun, WORKRUN_PHASE, type WorkrunHandle } from "./dream-workrun.ts";
import { buildIntentReview } from "./intent-review.ts";
+import { logShardWikilink } from "./log.ts";
import {
OPERATION,
progressCounter,
@@ -253,7 +254,12 @@ function dreamRun(
// nothing writes the decision back to `_brain.yaml`.
const healEnrichEnabled = opts.gates?.heal_enrich ?? cfg.dream.heal_enrich_enabled === true;
let runId = formatRunId(now);
- const wikilinkToRun = `[[Brain/log/${isoDate(now)}]]`;
+ // The retired_by stamp every retire in this pass carries. Built
+ // through the shared helper so it names the log shard the pass's own
+ // events land in - with a configured device id, the bare-date shape
+ // names a file this device never writes, and the doctor reads the
+ // provenance as a broken link.
+ const wikilinkToRun = logShardWikilink(isoDate(now));
const callerAgent = opts.agentName?.trim() ?? "";
const isNonPrimary =
diff --git a/src/core/brain/entities/person-variants.ts b/src/core/brain/entities/person-variants.ts
index 1e32232f..09d7db57 100644
--- a/src/core/brain/entities/person-variants.ts
+++ b/src/core/brain/entities/person-variants.ts
@@ -34,22 +34,36 @@
* lists the name's FULL normalized form among the values the
* nickname stands for.
*
- * Every emitted form meets a minimum-form-length floor (the same
- * discipline as the anchoring kernels' floor), the output order is
- * fixed (initial, reversed, then nicknames sorted), and the returned
- * array is frozen: same input, same bytes, on every peer. Derivation is
- * a re-ranking and resolution aid, never a retrieval mechanism and
- * never part of identity - `entityIdentityKey` is untouched by every
- * form this module emits.
+ * The two punctuation shapes normalization deliberately keeps - the
+ * dotted initial (`A. Lovelace`) and the comma spelling
+ * (`Lovelace, Ada`) - fold through {@link personNameComparisonForm}
+ * before any comparison: the kernel derives from the comparison form of
+ * its input, and every consumer compares query forms through the same
+ * fold, so both usual spellings meet the same derived forms the plain
+ * spelling does.
+ *
+ * Structural forms meet a minimum-form-length floor (the same
+ * discipline as the anchoring kernels' floor). Operator-declared
+ * nicknames are EXEMPT from that floor: they are data the operator
+ * chose, not forms this module invented, so a two-letter nickname the
+ * operator wrote down resolves instead of being silently dropped. The
+ * output order is fixed (the plain word order of a dotted or comma
+ * form, then initial, reversed, then nicknames sorted), and
+ * the returned array is frozen: same input, same bytes, on every peer.
+ * Derivation is a re-ranking and resolution aid, never a retrieval
+ * mechanism and never part of identity - `entityIdentityKey` is
+ * untouched by every form this module emits.
*/
/**
- * Minimum normalized form length that a derived person variant must
- * reach to be emitted. Mirrors `MIN_ANCHOR_FORM_LENGTH` in
+ * Minimum normalized form length that a derived STRUCTURAL person
+ * variant must reach to be emitted. Mirrors `MIN_ANCHOR_FORM_LENGTH` in
* `canonical.ts` (3): a derived form is an emitted needle in its own
* right, and an unfloored one would anchor standalone fragments no full
- * form ever matched. This module imports nothing, so the value is
- * restated here and pinned by tests rather than shared by import.
+ * form ever matched. Operator-declared nicknames are exempt - they are
+ * data the operator chose, and the doctor reports the unusable ones.
+ * This module imports nothing, so the value is restated here and pinned
+ * by tests rather than shared by import.
*/
export const MIN_PERSON_VARIANT_LENGTH = 3;
@@ -80,6 +94,93 @@ export type PersonNicknameMap = ReadonlyMap<string, ReadonlyArray<string>>;
/** The shared empty map, so callers without nickname data allocate once. */
export const EMPTY_PERSON_NICKNAME_MAP: PersonNicknameMap = Object.freeze(new Map());
+/**
+ * The nickname map INVERTED for derivation: keyed by a normalized FULL
+ * name, answering with the sorted nicknames it stands for. Building one
+ * index per nickname-file signature turns every derivation's scan of
+ * the whole operator map into a single lookup; the kernel consumes this
+ * shape, and the cache in `registry.ts` builds it once per signature.
+ */
+export interface PersonNicknameIndex {
+ /** The sorted nicknames the normalized FULL name `fullName` answers to. */
+ nicknamesOf(fullName: string): ReadonlyArray<string>;
+}
+
+const NO_NICKNAMES: ReadonlyArray<string> = Object.freeze([]);
+
+/** The shared empty index, so callers without nickname data allocate once. */
+export const EMPTY_PERSON_NICKNAME_INDEX: PersonNicknameIndex = {
+ nicknamesOf: () => NO_NICKNAMES,
+};
+
+/**
+ * Invert the operator nickname map once: nick-to-names becomes
+ * name-to-nicks, values deduplicated and sorted so the derivation's
+ * output stays independent of the operator map's insertion order. Pure.
+ */
+export function invertPersonNicknameMap(nicknames: PersonNicknameMap): PersonNicknameIndex {
+ const byFullName = new Map<string, string[]>();
+ for (const [nick, fullNames] of nicknames) {
+ for (const fullName of fullNames) {
+ const prior = byFullName.get(fullName);
+ if (prior === undefined) byFullName.set(fullName, [nick]);
+ else if (!prior.includes(nick)) prior.push(nick);
+ }
+ }
+ const frozen = new Map<string, ReadonlyArray<string>>();
+ for (const [fullName, nicks] of byFullName) {
+ frozen.set(fullName, Object.freeze(nicks.toSorted()));
+ }
+ return { nicknamesOf: (fullName) => frozen.get(fullName) ?? NO_NICKNAMES };
+}
+
+/**
+ * The DERIVED-COMPARISON form of an already-normalized person-name
+ * form: a single-code-point token drops a trailing full stop
+ * (`a. lovelace` -> `a lovelace`) and a token that carried a comma
+ * moves behind the rest (`lovelace, ada` -> `ada lovelace`). These are
+ * the two punctuation shapes `normalizeEntityName` deliberately keeps,
+ * and the two spellings in which people actually write names; folding
+ * them on BOTH sides of every derived comparison lets the usual
+ * spellings meet the same forms the plain spelling does. Pure, and a
+ * no-op on forms that carry neither shape.
+ */
+export function personNameComparisonForm(form: string): string {
+ const rest: string[] = [];
+ const carried: string[] = [];
+ for (const token of form.split(" ")) {
+ if (token === "") continue;
+ if (token.endsWith(".")) {
+ const head = token.slice(0, -1);
+ // Exactly one CODE POINT before the stop - measured in code points,
+ // so an astral initial folds too instead of leaking a lone surrogate.
+ if (head !== "" && Array.from(head).length === 1) {
+ rest.push(head);
+ continue;
+ }
+ }
+ if (token.endsWith(",")) {
+ const head = token.slice(0, -1);
+ if (head !== "") carried.push(head);
+ continue;
+ }
+ rest.push(token);
+ }
+ return [...rest, ...carried].join(" ");
+}
+
+/**
+ * The forms a normalized QUERY spelling is compared through in the
+ * derived lane: itself plus its comparison form, deduplicated. Pure.
+ */
+export function personVariantQueryForms(normalizedQuery: string): ReadonlyArray<string> {
+ const comparison = personNameComparisonForm(normalizedQuery);
+ if (comparison === "" || comparison === normalizedQuery) {
+ return Object.freeze([normalizedQuery]);
+ }
+ return Object.freeze([normalizedQuery, comparison]);
+}
+
/**
* The scripts written WITHOUT word spaces - Han, Hiragana, Katakana,
* Thai, Lao, Khmer, Myanmar. Mirrors `NO_SPACE_SCRIPT_RE` in
@@ -98,51 +199,60 @@ const NO_SPACE_SCRIPT_RE =
* order, nicknames sorted. Deterministic and frozen: the same input
* yields byte-identical output on every call and every peer.
*
+ * Structural variants derive from the comparison form of `name` (see
+ * {@link personNameComparisonForm}), so the dotted-initial and
+ * comma-spelled forms derive the same variants their plain spelling
+ * does. The initial is taken by CODE POINT, so an astral-script first
+ * letter reaches the emitted form whole.
+ *
* A name is split on single spaces (normalization collapsed any run).
* Structural variants require at least two tokens AND no
* unspaced-script character anywhere in the name; a variant equal to
- * the input name is never emitted, and every emitted form is at least
- * {@link MIN_PERSON_VARIANT_LENGTH} characters. Nickname expansion
- * consults ONLY `nicknames` - an absent or empty map yields no
+ * the input name or its comparison form is never emitted, and every
+ * emitted STRUCTURAL form is at least {@link MIN_PERSON_VARIANT_LENGTH}
+ * characters. Operator nicknames are exempt from that floor - they are
+ * declared data, not forms this module invented. Nickname expansion
+ * consults ONLY `nicknames` - an absent or empty index yields no
* nickname variants, and no hardcoded list exists anywhere in this
* module. Pure: no I/O, no state.
*/
export function derivePersonNameVariants(
name: string,
- nicknames: PersonNicknameMap = EMPTY_PERSON_NICKNAME_MAP,
+ nicknames: PersonNicknameIndex = EMPTY_PERSON_NICKNAME_INDEX,
): ReadonlyArray<string> {
const out: string[] = [];
const seen = new Set<string>();
- const consider = (form: string): void => {
- if (form.length < MIN_PERSON_VARIANT_LENGTH) return;
+ const comparison = personNameComparisonForm(name);
+ const consider = (form: string, floored: boolean): void => {
+ if (floored && form.length < MIN_PERSON_VARIANT_LENGTH) return;
if (form === name) return;
if (seen.has(form)) return;
seen.add(form);
out.push(form);
};
- const tokens = name.split(" ").filter((token) => token !== "");
- if (tokens.length > 0 && !NO_SPACE_SCRIPT_RE.test(name)) {
+ const tokens = comparison.split(" ").filter((token) => token !== "");
+ if (tokens.length > 0 && !NO_SPACE_SCRIPT_RE.test(comparison)) {
if (tokens.length >= 2) {
- // First-initial-plus-surname: the initial of the first token,
- // then the remaining tokens verbatim ("ada mary lovelace" ->
+ // The plain word-order spelling of a dotted or comma form
+ // ("lovelace, ada" -> "ada lovelace"); a no-op for a form that
+ // already carries neither shape.
+ consider(comparison, true);
+ // First-initial-plus-surname: the initial CODE POINT of the first
+ // token, then the remaining tokens verbatim ("ada mary lovelace" ->
// "a mary lovelace").
- consider(`${tokens[0]!.slice(0, 1)} ${tokens.slice(1).join(" ")}`);
+ const initial = Array.from(tokens[0]!)[0] ?? tokens[0]!;
+ consider(`${initial} ${tokens.slice(1).join(" ")}`, true);
// Reversed word order: "Last, First" read back as words.
- consider(tokens.slice().reverse().join(" "));
+ consider(tokens.slice().reverse().join(" "), true);
}
}
- if (nicknames.size > 0) {
- const nickHits: string[] = [];
- for (const [nick, fullNames] of nicknames) {
- if (!fullNames.includes(name)) continue;
- if (nick.length < MIN_PERSON_VARIANT_LENGTH || nick === name || seen.has(nick)) continue;
- nickHits.push(nick);
- }
- // Sorted so the output is independent of the operator map's
- // insertion order.
- for (const nick of nickHits.toSorted()) consider(nick);
+ for (const nick of nicknames.nicknamesOf(name)) {
+ // Operator-declared nicknames bypass the structural floor; the skips
+ // that remain are the ones that would resolve the person to a form
+ // they already answer to.
+ consider(nick, false);
}
return Object.freeze(out);
diff --git a/src/core/brain/entities/registry.ts b/src/core/brain/entities/registry.ts
index af927e86..0035d4c5 100644
--- a/src/core/brain/entities/registry.ts
+++ b/src/core/brain/entities/registry.ts
@@ -17,8 +17,8 @@
* frontmatter keys the operator may have added by hand.
*/
-import { dirname } from "node:path";
-import { mkdirSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { mkdirSync, statSync } from "node:fs";
import type { FrontmatterMap } from "../../types.ts";
import { parseFrontmatter, slugify, writeFrontmatterAtomic } from "../../vault.ts";
@@ -40,11 +40,20 @@ import {
} from "./canonical.ts";
import { resolveEntityLexicon } from "./lexicon.ts";
import { resolveEntityLabelDenylist } from "./label-hygiene.ts";
-import { resolvePersonNicknameMap } from "./person-nicknames.ts";
+import {
+ PERSON_NICKNAMES_ARTIFACT_REL,
+ PersonNicknameConfigError,
+ resolvePersonNicknameMap,
+} from "./person-nicknames.ts";
import {
derivePersonNameVariants,
+ EMPTY_PERSON_NICKNAME_INDEX,
EMPTY_PERSON_NICKNAME_MAP,
+ invertPersonNicknameMap,
isPersonCategory,
+ personNameComparisonForm,
+ personVariantQueryForms,
+ type PersonNicknameIndex,
type PersonNicknameMap,
} from "./person-variants.ts";
import {
@@ -192,70 +201,229 @@ function matchedLabel(entity: BrainEntity, query: string): string {
* operator to pass one would name an exit that cannot work. The quote
* fold is one way a vault arrives there, and when it is the cause the
* message says so and names the command that ends it.
+ *
+ * The DERIVED lane raises this same type for a different collision: a
+ * form that reaches more than one person only through derivation - an
+ * initial, a word order, an operator nickname. Those records are two
+ * real people, not duplicates, so the remedy is a more specific
+ * reference, not a merge; `derived` selects that message. The `name`
+ * stays the historic one so generic handlers keyed on the string keep
+ * working; callers that must tell the two collisions apart read the
+ * flag.
*/
export class EntityAmbiguityError extends Error {
- constructor(query: string, matches: ReadonlyArray<BrainEntity>) {
+ /** True when the collision exists only through derivation, not declaration. */
+ readonly derived: boolean;
+ constructor(
+ query: string,
+ matches: ReadonlyArray<BrainEntity>,
+ opts: { derived?: boolean } = {},
+ ) {
const ids = matches.map((m) => m.id).join(", ");
- const categories = new Set(matches.map((m) => m.category));
- if (categories.size > 1) {
- super(`entity reference '${query}' is ambiguous across categories: ${ids} - pass a category`);
+ let message: string;
+ if (opts.derived === true) {
+ message =
+ `entity reference '${query}' matches ${matches.length} person records through a ` +
+ `derived name form (an initial, a word order or an operator nickname): ${ids} - ` +
+ "that form alone does not name one person. Query the full name or a declared " +
+ "alias, or pass a category";
} else {
- const normalized = normalizeEntityName(query);
- const tail = quoteVariantCollisionTail(...matches.map((m) => matchedLabel(m, normalized)));
- super(
- `entity reference '${query}' is claimed by ${matches.length} canonical records in one ` +
+ const categories = new Set(matches.map((m) => m.category));
+ if (categories.size > 1) {
+ message = `entity reference '${query}' is ambiguous across categories: ${ids} - pass a category`;
+ } else {
+ const normalized = normalizeEntityName(query);
+ const tail = quoteVariantCollisionTail(...matches.map((m) => matchedLabel(m, normalized)));
+ message =
+ `entity reference '${query}' is claimed by ${matches.length} canonical records in one ` +
`category (${[...categories][0]}): ${ids} - no category separates them, so one must be ` +
- `archived or merged${tail}`,
- );
+ `archived or merged${tail}`;
+ }
}
+ super(message);
this.name = "EntityAmbiguityError";
+ this.derived = opts.derived === true;
+ }
+}
+
+// ----- Derived person-form index ---------------------------------------------
+
+/**
+ * The canonical records that answer to one comparison form in each
+ * lane: `declared` holds the records naming the form outright (name or
+ * alias, any category), `derived` the PERSON records a derivation
+ * kernel form of theirs reaches it through. Both lists are id-ordered,
+ * because the index walk they are built from is.
+ */
+export interface PersonFormClaimants {
+ readonly declared: ReadonlyArray<BrainEntity>;
+ readonly derived: ReadonlyArray<BrainEntity>;
+}
+
+/**
+ * Everything the derived person-form lanes need, built once per
+ * signature: the entity index it was built from (also the entity-change
+ * detector - the cache below reuses the index object identity the
+ * entity-index memo already maintains), the INVERTED nickname index,
+ * the config fault of an unusable nickname file if there was one, and
+ * the comparison-form -> claimants map.
+ */
+export interface PersonDerivedIndex {
+ readonly entityIndex: EntityIndex;
+ readonly nicknames: PersonNicknameIndex;
+ /** Present when `person-nicknames.json` exists but is unusable. */
+ readonly nicknameError: PersonNicknameConfigError | null;
+ readonly byForm: ReadonlyMap<string, PersonFormClaimants>;
+}
+
+function buildPersonDerivedIndex(
+ entityIndex: EntityIndex,
+ nicknames: PersonNicknameIndex,
+ nicknameError: PersonNicknameConfigError | null,
+): PersonDerivedIndex {
+ const byForm = new Map<string, { declared: BrainEntity[]; derived: BrainEntity[] }>();
+ const claim = (form: string, lane: "declared" | "derived", entity: BrainEntity): void => {
+ if (!form) return;
+ let entry = byForm.get(form);
+ if (entry === undefined) {
+ entry = { declared: [], derived: [] };
+ byForm.set(form, entry);
+ }
+ const list = entry[lane];
+ if (!list.some((e) => e.id === entity.id)) list.push(entity);
+ };
+ for (const entity of entityIndex.entities) {
+ if (!entityStatusInScope(entity.status, ENTITY_STATUS_SCOPE.canonical)) continue;
+ const declared = [entity.name, ...entity.aliases].map((f) => normalizeEntityName(f));
+ // Every canonical entity claims its declared forms; only person
+ // categories also derive (the kernel emits nothing for the others).
+ for (const form of declared) {
+ claim(personNameComparisonForm(form), "declared", entity);
+ }
+ if (!isPersonCategory(entity.category)) continue;
+ for (const form of declared) {
+ for (const variant of derivePersonNameVariants(form, nicknames)) {
+ claim(variant, "derived", entity);
+ }
+ }
}
+ return Object.freeze({
+ entityIndex,
+ nicknames,
+ nicknameError,
+ byForm: Object.freeze(byForm),
+ });
+}
+
+interface PersonDerivedCacheEntry {
+ readonly entityIndex: EntityIndex;
+ readonly nicknameSignature: string;
+ readonly value: PersonDerivedIndex;
}
+/** Per-vault memo. Small (a process serves one vault); capped defensively. */
+const PERSON_DERIVED_CACHE = new Map<string, PersonDerivedCacheEntry>();
+const PERSON_DERIVED_CACHE_MAX = 16;
+
+let unstableSignatureCounter = 0;
+
/**
- * Every derived person form `entity` answers to: the variants the
- * derivation kernel emits for each of its declared forms (name and
- * aliases), as a set. Empty by construction for non-person categories,
- * because the kernel emits nothing for them.
+ * The nickname file's change signature: nanosecond mtime plus byte size
+ * when it exists, a fixed marker when it does not. A stat failure other
+ * than a missing file yields a fresh marker each call, so an unreadable
+ * file is never cached - the read path keeps surfacing the fault.
*/
-function derivedPersonForms(
- entity: BrainEntity,
- nicknames: PersonNicknameMap,
-): ReadonlySet<string> {
- const forms = new Set<string>();
- for (const declared of [entity.name, ...entity.aliases]) {
- for (const variant of derivePersonNameVariants(normalizeEntityName(declared), nicknames)) {
- forms.add(variant);
+function personNicknameFileSignature(vault: string): string {
+ try {
+ const st = statSync(join(vault, PERSON_NICKNAMES_ARTIFACT_REL), { bigint: true });
+ return `${st.mtimeNs}\0${st.size}`;
+ } catch (err) {
+ if (typeof err === "object" && err !== null && (err as { code?: string }).code === "ENOENT") {
+ return "missing";
}
+ unstableSignatureCounter += 1;
+ return `unstable-${unstableSignatureCounter}`;
+ }
+}
+
+/**
+ * Resolve the derived person-form index for `vault`, reusing the last
+ * build while neither the entity files nor the nickname file changed.
+ * The entity side is detected through the entity index's own memo
+ * (object identity), the nickname side through its mtime/size
+ * signature - so a per-query expansion, a registry read or a doctor
+ * scan each pay the parse and the derivation once per signature instead
+ * of per call. An unusable nickname file is captured as
+ * `nicknameError` (consumers keep their documented degradations); any
+ * other fault propagates.
+ */
+export function resolvePersonDerivedIndex(vault: string): PersonDerivedIndex {
+ const entityIndex = buildEntityIndex(vault);
+ const nicknameSignature = personNicknameFileSignature(vault);
+ const cached = PERSON_DERIVED_CACHE.get(vault);
+ if (
+ cached !== undefined &&
+ cached.entityIndex === entityIndex &&
+ cached.nicknameSignature === nicknameSignature
+ ) {
+ return cached.value;
+ }
+ let map: PersonNicknameMap;
+ let nicknameError: PersonNicknameConfigError | null = null;
+ try {
+ map = resolvePersonNicknameMap(vault);
+ } catch (err) {
+ if (!(err instanceof PersonNicknameConfigError)) throw err;
+ nicknameError = err;
+ map = EMPTY_PERSON_NICKNAME_MAP;
+ }
+ const value = buildPersonDerivedIndex(
+ entityIndex,
+ map.size > 0 ? invertPersonNicknameMap(map) : EMPTY_PERSON_NICKNAME_INDEX,
+ nicknameError,
+ );
+ if (PERSON_DERIVED_CACHE.size >= PERSON_DERIVED_CACHE_MAX && !PERSON_DERIVED_CACHE.has(vault)) {
+ const oldest = PERSON_DERIVED_CACHE.keys().next().value;
+ if (oldest !== undefined) PERSON_DERIVED_CACHE.delete(oldest);
}
- return forms;
+ PERSON_DERIVED_CACHE.set(vault, { entityIndex, nicknameSignature, value });
+ return value;
}
/**
* The DERIVED lane of canonical read resolution: the person variants
* (write-honesty-egress-consent, F3) - first-initial-plus-surname,
* reversed word order, operator nicknames - consulted only after the
- * exact name/alias lanes above have missed. A unique derived match
+ * exact name/alias lanes above have missed. The lookup joins the query's
+ * forms (itself plus its comparison form - the dotted-initial and
+ * comma spellings) against the shared claimants map, over the declared
+ * lane too: a "Last, First" spelling that comparison-folds onto a
+ * canonical name resolves here rather than nowhere. A unique match
* resolves; more than one is the same ambiguity the exact lane reports,
- * naming both ids. Like the exact lane it resolves canonical-scope
+ * through the derived message - two real people sharing a form are not
+ * duplicates to merge. Like the exact lane it resolves canonical-scope
* records only, so a quarantined person's variants resolve nothing.
*/
-function resolveDerived(
- index: EntityIndex,
- ref: EntityRef,
- nicknames: PersonNicknameMap,
-): BrainEntity | null {
+function resolveDerived(derived: PersonDerivedIndex, ref: EntityRef): BrainEntity | null {
const query = normalizeEntityName(ref.query);
if (!query) return null;
const category = ref.category !== undefined ? validateEntityCategory(ref.category) : undefined;
- const matches = index.entities.filter(
- (e) =>
- entityStatusInScope(e.status, ENTITY_STATUS_SCOPE.canonical) &&
- isPersonCategory(e.category) &&
- (category === undefined || e.category === category) &&
- derivedPersonForms(e, nicknames).has(query),
- );
- if (matches.length > 1) throw new EntityAmbiguityError(ref.query, matches);
+ const matches: BrainEntity[] = [];
+ const seenIds = new Set<string>();
+ for (const form of personVariantQueryForms(query)) {
+ const claimants = derived.byForm.get(form);
+ if (claimants === undefined) continue;
+ for (const entity of [...claimants.declared, ...claimants.derived]) {
+ if (category !== undefined && entity.category !== category) continue;
+ if (seenIds.has(entity.id)) continue;
+ seenIds.add(entity.id);
+ matches.push(entity);
+ }
+ }
+ if (matches.length > 1) {
+ matches.sort((a, b) => a.id.localeCompare(b.id));
+ throw new EntityAmbiguityError(ref.query, matches, { derived: true });
+ }
return matches[0] ?? null;
}
@@ -365,13 +533,13 @@ export function getEntity(vault: string, ref: EntityRef): BrainEntity | null {
const index = buildEntityIndex(vault);
const exact = resolveActive(index, ref);
if (exact !== null) return exact;
- let nicknames = EMPTY_PERSON_NICKNAME_MAP;
+ let derived: PersonDerivedIndex;
try {
- nicknames = resolvePersonNicknameMap(vault);
+ derived = resolvePersonDerivedIndex(vault);
} catch {
return null;
}
- return resolveDerived(index, ref, nicknames);
+ return resolveDerived(derived, ref);
}
/**
diff --git a/src/core/brain/ingest/extract-source.ts b/src/core/brain/ingest/extract-source.ts
index 13f583b8..a86ac620 100644
--- a/src/core/brain/ingest/extract-source.ts
+++ b/src/core/brain/ingest/extract-source.ts
@@ -10,7 +10,7 @@
* section (`## Parts` or `## Table`); for every other source nothing.
*/
-import { pageVisibility } from "../../graph/visibility.ts";
+import { remoteDenyVisibilityTokens } from "../../graph/privacy-markers.ts";
import type { FrontmatterMap } from "../../types.ts";
import { FRONTMATTER_RE, parseFrontmatterText } from "../../vault.ts";
import { isSourceHidden, readSourceOrigin, SourceTrustError } from "../intake/source-trust.ts";
@@ -276,8 +276,15 @@ export function deriveSourceSection(
* construction) and decoded the way the reach predicate
* (`isPathReadableAtReach`) decodes a page: leniently, through the vault's
* frontmatter parser.
+ *
+ * The tokens are {@link remoteDenyVisibilityTokens}, not the declared
+ * `visibility:` alone: an informal privacy marker on the source
+ * (`access: private` and its siblings) is a claim about who may read the
+ * source, and a summary page that copies the source's rows or parts is at
+ * most as visible as the source. Reading the declaration alone would write
+ * those sections onto a default-visibility page - remotely served.
*/
function sourceVisibility(bytes: Uint8Array): readonly string[] {
const [meta] = parseFrontmatterText(UTF8_LENIENT.decode(bytes));
- return pageVisibility(meta);
+ return [...remoteDenyVisibilityTokens(meta)];
}
diff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.ts
index f3825ed1..0a55c8e1 100644
--- a/src/core/brain/ingest/ingest.ts
+++ b/src/core/brain/ingest/ingest.ts
@@ -44,6 +44,8 @@ import {
import { isoSecond } from "../time.ts";
import { sourcePagePath } from "../paths.ts";
import { assertVaultIdentityForWrite } from "../vault-identity.ts";
+import { originChannelStamp } from "../../origin-channel.ts";
+import { CONTENT_CLASS_FIELD, contentClassCreationStamp } from "../provenance/content-class.ts";
import {
intakeExtraction,
type ExtractionIntake,
@@ -427,6 +429,21 @@ export function ingestSource(
updated_at: stamp,
tags: ["brain", "brain/source"],
...visibilityFrontmatter(kept.visibility, derivation?.visibility),
+ // The content class, stamped server-side like every other
+ // deterministic prompt-derived writer's output (the creation merge in
+ // `createNote` is the pattern): the summary prose is the calling
+ // agent's writing, so when this ingest arrived over the MCP transport
+ // - the channel `contentClassCreationStamp` derives the class from -
+ // the page is marked and the shared exports leave it out by default.
+ // An operator-run ingest (the CLI) adds nothing. A page already
+ // carrying the key keeps it verbatim: a re-ingest REWRITES a page its
+ // first publish stamped, and re-deriving the class from the mutating
+ // channel would let a later CLI re-ingest unmark - and an MCP
+ // re-ingest overwrite - what the creating writer decided. The caller
+ // names no frontmatter on this path, so there is nothing to strip.
+ ...(kept.contentClass !== undefined
+ ? { [CONTENT_CLASS_FIELD]: kept.contentClass }
+ : stampContentClass(contentClassCreationStamp(originChannelStamp()))),
};
const body = [
@@ -999,20 +1016,39 @@ function resolvesInsideVault(vault: string, canonicalSource: string): boolean {
interface KeptFrontmatter {
readonly createdAt: string;
readonly visibility?: FrontmatterMap[string];
+ /**
+ * The page's existing content class, kept verbatim on a re-ingest -
+ * the creating writer's decision, whatever it spelled.
+ */
+ readonly contentClass?: FrontmatterMap[string];
}
/**
- * Read the stable `created_at` and an operator-set `visibility` from an
- * existing summary page; `created_at` falls back to `fallback`.
+ * The frontmatter spread for a freshly derived content class, or nothing
+ * when the channel derives no class - the honest byte is NO key.
+ */
+function stampContentClass(
+ contentClass: ReturnType<typeof contentClassCreationStamp>,
+): FrontmatterMap {
+ return contentClass !== null ? { [CONTENT_CLASS_FIELD]: contentClass } : {};
+}
+
+/**
+ * Read the stable `created_at`, an operator-set `visibility`, and any
+ * existing content class from an existing summary page; `created_at`
+ * falls back to `fallback`.
*/
function readKeptFrontmatter(absPath: string, fallback: string): KeptFrontmatter {
const [meta] = parseFrontmatter(absPath);
const value = meta["created_at"];
const createdAt = typeof value === "string" && value.length > 0 ? value : fallback;
const visibility = meta[VISIBILITY_FRONTMATTER_KEY];
- return visibility === undefined || visibility === null
- ? { createdAt }
- : { createdAt, visibility };
+ const contentClass = meta[CONTENT_CLASS_FIELD];
+ return {
+ createdAt,
+ ...(visibility === undefined || visibility === null ? {} : { visibility }),
+ ...(contentClass === undefined ? {} : { contentClass }),
+ };
}
/**
diff --git a/src/core/brain/label-suggest.ts b/src/core/brain/label-suggest.ts
index 9e293709..f725594c 100644
--- a/src/core/brain/label-suggest.ts
+++ b/src/core/brain/label-suggest.ts
@@ -35,7 +35,7 @@ import type { DecisionAnswer, DecisionChoiceQuestion } from "../decision-model/c
import { LABELS_QUESTIONS } from "../decision-model/questions.ts";
import { runDecision } from "../decision-model/run.ts";
import { estimateTokens, mayLeaveMachine } from "../decision-model/state.ts";
-import { pageVisibility } from "../graph/visibility.ts";
+import { remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
import { stripPrivateRegions } from "../redactor.ts";
import { parseFrontmatter } from "../vault.ts";
import { LabelVocabularyError, readLabels } from "./labels.ts";
@@ -178,7 +178,12 @@ export async function suggestNoteLabels(
const mode = decisionModelModeFor(cfg, "labels") as "shadow" | "enforce";
const dimensions = requestedDimensions(opts.pack, opts.dimensions);
- if (!mayLeaveMachine(pageVisibility(meta))) throw new LabelSuggestPrivateNoteError(relPath);
+ // The tokens fold in an informal privacy marker's reservation
+ // (graph/privacy-markers.ts): a marker-only note resolves to no
+ // caller-scope token at all, and without the fold `mayLeaveMachine`
+ // would read it as public and send it.
+ if (!mayLeaveMachine(remoteDenyVisibilityTokens(meta)))
+ throw new LabelSuggestPrivateNoteError(relPath);
const labels = readLabels(meta);
const asked: AskedDimension[] = [];
diff --git a/src/core/brain/log.ts b/src/core/brain/log.ts
index e2f2c198..d5fecfdc 100644
--- a/src/core/brain/log.ts
+++ b/src/core/brain/log.ts
@@ -95,6 +95,37 @@ export interface AppendLogEventResult {
readonly logPath: string;
}
+/**
+ * The wikilink a retirement record stamps as `retired_by`, naming the
+ * log shard THIS device appends to for `date`.
+ *
+ * The link and the event it cites must land on the same file. The
+ * appender resolves its shard with {@link resolveAppendShardId}; a link
+ * built from the bare date (the pre-shard shape) names a file this
+ * device never writes once a device id is configured, and the doctor
+ * reads the retirement's provenance as a broken link. This helper is
+ * the one place that spelling exists, so a writer cannot drift from the
+ * appender: with the empty shard id it emits the exact historical
+ * `[[Brain/log/<date>]]`, with a device id `[[Brain/log/<date>.<id>]]`
+ * - the Obsidian basename of the file {@link logShardPath} writes.
+ *
+ * Pass `deviceId` explicitly when the caller already resolved the shard
+ * for its own `appendLogEvent` call (merge does), so one resolution
+ * feeds both writes. Omitted, the helper resolves the same way the
+ * appender defaults to. Historical records are never rewritten: only
+ * links written after this ship carry the shard segment.
+ */
+export function logShardWikilink(date: string, deviceId?: string): string {
+ const shardId = deviceId ?? resolveAppendShardId();
+ if (shardId !== "" && !isValidDeviceId(shardId)) {
+ throw new Error(
+ `logShardWikilink: invalid deviceId ${JSON.stringify(shardId)} - ` +
+ "expected a lowercase slug matching the device_id config shape",
+ );
+ }
+ return shardId === "" ? `[[Brain/log/${date}]]` : `[[Brain/log/${date}.${shardId}]]`;
+}
+
// ----- Constants ------------------------------------------------------------
// `## HH:MM:SSZ โ€” <kind>` โ€” both the em dash (โ€”) and the hyphen are
diff --git a/src/core/brain/merge.ts b/src/core/brain/merge.ts
index 56d8c3f6..8c11db75 100644
--- a/src/core/brain/merge.ts
+++ b/src/core/brain/merge.ts
@@ -29,7 +29,8 @@ import { regenerateActiveQuiet } from "./active.ts";
import { computeContentHash } from "./content-hash.ts";
import { buildEntityIndex } from "./entities/index-builder.ts";
import { guardEntityMerge, type GuardEntityLike } from "./truth/merge-guard.ts";
-import { appendLogEvent } from "./log.ts";
+import { resolveAppendShardId } from "./ledger-shards.ts";
+import { appendLogEvent, logShardWikilink } from "./log.ts";
import { brainDirs, preferencePath } from "./paths.ts";
import { appendPrefAudit } from "./pref-audit.ts";
import { moveToRetired, parsePreference, writePreference } from "./preference.ts";
@@ -243,9 +244,14 @@ export function mergePreferences(
id: keep.id,
principle: keep.principle,
});
+ // The retired_by link names the log shard the `merge` event below is
+ // about to land in, so the retirement's provenance always resolves.
+ // Resolved once here and handed to both writes: two independent
+ // resolutions are two chances to disagree about the shard.
+ const shardId = resolveAppendShardId();
moveToRetired(vault, dropPath, BRAIN_RETIRED_REASON.mergedInto, {
now,
- retired_by: `[[Brain/log/${isoDate(now)}]]`,
+ retired_by: logShardWikilink(isoDate(now), shardId),
superseded_by: supersededBy,
audit: { agent: opts.agentName ?? "merge" },
});
@@ -273,19 +279,25 @@ export function mergePreferences(
);
}
- // 3. Append a `merge` log event with audit-grade payload.
- appendLogEvent(vault, {
- timestamp: isoSecond(now),
- eventType: BRAIN_LOG_EVENT_KIND.merge,
- body: {
- keep: renderPrefLink({ id: keep.id, principle: keep.principle }),
- drop: renderPrefLink({ id: drop.id, principle: drop.principle }),
- signal_union: `${merged_evidenced_by.length} (was ${keep.evidenced_by.length}, ${drop.evidenced_by.length})`,
- applied_sum: `${applied_sum} (was ${keep.applied_count}, ${drop.applied_count})`,
- violated_sum: `${violated_sum} (was ${keep.violated_count}, ${drop.violated_count})`,
- agent: opts.agentName ?? "unknown",
+ // 3. Append a `merge` log event with audit-grade payload. The same
+ // shard id the retired_by link above names, so the event is where
+ // the link says it is.
+ appendLogEvent(
+ vault,
+ {
+ timestamp: isoSecond(now),
+ eventType: BRAIN_LOG_EVENT_KIND.merge,
+ body: {
+ keep: renderPrefLink({ id: keep.id, principle: keep.principle }),
+ drop: renderPrefLink({ id: drop.id, principle: drop.principle }),
+ signal_union: `${merged_evidenced_by.length} (was ${keep.evidenced_by.length}, ${drop.evidenced_by.length})`,
+ applied_sum: `${applied_sum} (was ${keep.applied_count}, ${drop.applied_count})`,
+ violated_sum: `${violated_sum} (was ${keep.violated_count}, ${drop.violated_count})`,
+ agent: opts.agentName ?? "unknown",
+ },
},
- });
+ { deviceId: shardId },
+ );
// 4. Regenerate active.md so the operator sees the new state on
// their next session start.
diff --git a/src/core/brain/page-lint.ts b/src/core/brain/page-lint.ts
index 734886b1..997e540a 100644
--- a/src/core/brain/page-lint.ts
+++ b/src/core/brain/page-lint.ts
@@ -27,9 +27,11 @@
* - {@link UNDATED_QUANTITY_CODE}, the authored body scanned for a count
* or amount with no structural date on the page (t_fb442a24). The
* trigger shapes live in `undated-quantity.ts`; this module owns the
- * escape hatches - `valid_from` in frontmatter, any ISO date in the
- * body, or a `[Source: <name>, YYYY-MM-DD]` marker - and the finding
- * quotes the lexeme and its offset, never the line. Advisory only.
+ * escape hatches - `valid_from` in frontmatter, an ISO date in the
+ * body's claim-bearing prose, or a `[Source: <name>, YYYY-MM-DD]`
+ * marker - and the finding quotes the lexeme and its body offset,
+ * counted after frontmatter removal and named as such, never the
+ * line. Advisory only.
*
* `demote-stale-stable` is deliberately EXCLUDED, not overlooked: its
* trigger is the page's creation age against the staleness cap, so it can
@@ -116,7 +118,11 @@ import { VALID_FROM_KEY } from "./lifecycle/temporal-replace.ts";
import { tokenise } from "./similarity.ts";
import { parseCitations } from "./temporal/citations.ts";
import type { DoctorSeverity } from "./types.ts";
-import { containsIsoDate, findUndatedQuantities } from "./undated-quantity.ts";
+import {
+ containsIsoDate,
+ containsProseIsoDate,
+ findUndatedQuantities,
+} from "./undated-quantity.ts";
import { WIKILINK_TARGET_RE, isBrainArtifactId, normaliseWikilinkTarget } from "./wikilink.ts";
import { ARTIFACT_MAX_BYTES, validateArtifact } from "./write-session/validate.ts";
@@ -575,11 +581,17 @@ function hasReadableArtifact(vault: string, ctx: LintContext, target: string): b
/**
* Whether the page supplies a structural date that places its quantities
* in time. Three hatches, all structural: a `valid_from` boundary in
- * frontmatter, any ISO date or instant in the body, or a
- * `[Source: <name>, YYYY-MM-DD]` marker. The citation marker always
+ * frontmatter, an ISO date or instant in the body's claim-bearing prose,
+ * or a `[Source: <name>, YYYY-MM-DD]` marker. The citation marker always
* carries its own ISO date, so the date hatch already covers a well-formed
* one; it is checked explicitly through the canonical parser so the
* contract reads as three hatches rather than a coincidence.
+ *
+ * The body hatch reads the same masked view the quantity scanner does
+ * (`containsProseIsoDate`): a date inside a fenced block, an inline code
+ * span, a table row, a URL or a wikilink target is not a claim about the
+ * vault, so it must not quiet the undated quantities around it. The scope
+ * stays page-wide - one prose date anywhere still dates the page.
*/
function hasDatedQuantityContext(meta: FrontmatterMap, body: string): boolean {
const declared = meta[VALID_FROM_KEY];
@@ -587,7 +599,7 @@ function hasDatedQuantityContext(meta: FrontmatterMap, body: string): boolean {
// NAME a date. Any non-empty string ("TBD", "unknown", "soon") places
// nothing in time, and accepting it let a placeholder quiet the lint.
if (typeof declared === "string" && containsIsoDate(declared)) return true;
- if (containsIsoDate(body)) return true;
+ if (containsProseIsoDate(body)) return true;
return parseCitations(body).markers.length > 0;
}
@@ -595,7 +607,11 @@ function hasDatedQuantityContext(meta: FrontmatterMap, body: string): boolean {
* The undated-quantity findings for one written page: one warning per
* lexeme the body states with no date on the page. The lexeme and its
* offset are the whole evidence - the authored line is not quoted, so a
- * receipt cannot carry prose the caller did not ask it to repeat.
+ * receipt cannot carry prose the caller did not ask it to repeat. The
+ * offset is counted within the post-frontmatter body the scanner sees
+ * (neither a file offset nor an index into the raw write), and the
+ * message names that coordinate system so a reader can locate the
+ * lexeme honestly.
*/
function undatedQuantityFindings(
page: string,
@@ -609,7 +625,7 @@ function undatedQuantityFindings(
UNDATED_QUANTITY_CODE,
page,
"body",
- `quantity ${JSON.stringify(lexeme)} at offset ${offset} carries no date`,
+ `quantity ${JSON.stringify(lexeme)} at body offset ${offset} (after frontmatter) carries no date`,
),
);
}
diff --git a/src/core/brain/portability/knowledge-pack.ts b/src/core/brain/portability/knowledge-pack.ts
index 5ee4f796..add5e29e 100644
--- a/src/core/brain/portability/knowledge-pack.ts
+++ b/src/core/brain/portability/knowledge-pack.ts
@@ -46,7 +46,8 @@ import { atomicWriteFileSync } from "../../fs-atomic.ts";
import { EXCLUDED_DIRS, parseFrontmatterText } from "../../vault.ts";
import type { FrontmatterMap } from "../../types.ts";
import { pageVisibility } from "../../graph/visibility.ts";
-import { readContentClass } from "../provenance/content-class.ts";
+import { hasPrivacyDenyMarker } from "../../graph/privacy-markers.ts";
+import { omittedPageReferenceMatcher, readContentClass } from "../provenance/content-class.ts";
import {
PRIVATE_REGION_PLACEHOLDER,
REDACTION_PLACEHOLDER,
@@ -239,6 +240,11 @@ export const PRIVATE_REGION_WARNING = "privacy.private_region_stripped";
export const REDACTED_CONTENT_WARNING = "privacy.redacted_content";
/** The text carries secret-shaped content the export scan would redact. */
export const SECRET_SHAPED_WARNING = "privacy.secret_shaped_content";
+/**
+ * The entry's structured citations named a blocked page and were dropped
+ * from the manifest metadata; the entry's own prose is untouched.
+ */
+export const BLOCKED_PAGE_CITATION_WARNING = "scope.blocked_page_citation_dropped";
export interface KnowledgePackSelection {
readonly selection: ReadonlyArray<string>;
@@ -249,6 +255,11 @@ export interface KnowledgePackSelection {
readonly preferences: ReadonlyArray<ExportedPreferenceRow>;
/** Entries a selector matched but the privacy rules kept out. Never written. */
readonly blocked: ReadonlyArray<KnowledgePackBlocked>;
+ /**
+ * One carried entry's findings, as closed codes: the privacy scan's
+ * codes, and {@link BLOCKED_PAGE_CITATION_WARNING} once per entry whose
+ * manifest citations named a blocked page and were dropped.
+ */
readonly warnings: ReadonlyArray<KnowledgePackWarning>;
}
@@ -277,10 +288,33 @@ function exportWarnings(text: string, id: string): string[] {
* `warnings`. Carried preference rows drop the source vault's evidence
* links and rendered body; carried pages drop this vault's machinery
* frontmatter.
+ *
+ * A carried page's structured citations are not prose: a citation naming
+ * a BLOCKED page is the blocked page's name leaving through the carried
+ * page's manifest metadata, so such citations are dropped - the same
+ * shared matcher the okf, graph and bank exports apply to their own
+ * structured references - and the drop is named per carried entry in
+ * `warnings`. The page body that carries the link stays exactly as
+ * written, and the manifest's `sources` need no second pass: they are
+ * re-translated from the citations here, so a dropped citation resolves
+ * to nothing there either.
*/
+export interface SelectKnowledgePackOptions {
+ /**
+ * The pack's `--include-prompt-derived` override: a page the selection
+ * would block ONLY for the reserved `prompt-derived` class is carried.
+ * The operator-set scopes (`visibility`, `owner`) and the unreviewed
+ * lane are blocks in their own right, not class decisions, and stay
+ * blocks under the override - so a citation naming such a page is
+ * dropped under both paths.
+ */
+ readonly includePromptDerived?: boolean;
+}
+
export function selectKnowledgePack(
vault: string,
selectors: ReadonlyArray<string>,
+ opts: SelectKnowledgePackOptions = {},
): KnowledgePackSelection {
const selection = parseKnowledgePackSelectors(selectors);
if (selection.length === 0) {
@@ -317,7 +351,11 @@ export function selectKnowledgePack(
if (hits.length === 0) continue;
for (const sel of hits) matched.add(sel.raw);
const reason = pageBlockReason(page);
- if (reason !== null) {
+ // The override carries back ONLY the class block: the check itself
+ // stays in `pageBlockReason`, and the decision to keep a page the
+ // block scan marked `prompt-derived` is made here, the same way the
+ // guard's split turns into the identity when the policy includes.
+ if (reason !== null && !(reason === "prompt-derived" && opts.includePromptDerived === true)) {
blocked.push({ kind: "page", id: page.path, reason });
continue;
}
@@ -336,7 +374,28 @@ export function selectKnowledgePack(
);
}
- const okf = buildOkfSubsetBundle(vault, pages);
+ // The block scan is complete, so the blocked page paths are the omitted
+ // list the shared matcher judges citation targets against. A target
+ // that names a blocked page leaves the carried page's citations - the
+ // manifest is where the pack names its references - and each carried
+ // page that lost one is named in `warnings`.
+ const namesBlocked = omittedPageReferenceMatcher(
+ blocked.filter((entry) => entry.kind === "page").map((entry) => entry.id),
+ );
+ const packed: OkfPage[] = [];
+ for (const page of pages) {
+ const citations = page.citations.filter((target) => !namesBlocked(target));
+ if (citations.length !== page.citations.length) {
+ warnings.push({
+ kind: "page",
+ id: page.path,
+ reasons: [BLOCKED_PAGE_CITATION_WARNING],
+ });
+ }
+ packed.push({ ...page, citations });
+ }
+
+ const okf = buildOkfSubsetBundle(vault, packed);
return {
selection,
okfManifest: okf.manifest,
@@ -360,6 +419,12 @@ function preferenceOwner(vault: string, id: string): string | null {
function pageBlockReason(page: OkfPage): KnowledgePackBlockReason | null {
if (pageVisibility(page.frontmatter).length > 0) return "visibility";
+ // An informal privacy marker (`access: private` and its siblings) denies
+ // a remote read just as the reserved token does
+ // (graph/privacy-markers.ts), so a marker-only page - which declares no
+ // `visibility:` at all - is blocked under the same reason rather than
+ // shipped with its body.
+ if (hasPrivacyDenyMarker(page.frontmatter)) return "visibility";
const owner = page.frontmatter["owner"];
if (typeof owner === "string" && owner.trim() !== "") return "owner";
// The reserved class joins the operator-set scopes: a page stamped
diff --git a/src/core/brain/portability/okf.ts b/src/core/brain/portability/okf.ts
index 6d81e930..6c3c3729 100644
--- a/src/core/brain/portability/okf.ts
+++ b/src/core/brain/portability/okf.ts
@@ -78,6 +78,8 @@ import {
import { isoSecond } from "../time.ts";
import { vaultDisplayName } from "../templates.ts";
import { assertVaultIdentityForWrite } from "../vault-identity.ts";
+import { ORIGIN_CHANNEL_FIELD, isOriginChannelStamp } from "../../origin-channel.ts";
+import { contentClassCreationStamp } from "../provenance/content-class.ts";
import { KNOWLEDGE_PACK_FIELD, KNOWLEDGE_PACK_SHA_FIELD } from "./pack-stamp.ts";
import { OPEN_SECOND_BRAIN_VERSION } from "../../version.ts";
@@ -512,9 +514,61 @@ function listMarkdown(root: string, skipDirs: ReadonlyArray<string>): string[] {
* Shards of one day fold into ONE dated section, ordered by shard id
* (`listLogShardFiles` sorts by name), so the bundle is byte-identical
* whatever order Syncthing delivered the shards in.
+ *
+ * ## The content-class filter
+ *
+ * With a {@link OkfLogFilter} the log is NOT whole: the shards hold
+ * agent-written material, and a default export that carried it would
+ * undo, page by page, the omission the class policy just performed. Two
+ * kinds of entry leave, and each drop is counted into the returned
+ * `droppedEntries` so the calling verb can name the count:
+ *
+ * 1. an entry whose `origin_channel` bullet carries the channel
+ * {@link contentClassCreationStamp} derives the reserved class from -
+ * an entry the MCP transport appended is agent-dictated material. A
+ * log entry carries no `content_class` of its own, so the server-
+ * derived channel it is stamped with is the one provable marker it
+ * can carry; a channel this build does not know marks nothing.
+ * 2. an entry whose `target` bullet names a page in the filter's omitted
+ * set - the entry would disclose the very name the split withheld.
+ *
+ * Kept entries are preserved BYTE-EXACTLY: the shard is split at the
+ * event-block boundaries the log parser itself reads (`## ` headings),
+ * and only the dropped spans are removed - no reformatting, no
+ * re-serialisation of what stays. A shard whose every entry is dropped
+ * contributes nothing, and a day left with no contributing shard stops
+ * counting as a log day. Without a filter - and under
+ * `includePromptDerived` - each shard rides whole, exactly as before.
+ */
+export interface OkfLogFilter {
+ /**
+ * The operator override: when true the log rides whole, the same
+ * decision `--include-prompt-derived` makes for pages.
+ */
+ readonly includePromptDerived: boolean;
+ /** The vault-relative paths the page split left out of the bundle. */
+ readonly omittedPaths: ReadonlySet<string>;
+}
+
+export interface OkfLog {
+ /** The date-grouped change log, ready for the `log.md` bundle file. */
+ readonly contents: string;
+ /** Count of DAYS with at least one kept entry, across every shard. */
+ readonly days: number;
+ /** Entries the content-class filter dropped; zero without a filter. */
+ readonly droppedEntries: number;
+}
+
+/**
+ * One top-level log bullet, in the shape the log parser itself reads
+ * (`- key: value`). Spelled locally because the parser does not export
+ * it and this module reads no other log syntax.
*/
-function buildLog(vault: string): { contents: string; days: number } {
+const LOG_BULLET_RE = /^-\s+([A-Za-z_][A-Za-z0-9_-]*)\s*:\s*(.*)$/;
+
+export function buildLog(vault: string, filter?: OkfLogFilter): OkfLog {
const bodiesByDate = new Map<string, string[]>();
+ let droppedEntries = 0;
for (const shard of listLogShardFiles(vault)) {
// The bundle is markdown; the JSONL sidecar of a day carries the same
// entries in machine form and would duplicate every line.
@@ -531,14 +585,97 @@ function buildLog(vault: string): { contents: string; days: number } {
);
}
if (body.length === 0) continue;
+ const text =
+ filter === undefined || filter.includePromptDerived
+ ? body
+ : filterLogShard(body, filter, (dropped) => {
+ droppedEntries += dropped;
+ });
+ if (text.length === 0) continue;
const bodies = bodiesByDate.get(shard.date);
- if (bodies === undefined) bodiesByDate.set(shard.date, [body]);
- else bodies.push(body);
+ if (bodies === undefined) bodiesByDate.set(shard.date, [text]);
+ else bodies.push(text);
}
const sections = [...bodiesByDate.keys()]
.toSorted()
.map((date) => `## ${date}\n\n${bodiesByDate.get(date)!.join("\n\n")}`);
- return { contents: sections.join("\n\n"), days: bodiesByDate.size };
+ return { contents: sections.join("\n\n"), days: bodiesByDate.size, droppedEntries };
+}
+
+/**
+ * Drop the filtered entries from one shard's text, keeping every other
+ * byte where it was. The shard is split at the `## ` event-heading
+ * boundaries the log parser reads; the text before the first heading is
+ * the shard's own frontmatter and title and always stays, and each
+ * entry's span runs to the next heading, so removing an entry removes
+ * exactly its lines and nothing else. Splitting on `\n` and rejoining on
+ * `\n` keeps a hand-edited CRLF shard's kept bytes untouched.
+ */
+function filterLogShard(
+ body: string,
+ filter: OkfLogFilter,
+ recordDrop: (dropped: number) => void,
+): string {
+ const lines = body.split("\n");
+ let firstBlock = -1;
+ for (let i = 0; i < lines.length; i++) {
+ if (lines[i]!.startsWith("## ")) {
+ firstBlock = i;
+ break;
+ }
+ }
+ // No event headings: no entry can be provably marked, so the shard
+ // stays whole rather than being guessed about.
+ if (firstBlock < 0) return body;
+ const starts: number[] = [];
+ for (let i = firstBlock; i < lines.length; i++) {
+ if (lines[i]!.startsWith("## ")) starts.push(i);
+ }
+ const prologue = lines.slice(0, firstBlock).join("\n");
+ const kept: string[] = [];
+ for (let n = 0; n < starts.length; n++) {
+ const end = n + 1 < starts.length ? starts[n + 1]! : lines.length;
+ if (!logEntryIsFilteredOut(lines.slice(starts[n]!, end), filter)) {
+ kept.push(lines.slice(starts[n]!, end).join("\n"));
+ } else {
+ recordDrop(1);
+ }
+ }
+ // Every entry dropped: the shard has nothing left to say, and a bare
+ // frontmatter-plus-title shell would count as a log day that logs
+ // nothing.
+ if (kept.length === 0) return "";
+ return `${prologue}${kept.join("")}`.trimEnd();
+}
+
+/**
+ * Whether one event block leaves under the filter. The block's top-level
+ * bullets are read the way the log parser reads them (`- key: value`,
+ * later occurrences of a key winning), and the decision is fail-open: an
+ * entry that carries neither a known channel nor an omitted target stays.
+ */
+function logEntryIsFilteredOut(blockLines: ReadonlyArray<string>, filter: OkfLogFilter): boolean {
+ const bullets = new Map<string, string>();
+ for (const line of blockLines) {
+ const bullet = LOG_BULLET_RE.exec(line);
+ if (bullet !== null) bullets.set(bullet[1]!, bullet[2]!);
+ }
+ // The channel the creation stamp derives the reserved class from: an
+ // entry appended over the MCP transport is agent-dictated material,
+ // exactly as a page created there is. A bullet naming a channel this
+ // build does not know marks nothing.
+ const channel = bullets.get(ORIGIN_CHANNEL_FIELD);
+ if (
+ typeof channel === "string" &&
+ isOriginChannelStamp(channel) &&
+ contentClassCreationStamp(channel) !== null
+ ) {
+ return true;
+ }
+ // A write entry whose target is the page the split left out would name
+ // the omission right back into the bundle.
+ const target = bullets.get("target");
+ return typeof target === "string" && filter.omittedPaths.has(target);
}
/**
@@ -789,22 +926,33 @@ export function buildOkfSubsetBundle(vault: string, pages: ReadonlyArray<OkfPage
* page-input entry point, but a subset bundle carries no `log.md` by
* contract, so a caller that has ALREADY walked the vault (the shared
* export verbs, which split the collected pages through the egress
- * content-class policy before composition) had no way to keep the
- * change log without a second walk. Composing from the caller's page
- * array closes that double-walk window: between the caller's walk and a
+ * content-class policy before composition) had no way to keep the change
+ * log without a second walk. Composing from the caller's page array
+ * closes that double-walk window: between the caller's walk and a
* re-walk here, a page can change or appear, and the bundle would mix
* the two reads - or, with the class policy applied to the first walk
* only, re-admit a page the policy had just left out. The pages are
- * written as given; only the log is read from the vault here.
+ * written as given.
+ *
+ * The `log` argument is the change log to fold in, already read (and
+ * already filtered through the caller's content-class policy) by
+ * {@link buildLog}. Omitted, the log is read here unfiltered - the
+ * whole-log behavior the callers that have no split in hand rely on - so
+ * a verb that needs the filtered log passes its {@link buildLog} result
+ * through rather than re-reading the vault a second time.
*/
-export function buildOkfBundleFromPages(vault: string, pages: ReadonlyArray<OkfPage>): OkfBundle {
- return assembleOkfBundle(vault, pages, buildLog(vault));
+export function buildOkfBundleFromPages(
+ vault: string,
+ pages: ReadonlyArray<OkfPage>,
+ log: Pick<OkfLog, "contents" | "days"> = buildLog(vault),
+): OkfBundle {
+ return assembleOkfBundle(vault, pages, log);
}
function assembleOkfBundle(
vault: string,
pages: ReadonlyArray<OkfPage>,
- log: { contents: string; days: number } | null,
+ log: Pick<OkfLog, "contents" | "days"> | null,
): OkfBundle {
// ONE instant for the whole bundle: the manifest's `generated_at` and
// every page's `generated.at` share it, so whole-bundle byte equality
diff --git a/src/core/brain/provenance/content-class.ts b/src/core/brain/provenance/content-class.ts
index 6cd7bc89..b8fd59df 100644
--- a/src/core/brain/provenance/content-class.ts
+++ b/src/core/brain/provenance/content-class.ts
@@ -254,6 +254,66 @@ export function isPromptDerivedPage(vault: string, relPath: string): boolean {
return readContentClass(parseFrontmatterText(text)[0]) === CONTENT_CLASS.promptDerived;
}
+/**
+ * A citation target that names an absolute resource, not a vault page.
+ * The same shape the OKF source resolver keys on; a target a URL parser
+ * accepts cannot be a vault page, so it can never name an omitted one.
+ */
+const ABSOLUTE_RESOURCE_RE = /^[A-Za-z][A-Za-z0-9+.-]*:\/\//;
+
+/** Basename of a vault-relative or bare path, separators of either platform dropped. */
+function basenameOf(p: string): string {
+ return p.split(/[\\/]/).pop() ?? p;
+}
+
+/**
+ * The id a bare citation target names a page by: the basename minus one
+ * trailing `.md`. Pages end in `.md`, so this - not a strip of any
+ * extension - is the stem a reader resolves a bare target against.
+ */
+function pageStem(p: string): string {
+ const base = basenameOf(p);
+ return base.replace(/\.md$/i, "");
+}
+
+/**
+ * Build the predicate that answers whether one structured reference
+ * target - a graph link, a typed-relation target, a citation - names one
+ * of the pages a content-class split left out.
+ *
+ * The export verbs split their page collections BEFORE composition, and
+ * that is the whole story for the pages themselves; but a KEPT page can
+ * still name an omitted one in its structured references, and a
+ * reference is a disclosure of the omitted page's name. The verbs drop
+ * those references (never the kept page's prose) and count the drops
+ * into their omission notice; this matcher is the one reading of "does
+ * this target name an omitted page" they share, so the three exporters
+ * cannot grow three answers.
+ *
+ * The matching mirrors how a consumer resolves targets, in three shapes:
+ * an absolute resource (a URL) names no vault page; a path-shaped target
+ * names an omitted page on the exact path, with or without the `.md`
+ * suffix; a bare target names one on the stem. Dropping on ANY stem
+ * collision is the deliberate direction: a target that could name either
+ * a kept or an omitted page is dropped, because a wrong citation is
+ * worse than a missing one.
+ *
+ * Fail-open, like the split's page predicate: a target that matches
+ * nothing in the omitted list names a kept page or a dangling one, and
+ * stays exactly as the kept page wrote it.
+ */
+export function omittedPageReferenceMatcher(
+ omitted: ReadonlyArray<string>,
+): (target: string) => boolean {
+ const paths = new Set(omitted);
+ const stems = new Set(omitted.map(pageStem));
+ return (target: string): boolean => {
+ if (target.length === 0 || ABSOLUTE_RESOURCE_RE.test(target)) return false;
+ if (target.includes("/")) return paths.has(target) || paths.has(`${target}.md`);
+ return stems.has(target) || stems.has(pageStem(target));
+ };
+}
+
/** How deep {@link treeCarriesPromptDerived} descends before it stops. */
const TREE_SEARCH_DEPTH = 8;
diff --git a/src/core/brain/recall-inject-decision.ts b/src/core/brain/recall-inject-decision.ts
index 7afd071c..a28801ed 100644
--- a/src/core/brain/recall-inject-decision.ts
+++ b/src/core/brain/recall-inject-decision.ts
@@ -37,7 +37,7 @@ import type { DecisionProvider, DecisionResponse } from "../decision-model/contr
import { RECALL_INJECT_QUESTIONS } from "../decision-model/questions.ts";
import { runDecision } from "../decision-model/run.ts";
import { buildCandidateState, mayLeaveMachine } from "../decision-model/state.ts";
-import { pageVisibility } from "../graph/visibility.ts";
+import { remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
import { privateRegionTexts, stripPrivateRegions } from "../redactor.ts";
import { readCachedFrontmatterEntry, type FrontmatterCache } from "../search/result-filters.ts";
import { LOCAL_ORIGIN } from "./portability/origins.ts";
@@ -93,7 +93,11 @@ function defaultResolvePage(
return (note) => {
if (!isActiveVaultNote(note)) return { visibility: null, privateRegions: null };
const entry = readCachedFrontmatterEntry(cache, vault, note.path);
- const visibility = entry.unreadable ? null : pageVisibility(entry.meta);
+ // The tokens fold in an informal privacy marker's reservation
+ // (graph/privacy-markers.ts): a marker-only note resolves to no
+ // caller-scope token at all, and without the fold `mayLeaveMachine`
+ // would read it as public and send it.
+ const visibility = entry.unreadable ? null : remoteDenyVisibilityTokens(entry.meta);
if (!mayLeaveMachine(visibility)) return { visibility, privateRegions: null };
let privateRegions: ReadonlyArray<string> | null;
try {
diff --git a/src/core/brain/undated-quantity.ts b/src/core/brain/undated-quantity.ts
index ea3fe899..6736a10c 100644
--- a/src/core/brain/undated-quantity.ts
+++ b/src/core/brain/undated-quantity.ts
@@ -22,22 +22,33 @@
* - a bare thousands-separated count (`12,400`).
*
* Everything else is left alone. A number inside an ISO date or instant, a
- * version string (`v1.73.0`), an ordered-list marker, a table row, an
- * inline code span, a fenced block and the frontmatter block are masked
- * before the scan, because a number there is not a claim about the vault.
- * Precision is deliberate: a missed quantity costs one advisory, a
- * hallucinated one teaches the operator to ignore the list.
+ * version string (`v1.73.0`), a URL, a Markdown link destination, a
+ * wikilink target, an ordered-list marker, a table row, an inline code
+ * span, a fenced block and the frontmatter block are masked before the
+ * scan, because a number there is not a claim about the vault. Precision
+ * is deliberate: a missed quantity costs one advisory, a hallucinated one
+ * teaches the operator to ignore the list.
*
* Pure and deterministic: the same body yields the same lexemes and
- * offsets. An offset is a UTF-16 code-unit index into the caller's own
- * string.
+ * offsets. An offset is a UTF-16 code-unit index into the string handed
+ * to {@link findUndatedQuantities}. On the write path that string is the
+ * page body after the frontmatter block is stripped and the remainder
+ * trimmed, so an offset is neither a file offset nor an index into the
+ * raw text a write call received; the write-time lint names that
+ * coordinate system in its message.
*/
+import { WIKILINK_TARGET_RE } from "./wikilink.ts";
+
/** One quantity the authored body states without a date. */
export interface UndatedQuantity {
/** The matched text exactly as written. */
readonly lexeme: string;
- /** UTF-16 code-unit offset of the first character of {@link lexeme}. */
+ /**
+ * UTF-16 code-unit offset of the first character of {@link lexeme},
+ * counted within the string the caller passed in - after frontmatter
+ * removal on the write path, never a file offset.
+ */
readonly offset: number;
}
@@ -159,6 +170,25 @@ const QUANTITY_UNITS: ReadonlyArray<string> = Object.freeze([
"%",
]);
+/**
+ * Units that read as units with no space before them (`359ms`, `512 MB`,
+ * `35%`): every unit of two characters or more, plus the percent sign,
+ * whose glued spelling is the common one.
+ */
+const GLUED_QUANTITY_UNITS: ReadonlyArray<string> = Object.freeze(
+ QUANTITY_UNITS.filter((unit) => unit.length > 1 || unit === "%"),
+);
+
+/**
+ * The single-letter duration and size units (`3 d`, `72 h`). They need
+ * whitespace before them: glued to the number they stop being units, and
+ * a case-blind match with an optional space turned `3D` and `2D` (a
+ * dimension) and `2b` (a plan) into quantities.
+ */
+const SPACED_QUANTITY_UNITS: ReadonlyArray<string> = Object.freeze(
+ QUANTITY_UNITS.filter((unit) => unit.length === 1 && unit !== "%"),
+);
+
/** ISO currency codes admitted after an amount (`10 USD`). */
const CURRENCY_CODES: ReadonlyArray<string> = Object.freeze([
"USD",
@@ -206,29 +236,45 @@ const ISO_INSTANT_RUN =
* A version-like dotted run: `v1.73.0`, `1.73.0`, `192.168.1.1`. Two or
* more dots is the structural difference from a decimal (`3.14159`), so an
* IP address and a semantic version are both excluded without a table.
+ * The leading lookbehind admits no dot before the first digit: in a run
+ * like `1.1.1.1...` every position after a dot would otherwise be a legal
+ * start, each running the whole tail - a quadratic scan of the
+ * synchronous write path.
*/
-const VERSION_RUN = /(?<![0-9A-Za-z_])v?\d+(?:\.\d+){2,}(?![0-9A-Za-z_.])/iu;
+const VERSION_RUN = /(?<![0-9A-Za-z_.])v?\d+(?:\.\d+){2,}(?![0-9A-Za-z_.])/iu;
/** Currency amount led by a symbol, with an optional magnitude suffix. */
const CURRENCY_SYMBOL_RUN =
/(?<![0-9A-Za-z])[$โ‚ฌยฃยฅ]\s?\d[\d,]*(?:\.\d+)?(?:\s?[KMBT](?![A-Za-z]))?/iu;
-/** Currency amount followed by an ISO code. */
+/**
+ * Currency amount followed by an ISO code. The leading lookbehind admits
+ * no comma before the first digit, for the same bounded-scan reason as
+ * {@link VERSION_RUN}: in a comma-joined digit run every position after a
+ * comma would otherwise be a legal start.
+ */
const CURRENCY_CODE_RUN = new RegExp(
- `(?<![0-9A-Za-z_])[0-9][0-9,]*(?:\\.\\d+)?\\s?(?:${alternation(CURRENCY_CODES)})(?![A-Za-z])`,
+ `(?<![0-9A-Za-z_.,])[0-9][0-9,]*(?:\\.\\d+)?\\s?(?:${alternation(CURRENCY_CODES)})(?![A-Za-z])`,
"iu",
);
-/** Percentage. */
-const PERCENT_RUN = /(?<![0-9A-Za-z_.])\d[\d,]*(?:\.\d+)?\s?%/u;
+/** Percentage, with the same comma-guarded leading lookbehind. */
+const PERCENT_RUN = /(?<![0-9A-Za-z_.,])\d[\d,]*(?:\.\d+)?\s?%/u;
-/** Number followed by a unit from {@link QUANTITY_UNITS}. */
+/**
+ * Number followed by a unit from {@link QUANTITY_UNITS}, with the same
+ * comma-guarded leading lookbehind as the other run shapes. A multi-char
+ * unit (and `%`) may sit glued to the number; the single-letter units
+ * ({@link SPACED_QUANTITY_UNITS}) require whitespace.
+ */
const NUMBER_UNIT_RUN = new RegExp(
// The year guard: a bare four-digit 19xx/20xx number is a YEAR far
// more often than a count ("In 2024 users doubled"), and the lint is
// precision-first. A thousands separator (`2,024 users`) or a longer
// digit run is a number, not a year shape, and still fires.
- `(?<![0-9A-Za-z_.])(?!(?:19|20)\\d{2}\\b)\\d[\\d,]*(?:\\.\\d+)?\\s?(?:${alternation(QUANTITY_UNITS)})(?![0-9A-Za-z_])`,
+ `(?<![0-9A-Za-z_.,])(?!(?:19|20)\\d{2}\\b)\\d[\\d,]*(?:\\.\\d+)?` +
+ `(?:\\s?(?:${alternation(GLUED_QUANTITY_UNITS)})|\\s+(?:${alternation(SPACED_QUANTITY_UNITS)}))` +
+ `(?![0-9A-Za-z_])`,
"iu",
);
@@ -255,6 +301,19 @@ export function containsIsoDate(text: string): boolean {
return ISO_DATE_RUN.test(text) || ISO_INSTANT_RUN.test(text);
}
+/**
+ * A URL run: scheme plus every non-whitespace character after it. The run
+ * ends at whitespace, so it never crosses a line.
+ */
+const URL_RUN = /https?:\/\/\S+/giu;
+
+/**
+ * A Markdown inline link: `[text](destination)`. The bracketed text is
+ * authored prose and stays scanned; only the destination is machine
+ * addressing, where a number names a file or a fragment, not a count.
+ */
+const MD_LINK_RUN = /\[[^\]\n]*\]\(([^)\n]*)\)/gu;
+
interface Range {
readonly start: number;
readonly end: number;
@@ -282,12 +341,16 @@ function hasTablePipe(line: string): boolean {
}
/**
- * The masked spans of a document: frontmatter, fenced blocks, table rows,
- * ordered-list markers, inline code spans, ISO dates and instants, and
- * version-like tokens. Walked line by line because every one of those
- * exclusions except the date and version shapes is a LINE property.
+ * The spans of `text` that carry no authored claim: the frontmatter
+ * block, fenced blocks, table rows, ordered-list markers, inline code
+ * spans, URL runs, Markdown link destinations and wikilink targets.
+ * Walked line by line because most of those exclusions are a LINE
+ * property. The ISO-date and version masks are deliberately NOT applied
+ * here: a date in prose is a dated claim - exactly what the escape hatch
+ * in `page-lint.ts` looks for - so {@link containsProseIsoDate} can read
+ * this view unfiltered.
*/
-function exclusionRanges(text: string): Range[] {
+function structuralExclusionRanges(text: string): Range[] {
const ranges: Range[] = [];
// Frontmatter is a leading fence closed by another fence line. An
@@ -350,12 +413,65 @@ function exclusionRanges(text: string): Range[] {
lineStart = newline + 1;
}
+ // A URL run is machine addressing: its numbers name a path, a query or
+ // a fragment, not a count of vault things.
+ maskMatches(text, URL_RUN, ranges);
+
+ // Markdown link destinations: group 1 sits immediately before the
+ // closing paren of the match.
+ for (const match of text.matchAll(globalCopy(MD_LINK_RUN))) {
+ const dest = match[1] ?? "";
+ if (dest.length === 0) continue;
+ const start = match.index! + match[0].length - 1 - dest.length;
+ ranges.push({ start, end: start + dest.length });
+ }
+
+ // Wikilink targets: group 1 follows the two-character opening bracket.
+ // Only the target is masked here; the `|` alias and `#` anchor suffixes
+ // stay unmasked by this loop, though a line carrying a pipe is already
+ // masked whole as a table row above.
+ for (const match of text.matchAll(globalCopy(WIKILINK_TARGET_RE))) {
+ const target = match[1] ?? "";
+ if (target.length === 0) continue;
+ const start = match.index! + 2;
+ ranges.push({ start, end: start + target.length });
+ }
+
+ return ranges;
+}
+
+/**
+ * The masked spans of a document: every structural exclusion plus the
+ * ISO-date, instant and version-like tokens a quantity scan must not
+ * read as claims.
+ */
+function exclusionRanges(text: string): Range[] {
+ const ranges = structuralExclusionRanges(text);
maskMatches(text, ISO_DATE_RUN, ranges);
maskMatches(text, ISO_INSTANT_RUN, ranges);
maskMatches(text, VERSION_RUN, ranges);
return ranges;
}
+/**
+ * True when `text` carries an ISO date or instant in a region the scanner
+ * reads as authored prose. The escape hatch must see the page the way the
+ * trigger scan does: a date inside a fenced block, an inline code span, a
+ * table row, a URL or a wikilink target is not a claim about the vault,
+ * so it must not quiet the undated quantities around it. The scope stays
+ * page-wide - one prose date anywhere still dates the page.
+ */
+export function containsProseIsoDate(text: string): boolean {
+ const excluded = structuralExclusionRanges(text);
+ for (const pattern of [ISO_DATE_RUN, ISO_INSTANT_RUN]) {
+ for (const match of text.matchAll(globalCopy(pattern))) {
+ const start = match.index!;
+ if (!isExcluded(excluded, start, start + match[0].length)) return true;
+ }
+ }
+ return false;
+}
+
/** True when `[start, end)` overlaps any excluded span. */
function isExcluded(ranges: ReadonlyArray<Range>, start: number, end: number): boolean {
for (const range of ranges) {
diff --git a/src/core/brain/write-batch.ts b/src/core/brain/write-batch.ts
index 3bbf8d5a..9c09b100 100644
--- a/src/core/brain/write-batch.ts
+++ b/src/core/brain/write-batch.ts
@@ -46,7 +46,8 @@ import {
type NoteWriteOp,
} from "./notes/write-record.ts";
import { formatFrontmatter, parseFrontmatterWithNotices } from "../vault.ts";
-import { PRIVACY_MARKER_KEYS } from "../graph/privacy-markers.ts";
+import { hasPrivacyDenyMarker, PRIVACY_MARKER_KEYS } from "../graph/privacy-markers.ts";
+import { normToken } from "../graph/visibility.ts";
import { DEGRADATION_CODE } from "../integrity/degradation.ts";
import { writePathAdvisoryField, type WritePathAdvisoryField } from "./write-path-advisory.ts";
import {
@@ -760,35 +761,53 @@ function projectCreateNote(
}
/**
- * Frontmatter keys a caller-named UPDATE may not set. The first three are
- * read as boundary state, not note content: `visibility` carries the
- * page's read-scope tokens (the reserved `private` among them), and
+ * Frontmatter keys a caller-named UPDATE may not set, whatever the value.
+ * They are read as boundary state, not note content: `visibility` carries
+ * the page's read-scope tokens (the reserved `private` among them), and
* `origin_channel` and `content_class` are the server-derived stamps
- * creation writes once and mutation never rewrites. The marker keys
- * (`access`, `confidentiality`, `sensitivity`) are boundary state for the
- * same reason, one layer over: the read roots resolve a deny value under
- * any of them to the same reserved remote-deny token (`graph/privacy-
- * markers.ts`), so a merged `access: ""` would clear that reservation
- * without the caller ever naming the boundary - and a merged deny value
- * would forge one. The refusal posture is the visibility one: edit the
- * note directly. Creation needs no refusal - the server stamps are merged
- * last there and a caller-named class is dropped outright, and a marker a
- * caller authors at creation is the caller stating the claim about their
- * own new note - but the update merge inverts the order, so the guard
- * lives at the merge.
+ * creation writes once and mutation never rewrites. Keys are matched
+ * through {@link normToken}, so a capitalised spelling (`Visibility:`)
+ * meets the same refusal as the lower-case one.
*/
const RESERVED_UPDATE_FRONTMATTER_KEYS: ReadonlySet<string> = new Set([
"visibility",
ORIGIN_CHANNEL_FIELD,
CONTENT_CLASS_FIELD,
- ...PRIVACY_MARKER_KEYS,
]);
-/** The first reserved key the caller's map names, or `null`. */
-function reservedFrontmatterKey(frontmatter: FrontmatterMap | undefined): string | null {
+/**
+ * The informal privacy marker keys (`access`, `confidentiality`,
+ * `sensitivity`). Unlike {@link RESERVED_UPDATE_FRONTMATTER_KEYS} these are
+ * boundary state one layer over: the read roots resolve a deny value under
+ * any of them to the same reserved remote-deny token
+ * (`graph/privacy-markers.ts`). A caller-named update may rewrite a
+ * marker's VALUE freely while the note's deny status stands still - an
+ * `access: read-only`, or a move from `access: public` to `access: team`,
+ * changes nothing the read roots act on - but a merge that would CLEAR the
+ * note's deny value (a merged `access: ""`) or FORGE one is refused, in
+ * the visibility posture: edit the note directly.
+ *
+ * Creation needs no refusal - the server stamps are merged last there and
+ * a caller-named class is dropped outright, and a marker a caller authors
+ * at creation is the caller stating the claim about their own new note -
+ * but the update merge inverts the order, so the guard lives at the merge.
+ */
+const PRIVACY_MARKER_UPDATE_KEYS: ReadonlySet<string> = new Set(PRIVACY_MARKER_KEYS);
+
+/**
+ * The first key of `frontmatter` whose normalised spelling is in
+ * `reserved`, or `null`. Keys are compared through {@link normToken} - the
+ * same NFC + trim + lower-case normalisation the visibility layer gives a
+ * token - so a spelling the parser keeps distinct (`Access:`, `VISIBILITY:`)
+ * cannot slip a boundary claim past a guard that reads the lower-case form.
+ */
+function reservedFrontmatterKey(
+ frontmatter: FrontmatterMap | undefined,
+ reserved: ReadonlySet<string>,
+): string | null {
if (frontmatter === undefined) return null;
for (const key of Object.keys(frontmatter)) {
- if (RESERVED_UPDATE_FRONTMATTER_KEYS.has(key)) return key;
+ if (reserved.has(normToken(key))) return key;
}
return null;
}
@@ -862,8 +881,12 @@ function projectUpdateNote(
}
// The reserved-key guard rides the same seam: the merge below lets the
// caller's map win over the file's, so a boundary key arriving here
- // would silently rewrite it.
- const reservedKey = reservedFrontmatterKey(op.frontmatter);
+ // would silently rewrite it. The unconditional keys refuse on presence
+ // alone; a marker key refuses only when the merge would change the
+ // note's deny status (a marker the note's deny status does not move -
+ // `access: read-only`, or `access: public` becoming `access: team` -
+ // writes exactly as any other content key does).
+ const reservedKey = reservedFrontmatterKey(op.frontmatter, RESERVED_UPDATE_FRONTMATTER_KEYS);
if (reservedKey !== null) {
throw new WriteBatchError(
"reserved_frontmatter_key",
@@ -873,6 +896,20 @@ function projectUpdateNote(
{ path: target.relPath, key: reservedKey },
);
}
+ const markerKey = reservedFrontmatterKey(op.frontmatter, PRIVACY_MARKER_UPDATE_KEYS);
+ if (markerKey !== null) {
+ const merged: FrontmatterMap = { ...state.frontmatter, ...op.frontmatter };
+ if (hasPrivacyDenyMarker(merged) !== hasPrivacyDenyMarker(state.frontmatter)) {
+ throw new WriteBatchError(
+ "reserved_frontmatter_key",
+ index,
+ `operation ${index}: frontmatter key "${markerKey}" would change whether ` +
+ `${target.relPath} denies a remote read (a deny value under access / ` +
+ "confidentiality / sensitivity is boundary state); edit the note directly",
+ { path: target.relPath, key: markerKey },
+ );
+ }
+ }
const frontmatter =
op.frontmatter !== undefined ? { ...state.frontmatter, ...op.frontmatter } : state.frontmatter;
const body = op.body !== undefined ? op.body : state.body;
diff --git a/src/core/decision-model/advisory.ts b/src/core/decision-model/advisory.ts
index bec7fe71..02e8a409 100644
--- a/src/core/decision-model/advisory.ts
+++ b/src/core/decision-model/advisory.ts
@@ -15,7 +15,7 @@
import { readFileSync } from "node:fs";
import { discoverConfig } from "../config.ts";
-import { pageVisibility } from "../graph/visibility.ts";
+import { remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
import { privateRegionTexts } from "../redactor.ts";
import { parseFrontmatter } from "../vault.ts";
import {
@@ -84,7 +84,12 @@ export function pageEgressFacts(absPath: string | null): PageEgressFacts {
try {
const [meta] = parseFrontmatter(absPath);
const text = readFileSync(absPath, "utf8");
- return { visibility: pageVisibility(meta), privateRegions: privateRegionTexts(text) };
+ // The tokens fold in an informal privacy marker's reservation
+ // (graph/privacy-markers.ts), so a marker-only page cannot leave.
+ return {
+ visibility: remoteDenyVisibilityTokens(meta),
+ privateRegions: privateRegionTexts(text),
+ };
} catch {
return UNREADABLE;
}
diff --git a/src/core/decision-model/config-types.ts b/src/core/decision-model/config-types.ts
index 42597b08..b830b0dd 100644
--- a/src/core/decision-model/config-types.ts
+++ b/src/core/decision-model/config-types.ts
@@ -27,6 +27,44 @@ export type DecisionModelStatus =
export type DecisionModelUses = Readonly<Record<DecisionModelUse, DecisionModelMode>>;
+/**
+ * One entry of the provider chain (`decision_model_providers`, issue
+ * #264): a preset resolved against its own defaults, with the endpoint
+ * override keys applied to the first entry only. The top-level fields of
+ * {@link ResolvedDecisionModelConfig} mirror the FIRST entry, so readers
+ * written for the single-provider era keep working unchanged.
+ */
+export interface ResolvedDecisionProvider {
+ /** The preset name, as configured. */
+ readonly name: string;
+ readonly adapter: DecisionModelAdapter;
+ readonly baseUrl: string | null;
+ readonly model: string | null;
+ /** NAME of the env var holding this entry's key; never the value. */
+ readonly envKey: string | null;
+ readonly keyPresent: boolean;
+ /** False only for a loopback server whose preset makes the key optional. */
+ readonly keyRequired: boolean;
+ readonly maxStateTokens: number;
+ readonly maxChoiceOptions: number;
+ readonly calibrated: boolean;
+ readonly inputPriceUsdPerMtok: number | null;
+ readonly outputPriceUsdPerMtok: number | null;
+ /** The threshold profile this entry's answers are held to; null when unknown. */
+ readonly thresholdProfile: string | null;
+ readonly licenceNote: string | null;
+ /** Who processes the data; null when no preset resolves. */
+ readonly processor: string | null;
+ /**
+ * The effective per-use modes held to THIS entry's thresholds: an
+ * `enforce` setting without tuned thresholds for this entry's model is
+ * `shadow` here.
+ */
+ readonly uses: DecisionModelUses;
+ /** Uses configured `enforce` that run as `shadow` on this entry. */
+ readonly shadowOnlyUses: ReadonlyArray<DecisionModelUse>;
+}
+
export interface ResolvedDecisionModelConfig {
readonly status: DecisionModelStatus;
/** Config errors, each naming its key. Non-empty only when `invalid`. */
@@ -38,6 +76,13 @@ export interface ResolvedDecisionModelConfig {
readonly adapter: DecisionModelAdapter;
readonly baseUrl: string | null;
readonly model: string | null;
+ /**
+ * The ordered provider chain (`decision_model_providers`), first entry
+ * first. Absent on configs built before the chain existed (hand-built
+ * test configs); {@link resolvedProviderEntries} then derives the
+ * single implied entry from the top-level fields.
+ */
+ readonly providers?: ReadonlyArray<ResolvedDecisionProvider>;
/** NAME of the env var holding the key; never the key. */
readonly envKey: string | null;
readonly keyPresent: boolean;
diff --git a/src/core/decision-model/config.ts b/src/core/decision-model/config.ts
index ca609add..b9f568c3 100644
--- a/src/core/decision-model/config.ts
+++ b/src/core/decision-model/config.ts
@@ -48,6 +48,7 @@ import {
DECISION_MODEL_CHAT_SHAPED_ADAPTERS,
DECISION_MODEL_HOSTED_MAX_STATE_TOKENS,
DECISION_MODEL_PRESET_NAMES,
+ DECISION_MODEL_PRESETS,
DECISION_MODEL_WIRE_MAX_CHOICE_OPTIONS,
decisionModelPreset,
} from "./presets.ts";
@@ -57,6 +58,13 @@ import {
isDecisionThresholdProfile,
thresholdsTunedFor,
} from "./questions.ts";
+import {
+ decisionThresholdsPath,
+ EMPTY_THRESHOLD_STORE,
+ readThresholdsStore,
+ storedThresholdEntry,
+ type DecisionThresholdsStore,
+} from "./thresholds.ts";
export const DECISION_MODEL_DEFAULTS = Object.freeze({
timeoutMs: 3000,
@@ -69,10 +77,12 @@ export type {
DecisionModelUses,
ResolvedDecisionModelConfig,
} from "./config-types.ts";
+export type { ResolvedDecisionProvider } from "./config-types.ts";
import type {
DecisionModelStatus,
DecisionModelUses,
ResolvedDecisionModelConfig,
+ ResolvedDecisionProvider,
} from "./config-types.ts";
const ENV_PREFIX = "OPEN_SECOND_BRAIN_DECISION_MODEL_";
@@ -118,20 +128,16 @@ function isLoopbackUrl(raw: string | null): boolean {
/**
* The modes the uses actually run in: `enforce` becomes `shadow` for every
- * use whose thresholds are not tuned for `profile` - unless the
- * injected measured reader says the preregistered eval has marked the
- * pair tuned (the machine-side tuning mark, `eval/mark.ts`).
+ * use the `tunedFor` gate refuses. The gate (see {@link entryTunedGate})
+ * is the source table, the local thresholds store and the injected
+ * measured reader (the machine-side tuning mark, `eval/mark.ts`) OR-ed
+ * per use.
*/
function holdToProfile(
uses: DecisionModelUses,
- profile: string | null,
- measuredTuned?: ((use: DecisionModelUse) => boolean) | null,
+ tunedFor: (use: DecisionModelUse) => boolean,
): { readonly uses: DecisionModelUses; readonly shadowOnly: ReadonlyArray<DecisionModelUse> } {
- const shadowOnly = DECISION_MODEL_USES.filter(
- (use) =>
- uses[use] === "enforce" &&
- !thresholdsTunedFor(profile, use, measuredTuned ? () => measuredTuned(use) : null),
- );
+ const shadowOnly = DECISION_MODEL_USES.filter((use) => uses[use] === "enforce" && !tunedFor(use));
if (shadowOnly.length === 0) return { uses, shadowOnly: Object.freeze([]) };
const out: Record<DecisionModelUse, DecisionModelMode> = { ...uses };
for (const use of shadowOnly) out[use] = "shadow";
@@ -286,6 +292,56 @@ export interface ResolveDecisionModelOptions {
* test pins.
*/
readonly measuredTunedMark?: ((profile: string | null, use: DecisionModelUse) => boolean) | null;
+ /**
+ * Where the local thresholds store (`decision-thresholds.json`,
+ * `thresholds.ts`) lives. Undefined reads the file beside the default
+ * config path - the production behavior; a path reads that file (the
+ * CLI passes the `--config` sibling); null disables the store, which
+ * is the behavior the hand-edit tests pin.
+ */
+ readonly thresholdsPath?: string | null;
+}
+
+/**
+ * The thresholds store this resolution gates against, read lazily (only
+ * an enforce hold that the table refuses consults it) and TTL-cached
+ * inside the store module. `null` (the option set to null) disables the
+ * store: the empty store tunes nothing.
+ */
+function thresholdsStoreFor(opts: ResolveDecisionModelOptions): DecisionThresholdsStore {
+ if (opts.thresholdsPath === null) return EMPTY_THRESHOLD_STORE;
+ if (opts.thresholdsPath !== undefined) return readThresholdsStore(opts.thresholdsPath);
+ // The file beside the config path this process resolves: an explicit
+ // OPEN_SECOND_BRAIN_CONFIG in the resolution's environment names it, as
+ // it does for every other config read; otherwise the process's default
+ // discovery (env, XDG, home) applies. Lazy, as in the policy require
+ // above: core/config is a large module and only this store read needs
+ // its discovery chain.
+ const env = opts.env ?? process.env;
+ const explicit = env["OPEN_SECOND_BRAIN_CONFIG"];
+ const coreConfig = require("../config.ts") as typeof import("../config.ts");
+ const configPath =
+ typeof explicit === "string" && explicit !== "" ? explicit : coreConfig.defaultConfigPath();
+ return readThresholdsStore(decisionThresholdsPath(configPath));
+}
+
+/**
+ * The tuned gate for one chain entry: the source table (baseline or a
+ * hand-registered list) OR the local store entry for THIS entry's model
+ * id, keyed by the model the server reports - never the preset - OR the
+ * measured mark. The store is consulted only when the table refuses, so
+ * a baseline model pays no store read.
+ */
+function entryTunedGate(
+ store: DecisionThresholdsStore,
+ profile: string | null,
+ model: string | null,
+ vault: string | null,
+ measuredTuned: ((use: DecisionModelUse) => boolean) | null,
+): (use: DecisionModelUse) => boolean {
+ return (use) =>
+ thresholdsTunedFor(profile, use, measuredTuned === null ? null : () => measuredTuned(use)) ||
+ storedThresholdEntry(store, model, use, vault) !== null;
}
function defaultVaultOptOut(vault: string): boolean {
@@ -320,14 +376,66 @@ export function resolveDecisionModelConfig(
notes.push(`decision_model_enabled '${enabledRaw}' is not 'true', so the feature is off`);
}
+ // The ordered provider chain (`decision_model_providers`, issue #264).
+ // `decision_model_provider` stays the one-element form; when both are
+ // set the list wins and a note says so. Parsing collects errors naming
+ // the key; the list is only non-empty when every entry parsed.
+ const providersRaw = setting(env, config, "providers");
const providerRaw = setting(env, config, "provider");
- const preset = providerRaw !== null ? decisionModelPreset(providerRaw) : null;
- const explicitBaseUrl = setting(env, config, "base_url");
+ const chainNames: string[] = [];
+ if (providersRaw !== null) {
+ const seen = new Set<string>();
+ for (const part of providersRaw.split(",")) {
+ const name = part.trim();
+ if (name === "") continue;
+ if (!Object.hasOwn(DECISION_MODEL_PRESETS, name)) {
+ errors.push(
+ `decision_model_providers must be one of ${DECISION_MODEL_PRESET_NAMES.join(", ")}, ` +
+ `got '${name}'`,
+ );
+ } else if (seen.has(name)) {
+ errors.push(`decision_model_providers: provider '${name}' is listed more than once`);
+ } else {
+ seen.add(name);
+ chainNames.push(name);
+ }
+ }
+ if (chainNames.length === 0) {
+ errors.push(
+ `decision_model_providers must name at least one provider preset ` +
+ `(one of ${DECISION_MODEL_PRESET_NAMES.join(", ")})`,
+ );
+ }
+ if (providerRaw !== null && providerRaw !== (chainNames[0] ?? null)) {
+ notes.push(
+ `decision_model_provider '${providerRaw}' is ignored while decision_model_providers ` +
+ "is set; the list decides the order",
+ );
+ }
+ }
+ const primaryName = chainNames[0] ?? providerRaw;
+
+ // Per-entry endpoint overrides (`decision_model_providers_<preset>
+ // _base_url|_id|_env_key`): any entry can name its own endpoint, so a
+ // preset without one (compatible) can sit anywhere in the chain and a
+ // preset with a default (typesafe, laya) can be pointed elsewhere.
+ // Config only: the chain is a deployment fact, so these keys have no
+ // environment override.
+ const entryOverride = (name: string, suffix: string): string | null => {
+ const value = config[`decision_model_providers_${name}_${suffix}`];
+ return typeof value === "string" && value.trim() !== "" ? value : null;
+ };
+ const primaryBaseOverride = primaryName !== null ? entryOverride(primaryName, "base_url") : null;
+ const primaryIdOverride = primaryName !== null ? entryOverride(primaryName, "id") : null;
+ const primaryEnvOverride = primaryName !== null ? entryOverride(primaryName, "env_key") : null;
+
+ const preset = primaryName !== null ? decisionModelPreset(primaryName) : null;
+ const explicitBaseUrl = primaryBaseOverride ?? setting(env, config, "base_url");
const baseUrlRaw = explicitBaseUrl ?? preset?.baseUrl ?? null;
const cleanedUrl = baseUrlRaw !== null ? withoutUserinfo(baseUrlRaw.replace(/\/+$/, "")) : null;
const baseUrl = cleanedUrl?.url ?? null;
- const model = setting(env, config, "id") ?? preset?.model ?? null;
- const envKeyRaw = setting(env, config, "env_key") ?? preset?.envKey ?? null;
+ const model = primaryIdOverride ?? setting(env, config, "id") ?? preset?.model ?? null;
+ const envKeyRaw = primaryEnvOverride ?? setting(env, config, "env_key") ?? preset?.envKey ?? null;
const envKeyValid = envKeyRaw === null || ENV_VAR_NAME_RE.test(envKeyRaw);
// An invalid name is never looked up and never reported back.
const envKey = envKeyValid ? envKeyRaw : null;
@@ -345,7 +453,7 @@ export function resolveDecisionModelConfig(
errors: Object.freeze([]),
notes: Object.freeze(notes),
enabled: false,
- provider: providerRaw,
+ provider: primaryName,
adapter: preset?.adapter ?? "systemone",
baseUrl,
model,
@@ -375,30 +483,58 @@ export function resolveDecisionModelConfig(
});
}
- if (providerRaw === null) {
- errors.push(
- `decision_model_provider is required when decision_model_enabled is true ` +
- `(one of ${DECISION_MODEL_PRESET_NAMES.join(", ")})`,
- );
- } else if (preset === null) {
- errors.push(
- `decision_model_provider must be one of ${DECISION_MODEL_PRESET_NAMES.join(", ")}, ` +
- `got '${providerRaw}'`,
- );
+ if (providersRaw === null) {
+ if (providerRaw === null) {
+ errors.push(
+ `decision_model_provider is required when decision_model_enabled is true ` +
+ `(one of ${DECISION_MODEL_PRESET_NAMES.join(", ")})`,
+ );
+ } else if (preset === null) {
+ errors.push(
+ `decision_model_provider must be one of ${DECISION_MODEL_PRESET_NAMES.join(", ")}, ` +
+ `got '${providerRaw}'`,
+ );
+ }
}
+ // The first entry's endpoint keys: the single form's messages stay
+ // byte-identical; the list form names the per-entry key beside the
+ // global one, since both configure this entry.
if (preset !== null && baseUrl === null) {
- errors.push(`decision_model_base_url is required for provider '${preset.name}'`);
+ if (providersRaw === null) {
+ errors.push(`decision_model_base_url is required for provider '${preset.name}'`);
+ } else {
+ errors.push(
+ `decision_model_providers: provider '${preset.name}' needs an endpoint: set ` +
+ `decision_model_providers_${preset.name}_base_url (or decision_model_base_url, ` +
+ "which the first provider reads)",
+ );
+ }
}
if (preset !== null && model === null) {
- errors.push(`decision_model_id is required for provider '${preset.name}'`);
+ if (providersRaw === null) {
+ errors.push(`decision_model_id is required for provider '${preset.name}'`);
+ } else {
+ errors.push(
+ `decision_model_providers: provider '${preset.name}' needs a pinned model: set ` +
+ `decision_model_providers_${preset.name}_id (or decision_model_id, which the ` +
+ "first provider reads)",
+ );
+ }
}
if (model !== null && /(^|[-/])latest$/i.test(model)) {
errors.push(`decision_model_id must pin a model version, not a moving alias ('${model}')`);
}
if (!envKeyValid) {
+ const perEntryKey =
+ primaryName !== null &&
+ primaryEnvOverride !== null &&
+ !ENV_VAR_NAME_RE.test(primaryEnvOverride)
+ ? `decision_model_providers_${primaryName}_env_key`
+ : null;
errors.push(
- "decision_model_env_key must be the NAME of an environment variable " +
+ (perEntryKey ?? "decision_model_env_key") +
+ " must be the NAME of an environment variable " +
"(A-Z, 0-9 and _), not the key itself; the value is not shown",
);
}
@@ -507,7 +643,7 @@ export function resolveDecisionModelConfig(
if (enforced.length > 0) {
errors.push(
`decision_model_uses: enforce (${enforced.join(", ")}) needs a calibrated provider; ` +
- `set decision_model_allow_uncalibrated: true to allow provider '${providerRaw}'`,
+ `set decision_model_allow_uncalibrated: true to allow provider '${primaryName}'`,
);
}
}
@@ -529,27 +665,50 @@ export function resolveDecisionModelConfig(
} else if (profileRaw !== thresholdProfile) {
notes.push(
`decision_model_threshold_profile is read only for provider 'compatible'; ` +
- `provider '${providerRaw}' keeps its own profile (${thresholdProfile ?? "none"})`,
+ `provider '${primaryName}' keeps its own profile (${thresholdProfile ?? "none"})`,
);
}
}
+ // The measured-mark reader bound to one entry's endpoint. Disabled
+ // reader (null): the table alone decides. Default (undefined): the
+ // machine-side marks, TTL-cached inside the mark module, bound to THIS
+ // entry's endpoint - a mark earned by another provider, model or base
+ // URL never arms enforce here.
+ const markReaderFor = (
+ endpoint: {
+ readonly provider: string | null;
+ readonly model: string | null;
+ readonly baseUrl: string | null;
+ },
+ profile: string | null,
+ ): ((use: DecisionModelUse) => boolean) | null => {
+ if (profile === null) return null;
+ return (use) => {
+ if (opts.measuredTunedMark === null) return false;
+ const reader = opts.measuredTunedMark ?? machineTunedMarkReader(endpoint);
+ return reader(profile, use);
+ };
+ };
+
+ // The store is read only when a use is configured enforce: shadow and
+ // off installs pay no file read at all.
+ const needsGate = DECISION_MODEL_USES.some((use) => configured[use] === "enforce");
+ let storeCache: DecisionThresholdsStore | null = null;
+ const store = (): DecisionThresholdsStore => {
+ if (storeCache === null) storeCache = thresholdsStoreFor(opts);
+ return storeCache;
+ };
const held = holdToProfile(
configured,
- thresholdProfile,
- thresholdProfile === null
- ? null
- : (use) => {
- // Disabled reader (null): the table alone decides. Default
- // (undefined): the machine-side marks, TTL-cached inside the
- // mark module, bound to THIS resolution's endpoint - a mark
- // earned by another provider, model or base URL never arms
- // enforce here.
- if (opts.measuredTunedMark === null) return false;
- const reader =
- opts.measuredTunedMark ??
- machineTunedMarkReader({ provider: providerRaw, model, baseUrl });
- return reader(thresholdProfile, use);
- },
+ needsGate
+ ? entryTunedGate(
+ store(),
+ thresholdProfile,
+ model,
+ opts.vault,
+ markReaderFor({ provider: primaryName, model, baseUrl }, thresholdProfile),
+ )
+ : () => true,
);
const uses = held.uses;
@@ -557,9 +716,149 @@ export function resolveDecisionModelConfig(
// the machine does.
const keyRequired = !(preset?.keyOptionalOnLoopback === true && isLoopbackUrl(baseUrl));
+ // The first chain entry, as resolved above from the preset plus the
+ // endpoint override keys. Its fields mirror the config's top level, so
+ // every reader written for the single-provider era keeps working.
+ const firstEntry: ResolvedDecisionProvider = {
+ name: primaryName ?? "unknown",
+ adapter: preset?.adapter ?? "systemone",
+ baseUrl,
+ model,
+ envKey,
+ keyPresent,
+ keyRequired,
+ maxStateTokens,
+ maxChoiceOptions: preset?.maxChoiceOptions ?? DECISION_MODEL_WIRE_MAX_CHOICE_OPTIONS,
+ calibrated,
+ inputPriceUsdPerMtok,
+ outputPriceUsdPerMtok: outputPriceUsdPerMtok ?? null,
+ thresholdProfile,
+ licenceNote: preset?.licenceNote ?? null,
+ processor: preset?.processor ?? null,
+ uses,
+ shadowOnlyUses: held.shadowOnly,
+ };
+
+ // Entries after the first resolve like the first one, minus the global
+ // endpoint keys: preset defaults, overridden per entry by
+ // `decision_model_providers_<preset>_*`. An entry that cannot resolve
+ // an endpoint is a config error naming its key - a silently skipped
+ // fallback would hide the typo.
+ const laterEntries: ResolvedDecisionProvider[] = [];
+ for (const [index, name] of chainNames.entries()) {
+ if (index === 0) continue;
+ const laterPreset = DECISION_MODEL_PRESETS[name];
+ if (laterPreset === undefined) continue;
+ const laterUrlRaw = entryOverride(name, "base_url") ?? laterPreset.baseUrl;
+ const laterModel = entryOverride(name, "id") ?? laterPreset.model;
+ const laterEnvKeyRaw = entryOverride(name, "env_key") ?? laterPreset.envKey;
+ if (laterUrlRaw === null) {
+ errors.push(
+ `decision_model_providers: provider '${name}' needs an endpoint: set ` +
+ `decision_model_providers_${name}_base_url (its preset carries none)`,
+ );
+ continue;
+ }
+ if (laterModel === null) {
+ errors.push(
+ `decision_model_providers: provider '${name}' needs a pinned model: set ` +
+ `decision_model_providers_${name}_id`,
+ );
+ continue;
+ }
+ if (/(^|[-/])latest$/i.test(laterModel)) {
+ errors.push(
+ `decision_model_providers_${name}_id must pin a model version, ` +
+ `not a moving alias ('${laterModel}')`,
+ );
+ continue;
+ }
+ const laterEnvKeyValid = laterEnvKeyRaw === null || ENV_VAR_NAME_RE.test(laterEnvKeyRaw);
+ if (!laterEnvKeyValid) {
+ errors.push(
+ `decision_model_providers_${name}_env_key must be the NAME of an environment ` +
+ "variable (A-Z, 0-9 and _), not the key itself; the value is not shown",
+ );
+ continue;
+ }
+ const laterCleaned = withoutUserinfo(laterUrlRaw.replace(/\/+$/, ""));
+ if (laterCleaned.hadUserinfo) {
+ errors.push(
+ `decision_model_providers_${name}_base_url must not carry user:password@ ` +
+ "credentials; the key is sent in a header from its env_key",
+ );
+ continue;
+ }
+ try {
+ assertHttpEgressEndpoint(laterCleaned.url, `decision_model_providers_${name}_base_url`, {
+ allowInsecureHttp,
+ key: "decision_model_allow_insecure_http",
+ });
+ } catch (err) {
+ errors.push((err as Error).message);
+ continue;
+ }
+ const laterKeyValue = resolveKeyValue(
+ laterEnvKeyRaw !== null ? env[laterEnvKeyRaw] : undefined,
+ opts.secretsVault,
+ errors,
+ );
+ const laterKeyPresent = typeof laterKeyValue === "string" && laterKeyValue.trim() !== "";
+ const laterKeyRequired = !(
+ laterPreset.keyOptionalOnLoopback === true && isLoopbackUrl(laterCleaned.url)
+ );
+ const laterHeld = holdToProfile(
+ configured,
+ entryTunedGate(
+ store(),
+ laterPreset.thresholdProfile,
+ laterModel,
+ opts.vault,
+ markReaderFor(
+ { provider: laterPreset.name, model: laterModel, baseUrl: laterCleaned.url },
+ laterPreset.thresholdProfile,
+ ),
+ ),
+ );
+ laterEntries.push({
+ name: laterPreset.name,
+ adapter: laterPreset.adapter,
+ baseUrl: laterCleaned.url,
+ model: laterModel,
+ envKey: laterEnvKeyValid ? laterEnvKeyRaw : null,
+ keyPresent: laterKeyPresent,
+ keyRequired: laterKeyRequired,
+ maxStateTokens: laterPreset.defaultStateTokens ?? laterPreset.maxStateTokens,
+ maxChoiceOptions: laterPreset.maxChoiceOptions,
+ calibrated: laterPreset.calibrated,
+ inputPriceUsdPerMtok: laterPreset.inputPriceUsdPerMtok,
+ outputPriceUsdPerMtok: null,
+ thresholdProfile: laterPreset.thresholdProfile,
+ licenceNote: laterPreset.licenceNote,
+ processor: laterPreset.processor,
+ uses: laterHeld.uses,
+ shadowOnlyUses: laterHeld.shadowOnly,
+ });
+ }
+
+ const providers = Object.freeze([firstEntry, ...laterEntries]);
+ // An entry without its key is not configured: it is skipped at call
+ // time, and the feature is only off when NO entry is configured.
+ const anyConfigured = providers.some((p) => p.keyPresent || !p.keyRequired);
+ if (anyConfigured) {
+ for (const p of providers) {
+ if (!p.keyPresent && p.keyRequired) {
+ notes.push(
+ `decision_model_providers: provider '${p.name}' has no key in ` +
+ `${p.envKey ?? "its key variable"}; it is skipped until the variable is set`,
+ );
+ }
+ }
+ }
+
let status: DecisionModelStatus;
if (errors.length > 0) status = "invalid";
- else if (!keyPresent && keyRequired) status = "no_key";
+ else if (!anyConfigured) status = "no_key";
else if (opts.vault !== null && (opts.vaultOptOut ?? defaultVaultOptOut)(opts.vault)) {
status = "disabled_by_vault";
} else status = "active";
@@ -569,8 +868,9 @@ export function resolveDecisionModelConfig(
errors: Object.freeze(errors),
notes: Object.freeze(notes),
enabled: true,
- provider: providerRaw,
- adapter: preset?.adapter ?? "systemone",
+ provider: primaryName,
+ providers,
+ adapter: firstEntry.adapter,
baseUrl,
model,
envKey,
@@ -580,7 +880,7 @@ export function resolveDecisionModelConfig(
timeoutMs,
hookBudgetMs,
maxStateTokens,
- maxChoiceOptions: preset?.maxChoiceOptions ?? DECISION_MODEL_WIRE_MAX_CHOICE_OPTIONS,
+ maxChoiceOptions: firstEntry.maxChoiceOptions,
uses,
configuredUses: configured,
dailyCostGateUsd,
@@ -590,8 +890,8 @@ export function resolveDecisionModelConfig(
calibrated,
thresholdProfile,
shadowOnlyUses: held.shadowOnly,
- licenceNote: preset?.licenceNote ?? null,
- processor: preset?.processor ?? null,
+ licenceNote: firstEntry.licenceNote,
+ processor: firstEntry.processor,
vault: opts.vault,
});
}
@@ -607,3 +907,37 @@ export function decisionModelModeFor(
if (cfg === null || cfg === undefined || cfg.status !== "active") return "off";
return cfg.uses[use];
}
+
+/**
+ * The provider chain of a resolved config, first entry first. A config
+ * resolved from `decision_model_providers` carries its entries; a config
+ * built before the chain existed (hand-built test configs, older stored
+ * objects) derives the single implied entry from its top-level fields, so
+ * the callers of the chain behave identically on both shapes.
+ */
+export function resolvedProviderEntries(
+ cfg: ResolvedDecisionModelConfig,
+): ReadonlyArray<ResolvedDecisionProvider> {
+ if (cfg.providers !== undefined && cfg.providers.length > 0) return cfg.providers;
+ return Object.freeze([
+ {
+ name: cfg.provider ?? "unknown",
+ adapter: cfg.adapter,
+ baseUrl: cfg.baseUrl,
+ model: cfg.model,
+ envKey: cfg.envKey,
+ keyPresent: cfg.keyPresent,
+ keyRequired: cfg.keyRequired !== false,
+ maxStateTokens: cfg.maxStateTokens,
+ maxChoiceOptions: cfg.maxChoiceOptions ?? DECISION_MODEL_WIRE_MAX_CHOICE_OPTIONS,
+ calibrated: cfg.calibrated,
+ inputPriceUsdPerMtok: cfg.inputPriceUsdPerMtok,
+ outputPriceUsdPerMtok: cfg.outputPriceUsdPerMtok ?? null,
+ thresholdProfile: cfg.thresholdProfile ?? null,
+ licenceNote: cfg.licenceNote ?? null,
+ processor: cfg.processor,
+ uses: cfg.uses,
+ shadowOnlyUses: cfg.shadowOnlyUses ?? Object.freeze([]),
+ },
+ ]);
+}
diff --git a/src/core/decision-model/diagnostics.ts b/src/core/decision-model/diagnostics.ts
index f0428f62..09191f79 100644
--- a/src/core/decision-model/diagnostics.ts
+++ b/src/core/decision-model/diagnostics.ts
@@ -11,8 +11,12 @@
import { CHOICE_MAX_OPTIONS } from "./answers.ts";
import { DECISION_MODEL_USES, isDecisionModelUse, type DecisionModelUse } from "./contract.ts";
-import { resolveDecisionModelConfig, type ResolvedDecisionModelConfig } from "./config.ts";
-import { makeDecisionProvider } from "./provider.ts";
+import {
+ resolveDecisionModelConfig,
+ resolvedProviderEntries,
+ type ResolvedDecisionModelConfig,
+} from "./config.ts";
+import { makeEntryDecisionProvider } from "./provider.ts";
import { DECISION_MODEL_UNPRICED_ADAPTERS } from "./presets.ts";
import type { DecisionProvider } from "./contract.ts";
import { SecretReferenceError } from "../secret-ref.ts";
@@ -29,7 +33,7 @@ import {
type DecisionEvalGrade,
} from "./record.ts";
import type { ContinuityRecord } from "../brain/continuity/types.ts";
-import { DECISION_THRESHOLD_PROFILES, isDecisionThresholdProfile } from "./questions.ts";
+import { DECISION_THRESHOLD_PROFILES } from "./questions.ts";
import { decisionEvalSpecFor, decisionEvalSpecHash, type DecisionEvalSpec } from "./eval-spec.ts";
import { validatedTuningMark } from "./eval/mark.ts";
import { processDirsEnv, type PlatformDirsEnv } from "../platform-dirs.ts";
@@ -66,6 +70,12 @@ export interface DecisionModelCheckReport {
readonly adapter: string;
readonly base_url: string | null;
readonly model: string | null;
+ /**
+ * The ordered provider chain (`decision_model_providers`), first entry
+ * first; the top-level provider fields are the FIRST entry's. With
+ * `--ping`, each entry also carries its own live `ping` result.
+ */
+ readonly provider_order: ReadonlyArray<DecisionProviderCheck>;
/** NAME of the key variable, never its value. */
readonly env_key: string | null;
readonly key_set: boolean;
@@ -144,6 +154,35 @@ export interface DecisionModelCheckOptions {
* running process's platform dirs; tests inject a state root.
*/
readonly markSource?: PlatformDirsEnv | null;
+ /**
+ * Where the local thresholds store is read from. Undefined reads the
+ * file beside the default config path (the caller's `env` decides
+ * where that is); null disables the store, which is what the tests
+ * that pin the shadow hold use.
+ */
+ readonly thresholdsPath?: string | null;
+}
+
+/** One chain entry as `check` reports it, in configured order. */
+export interface DecisionProviderCheck {
+ readonly name: string;
+ readonly adapter: string;
+ readonly base_url: string | null;
+ readonly model: string | null;
+ /** NAME of the entry's key variable, never its value. */
+ readonly env_key: string | null;
+ readonly key_set: boolean;
+ readonly key_required: boolean;
+ /** Whether the entry would be tried: it has its key, or needs none. */
+ readonly configured: boolean;
+ readonly calibrated: boolean;
+ /** Present only for a `check --ping` of a configured entry. */
+ readonly ping?: {
+ readonly ok: boolean;
+ readonly model?: string;
+ readonly latency_ms?: number;
+ readonly reason?: string;
+ };
}
const RERANK_ENDPOINT_KEYS = [
@@ -198,7 +237,10 @@ export function measuredTuning(
source: PlatformDirsEnv = processDirsEnv(),
): MeasuredTuning {
const profile = cfg.thresholdProfile ?? null;
- if (profile === null || !isDecisionThresholdProfile(profile)) {
+ // A profile name the source table does not carry (a self-hosted family
+ // the code never registered, issue #264) is not "none": its marks are
+ // keyed by the same string, so the measured states still apply to it.
+ if (profile === null) {
return { profile, state: "none", marked_uses: [], elsewhere_uses: [] };
}
const tuned = DECISION_THRESHOLD_PROFILES[profile]?.tuned ?? [];
@@ -274,7 +316,8 @@ export function measuredTuningSummary(measured: MeasuredTuning): string {
case "untuned":
return (
`profile '${measured.profile}' is untuned; enforce holds as shadow ` +
- "(o2b decision-model eval run can measure a use)"
+ "(o2b decision-model eval run can measure a use, or " +
+ "o2b decision-model thresholds fit can tune one from shadow data)"
);
}
}
@@ -323,6 +366,7 @@ export async function buildDecisionModelCheck(
config: opts.config,
vault: opts.vault,
secretsVault: opts.vault ?? undefined,
+ ...(opts.thresholdsPath !== undefined ? { thresholdsPath: opts.thresholdsPath } : {}),
});
const warnings: string[] = [];
const notes: string[] = [];
@@ -352,13 +396,17 @@ export async function buildDecisionModelCheck(
// stay silent. Printed once per check.
const shadowOnly = cfg.shadowOnlyUses ?? [];
if (cfg.enabled && shadowOnly.length > 0) {
+ const profileNote =
+ cfg.thresholdProfile === null || cfg.thresholdProfile === undefined
+ ? ""
+ : ` (profile '${cfg.thresholdProfile}')`;
warnings.push(
- `enforce runs as shadow for ${shadowOnly.join(", ")}: threshold profile ` +
- `'${cfg.thresholdProfile ?? "unknown"}' has no tuned thresholds for ` +
+ `enforce runs as shadow for ${shadowOnly.join(", ")}: model ` +
+ `'${cfg.model ?? "unknown"}'${profileNote} has no tuned thresholds for ` +
(shadowOnly.length === 1 ? "it" : "them") +
- (cfg.thresholdProfile === null || cfg.thresholdProfile === undefined
- ? "; for a compatible server of a known family set decision_model_threshold_profile"
- : ""),
+ `; tune with \`o2b decision-model thresholds fit --use ${shadowOnly[0]!}\` against ` +
+ "this model's shadow data, or set values by hand with " +
+ "`o2b decision-model thresholds set`",
);
}
if (cfg.enabled && !cfg.calibrated) {
@@ -466,6 +514,17 @@ export async function buildDecisionModelCheck(
adapter: cfg.adapter,
base_url: cfg.baseUrl,
model: cfg.model,
+ provider_order: resolvedProviderEntries(cfg).map((entry): DecisionProviderCheck => ({
+ name: entry.name,
+ adapter: entry.adapter,
+ base_url: entry.baseUrl,
+ model: entry.model,
+ env_key: entry.envKey,
+ key_set: entry.keyPresent,
+ key_required: entry.keyRequired,
+ configured: entry.keyPresent || !entry.keyRequired,
+ calibrated: entry.calibrated,
+ })),
env_key: cfg.envKey,
key_set: cfg.keyPresent,
key_required: cfg.keyRequired !== false,
@@ -495,64 +554,95 @@ export async function buildDecisionModelCheck(
};
if (opts.ping !== true) return report;
- // The ping resolves the key the way the real request path does (the
+ // The ping resolves each key the way the real request path does (the
// factory re-resolves at call time). A store that locks between the
// config probe and here - or whose keyfile is missing while entries
- // survive - raises the named refusal, degraded into the ping's reason,
- // never sent as a Bearer key and never left to escape the check as an
- // exception.
- let provider: DecisionProvider | null;
- try {
- provider = makeDecisionProvider(cfg, env, opts.vault ?? undefined);
- } catch (err) {
- if (
- err instanceof SecretStoreLockedError ||
- err instanceof SecretStoreKeyfileMissingError ||
- err instanceof SecretReferenceError
- ) {
- return { ...report, ping: { ok: false, reason: `not sent: ${err.message}` } };
+ // survive - raises the named refusal, degraded into that entry's ping
+ // reason, never sent as a Bearer key and never left to escape the
+ // check as an exception.
+ const order = [...report.provider_order];
+ const entries = resolvedProviderEntries(cfg);
+ let firstPing: DecisionModelCheckReport["ping"] | undefined;
+ let anyConfigured = false;
+ for (const [index, entry] of entries.entries()) {
+ const slot = order[index];
+ if (slot === undefined || !slot.configured) continue;
+ anyConfigured = true;
+ let adapter: DecisionProvider | null;
+ try {
+ adapter = makeEntryDecisionProvider(
+ entry,
+ { allowInsecureHttp: cfg.allowInsecureHttp, timeoutMs: cfg.timeoutMs },
+ env,
+ opts.vault ?? undefined,
+ );
+ } catch (err) {
+ if (
+ err instanceof SecretStoreLockedError ||
+ err instanceof SecretStoreKeyfileMissingError ||
+ err instanceof SecretReferenceError
+ ) {
+ const ping = { ok: false, reason: `not sent: ${err.message}` };
+ order[index] = { ...slot, ping };
+ if (firstPing === undefined) firstPing = ping;
+ continue;
+ }
+ throw err;
}
- throw err;
- }
- if (provider === null) {
- return { ...report, ping: { ok: false, reason: `not sent: status ${cfg.status}` } };
- }
- const pong = await provider.ping();
- // A ping is a real request: record it (origin `ping`), so it counts
- // toward the daily gate and shows in the report.
- emitDecisionModelCall(cfg.vault, {
- use: "ping",
- mode: "shadow",
- provider: provider.name,
- model: pong.model ?? provider.model,
- calibrated: provider.calibrated,
- questionCount: 1,
- candidateCount: 0,
- ...(pong.usage !== undefined ? { usage: pong.usage } : {}),
- inputPriceUsdPerMtok: cfg.inputPriceUsdPerMtok,
- ...(cfg.outputPriceUsdPerMtok !== undefined && cfg.outputPriceUsdPerMtok !== null
- ? { outputPriceUsdPerMtok: cfg.outputPriceUsdPerMtok }
- : {}),
- latencyMs: pong.latencyMs ?? 0,
- outcome: pong.ok ? "ok" : (pong.reason ?? "network"),
- origin: "ping",
- ...(opts.now !== undefined ? { createdAt: opts.now.toISOString() } : {}),
- });
- return {
- ...report,
- ping: {
+ if (adapter === null) continue;
+ // eslint-disable-next-line no-await-in-loop -- the order is the point: one ping per entry, in order
+ const pong = await adapter.ping();
+ // A ping is a real request: record it (origin `ping`), so it counts
+ // toward the daily gate and shows in the report. Each entry's ping
+ // names its own provider and answering model, so the records never
+ // pool two routes.
+ emitDecisionModelCall(cfg.vault, {
+ use: "ping",
+ mode: "shadow",
+ provider: adapter.name,
+ model: pong.model ?? adapter.model,
+ calibrated: adapter.calibrated,
+ questionCount: 1,
+ candidateCount: 0,
+ ...(pong.usage !== undefined ? { usage: pong.usage } : {}),
+ inputPriceUsdPerMtok: entry.inputPriceUsdPerMtok,
+ ...(entry.outputPriceUsdPerMtok !== undefined && entry.outputPriceUsdPerMtok !== null
+ ? { outputPriceUsdPerMtok: entry.outputPriceUsdPerMtok }
+ : {}),
+ latencyMs: pong.latencyMs ?? 0,
+ outcome: pong.ok ? "ok" : (pong.reason ?? "network"),
+ origin: "ping",
+ ...(opts.now !== undefined ? { createdAt: opts.now.toISOString() } : {}),
+ });
+ const ping: DecisionModelCheckReport["ping"] = {
ok: pong.ok,
...(pong.model !== undefined ? { model: pong.model } : {}),
...(pong.latencyMs !== undefined ? { latency_ms: pong.latencyMs } : {}),
...(pong.reason !== undefined ? { reason: pong.reason } : {}),
- },
- };
+ };
+ order[index] = { ...slot, ping };
+ if (firstPing === undefined) firstPing = ping;
+ }
+ const withOrder: DecisionModelCheckReport = { ...report, provider_order: order };
+ if (!anyConfigured) {
+ return { ...withOrder, ping: { ok: false, reason: `not sent: status ${cfg.status}` } };
+ }
+ return { ...withOrder, ping: firstPing };
}
/** Non-zero only for a broken configured provider. */
export function decisionModelCheckExitCode(report: DecisionModelCheckReport): number {
if (report.status === "invalid") return 1;
- if (report.ping !== undefined && report.status === "active" && !report.ping.ok) return 1;
+ if (
+ report.ping !== undefined &&
+ report.status === "active" &&
+ !report.ping.ok &&
+ // With a chain, a dead FIRST entry is the situation the fallback
+ // exists for: the check fails only when no configured entry answers.
+ !report.provider_order.some((entry) => entry.ping?.ok === true)
+ ) {
+ return 1;
+ }
return 0;
}
@@ -566,6 +656,25 @@ export function renderDecisionModelCheck(report: DecisionModelCheckReport): stri
);
lines.push(` base url: ${report.base_url ?? "(not set)"}`);
lines.push(` model: ${report.model ?? "(not set)"}`);
+ const order = report.provider_order;
+ if (order.length > 1) {
+ lines.push(` providers (in order): ${order.map((p) => p.name).join(", ")}`);
+ for (const [i, p] of order.entries()) {
+ const key = p.env_key === null ? "" : `, key ${p.env_key} ${p.key_set ? "set" : "not set"}`;
+ lines.push(
+ ` ${i + 1}. ${p.name} (${p.adapter}): ${p.base_url ?? "(no base url)"}, ` +
+ `model ${p.model ?? "?"}${key}` +
+ `${p.configured ? "" : ", not configured (no key)"}`,
+ );
+ if (p.ping !== undefined) {
+ lines.push(
+ p.ping.ok
+ ? ` ping: ok, answered by ${p.ping.model ?? "?"} in ${p.ping.latency_ms ?? 0}ms`
+ : ` ping: failed (${p.ping.reason ?? "unknown"})`,
+ );
+ }
+ }
+ }
const optional = report.key_required ? "" : ", optional on this loopback server";
lines.push(
report.env_key === null
@@ -656,6 +765,17 @@ export interface DecisionModelReport {
* absent for `rerank`, whose metric is the summary's shadow agreement.
*/
readonly use_report?: { readonly use: DecisionModelUse; readonly report: unknown };
+ /**
+ * Present when the caller supplied the resolved config and a use
+ * configured `enforce` runs as `shadow` for want of tuned thresholds
+ * for the model: says so explicitly and names the tuning command.
+ */
+ readonly shadow?: {
+ readonly model: string | null;
+ readonly uses: ReadonlyArray<DecisionModelUse>;
+ /** The command that tunes the first shadow use. */
+ readonly command: string;
+ };
/**
* The cached verdict sets of the preregistered evaluation
* (`eval/grade.ts`), oldest first, with the named skip notes. Present
@@ -735,6 +855,8 @@ export function buildDecisionModelReport(
readonly origin?: string;
/** Attach the use's own report when `use` names one (default true). */
readonly useReport?: boolean;
+ /** The resolved config, for the shadow section; absent on record-only callers. */
+ readonly config?: ResolvedDecisionModelConfig;
} = {},
): DecisionModelReport {
const records = listDecisionModelCalls(vault, opts.since).filter(
@@ -782,7 +904,21 @@ export function buildDecisionModelReport(
...(agreement !== undefined ? { agreement } : {}),
});
}
- const summary: DecisionModelReport = { since: opts.since ?? null, total: records.length, uses };
+ const shadowOnly = opts.config?.shadowOnlyUses ?? [];
+ const shadow =
+ opts.config !== undefined && shadowOnly.length > 0
+ ? {
+ model: opts.config.model,
+ uses: shadowOnly,
+ command: `o2b decision-model thresholds fit --use ${shadowOnly[0]!}`,
+ }
+ : undefined;
+ const summary: DecisionModelReport = {
+ since: opts.since ?? null,
+ total: records.length,
+ uses,
+ ...(shadow !== undefined ? { shadow } : {}),
+ };
// The eval section is built for every report: `o2b decision-model
// report` is the audit surface for the grades a mark was written from.
const { grades, skipped } = listDecisionModelEvalGrades(vault, opts.since);
@@ -842,6 +978,16 @@ export function renderDecisionModelReport(report: DecisionModelReport): string {
);
}
}
+ if (report.shadow !== undefined) {
+ lines.push(
+ ` shadow (no tuned thresholds for model '${report.shadow.model ?? "unknown"}'): ` +
+ `${report.shadow.uses.join(", ")}`,
+ );
+ lines.push(
+ ` tune with \`${report.shadow.command}\`, or set values by hand with ` +
+ "`o2b decision-model thresholds set`",
+ );
+ }
if (report.eval !== undefined) {
lines.push(
`decision model eval grades: ${report.eval.grades.length} record(s)` +
diff --git a/src/core/decision-model/provider.ts b/src/core/decision-model/provider.ts
index ee91a127..33799c6b 100644
--- a/src/core/decision-model/provider.ts
+++ b/src/core/decision-model/provider.ts
@@ -7,8 +7,10 @@
* `search/rerank/provider.ts`, so a process that never enables the
* feature never loads an HTTP adapter.
*
- * The adapter is the one the configured preset names; nothing here picks
- * another one, and `llm-emulation` in particular is never a fallback.
+ * The adapter is the one the configured preset names. With an ordered
+ * chain (`decision_model_providers`) every entry is built from its own
+ * resolved fields; the run (`run.ts`) owns the order and the fallback,
+ * and `llm-emulation` in particular is never a fallback picked here.
*
* The key is read here, from the environment, at call time. It is handed
* to the adapter instance and never stored in any config object. When the
@@ -24,16 +26,32 @@
import { resolveNamedSecret } from "../secret-resolver.ts";
import type { DecisionProvider } from "./contract.ts";
-import type { ResolvedDecisionModelConfig } from "./config.ts";
+import { resolvedProviderEntries, type ResolvedDecisionModelConfig } from "./config.ts";
+import type { ResolvedDecisionProvider } from "./config-types.ts";
-export function makeDecisionProvider(
- cfg: ResolvedDecisionModelConfig | null | undefined,
+/**
+ * The transport settings shared by every entry of a chain: the network
+ * trust statement and the request budget are properties of the
+ * installation, not of one route.
+ */
+export interface EntryTransportSettings {
+ readonly allowInsecureHttp: boolean;
+ readonly timeoutMs: number;
+}
+
+/**
+ * Build one chain entry's adapter, or null when the entry is not
+ * configured (no endpoint, or a required key missing). Never consults
+ * any other entry: the fallback order is the caller's decision.
+ */
+export function makeEntryDecisionProvider(
+ entry: ResolvedDecisionProvider,
+ shared: EntryTransportSettings,
env: Readonly<Record<string, string | undefined>> = process.env,
secretsVault?: string | null,
): DecisionProvider | null {
- if (cfg === null || cfg === undefined || cfg.status !== "active") return null;
- if (cfg.baseUrl === null || cfg.model === null) return null;
- const raw = cfg.envKey !== null ? env[cfg.envKey] : undefined;
+ if (entry.baseUrl === null || entry.model === null) return null;
+ const raw = entry.envKey !== null ? env[entry.envKey] : undefined;
// Reference-shaped values resolve through the custody store; `null`
// (no vault) and plain values pass through untouched, byte-identical
// to the pre-routing factory.
@@ -43,18 +61,18 @@ export function makeDecisionProvider(
: raw;
const apiKey = typeof keyValue === "string" && keyValue.trim() !== "" ? keyValue.trim() : null;
// Only a loopback server whose preset allows it runs without a key.
- if (apiKey === null && cfg.keyRequired !== false) return null;
+ if (apiKey === null && entry.keyRequired !== false) return null;
const common = {
- name: cfg.provider ?? cfg.adapter,
- baseUrl: cfg.baseUrl,
- model: cfg.model,
- envKey: cfg.envKey,
- allowInsecureHttp: cfg.allowInsecureHttp,
- timeoutMs: cfg.timeoutMs,
- ...(cfg.maxChoiceOptions !== undefined ? { maxChoiceOptions: cfg.maxChoiceOptions } : {}),
+ name: entry.name,
+ baseUrl: entry.baseUrl,
+ model: entry.model,
+ envKey: entry.envKey,
+ allowInsecureHttp: shared.allowInsecureHttp,
+ timeoutMs: shared.timeoutMs,
+ ...(entry.maxChoiceOptions !== undefined ? { maxChoiceOptions: entry.maxChoiceOptions } : {}),
};
try {
- switch (cfg.adapter) {
+ switch (entry.adapter) {
case "llm-emulation": {
if (apiKey === null) return null;
// eslint-disable-next-line @typescript-eslint/no-require-imports
@@ -69,21 +87,21 @@ export function makeDecisionProvider(
return new VercelEvaluateDecisionProvider({
...common,
apiKey,
- calibrated: cfg.calibrated,
+ calibrated: entry.calibrated,
});
}
case "systemone": {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const { SystemOneDecisionProvider } =
require("./systemone.ts") as typeof import("./systemone.ts");
- return new SystemOneDecisionProvider({ ...common, apiKey, calibrated: cfg.calibrated });
+ return new SystemOneDecisionProvider({ ...common, apiKey, calibrated: entry.calibrated });
}
case "converse": {
if (apiKey === null) return null;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const { ConverseDecisionProvider } =
require("./converse.ts") as typeof import("./converse.ts");
- return new ConverseDecisionProvider({ ...common, apiKey, calibrated: cfg.calibrated });
+ return new ConverseDecisionProvider({ ...common, apiKey, calibrated: entry.calibrated });
}
}
} catch {
@@ -92,3 +110,25 @@ export function makeDecisionProvider(
return null;
}
}
+
+/**
+ * The first chain entry's adapter, the shape every pre-chain caller used.
+ * Kept as its own export because the diagnostics and single-provider use
+ * sites read exactly one entry; the run builds the whole chain.
+ */
+export function makeDecisionProvider(
+ cfg: ResolvedDecisionModelConfig | null | undefined,
+ env: Readonly<Record<string, string | undefined>> = process.env,
+ secretsVault?: string | null,
+): DecisionProvider | null {
+ if (cfg === null || cfg === undefined || cfg.status !== "active") return null;
+ const entries = resolvedProviderEntries(cfg);
+ const first = entries[0];
+ if (first === undefined) return null;
+ return makeEntryDecisionProvider(
+ first,
+ { allowInsecureHttp: cfg.allowInsecureHttp, timeoutMs: cfg.timeoutMs },
+ env,
+ secretsVault,
+ );
+}
diff --git a/src/core/decision-model/questions.ts b/src/core/decision-model/questions.ts
index 62686b4e..5e87ae9b 100644
--- a/src/core/decision-model/questions.ts
+++ b/src/core/decision-model/questions.ts
@@ -13,6 +13,7 @@
*/
import type { DecisionChoiceQuestion, DecisionModelUse, DecisionNoulQuestion } from "./contract.ts";
+import { SKILLS_FINAL_MIN, SKILLS_NEEDS_ANY_MIN } from "./questions-skills.ts";
/** Default per-candidate clip for any use that does not set its own. */
export const DEFAULT_CANDIDATE_CLIP_CHARS = 900;
@@ -89,18 +90,15 @@ export const RERANK_QUESTIONS = Object.freeze({
// ----- threshold profiles ------------------------------------------------------
/**
- * Threshold profiles, one per provider family. Probabilities from
- * different models are not comparable, so a threshold tuned against one
- * family says nothing about another. A use runs `enforce` only on a
- * profile that lists it as tuned; on any other profile (or on none, as
- * with a `compatible` server of unknown family) an `enforce` setting runs
- * as `shadow`, and `o2b decision-model check` says so. Hot paths never
- * warn.
- *
- * `baseline` marks the profile every threshold in this file was set
- * against: every use is tuned for it, so no use needs to be registered
- * here twice. Any other profile lists the
- * uses tuned for it explicitly, from its own shadow data.
+ * The built-in threshold profile: the hosted Jev 1.13 family, the family
+ * every threshold constant in this file was set against. It is the only
+ * entry the code carries (issue #264): every other model takes its
+ * thresholds from the local store (`thresholds.ts`, keyed by the model id
+ * the server reports), so supporting a model never needs a release. A use
+ * runs `enforce` when the profile is this baseline, or when the store
+ * carries an entry for the answering model, or when the preregistered
+ * eval's machine-side mark says the pair is measured; otherwise it runs
+ * as `shadow`, and `check`/`report` say so and name the tuning command.
*/
export interface DecisionThresholdProfile {
readonly name: string;
@@ -112,15 +110,6 @@ export const DECISION_THRESHOLD_PROFILES: Readonly<Record<string, DecisionThresh
Object.freeze({
// Every threshold above was set against the hosted Jev 1.13 family.
"jev-1.13": { name: "jev-1.13", tuned: "baseline" },
- // Self-hosted open-weight models: their own calibration, none tuned yet.
- laya: { name: "laya", tuned: Object.freeze([]) },
- openjev: { name: "openjev", tuned: Object.freeze([]) },
- // Self-reported probabilities of a generative model: none tuned.
- "llm-emulation": { name: "llm-emulation", tuned: Object.freeze([]) },
- // A Converse-protocol gateway fronting the operator's own judgment
- // model family: calibrated per that deployment, none tuned here, so an
- // `enforce` setting runs as `shadow` until tuned from shadow data.
- converse: { name: "converse", tuned: Object.freeze([]) },
});
export const DECISION_THRESHOLD_PROFILE_NAMES: ReadonlyArray<string> = Object.freeze(
@@ -132,23 +121,28 @@ export function isDecisionThresholdProfile(name: string): boolean {
}
/**
- * Whether `use` has tuned thresholds on `profile`; false for an unknown or absent profile.
+ * Whether `use` has tuned thresholds on `profile`.
+ *
+ * The source table decides first: the baseline profile ships tuned for
+ * every use, a hand-registered list carries its own. An unknown profile
+ * name (a self-hosted family the code does not carry) has no table entry,
+ * so the measured mark decides, exactly as for a known-but-untuned one -
+ * the store gate (`thresholds.ts`, keyed by model id) is applied beside
+ * this by the config resolver.
*
- * The source table decides first. Only when it says untuned is the
- * optional {@link measuredTuned} reader consulted - the machine-side
- * tuning mark of the preregistered evaluation (`eval/mark.ts`), whose read is
- * therefore never paid for a hand-registered or baseline use, and never
- * paid at all when no reader is injected (today's behavior, which the
- * hand-edit tests pin).
+ * The measured reader is consulted only when the table leaves the pair
+ * untuned, so its read is never paid for a hand-registered or baseline
+ * use, and never paid at all when no reader is injected.
*/
export function thresholdsTunedFor(
profile: string | null,
use: DecisionModelUse,
measuredTuned?: (() => boolean) | null,
): boolean {
- if (profile === null || !isDecisionThresholdProfile(profile)) return false;
- const tuned = DECISION_THRESHOLD_PROFILES[profile]!.tuned;
- if (tuned === "baseline" || tuned.includes(use)) return true;
+ if (profile === null) return false;
+ const tuned = DECISION_THRESHOLD_PROFILES[profile]?.tuned;
+ if (tuned === "baseline") return true;
+ if (tuned !== undefined && tuned.includes(use)) return true;
return measuredTuned !== null && measuredTuned !== undefined && measuredTuned();
}
@@ -342,3 +336,37 @@ export const RECALL_INJECT_QUESTIONS = Object.freeze({
};
},
});
+
+/**
+ * The threshold values each use applies, by name, when nothing is stored
+ * for the model: the hosted baseline family's compiled constants, the
+ * fallback {@link effectiveThresholds} merges stored values over. Derived
+ * from the constants above, never re-declared, so the two cannot drift.
+ */
+export const BUILTIN_USE_THRESHOLDS: Readonly<
+ Record<DecisionModelUse, Readonly<Record<string, number>>>
+> = Object.freeze({
+ rerank: Object.freeze({ injection_flag_min: RERANK_QUESTIONS.injectionFlagMin }),
+ answerable: Object.freeze({ low: ANSWERABLE_LOW, high: ANSWERABLE_HIGH }),
+ skills: Object.freeze({ needs_any_min: SKILLS_NEEDS_ANY_MIN, final_min: SKILLS_FINAL_MIN }),
+ extract_prefilter: Object.freeze({ drop_below: EXTRACT_PREFILTER_DROP_BELOW }),
+ dedup: Object.freeze({ low_priority_min: PAIR_VERDICT_LIMITS.lowPriorityMin }),
+ tension: Object.freeze({ low_priority_min: PAIR_VERDICT_LIMITS.lowPriorityMin }),
+ labels: Object.freeze({ suggest_min: LABELS_QUESTIONS.suggestMin }),
+ recall_inject: Object.freeze({
+ any_min: RECALL_INJECT_ANY_MIN,
+ note_min: RECALL_INJECT_NOTE_MIN,
+ }),
+});
+
+/** The one value a threshold `fit` derives for a use: the action threshold. */
+export const PRIMARY_THRESHOLD_NAME: Readonly<Record<DecisionModelUse, string>> = Object.freeze({
+ rerank: "injection_flag_min",
+ answerable: "low",
+ skills: "needs_any_min",
+ extract_prefilter: "drop_below",
+ dedup: "low_priority_min",
+ tension: "low_priority_min",
+ labels: "suggest_min",
+ recall_inject: "any_min",
+});
diff --git a/src/core/decision-model/reports/extract-prefilter.ts b/src/core/decision-model/reports/extract-prefilter.ts
index d2f61d23..c7272f0a 100644
--- a/src/core/decision-model/reports/extract-prefilter.ts
+++ b/src/core/decision-model/reports/extract-prefilter.ts
@@ -71,17 +71,73 @@ interface PlanAgg {
readonly scores: Map<string, number>;
readonly chars: Map<string, number>;
readonly threshold: number;
+ /** The answering model of the record the plan came from. */
+ readonly model: string;
}
function share(part: number, whole: number): number | null {
return whole === 0 ? null : part / whole;
}
-export function buildExtractPrefilterReport(
+/** One scored turn of a shadow plan, joined to whether a commit cited it. */
+export interface ExtractRegretSample {
+ readonly probability: number;
+ readonly cited: boolean;
+ /** The answering model of the plan the score came from. */
+ readonly model: string;
+}
+
+/**
+ * Every scored turn of every shadow plan, joined to whether a later
+ * extract commit cited it - the regret data both the report and the
+ * threshold fit derive from. A turn cited before its newest prior plan
+ * exists is not evaluable and appears with `cited: false`, exactly as
+ * the report's regret join leaves it out. Read-only.
+ */
+export function buildExtractRegretSamples(
vault: string,
opts: { readonly since?: string } = {},
-): ExtractPrefilterReport {
- const calls = listDecisionModelCalls(vault, opts.since).filter(
+): { readonly samples: ReadonlyArray<ExtractRegretSample> } {
+ const plans = collectPlans(vault, opts.since);
+ // Regret: each cited turn against the latest shadow plan of its session
+ // created at or before the commit.
+ const shadowBySession = new Map<string, PlanAgg[]>();
+ for (const plan of plans.values()) {
+ if (plan.mode !== "shadow") continue;
+ const list = shadowBySession.get(plan.sessionId) ?? [];
+ list.push(plan);
+ shadowBySession.set(plan.sessionId, list);
+ }
+ for (const list of shadowBySession.values()) {
+ list.sort((a, b) => b.createdAt.localeCompare(a.createdAt));
+ }
+ const cited = new Set<PlanAgg>();
+ for (const commit of listDecisionModelExtractCommits(vault, opts.since)) {
+ const sessionId = commit.payload["session_id"];
+ const turns = strings(commit.payload["source_turns"]);
+ if (typeof sessionId !== "string" || turns === null) continue;
+ const candidates = (shadowBySession.get(sessionId) ?? []).filter(
+ (plan) => plan.createdAt <= commit.createdAt,
+ );
+ for (const turn of turns) {
+ const plan = candidates.find((c) => c.scores.has(turn));
+ if (plan === undefined) continue;
+ cited.add(plan);
+ }
+ }
+ const samples: ExtractRegretSample[] = [];
+ for (const list of shadowBySession.values()) {
+ for (const plan of list) {
+ for (const probability of plan.scores.values()) {
+ samples.push({ probability, cited: cited.has(plan), model: plan.model });
+ }
+ }
+ }
+ return { samples };
+}
+
+function collectPlans(vault: string, since?: string): Map<string, PlanAgg> {
+ const calls = listDecisionModelCalls(vault, since).filter(
(r) =>
r.payload["use"] === USE &&
r.payload["outcome"] === "ok" &&
@@ -104,6 +160,7 @@ export function buildExtractPrefilterReport(
scores: new Map(),
chars: new Map(),
threshold: typeof p["threshold"] === "number" ? p["threshold"] : EXTRACT_PREFILTER_DROP_BELOW,
+ model: typeof p["model"] === "string" ? p["model"] : "unknown",
};
ids.forEach((id, k) => {
const prob = probs[k];
@@ -113,6 +170,14 @@ export function buildExtractPrefilterReport(
});
plans.set(planId, plan);
}
+ return plans;
+}
+
+export function buildExtractPrefilterReport(
+ vault: string,
+ opts: { readonly since?: string } = {},
+): ExtractPrefilterReport {
+ const plans = collectPlans(vault, opts.since);
let scored = 0;
let dropped = 0;
diff --git a/src/core/decision-model/reports/labels.ts b/src/core/decision-model/reports/labels.ts
index 3e384f5c..00f87347 100644
--- a/src/core/decision-model/reports/labels.ts
+++ b/src/core/decision-model/reports/labels.ts
@@ -53,6 +53,10 @@ interface Suggested {
readonly dimension: string;
readonly current: string | null;
readonly suggested: string | null;
+ /** The model's confidence in the suggestion, as the record carries it. */
+ readonly confidence: number | null;
+ /** The answering model of the record the suggestion came from. */
+ readonly model: string;
}
function emptyOutcomes(): Record<LabelOutcome, number> {
@@ -79,10 +83,28 @@ function nowValue(
return { value: token === undefined ? null : token.slice(dimension.length + 1) };
}
-export function buildLabelsReport(
+/** One suggestion joined to its later outcome, with the model's confidence. */
+export interface LabelSample {
+ readonly mode: string;
+ /** Whether the model made a suggestion at all (`none` or below threshold otherwise). */
+ readonly suggested: boolean;
+ readonly confidence: number | null;
+ readonly outcome: LabelOutcome;
+ /** The answering model of the record the suggestion came from. */
+ readonly model: string;
+}
+
+/**
+ * Every latest suggestion per note and dimension joined to its outcome -
+ * the data both the acceptance report and the threshold fit derive from.
+ * A suggestion without one (`none` or below the threshold) carries
+ * `suggested: false` and outcome `missing` (there is nothing the note
+ * could accept or override). Read-only.
+ */
+export function buildLabelSamples(
vault: string,
opts: { readonly since?: string } = {},
-): LabelsReport {
+): { readonly samples: ReadonlyArray<LabelSample> } {
const latest = new Map<string, Suggested>();
for (const record of listDecisionModelCalls(vault, opts.since)) {
const p = record.payload;
@@ -105,19 +127,27 @@ export function buildLabelsReport(
dimension: d["dimension"],
current: typeof d["current"] === "string" ? d["current"] : null,
suggested: typeof d["suggested"] === "string" ? d["suggested"] : null,
+ confidence:
+ typeof d["confidence"] === "number" && Number.isFinite(d["confidence"])
+ ? d["confidence"]
+ : null,
+ model: typeof p["model"] === "string" ? p["model"] : "unknown",
});
}
}
const cache = new Map<string, ReadonlyArray<string> | null>();
- const byMode = new Map<string, Record<LabelOutcome, number>>();
- let suggestions = 0;
- let none = 0;
+ const samples: LabelSample[] = [];
for (const s of latest.values()) {
if (s.suggested === null) {
- none++;
+ samples.push({
+ mode: s.mode,
+ suggested: false,
+ confidence: s.confidence,
+ outcome: "missing",
+ model: s.model,
+ });
continue;
}
- suggestions++;
const now = nowValue(vault, s.path, s.dimension, cache);
let outcome: LabelOutcome;
if (now === null) outcome = "missing";
@@ -125,8 +155,33 @@ export function buildLabelsReport(
else if (now.value === s.current) outcome = "unchanged";
else if (now.value === s.suggested) outcome = "accepted";
else outcome = "overridden";
+ samples.push({
+ mode: s.mode,
+ suggested: true,
+ confidence: s.confidence,
+ outcome,
+ model: s.model,
+ });
+ }
+ return { samples };
+}
+
+export function buildLabelsReport(
+ vault: string,
+ opts: { readonly since?: string } = {},
+): LabelsReport {
+ const { samples } = buildLabelSamples(vault, opts);
+ const byMode = new Map<string, Record<LabelOutcome, number>>();
+ let suggestions = 0;
+ let none = 0;
+ for (const s of samples) {
+ if (!s.suggested) {
+ none++;
+ continue;
+ }
+ suggestions++;
const counts = byMode.get(s.mode) ?? emptyOutcomes();
- counts[outcome]++;
+ counts[s.outcome]++;
byMode.set(s.mode, counts);
}
return {
@@ -134,7 +189,7 @@ export function buildLabelsReport(
suggestions,
no_suggestion: none,
by_mode: [...byMode]
- .toSorted((x, y) => x[0].localeCompare(y[0]))
+ .toSorted((a, b) => a[0].localeCompare(b[0]))
.map(([mode, outcomes]) => {
const decided = outcomes.accepted + outcomes.overridden;
return {
diff --git a/src/core/decision-model/reports/pair-verdict.ts b/src/core/decision-model/reports/pair-verdict.ts
index 4dfa64ab..72a994fc 100644
--- a/src/core/decision-model/reports/pair-verdict.ts
+++ b/src/core/decision-model/reports/pair-verdict.ts
@@ -57,6 +57,8 @@ interface RecordedPair {
readonly b: string;
readonly verdict: string;
readonly probability: number;
+ /** The answering model of the record the verdict came from. */
+ readonly model: string;
}
function band(verdict: string, p: number): string {
@@ -90,6 +92,7 @@ function recordedPairs(record: ContinuityRecord): RecordedPair[] {
b: r["b"],
verdict: r["verdict"],
probability: r["probability"],
+ model: typeof record.payload["model"] === "string" ? record.payload["model"] : "unknown",
});
}
return out;
@@ -143,12 +146,25 @@ function emptyOutcomes(): Record<PairOutcome, number> {
return { merged: 0, resolved: 0, dismissed: 0, confirmed: 0, open: 0, missing: 0 };
}
-/** Per-band operator outcomes for the `dedup` or `tension` use. */
-export function buildPairVerdictReport(
+/** One verdicted pair joined to what the operator did with it afterwards. */
+export interface PairVerdictSample {
+ readonly probability: number;
+ readonly verdict: string;
+ readonly outcome: PairOutcome;
+ /** The answering model of the record the verdict came from. */
+ readonly model: string;
+}
+
+/**
+ * Every latest-verdict pair of the use, joined to its later outcome -
+ * the data both the band report and the threshold fit derive from.
+ * Read-only.
+ */
+export function buildPairVerdictSamples(
vault: string,
use: "dedup" | "tension",
opts: { readonly since?: string } = {},
-): PairVerdictUseReport {
+): { readonly samples: ReadonlyArray<PairVerdictSample> } {
const latest = new Map<string, RecordedPair>();
for (const record of listDecisionModelCalls(vault, opts.since)) {
if (record.payload["use"] !== use) continue;
@@ -161,7 +177,7 @@ export function buildPairVerdictReport(
}
}
let entities: Map<string, BrainEntity> | null = null;
- const bands = new Map<string, Record<PairOutcome, number>>();
+ const samples: PairVerdictSample[] = [];
for (const pair of latest.values()) {
let outcome: PairOutcome;
if (pair.kind === "tension") outcome = tensionOutcome(vault, pair);
@@ -169,14 +185,33 @@ export function buildPairVerdictReport(
entities ??= new Map(buildEntityIndex(vault).entities.map((e) => [e.id, e]));
outcome = entityOutcome(entities, pair);
} else outcome = preferenceOutcome(vault, pair);
- const key = band(pair.verdict, pair.probability);
+ samples.push({
+ probability: pair.probability,
+ verdict: pair.verdict,
+ outcome,
+ model: pair.model,
+ });
+ }
+ return { samples };
+}
+
+/** Per-band operator outcomes for the `dedup` or `tension` use. */
+export function buildPairVerdictReport(
+ vault: string,
+ use: "dedup" | "tension",
+ opts: { readonly since?: string } = {},
+): PairVerdictUseReport {
+ const { samples } = buildPairVerdictSamples(vault, use, opts);
+ const bands = new Map<string, Record<PairOutcome, number>>();
+ for (const sample of samples) {
+ const key = band(sample.verdict, sample.probability);
const counts = bands.get(key) ?? emptyOutcomes();
- counts[outcome]++;
+ counts[sample.outcome]++;
bands.set(key, counts);
}
return {
use,
- pairs: latest.size,
+ pairs: samples.length,
bands: [...bands]
.toSorted((x, y) => x[0].localeCompare(y[0]))
.map(([name, outcomes]) => ({
diff --git a/src/core/decision-model/run.ts b/src/core/decision-model/run.ts
index ea2605e4..c50708bf 100644
--- a/src/core/decision-model/run.ts
+++ b/src/core/decision-model/run.ts
@@ -8,9 +8,14 @@
* no state is built, nothing is sent, nothing is recorded.
* - the daily cost gate, checked before the state is built;
* - the state builder's own budget refusal;
- * - the request itself, with the configured timeout;
- * - exactly one `decision_model_call` record per attempted request, with
- * the outcome (`ok` or the degrade reason);
+ * - the request itself, over the configured provider chain in order:
+ * an entry that is not configured is skipped, a connection error or
+ * timeout falls through to the next entry, and the whole chain
+ * shares the caller's one timeout budget (each attempt gets an even
+ * slice of what remains, so a dead first server can never stretch
+ * the call);
+ * - exactly one `decision_model_call` record per attempted request,
+ * each naming the provider and model that attempted or answered it;
* - the per-route latency component for MCP route metrics.
*
* It never throws. A degrade is returned as a typed reason and the use
@@ -28,9 +33,14 @@ import {
type DecisionResponse,
type DecisionState,
} from "./contract.ts";
-import { decisionModelModeFor, type ResolvedDecisionModelConfig } from "./config.ts";
+import {
+ decisionModelModeFor,
+ resolvedProviderEntries,
+ type ResolvedDecisionModelConfig,
+} from "./config.ts";
+import type { ResolvedDecisionProvider } from "./config-types.ts";
import { noteDecisionLatency } from "./latency.ts";
-import { makeDecisionProvider } from "./provider.ts";
+import { makeEntryDecisionProvider } from "./provider.ts";
import { emitDecisionModelCall, todaySpendUsd } from "./record.ts";
import { estimateTokens } from "./state.ts";
@@ -84,7 +94,10 @@ export type DecisionRunResult<C> =
export interface RunDecisionOptions<C> {
readonly config: ResolvedDecisionModelConfig | null | undefined;
- /** Injected provider (tests). Defaults to the configured adapter. */
+ /**
+ * Injected provider (tests): replaces the whole chain with this one
+ * adapter. Defaults to the configured chain, first entry first.
+ */
readonly provider?: DecisionProvider;
readonly env?: Readonly<Record<string, string | undefined>>;
/**
@@ -114,6 +127,65 @@ export interface RunDecisionOptions<C> {
) => Readonly<Record<string, unknown>> | undefined;
}
+/** One attempt of the chain: an adapter plus the mode it answers under. */
+interface ChainAttempt {
+ readonly adapter: DecisionProvider;
+ readonly entry: ResolvedDecisionProvider;
+ readonly mode: Exclude<DecisionModelMode, "off">;
+}
+
+/** The chain options `buildChain` reads; a subset of {@link RunDecisionOptions}. */
+interface ChainOptions {
+ readonly provider?: DecisionProvider;
+ readonly modeOverride?: Exclude<DecisionModelMode, "off">;
+}
+
+/**
+ * The chain of attempts for one call, in configured order. An entry the
+ * factory cannot build (no endpoint, a missing key) is skipped: it is
+ * not configured, exactly as the resolver's status rules say. An
+ * injected provider (tests) is the whole chain.
+ */
+function buildChain(
+ cfg: ResolvedDecisionModelConfig,
+ use: DecisionModelUse,
+ mode: Exclude<DecisionModelMode, "off">,
+ opts: ChainOptions,
+ env: Readonly<Record<string, string | undefined>>,
+ secretsVault: string | null | undefined,
+): ReadonlyArray<ChainAttempt> {
+ if (opts.provider !== undefined) {
+ const entry = resolvedProviderEntries(cfg)[0];
+ if (entry === undefined) return [];
+ return [{ adapter: opts.provider, entry, mode }];
+ }
+ const attempts: ChainAttempt[] = [];
+ for (const [index, entry] of resolvedProviderEntries(cfg).entries()) {
+ const adapter = makeEntryDecisionProvider(
+ entry,
+ { allowInsecureHttp: cfg.allowInsecureHttp, timeoutMs: cfg.timeoutMs },
+ env,
+ secretsVault,
+ );
+ if (adapter === null) continue;
+ // The first entry answers for the config as written: the top-level
+ // modes are its held modes, and a caller that overrode them (the
+ // eval gate forces enforce) means it for the primary. A later entry
+ // is judged by its own thresholds - a fallback model without tuned
+ // thresholds answers as shadow even when the primary enforces. A
+ // held mode is never off (off-ness is the global config's decision);
+ // anything but enforce is shadow.
+ const held = opts.modeOverride ?? (index === 0 ? cfg.uses[use] : (entry.uses[use] ?? "shadow"));
+ attempts.push({ adapter, entry, mode: held === "enforce" ? "enforce" : "shadow" });
+ }
+ return attempts;
+}
+
+/** The degrade reasons that mean "this route could not answer": fall through. */
+function isFailoverReason(reason: DecisionDegradeReason): boolean {
+ return reason === "network" || reason === "timeout";
+}
+
export async function runDecision<C>(
use: DecisionModelUse,
buildState: () => BuiltDecisionState<C>,
@@ -126,12 +198,14 @@ export async function runDecision<C>(
const configured = decisionModelModeFor(cfg, use);
if (configured === "off" || cfg === null || cfg === undefined) return { status: "off" };
const mode = opts.modeOverride ?? configured;
- const provider =
- opts.provider ?? makeDecisionProvider(cfg, opts.env ?? process.env, opts.secretsVault);
- if (provider === null) return { status: "off" };
+ const env = opts.env ?? process.env;
+ const chain = buildChain(cfg, use, mode, opts, env, opts.secretsVault);
+ if (chain.length === 0 || chain[0] === undefined) return { status: "off" };
+ const primary = chain[0];
const now = opts.now ?? (() => new Date());
const record = (
+ attempt: ChainAttempt,
outcome: string,
extra: {
readonly questionCount?: number;
@@ -149,16 +223,17 @@ export async function runDecision<C>(
}
emitDecisionModelCall(cfg.vault, {
use,
- mode,
- provider: provider.name,
- model: extra.response?.model ?? provider.model,
- calibrated: extra.response?.calibrated ?? provider.calibrated,
+ mode: attempt.mode,
+ provider: attempt.adapter.name,
+ model: extra.response?.model ?? attempt.adapter.model,
+ calibrated: extra.response?.calibrated ?? attempt.adapter.calibrated,
questionCount: extra.questionCount ?? 0,
candidateCount: extra.candidateCount ?? 0,
...(extra.response !== undefined ? { usage: extra.response.usage } : {}),
- inputPriceUsdPerMtok: cfg.inputPriceUsdPerMtok,
- ...(cfg.outputPriceUsdPerMtok !== undefined && cfg.outputPriceUsdPerMtok !== null
- ? { outputPriceUsdPerMtok: cfg.outputPriceUsdPerMtok }
+ inputPriceUsdPerMtok: attempt.entry.inputPriceUsdPerMtok,
+ ...(attempt.entry.outputPriceUsdPerMtok !== undefined &&
+ attempt.entry.outputPriceUsdPerMtok !== null
+ ? { outputPriceUsdPerMtok: attempt.entry.outputPriceUsdPerMtok }
: {}),
latencyMs: extra.latencyMs ?? 0,
outcome,
@@ -176,6 +251,8 @@ export async function runDecision<C>(
// requests cannot all pass against the same recorded spend. Separate
// processes still share only the recorded spend, so the gate is a
// soft limit that can be passed by the requests in flight elsewhere.
+ // The estimate uses the FIRST entry's price; a fallback attempt is
+ // judged by its own price only in its own record.
const gated = cfg.dailyCostGateUsd > 0 && cfg.vault !== null;
if (gated) {
let spend: number;
@@ -185,7 +262,7 @@ export async function runDecision<C>(
spend = Number.POSITIVE_INFINITY;
}
if (spend >= cfg.dailyCostGateUsd) {
- record("cost_gate");
+ record(primary, "cost_gate");
return { status: "degraded", mode, reason: "cost_gate" };
}
}
@@ -198,7 +275,7 @@ export async function runDecision<C>(
}
if (built.kind === "empty") return { status: "empty", mode };
if (built.kind === "budget") {
- record("budget");
+ record(primary, "budget");
return { status: "degraded", mode, reason: "budget" };
}
@@ -210,40 +287,69 @@ export async function runDecision<C>(
? (estimateTokens({ state: ready.state, questions: qs }) * cfg.inputPriceUsdPerMtok) /
1_000_000
: 0;
+
+ // The whole chain shares the caller's one budget (`timeout_ms`, or the
+ // hook sub-budget): each attempt gets an even slice of what remains, so
+ // a dead first server can never stretch the call past the budget, and
+ // the last entry always keeps a real slice to answer in.
+ const budgetMs = opts.timeoutMs ?? cfg.timeoutMs;
const send = async (): Promise<DecisionRunResult<C>> => {
- const started = Date.now();
- let response: DecisionResponse;
- try {
- response = await provider.decide(
- { use, state: ready.state, questions: qs },
- {
- timeoutMs: opts.timeoutMs ?? cfg.timeoutMs,
- ...(opts.signal !== undefined ? { signal: opts.signal } : {}),
- },
- );
- } catch (e) {
+ const deadline = Date.now() + budgetMs;
+ let last: {
+ readonly reason: DecisionDegradeReason;
+ readonly mode: Exclude<DecisionModelMode, "off">;
+ } | null = null;
+ for (const [index, attempt] of chain.entries()) {
+ const remaining = chain.length - index;
+ const slice = Math.max(1, Math.floor((deadline - Date.now()) / remaining));
+ const started = Date.now();
+ let response: DecisionResponse;
+ try {
+ // eslint-disable-next-line no-await-in-loop -- the fallback is sequential by design: an attempt must settle before the next starts
+ response = await attempt.adapter.decide(
+ { use, state: ready.state, questions: qs },
+ {
+ timeoutMs: slice,
+ ...(opts.signal !== undefined ? { signal: opts.signal } : {}),
+ },
+ );
+ } catch (e) {
+ const latencyMs = Date.now() - started;
+ noteDecisionLatency(latencyMs);
+ const reason: DecisionDegradeReason =
+ e instanceof DecisionProviderError ? e.reason : "network";
+ record(attempt, reason, {
+ questionCount,
+ candidateCount: ready.candidateCount,
+ context: ready.context,
+ latencyMs,
+ });
+ // Only a route that could not answer falls through; an HTTP or
+ // reply-shape failure is the endpoint's answer about THIS
+ // request, and retrying it elsewhere would multiply the same
+ // refusal.
+ if (!isFailoverReason(reason)) {
+ return { status: "degraded", mode: attempt.mode, reason };
+ }
+ last = { reason, mode: attempt.mode };
+ continue;
+ }
const latencyMs = Date.now() - started;
noteDecisionLatency(latencyMs);
- const reason: DecisionDegradeReason =
- e instanceof DecisionProviderError ? e.reason : "network";
- record(reason, {
+ record(attempt, "ok", {
questionCount,
candidateCount: ready.candidateCount,
+ response,
context: ready.context,
latencyMs,
});
- return { status: "degraded", mode, reason };
+ return { status: "ok", mode: attempt.mode, response, context: ready.context, latencyMs };
}
- const latencyMs = Date.now() - started;
- noteDecisionLatency(latencyMs);
- record("ok", {
- questionCount,
- candidateCount: ready.candidateCount,
- response,
- context: ready.context,
- latencyMs,
- });
- return { status: "ok", mode, response, context: ready.context, latencyMs };
+ return {
+ status: "degraded",
+ mode: last?.mode ?? mode,
+ reason: last?.reason ?? "network",
+ };
};
const release = reserveInFlight(gated ? cfg.vault : null, reservation);
try {
diff --git a/src/core/decision-model/thresholds-fit.ts b/src/core/decision-model/thresholds-fit.ts
new file mode 100644
index 00000000..9bdf940d
--- /dev/null
+++ b/src/core/decision-model/thresholds-fit.ts
@@ -0,0 +1,292 @@
+/**
+ * Threshold fitting: derive one use's action threshold from the use's own
+ * labelled report data against a false-action target (issue #264).
+ *
+ * The data is what the per-use reports already join - dedup and tension
+ * verdicts to the operator's later status, label suggestions to the
+ * note's later acceptance, extract plans to later citations. A sample is
+ * one decidable observation: the model's score and whether acting on it
+ * was right. The fit sweeps the threshold over the observed scores and
+ * keeps the value that acts on the most samples while holding the share
+ * of false actions at or under the target. Nothing meets the target, or
+ * too few samples decide, and nothing is written - the refusal names the
+ * best the data could do.
+ *
+ * The value is written for the ANSWERING model only: a fit whose records
+ * pool several models is refused by name, so one model's calibration
+ * never leaks into another's thresholds.
+ */
+
+import type { DecisionModelUse } from "./contract.ts";
+import { DEDUP_QUESTIONS, PRIMARY_THRESHOLD_NAME, TENSION_QUESTIONS } from "./questions.ts";
+import {
+ readThresholdsStore,
+ storedThresholdEntry,
+ writeThresholdEntry,
+ type DecisionThresholdEntry,
+} from "./thresholds.ts";
+import { buildPairVerdictSamples } from "./reports/pair-verdict.ts";
+import { buildLabelSamples } from "./reports/labels.ts";
+import { buildExtractRegretSamples } from "./reports/extract-prefilter.ts";
+
+/** Fewer decidable samples than this and a fit refuses to speak. */
+export const MIN_FIT_SAMPLES = 8;
+
+/** One decidable observation: the model's score and whether acting was wrong. */
+export interface ThresholdFitSample {
+ readonly score: number;
+ readonly falseAction: boolean;
+}
+
+/**
+ * Which side of the threshold an action lies on: `at_or_above` acts on
+ * scores at or above the value (a demotion, a suggestion), `below` acts
+ * on scores below it (a dropped turn).
+ */
+export type ThresholdFitDirection = "at_or_above" | "below";
+
+/** A threshold the fit evaluated, feasible or not. */
+export interface ThresholdFitCandidate {
+ readonly threshold: number;
+ readonly acted: number;
+ readonly falseActions: number;
+ readonly falseShare: number | null;
+}
+
+export type ThresholdFitOutcome =
+ | {
+ readonly ok: true;
+ readonly candidate: ThresholdFitCandidate;
+ }
+ | {
+ readonly ok: false;
+ readonly note: string;
+ /** The closest try, for the refusal to name. */
+ readonly best: ThresholdFitCandidate | null;
+ };
+
+/**
+ * Sweep the observed scores and keep the threshold that acts on the most
+ * samples while holding the false-action share at or under `target`.
+ * Every threshold considered needs at least `minSamples` actions behind
+ * it: a share measured on two demotions says nothing.
+ */
+export function fitThresholdAgainstTarget(
+ samples: ReadonlyArray<ThresholdFitSample>,
+ direction: ThresholdFitDirection,
+ target: number,
+ minSamples: number = MIN_FIT_SAMPLES,
+): ThresholdFitOutcome {
+ const evaluate = (threshold: number): ThresholdFitCandidate => {
+ const acted =
+ direction === "at_or_above"
+ ? samples.filter((s) => s.score >= threshold)
+ : samples.filter((s) => s.score < threshold);
+ const falseActions = acted.filter((s) => s.falseAction).length;
+ return {
+ threshold,
+ acted: acted.length,
+ falseActions,
+ falseShare: acted.length === 0 ? null : falseActions / acted.length,
+ };
+ };
+ const scores = [...new Set(samples.map((s) => s.score))].toSorted((a, b) => a - b);
+ const candidates = direction === "at_or_above" ? scores : [...scores, Math.max(1, ...scores)];
+ let best: ThresholdFitCandidate | null = null;
+ let feasible: ThresholdFitCandidate | null = null;
+ for (const threshold of candidates) {
+ const candidate = evaluate(threshold);
+ if (candidate.acted > (best?.acted ?? -1)) best = candidate;
+ if (
+ candidate.acted >= minSamples &&
+ candidate.falseShare !== null &&
+ candidate.falseShare <= target
+ ) {
+ // The most action is the goal; ties keep the earlier (lower or
+ // tighter) threshold, which acts on exactly the same set.
+ if (feasible === null || candidate.acted > feasible.acted) feasible = candidate;
+ }
+ }
+ if (samples.length < minSamples) {
+ return {
+ ok: false,
+ note: `only ${samples.length} decidable sample(s), ${minSamples} needed`,
+ best,
+ };
+ }
+ if (feasible === null) {
+ const bestNote =
+ best === null || best.acted === 0 || best.falseShare === null
+ ? "no threshold in the data acts on enough samples"
+ : `the closest is ${best.threshold} with ${(best.falseShare * 100).toFixed(1)}% false ` +
+ `actions over ${best.acted} action(s)`;
+ return {
+ ok: false,
+ note: `no threshold meets a ${target} false-action target: ${bestNote}`,
+ best,
+ };
+ }
+ return { ok: true, candidate: feasible };
+}
+
+/** What each use fits from, and how. The specless uses are absent. */
+export const FITTABLE_USES: ReadonlyArray<DecisionModelUse> = Object.freeze([
+ "dedup",
+ "tension",
+ "labels",
+ "extract_prefilter",
+]);
+
+export function isFittableUse(use: DecisionModelUse): boolean {
+ return FITTABLE_USES.includes(use);
+}
+
+/**
+ * The fit samples of one use from the vault's labelled records, with the
+ * answering models the records name. A model of `unknown` is skipped for
+ * the model identity but keeps its sample: the data still counts.
+ */
+export function fitSamplesForUse(
+ vault: string,
+ use: DecisionModelUse,
+ opts: { readonly since?: string } = {},
+): { readonly samples: ReadonlyArray<ThresholdFitSample>; readonly models: ReadonlyArray<string> } {
+ if (use === "dedup" || use === "tension") {
+ const lowPriority: ReadonlyArray<string> =
+ use === "dedup" ? DEDUP_QUESTIONS.lowPriority : TENSION_QUESTIONS.lowPriority;
+ const { samples } = buildPairVerdictSamples(vault, use, opts);
+ const usable = samples.filter((s) => lowPriority.includes(s.verdict));
+ return {
+ samples: usable.map((s) => ({
+ score: s.probability,
+ // The action said "not this" and the operator later said it was:
+ // a merged pair was the same rule, a confirmed (or resolved)
+ // tension was real.
+ falseAction: use === "dedup" ? s.outcome === "merged" : s.outcome === "confirmed",
+ })),
+ models: usable.map((s) => s.model),
+ };
+ }
+ if (use === "labels") {
+ const { samples } = buildLabelSamples(vault, opts);
+ const usable = samples.filter((s) => s.suggested && s.confidence !== null);
+ return {
+ samples: usable.map((s) => ({
+ score: s.confidence!,
+ // Suggesting a value the operator later replaced was the false
+ // action; accepting it was the true one. Unchanged, already and
+ // missing do not decide either way and drop out of the share.
+ falseAction: s.outcome === "overridden",
+ })),
+ models: usable.map((s) => s.model),
+ };
+ }
+ if (use === "extract_prefilter") {
+ const { samples } = buildExtractRegretSamples(vault, opts);
+ return {
+ samples: samples.map((s) => ({ score: s.probability, falseAction: s.cited })),
+ models: samples.map((s) => s.model),
+ };
+ }
+ return { samples: [], models: [] };
+}
+
+/** The named threshold a use fits: the one value the action reads. */
+export function fittedThresholdName(use: DecisionModelUse): string {
+ return PRIMARY_THRESHOLD_NAME[use];
+}
+
+export type ThresholdFitRun =
+ | {
+ readonly ok: true;
+ readonly model: string;
+ readonly use: DecisionModelUse;
+ readonly name: string;
+ readonly entry: DecisionThresholdEntry;
+ readonly alreadyTuned: boolean;
+ }
+ | { readonly ok: false; readonly note: string };
+
+/**
+ * Run the fit for one use and, when it and the data agree, write the
+ * entry for the answering model. `alreadyTuned` says whether the model
+ * had an entry for this use before (the fit refreshed it).
+ */
+export function runThresholdFit(
+ storePath: string,
+ input: {
+ readonly vault: string;
+ readonly use: DecisionModelUse;
+ readonly since?: string;
+ /** Highest false-action share the fitted threshold may carry. */
+ readonly target?: number;
+ },
+ now: Date = new Date(),
+): ThresholdFitRun {
+ if (!isFittableUse(input.use)) {
+ return {
+ ok: false,
+ note:
+ `use '${input.use}' has no labelled report to fit from; fittable uses: ` +
+ FITTABLE_USES.join(", "),
+ };
+ }
+ const target = input.target ?? DEFAULT_FIT_TARGET;
+ const { samples, models } = fitSamplesForUse(input.vault, input.use, {
+ since: input.since,
+ });
+ const named = [...new Set(models.filter((m) => m !== "unknown" && m !== ""))];
+ if (named.length === 0) {
+ return {
+ ok: false,
+ note: `no ok ordinary records for use '${input.use}' name a model; run it in shadow first`,
+ };
+ }
+ if (named.length > 1) {
+ return {
+ ok: false,
+ note:
+ `the records pool several models (${named.join(", ")}); a threshold is one model's ` +
+ "calibration, so narrow the window with --since until one model remains",
+ };
+ }
+ const model = named[0]!;
+ const fit = fitThresholdAgainstTarget(samples, fitDirection(input.use), target);
+ if (!fit.ok) return { ok: false, note: fit.note };
+ const alreadyTuned =
+ storedThresholdEntry(readStoreFor(storePath), model, input.use, null) !== null;
+ const entry = writeThresholdEntry(
+ storePath,
+ model,
+ input.use,
+ {
+ values: { [fittedThresholdName(input.use)]: fit.candidate.threshold },
+ source: "fit",
+ target,
+ samples: fit.candidate.acted,
+ },
+ null,
+ now,
+ );
+ return {
+ ok: true,
+ model,
+ use: input.use,
+ name: fittedThresholdName(input.use),
+ entry,
+ alreadyTuned,
+ };
+}
+
+/** The default false-action target: five wrong actions in a hundred. */
+export const DEFAULT_FIT_TARGET = 0.05;
+
+function fitDirection(use: DecisionModelUse): ThresholdFitDirection {
+ return use === "extract_prefilter" ? "below" : "at_or_above";
+}
+
+function readStoreFor(storePath: string): ReturnType<typeof readThresholdsStore> {
+ // A cache-busting read (nowMs 0 is older than any TTL window) so a fit
+ // that follows a hand edit sees the file as it is on disk.
+ return readThresholdsStore(storePath, 0);
+}
diff --git a/src/core/decision-model/thresholds.ts b/src/core/decision-model/thresholds.ts
new file mode 100644
index 00000000..d1c8373f
--- /dev/null
+++ b/src/core/decision-model/thresholds.ts
@@ -0,0 +1,323 @@
+/**
+ * The local per-model thresholds store (issue #264).
+ *
+ * Thresholds are a property of the installation, not of the code. They
+ * live in `decision-thresholds.json`, next to `config.yaml` in the user
+ * config directory, keyed by `(model id, use)` - the model id the server
+ * reports, never the provider preset, so a self-hosted server that swaps
+ * models never inherits the previous model's thresholds.
+ *
+ * Semantics the whole feature relies on:
+ *
+ * - An entry exists for `(model, use)` exactly when that use may run
+ * `enforce` for that model. A model without an entry for a use runs
+ * that use as `shadow`, and `check`/`report` name the tuning command.
+ * - Entries are never deleted by a switch: the file keeps every model
+ * ever tuned, so switching back restores enforce without re-tuning.
+ * Writing an entry keeps the previous value in the entry's history.
+ * - A `vaults` section overrides the global entry for one vault, keyed
+ * by the vault's absolute path.
+ * - The code's built-in threshold values ({@link BUILTIN_USE_THRESHOLDS}
+ * in `questions.ts`, the hosted baseline family's) remain the
+ * fallback; a stored value for the same name wins.
+ *
+ * The file is small and read through a TTL cache (the tuning-mark
+ * precedent, `eval/mark.ts`), so a hot path pays one cached read per
+ * minute. Every write is atomic and busts the cache. A missing, unreadable
+ * or malformed file reads as an empty store: fail closed to shadow, and
+ * never throw - the store only ever widens what may enforce.
+ */
+
+import { mkdirSync, readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+
+import { atomicWriteFileSync } from "../fs-atomic.ts";
+import { DECISION_MODEL_USES, type DecisionModelUse } from "./contract.ts";
+import { BUILTIN_USE_THRESHOLDS } from "./questions.ts";
+
+/** The file name, read next to the operator's `config.yaml`. */
+export const DECISION_THRESHOLDS_FILE = "decision-thresholds.json";
+
+/** How long a loaded store is trusted before its file is read again. */
+export const THRESHOLDS_CACHE_TTL_MS = 60_000;
+
+/** How many previous values one entry's history keeps. */
+export const THRESHOLD_HISTORY_MAX = 20;
+
+/** Where a threshold value came from: the fit command or a hand set. */
+export type DecisionThresholdSource = "fit" | "set";
+
+export interface DecisionThresholdHistoryItem {
+ readonly value: number;
+ readonly source: string;
+ readonly updated_at: string;
+}
+
+export interface DecisionThresholdEntry {
+ /** The threshold values this entry carries, by name (`low_priority_min`). */
+ readonly values: Readonly<Record<string, number>>;
+ readonly source: DecisionThresholdSource;
+ readonly updated_at: string;
+ /** The false-action target the fit derived the value against. */
+ readonly target?: number;
+ /** How many labelled samples the fit saw. */
+ readonly samples?: number;
+ /** Previous values of this entry, most recent first. Never trimmed by a switch. */
+ readonly history?: ReadonlyArray<DecisionThresholdHistoryItem>;
+}
+
+/** One model's entries by use; a use without an entry is untuned for it. */
+export type DecisionThresholdModelEntries = Readonly<
+ Partial<Record<DecisionModelUse, DecisionThresholdEntry>>
+>;
+
+export interface DecisionThresholdsStore {
+ readonly version: 1;
+ readonly models: Readonly<Record<string, DecisionThresholdModelEntries>>;
+ /** Per-vault overrides, keyed by the vault's absolute path. */
+ readonly vaults: Readonly<
+ Record<string, Readonly<Record<string, DecisionThresholdModelEntries>>>
+ >;
+}
+
+/** The path of the thresholds file beside `configPath`. */
+export function decisionThresholdsPath(configPath: string): string {
+ return join(dirname(configPath), DECISION_THRESHOLDS_FILE);
+}
+
+export const EMPTY_THRESHOLD_STORE: DecisionThresholdsStore = Object.freeze({
+ version: 1,
+ models: Object.freeze({}),
+ vaults: Object.freeze({}),
+});
+
+/** Parse one store document. Anything malformed parses as empty (fail closed). */
+export function parseThresholdsStore(text: string): DecisionThresholdsStore {
+ let raw: unknown;
+ try {
+ raw = JSON.parse(text);
+ } catch {
+ return EMPTY_THRESHOLD_STORE;
+ }
+ if (raw === null || typeof raw !== "object" || Array.isArray(raw)) {
+ return EMPTY_THRESHOLD_STORE;
+ }
+ const r = raw as Record<string, unknown>;
+ const models = parseModelSection(r["models"]);
+ const vaults: Record<string, DecisionThresholdsStore["vaults"][string]> = {};
+ if (r["vaults"] !== null && typeof r["vaults"] === "object" && !Array.isArray(r["vaults"])) {
+ for (const [vaultPath, section] of Object.entries(r["vaults"] as Record<string, unknown>)) {
+ const parsed = parseModelSection(section);
+ if (parsed !== null) vaults[vaultPath] = parsed;
+ }
+ }
+ return { version: 1, models: models ?? {}, vaults };
+}
+
+function parseModelSection(raw: unknown): DecisionThresholdsStore["models"] | null {
+ if (raw === null || typeof raw !== "object" || Array.isArray(raw)) return null;
+ const models: Record<string, DecisionThresholdModelEntries> = {};
+ for (const [model, uses] of Object.entries(raw as Record<string, unknown>)) {
+ if (typeof model !== "string" || model === "" || uses === null || typeof uses !== "object") {
+ continue;
+ }
+ const entries: Record<string, DecisionThresholdEntry> = {};
+ for (const use of DECISION_MODEL_USES) {
+ const entry = parseEntry((uses as Record<string, unknown>)[use]);
+ if (entry !== null) entries[use] = entry;
+ }
+ if (Object.keys(entries).length > 0) models[model] = entries;
+ }
+ return models;
+}
+
+function parseEntry(raw: unknown): DecisionThresholdEntry | null {
+ if (raw === null || typeof raw !== "object" || Array.isArray(raw)) return null;
+ const e = raw as Record<string, unknown>;
+ if (
+ e["values"] === null ||
+ typeof e["values"] !== "object" ||
+ Array.isArray(e["values"]) ||
+ !Object.values(e["values"]).every((v) => typeof v === "number" && Number.isFinite(v)) ||
+ Object.keys(e["values"]).length === 0 ||
+ (e["source"] !== "fit" && e["source"] !== "set") ||
+ typeof e["updated_at"] !== "string"
+ ) {
+ return null;
+ }
+ const history: DecisionThresholdHistoryItem[] = [];
+ if (
+ e["history"] !== null &&
+ Array.isArray(e["history"]) &&
+ e["history"].every(
+ (h) =>
+ h !== null &&
+ typeof h === "object" &&
+ typeof (h as Record<string, unknown>)["value"] === "number" &&
+ typeof (h as Record<string, unknown>)["updated_at"] === "string",
+ )
+ ) {
+ for (const h of e["history"] as Record<string, unknown>[]) {
+ history.push({
+ value: h["value"] as number,
+ source: typeof h["source"] === "string" ? h["source"] : "unknown",
+ updated_at: h["updated_at"] as string,
+ });
+ }
+ }
+ return {
+ values: Object.freeze({ ...(e["values"] as Record<string, number>) }),
+ source: e["source"],
+ updated_at: e["updated_at"],
+ ...(typeof e["target"] === "number" && Number.isFinite(e["target"])
+ ? { target: e["target"] }
+ : {}),
+ ...(typeof e["samples"] === "number" && Number.isFinite(e["samples"])
+ ? { samples: e["samples"] }
+ : {}),
+ ...(history.length > 0 ? { history } : {}),
+ };
+}
+
+const STORE_CACHE = new Map<string, { store: DecisionThresholdsStore; loadedAtMs: number }>();
+
+/** Tests only: forget every loaded store. */
+export function resetThresholdsStoreCache(): void {
+ STORE_CACHE.clear();
+}
+
+/**
+ * Read the store at `path` through the TTL cache. Anything unreadable or
+ * malformed reads as the empty store, and that absence is cached too, so
+ * a missing file costs one stat-and-read per {@link THRESHOLDS_CACHE_TTL_MS}.
+ */
+export function readThresholdsStore(
+ path: string,
+ nowMs: number = Date.now(),
+): DecisionThresholdsStore {
+ const hit = STORE_CACHE.get(path);
+ if (hit !== undefined && nowMs - hit.loadedAtMs < THRESHOLDS_CACHE_TTL_MS) return hit.store;
+ let store: DecisionThresholdsStore;
+ try {
+ store = parseThresholdsStore(readFileSync(path, "utf8"));
+ } catch {
+ store = EMPTY_THRESHOLD_STORE;
+ }
+ STORE_CACHE.set(path, { store, loadedAtMs: nowMs });
+ return store;
+}
+
+/**
+ * The stored entry for `(model, use)`: the vault's override when the
+ * vault declares one, else the global entry, else null.
+ */
+export function storedThresholdEntry(
+ store: DecisionThresholdsStore,
+ model: string | null,
+ use: DecisionModelUse,
+ vault?: string | null,
+): DecisionThresholdEntry | null {
+ if (model === null || model === "") return null;
+ const entry = store.models[model]?.[use];
+ const override = vault != null ? store.vaults[vault]?.[model]?.[use] : undefined;
+ return override ?? entry ?? null;
+}
+
+/**
+ * The threshold values a use applies for `model`: the built-in defaults
+ * with every stored value of the same name winning. Without a stored
+ * entry this is exactly the built-in baseline family's set.
+ */
+export function effectiveThresholds(
+ store: DecisionThresholdsStore,
+ model: string | null,
+ use: DecisionModelUse,
+ vault?: string | null,
+): Readonly<Record<string, number>> {
+ const entry = storedThresholdEntry(store, model, use, vault);
+ if (entry === null) return Object.freeze({ ...BUILTIN_USE_THRESHOLDS[use] });
+ return Object.freeze({ ...BUILTIN_USE_THRESHOLDS[use], ...entry.values });
+}
+
+/**
+ * Merge `entry` into the store at `path` atomically. A previous entry for
+ * the same `(model, use)` keeps its current value in the entry's history
+ * (most recent first, capped); nothing is ever deleted, so switching
+ * models and switching back never loses a tune. The cache entry is
+ * dropped so the next read sees the write.
+ */
+export function writeThresholdEntry(
+ path: string,
+ model: string,
+ use: DecisionModelUse,
+ entry: {
+ readonly values: Readonly<Record<string, number>>;
+ readonly source: DecisionThresholdSource;
+ readonly target?: number;
+ readonly samples?: number;
+ },
+ vaultPath?: string | null,
+ now: Date = new Date(),
+): DecisionThresholdEntry {
+ let store: DecisionThresholdsStore;
+ try {
+ store = parseThresholdsStore(readFileSync(path, "utf8"));
+ } catch {
+ store = EMPTY_THRESHOLD_STORE;
+ }
+ // History first: the entry being replaced keeps its old value, latest
+ // first, capped - nothing is ever deleted, so switching models and
+ // switching back never loses a tune.
+ const previous =
+ vaultPath == null
+ ? store.models[model]?.[use]
+ : (store.vaults[vaultPath]?.[model]?.[use] ?? store.models[model]?.[use]);
+ const history =
+ previous === undefined || previous === null
+ ? undefined
+ : Object.freeze(
+ [
+ {
+ value: primaryValueOf(previous),
+ source: previous.source,
+ updated_at: previous.updated_at,
+ },
+ ...(previous.history ?? []),
+ ].slice(0, THRESHOLD_HISTORY_MAX),
+ );
+ const updated: DecisionThresholdEntry = {
+ values: Object.freeze({ ...entry.values }),
+ source: entry.source,
+ updated_at: now.toISOString(),
+ ...(entry.target !== undefined ? { target: entry.target } : {}),
+ ...(entry.samples !== undefined ? { samples: entry.samples } : {}),
+ ...(history !== undefined ? { history } : {}),
+ };
+ const section: Record<string, DecisionThresholdEntry> = {
+ ...(vaultPath == null
+ ? store.models[model]
+ : (store.vaults[vaultPath]?.[model] ?? store.models[model])),
+ };
+ section[use] = updated;
+ const sectionEntries: DecisionThresholdModelEntries = section;
+ const vaults: DecisionThresholdsStore["vaults"] =
+ vaultPath == null
+ ? store.vaults
+ : {
+ ...store.vaults,
+ [vaultPath]: { ...store.vaults[vaultPath], [model]: sectionEntries },
+ };
+ const globalModels: DecisionThresholdsStore["models"] =
+ vaultPath == null ? { ...store.models, [model]: sectionEntries } : store.models;
+ const next: DecisionThresholdsStore = { version: 1, models: globalModels, vaults };
+ mkdirSync(dirname(path), { recursive: true });
+ atomicWriteFileSync(path, `${JSON.stringify(next, null, 2)}\n`);
+ STORE_CACHE.delete(path);
+ return updated;
+}
+
+/** The entry's primary stored value, for the history line. */
+function primaryValueOf(entry: DecisionThresholdEntry): number {
+ const first = Object.values(entry.values)[0];
+ return typeof first === "number" ? first : Number.NaN;
+}
diff --git a/src/core/doctor-readiness.ts b/src/core/doctor-readiness.ts
index 37ec51ab..54b9324c 100644
--- a/src/core/doctor-readiness.ts
+++ b/src/core/doctor-readiness.ts
@@ -95,6 +95,8 @@ import { providerProducesVectors } from "./search/embeddings/contract.ts";
import { makeProvider } from "./search/embeddings/provider.ts";
import { resolveSearchConfig } from "./search/index.ts";
import { providerConsentIsDeclined, resolveProviderConsent } from "./search/provider-consent.ts";
+import { envKeyList, loadProviderRegistry } from "./search/embeddings/registry.ts";
+import { loadRerankRegistry } from "./search/rerank/registry.ts";
import {
COMMAND_PROBE_VERDICT,
CommandProbeError,
@@ -141,21 +143,24 @@ export const READINESS_PROBE = {
const PROVIDERS_REQUIRING_API_KEY: ReadonlySet<string> = new Set(["openai-compat", "zeroentropy"]);
/**
- * The environment-variable PREFIXES that name an embedding-provider
- * setting.
+ * The environment-variable PREFIXES that name an embedding-provider or
+ * rerank setting.
*
* A prefix rather than a hand-written list, because a list would be a
* second copy of the resolver's spelling table and would drift: a new
* `OPEN_SECOND_BRAIN_EMBEDDING_*` key added to the resolver would not
* appear here, and the probe would then under-report what is present in
- * the environment - which is the one thing this probe exists to say.
+ * the environment - which is the one thing this probe exists to say. The
+ * rerank resolver spells its settings `OPEN_SECOND_BRAIN_SEARCH_RERANK_*`
+ * and is missed by the embedding prefix alone.
*/
const PROVIDER_ENV_PREFIXES: ReadonlyArray<string> = Object.freeze([
"OPEN_SECOND_BRAIN_EMBEDDING_",
+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_",
]);
/**
- * The provider environment variables that do NOT carry the prefix above.
+ * The provider environment variables that do NOT carry a prefix above.
* The switch is one of them: it is the environment twin of the decline
* record, and an operator who set it in the environment rather than in
* the config file is exactly the case this probe has to reach.
@@ -164,9 +169,48 @@ const PROVIDER_ENV_NAMES: ReadonlyArray<string> = Object.freeze([
"OPEN_SECOND_BRAIN_SEARCH_SEMANTIC",
]);
+/**
+ * The env-var NAMES the operator's own provider registrations and rerank
+ * settings point at: every embedding profile's `envKey` in
+ * `Brain/search/embedding-providers.json`, every rerank profile's
+ * `envKey` in `Brain/search/rerank-providers.json`, and the variable the
+ * `search_rerank_env_key` setting names. A registry profile keeps its
+ * credential in an operator-named variable (`OPENROUTER_API_KEY`, say),
+ * which no prefix above can reach - and the probe that misses it reports
+ * exactly the credentials an operator set as absent.
+ *
+ * Reads are fail-soft: a missing registry contributes nothing, and a
+ * config that cannot be read contributes its env spelling only.
+ */
+function operatorNamedProviderEnvNames(
+ opts: ReadinessOptions,
+ env: Readonly<Record<string, string | undefined>>,
+): ReadonlyArray<string> {
+ const names = new Set<string>();
+ for (const profile of loadProviderRegistry(opts.vault)) {
+ for (const name of envKeyList(profile.envKey)) names.add(name);
+ }
+ for (const profile of loadRerankRegistry(opts.vault)) names.add(profile.envKey);
+ let config: Readonly<Record<string, string>> = {};
+ try {
+ config = discoverConfig(resolveConfigPath(opts)).data;
+ } catch {
+ // The consent resolver reports the unreadable record by name; the
+ // env census here simply cannot add its config half.
+ }
+ const rerankEnvKey =
+ env["OPEN_SECOND_BRAIN_SEARCH_RERANK_ENV_KEY"] ?? config["search_rerank_env_key"];
+ if (typeof rerankEnvKey === "string" && rerankEnvKey.trim() !== "") {
+ names.add(rerankEnvKey.trim());
+ }
+ return Object.freeze([...names]);
+}
+
/**
* The NAMES of the provider environment variables present in `env`, in
- * alphabetical order.
+ * alphabetical order. `operatorNamed` extends the membership with the
+ * env-var names the operator's own provider and rerank registrations
+ * point at.
*
* NAMES ONLY, never values: a readiness detail travels into logs, into a
* JSON payload an operator pastes into a bug report and into the
@@ -180,6 +224,7 @@ const PROVIDER_ENV_NAMES: ReadonlyArray<string> = Object.freeze([
*/
function presentProviderEnvNames(
env: Readonly<Record<string, string | undefined>>,
+ operatorNamed: ReadonlyArray<string> = [],
): ReadonlyArray<string> {
const names: string[] = [];
for (const name of Object.keys(env).toSorted()) {
@@ -187,6 +232,7 @@ function presentProviderEnvNames(
if (value === undefined || value === "") continue;
const isProviderEnv =
PROVIDER_ENV_NAMES.includes(name) ||
+ operatorNamed.includes(name) ||
PROVIDER_ENV_PREFIXES.some((prefix) => name.startsWith(prefix));
if (isProviderEnv) names.push(name);
}
@@ -405,30 +451,41 @@ export async function probeLlmKey(opts: ReadinessOptions): Promise<ReadinessVerd
* first one is named.
*
* `skipped`, not `fail`: a decline is a decision, and a decision is not
- * a fault. The detail names the two spellings a decline is written in,
- * so the operator can see which one is in force, and lists the provider
- * environment variables that are PRESENT - by NAME. The names are the
- * point: an operator who set `OPEN_SECOND_BRAIN_EMBEDDING_KEY` in their
- * environment and then wrote a decline in the config file needs to know
- * that the key is still there, and a readiness detail is the wrong
- * place to publish a credential.
+ * a fault. The detail names the keys the decline was written in, so the
+ * operator can see which one is in force, and lists the provider
+ * environment variables that are PRESENT - by NAME, on every arm,
+ * because a key sitting in the environment beside a recorded decline (or
+ * beside no decision at all) is exactly the fact that explains the
+ * configuration's behaviour. The names are the point: an operator who set
+ * `OPEN_SECOND_BRAIN_EMBEDDING_KEY` in their environment and then wrote a
+ * decline in the config file needs to know that the key is still there,
+ * and a readiness detail is the wrong place to publish a credential.
+ *
+ * `unknown`, not `skipped`, when the record could not be read: the
+ * resolver fails that arm closed to declined (an unreadable record is
+ * not consent), but this probe answers whether a decision EXISTS, and a
+ * record nobody could read is a measurement that did not complete. The
+ * CLI already carries `unknown` for exactly that, and reporting the
+ * fail-closed decline as the operator's choice showed a broken machine
+ * configuration as 0 failed / 0 unknown - a clean bill nobody earned.
*/
export async function probeProviderConsent(opts: ReadinessOptions): Promise<ReadinessVerdict> {
const configPath = resolveConfigPath(opts);
const env = opts.env ?? process.env;
const consent = resolveProviderConsent({ configPath, env });
- if (!providerConsentIsDeclined(consent)) {
+ const present = presentProviderEnvNames(env, operatorNamedProviderEnvNames(opts, env));
+ const presentSentence =
+ ` Present provider environment variables: ` +
+ `${present.length > 0 ? present.join(", ") : "none"}.`;
+ if (consent.unreadableReason !== null) {
return {
- status: READINESS_STATUS.skipped,
- detail: consent.detail,
+ status: READINESS_STATUS.unknown,
+ detail: `${consent.detail}${presentSentence}`,
};
}
- const present = presentProviderEnvNames(env);
return {
status: READINESS_STATUS.skipped,
- detail:
- `${consent.detail} Present provider environment variables: ` +
- `${present.length > 0 ? present.join(", ") : "none"}.`,
+ detail: `${consent.detail}${presentSentence}`,
};
}
diff --git a/src/core/graph/privacy-markers.ts b/src/core/graph/privacy-markers.ts
index 7596e5c0..01c97426 100644
--- a/src/core/graph/privacy-markers.ts
+++ b/src/core/graph/privacy-markers.ts
@@ -29,7 +29,9 @@
*
* The values are token-matched through {@link normToken} - the same NFC +
* trim + lower-case normalisation the visibility layer uses - rather than
- * through a second normaliser. A value that spells something else (a
+ * through a second normaliser, and the KEYS are matched through it too: a
+ * vault that spells the key `Access` or `CONFIDENTIALITY` makes the same
+ * claim as the lower-case spelling. A value that spells something else (a
* phrase, a negation, a Markdown sentence) is not a deny marker: this
* module recognises the three token spellings the design pins and does not
* guess at prose.
@@ -67,16 +69,42 @@ function declaredValues(raw: unknown): string[] {
return raw.filter((v): v is string => typeof v === "string");
}
+/**
+ * The marker keys in the normal form {@link normToken} produces. The values
+ * are matched after normalisation, so the KEYS are too: a vault that spells
+ * the key `Access`, `ACCESS` or with stray surrounding whitespace makes the
+ * same claim as the lower-case spelling, and a reader that matched keys by
+ * exact case would hand that claim to a remote reader.
+ */
+const PRIVACY_MARKER_KEY_TOKENS: ReadonlySet<string> = Object.freeze(
+ new Set(PRIVACY_MARKER_KEYS.map((key) => normToken(key))),
+);
+
+/**
+ * Does this frontmatter carry ANY key under a marker key spelling, whatever
+ * the value? The migration re-stamping the indexed visibility column gates
+ * on this, because a row whose frontmatter carries a marker key at all is a
+ * row the column's pre-marker stamp may understate.
+ */
+export function hasPrivacyMarkerKey(meta: FrontmatterMap): boolean {
+ for (const key of Object.keys(meta)) {
+ if (PRIVACY_MARKER_KEY_TOKENS.has(normToken(key))) return true;
+ }
+ return false;
+}
+
/**
* Does this frontmatter carry a privacy-denying informal marker value?
*
- * True when any of {@link PRIVACY_MARKER_KEYS} declares a string (or an
- * array holding a string) whose {@link normToken} form is one of
+ * True when any key whose normalised spelling is one of
+ * {@link PRIVACY_MARKER_KEYS} declares a string (or an array holding a
+ * string) whose {@link normToken} form is one of
* {@link PRIVACY_DENY_MARKER_VALUES}. Absent keys, non-string scalars,
* empty values and every other token answer false.
*/
export function hasPrivacyDenyMarker(meta: FrontmatterMap): boolean {
- for (const key of PRIVACY_MARKER_KEYS) {
+ for (const key of Object.keys(meta)) {
+ if (!PRIVACY_MARKER_KEY_TOKENS.has(normToken(key))) continue;
for (const value of declaredValues(meta[key])) {
if (PRIVACY_DENY_MARKER_VALUES.includes(normToken(value))) return true;
}
diff --git a/src/core/install/ownership.ts b/src/core/install/ownership.ts
index 62e7e2b7..257c7a9d 100644
--- a/src/core/install/ownership.ts
+++ b/src/core/install/ownership.ts
@@ -372,6 +372,22 @@ export const OUT_OF_VAULT_STATE: ReadonlyArray<OutOfVaultState> = Object.freeze(
"merely puts that use back in shadow until the eval promotes it again.",
sources: ["src/core/decision-model/eval/mark.ts", "src/core/platform-dirs.ts"],
},
+ {
+ id: "decision_model_thresholds",
+ label: "decision-model thresholds",
+ location: WINDOWS
+ ? `the directory %OPEN_SECOND_BRAIN_CONFIG% names, else (${describeBaseDir("config", "win32")})\\open-second-brain\\decision-thresholds.json`
+ : "beside $OPEN_SECOND_BRAIN_CONFIG, else ${XDG_CONFIG_HOME:-~/.config}/open-second-brain/decision-thresholds.json",
+ carries_memory: false,
+ created_by: "`o2b decision-model thresholds fit`, or `thresholds set` by hand",
+ removed_by: "deleting the file yourself; the kept history in it goes with it",
+ note:
+ "Per-use enforce thresholds keyed by the model id the server reports, so a self-hosted " +
+ "server that swaps models never inherits the previous model's numbers. Entries of every " +
+ "model ever tuned are kept across switches, with an optional per-vault override. It " +
+ "carries numbers, names and timestamps - never vault text.",
+ sources: ["src/core/decision-model/thresholds.ts", "src/core/decision-model/config.ts"],
+ },
]);
/**
@@ -636,6 +652,20 @@ export const SOURCES_INVISIBLE_TO_THE_SWEEP: ReadonlyMap<string, string> = new M
"so the root arrives as an argument and no home or XDG token appears in the source. The " +
"config path itself is built by src/core/config.ts, which IS swept",
],
+ [
+ "src/core/decision-model/thresholds.ts",
+ "the thresholds file is `join(dirname(configPath), DECISION_THRESHOLDS_FILE)` - the root " +
+ "arrives as an argument and no home or XDG token appears in the source. The config path " +
+ "itself is built by src/core/config.ts, which IS swept; the location this file writes is " +
+ "the decision_model_thresholds row",
+ ],
+ [
+ "src/core/decision-model/config.ts",
+ "the store path arrives from the resolved config path - `decisionThresholdsPath(configPath)` " +
+ "derives `dirname(configPath)` for the store in src/core/decision-model/thresholds.ts - so " +
+ "no home or XDG token appears in this file's own source; the location it gates on is the " +
+ "decision_model_thresholds row",
+ ],
[
"hooks/active-inject.ts",
"it re-points the CLI symlinks by calling `healCliSymlinks` in src/cli/install-cli.ts, which " +
diff --git a/src/core/search/chunker.ts b/src/core/search/chunker.ts
index e51c9770..b9c1a70d 100644
--- a/src/core/search/chunker.ts
+++ b/src/core/search/chunker.ts
@@ -8,13 +8,25 @@
* Thai, Lao, Khmer, CJK punctuation and fullwidth forms), where every
* character is one token - a paragraph of Chinese has no whitespace, and
* counting it as one "word" sent 9,000-character chunks to providers that
- * reject them (issue #186). Deterministic across Bun/Node,
- * dependency-free, machine-independent โ€” so the same vault hashes the
- * same chunks on every Syncthing peer.
+ * reject them (issue #186). Long whitespace-free runs on the Latin side
+ * are the same trap with the sign flipped: a subword tokenizer spends
+ * roughly one token per two to four characters on a URL, a file path or a
+ * hash, so a run of sixteen or more such characters is charged one token
+ * per three characters instead of one (issue #272). Deterministic across
+ * Bun/Node, dependency-free, machine-independent โ€” so the same vault
+ * hashes the same chunks on every Syncthing peer.
*
* Budget invariant: every chunk's `tokenCount`, overlap included, is at
* most `maxTokens`. A block too large for one chunk is split by line, then
- * after sentence punctuation, then at a token boundary.
+ * after sentence punctuation, then at a token boundary - and inside a
+ * long whitespace-free run those token boundaries fall every few
+ * characters, so one giant line is still distributed across chunks
+ * without dropping or duplicating source text. A cut that would land
+ * inside a `[[...]]` wikilink moves out of the link whenever any
+ * alternative exists, and severs the link only when the link alone
+ * exceeds the whole budget: the indexer extracts links per chunk, so a
+ * span cut in half would vanish from the link index even though its text
+ * is still indexed.
*/
/**
@@ -26,8 +38,14 @@
* 1 - whitespace word count, overlap outside the budget (through v1.57).
* 2 - per-character count for unspaced scripts; overlap inside the
* budget; oversize blocks split (issue #186).
+ * 3 - whitespace-free runs of sixteen or more characters charged about
+ * one token per three characters and split inside the run, so a
+ * long URL or hash can no longer pass the budget as one "word"
+ * (issue #272). Cut placement keeps `[[...]]` wikilinks whole
+ * whenever any alternative cut exists, because links are extracted
+ * per chunk.
*/
-export const CHUNKER_VERSION = 2;
+export const CHUNKER_VERSION = 3;
/**
* Token budget of one chunk when the caller names none. Exported so a
@@ -96,19 +114,67 @@ const MARK_RE = /\p{M}/uy;
/** Every unspaced-script code point is at or above Thai (U+0E00). */
const FIRST_UNSPACED_CODE_POINT = 0x0e00;
+/**
+ * A whitespace-free run shorter than this many code units keeps the plain
+ * word count: ordinary prose words, however long, stay one estimated
+ * token, and the space-separated corpus hashes exactly as before. The
+ * threshold sits above any real word with its punctuation attached, where
+ * a run stops looking like prose and starts looking like a URL, a path or
+ * a hash - the shapes that hide hundreds of real tokens behind a
+ * one-"word" estimate.
+ */
+const DENSE_RUN_MIN_CHARS = 16;
+
+/**
+ * Code units charged per token inside a whitespace-free run of at least
+ * {@link DENSE_RUN_MIN_CHARS}. Subword tokenizers break such text into
+ * small fragments - about one token per two to four characters - so
+ * counting a 3,700-character URL as one "word" under-reported reality by
+ * three orders of magnitude and let a single chunk thousands of tokens
+ * past the model's input window through the budget (issue #272). Three is
+ * the conservative end of that range, chosen in the spirit of the
+ * unspaced scripts' one token per character: over-charging only splits a
+ * chunk earlier, while under-charging silently breaks the window.
+ */
+const DENSE_CHARS_PER_TOKEN = 3;
+
function matchesAt(re: RegExp, text: string, i: number): boolean {
re.lastIndex = i;
return re.test(text);
}
+/**
+ * End (exclusive) of the whitespace-free run of code points below
+ * {@link FIRST_UNSPACED_CODE_POINT} that starts at `from`: the run stops
+ * at whitespace or at the first character of an unspaced script - exactly
+ * the places the word walk stops, so the measured run is the same span
+ * the old code counted as one "word". Cyrillic, Greek and the other
+ * scripts below Thai take part, as they took part in the word they
+ * extended.
+ */
+function wordRunEnd(text: string, from: number): number {
+ let end = from;
+ while (end < text.length) {
+ const c = text.charCodeAt(end);
+ if (c >= FIRST_UNSPACED_CODE_POINT) break;
+ if (c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d) break;
+ end++;
+ }
+ return end;
+}
+
/**
* Walk `text`'s tokens: a run of non-whitespace is one token, except that
* each unspaced-script character is a token of its own (a combining mark
- * after one stays part of it). Pushes each token's start offset onto
- * `starts` when given; returns the count.
+ * after one stays part of it), and a whitespace-free run of at least
+ * {@link DENSE_RUN_MIN_CHARS} code units is charged one token per
+ * {@link DENSE_CHARS_PER_TOKEN} code units, with a token start every few
+ * characters so a cut can fall inside the run. Pushes each token's start
+ * offset onto `starts` when given; returns the count.
*
* Whitespace is exactly space, tab, LF and CR, as it always was: text
- * with no unspaced-script character counts exactly as before.
+ * with no unspaced-script character and no long unbroken run counts
+ * exactly as before.
*/
function walkTokens(text: string, starts: number[] | null): number {
let count = 0;
@@ -129,6 +195,20 @@ function walkTokens(text: string, starts: number[] | null): number {
continue;
}
if (!inWord) {
+ const runEnd = wordRunEnd(text, i);
+ if (runEnd - i >= DENSE_RUN_MIN_CHARS) {
+ const runLen = runEnd - i;
+ count += Math.ceil(runLen / DENSE_CHARS_PER_TOKEN);
+ if (starts) {
+ for (let p = i; p < runEnd; p += DENSE_CHARS_PER_TOKEN) starts.push(p);
+ }
+ i = runEnd;
+ // The run stopped on whitespace or an unspaced-script
+ // character; both start a fresh token, so the word state resets.
+ inWord = false;
+ afterUnspaced = false;
+ continue;
+ }
count++;
starts?.push(i);
inWord = true;
@@ -176,12 +256,44 @@ function tokenStarts(text: string): number[] {
return starts;
}
-/** The longest suffix of `text` holding at most `budget` tokens. */
+/**
+ * Half-open offsets of every `[[...]]` wikilink span in `text`, in order.
+ * Wikilinks do not nest, so the first `]]` closes a span, and an
+ * unterminated `[[` is no span at all - the link extractor cannot resolve
+ * one either. The indexer extracts links per chunk, so a span severed at
+ * a chunk boundary disappears from the index even though its text is
+ * still indexed.
+ */
+function wikilinkSpans(text: string): Array<{ readonly start: number; readonly end: number }> {
+ const spans: Array<{ start: number; end: number }> = [];
+ let i = 0;
+ while (i < text.length) {
+ const open = text.indexOf("[[", i);
+ if (open === -1) break;
+ const close = text.indexOf("]]", open + 2);
+ if (close === -1) break;
+ spans.push({ start: open, end: close + 2 });
+ i = close + 2;
+ }
+ return spans;
+}
+
+/**
+ * The longest suffix of `text` holding at most `budget` tokens. A cut
+ * that would open the suffix inside a wikilink retreats past the link's
+ * closing brackets: the link stays whole in the chunk it came from and
+ * the next chunk never opens with a severed fragment of it. The suffix
+ * only shrinks, so the budget holds.
+ */
function tailTokens(text: string, budget: number): string {
if (budget <= 0) return "";
const starts = tokenStarts(text);
if (starts.length <= budget) return text;
- return text.slice(starts[starts.length - budget]);
+ const from = starts[starts.length - budget]!;
+ for (const span of wikilinkSpans(text)) {
+ if (from > span.start && from < span.end) return text.slice(span.end);
+ }
+ return text.slice(from);
}
function tokensOfLines(lines: ReadonlyArray<Line>): number {
@@ -469,17 +581,34 @@ function sentences(text: string): string[] {
}
/**
- * `text` cut into pieces of exactly `budget` tokens (the last may be
+ * `text` cut into pieces of about `budget` tokens (the last may be
* shorter), at token starts: never inside a word, a surrogate pair, or
- * between a character and its combining mark. Concatenates back to `text`.
+ * between a character and its combining mark. A boundary that would fall
+ * inside a `[[...]]` wikilink moves out of the link whenever any
+ * alternative exists - the whole link is absorbed into the piece when it
+ * still fits the budget, else the cut retreats to before the link - and
+ * only a link too large for the budget on its own is severed. Piece
+ * boundaries advance monotonically, so pieces still concatenate back to
+ * `text`.
*/
function hardCut(text: string, budget: number): string[] {
const starts = tokenStarts(text);
+ const spans = wikilinkSpans(text);
const out: string[] = [];
- for (let k = 0; k < starts.length; k += budget) {
- const from = k === 0 ? 0 : starts[k]!;
- const to = k + budget < starts.length ? starts[k + budget]! : text.length;
+ let from = 0;
+ let ki = 0; // first token start at or after `from`; advances monotonically
+ while (from < text.length) {
+ while (ki < starts.length && starts[ki]! < from) ki++;
+ let to = ki + budget < starts.length ? starts[ki + budget]! : text.length;
+ for (const span of spans) {
+ if (to > span.start && to < span.end) {
+ if (countChunkTokens(text.slice(from, span.end)) <= budget) to = span.end;
+ else if (span.start > from) to = span.start;
+ break; // spans are ordered and disjoint: at most one holds `to`
+ }
+ }
out.push(text.slice(from, to));
+ from = to;
}
return out;
}
diff --git a/src/core/search/embeddings/signature.ts b/src/core/search/embeddings/signature.ts
index 1fc1dbea..aa5e997c 100644
--- a/src/core/search/embeddings/signature.ts
+++ b/src/core/search/embeddings/signature.ts
@@ -182,17 +182,37 @@ function maxNonAsciiCodePoints(extent: TextExtent): number {
}
/**
- * Fewest tokens the text can plausibly cost: every code point at a
- * quarter token, which is {@link estimateTokens} generalised from UTF-16
- * code units to code points.
+ * The code-point cost the chunk-hold proof charges a single token.
+ *
+ * Chars-over-four is an AVERAGE, not a bound: real tokenizer rates for
+ * the shipped preset families run roughly 4.5-9 code points per token on
+ * prose, tables and code, so a text estimated at `ceil(codePoints / 4)`
+ * tokens usually costs fewer, and holding on that number holds chunks
+ * that fit. This constant is the widest per-token cost the hold's proof
+ * will grant - a full code point is charged at no more than a twelfth of
+ * a token - so when the resulting bound still exceeds the window, the
+ * real request is over it whatever the tokenizer does.
+ */
+export const MAX_CODE_POINTS_PER_TOKEN = 12;
+
+/**
+ * A LOWER bound on the tokens the text can cost: every code point at no
+ * more than a twelfth of a token ({@link MAX_CODE_POINTS_PER_TOKEN}).
+ *
+ * This is the quantity the chunk skip-and-hold acts on, and the word
+ * bound is the whole point: the chars-over-four average read LOW chunks
+ * that fit the window as over it and held them from embedding for
+ * nothing. Only when even this conservative bound exceeds the window is
+ * the chunk provably over; everything between this bound and the
+ * average-rate estimate is undecided, reported as such, and still sent.
*
* Code points rather than code units because that is the unit SQLite
* counts, and because it is the SMALLER of the two - a supplementary-plane
- * character is one code point and two code units - so the floor stays a
- * floor on both sides of the seam.
+ * character is one code point and two code units - so the bound stays a
+ * bound on both sides of the seam.
*/
-export function tokenEstimateFloor(extent: TextExtent): number {
- return Math.ceil(extent.codePoints / CHARS_PER_ESTIMATED_TOKEN);
+export function tokenHoldFloor(extent: TextExtent): number {
+ return Math.ceil(extent.codePoints / MAX_CODE_POINTS_PER_TOKEN);
}
/**
@@ -200,9 +220,11 @@ export function tokenEstimateFloor(extent: TextExtent): number {
* {@link MAX_TOKENS_PER_NON_ASCII_CODE_POINT}, the rest at a quarter
* token each.
*
- * Equal to {@link tokenEstimateFloor} for pure ASCII, which is what keeps
- * an all-Latin index reporting exactly what it reported before this
- * two-sided bound existed.
+ * For pure ASCII this equals the average-rate estimate - every code point
+ * at a quarter token, {@link estimateTokens} generalised from UTF-16 code
+ * units to code points - which is what keeps an all-Latin index reporting
+ * exactly what the high arm reported before the conservative hold bound
+ * existed beside it.
*/
export function tokenEstimateCeiling(extent: TextExtent): number {
const nonAscii = maxNonAsciiCodePoints(extent);
diff --git a/src/core/search/entity-alias.ts b/src/core/search/entity-alias.ts
index 83c7d58a..652e4ad3 100644
--- a/src/core/search/entity-alias.ts
+++ b/src/core/search/entity-alias.ts
@@ -17,31 +17,50 @@
* can tell a declared-form match from a derived-variant match; the
* derivation kernel is the leaf module `person-variants.ts`, and the
* nickname data is operator-supplied only (`person-nicknames.json` in
- * the vault, resolved by `resolvePersonNicknameMap` in
- * `person-nicknames.ts` - never a shipped word list).
+ * the vault - never a shipped word list).
+ *
+ * A form that reaches more than one person ONLY through derivation is
+ * reported, never guessed: neither person participates through it and
+ * the ambiguity rides the expansion result (`derivedAmbiguities`, under
+ * {@link DERIVED_PERSON_AMBIGUITY_CODE}) instead of silently adding
+ * several people's names to the query. The retrieval-trail wiring for
+ * that code lives with the pipeline's other degradations.
*/
-import { buildEntityIndex } from "../brain/entities/index-builder.ts";
import { normalizeEntityName } from "../brain/entities/canonical.ts";
import {
- PERSON_NICKNAMES_ARTIFACT_REL,
- PersonNicknameConfigError,
- resolvePersonNicknameMap,
-} from "../brain/entities/person-nicknames.ts";
-import {
- derivePersonNameVariants,
- EMPTY_PERSON_NICKNAME_MAP,
isPersonCategory,
- type PersonNicknameMap,
+ personNameComparisonForm,
+ derivePersonNameVariants,
} from "../brain/entities/person-variants.ts";
-import { ENTITY_STATUS_SCOPE, entityStatusInScope } from "../brain/entities/status-scope.ts";
+import { resolvePersonDerivedIndex, type PersonDerivedIndex } from "../brain/entities/registry.ts";
+import type { BrainEntity } from "../brain/entities/types.ts";
// The nickname reader lives on the brain side (`person-nicknames.ts`),
// beside the kernel it feeds: the registry's derived lookup lane and the
// doctor's collision scan consume it from there, and the brain must not
// import from `src/core/search/`. The re-export keeps this module's
// public surface byte-identical to C2.
-export { PERSON_NICKNAMES_ARTIFACT_REL, PersonNicknameConfigError, resolvePersonNicknameMap };
+export {
+ PERSON_NICKNAMES_ARTIFACT_REL,
+ PersonNicknameConfigError,
+ resolvePersonNicknameMap,
+} from "../brain/entities/person-nicknames.ts";
+
+/**
+ * The degradation code naming a person form that reaches more than one
+ * person only through derivation. The expansion result carries the form
+ * and the ids under this code; a form two people DECLARED is the
+ * doctor's alias-merge candidate instead, and a uniquely-derived form
+ * resolves without a word.
+ */
+export const DERIVED_PERSON_AMBIGUITY_CODE = "person-derived-ambiguity";
+
+/** One derived-only ambiguity: the query form and every person it reaches. */
+export interface DerivedPersonAmbiguity {
+ readonly form: string;
+ readonly ids: ReadonlyArray<string>;
+}
export interface QueryEntityExpansion {
/** Original query entities plus every form added by the registry. */
@@ -58,6 +77,14 @@ export interface QueryEntityExpansion {
readonly derived: ReadonlyArray<string>;
/** Ids of the canonical entities whose forms were added, sorted. */
readonly sourceIds: ReadonlyArray<string>;
+ /**
+ * Query forms that reach more than one person only through
+ * derivation. Nothing is expanded for them - the design reports
+ * derived ambiguity rather than guessing one person - and each entry
+ * names the form and every person id it reaches, so the pipeline can
+ * surface the fact under {@link DERIVED_PERSON_AMBIGUITY_CODE}.
+ */
+ readonly derivedAmbiguities: ReadonlyArray<DerivedPersonAmbiguity>;
}
/**
@@ -66,7 +93,9 @@ export interface QueryEntityExpansion {
* (name or alias) appears among the query entities - or, for a
* person-category entity, when any DERIVED person variant of its forms
* does - and the entity's other forms (declared and derived) are then
- * added.
+ * added. Query forms are compared through their comparison forms too
+ * (the dotted-initial and "Last, First" spellings), exactly as the
+ * registry's derived lookup lane compares them.
*/
export function expandQueryEntities(
vault: string,
@@ -77,66 +106,83 @@ export function expandQueryEntities(
added: Object.freeze([] as string[]),
derived: Object.freeze([] as string[]),
sourceIds: Object.freeze([] as string[]),
+ derivedAmbiguities: Object.freeze([] as DerivedPersonAmbiguity[]),
});
if (queryEntities.length === 0) return identity;
- let index;
+ let derivedIndex: PersonDerivedIndex;
try {
- index = buildEntityIndex(vault);
+ derivedIndex = resolvePersonDerivedIndex(vault);
} catch {
return identity; // fail-soft: search never breaks on a bad registry
}
+ const index = derivedIndex.entityIndex;
if (index.entities.length === 0) return identity;
- let nicknames = EMPTY_PERSON_NICKNAME_MAP;
- try {
- nicknames = resolvePersonNicknameMap(vault);
- } catch (err) {
- // Fail-soft for the operator data alone: an unusable nickname file
- // drops the NICKNAME lane, not the expansion the registry provides
- // without it - the declared name/alias forms and the structural
- // variants keep working, exactly as they did before operator
- // nickname data existed. Anything that is not a config fault is not
- // a degradation, it is a bug, and it surfaces by name.
- if (!(err instanceof PersonNicknameConfigError)) throw err;
+ // Every form the query is answered through: each query entity's
+ // normalized form plus its comparison form, so "A. Lovelace" and
+ // "Lovelace, Ada" meet the same claimants "a lovelace" does. The raw
+ // normalized forms stay a separate set: they are what the attribution
+ // skips below test against, because the comparison forms are lookup
+ // aids, not query content - a canonical form reached only through the
+ // query's punctuation still has to join `added` for chunks naming it
+ // to match.
+ const rawQueryForms = new Set<string>();
+ const queryForms = new Set<string>();
+ for (const entity of queryEntities) {
+ const normalized = normalizeEntityName(entity);
+ if (!normalized) continue;
+ rawQueryForms.add(normalized);
+ queryForms.add(normalized);
+ const comparison = personNameComparisonForm(normalized);
+ if (comparison) queryForms.add(comparison);
+ }
+
+ // Participation per form. A form reaching more than one person ONLY
+ // through derivation expands nothing and is reported instead; a form
+ // anyone declared answers through the declared lane, and the derived
+ // claimants beside it keep today's behaviour (the doctor nominates
+ // that declared-vs-derived pair as an alias-merge candidate).
+ const derivedAmbiguities: DerivedPersonAmbiguity[] = [];
+ const participants = new Map<string, BrainEntity>();
+ for (const form of queryForms) {
+ const claimants = derivedIndex.byForm.get(form);
+ if (claimants === undefined) continue;
+ if (claimants.declared.length === 0 && claimants.derived.length > 1) {
+ derivedAmbiguities.push({ form, ids: claimants.derived.map((e) => e.id) });
+ continue;
+ }
+ for (const entity of [...claimants.declared, ...claimants.derived]) {
+ participants.set(entity.id, entity);
+ }
}
- const querySet = new Set(queryEntities.map((e) => normalizeEntityName(e)));
const added: string[] = [];
const addedSet = new Set<string>();
const derived: string[] = [];
const derivedSet = new Set<string>();
const sourceIds: string[] = [];
- for (const entity of index.entities) {
- if (!entityStatusInScope(entity.status, ENTITY_STATUS_SCOPE.canonical)) continue;
+ // Id order, like the registry walk the participants came from, so the
+ // expansion is independent of the query's own form order.
+ const ordered = [...participants.values()].toSorted((a, b) => a.id.localeCompare(b.id));
+ for (const entity of ordered) {
const forms = [entity.name, ...entity.aliases].map((f) => normalizeEntityName(f));
- // Person-category entities derive variants per declared form; the
- // variants join the PARTICIPATION test (a query naming a derived
- // form picks the entity up) and, below, the expanded set.
- const derivedByForm = isPersonCategory(entity.category)
- ? forms.map((f) => derivePersonNameVariants(f, nicknames))
- : null;
- const participates =
- forms.some((f) => querySet.has(f)) ||
- (derivedByForm !== null &&
- derivedByForm.some((variants) => variants.some((v) => querySet.has(v))));
- if (!participates) continue;
let contributed = false;
for (const form of forms) {
- if (querySet.has(form) || addedSet.has(form) || form.length < 2) continue;
+ if (form.length < 2 || rawQueryForms.has(form) || addedSet.has(form)) continue;
addedSet.add(form);
added.push(form);
contributed = true;
}
- if (derivedByForm !== null) {
- for (const variants of derivedByForm) {
- for (const variant of variants) {
+ if (isPersonCategory(entity.category)) {
+ for (const form of forms) {
+ for (const variant of derivePersonNameVariants(form, derivedIndex.nicknames)) {
// A derived form another entity (or this one) already DECLARED
- // is tracked as declared: declared attribution wins. The guard
- // sees declarations collected SO FAR; a declaration collected
- // LATER in the walk is claimed by the post-loop filter below.
- if (querySet.has(variant) || addedSet.has(variant) || derivedSet.has(variant)) continue;
+ // is tracked as declared: declared attribution wins.
+ if (rawQueryForms.has(variant) || addedSet.has(variant) || derivedSet.has(variant)) {
+ continue;
+ }
derivedSet.add(variant);
derived.push(variant);
contributed = true;
@@ -152,11 +198,14 @@ export function expandQueryEntities(
// counted - or boosted - twice.
const derivedOnly = derived.filter((v) => !addedSet.has(v));
- if (added.length === 0 && derivedOnly.length === 0) return identity;
+ if (added.length === 0 && derivedOnly.length === 0 && derivedAmbiguities.length === 0) {
+ return identity;
+ }
return Object.freeze({
expanded: Object.freeze([...queryEntities, ...added, ...derivedOnly]),
added: Object.freeze(added),
derived: Object.freeze(derivedOnly),
sourceIds: Object.freeze(sourceIds.toSorted()),
+ derivedAmbiguities: Object.freeze(derivedAmbiguities),
});
}
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
index a9b9da4e..b1b518b5 100644
--- a/src/core/search/indexer.ts
+++ b/src/core/search/indexer.ts
@@ -49,13 +49,13 @@ import {
import { embeddingPriceRecommendations } from "./price-recommendations.ts";
import type { EmbeddingPriceSource } from "./embeddings/pricing.ts";
import { makeProvider } from "./embeddings/provider.ts";
-import { resolveProviderConsent } from "./provider-consent.ts";
+import { resolveProviderConsent, providerConsentIsDeclined } from "./provider-consent.ts";
import {
embeddingSignature,
isStaleSignature,
MODEL_NATIVE_DIMENSION,
textExtent,
- tokenEstimateFloor,
+ tokenHoldFloor,
type EmbeddingIdentity,
} from "./embeddings/signature.ts";
import { LOCAL_DEFAULT_DIMENSION } from "./embeddings/local-provider.ts";
@@ -120,7 +120,7 @@ import {
excludedCallableVisibilitySurfaces,
} from "./visibility-surface-registry.ts";
import type { ChunkInput, LinkInput } from "./store.ts";
-import type { PendingVectorScope } from "./store/chunks.ts";
+import type { ChunkHoldExclusion, PendingVectorScope } from "./store/chunks.ts";
import type {
ChunkWindowCensus,
EmbedderRecordCensus,
@@ -936,7 +936,7 @@ const CHUNK_SIZE_CONFIG_KEY = "search_chunk_size";
*
* The silences it does NOT produce are the two states the estimate
* cannot back: a model with no declared window (`window-undeclared`),
- * and chunks the estimate cannot place on either side of a declared one
+ * and chunks the census cannot place inside a declared one
* (`estimate-undecided`). A check that could not run and a check that
* passed are not the same statement.
*/
@@ -991,7 +991,7 @@ function censusChunkWindow(
/**
* The operator-facing warning line for a census that MEASURED something
- * - chunks proved over the window, or chunks its estimate cannot place
+ * - chunks proved over the window, or chunks its estimates cannot place
* inside one - ending in the command its REGISTERED diagnostic names.
* The command is resolved from the registry rather than written here, so
* this sentence cannot name an exit the registry does not hold; an
@@ -1018,22 +1018,23 @@ async function formatChunkWindowMeasured(
return (
`${census.chunksUndecided} of ${census.chunksMeasured} indexed chunk(s) cannot be ` +
`placed inside the ${census.windowTokens}-token input window declared for ` +
- `${census.model}: the census estimates tokens from character counts, which does not ` +
- `bound a token window for non-Latin scripts, so this is not a pass. Undecided chunks ` +
- `are still sent, and whatever does not fit is truncated by the provider, silently - ` +
- `${levers}`
+ `${census.model}: the census cannot bound a token window with character counts - the ` +
+ `average-rate estimate reads high on ordinary prose, and non-Latin scripts and dense ` +
+ `symbols cost far more than four characters per token - so this is not a pass. ` +
+ `Undecided chunks are still sent, and whatever does not fit is truncated by the ` +
+ `provider, silently - ${levers}`
);
}
const undecided =
census.chunksUndecided === undefined
? ""
- : ` A further ${census.chunksUndecided} chunk(s) the estimate cannot place either way.`;
+ : ` A further ${census.chunksUndecided} chunk(s) the census cannot place either way.`;
return (
- `${census.chunksOverWindow} of ${census.chunksMeasured} indexed chunk(s) estimate above ` +
- `the ${census.windowTokens}-token input window declared for ${census.model}; chunks ` +
- `proved over the window are held back from embedding rather than sent and truncated, ` +
- `and \`o2b search status\` names the held ones in its held-chunks report. ${levers}` +
- `${undecided}`
+ `${census.chunksOverWindow} of ${census.chunksMeasured} indexed chunk(s) cannot fit the ` +
+ `${census.windowTokens}-token input window declared for ${census.model} even at a ` +
+ `conservative estimate; chunks proved over the window are held back from embedding ` +
+ `rather than sent and truncated, and \`o2b search status\` names the held ones in its ` +
+ `held-chunks report. ${levers}${undecided}`
);
}
@@ -1128,17 +1129,26 @@ type PendingCensusEntry = EmbeddingSpendPlan["pending"][number];
* Split a pending census into what the embedding phase may send and what
* it must HOLD back (chunk skip-and-hold, F2).
*
- * The hold acts on proof only. A chunk is held when the estimator's FLOOR
- * - code points over four, charged through the prefix the configured
- * backend actually prepends - already exceeds the model's effective
- * declared window: then the real request is over it whatever tokenizer the
- * provider runs. The undecided band between the floor and the ceiling
- * stays embedded exactly as before this feature, because acting there
- * would hold work on a guess, and a model with no declared window (or the
- * local embedder, which has no window by construction) proves nothing at
- * all - for them nothing is held and every chunk stays pending.
+ * The hold acts on proof only. A chunk is held when the conservative
+ * bound - code points charged at no more than one token per
+ * `MAX_CODE_POINTS_PER_TOKEN` (embeddings/signature.ts), through the
+ * prefix the configured backend actually prepends - already exceeds the
+ * model's effective declared window: real tokenizers absorb several code
+ * points per token, so a text over even that generous charge cannot fit.
+ * The chars-over-four
+ * average is deliberately NOT the trigger: it is an average, not a bound,
+ * and holding on it read whole English chunks that fit the window as over
+ * it - recall lost on an arithmetic guess.
*
- * The predicate is the same measurement the oversize-chunk census makes
+ * Everything between that bound and the average-rate estimate is the
+ * undecided band: those chunks stay embedded exactly as before this
+ * feature, because acting there would hold work on a guess, and the window
+ * census beside this phase reports the band rather than passing it in
+ * silence. A model with no declared window (or the local embedder, which
+ * has no window by construction) proves nothing at all - for them nothing
+ * is held and every chunk stays pending.
+ *
+ * The hold bound is the same measurement the oversize-chunk census makes
* its `over-window` arm with, so the phase and the report can never
* disagree about which chunks are provably over.
*/
@@ -1156,12 +1166,12 @@ function partitionHeldChunks(
const embeddable: Array<PendingCensusEntry> = [];
const held: HeldChunkRecord[] = [];
for (const chunk of pending) {
- const estimatedTokens = tokenEstimateFloor(textExtent(prefix + chunk.content));
- if (estimatedTokens > windowTokens) {
+ const heldTokens = tokenHoldFloor(textExtent(prefix + chunk.content));
+ if (heldTokens > windowTokens) {
held.push(
Object.freeze({
chunkId: chunk.chunkId,
- estimatedTokens,
+ estimatedTokens: heldTokens,
windowTokens,
reason: CHUNK_HOLD_REASON.overDeclaredWindow,
}),
@@ -2285,9 +2295,15 @@ export async function indexCheck(
warnings.push(await semanticCapabilityLabel(capability.code));
}
- // The live probe (wiring-what-exists, E1). Three decisions, each one
+ // The live probe (wiring-what-exists, E1). Four decisions, each one
// the difference between two states a single verdict used to blur:
//
+ // - A recorded DECLINE is `skipped`, with the consent's own sentence
+ // as the reason - decided, before the tiers are consulted. The two
+ // decline keys refuse independently, and the operator who wrote one
+ // made the same decision either way; reporting that as
+ // `not-configured` merged a decision with an absence and read a
+ // broken configuration as one that was never finished.
// - Nothing configured is `not-configured`, not a failed probe. The
// absence of a provider is not a broken provider, and it costs no
// network call: `probeProvider` is never reached on that arm.
@@ -2303,11 +2319,14 @@ export async function indexCheck(
env: process.env,
semantic: config.semantic,
});
- const probe = !embeddingKeyResolved
- ? PROVIDER_PROBE_NOT_CONFIGURED
- : opts?.probeProvider === false
- ? PROVIDER_PROBE_SKIPPED
- : await probeProvider(() => makeProvider(config.semantic, consent));
+ const consentDeclined = providerConsentIsDeclined(consent);
+ const probe = consentDeclined
+ ? Object.freeze({ state: PROVIDER_PROBE.skipped, reason: consent.detail })
+ : !embeddingKeyResolved
+ ? PROVIDER_PROBE_NOT_CONFIGURED
+ : opts?.probeProvider === false
+ ? PROVIDER_PROBE_SKIPPED
+ : await probeProvider(() => makeProvider(config.semantic, consent));
const providerProbe = probe.state;
const providerReason = probe.reason;
if (providerProbe === PROVIDER_PROBE.unreachable) {
@@ -2334,8 +2353,14 @@ export async function indexCheck(
// The measured pending-vector count (nothing-writes-silently, unit A),
// read through the same explicit peek the ABI tokens use and for the
// same reason. Two `COUNT(*)`s on a read-only handle; the report is a
- // diagnostic an operator ran, never the query path.
- const pendingVectors = readPendingVectorCensus(config.dbPath);
+ // diagnostic an operator ran, never the query path. The config's own
+ // chunk-hold exclusion rides along: a held chunk is not work the index
+ // owes a vector for, and counting it here would recommend a backfill
+ // that finds nothing to do and never clears.
+ const pendingVectors = readPendingVectorCensus(
+ config.dbPath,
+ config.semantic.enabled ? (chunkHoldExclusion(config) ?? undefined) : undefined,
+ );
// The record-vs-data embedder audit (nothing-writes-silently, unit G).
// Ungated by the semantic switch on purpose: it compares the index's
@@ -2400,12 +2425,20 @@ export async function indexCheck(
* UNRECORDED - they differ in their reason, not in what they prove -
* and neither is a count of zero.
*
+ * `hold` is the configuration's chunk-hold exclusion key; passing it
+ * keeps held chunks out of the pending count, the same way the spend
+ * plan and the backfill exclude them. Callers without a configuration
+ * in hand omit it and read the raw anti-join.
+ *
* Exported for the brain doctor's embeddings-health check, which
* consolidates this census beside the index counts instead of deriving
* a private copy whose unrecorded contract could drift from this one.
*/
-export function readPendingVectorCensus(dbPath: string): PendingVectorCensus {
- const peek = peekPendingVectorsSync(dbPath);
+export function readPendingVectorCensus(
+ dbPath: string,
+ hold?: ChunkHoldExclusion,
+): PendingVectorCensus {
+ const peek = peekPendingVectorsSync(dbPath, hold);
if (peek.kind === "read") {
return Object.freeze({
verdict: "measured" as const,
diff --git a/src/core/search/pipeline/assemble.ts b/src/core/search/pipeline/assemble.ts
index 15da0317..791e7e51 100644
--- a/src/core/search/pipeline/assemble.ts
+++ b/src/core/search/pipeline/assemble.ts
@@ -480,10 +480,32 @@ export function assembleRankedResults(input: AssemblyInput): ReadonlyArray<Brain
effectiveCap = wideCap;
}
}
+ noteDerivedPersonAmbiguities(input.degraded, input.signals);
noteAssemblyNarrowings(input.degraded, assembled, effectiveCap, poolCap);
return assembled.visible;
}
+/**
+ * Name the query person forms that reached several people ONLY through
+ * derivation, and therefore expanded nothing - the trail entry is what
+ * keeps "reported rather than guessed" from reading as silence. One entry
+ * per ambiguous form, recorded once per answer beside the assembly
+ * narrowings: the detail maps the form to its competing person ids,
+ * dot-joined. Entity ids never carry a dot, so the join is exact, and the
+ * form rides the detail KEY because a name form is not an identifier and
+ * the detail value lane carries identifiers and integers only.
+ */
+function noteDerivedPersonAmbiguities(
+ sink: RetrievalDegradationSink,
+ signals: CandidateSignals,
+): void {
+ for (const ambiguity of signals.derivedAmbiguities ?? []) {
+ noteDegradation(sink, RETRIEVAL_DEGRADATION.personDerivedAmbiguity, {
+ [ambiguity.form]: ambiguity.ids.join("."),
+ });
+ }
+}
+
/**
* Name what this stage removed, once, against the assembly that actually
* produced the answer. Recorded here rather than inside `assemble` because
diff --git a/src/core/search/pipeline/candidate-signals.ts b/src/core/search/pipeline/candidate-signals.ts
index 665cad5a..daca9cb7 100644
--- a/src/core/search/pipeline/candidate-signals.ts
+++ b/src/core/search/pipeline/candidate-signals.ts
@@ -15,7 +15,7 @@ import { effectiveActivation, halfLifeDays, resolveActivationKind } from "../act
import { CO_ACCESS_MIN_COUNT, readActivationState } from "../activation/store.ts";
import type { ActivationPathState, CoAccessPair } from "../activation/types.ts";
import { extractEntities } from "../entities.ts";
-import { expandQueryEntities } from "../entity-alias.ts";
+import { expandQueryEntities, type DerivedPersonAmbiguity } from "../entity-alias.ts";
import { readCachedFrontmatter, type FrontmatterCache } from "../result-filters.ts";
import type { HydratedChunk, Store } from "../store.ts";
import type { ResolvedSearchConfig } from "../types.ts";
@@ -53,6 +53,14 @@ export interface CandidateSignals {
* score and breakdown shape is byte-identical.
*/
readonly derivedEntityMatchByChunk: ReadonlyMap<number, number> | undefined;
+ /**
+ * The query person forms that reached more than one person ONLY
+ * through derivation (F3): the expansion added nothing for them, and
+ * the trail says so instead of the silence a healthy answer carries -
+ * reported rather than guessed. Undefined when the expansion reported
+ * no ambiguity, per the module-wide omit-when-empty rule.
+ */
+ readonly derivedAmbiguities: ReadonlyArray<DerivedPersonAmbiguity> | undefined;
readonly activationByChunk: ReadonlyMap<number, number> | undefined;
readonly coAccessByChunk: ReadonlyMap<number, ReadonlyMap<number, number>> | undefined;
readonly trendByDoc: ReadonlyMap<number, string> | undefined;
@@ -130,6 +138,10 @@ export function collectCandidateSignals(input: CandidateSignalsInput): Candidate
canonicalMatchByChunk,
canonicalSourceIds: entityExpansion.sourceIds,
derivedEntityMatchByChunk,
+ derivedAmbiguities:
+ entityExpansion.derivedAmbiguities.length > 0
+ ? entityExpansion.derivedAmbiguities
+ : undefined,
activationByChunk: activation.activationByChunk,
coAccessByChunk: activation.coAccessByChunk,
trendByDoc: collectFreshnessTrend(input),
diff --git a/src/core/search/pipeline/post-rank.ts b/src/core/search/pipeline/post-rank.ts
index 97696858..cf6e5c5c 100644
--- a/src/core/search/pipeline/post-rank.ts
+++ b/src/core/search/pipeline/post-rank.ts
@@ -29,6 +29,7 @@ import { collectTrustGateRefusals, trustGateAdjuster } from "../../brain/trust/r
import { applyRelationPolarityPhase } from "../graph-phases.ts";
import { clamp01 } from "../../math.ts";
import { isVisible, pageVisibility } from "../../graph/visibility.ts";
+import { remoteDenyVisibilityTokens } from "../../graph/privacy-markers.ts";
import { SUPERSEDED_BY_RELATION } from "../../graph/relation-vocab.ts";
import { couplingVerdict, type CouplingVerdict } from "../correction-coupling.ts";
import { SUCCESSOR_CARRY } from "../relation-polarity.ts";
@@ -551,10 +552,13 @@ export async function applyPostRankPhases(input: PostRankInput): Promise<PostRan
});
},
// Decision-model kind only: a candidate leaves the machine only when
- // its page's visibility resolves and does not carry `private`.
+ // its page's visibility resolves and does not carry `private`. The
+ // tokens fold in an informal privacy marker's reservation
+ // (graph/privacy-markers.ts), so a marker-only page - which resolves
+ // to no caller-scope token at all - is withheld like a formal one.
resolveVisibility: (path) => {
const entry = readCachedFrontmatterEntry(frontmatterCache, config.vault, path);
- return entry.unreadable ? null : pageVisibility(entry.meta);
+ return entry.unreadable ? null : remoteDenyVisibilityTokens(entry.meta);
},
// The index keeps a page's text whole, and a long `<private>` region
// can be split across chunks, so the page's own regions are read to
diff --git a/src/core/search/provider-consent.ts b/src/core/search/provider-consent.ts
index 6fd21380..5cd85827 100644
--- a/src/core/search/provider-consent.ts
+++ b/src/core/search/provider-consent.ts
@@ -41,7 +41,12 @@
* default `search_semantic_enabled` is `false`, so "unset" and "off" are
* the same configuration; a record that cannot be read must not be allowed
* to look like a record that says nothing, because only one of those two
- * is a decision the operator made.
+ * is a decision the operator made. The fail-closed decline is for SPEND
+ * surfaces - nothing may be sent on a record nobody read - while
+ * {@link ProviderConsent.unreadableReason} lets a diagnostic report the
+ * same arm as unmeasured rather than as the operator's decision, and the
+ * refusal names the file to repair instead of keys to flip, because a
+ * record nobody has read has named no key to clear.
*
* ## The two witnesses of a decline
*
@@ -70,7 +75,7 @@
* place - the frozen object and the derived union - are what ship.
*/
-import { discoverConfig } from "../config.ts";
+import { discoverConfig, defaultConfigPath, ConfigReadError } from "../config.ts";
import { envOrConfig, parseBool } from "../validate.ts";
import { resolveSemanticCapability, type SemanticCapability } from "./capability-tier.ts";
import type { ResolvedEmbeddingConfig } from "./types.ts";
@@ -165,20 +170,55 @@ export interface ProviderConsent {
readonly declineKeys: ReadonlyArray<string>;
/** The keys that would re-enable a declined provider. */
readonly reenableKeys: ReadonlyArray<string>;
+ /**
+ * Why the record could not be read, on the arm where it could not.
+ * The resolver still fails closed to {@link PROVIDER_CONSENT_STATE.declined}
+ * - an unreadable record is not consent - but a diagnostic that renders
+ * that arm as a DECISION would pass a broken machine configuration as
+ * one the operator chose. Null on every arm whose record was read.
+ */
+ readonly unreadableReason: string | null;
/** The operator-facing sentence. Never empty. */
readonly detail: string;
}
/**
- * The sentence a refusal carries. Named in the message as a KEY, so an
+ * The value that clears each decline key, keyed the way
+ * {@link PROVIDER_CONSENT_KEY} is. The two keys decline INDEPENDENTLY -
+ * clearing one leaves the other's decline in force - which is why the
+ * re-enable sentence is built from the keys this record actually used
+ * rather than naming both spellings and letting an operator clear only
+ * one of them.
+ */
+const DECLINE_CLEARING_VALUE: Readonly<Record<string, string>> = Object.freeze({
+ [PROVIDER_CONSENT_KEY.semantic]: "true",
+ [PROVIDER_CONSENT_KEY.provider]: "<provider>",
+});
+
+/**
+ * The sentence a refusal carries. Built from `declineKeys`, naming every
+ * key that recorded THIS decline and the value that clears it, so the
+ * advice is one the operator can follow to the end: following it must
+ * never leave them refused by the other key. Named as a KEY, so an
* operator reading it can search the configuration file for the spelling
* rather than for a phrase.
*/
-function reenableSentence(): string {
- return (
- `re-enable it with ${PROVIDER_CONSENT_KEY.semantic}=true ` +
- `or ${PROVIDER_CONSENT_KEY.provider}=<provider>`
- );
+function reenableSentence(declineKeys: ReadonlyArray<string>): string {
+ const clears = declineKeys.flatMap((key) => {
+ const value = DECLINE_CLEARING_VALUE[key];
+ return value === undefined ? [] : [`${key}=${value}`];
+ });
+ return `re-enable it with ${clears.join(" or ")}`;
+}
+
+/**
+ * The advice an UNREADABLE record carries instead of a re-enable
+ * sentence: an unreadable record named no key, so there is no key to
+ * clear. The repair is the record itself, and the sentence names the
+ * file it lives in.
+ */
+function repairRecordSentence(configPath: string): string {
+ return `repair the configuration file at ${configPath}`;
}
/**
@@ -194,7 +234,7 @@ function declineSpellingsSentence(): string {
);
}
-/** A declined consent, with the reason it was read that way. */
+/** A declined consent whose record was read, with the keys that did it. */
function declined(
capability: SemanticCapability | null,
declineKeys: ReadonlyArray<string>,
@@ -207,10 +247,38 @@ function declined(
capability,
declineKeys: Object.freeze([...declineKeys]),
reenableKeys: PROVIDER_CONSENT_REENABLE_KEYS,
+ unreadableReason: null,
detail:
`provider consent declined: no embedding provider will be contacted. ` +
`${reason}${declineSpellingsSentence()}${recorded}. ` +
- `${reenableSentence()}.`,
+ `${reenableSentence(declineKeys)}.`,
+ });
+}
+
+/**
+ * A declined consent whose record could not be READ. Same fail-closed
+ * state as {@link declined}, but no key is named - the record named none
+ * - and the advice is a repair of the file, not a key to flip: a
+ * re-enable sentence would be advice about a record nobody has read.
+ * `unreadableReason` is what lets a diagnostic report this arm as
+ * unmeasured rather than as an operator's decision.
+ */
+function declinedUnreadable(
+ capability: SemanticCapability | null,
+ unreadableReason: string,
+ configPath: string,
+): ProviderConsent {
+ return Object.freeze({
+ state: PROVIDER_CONSENT_STATE.declined,
+ capability,
+ declineKeys: Object.freeze([]),
+ reenableKeys: PROVIDER_CONSENT_REENABLE_KEYS,
+ unreadableReason,
+ detail:
+ `provider consent declined: no embedding provider will be contacted. ` +
+ `The provider-consent record could not be read (${unreadableReason}), and an ` +
+ `unreadable record is not consent. ${declineSpellingsSentence()}. ` +
+ `${repairRecordSentence(configPath)}.`,
});
}
@@ -230,16 +298,20 @@ export function resolveProviderConsent(request: ProviderConsentRequest): Provide
// The record. An unreadable one is a decline, never an absence: the
// operator's settings exist and are not the ones in force, which is
- // exactly what the configuration reader refuses to paper over.
+ // exactly what the configuration reader refuses to paper over. The
+ // reason rides on the verdict, so a diagnostic can report this arm as
+ // unmeasured rather than as a decision the operator made.
let config: Readonly<Record<string, string>>;
try {
config = request.configPath === null ? {} : discoverConfig(request.configPath).data;
- } catch {
- return declined(
- capability,
- [],
- "The provider-consent record could not be read, and an unreadable record is not consent. ",
- );
+ } catch (e) {
+ const reason =
+ e instanceof ConfigReadError
+ ? "the config file exists but could not be read"
+ : e instanceof Error
+ ? e.message
+ : String(e);
+ return declinedUnreadable(capability, reason, request.configPath ?? defaultConfigPath());
}
const declineKeys: string[] = [];
@@ -291,6 +363,7 @@ export function resolveProviderConsent(request: ProviderConsentRequest): Provide
capability,
declineKeys: Object.freeze([]),
reenableKeys: PROVIDER_CONSENT_REENABLE_KEYS,
+ unreadableReason: null,
detail: "no provider-consent decision is recorded: the embedding provider was never declined.",
});
}
diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.ts
index 602c73e7..365a4d0a 100644
--- a/src/core/search/ranker.ts
+++ b/src/core/search/ranker.ts
@@ -907,15 +907,21 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea
const aRelationalOnly = relationalOnlyIds.has(a.chunkId);
const bRelationalOnly = relationalOnlyIds.has(b.chunkId);
if (aRelationalOnly !== bRelationalOnly) return aRelationalOnly ? 1 : -1;
- // Structural declared-before-derived entity precedence: candidates
- // are partitioned into TIERS - declared, then neutral, then
- // derived-only - and the tiers compare before scores, so a declared
- // match always outranks a derived-only one regardless of entityMul.
- // A tier, not a pairwise rule: declared-beats-derived-only answered
- // pairwise while neutral candidates fell through to the score, which
- // is not a strict weak ordering (declared A < derived B, derived B <
- // neutral C, neutral C < declared A) and made the sort output depend
- // on the input order.
+ if (b.score !== a.score) return b.score - a.score;
+ // Structural declared-before-derived entity precedence, as a TIE-BREAK
+ // on exact score ties only: candidates are partitioned into TIERS -
+ // declared, then neutral, then derived-only - and the tiers compare
+ // only after the composite scores have had their say, so the capped
+ // boost lanes stay the ranking mechanism and a single entity match can
+ // never float a low-scoring chunk above a higher-scoring one (nor sink
+ // a derived-only direct hit below the cut). The lane sizes (declared
+ // 0.02 per match capped at 0.04, derived 0.01 capped at 0.02) already
+ // rank declared above derived among otherwise-equal candidates; the
+ // tier orders what the lanes could not separate - ties left by a zeroed
+ // boost or a saturated cap. A tier, not a pairwise rule: within one
+ // score value the partition is a strict weak ordering, and across
+ // values the score decides, so the sort never depends on the input
+ // order.
const aDeclared = (inputs.entityMatchByChunk?.get(a.chunkId) ?? 0) > 0;
const bDeclared = (inputs.entityMatchByChunk?.get(b.chunkId) ?? 0) > 0;
const aDerivedOnly =
@@ -926,7 +932,6 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea
declared ? 0 : derivedOnly ? 2 : 1;
const tierDelta = tierOf(aDeclared, aDerivedOnly) - tierOf(bDeclared, bDerivedOnly);
if (tierDelta !== 0) return tierDelta;
- if (b.score !== a.score) return b.score - a.score;
if (temporalIntent !== null) {
const aTemporal = a.breakdown?.temporal ?? 0;
const bTemporal = b.breakdown?.temporal ?? 0;
diff --git a/src/core/search/retrieval-trail.ts b/src/core/search/retrieval-trail.ts
index 85857382..aa7f357c 100644
--- a/src/core/search/retrieval-trail.ts
+++ b/src/core/search/retrieval-trail.ts
@@ -152,6 +152,19 @@ export const RETRIEVAL_DEGRADATION = Object.freeze({
* fired.
*/
hybridDeadlineExceeded: "hybrid-deadline-exceeded",
+ /**
+ * A person form in the query reached more than one person ONLY through
+ * derivation - an initial, a word order or an operator nickname - so the
+ * registry expansion added nothing for it: the design reports derived
+ * ambiguity rather than guessing one person (write-honesty-egress-consent,
+ * F3). Produced by the pipeline's candidate-signal collection, emitted in
+ * assembly. One entry per ambiguous form; `detail` maps that form to the
+ * competing person ids, dot-joined - entity ids never carry a dot, so the
+ * join is exact. The form is the detail KEY, not a value, because a name
+ * form is not an identifier and the value lane carries identifiers and
+ * integers only.
+ */
+ personDerivedAmbiguity: "person-derived-ambiguity",
/**
* The rank cap truncated the candidate pool: matches existed below the
* cut and were never ranked. `detail.cap` is the cap that bit.
@@ -233,6 +246,7 @@ export const RETRIEVAL_DEGRADATION_CODES: ReadonlyArray<RetrievalDegradationCode
RETRIEVAL_DEGRADATION.semanticEmbeddingAbiDrift,
RETRIEVAL_DEGRADATION.hybridDegraded,
RETRIEVAL_DEGRADATION.hybridDeadlineExceeded,
+ RETRIEVAL_DEGRADATION.personDerivedAmbiguity,
RETRIEVAL_DEGRADATION.rankCapTruncatedPool,
RETRIEVAL_DEGRADATION.rerankProviderUnavailable,
RETRIEVAL_DEGRADATION.rerankModelSunset,
@@ -445,6 +459,8 @@ export function describeRetrievalDegradation(code: RetrievalDegradationCode): st
return "the semantic lane did not run, so this answer is keyword-only";
case RETRIEVAL_DEGRADATION.hybridDeadlineExceeded:
return "the composite search outlived its wall-clock deadline, so the phases past the budget were cut and this answer is keyword-only";
+ case RETRIEVAL_DEGRADATION.personDerivedAmbiguity:
+ return "a person name in the query matched several people only through a derived form - an initial, a word order or a nickname - so nothing was expanded for it rather than guess one person";
case RETRIEVAL_DEGRADATION.rankCapTruncatedPool:
return "the rank cap truncated the candidate pool, so lower-ranked matches were never considered";
case RETRIEVAL_DEGRADATION.rerankProviderUnavailable:
diff --git a/src/core/search/schema.ts b/src/core/search/schema.ts
index 01acedee..e5a9539e 100644
--- a/src/core/search/schema.ts
+++ b/src/core/search/schema.ts
@@ -10,7 +10,7 @@
import { Database } from "bun:sqlite";
import { SearchError } from "./types.ts";
-import { remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
+import { hasPrivacyMarkerKey, remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";
import { parseFrontmatterText } from "../vault.ts";
import { closeDatabase } from "../sqlite-close.ts";
@@ -19,7 +19,7 @@ import { closeDatabase } from "../sqlite-close.ts";
* this raises `SCHEMA_MISMATCH` on open โ€” the operator must reindex
* with a newer binary.
*/
-export const LATEST_SCHEMA_VERSION = 15;
+export const LATEST_SCHEMA_VERSION = 16;
/**
* The one command that rebuilds an index this binary cannot read. Every
@@ -746,6 +746,47 @@ export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([
`);
},
},
+ {
+ // v16 (privacy-marker visibility restamp) - re-measure the
+ // {@link DOCUMENT_VISIBILITY_COLUMN} of every row whose chunk-zero
+ // frontmatter carries an informal privacy marker key (`access`,
+ // `confidentiality`, `sensitivity`).
+ //
+ // WHY A RE-STAMP, NOT A BACKFILL: the column has existed since v12,
+ // and the v12 backfill ran once, for `IS NULL` rows, under whatever
+ // measurement its binary knew. An index migrated to v12-v14 by a
+ // pre-marker binary holds rows stamped from the declared `visibility:`
+ // alone, and those rows are NOT NULL anymore - the v12 gate never
+ // opens for them again. The marker-aware measurement
+ // ({@link remoteDenyVisibilityTokens}) arrived after, and the
+ // incremental indexer skips an unchanged file, so a marker page's
+ // column can keep saying `[]` - measured, and declaring nothing -
+ // while the live read roots deny the page. Every consumer that reads
+ // the column alone (the belief-semantic lane, the visibility-honesty
+ // census) then undercounts the very pages this column exists to make
+ // countable. A new migration is the only frame that runs on those
+ // indexes: editing v15 in place would rewrite released migration
+ // history and never execute on an index already at 15.
+ //
+ // The gate is {@link hasPrivacyMarkerKey}, not the deny predicate:
+ // any marker key at all marks a row whose pre-marker stamp may
+ // understate it, and the update writes the full marker-aware token
+ // list, so a row already carrying the reserved token re-derives the
+ // same value and a non-marker row is not touched. Re-running the v12
+ // recipe over EVERY row would be equivalent but would rewrite rows
+ // this migration has no reason to touch; a NULL row (no chunk zero)
+ // stays NULL - there is still nothing to measure. No DDL: the
+ // manifest this migration produces is v15's, unchanged.
+ //
+ // It runs INSIDE `applyMigrations`' single open transaction, like the
+ // v12 backfill: a half-restamped column would leave sibling rows
+ // disagreeing about the same kind of claim, and a crash mid-pass
+ // would leave exactly that.
+ version: 16,
+ up(db) {
+ restampMarkerVisibility(db);
+ },
+ },
]);
/**
@@ -991,6 +1032,42 @@ function backfillDocumentVisibility(db: Database): void {
}
}
+/**
+ * Re-measure {@link DOCUMENT_VISIBILITY_COLUMN} for every row whose
+ * chunk-zero frontmatter carries an informal privacy marker key, from that
+ * same chunk, through {@link remoteDenyVisibilityTokens} - the v16
+ * migration's whole body.
+ *
+ * The rows it targets are exactly the ones a pre-marker stamp can
+ * understate: a marker page's column reads `[]` (measured, declares
+ * nothing) while the live read roots deny the page, and the incremental
+ * indexer never revisits an unchanged file to correct it. A row already
+ * carrying the marker-aware value re-derives that value and is written
+ * with what it already was, so the pass is idempotent; a row whose column
+ * disagrees with its frontmatter for any other reason is left to a
+ * re-index, which remains the authoritative rewrite.
+ */
+function restampMarkerVisibility(db: Database): void {
+ const rows = db
+ .query<{ id: number; content: string | null; current: string | null }, [number]>(
+ `SELECT d.id AS id, c.content AS content, d.${DOCUMENT_VISIBILITY_COLUMN} AS current ` +
+ `FROM documents d ` +
+ `LEFT JOIN chunks c ON c.document_id = d.id AND c.chunk_index = ?1`,
+ )
+ .all(FRONTMATTER_CHUNK_INDEX);
+ if (rows.length === 0) return;
+ const update = db.query<never, [string, number]>(
+ `UPDATE documents SET ${DOCUMENT_VISIBILITY_COLUMN} = ?1 WHERE id = ?2`,
+ );
+ for (const row of rows) {
+ if (row.content === null) continue;
+ const [meta] = parseFrontmatterText(row.content);
+ if (!hasPrivacyMarkerKey(meta)) continue;
+ const next = encodeDocumentVisibility(remoteDenyVisibilityTokens(meta));
+ if (row.current !== next) update.run(next, row.id);
+ }
+}
+
/**
* Fill {@link DOCUMENT_ATTRIBUTES_TABLE} for every document that has no
* attribute rows yet, from the frontmatter chunk the index already holds -
diff --git a/src/core/search/store/counts.ts b/src/core/search/store/counts.ts
index 17c69f1d..4ec276ec 100644
--- a/src/core/search/store/counts.ts
+++ b/src/core/search/store/counts.ts
@@ -9,6 +9,7 @@ import { Database } from "bun:sqlite";
import { SearchError } from "../search-error.ts";
import { EVENT_TIME_MAX_COLUMN, EVENT_TIME_MIN_COLUMN } from "../schema.ts";
import { countChunks, countChunksWithoutEmbeddings } from "./chunks.ts";
+import type { ChunkHoldExclusion } from "./chunks.ts";
import { countDocuments } from "./documents.ts";
import {
EMBEDDING_DIMENSION_STATE_KEY,
@@ -19,6 +20,7 @@ import {
import { countEmbeddings, staleEmbeddings } from "./vectors.ts";
import {
charLengthOverTokenBudget,
+ MAX_CODE_POINTS_PER_TOKEN,
NON_ASCII_CEILING_COEFFICIENT,
textExtent,
utf8ByteFloorUnderTokenBudget,
@@ -184,16 +186,24 @@ export function eventTimeWindowCensus(
/** Chunks that carry no vector, against the chunks there are. */
export interface PendingVectorTally {
- /** Chunks with no row in `embeddings`. */
+ /** Chunks with no row in `embeddings`, held chunks excluded when asked. */
readonly pending: number;
/** Chunks in the index at all, so the pending count has a denominator. */
readonly chunks: number;
}
-/** Both counts from one connection, so they describe one instant. */
-export function pendingVectorTally(db: Database): PendingVectorTally {
+/**
+ * Both counts from one connection, so they describe one instant.
+ *
+ * `hold` is the config's chunk-hold exclusion key: a chunk held out of
+ * embedding under that key is not work the index owes a vector for, so a
+ * tally that counted it as pending would recommend a backfill that finds
+ * nothing to do and never clears. Callers that have no configuration in
+ * hand (the key-free diagnostic reads) omit it and see the raw count.
+ */
+export function pendingVectorTally(db: Database, hold?: ChunkHoldExclusion): PendingVectorTally {
return Object.freeze({
- pending: countChunksWithoutEmbeddings(db),
+ pending: countChunksWithoutEmbeddings(db, undefined, hold),
chunks: countChunks(db),
});
}
@@ -202,22 +212,26 @@ export function pendingVectorTally(db: Database): PendingVectorTally {
* {@link pendingVectorTally} taken WITHOUT opening a `Store`, for
* `indexCheck` - which deliberately probes an in-memory database and
* never touches the real index, so anything it reports about stored
- * state has to be read explicitly.
+ * state has to be read explicitly. `hold` threads the config's chunk-hold
+ * exclusion through, exactly as {@link pendingVectorTally} takes it.
*
* The peek keeps its three outcomes rather than collapsing to a number:
* "no index here" and "an index that will not open" are not a pending
* count of zero, and zero is exactly what a caller would read as a
* fully-embedded vault.
*/
-export function peekPendingVectorsSync(dbPath: string): IndexPeek<PendingVectorTally> {
- return peekReadonlyIndex(dbPath, (_read, db) => pendingVectorTally(db));
+export function peekPendingVectorsSync(
+ dbPath: string,
+ hold?: ChunkHoldExclusion,
+): IndexPeek<PendingVectorTally> {
+ return peekReadonlyIndex(dbPath, (_read, db) => pendingVectorTally(db, hold));
}
/** Split of the chunk table by what the census can decide about it. */
export interface ChunkWindowTally {
- /** Chunks whose LOW estimate already exceeds the window. */
+ /** Chunks over the window even at the conservative hold bound. */
readonly overWindow: number;
- /** Chunks the low estimate clears and the high estimate does not. */
+ /** Chunks the hold bound does not prove over, whose estimates still read over. */
readonly undecided: number;
}
@@ -243,34 +257,38 @@ export const NO_REQUEST_PREFIX = "";
* different unit from a model's token window, and comparing the two as
* though they matched is the arithmetic this census exists to stop.
*
- * ## Why two aggregates and not one
+ * ## Why two aggregates, and where the boundary sits
*
- * Characters-over-four is a Latin-script heuristic. Two populations
- * break it in the same direction - the estimate reads LOW, so a single
- * count reports a clean census over chunks the provider truncates:
+ * `overWindow` is the HOLD population, so it uses the hold's own bound:
+ * code points charged at no more than one token per
+ * {@link MAX_CODE_POINTS_PER_TOKEN}. Only that conservative bound proves a
+ * chunk over the window - the chars-over-four average is not a bound, and
+ * the phase that embeds must never hold on it.
*
+ * The undecided band is everything the proof cannot settle: chunks whose
+ * conservative bound still fits the window but whose HIGH estimate - the
+ * average rate for Latin text, a near-character-level charge for Han and
+ * supplementary-plane text - reads over it. Three populations land there:
+ *
+ * - Latin chunks between the average estimate and the sound bound. The
+ * average reads HIGH on real prose (real tokenizers absorb more than
+ * four code points per token), so acting on it would hold work on a
+ * guess - exactly the defect that moved the hold to the conservative
+ * bound;
* - Han text, where a BERT-family tokenizer is roughly
* character-level. `BAAI/bge-small-zh-v1.5` declares a 512-token
* window, and a Chinese chunk that estimates at 512 costs about four
* times that;
* - supplementary-plane text, where SQLite's `length()` counts one per
* character and `String.length` counts two, so the database reads
- * half the estimated size and a chunk estimated at twice the window
- * lands under a threshold expressed in code points.
+ * half the estimated size.
*
- * Neither is fixable by a tighter constant, and neither is observable in
- * SQL as a script test - bun:sqlite exposes no user-defined-function
- * hook, and materialising every chunk body costs more than the answer is
- * worth. What IS available in SQL is `octet_length()`, and UTF-8 makes
- * the byte count a structural script signal: every non-ASCII code point
- * costs at least one extra byte, so `octet_length - length` bounds the
- * non-ASCII code points exactly. Charging those at
+ * None of the three is fixable by a tighter constant on its own, and all
+ * are observable in SQL through the same two measurements: `length()`
+ * (code points) and `octet_length()` (UTF-8 bytes), whose difference
+ * bounds the non-ASCII code points exactly. Charging those at
* `MAX_TOKENS_PER_NON_ASCII_CODE_POINT` gives the high estimate, and the
- * gap between the two is reported as undecided rather than passed.
- *
- * For pure ASCII the two estimates coincide - `octet_length` equals
- * `length`, the band is empty - so an all-Latin index is counted exactly
- * as it was before the high estimate existed.
+ * gap between the two arms is reported as undecided rather than passed.
*
* ## Cost
*
@@ -294,6 +312,7 @@ export function chunkWindowTally(
{ over_window: number; undecided: number },
{
$budget: number;
+ $holdBudget: number;
$byteFloor: number;
$coefficient: number;
$prefixCodePoints: number;
@@ -310,15 +329,16 @@ export function chunkWindowTally(
WHERE octet_length(content) + $prefixUtf8Bytes > $byteFloor
)
SELECT
- COUNT(*) FILTER (WHERE code_points > $budget) AS over_window,
+ COUNT(*) FILTER (WHERE code_points > $holdBudget) AS over_window,
COUNT(*) FILTER (
- WHERE code_points <= $budget
+ WHERE code_points <= $holdBudget
AND code_points + $coefficient * min(code_points, utf8_bytes - code_points) > $budget
) AS undecided
FROM extents`,
)
.get({
$budget: charLengthOverTokenBudget(windowTokens),
+ $holdBudget: windowTokens * MAX_CODE_POINTS_PER_TOKEN,
$byteFloor: utf8ByteFloorUnderTokenBudget(windowTokens),
$coefficient: NON_ASCII_CEILING_COEFFICIENT,
$prefixCodePoints: prefix.codePoints,
diff --git a/tests/cli/brain-help-write-honesty.test.ts b/tests/cli/brain-help-write-honesty.test.ts
index 260342e7..c637281e 100644
--- a/tests/cli/brain-help-write-honesty.test.ts
+++ b/tests/cli/brain-help-write-honesty.test.ts
@@ -6,10 +6,20 @@
* usage error. Both defects pinned here were discoverability dead ends:
* the claims surface said nothing about the `undated-quantity` finding it
* is the exit for, and `privacy-marker-restamp` had no entry at all.
+ *
+ * The restamp describe also pins its forward pointer: the local-search
+ * consequence of a stamp travels as prose beside the advisory rail's
+ * structural line (`restamp-local-visibility`), never as a hand-written
+ * invocation literal - the forward-pointer rail's own test is the ratchet
+ * for that, this is the migrated verb's behaviour pin.
*/
-import { describe, expect, test } from "bun:test";
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
import { runCli } from "../helpers/run-cli.ts";
describe("o2b brain claims --help", () => {
@@ -48,3 +58,69 @@ describe("o2b brain privacy-marker-restamp --help", () => {
expect(res.stdout).toContain("privacy-marker-restamp");
});
});
+
+describe("o2b brain privacy-marker-restamp emits through the advisory rail", () => {
+ let tmp: string;
+ let vault: string;
+ let configPath: string;
+ const savedEnv: Record<string, string | undefined> = {};
+
+ beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-restamp-rail-"));
+ vault = join(tmp, "vault");
+ configPath = join(tmp, "config.yaml");
+ for (const key of ["VAULT_DIR", "OPEN_SECOND_BRAIN_CONFIG", "VAULT_AGENT_NAME"]) {
+ savedEnv[key] = process.env[key];
+ delete process.env[key];
+ }
+ writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ mkdirSync(vault, { recursive: true });
+ writeFileSync(join(vault, "marked.md"), "---\naccess: private\n---\n\nMarked.\n");
+ });
+
+ afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ for (const [key, value] of Object.entries(savedEnv)) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ });
+
+ test("the dry run states the consequence and defers the command to the rail", async () => {
+ const res = await runCli(["brain", "privacy-marker-restamp", "--vault", vault]);
+ expect(res.returncode).toBe(0);
+ // The consequence is prose; the command is the rail's structural line.
+ expect(res.stdout).toContain(
+ "note: after --apply each stamped page is also excluded from default-scope local search",
+ );
+ expect(res.stdout).toContain("next: o2b search --visibility=private\n");
+ // No hand-written invocation beside the rail.
+ expect(res.stdout).not.toContain("reach it again with");
+ });
+
+ test("--apply names the same exit through the rail", async () => {
+ const res = await runCli(["brain", "privacy-marker-restamp", "--vault", vault, "--apply"]);
+ expect(res.returncode).toBe(0);
+ expect(res.stdout).toContain("updated 1 of 1 candidate(s)");
+ expect(res.stdout).toContain(
+ "note: each stamped page is now also excluded from default-scope local search",
+ );
+ expect(res.stdout).toContain("next: o2b search --visibility=private\n");
+ });
+
+ test("the rail line stays off the machine stream", async () => {
+ const res = await runCli(["brain", "privacy-marker-restamp", "--vault", vault, "--json"]);
+ expect(res.returncode).toBe(0);
+ expect(res.stdout).not.toContain("next: ");
+ expect(() => JSON.parse(res.stdout)).not.toThrow();
+ });
+
+ test("a vault with no marker pages names no exit", async () => {
+ rmSync(join(vault, "marked.md"));
+ const res = await runCli(["brain", "privacy-marker-restamp", "--vault", vault]);
+ expect(res.returncode).toBe(0);
+ expect(res.stdout).not.toContain("next: ");
+ expect(res.stdout).not.toContain("excluded from default-scope local search");
+ });
+});
diff --git a/tests/cli/brain-reject-log-shard.test.ts b/tests/cli/brain-reject-log-shard.test.ts
new file mode 100644
index 00000000..09699625
--- /dev/null
+++ b/tests/cli/brain-reject-log-shard.test.ts
@@ -0,0 +1,115 @@
+/**
+ * `o2b brain reject` writes its retirement provenance into the log shard
+ * this device actually appends to (issue #271, CLI side).
+ *
+ * The merge and dream halves live in
+ * tests/core/brain/retirement-log-shard.test.ts; this file pins the same
+ * contract for the CLI verb: with a configured device id the retired_by
+ * link names the sharded day file the reject append creates, and with the
+ * documented empty-device-id opt-out the link keeps the exact historical
+ * un-sharded shape. The link is resolved from the written page itself, so
+ * the assertion never depends on the wall-clock date the verb stamped.
+ *
+ * Every hook lives INSIDE the describe block, on purpose: bun's hook
+ * registry spans the files one worker loads, and a top-level beforeEach
+ * here would mint a vault around other files' tests too - the temp guard
+ * then names this prefix from a file that finished long ago. The
+ * OPEN_SECOND_BRAIN_CONFIG pointer reaches the CLI through runCli's own
+ * isolated-env channel instead of a process.env write, so nothing this
+ * file sets outlives a test.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, rmSync } from "node:fs";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { brainDirs, retiredPath } from "../../src/core/brain/paths.ts";
+import { parseRetired, writePreference } from "../../src/core/brain/preference.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { tempDirs } from "../helpers/temp-dir.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+const DEVICE_ID = "fixturedev";
+
+describe("brain reject retirement log shard", () => {
+ const mkTemp = tempDirs();
+ const savedDeviceEnv = process.env["O2B_DEVICE_ID"];
+
+ let tmp: string;
+ let vault: string;
+ let config: string;
+
+ beforeEach(() => {
+ tmp = mkTemp("o2b-reject-shard-");
+ vault = join(tmp, "vault");
+ config = join(tmp, "config.yaml");
+ atomicWriteFileSync(config, `vault: ${vault}\n`);
+ bootstrapBrain(vault, { configPath: config });
+ writePreference(vault, {
+ slug: "old",
+ topic: "formatting",
+ principle: "always run the formatter before every commit in this repository",
+ created_at: "2026-06-01T00:00:00Z",
+ confirmed_at: "2026-06-02T00:00:00Z",
+ unconfirmed_until: "2026-06-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: ["[[sig-1]]"],
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-06-02T00:00:00Z",
+ confidence: "low",
+ });
+ });
+
+ afterEach(() => {
+ // Clean here, not only in the tempDirs() safety net: the guard's
+ // afterAll must never see a vault this file minted, whatever order
+ // the file-level nets run in.
+ rmSync(tmp, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
+ if (savedDeviceEnv === undefined) delete process.env["O2B_DEVICE_ID"];
+ else process.env["O2B_DEVICE_ID"] = savedDeviceEnv;
+ });
+
+ /**
+ * Write the per-test `device_id` line (sharded, or the empty env opt-out
+ * that keeps the legacy un-sharded pair). The config path reaches the
+ * CLI through runCli's isolated env, never a process.env write.
+ */
+ function registerConfig(deviceId: string | null): void {
+ const deviceLine = deviceId === null ? "" : `device_id: "${deviceId}"\n`;
+ atomicWriteFileSync(config, `vault: ${vault}\n${deviceLine}`);
+ if (deviceId === null) process.env["O2B_DEVICE_ID"] = "";
+ else delete process.env["O2B_DEVICE_ID"];
+ }
+
+ /** The retired page's link must have the expected shape and resolve on disk. */
+ function assertRetiredByResolves(shardSuffix: string): void {
+ const ret = parseRetired(retiredPath(vault, "old"));
+ const pattern = `^\\[\\[Brain/log/\\d{4}-\\d{2}-\\d{2}${shardSuffix}\\]\\]$`;
+ expect(ret.retired_by).toMatch(new RegExp(pattern));
+ const target = String(ret.retired_by).slice("[[Brain/log/".length, -2);
+ expect(existsSync(join(brainDirs(vault).log, `${target}.md`))).toBe(true);
+ }
+
+ function reject(): Promise<{ stdout: string; stderr: string; returncode: number }> {
+ return runCli(
+ ["brain", "reject", "--vault", vault, "--id", "pref-old", "--reason", "obsolete"],
+ { env: { OPEN_SECOND_BRAIN_CONFIG: config } },
+ );
+ }
+
+ test("with a configured device id, retired_by names the shard the append creates", async () => {
+ registerConfig(DEVICE_ID);
+ const r = await reject();
+ expect(r.returncode).toBe(0);
+ assertRetiredByResolves(`\\.${DEVICE_ID}`);
+ });
+
+ test("with the empty-device-id opt-out, retired_by keeps the historical shape", async () => {
+ registerConfig(null);
+ const r = await reject();
+ expect(r.returncode).toBe(0);
+ assertRetiredByResolves("");
+ });
+});
diff --git a/tests/cli/decision-model-eval.test.ts b/tests/cli/decision-model-eval.test.ts
index 1016a3cc..b35268ff 100644
--- a/tests/cli/decision-model-eval.test.ts
+++ b/tests/cli/decision-model-eval.test.ts
@@ -67,15 +67,12 @@ function childEnv(extra: Record<string, string> = {}): Record<string, string> {
/** The active eval config; the loopback base URL is added at call time. */
const ENABLED = {
decision_model_enabled: "true",
- decision_model_provider: "compatible",
- decision_model_id: "fake-model-1",
- decision_model_env_key: KEY_VAR,
- decision_model_threshold_profile: "laya",
+ decision_model_provider: "laya",
decision_model_uses: "rerank:shadow",
};
function writeEnabledConfig(): void {
- writeConfig({ ...ENABLED, decision_model_base_url: server.url });
+ writeConfig({ ...ENABLED, decision_model_base_url: server.url, decision_model_id: "english" });
}
/**
@@ -135,7 +132,16 @@ describe("o2b decision-model eval", () => {
expect(specless.returncode).toBe(2);
expect(specless.stderr).toContain("no preregistered spec");
- writeEnabledConfig();
+ // An inactive config is a refusal: a keyed preset without its key is
+ // no_key (the keyless laya loopback above cannot produce one).
+ writeConfig({
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: server.url,
+ decision_model_id: "fake-model-1",
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "rerank:shadow",
+ });
const noKey = await runCli(
["decision-model", "eval", "run", "--use", "rerank", "--config", configPath],
{ env: { OPEN_SECOND_BRAIN_CONFIG: configPath, XDG_STATE_HOME: stateDir } },
diff --git a/tests/cli/decision-model-thresholds.test.ts b/tests/cli/decision-model-thresholds.test.ts
new file mode 100644
index 00000000..e35fd9e7
--- /dev/null
+++ b/tests/cli/decision-model-thresholds.test.ts
@@ -0,0 +1,529 @@
+/**
+ * `o2b decision-model thresholds fit|list|set` (issue #264) and the
+ * shadow messaging: check and report name the model that has no stored
+ * thresholds and the command that tunes it. The store lives beside the
+ * test's own config.yaml; the fit reads records the test seeds into the
+ * vault's continuity log. No network, no credentials.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ buildDecisionModelCheck,
+ buildDecisionModelReport,
+ renderDecisionModelCheck,
+ renderDecisionModelReport,
+} from "../../src/core/decision-model/diagnostics.ts";
+import {
+ decisionThresholdsPath,
+ parseThresholdsStore,
+ resetThresholdsStoreCache,
+} from "../../src/core/decision-model/thresholds.ts";
+import { emitDecisionModelCall } from "../../src/core/decision-model/record.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+const KEY_VAR = "O2B_TEST_THR_CLI_KEY";
+const MODEL = "self-hosted-cli-1";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+let storePath: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-dm-thr-cli-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ storePath = decisionThresholdsPath(configPath);
+ writeConfig({
+ vault: `"${vault}"`,
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: "http://127.0.0.1:9",
+ decision_model_id: MODEL,
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "dedup:enforce,tension:enforce",
+ });
+ resetThresholdsStoreCache();
+});
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ resetThresholdsStoreCache();
+});
+
+function writeConfig(entries: Record<string, string>): void {
+ const lines = Object.entries(entries).map(([k, v]) => `${k}: ${v}`);
+ writeFileSync(configPath, lines.join("\n") + "\n");
+}
+
+function cliEnv(): Record<string, string> {
+ return { OPEN_SECOND_BRAIN_CONFIG: configPath, [KEY_VAR]: "unused" };
+}
+
+function seedDedupRecords(model: string, mergedProbability = 0.6): void {
+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });
+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });
+ // Eight true negatives at or above 0.9 and one merged pair (one side
+ // retired) at 0.6: the fit lands on 0.9 with zero false actions. A
+ // merged pair recorded at 0.95 instead sits inside every wide window,
+ // so no threshold survives a small target.
+ writeFileSync(join(vault, "Brain", "preferences", "pref-fa.md"), "---\nid: pref-fa\n---\n");
+ writeFileSync(join(vault, "Brain", "retired", "ret-fb.md"), "---\nid: ret-fb\n---\n");
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model,
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: "f",
+ a: "pref-fa",
+ b: "pref-fb",
+ verdict: "different",
+ probability: mergedProbability,
+ },
+ ],
+ },
+ });
+ for (let i = 0; i < 8; i++) {
+ writeFileSync(
+ join(vault, "Brain", "preferences", `pref-o${i}a.md`),
+ `---\nid: pref-o${i}a\n---\n`,
+ );
+ writeFileSync(
+ join(vault, "Brain", "preferences", `pref-o${i}b.md`),
+ `---\nid: pref-o${i}b\n---\n`,
+ );
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model,
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: `o${i}`,
+ a: `pref-o${i}a`,
+ b: `pref-o${i}b`,
+ verdict: "different",
+ probability: 0.9 + i / 100,
+ },
+ ],
+ },
+ });
+ }
+}
+
+describe("o2b decision-model thresholds set and list", () => {
+ test("set writes the entry beside config.yaml; list shows it with history", async () => {
+ const set = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "dedup",
+ "--value",
+ "0.8",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(set.returncode).toBe(0);
+ expect(set.stdout).toContain(`low_priority_min = 0.8 for model '${MODEL}'`);
+ expect(existsSync(storePath)).toBe(true);
+
+ const update = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "dedup",
+ "--value",
+ "0.85",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(update.returncode).toBe(0);
+
+ const list = await runCli(["decision-model", "thresholds", "list", "--config", configPath], {
+ env: cliEnv(),
+ });
+ expect(list.returncode).toBe(0);
+ expect(list.stdout).toContain(storePath);
+ expect(list.stdout).toContain(`${MODEL} / dedup: low_priority_min=0.85 (set,`);
+ expect(list.stdout).toContain("was 0.8 (set,");
+
+ const json = await runCli(
+ ["decision-model", "thresholds", "list", "--config", configPath, "--json"],
+ { env: cliEnv() },
+ );
+ expect(json.returncode).toBe(0);
+ const parsed = JSON.parse(json.stdout) as {
+ path: string;
+ store: { models: Record<string, unknown> };
+ };
+ expect(parsed.path).toBe(storePath);
+ expect(Object.keys(parsed.store.models)).toEqual([MODEL]);
+ });
+
+ test("set refuses a value, name or use that cannot be written", async () => {
+ const badValue = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "dedup",
+ "--value",
+ "7",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(badValue.returncode).toBe(2);
+ expect(badValue.stderr).toContain("--value");
+
+ const badName = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "dedup",
+ "--value",
+ "0.5",
+ "--name",
+ "drop_below",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(badName.returncode).toBe(2);
+ expect(badName.stderr).toContain("low_priority_min");
+
+ const badUse = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "nonsense",
+ "--value",
+ "0.5",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(badUse.returncode).toBe(2);
+ expect(existsSync(storePath)).toBe(false);
+ });
+
+ test("list on a fresh install says none stored and exits 0", async () => {
+ const list = await runCli(["decision-model", "thresholds", "list", "--config", configPath], {
+ env: cliEnv(),
+ });
+ expect(list.returncode).toBe(0);
+ expect(list.stdout).toContain("none stored");
+ });
+
+ test("a per-vault override is written with --vault and listed under it", async () => {
+ const set = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "tension",
+ "--value",
+ "0.7",
+ "--vault",
+ vault,
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(set.returncode).toBe(0);
+ const store = parseThresholdsStore(readFileSync(storePath, "utf8"));
+ expect(store.vaults[vault]?.[MODEL]?.tension?.values).toEqual({ low_priority_min: 0.7 });
+ expect(store.models[MODEL]?.tension).toBeUndefined();
+ const list = await runCli(["decision-model", "thresholds", "list", "--config", configPath], {
+ env: cliEnv(),
+ });
+ expect(list.stdout).toContain("vault overrides:");
+ expect(list.stdout).toContain(vault);
+ });
+});
+
+describe("o2b decision-model thresholds fit", () => {
+ test("fit writes the derived threshold for the answering model and says what it wrote", async () => {
+ seedDedupRecords("fit-model-9");
+ const fit = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "fit",
+ "--use",
+ "dedup",
+ "--config",
+ configPath,
+ "--vault",
+ vault,
+ ],
+ { env: cliEnv() },
+ );
+ expect(fit.returncode).toBe(0);
+ expect(fit.stdout).toContain("wrote low_priority_min = 0.9 for model 'fit-model-9'");
+ expect(fit.stdout).toContain("at most 5.0%");
+ expect(fit.stdout).toContain(storePath);
+ const store = parseThresholdsStore(readFileSync(storePath, "utf8"));
+ expect(store.models["fit-model-9"]?.dedup?.values).toEqual({ low_priority_min: 0.9 });
+ expect(store.models["fit-model-9"]?.dedup?.source).toBe("fit");
+ // A different model on the same preset inherits nothing.
+ expect(store.models[MODEL]).toBeUndefined();
+ });
+
+ test("fit --json prints the written entry", async () => {
+ seedDedupRecords("fit-model-9");
+ const fit = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "fit",
+ "--use",
+ "dedup",
+ "--config",
+ configPath,
+ "--vault",
+ vault,
+ "--json",
+ ],
+ { env: cliEnv() },
+ );
+ expect(fit.returncode).toBe(0);
+ const parsed = JSON.parse(fit.stdout) as {
+ model: string;
+ name: string;
+ value: number;
+ target: number;
+ already_tuned: boolean;
+ };
+ expect(parsed).toMatchObject({
+ model: "fit-model-9",
+ name: "low_priority_min",
+ value: 0.9,
+ target: 0.05,
+ already_tuned: false,
+ });
+ });
+
+ test("a fit that misses the target is a named refusal, and nothing is written", async () => {
+ // The merged pair at 0.95 sits inside the only wide window, so no
+ // threshold both acts on eight samples and keeps a 0.1% false share.
+ seedDedupRecords("fit-model-9", 0.95);
+ const fit = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "fit",
+ "--use",
+ "dedup",
+ "--target",
+ "0.001",
+ "--config",
+ configPath,
+ "--vault",
+ vault,
+ ],
+ { env: cliEnv() },
+ );
+ expect(fit.returncode).toBe(2);
+ expect(fit.stderr).toContain("no threshold meets");
+ expect(fit.stderr).toContain("nothing was written");
+ expect(existsSync(storePath)).toBe(false);
+ });
+
+ test("a use without a labelled report is refused with the fittable list", async () => {
+ const fit = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "fit",
+ "--use",
+ "rerank",
+ "--config",
+ configPath,
+ "--vault",
+ vault,
+ ],
+ { env: cliEnv() },
+ );
+ expect(fit.returncode).toBe(2);
+ expect(fit.stderr).toContain("no labelled report");
+ });
+});
+
+describe("shadow messaging", () => {
+ test("check names the model without stored thresholds and the tuning command", async () => {
+ const report = await buildDecisionModelCheck({
+ config: {
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: "http://127.0.0.1:9",
+ decision_model_id: MODEL,
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "dedup:enforce",
+ },
+ vault: null,
+ env: cliEnv() as unknown as NodeJS.ProcessEnv,
+ markSource: null,
+ });
+ expect(report.shadow_only_uses).toEqual(["dedup"]);
+ const text = renderDecisionModelCheck(report);
+ expect(text).toContain("enforce runs as shadow for dedup");
+ expect(text).toContain(`model '${MODEL}'`);
+ expect(text).toContain("o2b decision-model thresholds fit --use dedup");
+ expect(text).toContain("o2b decision-model thresholds set");
+ });
+
+ test("after thresholds set, the use enforces and the warning is gone", async () => {
+ const set = await runCli(
+ [
+ "decision-model",
+ "thresholds",
+ "set",
+ "--model",
+ MODEL,
+ "--use",
+ "dedup",
+ "--value",
+ "0.8",
+ "--config",
+ configPath,
+ ],
+ { env: cliEnv() },
+ );
+ expect(set.returncode).toBe(0);
+ const report = await buildDecisionModelCheck({
+ config: {
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: "http://127.0.0.1:9",
+ decision_model_id: MODEL,
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "dedup:enforce",
+ },
+ vault: null,
+ env: cliEnv() as unknown as NodeJS.ProcessEnv,
+ markSource: null,
+ });
+ expect(report.shadow_only_uses).toEqual([]);
+ expect(report.uses.dedup).toBe("enforce");
+ expect(renderDecisionModelCheck(report)).not.toContain("enforce runs as shadow");
+
+ // Another model on the same preset is still shadow: the key is the
+ // model id, never the provider preset.
+ const other = await buildDecisionModelCheck({
+ config: {
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: "http://127.0.0.1:9",
+ decision_model_id: "other-model",
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "dedup:enforce",
+ },
+ vault: null,
+ env: cliEnv() as unknown as NodeJS.ProcessEnv,
+ markSource: null,
+ });
+ expect(other.shadow_only_uses).toEqual(["dedup"]);
+ });
+
+ test("report names the shadow uses and the fit command when the config is supplied", () => {
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model: MODEL,
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ });
+ const report = buildDecisionModelReport(vault, {
+ config: {
+ status: "active",
+ errors: [],
+ notes: [],
+ enabled: true,
+ provider: "compatible",
+ adapter: "systemone",
+ baseUrl: "http://127.0.0.1:9",
+ model: MODEL,
+ envKey: KEY_VAR,
+ keyPresent: true,
+ keyRequired: true,
+ allowInsecureHttp: false,
+ timeoutMs: 3000,
+ hookBudgetMs: 700,
+ maxStateTokens: 32000,
+ uses: { dedup: "shadow" } as never,
+ configuredUses: { dedup: "enforce" } as never,
+ dailyCostGateUsd: 0.5,
+ inputPriceUsdPerMtok: null,
+ allowUncalibrated: false,
+ calibrated: true,
+ thresholdProfile: null,
+ shadowOnlyUses: ["dedup"],
+ processor: null,
+ vault,
+ },
+ });
+ expect(report.shadow).toMatchObject({ model: MODEL, uses: ["dedup"] });
+ expect(report.shadow?.command).toContain("thresholds fit --use dedup");
+ const rendered = renderDecisionModelReport(report);
+ expect(rendered).toContain(`no tuned thresholds for model '${MODEL}'`);
+ expect(rendered).toContain("thresholds fit --use dedup");
+ });
+});
diff --git a/tests/cli/search-check-provider.test.ts b/tests/cli/search-check-provider.test.ts
index 9cf24813..2163e5fa 100644
--- a/tests/cli/search-check-provider.test.ts
+++ b/tests/cli/search-check-provider.test.ts
@@ -94,8 +94,13 @@ async function writeConfiguredProvider(extra: ReadonlyArray<string> = []): Promi
);
}
-/** A config with no embedding provider at all. */
+/** A config with no embedding provider at all, and no decision about one. */
async function writeUnconfiguredProvider(): Promise<void> {
+ await Bun.write(configPath, [`vault: "${vault}"`, ""].join("\n"));
+}
+
+/** A config that records a decline of the embedding provider. */
+async function writeDeclinedProvider(): Promise<void> {
await Bun.write(
configPath,
[`vault: "${vault}"`, "search_semantic_enabled: false", ""].join("\n"),
@@ -170,6 +175,27 @@ test("a provider that is not configured is unchanged, exits 0, and is never aske
expect(server.callCount()).toBe(0);
});
+test("a declined vault is skipped with the decline named, exits 0, and is never asked", async () => {
+ // A decline is a decision, not an absence: reporting it as
+ // not-configured merged the two and read a decided configuration as
+ // one that was never finished. The verdict is the skipped state - a
+ // decision is not a fault, so the exit stays 0 - and the reason is the
+ // consent's own sentence naming the keys that would re-enable it.
+ await writeDeclinedProvider();
+
+ const run = await runCheck("--json");
+
+ expect(run.payload["provider_probe"]).toBe(PROVIDER_PROBE.skipped);
+ const reason = run.payload["provider_reason"];
+ expect(typeof reason).toBe("string");
+ expect(reason).toContain("provider consent declined");
+ expect(reason).toContain("search_semantic_enabled=true");
+ expect(run.returncode).toBe(SEARCH_CHECK_EXIT.ok);
+ expect(run.payload["fatal"]).toEqual([]);
+ // The decline is honoured before any network call.
+ expect(server.callCount()).toBe(0);
+});
+
test("a provider that answers exits 0", async () => {
await writeConfiguredProvider();
diff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.ts
index d56c099c..d45ae4b2 100644
--- a/tests/core/architecture/write-site-census.test.ts
+++ b/tests/core/architecture/write-site-census.test.ts
@@ -1543,6 +1543,7 @@ const SHARED_HELPER_ROWS = 130;
*/
const ORIGIN_CHANNEL_STAMPED: ReadonlyArray<string> = Object.freeze([
"src/core/brain/continuity/store.ts",
+ "src/core/brain/ingest/ingest.ts",
"src/core/brain/log.ts",
"src/core/brain/notes/create-note.ts",
"src/core/brain/signal.ts",
@@ -1680,7 +1681,7 @@ const UNSTAMPED_DIRECT_ROWS = 84;
* stamps an origin channel - one writes a class the writer derives, the
* other a visibility token the page's own informal marker declares.
*/
-const UNSTAMPED_SHARED_ROWS = 127;
+const UNSTAMPED_SHARED_ROWS = 126;
describe("in-vault write-site census", () => {
test("every direct-fs write site carries a written exclusion", () => {
@@ -1772,6 +1773,7 @@ describe("origin-channel coverage boundary", () => {
.filter((row) => STAMPED_PATHS.has(row.path))
.map((row) => row.path);
expect(stampedShared.toSorted()).toEqual([
+ "src/core/brain/ingest/ingest.ts",
"src/core/brain/log.ts",
"src/core/brain/notes/create-note.ts",
"src/core/brain/signal.ts",
diff --git a/tests/core/brain/doctor.entities.test.ts b/tests/core/brain/doctor.entities.test.ts
index 65530438..4fdb90dc 100644
--- a/tests/core/brain/doctor.entities.test.ts
+++ b/tests/core/brain/doctor.entities.test.ts
@@ -255,4 +255,50 @@ describe("doctor derived-variant collisions (write-honesty-egress-consent F3/C5)
runDoctor(vault).warnings.filter((i) => i.code === "person-nicknames-unreadable"),
).toEqual([]);
});
+
+ test("a two-letter nickname never reads as a skip: declared nicknames are exempt from the floor", () => {
+ person("Alan Lovelace");
+ atomicWriteFileSync(
+ join(vault, PERSON_NICKNAMES_ARTIFACT_REL),
+ JSON.stringify({ al: ["Alan Lovelace"] }),
+ );
+ expect(
+ runDoctor(vault).warnings.filter((i) => i.code === "person-nicknames-entry-skipped"),
+ ).toEqual([]);
+ });
+
+ test("an entry the parser folds away surfaces once, by position", () => {
+ person("Ada Lovelace");
+ atomicWriteFileSync(
+ join(vault, PERSON_NICKNAMES_ARTIFACT_REL),
+ JSON.stringify({
+ " ": ["Ada Lovelace"],
+ "no names": [" ", ""],
+ al: ["Alan Lovelace"],
+ loop: ["Loop"],
+ }),
+ );
+
+ const warnings = runDoctor(vault).warnings.filter(
+ (i) => i.code === "person-nicknames-entry-skipped",
+ );
+ // Entries #1 (empty nickname), #2 (no usable value) and #4 (the
+ // nickname equals the name it stands for) are each named by
+ // position; #3, the usable two-letter nickname, is not a skip.
+ expect(warnings).toHaveLength(3);
+ expect(warnings.map((w) => w.message).join(" ")).toContain("#1");
+ expect(warnings.map((w) => w.message).join(" ")).toContain("#2");
+ expect(warnings.map((w) => w.message).join(" ")).toContain("#4");
+ expect(warnings.every((w) => w.severity === "warning")).toBe(true);
+ for (const w of warnings) {
+ expect(w.path).toContain("person-nicknames.json");
+ }
+ });
+
+ test("a missing nickname file raises no skipped-entry warning", () => {
+ person("Ada Lovelace");
+ expect(
+ runDoctor(vault).warnings.filter((i) => i.code === "person-nicknames-entry-skipped"),
+ ).toEqual([]);
+ });
});
diff --git a/tests/core/brain/entities/entity-read-census.test.ts b/tests/core/brain/entities/entity-read-census.test.ts
index f53259f1..062d19a0 100644
--- a/tests/core/brain/entities/entity-read-census.test.ts
+++ b/tests/core/brain/entities/entity-read-census.test.ts
@@ -148,6 +148,12 @@ const UNSCOPED_WITH_REASON: Readonly<Record<string, string>> = Object.freeze({
"agent-facing form of the same operator ask, validated against the same " +
"vocabulary. Its `get` view resolves through `getEntity`, which applies the " +
"canonical scope inside the registry.",
+ "src/core/brain/doctor/entity-checks.ts":
+ "the doctor is the surfacing point for what a read scope hides: its duplicate " +
+ "and broken-relation findings deliberately read what is STORED, so scoping " +
+ "them would stop the doctor reporting on exactly the records a read path " +
+ "hides. Its derived person-variant scan resolves through the registry's " +
+ "canonical-scope derived index, so no second filter belongs in the check.",
});
interface CensusRow {
diff --git a/tests/core/brain/entities/person-variants.test.ts b/tests/core/brain/entities/person-variants.test.ts
index e9a19ead..86f594ed 100644
--- a/tests/core/brain/entities/person-variants.test.ts
+++ b/tests/core/brain/entities/person-variants.test.ts
@@ -2,18 +2,22 @@
* Person-name variant derivation (write-honesty-egress-consent, F3 / C1).
*
* The module is a leaf: it imports nothing, so these tests exercise the
- * pure kernel only - determinism, the form-length floor, script
- * neutrality, and the rule that nickname expansion comes exclusively
- * from the operator-supplied map argument.
+ * pure kernel only - determinism, the structural form-length floor, the
+ * nickname floor exemption, script neutrality, the comparison forms for
+ * the dotted-initial and comma spellings, and the rule that nickname
+ * expansion comes exclusively from the operator-supplied argument.
*/
import { describe, expect, test } from "bun:test";
import {
derivePersonNameVariants,
+ invertPersonNicknameMap,
isPersonCategory,
MIN_PERSON_VARIANT_LENGTH,
+ personNameComparisonForm,
PERSON_ENTITY_CATEGORIES,
+ personVariantQueryForms,
type PersonNicknameMap,
} from "../../../../src/core/brain/entities/person-variants.ts";
@@ -23,6 +27,11 @@ function mapOf(entries: Record<string, string[]>): PersonNicknameMap {
);
}
+/** The operator map, inverted the way every real consumer does. */
+function indexOf(entries: Record<string, string[]>) {
+ return invertPersonNicknameMap(mapOf(entries));
+}
+
describe("derivePersonNameVariants", () => {
test("same input yields byte-identical frozen output", () => {
const a = derivePersonNameVariants("ada lovelace");
@@ -55,22 +64,24 @@ describe("derivePersonNameVariants", () => {
expect([...derivePersonNameVariants("smith")]).toEqual([]);
});
- test("nickname expansion comes only from the operator map argument", () => {
+ test("nickname expansion comes only from the operator index argument", () => {
// No map: no nickname data exists, so nothing beyond structural forms.
const bare = derivePersonNameVariants("robert smith");
expect([...bare]).toEqual(["r smith", "smith robert"]);
expect([...bare]).not.toContain("bob");
- // With the operator map (normalized keys and values, the shape the
- // config reader hands over): the nickname joins, sorted after the
- // structural forms.
- const mapped = derivePersonNameVariants("robert smith", mapOf({ bob: ["robert smith"] }));
+ // With the operator data (inverted from the config reader's shape):
+ // the nickname joins, sorted after the structural forms.
+ const mapped = derivePersonNameVariants("robert smith", indexOf({ bob: ["robert smith"] }));
expect([...mapped]).toEqual(["r smith", "smith robert", "bob"]);
});
test("nickname matching is exact over the full normalized name", () => {
- const map = mapOf({ bob: ["robert jones", "roberta stone"] });
- expect([...derivePersonNameVariants("robert smith", map)]).toEqual(["r smith", "smith robert"]);
- expect([...derivePersonNameVariants("roberta stone", map)]).toEqual([
+ const index = indexOf({ bob: ["robert jones", "roberta stone"] });
+ expect([...derivePersonNameVariants("robert smith", index)]).toEqual([
+ "r smith",
+ "smith robert",
+ ]);
+ expect([...derivePersonNameVariants("roberta stone", index)]).toEqual([
"r stone",
"stone roberta",
"bob",
@@ -78,20 +89,32 @@ describe("derivePersonNameVariants", () => {
});
test("multiple matching nicknames come out sorted", () => {
- const map = mapOf({
+ const index = indexOf({
roberto: ["robert smith"],
bob: ["robert smith"],
bobby: ["robert smith"],
});
- const out = derivePersonNameVariants("robert smith", map);
+ const out = derivePersonNameVariants("robert smith", index);
expect([...out]).toEqual(["r smith", "smith robert", "bob", "bobby", "roberto"]);
});
- test(`forms shorter than MIN_PERSON_VARIANT_LENGTH (${MIN_PERSON_VARIANT_LENGTH}) are dropped`, () => {
+ test(`structural forms shorter than MIN_PERSON_VARIANT_LENGTH (${MIN_PERSON_VARIANT_LENGTH}) are dropped`, () => {
expect(MIN_PERSON_VARIANT_LENGTH).toBe(3);
- // A two-letter nickname is below the floor and never emitted.
- const map = mapOf({ ed: ["ma li"] });
- expect([...derivePersonNameVariants("ma li", map)]).toEqual(["m li", "li ma"]);
+ // A two-token name whose initial form would be below the floor
+ // ("a b" is 3 - use a name whose initial form stays under it).
+ expect([...derivePersonNameVariants("a bo")]).toEqual(["bo a"]);
+ });
+
+ test("operator-declared nicknames are exempt from the structural floor", () => {
+ // The operator chose these forms; the kernel's floor is for forms
+ // the kernel invented. A two-letter nickname resolves.
+ const index = indexOf({ ed: ["ma li"], al: ["alan lovelace"] });
+ expect([...derivePersonNameVariants("ma li", index)]).toEqual(["m li", "li ma", "ed"]);
+ expect([...derivePersonNameVariants("alan lovelace", index)]).toEqual([
+ "a lovelace",
+ "lovelace alan",
+ "al",
+ ]);
});
test("the floor is measured on the whole emitted form, not per token", () => {
@@ -101,6 +124,13 @@ describe("derivePersonNameVariants", () => {
expect([...out]).toEqual(["a li", "li ada"]);
});
+ test("the initial is taken by code point, so an astral first letter stays whole", () => {
+ // U+10414 DESERET CAPITAL LETTER AIL - a surrogate pair in UTF-16.
+ const out = derivePersonNameVariants("\u{10414}avid smith");
+ expect([...out]).toContain("\u{10414} smith");
+ expect([...out].join("")).not.toMatch(/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/);
+ });
+
test("unspaced-script names emit no word-order or initial variants", () => {
expect([...derivePersonNameVariants("็”ฐไธญๅคช้ƒŽ")]).toEqual([]);
// Even when the operator spells the family and given names with a
@@ -113,14 +143,85 @@ describe("derivePersonNameVariants", () => {
});
test("operator nicknames still apply to unspaced-script names", () => {
- const map = mapOf({ taro: ["็”ฐไธญ ๅคช้ƒŽ"] });
- expect([...derivePersonNameVariants("็”ฐไธญ ๅคช้ƒŽ", map)]).toEqual(["taro"]);
+ const index = indexOf({ taro: ["็”ฐไธญ ๅคช้ƒŽ"] });
+ expect([...derivePersonNameVariants("็”ฐไธญ ๅคช้ƒŽ", index)]).toEqual(["taro"]);
});
test("an empty or whitespace name derives nothing", () => {
expect([...derivePersonNameVariants("")]).toEqual([]);
expect([...derivePersonNameVariants(" ")]).toEqual([]);
});
+
+ test("the dotted-initial and comma spellings derive the same variants as the plain form", () => {
+ // "A. Lovelace" normalizes to "a. lovelace": the comparison form
+ // folds the dotted initial, so the record also answers to the plain
+ // initial spelling, and the reversed form keeps the punctuation off.
+ expect([...derivePersonNameVariants("a. lovelace")]).toEqual(["a lovelace", "lovelace a"]);
+ // "Lovelace, Ada" normalizes to "lovelace, ada": the comma folds,
+ // so the plain word order joins the plain spelling's variants.
+ expect([...derivePersonNameVariants("lovelace, ada")]).toEqual([
+ "ada lovelace",
+ "a lovelace",
+ "lovelace ada",
+ ]);
+ });
+
+ test("a nickname declared for a comma-spelled full name still resolves", () => {
+ // Operator data lists the full name exactly as the record spells it.
+ const index = indexOf({ ada: ["lovelace, ada"] });
+ expect([...derivePersonNameVariants("lovelace, ada", index)]).toEqual([
+ "ada lovelace",
+ "a lovelace",
+ "lovelace ada",
+ "ada",
+ ]);
+ });
+});
+
+describe("personNameComparisonForm", () => {
+ test("a single-code-point token drops a trailing full stop", () => {
+ expect(personNameComparisonForm("a. lovelace")).toBe("a lovelace");
+ expect(personNameComparisonForm("ada m. lovelace")).toBe("ada m lovelace");
+ });
+
+ test("the dot fold is measured in code points, not UTF-16 units", () => {
+ expect(personNameComparisonForm("\u{10414}. smith")).toBe("\u{10414} smith");
+ });
+
+ test("a comma-carrying token moves behind the rest", () => {
+ expect(personNameComparisonForm("lovelace, ada")).toBe("ada lovelace");
+ expect(personNameComparisonForm("lovelace, ada mary")).toBe("ada mary lovelace");
+ });
+
+ test("multi-token dotted forms and bare tokens are untouched", () => {
+ // A dotted abbreviation that is not a single letter stays verbatim,
+ // and a clean form is a no-op.
+ expect(personNameComparisonForm("st. john smith")).toBe("st. john smith");
+ expect(personNameComparisonForm("ada lovelace")).toBe("ada lovelace");
+ expect(personNameComparisonForm("")).toBe("");
+ });
+});
+
+describe("personVariantQueryForms", () => {
+ test("a plain spelling compares through itself alone", () => {
+ expect([...personVariantQueryForms("ada lovelace")]).toEqual(["ada lovelace"]);
+ });
+
+ test("the dotted and comma spellings compare through both forms", () => {
+ expect([...personVariantQueryForms("a. lovelace")]).toEqual(["a. lovelace", "a lovelace"]);
+ expect([...personVariantQueryForms("lovelace, ada")]).toEqual([
+ "lovelace, ada",
+ "ada lovelace",
+ ]);
+ });
+});
+
+describe("invertPersonNicknameMap", () => {
+ test("the inverted index answers sorted nicknames per full name", () => {
+ const index = invertPersonNicknameMap(mapOf({ zed: ["ada lovelace"], ace: ["ada lovelace"] }));
+ expect([...index.nicknamesOf("ada lovelace")]).toEqual(["ace", "zed"]);
+ expect(index.nicknamesOf("nobody here")).toEqual([]);
+ });
});
describe("isPersonCategory", () => {
diff --git a/tests/core/brain/entities/registry-derived-lookup.test.ts b/tests/core/brain/entities/registry-derived-lookup.test.ts
index 0bcb87a9..f68d6bc8 100644
--- a/tests/core/brain/entities/registry-derived-lookup.test.ts
+++ b/tests/core/brain/entities/registry-derived-lookup.test.ts
@@ -23,9 +23,13 @@ import {
archiveEntity,
getEntity,
listEntities,
+ resolvePersonDerivedIndex,
upsertEntity,
} from "../../../../src/core/brain/entities/registry.ts";
-import { PERSON_NICKNAMES_ARTIFACT_REL } from "../../../../src/core/brain/entities/person-nicknames.ts";
+import {
+ PERSON_NICKNAMES_ARTIFACT_REL,
+ PersonNicknameConfigError,
+} from "../../../../src/core/brain/entities/person-nicknames.ts";
import { BRAIN_ENTITY_STATUS } from "../../../../src/core/brain/entities/types.ts";
let vault: string;
@@ -82,6 +86,31 @@ describe("getEntity resolves a unique derived match after the exact miss", () =>
expect(getEntity(vault, { category: "people", query: "r smith" })!.id).toBe(id);
});
+ test("the dotted-initial and comma spellings resolve the plain-named person", () => {
+ const id = upsertPerson("Ada Lovelace");
+ // "A. Lovelace" and "Lovelace, Ada" are the two punctuation shapes
+ // normalization keeps; both fold onto the derived forms of the
+ // plain spelling instead of missing.
+ expect(getEntity(vault, { query: "A. Lovelace" })!.id).toBe(id);
+ expect(getEntity(vault, { query: "Lovelace, Ada" })!.id).toBe(id);
+ expect(getEntity(vault, { category: "people", query: "lovelace, ada" })!.id).toBe(id);
+ });
+
+ test("an astral initial resolves through the derived initial form", () => {
+ // U+10414 DESERET CAPITAL LETTER AIL - a surrogate pair in UTF-16,
+ // so a UTF-16-unit initial slice would emit a lone surrogate.
+ const id = upsertPerson("\u{10414}avid Smith");
+ expect(getEntity(vault, { query: "\u{10414} smith" })!.id).toBe(id);
+ });
+
+ test("a two-letter operator nickname resolves the person it stands for", () => {
+ // Operator-declared nicknames are exempt from the structural floor.
+ const id = upsertPerson("Alan Lovelace");
+ writeNicknames({ al: ["Alan Lovelace"] });
+ expect(getEntity(vault, { query: "Al" })!.id).toBe(id);
+ expect(getEntity(vault, { query: "a lovelace" })!.id).toBe(id);
+ });
+
test("an exact declared form still wins before the derived lane is consulted", () => {
const id = upsertPerson("Robert Smith", { aliases: ["Rob"] });
writeNicknames({ bob: ["Robert Smith"], rob: ["Robert Smith"] });
@@ -110,6 +139,29 @@ describe("a shared derived form is an ambiguity, not a silent pick", () => {
expect(amb.message).toContain(jones);
});
+ test("the derived ambiguity names the remedy of two real people, not a merge", () => {
+ // Two persons reaching one initial form are not duplicates: the
+ // message must offer a more specific reference, never the
+ // archive-or-merge exit the exact lane's duplicates get.
+ upsertPerson("Robert Smith");
+ upsertPerson("Robert Jones");
+ writeNicknames({ bob: ["Robert Smith", "Robert Jones"] });
+
+ let err: unknown;
+ try {
+ getEntity(vault, { query: "bob" });
+ } catch (e) {
+ err = e;
+ }
+ const amb = err as EntityAmbiguityError;
+ expect(amb.derived).toBe(true);
+ expect(amb.message).toContain("derived name form");
+ expect(amb.message).toContain("full name");
+ expect(amb.message).toContain("alias");
+ expect(amb.message).toContain("category");
+ expect(amb.message).not.toContain("archived or merged");
+ });
+
test("the exact forms of the same two persons still resolve uniquely", () => {
const smith = upsertPerson("Robert Smith");
const jones = upsertPerson("Robert Jones");
@@ -209,3 +261,35 @@ describe("a malformed nickname file degrades reads to exact-only", () => {
expect(getEntity(vault, { query: "bob" })).toBeNull();
});
});
+
+describe("the derived person-form index is cached per signature", () => {
+ test("unchanged inputs return the same object; a nickname edit forces a rebuild", () => {
+ const smith = upsertPerson("Robert Smith");
+ writeNicknames({ bob: ["Robert Smith"] });
+
+ const first = resolvePersonDerivedIndex(vault);
+ expect(resolvePersonDerivedIndex(vault)).toBe(first);
+
+ // A nicknames edit shifts the mtime/size signature and forces a
+ // fresh build that answers through the new entry.
+ writeNicknames({ bob: ["Robert Smith"], rob: ["Robert Smith"] });
+ const second = resolvePersonDerivedIndex(vault);
+ expect(second).not.toBe(first);
+ expect(second.byForm.get("rob")?.derived.map((e) => e.id)).toEqual([smith]);
+
+ // A present-but-unusable file is captured, not thrown: the derived
+ // lane empties while the structure stays available to consumers
+ // that degrade by posture.
+ writeFileSync(join(vault, PERSON_NICKNAMES_ARTIFACT_REL), "{ not json");
+ const broken = resolvePersonDerivedIndex(vault);
+ expect(broken).not.toBe(second);
+ expect(broken.nicknameError).toBeInstanceOf(PersonNicknameConfigError);
+ expect(broken.byForm.get("bob")).toBeUndefined();
+ // An entity write still shows up: the index identity is part of the
+ // cache key.
+ upsertPerson("Robert Jones");
+ const afterWrite = resolvePersonDerivedIndex(vault);
+ expect(afterWrite).not.toBe(broken);
+ expect(afterWrite.nicknameError).toBeInstanceOf(PersonNicknameConfigError);
+ });
+});
diff --git a/tests/core/brain/ingest/ingest-derived-section.test.ts b/tests/core/brain/ingest/ingest-derived-section.test.ts
index 24d890cc..efbd2935 100644
--- a/tests/core/brain/ingest/ingest-derived-section.test.ts
+++ b/tests/core/brain/ingest/ingest-derived-section.test.ts
@@ -338,6 +338,22 @@ describe("visibility on a summary page", () => {
expect(frontmatterLines(page(vault, res.summaryPath))).toContain("visibility: [private]");
});
+ test("an informally marked source reserves the page its derived section rides on", () => {
+ // `access: private` declares no `visibility:` the reach predicate's
+ // `pageVisibility` would read. Reading the declaration alone stamped no
+ // tokens, and the page carrying the source's rows went out at default
+ // visibility - remotely served.
+ seed(vault, "Clips/parts.csv", `---\naccess: private\n---\n${PARTS_CSV}`);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(res.table).toMatchObject({ rendered: true, rows: 2 });
+ const text = page(vault, res.summaryPath);
+ expect(frontmatterLines(text)).toContain("visibility: [private]");
+ // The copied source content rides a page that now carries the reserved
+ // token, not a default-visibility one.
+ expect(text).toContain("## Table");
+ expect(text).toContain("bolt | 4");
+ });
+
test("a source moved to another audience withholds the page rather than widening it", () => {
seed(vault, "Clips/parts.csv", `---\nvisibility: team-a\n---\n${PARTS_CSV}`);
const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
diff --git a/tests/core/brain/page-lint.test.ts b/tests/core/brain/page-lint.test.ts
index b0b4a629..08a5dc10 100644
--- a/tests/core/brain/page-lint.test.ts
+++ b/tests/core/brain/page-lint.test.ts
@@ -152,8 +152,9 @@ describe("lintWrittenPages - findings", () => {
* The fifth detector (write-honesty, F4, t_fb442a24). A count or amount
* with no structural date is an advisory warning on the `body` path: the
* page states a quantity a reader cannot place in time. The escape hatches
- * are structural, never natural language - `valid_from` in frontmatter, any
- * ISO date in the body, or a `[Source: <name>, YYYY-MM-DD]` marker.
+ * are structural, never natural language - `valid_from` in frontmatter, an
+ * ISO date in the body's claim-bearing prose, or a
+ * `[Source: <name>, YYYY-MM-DD]` marker.
*/
describe("lintWrittenPages - undated-quantity findings", () => {
const BODY = "We onboarded 5,000 users.";
@@ -214,6 +215,35 @@ describe("lintWrittenPages - undated-quantity findings", () => {
expect(pageLintField(report)).toEqual({});
});
+ test("a date inside a fence, a URL or a wikilink target silences nothing", () => {
+ // The hatch reads the masked view the scanner scans: a date in a
+ // region that carries no claim must not quiet the quantities in the
+ // prose around it.
+ const rel = writeNote(
+ "Notes/MaskedDate.md",
+ "---\ntitle: M\n---\n\n```\n2026-10-01\n```\n\n" +
+ "see https://x.com/posts/2026-10-01 and [[Report 2026-10-01]]\n\n" +
+ `${BODY}\n`,
+ );
+ const findings = undatedFindings(lintWrittenPages(vault, [rel], READ_ALL));
+ expect(findings).toHaveLength(1);
+ expect(findings[0]!.message).toContain(JSON.stringify(QUANTITY));
+ });
+
+ test("the offset is named as a post-frontmatter body coordinate, not a file offset", () => {
+ const raw = `---\ntitle: U\n---\n\n ${BODY}\n`;
+ const rel = writeNote("Notes/OffsetCoordinate.md", raw);
+ const finding = undatedFindings(lintWrittenPages(vault, [rel], READ_ALL))[0]!;
+ // The scanner sees the trimmed body, so the lexeme sits at its body
+ // offset; in the raw file it sits later by the frontmatter and the
+ // leading blank line and spaces. The message names the coordinate
+ // system, so the number can be located honestly.
+ expect(finding.message).toContain(
+ `at body offset ${BODY.indexOf(QUANTITY)} (after frontmatter)`,
+ );
+ expect(finding.message).not.toContain(`offset ${raw.indexOf(QUANTITY)}`);
+ });
+
test("a [Source: <name>, YYYY-MM-DD] marker silences the finding", () => {
const rel = writeNote(
"Notes/Cited.md",
diff --git a/tests/core/brain/portability/knowledge-pack.test.ts b/tests/core/brain/portability/knowledge-pack.test.ts
index 95f63c2c..1d4043ce 100644
--- a/tests/core/brain/portability/knowledge-pack.test.ts
+++ b/tests/core/brain/portability/knowledge-pack.test.ts
@@ -165,6 +165,26 @@ describe("export", () => {
);
expect(() => selectKnowledgePack(src, [])).toThrow(KnowledgePackError);
});
+
+ test("an informal privacy marker blocks a page that declares no visibility", () => {
+ seedSource();
+ page(
+ src,
+ "Notes/marked.md",
+ "---\ntags: [ops]\naccess: private\n---\nThe informally marked incident notes.\n",
+ );
+ // A marker-only page resolves to no `visibility:` token at all, so a
+ // declaration-only gate would ship it, body included, to a shared
+ // export. The marker denies a remote read all the same.
+ const selected = selectKnowledgePack(src, ["tag:ops"]);
+ expect(selected.okfManifest.pages.map((p) => p.path)).toEqual(["Notes/runbook.md"]);
+ expect(selected.blocked).toContainEqual({
+ kind: "page",
+ id: "Notes/marked.md",
+ reason: "visibility",
+ });
+ expect(JSON.stringify(selected)).not.toContain("informally marked incident notes");
+ });
});
describe("preview", () => {
diff --git a/tests/core/brain/provenance/content-class-paths.test.ts b/tests/core/brain/provenance/content-class-paths.test.ts
index 06618d4f..60fbae39 100644
--- a/tests/core/brain/provenance/content-class-paths.test.ts
+++ b/tests/core/brain/provenance/content-class-paths.test.ts
@@ -1,9 +1,11 @@
/**
* The deterministic prompt-derived write paths (F5, task E2).
*
- * Three write paths are definitionally prompt-derived and stamp
+ * Four write paths are definitionally prompt-derived and stamp
* `content_class: prompt-derived` at creation, server-side:
* - a note created over the MCP transport (the mcp-tool origin channel),
+ * - a source summary ingested over the same channel (the ingest pipeline
+ * composes its own frontmatter, and the calling agent wrote the prose),
* - a signal replayed from a session by the import,
* - a research report the in-process pipeline synthesised.
*
@@ -16,7 +18,7 @@
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
-import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";
+import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { readFileSync as readSource } from "node:fs";
@@ -28,6 +30,7 @@ import { importSession } from "../../../../src/core/brain/sessions/import.ts";
import { writeResearchReport } from "../../../../src/core/brain/research/research.ts";
import { bootstrapBrain } from "../../../../src/core/brain/init.ts";
import { createNote } from "../../../../src/core/brain/notes/create-note.ts";
+import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts";
import {
CONTENT_CLASS_FIELD,
CONTENT_CLASS,
@@ -105,6 +108,54 @@ describe("research reports stamp the class", () => {
});
});
+/** Seed and ingest one local source, the shape every ingest lane writes. */
+function ingestLocalSource() {
+ writeFileSync(join(vault, "source.md"), "raw source bytes\n", "utf8");
+ return ingestSource(
+ vault,
+ {
+ sourcePath: "source.md",
+ summary: "Agent-written summary prose.",
+ extraction: { entities: [{ category: "concept", name: "Bolts" }], relations: [] },
+ },
+ { agent: "claude", now: new Date("2026-06-13T12:00:00Z") },
+ );
+}
+
+describe("ingest stamps the class", () => {
+ test("a summary page ingested over the MCP channel carries content_class", () => {
+ setOriginChannel(ORIGIN_CHANNEL.mcpTool);
+ const res = ingestLocalSource();
+ const text = readFileSync(join(vault, res.summaryPath), "utf8");
+ expect(text).toContain(`${CONTENT_CLASS_FIELD}: ${CONTENT_CLASS.promptDerived}`);
+ });
+
+ test("a re-ingest over the CLI keeps the creating writer's stamp", () => {
+ setOriginChannel(ORIGIN_CHANNEL.mcpTool);
+ const first = ingestLocalSource();
+ setOriginChannel(ORIGIN_CHANNEL.cli);
+ const second = ingestSource(
+ vault,
+ {
+ sourcePath: "source.md",
+ summary: "Rewritten summary prose.",
+ extraction: { entities: [{ category: "concept", name: "Bolts" }], relations: [] },
+ },
+ { agent: "claude", now: new Date("2026-06-14T12:00:00Z") },
+ );
+ expect(second.summaryPath).toBe(first.summaryPath);
+ const text = readFileSync(join(vault, second.summaryPath), "utf8");
+ expect(text).toContain(`${CONTENT_CLASS_FIELD}: ${CONTENT_CLASS.promptDerived}`);
+ });
+
+ test("an operator-run CLI ingest stays unmarked", () => {
+ setOriginChannel(ORIGIN_CHANNEL.cli);
+ const res = ingestLocalSource();
+ const text = readFileSync(join(vault, res.summaryPath), "utf8");
+ expect(text).not.toContain(CONTENT_CLASS_FIELD);
+ });
+});
+
describe("operator-authored CLI writes stay unmarked", () => {
test("a note created under a claimed cli channel carries no class", () => {
setOriginChannel(ORIGIN_CHANNEL.cli);
diff --git a/tests/core/brain/provenance/content-class.test.ts b/tests/core/brain/provenance/content-class.test.ts
index 0c55da62..ade14d41 100644
--- a/tests/core/brain/provenance/content-class.test.ts
+++ b/tests/core/brain/provenance/content-class.test.ts
@@ -31,6 +31,7 @@ import {
contentClassCreationStamp,
ContentClassValueError,
isContentClass,
+ omittedPageReferenceMatcher,
readContentClass,
resolveContentClass,
} from "../../../../src/core/brain/provenance/content-class.ts";
@@ -41,7 +42,6 @@ import {
ORIGIN_CHANNEL,
setOriginChannel,
} from "../../../../src/core/origin-channel.ts";
-
let vault: string;
beforeEach(() => {
vault = mkdtempSync(join(tmpdir(), "o2b-content-class-"));
@@ -173,3 +173,28 @@ describe("an update cannot forge or strip the class", () => {
expect(written).toContain(`${CONTENT_CLASS_FIELD}: prompt-derived`);
});
});
+
+describe("the omitted-page reference matcher", () => {
+ const namesOmitted = omittedPageReferenceMatcher(["Agent Page.md", "Notes/Nested Plan.md"]);
+
+ test("a bare target names an omitted page with or without the .md suffix", () => {
+ expect(namesOmitted("Agent Page")).toBe(true);
+ expect(namesOmitted("Agent Page.md")).toBe(true);
+ });
+
+ test("a path-shaped target matches the exact path, suffix optional", () => {
+ expect(namesOmitted("Notes/Nested Plan")).toBe(true);
+ expect(namesOmitted("Notes/Nested Plan.md")).toBe(true);
+ // A different folder holding the same basename is a different page.
+ expect(namesOmitted("Other/Agent Page.md")).toBe(false);
+ });
+
+ test("an absolute resource names no vault page", () => {
+ expect(namesOmitted("https://example.org/Agent Page")).toBe(false);
+ });
+
+ test("anything else names a kept page and stays", () => {
+ expect(namesOmitted("Operator Page")).toBe(false);
+ expect(namesOmitted("")).toBe(false);
+ });
+});
diff --git a/tests/core/brain/recall-inject-decision.test.ts b/tests/core/brain/recall-inject-decision.test.ts
index 48b9b89d..873ad478 100644
--- a/tests/core/brain/recall-inject-decision.test.ts
+++ b/tests/core/brain/recall-inject-decision.test.ts
@@ -48,6 +48,7 @@ const NOTE_TEXT = {
b: "Grocery list with bread and milk, nothing about herons.",
c: "Heron nesting sites along the river, used every spring.",
s: "Secret heron counts kept for the landowner only.",
+ m: "Informally marked heron counts, reserved by an access key.",
};
const vaults: string[] = [];
@@ -67,6 +68,7 @@ function tempVault(): string {
write("notes/b.md", `# Groceries\n\n${NOTE_TEXT.b}\n`);
write("notes/c.md", `# Heron nests\n\n${NOTE_TEXT.c}\n`);
write("notes/s.md", `---\nvisibility: private\n---\n# Counts\n\n${NOTE_TEXT.s}\n`);
+ write("notes/m.md", `---\naccess: private\n---\n# Marked\n\n${NOTE_TEXT.m}\n`);
return vault;
}
@@ -149,7 +151,7 @@ function expectNoText(value: unknown): void {
expect(text).not.toContain("heron migration");
expect(text).not.toContain(PROMPT);
for (const t of Object.values(NOTE_TEXT)) expect(text).not.toContain(t);
- for (const k of ["a", "b", "c", "s"]) expect(text).not.toContain(`Title ${k}`);
+ for (const k of ["a", "b", "c", "s", "m"]) expect(text).not.toContain(`Title ${k}`);
}
describe("recall_inject off: no filter, no request, no field", () => {
@@ -358,6 +360,28 @@ describe("recall_inject enforce", () => {
expect(record["withheld_count"]).toBe(1);
});
+ test("an informally marked note (access: private) is withheld like a formal one", async () => {
+ // `access: private` resolves to no caller-scope token, so a gate fed
+ // by `pageVisibility` alone read the note as public and sent it.
+ const vault = tempVault();
+ const provider = new FakeDecisionProvider({ answer: answerBy({ b: 0.05 }, 0.05) });
+ const filter = createRecallInjectDecisionFilter({
+ config: config(vault, "enforce"),
+ vault,
+ provider,
+ });
+ const notes = [note("a", 0.9), note("m", 0.85), note("b", 0.8)];
+ const decision = await decide(notes, filter);
+ expect(allText(provider.requests)).not.toContain(NOTE_TEXT.m);
+ expect(Object.keys(provider.requests[0]!.questions)).toHaveLength(3); // two notes + inject_any
+ expect(decision.kind).toBe("inject");
+ if (decision.kind !== "inject") return;
+ // The mask hides the withheld note from the model, as for a formal one.
+ const record = listDecisionModelCalls(vault)[0]!.payload;
+ expect(record["note_paths"]).toEqual(["notes/a.md", "(withheld)", "notes/b.md"]);
+ expect(record["withheld_count"]).toBe(1);
+ });
+
test("notes from another origin and every note private: nothing sent, today's brief", async () => {
const vault = tempVault();
const provider = new FakeDecisionProvider();
diff --git a/tests/core/brain/retirement-log-shard.test.ts b/tests/core/brain/retirement-log-shard.test.ts
new file mode 100644
index 00000000..adb45fbc
--- /dev/null
+++ b/tests/core/brain/retirement-log-shard.test.ts
@@ -0,0 +1,232 @@
+/**
+ * Regression tests for issue #271: the `retired_by` provenance a merge
+ * or dream retirement stamps must name the log shard this device
+ * actually appends to, not the legacy un-sharded day file.
+ *
+ * The device id is configured the way the issue's repro does it - a
+ * temp `config.yaml` with `device_id` reached through
+ * `OPEN_SECOND_BRAIN_CONFIG`, with `O2B_DEVICE_ID` cleared so the
+ * config file is what resolves. Every test registers its own config and
+ * vault and restores the environment in `afterEach`, including on
+ * failure.
+ *
+ * The backward-compatibility half pins the historical shape: with the
+ * empty device id (the documented `O2B_DEVICE_ID=""` opt-out) the link
+ * stays exactly `[[Brain/log/<date>]]` - no historical record is
+ * rewritten by the fix, and un-sharded vaults keep writing the same
+ * bytes they always did.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { dream } from "../../../src/core/brain/dream.ts";
+import { bootstrapBrain } from "../../../src/core/brain/init.ts";
+import { logShardWikilink } from "../../../src/core/brain/log.ts";
+import { resolveAppendShardId } from "../../../src/core/brain/ledger-shards.ts";
+import { mergePreferences } from "../../../src/core/brain/merge.ts";
+import { brainDirs, retiredPath } from "../../../src/core/brain/paths.ts";
+import { parseRetired, writePreference } from "../../../src/core/brain/preference.ts";
+import { runDoctor } from "../../../src/core/brain/doctor.ts";
+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";
+import { normaliseWikilinkTarget } from "../../../src/core/brain/wikilink.ts";
+
+const DEVICE_ID = "fixturedev";
+
+let vault: string;
+let configHome: string;
+const savedEnv: Record<string, string | undefined> = {};
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-retire-shard-"));
+ configHome = mkdtempSync(join(tmpdir(), "o2b-retire-shard-cfg-"));
+ savedEnv["OPEN_SECOND_BRAIN_CONFIG"] = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ savedEnv["O2B_DEVICE_ID"] = process.env["O2B_DEVICE_ID"];
+ // Clear the env override so the device id resolves from the config
+ // file, like the issue's repro.
+ delete process.env["O2B_DEVICE_ID"];
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+ rmSync(configHome, { recursive: true, force: true });
+ for (const key of ["OPEN_SECOND_BRAIN_CONFIG", "O2B_DEVICE_ID"]) {
+ const value = savedEnv[key];
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+});
+
+/**
+ * Point `OPEN_SECOND_BRAIN_CONFIG` at a temp config.yaml for `vault`,
+ * with `device_id` present (sharded) or absent plus the empty env
+ * opt-out (legacy un-sharded pair).
+ */
+function registerConfig(deviceId: string | null): void {
+ const configPath = join(configHome, "config.yaml");
+ const deviceLine = deviceId === null ? "" : `device_id: "${deviceId}"\n`;
+ atomicWriteFileSync(configPath, `vault: ${vault}\n${deviceLine}`);
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;
+ if (deviceId === null) process.env["O2B_DEVICE_ID"] = "";
+ bootstrapBrain(vault, { configPath });
+}
+
+function confirmedPref(slug: string, topic: string): void {
+ writePreference(vault, {
+ slug,
+ topic,
+ principle: `Principle for ${slug}`,
+ created_at: "2026-05-01T00:00:00Z",
+ unconfirmed_until: "2026-05-08T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: [],
+ confirmed_at: "2026-05-02T00:00:00Z",
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-05-02T00:00:00Z",
+ confidence: "high",
+ confidence_value: 0.8,
+ pinned: false,
+ });
+}
+
+/**
+ * The issue's core assertion: the retired_by link's target must exist
+ * as a file, and the doctor must report no broken-wikilink finding for
+ * the retired_by field.
+ */
+function assertProvenanceResolves(retiredBy: string): void {
+ const target = normaliseWikilinkTarget(retiredBy);
+ expect(existsSync(join(brainDirs(vault).log, `${target}.md`))).toBe(true);
+
+ const doc = runDoctor(vault, { now: new Date("2026-10-22T12:00:00Z") });
+ const brokenRetiredBy = doc.warnings.filter(
+ (w) => w.code === "broken-wikilink" && w.field === "retired_by",
+ );
+ expect(brokenRetiredBy).toEqual([]);
+}
+
+// ----- Shared helper --------------------------------------------------------
+
+describe("logShardWikilink", () => {
+ test("empty shard id keeps the exact historical shape", () => {
+ registerConfig(null);
+ expect(logShardWikilink("2026-10-22")).toBe("[[Brain/log/2026-10-22]]");
+ });
+
+ test("configured device id names the shard file the appender writes", () => {
+ registerConfig(DEVICE_ID);
+ expect(resolveAppendShardId()).toBe(DEVICE_ID);
+ expect(logShardWikilink("2026-10-22")).toBe(`[[Brain/log/2026-10-22.${DEVICE_ID}]]`);
+ });
+
+ test("an explicit shard id wins over config resolution", () => {
+ registerConfig(DEVICE_ID);
+ expect(logShardWikilink("2026-10-22", "otherdev")).toBe("[[Brain/log/2026-10-22.otherdev]]");
+ });
+
+ test("rejects a shard id the appender would reject", () => {
+ registerConfig(DEVICE_ID);
+ expect(() => logShardWikilink("2026-10-22", "Sync-Conflict")).toThrow();
+ });
+});
+
+// ----- Merge retirement -----------------------------------------------------
+
+describe("mergePreferences retirement provenance under a configured device id", () => {
+ test("retired_by names the existing device shard and the doctor reports no broken link", () => {
+ registerConfig(DEVICE_ID);
+ confirmedPref("alpha", "commits");
+ confirmedPref("beta", "commits");
+
+ const now = new Date("2026-10-22T10:00:00Z");
+ mergePreferences(vault, "pref-alpha", "pref-beta", { now });
+
+ const ret = parseRetired(retiredPath(vault, "beta"));
+ expect(ret.retired_by).toBe(`[[Brain/log/2026-10-22.${DEVICE_ID}]]`);
+
+ const logDir = brainDirs(vault).log;
+ expect(existsSync(join(logDir, `2026-10-22.${DEVICE_ID}.md`))).toBe(true);
+ expect(existsSync(join(logDir, "2026-10-22.md"))).toBe(false);
+
+ assertProvenanceResolves(ret.retired_by);
+ });
+
+ test("empty device id keeps the historical un-sharded shape", () => {
+ registerConfig(null);
+ confirmedPref("alpha", "commits");
+ confirmedPref("beta", "commits");
+
+ const now = new Date("2026-10-22T10:00:00Z");
+ mergePreferences(vault, "pref-alpha", "pref-beta", { now });
+
+ const ret = parseRetired(retiredPath(vault, "beta"));
+ expect(ret.retired_by).toBe("[[Brain/log/2026-10-22]]");
+ expect(existsSync(join(brainDirs(vault).log, "2026-10-22.md"))).toBe(true);
+
+ assertProvenanceResolves(ret.retired_by);
+ });
+});
+
+// ----- Dream retirement -----------------------------------------------------
+
+describe("dream retirement provenance under a configured device id", () => {
+ test("retired_by names the existing device shard and the doctor reports no broken link", () => {
+ registerConfig(DEVICE_ID);
+ writePreference(vault, {
+ slug: "stale-rule",
+ topic: "stale-rule",
+ principle: "Old rule",
+ created_at: "2026-01-01T00:00:00Z",
+ unconfirmed_until: "2026-01-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: [],
+ confirmed_at: "2026-01-05T00:00:00Z",
+ applied_count: 3,
+ violated_count: 0,
+ last_evidence_at: "2026-01-10T00:00:00Z",
+ confidence: "medium",
+ pinned: false,
+ });
+
+ dream(vault, { now: new Date("2026-08-01T00:00:00Z") });
+
+ const ret = parseRetired(retiredPath(vault, "stale-rule"));
+ expect(ret.retired_by).toBe(`[[Brain/log/2026-08-01.${DEVICE_ID}]]`);
+
+ const logDir = brainDirs(vault).log;
+ expect(existsSync(join(logDir, `2026-08-01.${DEVICE_ID}.md`))).toBe(true);
+ expect(existsSync(join(logDir, "2026-08-01.md"))).toBe(false);
+
+ assertProvenanceResolves(ret.retired_by);
+ });
+
+ test("empty device id keeps the historical un-sharded shape", () => {
+ registerConfig(null);
+ writePreference(vault, {
+ slug: "stale-rule",
+ topic: "stale-rule",
+ principle: "Old rule",
+ created_at: "2026-01-01T00:00:00Z",
+ unconfirmed_until: "2026-01-15T00:00:00Z",
+ status: "confirmed",
+ evidenced_by: [],
+ confirmed_at: "2026-01-05T00:00:00Z",
+ applied_count: 3,
+ violated_count: 0,
+ last_evidence_at: "2026-01-10T00:00:00Z",
+ confidence: "medium",
+ pinned: false,
+ });
+
+ dream(vault, { now: new Date("2026-08-01T00:00:00Z") });
+
+ const ret = parseRetired(retiredPath(vault, "stale-rule"));
+ expect(ret.retired_by).toBe("[[Brain/log/2026-08-01]]");
+ expect(existsSync(join(brainDirs(vault).log, "2026-08-01.md"))).toBe(true);
+
+ assertProvenanceResolves(ret.retired_by);
+ });
+});
diff --git a/tests/core/brain/undated-quantity.test.ts b/tests/core/brain/undated-quantity.test.ts
index 58f96211..6a295c9f 100644
--- a/tests/core/brain/undated-quantity.test.ts
+++ b/tests/core/brain/undated-quantity.test.ts
@@ -16,8 +16,10 @@ import { describe, expect, test } from "bun:test";
import {
containsIsoDate,
+ containsProseIsoDate,
findUndatedQuantities,
} from "../../../src/core/brain/undated-quantity.ts";
+import { LINEAR_CEILING_MS } from "../../helpers/linear-time.ts";
interface PositiveCase {
readonly name: string;
@@ -87,6 +89,11 @@ const POSITIVE: ReadonlyArray<PositiveCase> = [
body: "2,024 users were migrated",
lexemes: ["2,024 users"],
},
+ {
+ name: "a single-letter unit after whitespace",
+ body: "the job ran 3 d",
+ lexemes: ["3 d"],
+ },
];
const NEGATIVE: ReadonlyArray<PositiveCase> = [
@@ -170,6 +177,31 @@ const NEGATIVE: ReadonlyArray<PositiveCase> = [
body: "In 2024 users doubled",
lexemes: [],
},
+ {
+ name: "a dimension letter glued to a number",
+ body: "a 3D printer outputs 2D jigs",
+ lexemes: [],
+ },
+ {
+ name: "a plan letter glued to a number",
+ body: "plan 2b covers the rollout",
+ lexemes: [],
+ },
+ {
+ name: "a count inside a wikilink target",
+ body: "see [[Pricing 12,400 plan]] for details",
+ lexemes: [],
+ },
+ {
+ name: "numbers in a URL run",
+ body: "fetched https://x.com/item/5000users?q=50% just now",
+ lexemes: [],
+ },
+ {
+ name: "a markdown link destination",
+ body: "the rows live in [quarterly export](12,400-rows.csv) here",
+ lexemes: [],
+ },
];
describe("findUndatedQuantities - the trigger classes", () => {
@@ -233,3 +265,64 @@ describe("containsIsoDate", () => {
expect(containsIsoDate("upgraded to v1.73.0")).toBe(false);
});
});
+
+describe("containsProseIsoDate - the escape hatch reads the masked view", () => {
+ test("a prose date hatches", () => {
+ expect(containsProseIsoDate("published 2026-10-22 by the team")).toBe(true);
+ });
+
+ test("an instant in prose hatches", () => {
+ expect(containsProseIsoDate("ended 2026-10-22T10:30:00Z on the dot")).toBe(true);
+ });
+
+ test("a date inside a fenced block is not prose", () => {
+ expect(containsProseIsoDate("```\n2026-10-22\n```\n\n5,000 users\n")).toBe(false);
+ });
+
+ test("a date inside an inline code span is not prose", () => {
+ expect(containsProseIsoDate("the stamp `2026-10-22` is quoted")).toBe(false);
+ });
+
+ test("a date inside a table row is not prose", () => {
+ expect(containsProseIsoDate("| 2026-10-22 | 35% |")).toBe(false);
+ });
+
+ test("a date inside a URL is not prose", () => {
+ expect(containsProseIsoDate("see https://x.com/posts/2026-10-22 today")).toBe(false);
+ });
+
+ test("a date inside a wikilink target is not prose", () => {
+ expect(containsProseIsoDate("see [[Report 2026-10-22]] today")).toBe(false);
+ });
+
+ test("text with no date anywhere does not hatch", () => {
+ expect(containsProseIsoDate("no date in here at all")).toBe(false);
+ });
+});
+
+/**
+ * The run shapes admit no comma or dot before the first digit. Without
+ * that guard, in a comma-joined digit run every position after a comma
+ * was a legal match start, each running `[\d,]*` to the end and
+ * backtracking through the tail: 45 s on 50,000 chars, in the
+ * synchronous write receipt. A quadratic pass over a 256 KiB body takes
+ * minutes, the fixed scan takes milliseconds, and the ceiling is the
+ * shared guard sized for a loaded CI runner.
+ */
+describe("findUndatedQuantities - bounded scan time", () => {
+ const QUARTER_MIB_CHARS = 256 * 1024;
+
+ test("a comma-joined digit run the size of the artifact cap stays linear", () => {
+ const body = "1,".repeat(QUARTER_MIB_CHARS / 2);
+ const started = performance.now();
+ expect(findUndatedQuantities(body)).toEqual([]);
+ expect(performance.now() - started).toBeLessThan(LINEAR_CEILING_MS);
+ });
+
+ test("a dotted version-like run the size of the artifact cap stays linear", () => {
+ const body = "1.1.".repeat(QUARTER_MIB_CHARS / 4);
+ const started = performance.now();
+ expect(findUndatedQuantities(body)).toEqual([]);
+ expect(performance.now() - started).toBeLessThan(LINEAR_CEILING_MS);
+ });
+});
diff --git a/tests/core/brain/write-batch.test.ts b/tests/core/brain/write-batch.test.ts
index 0313e89f..0ed3f801 100644
--- a/tests/core/brain/write-batch.test.ts
+++ b/tests/core/brain/write-batch.test.ts
@@ -253,6 +253,111 @@ describe("applyWriteBatch note operations", () => {
expect(readFileSync(join(vault, "Notes/Authored.md"), "utf8")).toContain("access: private");
});
+ test("update_note writes a marker key whose deny status stands still", () => {
+ // The guard is about the note's deny status, not the key's presence:
+ // a value that denies nothing (`read-only`), or a move between two
+ // non-deny values, is note content like any other key's.
+ seedNote("Notes/ReadOnly.md", "body", "title: R\naccess: read-only");
+ const res = applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/ReadOnly.md", frontmatter: { access: "read-only" } },
+ ]);
+ expect(res.applied).toBe(1);
+ expect(readFileSync(join(vault, "Notes/ReadOnly.md"), "utf8")).toContain("access: read-only");
+
+ seedNote("Notes/Move.md", "body", "title: M\naccess: public");
+ const moved = applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/Move.md", frontmatter: { access: "team" } },
+ ]);
+ expect(moved.applied).toBe(1);
+ expect(readFileSync(join(vault, "Notes/Move.md"), "utf8")).toContain("access: team");
+
+ // A marker note is otherwise a normal note: an update that leaves the
+ // deny status standing still writes its other content keys as usual.
+ seedNote("Notes/Marked.md", "body", "title: K\naccess: private");
+ const sideBySide = applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/Marked.md", frontmatter: { status: "final" } },
+ ]);
+ expect(sideBySide.applied).toBe(1);
+ const marked = readFileSync(join(vault, "Notes/Marked.md"), "utf8");
+ expect(marked).toContain("access: private");
+ expect(marked).toContain("status: final");
+ });
+
+ test("update_note refuses a merge that clears or forges the note's deny value", () => {
+ // Clearing: `access: private` rewritten to a non-deny value would lift
+ // the reservation without the caller ever naming the boundary.
+ seedNote("Notes/Clear.md", "body", "title: C\naccess: private");
+ try {
+ applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/Clear.md", frontmatter: { access: "public" } },
+ ]);
+ throw new Error("expected throw");
+ } catch (err) {
+ expect(err).toBeInstanceOf(WriteBatchError);
+ expect((err as WriteBatchError).code).toBe("reserved_frontmatter_key");
+ expect((err as WriteBatchError).message).toContain("access");
+ expect((err as WriteBatchError).message).toContain("edit the note directly");
+ expect((err as WriteBatchError).details).toMatchObject({
+ path: "Notes/Clear.md",
+ key: "access",
+ });
+ }
+ expect(readFileSync(join(vault, "Notes/Clear.md"), "utf8")).toContain("access: private");
+
+ // Forging: a deny value the note did not carry arrives the same way,
+ // silently, through the same merge.
+ seedNote("Notes/Forge.md", "body", "title: F\nsensitivity: internal");
+ try {
+ applyWriteBatch(vault, [
+ {
+ kind: "update_note",
+ path: "Notes/Forge.md",
+ frontmatter: { sensitivity: "confidential" },
+ },
+ ]);
+ throw new Error("expected throw");
+ } catch (err) {
+ expect(err).toBeInstanceOf(WriteBatchError);
+ expect((err as WriteBatchError).code).toBe("reserved_frontmatter_key");
+ expect((err as WriteBatchError).details).toMatchObject({
+ path: "Notes/Forge.md",
+ key: "sensitivity",
+ });
+ }
+ expect(readFileSync(join(vault, "Notes/Forge.md"), "utf8")).toContain("sensitivity: internal");
+ });
+
+ test("update_note matches reserved keys whatever their case", () => {
+ // The frontmatter parser keeps a key's spelling, so `Access:` is a
+ // different map key from `access:` - but it states the same boundary
+ // claim, and a guard reading only the lower-case form would wave it
+ // through the merge.
+ seedNote("Notes/Cased.md", "body", "title: X");
+ try {
+ applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/Cased.md", frontmatter: { Access: "private" } },
+ ]);
+ throw new Error("expected throw");
+ } catch (err) {
+ expect(err).toBeInstanceOf(WriteBatchError);
+ expect((err as WriteBatchError).code).toBe("reserved_frontmatter_key");
+ expect((err as WriteBatchError).details).toMatchObject({
+ path: "Notes/Cased.md",
+ key: "Access",
+ });
+ }
+
+ seedNote("Notes/Privately.md", "body", "title: P\nvisibility: private");
+ expect(() =>
+ applyWriteBatch(vault, [
+ { kind: "update_note", path: "Notes/Privately.md", frontmatter: { Visibility: "team" } },
+ ]),
+ ).toThrow(WriteBatchError);
+ expect(readFileSync(join(vault, "Notes/Privately.md"), "utf8")).toContain(
+ "visibility: private",
+ );
+ });
+
test("create_note refuses to clobber an existing note", () => {
seedNote("Notes/Dup.md", "original");
try {
diff --git a/tests/core/content-class-egress.test.ts b/tests/core/content-class-egress.test.ts
index 1be27440..75773ae1 100644
--- a/tests/core/content-class-egress.test.ts
+++ b/tests/core/content-class-egress.test.ts
@@ -87,6 +87,39 @@ function seedLogDay(): void {
);
}
+/**
+ * A log day with one entry of each shape: an agent-channel narrative
+ * line, a write event whose target is the left-out page, and an
+ * operator-channel entry that must ride the bundle byte-exactly. The
+ * blocks are written in the shape `renderLogEntryBlock` produces.
+ */
+function seedMixedLogDay(): void {
+ const logDir = join(vault, "Brain", "log");
+ mkdirSync(logDir, { recursive: true });
+ writeFileSync(
+ join(logDir, "2026-06-13.md"),
+ [
+ "## 12:00:00Z โ€” note",
+ "- text: agent narrative line",
+ "- agent: claude",
+ "- origin_channel: mcp-tool",
+ "",
+ "## 12:00:01Z โ€” note-write",
+ "- write_id: nw_20260613_1",
+ "- op: create",
+ "- target: Agent Page.md",
+ "- origin_channel: cli",
+ "",
+ "## 12:00:02Z โ€” note",
+ "- text: operator narrative line",
+ "- agent: operator",
+ "- origin_channel: cli",
+ "",
+ ].join("\n"),
+ "utf8",
+ );
+}
+
describe("the guard seam", () => {
const items = ["a.md", "b.md", "c.md"];
@@ -150,6 +183,119 @@ describe("okf-export applies the default and the override", () => {
);
expect(r.stderr).toContain(EGRESS_CONTENT_CLASS_INCLUDE_NOTICE);
});
+
+ test("a marked ingested source page is left out of the references lane and named", async () => {
+ await bootstrap();
+ const sourcesDir = join(vault, "Brain", "sources");
+ mkdirSync(sourcesDir, { recursive: true });
+ writeFileSync(
+ join(sourcesDir, "src-stamped.md"),
+ `---\ntitle: Ingested Source\nkind: brain-source\n${CONTENT_CLASS_FIELD}: ${CONTENT_CLASS.promptDerived}\n---\n\nAgent-written summary prose.\n`,
+ "utf8",
+ );
+ const out = join(tmp, "bundle-sources");
+ const r = await runCli(["brain", "okf-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ expect(existsSync(join(out, "references", "src-stamped.md"))).toBe(false);
+ expect(r.stderr).toContain("Brain/sources/src-stamped.md (content_class: prompt-derived)");
+ });
+});
+
+describe("okf-export applies the class policy to the change log", () => {
+ test("agent-channel entries and entries naming a left-out page leave log.md, counted", async () => {
+ await bootstrap();
+ seedMixedLogDay();
+ const out = join(tmp, "bundle-log");
+ const r = await runCli(["brain", "okf-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ const log = readFileSync(join(out, "log.md"), "utf8");
+ expect(log).not.toContain("agent narrative line");
+ expect(log).not.toContain("Agent Page.md");
+ expect(log).not.toContain("12:00:00Z");
+ expect(log).not.toContain("12:00:01Z");
+ // The kept entry rides byte-exactly: no reformatting of what stays.
+ expect(log).toContain(
+ "## 12:00:02Z โ€” note\n- text: operator narrative line\n- agent: operator\n- origin_channel: cli",
+ );
+ expect(r.stderr).toContain("2 log line(s) left out by the same policy");
+ expect(r.stderr).toContain(contentClassOmissionNotice([MARKED]));
+ });
+
+ test("a day whose every entry leaves stops counting as a log day", async () => {
+ await bootstrap();
+ const logDir = join(vault, "Brain", "log");
+ mkdirSync(logDir, { recursive: true });
+ writeFileSync(
+ join(logDir, "2026-06-14.md"),
+ "## 09:00:00Z โ€” note\n- text: only agent words\n- agent: claude\n- origin_channel: mcp-tool\n",
+ "utf8",
+ );
+ const out = join(tmp, "bundle-log-empty");
+ const r = await runCli(["brain", "okf-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ expect(readFileSync(join(out, "log.md"), "utf8")).not.toContain("2026-06-14");
+ expect(readFileSync(join(out, "log.md"), "utf8")).not.toContain("only agent words");
+ expect(r.stdout).toContain("0 log day(s)");
+ });
+
+ test("the override keeps the log whole and names no log omission", async () => {
+ await bootstrap();
+ seedMixedLogDay();
+ const out = join(tmp, "bundle-log-override");
+ const r = await runCli(["brain", "okf-export", "--out", out, "--include-prompt-derived"], {
+ env: env(),
+ });
+ expect(r.returncode).toBe(0);
+ const log = readFileSync(join(out, "log.md"), "utf8");
+ expect(log).toContain("agent narrative line");
+ expect(log).toContain("target: Agent Page.md");
+ expect(log).toContain("operator narrative line");
+ expect(r.stderr).toContain(EGRESS_CONTENT_CLASS_INCLUDE_NOTICE);
+ expect(r.stderr).not.toContain("log line(s) left out");
+ });
+});
+
+describe("okf-export drops kept pages' manifest citations to left-out pages", () => {
+ test("the citation leaves the manifest, counted; the page prose is untouched", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ '---\ntitle: Operator Page\nrelated: "[[Agent Page]]"\n---\n\nSee [[Agent Page]] and [[https://example.org/one]].\n',
+ "utf8",
+ );
+ const out = join(tmp, "bundle-citations");
+ const r = await runCli(["brain", "okf-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ const manifest = JSON.parse(readFileSync(join(out, "okf.json"), "utf8")) as {
+ pages: ReadonlyArray<{ path: string; citations: ReadonlyArray<string> }>;
+ };
+ const operator = manifest.pages.find((p) => p.path === "Operator Page.md");
+ expect(operator?.citations).toEqual(["https://example.org/one"]);
+ // The page body is not prose-edited: the link the operator wrote stays.
+ const page = readFileSync(join(out, "concepts", "Operator Page.md"), "utf8");
+ expect(page).toContain("See [[Agent Page]] and [[https://example.org/one]].");
+ expect(r.stderr).toContain("1 citation(s) to left-out page(s) dropped");
+ });
+
+ test("the override keeps the citations", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ "---\ntitle: Operator Page\n---\n\nSee [[Agent Page]].\n",
+ "utf8",
+ );
+ const out = join(tmp, "bundle-citations-override");
+ const r = await runCli(["brain", "okf-export", "--out", out, "--include-prompt-derived"], {
+ env: env(),
+ });
+ expect(r.returncode).toBe(0);
+ const manifest = JSON.parse(readFileSync(join(out, "okf.json"), "utf8")) as {
+ pages: ReadonlyArray<{ path: string; citations: ReadonlyArray<string> }>;
+ };
+ const operator = manifest.pages.find((p) => p.path === "Operator Page.md");
+ expect(operator?.citations).toEqual(["Agent Page"]);
+ expect(r.stderr).not.toContain("citation(s) to left-out page(s) dropped");
+ });
});
describe("okf-export writes the reserved bundle files", () => {
@@ -230,6 +376,61 @@ describe("graph-export applies the default and the override", () => {
});
});
+describe("graph-export drops kept nodes' references to left-out pages", () => {
+ test("links and relation targets naming a left-out page leave the node, counted", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ '---\ntitle: Operator Page\nrelated: "[[Agent Page]]"\n---\n\nSee [[Agent Page]] and [[Operator Journal]].\n',
+ "utf8",
+ );
+ writeFileSync(
+ join(vault, "Operator Journal.md"),
+ "---\ntitle: Operator Journal\n---\n\nOperator words.\n",
+ "utf8",
+ );
+ const out = join(tmp, "graph-refs.json");
+ const r = await runCli(["brain", "graph-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ const graph = JSON.parse(readFileSync(out, "utf8")) as {
+ nodes: ReadonlyArray<{
+ path: string;
+ links: ReadonlyArray<string>;
+ relations: Readonly<Record<string, ReadonlyArray<string>>>;
+ }>;
+ };
+ const operator = graph.nodes.find((n) => n.path === "Operator Page.md");
+ expect(operator?.links).toEqual(["Operator Journal"]);
+ // The relation left with no targets drops with them, matching the
+ // exporter's own shape of carrying only non-empty relation fields.
+ expect(operator?.relations).toEqual({});
+ expect(r.stderr).toContain(contentClassOmissionNotice([MARKED]));
+ expect(r.stderr).toContain(
+ "2 link/relation target(s) to left-out page(s) dropped from kept nodes",
+ );
+ });
+
+ test("the override keeps every target", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ "---\ntitle: Operator Page\n---\n\nSee [[Agent Page]].\n",
+ "utf8",
+ );
+ const out = join(tmp, "graph-refs-override.json");
+ const r = await runCli(["brain", "graph-export", "--out", out, "--include-prompt-derived"], {
+ env: env(),
+ });
+ expect(r.returncode).toBe(0);
+ const graph = JSON.parse(readFileSync(out, "utf8")) as {
+ nodes: ReadonlyArray<{ path: string; links: ReadonlyArray<string> }>;
+ };
+ const operator = graph.nodes.find((n) => n.path === "Operator Page.md");
+ expect(operator?.links).toEqual(["Agent Page"]);
+ expect(r.stderr).not.toContain("link/relation target(s) to left-out page(s) dropped");
+ });
+});
+
describe("bank-export applies the default and the override", () => {
test("a marked page leaves both page sections, named once", async () => {
await bootstrap();
@@ -263,6 +464,60 @@ describe("bank-export applies the default and the override", () => {
});
});
+describe("bank-export drops kept pages' structured references to left-out pages", () => {
+ test("graph links, relation targets and contract citations leave, counted", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ '---\ntitle: Operator Page\nrelated: "[[Agent Page]]"\n---\n\nSee [[Agent Page]].\n',
+ "utf8",
+ );
+ const out = join(tmp, "bank-refs.json");
+ const r = await runCli(["brain", "bank-export", "--out", out], { env: env() });
+ expect(r.returncode).toBe(0);
+ const bundle = JSON.parse(readFileSync(out, "utf8")) as {
+ graph: {
+ nodes: ReadonlyArray<{
+ links: ReadonlyArray<string>;
+ relations: Readonly<Record<string, ReadonlyArray<string>>>;
+ }>;
+ };
+ pages: ReadonlyArray<{ citations: ReadonlyArray<string> }>;
+ };
+ expect(bundle.graph.nodes[0]?.links).toEqual([]);
+ expect(bundle.graph.nodes[0]?.relations).toEqual({});
+ expect(bundle.pages[0]?.citations).toEqual([]);
+ // One body link, one relation target on the graph node, and one
+ // de-duplicated citation on the contract.
+ expect(r.stderr).toContain(
+ "3 link/relation/citation target(s) to left-out page(s) dropped from kept pages' structured references",
+ );
+ });
+
+ test("the override keeps every structured reference", async () => {
+ await bootstrap();
+ writeFileSync(
+ join(vault, "Operator Page.md"),
+ "---\ntitle: Operator Page\n---\n\nSee [[Agent Page]].\n",
+ "utf8",
+ );
+ const out = join(tmp, "bank-refs-override.json");
+ const r = await runCli(["brain", "bank-export", "--out", out, "--include-prompt-derived"], {
+ env: env(),
+ });
+ expect(r.returncode).toBe(0);
+ const bundle = JSON.parse(readFileSync(out, "utf8")) as {
+ graph: { nodes: ReadonlyArray<{ path: string; links: ReadonlyArray<string> }> };
+ pages: ReadonlyArray<{ path: string; citations: ReadonlyArray<string> }>;
+ };
+ const operatorNode = bundle.graph.nodes.find((n) => n.path === "Operator Page.md");
+ expect(operatorNode?.links).toEqual(["Agent Page"]);
+ const operatorPage = bundle.pages.find((p) => p.path === "Operator Page.md");
+ expect(operatorPage?.citations).toEqual(["Agent Page"]);
+ expect(r.stderr).not.toContain("dropped from kept pages' structured references");
+ });
+});
+
/**
* A stand-in corpus stream: the real one reads runtime session
* directories; the loop under test only consumes records and a summary.
diff --git a/tests/core/content-class-knowledge-pack.test.ts b/tests/core/content-class-knowledge-pack.test.ts
index 95215031..cfcdafd6 100644
--- a/tests/core/content-class-knowledge-pack.test.ts
+++ b/tests/core/content-class-knowledge-pack.test.ts
@@ -25,7 +25,10 @@ import {
CONTENT_CLASS_FIELD,
stampPromptDerivedLine,
} from "../../src/core/brain/provenance/content-class.ts";
-import { selectKnowledgePack } from "../../src/core/brain/portability/knowledge-pack.ts";
+import {
+ BLOCKED_PAGE_CITATION_WARNING,
+ selectKnowledgePack,
+} from "../../src/core/brain/portability/knowledge-pack.ts";
import { brainDirs } from "../../src/core/brain/paths.ts";
let src: string;
@@ -102,6 +105,68 @@ describe("the knowledge-pack block list", () => {
});
});
+describe("kept pages' citations to blocked pages", () => {
+ test("citations naming a blocked page leave the manifest, named; the prose is untouched", () => {
+ page(
+ "Notes/handbook.md",
+ ["tags: [ops]"],
+ "See [[agent-draft]] and [[scoped]] plus [[https://example.org/guide]].\n",
+ );
+ page(
+ "Notes/agent-draft.md",
+ ["tags: [ops]", `${CONTENT_CLASS_FIELD}: ${CONTENT_CLASS.promptDerived}`],
+ "Synthesised from a prompt.\n",
+ );
+ page("Notes/scoped.md", ["tags: [ops]", "visibility: private"], "Operator-scoped.\n");
+
+ const selected = selectKnowledgePack(src, ["tag:ops"]);
+ expect(selected.blocked).toEqual([
+ { kind: "page", id: "Notes/agent-draft.md", reason: "prompt-derived" },
+ { kind: "page", id: "Notes/scoped.md", reason: "visibility" },
+ ]);
+ // The carried page's manifest metadata names neither blocked page;
+ // the URL target never names a vault page and rides.
+ const handbook = selected.okfManifest.pages.find((p) => p.path === "Notes/handbook.md");
+ expect(handbook?.citations).toEqual(["https://example.org/guide"]);
+ expect(handbook?.sources).toEqual([{ resource: "https://example.org/guide" }]);
+ // The page body is not prose-edited: the links the operator wrote stay.
+ const packed = selected.pageFiles.find((f) => f.path.endsWith("handbook.md"))!;
+ expect(packed.contents).toContain(
+ "See [[agent-draft]] and [[scoped]] plus [[https://example.org/guide]].",
+ );
+ // The drops are named on the carried page, in the report the export
+ // already renders entry by entry.
+ expect(selected.warnings).toEqual([
+ { kind: "page", id: "Notes/handbook.md", reasons: [BLOCKED_PAGE_CITATION_WARNING] },
+ ]);
+ });
+
+ test("the override carries the prompt-derived page and its citation back", () => {
+ page("Notes/handbook.md", ["tags: [ops]"], "See [[agent-draft]] and [[scoped]].\n");
+ page(
+ "Notes/agent-draft.md",
+ ["tags: [ops]", `${CONTENT_CLASS_FIELD}: ${CONTENT_CLASS.promptDerived}`],
+ "Synthesised from a prompt.\n",
+ );
+ page("Notes/scoped.md", ["tags: [ops]", "visibility: private"], "Operator-scoped.\n");
+
+ const selected = selectKnowledgePack(src, ["tag:ops"], { includePromptDerived: true });
+ // Only the operator-set scope still blocks: the class block is the
+ // one the override carries back.
+ expect(selected.blocked).toEqual([
+ { kind: "page", id: "Notes/scoped.md", reason: "visibility" },
+ ]);
+ expect(selected.pageFiles.some((f) => f.path.endsWith("agent-draft.md"))).toBe(true);
+ const handbook = selected.okfManifest.pages.find((p) => p.path === "Notes/handbook.md");
+ expect(handbook?.citations).toEqual(["agent-draft"]);
+ // The visibility-scoped page stays blocked under the override, so its
+ // citation is still a drop and still named.
+ expect(selected.warnings).toEqual([
+ { kind: "page", id: "Notes/handbook.md", reasons: [BLOCKED_PAGE_CITATION_WARNING] },
+ ]);
+ });
+});
+
describe("the no-backfill era boundary", () => {
test("a pre-stamp page exports unchanged; absence of the mark is the documented era", () => {
page("Notes/runbook.md", ["tags: [ops]"], "Written before the class shipped.\n");
diff --git a/tests/core/decision-model/eval-mark.test.ts b/tests/core/decision-model/eval-mark.test.ts
index 4515ec6a..3c161e5a 100644
--- a/tests/core/decision-model/eval-mark.test.ts
+++ b/tests/core/decision-model/eval-mark.test.ts
@@ -92,11 +92,11 @@ function evalConfig(profile: string | null = "laya"): ResolvedDecisionModelConfi
errors: [],
notes: [],
enabled: true,
- provider: "compatible",
+ provider: "laya",
adapter: "systemone",
- baseUrl: "http://127.0.0.1:9",
- model: "fake-model-1",
- envKey: "FAKE_DECISION_KEY_UNUSED",
+ baseUrl: "http://127.0.0.1:8000",
+ model: "english",
+ envKey: null,
keyPresent: true,
allowInsecureHttp: false,
timeoutMs: 3000,
@@ -155,15 +155,15 @@ describe("the machine-side mark file", () => {
baseline_error: 0.5,
verdict: "promote",
created_at: "2026-10-20T00:00:00.000Z",
- provider: "compatible",
- model: "fake-model-1",
- base_url_hash: sha256Hex("http://127.0.0.1:9"),
+ provider: "laya",
+ model: "english",
+ base_url_hash: sha256Hex("http://127.0.0.1:8000"),
});
const mark = readTuningMark(path);
expect(mark?.profile).toBe("laya");
expect(mark?.verdict).toBe("promote");
- expect(mark?.model).toBe("fake-model-1");
- expect(mark?.base_url_hash).toBe(sha256Hex("http://127.0.0.1:9"));
+ expect(mark?.model).toBe("english");
+ expect(mark?.base_url_hash).toBe(sha256Hex("http://127.0.0.1:8000"));
const broken = join(tmp, "broken.json");
writeFileSync(broken, "{not json", "utf8");
expect(readTuningMark(broken)).toBeNull();
@@ -183,9 +183,9 @@ describe("the machine-side mark file", () => {
counted_error: 0,
baseline_error: 0.5,
verdict: "promote" as const,
- provider: "compatible",
- model: "fake-model-1",
- base_url_hash: sha256Hex("http://127.0.0.1:9"),
+ provider: "laya",
+ model: "english",
+ base_url_hash: sha256Hex("http://127.0.0.1:8000"),
};
writeTuningMark(path, { ...base, created_at: "2026-10-20T00:00:00.000Z" });
expect(readTuningMark(path)?.created_at).toBe("2026-10-20T00:00:00.000Z");
@@ -216,9 +216,9 @@ describe("retractTuningMark", () => {
baseline_error: 0.5,
verdict: "promote",
created_at: "2026-10-20T00:00:00.000Z",
- provider: "compatible",
- model: "fake-model-1",
- base_url_hash: sha256Hex("http://127.0.0.1:9"),
+ provider: "laya",
+ model: "english",
+ base_url_hash: sha256Hex("http://127.0.0.1:8000"),
});
}
@@ -369,10 +369,10 @@ describe("the eval runner writes the mark only on a promote", () => {
const raw = JSON.parse(
readFileSync(decisionModelTuningMarkPath(stateEnv, "laya", "rerank"), "utf8"),
) as Record<string, unknown>;
- expect(raw["provider"]).toBe("compatible");
- expect(raw["model"]).toBe("fake-model-1");
+ expect(raw["provider"]).toBe("laya");
+ expect(raw["model"]).toBe("english");
// A hash, never the URL itself.
- expect(raw["base_url_hash"]).toBe(sha256Hex("http://127.0.0.1:9"));
+ expect(raw["base_url_hash"]).toBe(sha256Hex("http://127.0.0.1:8000"));
expect(JSON.stringify(raw)).not.toContain("127.0.0.1");
});
@@ -415,28 +415,25 @@ describe("the eval runner writes the mark only on a promote", () => {
describe("the measured gate", () => {
/** The endpoint `evalConfig` measures on - what a mark gets bound to. */
const ENDPOINT = {
- provider: "compatible",
- model: "fake-model-1",
- baseUrl: "http://127.0.0.1:9",
+ provider: "laya",
+ model: "english",
+ baseUrl: "http://127.0.0.1:8000",
};
- const ENV = { O2B_TEST_EVAL_MARK_KEY: "unused" };
- // Resolves to exactly the endpoint the measurement ran on (and a key,
- // since the compatible provider always needs one).
+ // Resolves to exactly the endpoint the measurement ran on: the laya
+ // preset, whose threshold profile the preset itself names (a
+ // non-baseline family is never registered in the source table).
const CONFIG = {
decision_model_enabled: "true",
- decision_model_provider: "compatible",
- decision_model_base_url: "http://127.0.0.1:9",
- decision_model_id: "fake-model-1",
- decision_model_env_key: "O2B_TEST_EVAL_MARK_KEY",
- decision_model_threshold_profile: "laya",
+ decision_model_provider: "laya",
decision_model_uses: "rerank:enforce",
};
test("with no mark the use stays shadow exactly as today", () => {
const cfg = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: CONFIG,
vault,
+ thresholdsPath: null,
measuredTunedMark: machineTunedMarkReader(ENDPOINT, stateEnv),
});
expect(cfg.status).toBe("active");
@@ -453,9 +450,10 @@ describe("the measured gate", () => {
});
const reader = machineTunedMarkReader(ENDPOINT, stateEnv);
const cfg = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: CONFIG,
vault,
+ thresholdsPath: null,
measuredTunedMark: reader,
});
expect(cfg.shadowOnlyUses).toEqual([]);
@@ -464,7 +462,7 @@ describe("the measured gate", () => {
// Only the marked use is measured: an untuned use without a mark
// stays shadow.
const both = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: { ...CONFIG, decision_model_uses: "rerank:enforce,dedup:enforce" },
vault,
measuredTunedMark: reader,
@@ -487,7 +485,7 @@ describe("the measured gate", () => {
process.env["XDG_STATE_HOME"] = stateEnv.env["XDG_STATE_HOME"]!;
try {
const resolve = (config: Record<string, string>) =>
- resolveDecisionModelConfig({ env: ENV, config, vault });
+ resolveDecisionModelConfig({ env: {}, config, vault, thresholdsPath: null });
expect(resolve(CONFIG).uses["rerank"]).toBe("enforce");
// A model that was never measured runs shadow.
@@ -499,7 +497,9 @@ describe("the measured gate", () => {
resolve({ ...CONFIG, decision_model_base_url: "http://127.0.0.1:10" }).uses["rerank"],
).toBe("shadow");
// And another provider id is a different decision surface outright.
- expect(resolve({ ...CONFIG, decision_model_provider: "laya" }).uses["rerank"]).toBe("shadow");
+ expect(resolve({ ...CONFIG, decision_model_provider: "openjev" }).uses["rerank"]).toBe(
+ "shadow",
+ );
// Switching back to the measured endpoint re-arms enforce.
expect(resolve(CONFIG).uses["rerank"]).toBe("enforce");
@@ -526,9 +526,10 @@ describe("the measured gate", () => {
resetTuningMarkCache();
expect(validatedTuningMark("laya", "rerank", stateEnv)).toBeNull();
const cfg = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: CONFIG,
vault,
+ thresholdsPath: null,
measuredTunedMark: machineTunedMarkReader(ENDPOINT, stateEnv),
});
expect(cfg.uses["rerank"]).toBe("shadow");
@@ -544,7 +545,13 @@ describe("the measured gate", () => {
});
const reader = machineTunedMarkReader(ENDPOINT, stateEnv);
const gated = () =>
- resolveDecisionModelConfig({ env: ENV, config: CONFIG, vault, measuredTunedMark: reader });
+ resolveDecisionModelConfig({
+ env: {},
+ config: CONFIG,
+ vault,
+ measuredTunedMark: reader,
+ thresholdsPath: null,
+ });
// Run 1 promotes: the gate arms enforce.
const first = await run(new FakeDecisionProvider({ answer: byLabelRule }));
@@ -587,9 +594,10 @@ describe("the measured gate", () => {
writeFileSync(path, JSON.stringify({ ...valid, spec_hash: "0".repeat(64) }), "utf8");
resetTuningMarkCache();
const cfg = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: CONFIG,
vault,
+ thresholdsPath: null,
measuredTunedMark: machineTunedMarkReader(ENDPOINT, stateEnv),
});
expect(cfg.shadowOnlyUses).toEqual(["rerank"]);
@@ -600,7 +608,7 @@ describe("the measured gate", () => {
const previous = process.env["XDG_STATE_HOME"];
process.env["XDG_STATE_HOME"] = stateEnv.env["XDG_STATE_HOME"]!;
try {
- const before = resolveDecisionModelConfig({ env: ENV, config: CONFIG, vault });
+ const before = resolveDecisionModelConfig({ env: {}, config: CONFIG, vault });
expect(before.shadowOnlyUses).toEqual(["rerank"]);
await runDecisionModelEval("rerank", {
config: evalConfig(),
@@ -608,12 +616,12 @@ describe("the measured gate", () => {
markSource: stateEnv,
});
resetTuningMarkCache();
- const after = resolveDecisionModelConfig({ env: ENV, config: CONFIG, vault });
+ const after = resolveDecisionModelConfig({ env: {}, config: CONFIG, vault });
expect(after.shadowOnlyUses).toEqual([]);
expect(after.uses["rerank"]).toBe("enforce");
// An explicit null reader is today's hand-edit behavior.
const disabled = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: CONFIG,
vault,
measuredTunedMark: null,
@@ -632,7 +640,7 @@ describe("the measured gate", () => {
markSource: stateEnv,
});
const cfg = resolveDecisionModelConfig({
- env: ENV,
+ env: {},
config: {
decision_model_enabled: "true",
decision_model_provider: "llm-emulation",
diff --git a/tests/core/decision-model/eval-surfaces.test.ts b/tests/core/decision-model/eval-surfaces.test.ts
index eee4845e..63458e92 100644
--- a/tests/core/decision-model/eval-surfaces.test.ts
+++ b/tests/core/decision-model/eval-surfaces.test.ts
@@ -85,11 +85,11 @@ function evalConfig(): ResolvedDecisionModelConfig {
errors: [],
notes: [],
enabled: true,
- provider: "compatible",
+ provider: "laya",
adapter: "systemone",
- baseUrl: "http://127.0.0.1:9",
- model: "fake-model-1",
- envKey: "FAKE_DECISION_KEY_UNUSED",
+ baseUrl: "http://127.0.0.1:8000",
+ model: "english",
+ envKey: null,
keyPresent: true,
allowInsecureHttp: false,
timeoutMs: 3000,
@@ -110,15 +110,17 @@ function evalConfig(): ResolvedDecisionModelConfig {
const CHECK_CONFIG = {
decision_model_enabled: "true",
- decision_model_provider: "compatible",
- decision_model_base_url: "http://127.0.0.1:9",
- decision_model_id: "fake-model-1",
- decision_model_threshold_profile: "laya",
+ decision_model_provider: "laya",
decision_model_uses: "rerank:shadow",
} as const;
function checkWith(markSource: PlatformDirsEnv | null) {
- return buildDecisionModelCheck({ config: { ...CHECK_CONFIG }, vault, markSource });
+ return buildDecisionModelCheck({
+ config: { ...CHECK_CONFIG },
+ vault,
+ markSource,
+ thresholdsPath: null,
+ });
}
describe("the check's measured tuning state", () => {
@@ -163,6 +165,7 @@ describe("the check's measured tuning state", () => {
config: { ...CHECK_CONFIG, decision_model_id: "fake-model-2" },
vault,
markSource: stateEnv,
+ thresholdsPath: null,
});
expect(elsewhere.tuning.state).toBe("measured_elsewhere");
expect(elsewhere.tuning.marked_uses).toEqual([]);
@@ -177,6 +180,7 @@ describe("the check's measured tuning state", () => {
config: { ...CHECK_CONFIG, decision_model_base_url: "http://127.0.0.1:10" },
vault,
markSource: stateEnv,
+ thresholdsPath: null,
});
expect(movedUrl.tuning.state).toBe("measured_elsewhere");
expect(renderDecisionModelCheck(movedUrl)).toContain("on another endpoint");
diff --git a/tests/core/decision-model/presets-profiles.test.ts b/tests/core/decision-model/presets-profiles.test.ts
index 6860a036..02c32851 100644
--- a/tests/core/decision-model/presets-profiles.test.ts
+++ b/tests/core/decision-model/presets-profiles.test.ts
@@ -206,6 +206,7 @@ describe("threshold profiles", () => {
env: {},
config: preset("laya", { decision_model_uses: "rerank:enforce,skills:shadow" }),
vault: null,
+ thresholdsPath: null,
});
expect(cfg.status).toBe("active");
expect(cfg.errors).toEqual([]);
@@ -249,6 +250,7 @@ describe("threshold profiles", () => {
decision_model_threshold_profile: "jev-1.13",
}),
vault: null,
+ thresholdsPath: null,
});
expect(cfg.thresholdProfile).toBe("laya");
expect(decisionModelModeFor(cfg, "rerank")).toBe("shadow");
diff --git a/tests/core/decision-model/profiles-all-uses.test.ts b/tests/core/decision-model/profiles-all-uses.test.ts
index adecd5a5..f7c1d624 100644
--- a/tests/core/decision-model/profiles-all-uses.test.ts
+++ b/tests/core/decision-model/profiles-all-uses.test.ts
@@ -64,7 +64,12 @@ const ENV = { [KEY_VAR]: FAKE_DECISION_KEY };
describe("enforce on an untuned threshold profile runs as shadow for every use", () => {
for (const profile of Object.keys(PROFILES)) {
test(profile, async () => {
- const cfg = resolveDecisionModelConfig({ env: ENV, config: config(profile), vault });
+ const cfg = resolveDecisionModelConfig({
+ env: ENV,
+ config: config(profile),
+ vault,
+ thresholdsPath: null,
+ });
expect(cfg.status).toBe("active");
expect(cfg.thresholdProfile).toBe(profile);
expect(cfg.shadowOnlyUses).toEqual([...DECISION_MODEL_USES]);
diff --git a/tests/core/decision-model/providers-chain.test.ts b/tests/core/decision-model/providers-chain.test.ts
new file mode 100644
index 00000000..1b115a84
--- /dev/null
+++ b/tests/core/decision-model/providers-chain.test.ts
@@ -0,0 +1,407 @@
+/**
+ * The ordered provider chain (`decision_model_providers`, issue #264):
+ * resolution and validation of the list, the fallback in `runDecision`
+ * (a dead or hanging first entry falls through within the one budget),
+ * the per-attempt records, and the check's order and reachability lines.
+ *
+ * Every request lands on a loopback fake (`startFakeSystemOne`); a dead
+ * entry points at a port on 127.0.0.1 that refuses connections, so no
+ * test waits on a real network.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ decisionModelModeFor,
+ resolveDecisionModelConfig,
+} from "../../../src/core/decision-model/config.ts";
+import {
+ buildDecisionModelCheck,
+ decisionModelCheckExitCode,
+ renderDecisionModelCheck,
+} from "../../../src/core/decision-model/diagnostics.ts";
+import { runDecision } from "../../../src/core/decision-model/run.ts";
+import { listDecisionModelCalls } from "../../../src/core/decision-model/record.ts";
+import { FAKE_DECISION_KEY, fakeCredential } from "../../helpers/fake-credentials.ts";
+import { startFakeSystemOne } from "../../helpers/fake-decision-provider.ts";
+
+const KEY_VAR = "O2B_TEST_CHAIN_LOCAL_KEY";
+const FALLBACK_KEY_VAR = "O2B_TEST_CHAIN_FALLBACK_KEY";
+
+const dirs: string[] = [];
+afterEach(() => {
+ for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true });
+});
+function tempVault(): string {
+ const v = mkdtempSync(join(tmpdir(), "osb-dm-chain-"));
+ dirs.push(v);
+ return v;
+}
+
+/** A chain config: a compatible server first, a hosted preset second. */
+function chainConfig(extra: Record<string, string> = {}): Record<string, string> {
+ return {
+ decision_model_enabled: "true",
+ decision_model_providers: "compatible,typesafe",
+ decision_model_base_url: "http://127.0.0.1:8080",
+ decision_model_id: "local-model-1",
+ decision_model_env_key: KEY_VAR,
+ decision_model_uses: "rerank:enforce",
+ ...extra,
+ };
+}
+
+const CHAIN_ENV = {
+ [KEY_VAR]: FAKE_DECISION_KEY,
+ TYPESAFE_API_KEY: fakeCredential("O2B_TEST_CHAIN"),
+} as NodeJS.ProcessEnv;
+
+/** A state/questions pair small enough for every fake server. */
+function build() {
+ return {
+ kind: "ok" as const,
+ state: { query: "q", passages: { P0: "text" } },
+ candidateCount: 1,
+ context: null,
+ };
+}
+const QUESTIONS = () => ({ rel_0: { type: "noul" as const, instructions: "relevant?" } });
+
+describe("decision_model_providers resolution", () => {
+ test("the list resolves in order; the first entry reads the global endpoint keys", () => {
+ const cfg = resolveDecisionModelConfig({ env: CHAIN_ENV, config: chainConfig(), vault: null });
+ expect(cfg.status).toBe("active");
+ expect(cfg.provider).toBe("compatible");
+ expect(cfg.baseUrl).toBe("http://127.0.0.1:8080");
+ expect(cfg.model).toBe("local-model-1");
+ const names = cfg.providers?.map((p) => p.name);
+ expect(names).toEqual(["compatible", "typesafe"]);
+ const second = cfg.providers?.[1];
+ expect(second?.baseUrl).toBe("https://api.typesafe.ai");
+ expect(second?.model).toBe("jev-1.13.0");
+ expect(second?.envKey).toBe("TYPESAFE_API_KEY");
+ expect(second?.keyPresent).toBe(true);
+ });
+
+ test("a single decision_model_provider resolves as a one-entry chain", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: {
+ decision_model_enabled: "true",
+ decision_model_provider: "typesafe",
+ decision_model_uses: "rerank:shadow",
+ },
+ vault: null,
+ });
+ expect(cfg.status).toBe("active");
+ expect(cfg.providers?.map((p) => p.name)).toEqual(["typesafe"]);
+ expect(cfg.providers?.[0]?.baseUrl).toBe("https://api.typesafe.ai");
+ });
+
+ test("the list wins over decision_model_provider, with a note naming both", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({ decision_model_provider: "openrouter" }),
+ vault: null,
+ });
+ expect(cfg.providers?.map((p) => p.name)).toEqual(["compatible", "typesafe"]);
+ expect(cfg.notes.join()).toContain("decision_model_provider");
+ expect(cfg.notes.join()).toContain("decision_model_providers");
+ });
+
+ test("an unknown preset and a duplicate are errors naming the new key", () => {
+ const unknown = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({ decision_model_providers: "compatible,somewhere" }),
+ vault: null,
+ });
+ expect(unknown.status).toBe("invalid");
+ expect(unknown.errors.join()).toContain("decision_model_providers");
+ expect(unknown.errors.join()).toContain("'somewhere'");
+ const duplicate = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({ decision_model_providers: "typesafe,typesafe" }),
+ vault: null,
+ });
+ expect(duplicate.status).toBe("invalid");
+ expect(duplicate.errors.join()).toContain("more than once");
+ });
+
+ test("an empty list is an error naming the key", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({ decision_model_providers: " , " }),
+ vault: null,
+ });
+ expect(cfg.status).toBe("invalid");
+ expect(cfg.errors.join()).toContain("decision_model_providers");
+ });
+
+ test("a later entry without any endpoint is an error naming its per-entry key", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({ decision_model_providers: "typesafe,compatible" }),
+ vault: null,
+ });
+ expect(cfg.status).toBe("invalid");
+ expect(cfg.errors.join()).toContain("decision_model_providers_compatible_base_url");
+ });
+
+ test("per-entry endpoint keys configure a later slot", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({
+ decision_model_providers: "typesafe,compatible",
+ decision_model_providers_compatible_base_url: "http://127.0.0.1:9",
+ decision_model_providers_compatible_id: "second-local-model",
+ decision_model_providers_compatible_env_key: FALLBACK_KEY_VAR,
+ }),
+ vault: null,
+ });
+ expect(cfg.status).toBe("active");
+ const second = cfg.providers?.[1];
+ expect(second?.baseUrl).toBe("http://127.0.0.1:9");
+ expect(second?.model).toBe("second-local-model");
+ expect(second?.envKey).toBe(FALLBACK_KEY_VAR);
+ });
+
+ test("no_key only when no entry is configured; a keyed later entry keeps the feature on", () => {
+ const none = resolveDecisionModelConfig({
+ env: {},
+ config: chainConfig(),
+ vault: null,
+ });
+ expect(none.status).toBe("no_key");
+ const secondHasKey = resolveDecisionModelConfig({
+ env: { TYPESAFE_API_KEY: fakeCredential("O2B_TEST_CHAIN") } as NodeJS.ProcessEnv,
+ config: chainConfig(),
+ vault: null,
+ });
+ expect(secondHasKey.status).toBe("active");
+ expect(secondHasKey.notes.join()).toContain("'compatible'");
+ expect(secondHasKey.notes.join()).toContain(KEY_VAR);
+ expect(secondHasKey.providers?.[0]?.keyPresent).toBe(false);
+ });
+
+ test("enforce is held per entry: the baseline fallback enforces where an unknown model cannot", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig(),
+ vault: null,
+ thresholdsPath: null,
+ });
+ expect(cfg.status).toBe("active");
+ const first = cfg.providers?.[0];
+ const second = cfg.providers?.[1];
+ // compatible has no threshold profile and no stored thresholds, so
+ // its rerank enforce holds as shadow; the hosted fallback is the
+ // baseline family and keeps enforce.
+ expect(first?.uses.rerank).toBe("shadow");
+ expect(first?.shadowOnlyUses).toContain("rerank");
+ expect(second?.uses.rerank).toBe("enforce");
+ expect(second?.shadowOnlyUses).toEqual([]);
+ // The top level mirrors the first entry.
+ expect(cfg.uses.rerank).toBe("shadow");
+ expect(decisionModelModeFor(cfg, "rerank")).toBe("shadow");
+ });
+});
+
+describe("chain fallback in runDecision", () => {
+ test("a dead first entry falls through; records name each provider and the answering model", async () => {
+ const vault = tempVault();
+ const server = await startFakeSystemOne();
+ try {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({
+ decision_model_base_url: "http://127.0.0.1:1",
+ decision_model_providers_typesafe_base_url: server.url,
+ decision_model_providers_typesafe_id: "fake-model-1.0",
+ decision_model_providers_typesafe_env_key: "TYPESAFE_API_KEY",
+ decision_model_uses: "rerank:shadow",
+ }),
+ vault,
+ });
+ expect(cfg.status).toBe("active");
+ const result = await runDecision("rerank", build, QUESTIONS, { config: cfg, env: CHAIN_ENV });
+ expect(result.status).toBe("ok");
+ const records = listDecisionModelCalls(vault);
+ expect(records).toHaveLength(2);
+ expect(records[0]?.payload).toMatchObject({
+ provider: "compatible",
+ outcome: "network",
+ });
+ expect(records[1]?.payload).toMatchObject({
+ provider: "typesafe",
+ model: "fake-model-1.0",
+ outcome: "ok",
+ });
+ } finally {
+ await server.close();
+ }
+ });
+
+ test("a hanging first entry cannot stretch the call past the budget", async () => {
+ const vault = tempVault();
+ const hang = await startFakeSystemOne();
+ const fast = await startFakeSystemOne();
+ hang.setReply(() => ({ hang: true }));
+ try {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({
+ decision_model_base_url: hang.url,
+ decision_model_providers_typesafe_base_url: fast.url,
+ decision_model_providers_typesafe_id: "fake-model-1.0",
+ decision_model_uses: "rerank:shadow",
+ decision_model_timeout_ms: "500",
+ }),
+ vault,
+ });
+ const started = Date.now();
+ const result = await runDecision("rerank", build, QUESTIONS, { config: cfg, env: CHAIN_ENV });
+ const elapsed = Date.now() - started;
+ expect(result.status).toBe("ok");
+ // The whole chain shares the one budget: two entries, 500 ms, so
+ // the first attempt is cut at about half and the second answers.
+ // Anything near or above the full budget twice over would mean a
+ // hook waiting on a dead server.
+ expect(elapsed).toBeLessThan(1500);
+ expect(elapsed).toBeGreaterThanOrEqual(200);
+ const records = listDecisionModelCalls(vault);
+ expect(records[0]?.payload).toMatchObject({ provider: "compatible", outcome: "timeout" });
+ expect(records[1]?.payload).toMatchObject({ provider: "typesafe", outcome: "ok" });
+ expect(fast.requests).toHaveLength(1);
+ } finally {
+ await hang.close();
+ await fast.close();
+ }
+ });
+
+ test("an HTTP failure is an answer, not a fall-through: the next entry is not asked", async () => {
+ const vault = tempVault();
+ const refusing = await startFakeSystemOne();
+ const next = await startFakeSystemOne();
+ refusing.setReply(() => ({ status: 401 }));
+ try {
+ const cfg = resolveDecisionModelConfig({
+ env: CHAIN_ENV,
+ config: chainConfig({
+ decision_model_base_url: refusing.url,
+ decision_model_providers_typesafe_base_url: next.url,
+ decision_model_providers_typesafe_id: "fake-model-1.0",
+ decision_model_uses: "rerank:shadow",
+ }),
+ vault,
+ });
+ const result = await runDecision("rerank", build, QUESTIONS, { config: cfg, env: CHAIN_ENV });
+ expect(result).toMatchObject({ status: "degraded", reason: "http_401" });
+ expect(next.requests).toHaveLength(0);
+ expect(listDecisionModelCalls(vault)).toHaveLength(1);
+ } finally {
+ await refusing.close();
+ await next.close();
+ }
+ });
+
+ test("an entry whose key is missing is skipped, not fatal", async () => {
+ const vault = tempVault();
+ const server = await startFakeSystemOne();
+ try {
+ const cfg = resolveDecisionModelConfig({
+ env: { TYPESAFE_API_KEY: fakeCredential("O2B_TEST_CHAIN") } as NodeJS.ProcessEnv,
+ config: chainConfig({
+ decision_model_providers: "compatible,typesafe",
+ decision_model_base_url: server.url,
+ decision_model_id: "fake-model-1.0",
+ // No local key: the compatible entry is not configured.
+ decision_model_providers_typesafe_base_url: server.url,
+ decision_model_providers_typesafe_id: "jev-1.13.0",
+ decision_model_uses: "rerank:shadow",
+ }),
+ vault,
+ });
+ expect(cfg.status).toBe("active");
+ const result = await runDecision("rerank", build, QUESTIONS, {
+ config: cfg,
+ env: { TYPESAFE_API_KEY: fakeCredential("O2B_TEST_CHAIN") } as NodeJS.ProcessEnv,
+ });
+ expect(result.status).toBe("ok");
+ const records = listDecisionModelCalls(vault);
+ expect(records).toHaveLength(1);
+ expect(records[0]?.payload).toMatchObject({ provider: "typesafe", outcome: "ok" });
+ } finally {
+ await server.close();
+ }
+ });
+});
+
+describe("check with a chain", () => {
+ test("the report lists the order and per-entry key state without any request", async () => {
+ const report = await buildDecisionModelCheck({
+ config: chainConfig(),
+ vault: null,
+ env: CHAIN_ENV,
+ });
+ expect(report.provider_order.map((p) => p.name)).toEqual(["compatible", "typesafe"]);
+ expect(report.provider_order[0]?.configured).toBe(true);
+ expect(report.provider_order[0]?.ping).toBeUndefined();
+ const text = renderDecisionModelCheck(report);
+ expect(text).toContain("providers (in order): compatible, typesafe");
+ expect(text).toContain("1. compatible (systemone): http://127.0.0.1:8080");
+ expect(text).toContain("2. typesafe (systemone): https://api.typesafe.ai");
+ });
+
+ test("--ping answers per entry; a dead first entry with a live second exits 0", async () => {
+ const vault = tempVault();
+ const dead = await startFakeSystemOne();
+ const deadUrl = dead.url;
+ await dead.close();
+ const live = await startFakeSystemOne();
+ try {
+ const report = await buildDecisionModelCheck({
+ config: chainConfig({
+ decision_model_base_url: deadUrl,
+ decision_model_providers_typesafe_base_url: live.url,
+ decision_model_providers_typesafe_id: "fake-model-1.0",
+ decision_model_uses: "rerank:shadow",
+ }),
+ vault,
+ env: CHAIN_ENV,
+ ping: true,
+ });
+ expect(report.ping?.ok).toBe(false);
+ expect(report.provider_order[0]?.ping?.ok).toBe(false);
+ expect(report.provider_order[1]?.ping?.ok).toBe(true);
+ expect(report.provider_order[1]?.ping?.model).toBe("fake-model-1.0");
+ expect(decisionModelCheckExitCode(report)).toBe(0);
+ // Each ping is its own record, naming its own provider.
+ const pings = listDecisionModelCalls(vault).filter((r) => r.payload["use"] === "ping");
+ expect(pings.map((r) => r.payload["provider"])).toEqual(["compatible", "typesafe"]);
+ expect(pings.map((r) => r.payload["outcome"])).toEqual(["network", "ok"]);
+ } finally {
+ await live.close();
+ }
+ });
+
+ test("--ping exits 1 when every configured entry fails", async () => {
+ const dead = await startFakeSystemOne();
+ const deadUrl = dead.url;
+ await dead.close();
+ const report = await buildDecisionModelCheck({
+ config: chainConfig({
+ decision_model_base_url: deadUrl,
+ decision_model_providers_typesafe_base_url: deadUrl,
+ decision_model_providers_typesafe_id: "fake-model-1.0",
+ decision_model_uses: "rerank:shadow",
+ }),
+ vault: null,
+ env: CHAIN_ENV,
+ ping: true,
+ });
+ expect(decisionModelCheckExitCode(report)).toBe(1);
+ });
+});
diff --git a/tests/core/decision-model/thresholds-store.test.ts b/tests/core/decision-model/thresholds-store.test.ts
new file mode 100644
index 00000000..589016ba
--- /dev/null
+++ b/tests/core/decision-model/thresholds-store.test.ts
@@ -0,0 +1,511 @@
+/**
+ * The local per-model thresholds store (issue #264): file shape,
+ * append-only history, per-vault overrides, and the enforce gate keyed by
+ * the model id - never the provider preset - so a swapped model inherits
+ * nothing and switching back needs no re-tuning.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ decisionModelModeFor,
+ resolveDecisionModelConfig,
+} from "../../../src/core/decision-model/config.ts";
+import {
+ decisionThresholdsPath,
+ effectiveThresholds,
+ EMPTY_THRESHOLD_STORE,
+ parseThresholdsStore,
+ readThresholdsStore,
+ resetThresholdsStoreCache,
+ storedThresholdEntry,
+ writeThresholdEntry,
+} from "../../../src/core/decision-model/thresholds.ts";
+import {
+ fitThresholdAgainstTarget,
+ runThresholdFit,
+} from "../../../src/core/decision-model/thresholds-fit.ts";
+import { emitDecisionModelCall } from "../../../src/core/decision-model/record.ts";
+
+function tempVault(): string {
+ const v = mkdtempSync(join(tmpdir(), "osb-dm-thr-vault-"));
+ extraDirs.push(v);
+ mkdirSync(join(v, "Brain", "preferences"), { recursive: true });
+ mkdirSync(join(v, "Brain", "retired"), { recursive: true });
+ return v;
+}
+
+/**
+ * Writes one pair's sides: both active in preferences, or `bRetired`
+ * moves b to the retired directory, which the report reads as merged.
+ */
+function preferencePair(vault: string, id: string, aActive: boolean, bRetired: boolean): void {
+ if (aActive) {
+ writeFileSync(
+ join(vault, "Brain", "preferences", `pref-${id}a.md`),
+ `---\nid: pref-${id}a\n---\n`,
+ );
+ }
+ if (bRetired) {
+ writeFileSync(join(vault, "Brain", "retired", `ret-${id}b.md`), `---\nid: ret-${id}b\n---\n`);
+ } else {
+ writeFileSync(
+ join(vault, "Brain", "preferences", `pref-${id}b.md`),
+ `---\nid: pref-${id}b\n---\n`,
+ );
+ }
+}
+
+let tmp: string;
+let configPath: string;
+let storePath: string;
+const extraDirs: string[] = [];
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-dm-thr-"));
+ configPath = join(tmp, "config.yaml");
+ storePath = decisionThresholdsPath(configPath);
+ resetThresholdsStoreCache();
+});
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ for (const d of extraDirs.splice(0)) rmSync(d, { recursive: true, force: true });
+ resetThresholdsStoreCache();
+});
+
+const COMPATIBLE_ENFORCE = {
+ decision_model_enabled: "true",
+ decision_model_provider: "compatible",
+ decision_model_base_url: "http://127.0.0.1:9",
+ decision_model_id: "self-hosted-1",
+ decision_model_env_key: "O2B_TEST_THR_KEY",
+ decision_model_uses: "dedup:enforce,tension:enforce,labels:enforce",
+};
+
+function resolveWithStore(config: Record<string, string>, vault: string | null = null) {
+ return resolveDecisionModelConfig({
+ env: { O2B_TEST_THR_KEY: "unused" } as NodeJS.ProcessEnv,
+ config,
+ vault,
+ thresholdsPath: storePath,
+ measuredTunedMark: null,
+ });
+}
+
+describe("the thresholds store file", () => {
+ test("a missing file reads as the empty store and gates everything off", () => {
+ expect(readThresholdsStore(storePath)).toEqual(EMPTY_THRESHOLD_STORE);
+ expect(existsSync(storePath)).toBe(false);
+ });
+
+ test("a malformed file reads as empty (fail closed), never throws", () => {
+ writeFileSync(storePath, "{not json", "utf8");
+ expect(readThresholdsStore(storePath)).toEqual(EMPTY_THRESHOLD_STORE);
+ expect(parseThresholdsStore("[]")).toEqual(EMPTY_THRESHOLD_STORE);
+ });
+
+ test("write then read round-trips; an entry keeps its history on rewrite", () => {
+ writeThresholdEntry(
+ storePath,
+ "self-hosted-1",
+ "dedup",
+ { values: { low_priority_min: 0.7 }, source: "set" },
+ null,
+ new Date("2026-10-20T10:00:00Z"),
+ );
+ writeThresholdEntry(
+ storePath,
+ "self-hosted-1",
+ "dedup",
+ { values: { low_priority_min: 0.9 }, source: "fit", target: 0.05, samples: 12 },
+ null,
+ new Date("2026-10-21T10:00:00Z"),
+ );
+ const store = readThresholdsStore(storePath);
+ const entry = storedThresholdEntry(store, "self-hosted-1", "dedup", null);
+ expect(entry?.values).toEqual({ low_priority_min: 0.9 });
+ expect(entry?.source).toBe("fit");
+ expect(entry?.target).toBe(0.05);
+ expect(entry?.samples).toBe(12);
+ expect(entry?.history).toEqual([
+ { value: 0.7, source: "set", updated_at: "2026-10-20T10:00:00.000Z" },
+ ]);
+ });
+
+ test("entries of every model ever tuned are kept; nothing is deleted on a switch", () => {
+ writeThresholdEntry(storePath, "model-a", "dedup", {
+ values: { low_priority_min: 0.8 },
+ source: "set",
+ });
+ writeThresholdEntry(storePath, "model-b", "dedup", {
+ values: { low_priority_min: 0.6 },
+ source: "set",
+ });
+ // The operator moves to model-b; model-a's entry survives untouched.
+ expect(
+ storedThresholdEntry(readThresholdsStore(storePath), "model-a", "dedup", null)?.values,
+ ).toEqual({
+ low_priority_min: 0.8,
+ });
+ expect(
+ storedThresholdEntry(readThresholdsStore(storePath), "model-b", "dedup", null)?.values,
+ ).toEqual({
+ low_priority_min: 0.6,
+ });
+ });
+
+ test("history is capped", () => {
+ for (let i = 0; i < 25; i++) {
+ writeThresholdEntry(
+ storePath,
+ "m",
+ "dedup",
+ { values: { low_priority_min: 0.5 + i / 100 }, source: "set" },
+ null,
+ new Date(Date.UTC(2026, 9, 1, 0, 0, i)),
+ );
+ }
+ const entry = storedThresholdEntry(readThresholdsStore(storePath), "m", "dedup", null);
+ expect(entry?.history?.length).toBe(20);
+ });
+
+ test("a per-vault override wins for that vault and leaves the global entry", () => {
+ writeThresholdEntry(storePath, "m", "dedup", {
+ values: { low_priority_min: 0.8 },
+ source: "set",
+ });
+ writeThresholdEntry(
+ storePath,
+ "m",
+ "dedup",
+ { values: { low_priority_min: 0.6 }, source: "set" },
+ "/vaults/work",
+ );
+ const store = readThresholdsStore(storePath);
+ expect(storedThresholdEntry(store, "m", "dedup", "/vaults/work")?.values).toEqual({
+ low_priority_min: 0.6,
+ });
+ expect(storedThresholdEntry(store, "m", "dedup", "/vaults/other")?.values).toEqual({
+ low_priority_min: 0.8,
+ });
+ expect(storedThresholdEntry(store, "m", "dedup", null)?.values).toEqual({
+ low_priority_min: 0.8,
+ });
+ });
+
+ test("effectiveThresholds merges stored values over the built-in defaults", () => {
+ expect(effectiveThresholds(EMPTY_THRESHOLD_STORE, "m", "dedup", null)).toEqual({
+ low_priority_min: 0.9,
+ });
+ writeThresholdEntry(storePath, "m", "dedup", {
+ values: { low_priority_min: 0.75 },
+ source: "fit",
+ });
+ expect(effectiveThresholds(readThresholdsStore(storePath), "m", "dedup", null)).toEqual({
+ low_priority_min: 0.75,
+ });
+ });
+});
+
+describe("the enforce gate keys on the model id", () => {
+ test("a stored entry arms enforce for its model; another model stays shadow", () => {
+ writeThresholdEntry(storePath, "self-hosted-1", "dedup", {
+ values: { low_priority_min: 0.85 },
+ source: "fit",
+ });
+ const tuned = resolveWithStore(COMPATIBLE_ENFORCE);
+ expect(tuned.status).toBe("active");
+ expect(tuned.shadowOnlyUses).toEqual(["tension", "labels"]);
+ expect(tuned.uses.dedup).toBe("enforce");
+ expect(decisionModelModeFor(tuned, "dedup")).toBe("enforce");
+ expect(decisionModelModeFor(tuned, "tension")).toBe("shadow");
+
+ // The same preset pointing at a different model inherits nothing.
+ const swapped = resolveWithStore({
+ ...COMPATIBLE_ENFORCE,
+ decision_model_id: "self-hosted-2",
+ });
+ expect(swapped.status).toBe("active");
+ expect(swapped.shadowOnlyUses).toEqual(["dedup", "tension", "labels"]);
+ expect(decisionModelModeFor(swapped, "dedup")).toBe("shadow");
+
+ // Switching back restores the tune without re-tuning.
+ const back = resolveWithStore(COMPATIBLE_ENFORCE);
+ expect(back.uses.dedup).toBe("enforce");
+ });
+
+ test("the baseline family keeps enforcing with no store at all", () => {
+ const cfg = resolveDecisionModelConfig({
+ env: { TYPESAFE_API_KEY: "unused" } as NodeJS.ProcessEnv,
+ config: {
+ decision_model_enabled: "true",
+ decision_model_provider: "typesafe",
+ decision_model_uses: "dedup:enforce",
+ },
+ vault: null,
+ thresholdsPath: storePath,
+ measuredTunedMark: null,
+ });
+ expect(cfg.uses.dedup).toBe("enforce");
+ expect(cfg.shadowOnlyUses).toEqual([]);
+ });
+
+ test("the store arms a use even when no threshold profile resolves", () => {
+ // compatible carries no profile: before the store, every enforce use
+ // silently ran as shadow on it.
+ writeThresholdEntry(storePath, "self-hosted-1", "dedup", {
+ values: { low_priority_min: 0.85 },
+ source: "set",
+ });
+ const cfg = resolveWithStore(COMPATIBLE_ENFORCE);
+ expect(cfg.thresholdProfile).toBeNull();
+ expect(cfg.uses.dedup).toBe("enforce");
+ });
+
+ test("a per-vault override arms enforce for that vault only", () => {
+ writeThresholdEntry(
+ storePath,
+ "self-hosted-1",
+ "dedup",
+ { values: { low_priority_min: 0.6 }, source: "set" },
+ "/vaults/work",
+ );
+ const work = resolveWithStore(COMPATIBLE_ENFORCE, "/vaults/work");
+ expect(work.uses.dedup).toBe("enforce");
+ const other = resolveWithStore(COMPATIBLE_ENFORCE, "/vaults/other");
+ expect(other.uses.dedup).toBe("shadow");
+ });
+
+ test("thresholdsPath: null disables the store entirely", () => {
+ writeThresholdEntry(storePath, "self-hosted-1", "dedup", {
+ values: { low_priority_min: 0.85 },
+ source: "set",
+ });
+ const cfg = resolveDecisionModelConfig({
+ env: { O2B_TEST_THR_KEY: "unused" } as NodeJS.ProcessEnv,
+ config: COMPATIBLE_ENFORCE,
+ vault: null,
+ thresholdsPath: null,
+ measuredTunedMark: null,
+ });
+ expect(cfg.uses.dedup).toBe("shadow");
+ });
+});
+
+describe("the threshold fit", () => {
+ test("sweeps to the threshold that holds the false-action target", () => {
+ const vault = tempVault();
+ // Eight true negatives at or above 0.9; two false actions below 0.9
+ // (a demoted pair that was later merged anyway).
+ for (let i = 0; i < 8; i++) {
+ preferencePair(vault, `o${i}`, true, false);
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model: "fit-model-1",
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: `o${i}`,
+ a: `pref-o${i}a`,
+ b: `pref-o${i}b`,
+ verdict: "different",
+ probability: 0.9 + i / 100,
+ },
+ ],
+ },
+ });
+ }
+ for (const [i, p] of [0.6, 0.65].entries()) {
+ preferencePair(vault, `f${i}`, true, true); // one side retired: merged
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model: "fit-model-1",
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: `f${i}`,
+ a: `pref-f${i}a`,
+ b: `pref-f${i}b`,
+ verdict: "different",
+ probability: p,
+ },
+ ],
+ },
+ });
+ }
+ const result = runThresholdFit(storePath, { vault, use: "dedup" });
+ expect(result.ok).toBe(true);
+ if (!result.ok) return;
+ expect(result.model).toBe("fit-model-1");
+ expect(result.name).toBe("low_priority_min");
+ // At 0.9 the demotions are the eight open pairs: zero false actions,
+ // the most action any feasible threshold in this data achieves.
+ expect(result.entry.values).toEqual({ low_priority_min: 0.9 });
+ expect(result.entry.source).toBe("fit");
+ expect(result.entry.target).toBe(0.05);
+ expect(result.entry.samples).toBe(8);
+ });
+
+ test("nothing is written when no threshold meets the target", () => {
+ const vault = tempVault();
+ preferencePair(vault, "t", true, true);
+ // Eight true negatives at 0.80-0.87 and one merged pair at 0.95: the
+ // merged pair sits inside no feasible window, so no threshold both
+ // acts on eight samples and keeps a 1% false share.
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model: "fit-model-1",
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [{ id: "f", a: "pref-fa", b: "pref-fb", verdict: "different", probability: 0.95 }],
+ },
+ });
+ for (let i = 0; i < 8; i++) {
+ preferencePair(vault, `p${i}`, true, true);
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model: "fit-model-1",
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: `p${i}`,
+ a: `pref-p${i}a`,
+ b: `pref-p${i}b`,
+ verdict: "different",
+ probability: 0.8 + i / 100,
+ },
+ ],
+ },
+ });
+ }
+ const before = existsSync(storePath) ? readThresholdsStore(storePath) : EMPTY_THRESHOLD_STORE;
+ const result = runThresholdFit(storePath, { vault, use: "dedup", target: 0.01 });
+ expect(result.ok).toBe(false);
+ if (result.ok) return;
+ expect(result.note).toContain("no threshold meets");
+ expect(readThresholdsStore(storePath)).toEqual(before);
+ });
+
+ test("records naming several models are refused: one model's calibration per entry", () => {
+ const vault = tempVault();
+ preferencePair(vault, "t", true, true);
+ for (const model of ["a", "b"]) {
+ emitDecisionModelCall(vault, {
+ use: "dedup",
+ mode: "shadow",
+ provider: "fake",
+ model,
+ calibrated: true,
+ questionCount: 1,
+ candidateCount: 1,
+ inputPriceUsdPerMtok: null,
+ latencyMs: 1,
+ outcome: "ok",
+ details: {
+ pair_kind: "preference",
+ pairs: [
+ {
+ id: `x-${model}`,
+ a: "pref-ta",
+ b: "pref-tb",
+ verdict: "different",
+ probability: 0.9,
+ },
+ ],
+ },
+ });
+ }
+ const result = runThresholdFit(storePath, { vault, use: "dedup" });
+ expect(result.ok).toBe(false);
+ if (result.ok) return;
+ expect(result.note).toContain("several models");
+ expect(result.note).toContain("a");
+ expect(result.note).toContain("b");
+ });
+
+ test("a use without a labelled report is refused by name", () => {
+ const vault = tempVault();
+ const result = runThresholdFit(storePath, { vault, use: "rerank" });
+ expect(result.ok).toBe(false);
+ if (result.ok) return;
+ expect(result.note).toContain("no labelled report");
+ expect(result.note).toContain("dedup, tension, labels, extract_prefilter");
+ });
+});
+
+describe("the fit sweep itself", () => {
+ test("at_or_above keeps the most action under the target", () => {
+ const samples = [
+ ...Array.from({ length: 8 }, (_, i) => ({ score: 0.9 + i / 100, falseAction: false })),
+ { score: 0.6, falseAction: true },
+ ];
+ const fit = fitThresholdAgainstTarget(samples, "at_or_above", 0.05);
+ expect(fit.ok).toBe(true);
+ if (!fit.ok) return;
+ expect(fit.candidate.threshold).toBe(0.9);
+ expect(fit.candidate.acted).toBe(8);
+ expect(fit.candidate.falseShare).toBe(0);
+ });
+
+ test("below acts under the threshold and drops nothing else", () => {
+ const samples = [
+ ...Array.from({ length: 9 }, (_, i) => ({ score: 0.1 + i / 100, falseAction: false })),
+ { score: 0.6, falseAction: true },
+ ];
+ const fit = fitThresholdAgainstTarget(samples, "below", 0.05);
+ expect(fit.ok).toBe(true);
+ if (!fit.ok) return;
+ expect(fit.candidate.threshold).toBe(0.6);
+ expect(fit.candidate.acted).toBe(9);
+ expect(fit.candidate.falseActions).toBe(0);
+ });
+
+ test("too few samples refuse with the count", () => {
+ const fit = fitThresholdAgainstTarget(
+ [{ score: 0.9, falseAction: false }],
+ "at_or_above",
+ 0.05,
+ );
+ expect(fit.ok).toBe(false);
+ if (fit.ok) return;
+ expect(fit.note).toContain("1 decidable sample(s)");
+ });
+});
diff --git a/tests/core/doctor-consent-probe.test.ts b/tests/core/doctor-consent-probe.test.ts
index 60bfe35e..4c4ebf74 100644
--- a/tests/core/doctor-consent-probe.test.ts
+++ b/tests/core/doctor-consent-probe.test.ts
@@ -20,7 +20,7 @@
*/
import { afterEach, beforeEach, describe, expect, test } from "bun:test";
-import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -98,6 +98,81 @@ describe("the provider-consent readiness probe", () => {
expect(verdict.detail).not.toContain("embedding_provider=disabled");
});
+ test("the present provider environment variables are listed on the no-decision arm too", async () => {
+ // The env census travels with every arm: an operator who set the key
+ // in the environment and never wrote a decision needs to see that
+ // the key is there all the same.
+ writeConfig("");
+ const verdict = await probeProviderConsent(probeOpts({ [PROVIDER_KEY_ENV]: PRIVATE_BODY }));
+ expect(verdict.status).toBe("skipped");
+ expect(verdict.detail).toContain(PROVIDER_KEY_ENV);
+ expect(verdict.detail).not.toContain(PRIVATE_BODY);
+ });
+
+ test("a registered profile's env-key name is listed when a variable of that name is present", async () => {
+ // A registry profile keeps its credential in an OPERATOR-NAMED
+ // variable (envKey in Brain/search/embedding-providers.json), which
+ // no OPEN_SECOND_BRAIN_* prefix reaches. The probe reads the NAMES
+ // out of the registry and reports the ones present - never values.
+ const profileEnv = "OPENROUTER_API_KEY";
+ mkdirSync(join(tmp, "Brain", "search"), { recursive: true });
+ writeFileSync(
+ join(tmp, "Brain", "search", "embedding-providers.json"),
+ JSON.stringify([
+ {
+ name: "openrouter",
+ baseUrl: "https://openrouter.invalid/v1",
+ defaultModel: "acme/embed-1",
+ envKey: profileEnv,
+ },
+ ]),
+ );
+ writeConfig("search_semantic_enabled: false\n");
+ const verdict = await probeProviderConsent(probeOpts({ [profileEnv]: PRIVATE_BODY }));
+ expect(verdict.status).toBe("skipped");
+ expect(verdict.detail).toContain(profileEnv);
+ expect(verdict.detail).not.toContain(PRIVATE_BODY);
+ // A profile name whose variable is absent lists nothing.
+ const without = await probeProviderConsent(probeOpts());
+ expect(without.detail).not.toContain(profileEnv);
+ });
+
+ test("the rerank environment is part of the census", async () => {
+ // The rerank resolver spells its settings with their own prefix, and
+ // its credential lives in the variable `search_rerank_env_key`
+ // NAMES - both were invisible to a probe that read the embedding
+ // prefix alone.
+ const rerankCredentialEnv = "MY_RERANK_KEY";
+ writeConfig(`search_semantic_enabled: false\nsearch_rerank_env_key: ${rerankCredentialEnv}\n`);
+ const verdict = await probeProviderConsent(
+ probeOpts({
+ OPEN_SECOND_BRAIN_SEARCH_RERANK_MODEL: "acme/rerank-1",
+ [rerankCredentialEnv]: PRIVATE_BODY,
+ }),
+ );
+ expect(verdict.status).toBe("skipped");
+ expect(verdict.detail).toContain("OPEN_SECOND_BRAIN_SEARCH_RERANK_MODEL");
+ expect(verdict.detail).toContain(rerankCredentialEnv);
+ expect(verdict.detail).not.toContain(PRIVATE_BODY);
+ });
+
+ test("a record that cannot be read is unknown, not a skipped decision", async () => {
+ // A DIRECTORY in the config's place: the resolver fails that arm
+ // closed to declined, but the probe could not measure whether a
+ // decision exists, which is the `unknown` vocabulary - the CLI's
+ // own "could not measure" - and never a clean skip.
+ mkdirSync(configPath, { recursive: true });
+ const verdict = await probeProviderConsent(probeOpts());
+ expect(verdict.status).toBe("unknown");
+ expect(verdict.detail).toContain("could not be read");
+ // The counts follow the verdict: a broken machine configuration is
+ // 0 failed but also 0 skipped-for-this-probe and 1 unknown.
+ const report = await runReadinessProbes(probeOpts());
+ expect(report.unknown).toBeGreaterThan(0);
+ const consent = report.probes.find((p) => p.name === READINESS_PROBE.providerConsent);
+ expect(consent?.status).toBe("unknown");
+ });
+
test("the value of a present environment variable never reaches the JSON output", async () => {
writeConfig("search_semantic_enabled: false\n");
const report = await runReadinessProbes(probeOpts({ [PROVIDER_KEY_ENV]: PRIVATE_BODY }));
diff --git a/tests/core/graph/privacy-markers.test.ts b/tests/core/graph/privacy-markers.test.ts
index 8505fc98..8bf8e8b3 100644
--- a/tests/core/graph/privacy-markers.test.ts
+++ b/tests/core/graph/privacy-markers.test.ts
@@ -23,6 +23,7 @@ import {
PRIVACY_DENY_MARKER_VALUES,
PRIVACY_MARKER_KEYS,
hasPrivacyDenyMarker,
+ hasPrivacyMarkerKey,
markerDenyTokens,
remoteDenyVisibilityTokens,
} from "../../../src/core/graph/privacy-markers.ts";
@@ -108,6 +109,32 @@ describe("normalisation is the visibility module's own", () => {
expect(hasPrivacyDenyMarker({ sensitivity: ["CONFIDENTIAL"] })).toBe(true);
});
+ test("a capitalised KEY spelling is the same marker key", () => {
+ // A vault that spells the key `Access` makes the same claim as the
+ // lower-case one; a reader matching keys by exact case would miss it
+ // and hand the page to a remote reader.
+ expect(hasPrivacyDenyMarker({ Access: "private" })).toBe(true);
+ expect(hasPrivacyDenyMarker({ ACCESS: "Private" })).toBe(true);
+ expect(hasPrivacyDenyMarker({ Confidentiality: "Secret" })).toBe(true);
+ expect(hasPrivacyDenyMarker({ "Sensitivity ": ["Confidential"] })).toBe(true);
+ expect(markerDenyTokens({ Access: "private" })).toEqual([REMOTE_DENY_VISIBILITY_TOKEN]);
+ expect(remoteDenyVisibilityTokens({ ACCESS: "Private" })).toEqual([
+ REMOTE_DENY_VISIBILITY_TOKEN,
+ ]);
+ });
+
+ test("a capitalised key carrying a non-deny value is still no marker", () => {
+ expect(hasPrivacyDenyMarker({ Access: "public" })).toBe(false);
+ expect(hasPrivacyDenyMarker({ Topic: "private" })).toBe(false);
+ });
+
+ test("hasPrivacyMarkerKey answers for any marker key, deny value or not", () => {
+ expect(hasPrivacyMarkerKey({ access: "public" })).toBe(true);
+ expect(hasPrivacyMarkerKey({ Confidentiality: 42 })).toBe(true);
+ expect(hasPrivacyMarkerKey({ visibility: "private" })).toBe(false);
+ expect(hasPrivacyMarkerKey({})).toBe(false);
+ });
+
test("comparison happens after NFC composition, not by stripping marks", () => {
// `private` + a combining acute composes to a DIFFERENT token under
// NFC. A normaliser that stripped the mark would answer "private".
diff --git a/tests/core/search/chunk-hold-persistence.test.ts b/tests/core/search/chunk-hold-persistence.test.ts
index 6fe2a3c1..d19f8ff8 100644
--- a/tests/core/search/chunk-hold-persistence.test.ts
+++ b/tests/core/search/chunk-hold-persistence.test.ts
@@ -32,8 +32,9 @@ import {
RECOMMENDED_EMBEDDING_MODEL,
} from "../../../src/core/search/embeddings/presets.ts";
import {
- charLengthOverTokenBudget,
embeddingSignature,
+ textExtent,
+ tokenHoldFloor,
} from "../../../src/core/search/embeddings/signature.ts";
import type {
ResolvedEmbeddingConfig,
@@ -45,9 +46,24 @@ import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";
const WINDOW = declaredInputWindowTokens(RECOMMENDED_EMBEDDING_MODEL)!;
-/** The LOW estimate exceeds the window: provably over, so it gets held. */
-const OVER_CONTENT = "b".repeat(charLengthOverTokenBudget(WINDOW) + 1);
-const OVER_FLOOR = Math.ceil(OVER_CONTENT.length / 4);
+/**
+ * An ordinary English sentence, assembled into a fixture of an exact
+ * code-point size: the hold's proof reads code points, and a fixture
+ * shaped like the corpus keeps the persistence tests honest about which
+ * chunks a real tokenizer cannot fit.
+ */
+const PROSE_SENTENCE =
+ "The survey team recorded every measurement in the ledger before the storm reached the harbour wall. ";
+
+/**
+ * The conservative hold bound already exceeds the window: even at one
+ * token per twelve code points - far below what this prose costs a real
+ * tokenizer - it cannot fit, so it gets held.
+ */
+const OVER_CONTENT = PROSE_SENTENCE.repeat(
+ Math.ceil((WINDOW * 12 + 1) / PROSE_SENTENCE.length),
+).slice(0, WINDOW * 12 + 1);
+const HELD_TOKENS = tokenHoldFloor(textExtent(OVER_CONTENT));
const FITTING = "small chunk of ordinary prose";
let vault: string;
@@ -186,7 +202,7 @@ test("the phase persists a hold row keyed by model, window and signature", async
dimension: config.semantic.dimension,
}),
);
- expect(rows[0]!.estimated_tokens).toBe(OVER_FLOOR);
+ expect(rows[0]!.estimated_tokens).toBe(HELD_TOKENS);
expect(Number.isNaN(Date.parse(rows[0]!.held_at))).toBe(false);
});
@@ -283,7 +299,7 @@ test("a model change releases the hold under the old model's key", async () => {
}
// bge-m3's declared window bounds the proof the other way: under the
// new model the chunk simply fits, so the releasing run embeds it.
- expect(declaredInputWindowTokens("BAAI/bge-m3")! > OVER_FLOOR).toBe(true);
+ expect(declaredInputWindowTokens("BAAI/bge-m3")! > HELD_TOKENS).toBe(true);
const store2 = await Store.open(other, { mode: "write" });
const tally = EMPTY_TALLY();
try {
@@ -306,7 +322,7 @@ test("holding again refreshes the row rather than duplicating it", async () => {
// Same-key re-decision re-records through the primary key rather than
// stacking a second row for one chunk.
store.recordChunkHolds(
- [{ chunkId: heldId, estimatedTokens: OVER_FLOOR, windowTokens: WINDOW }],
+ [{ chunkId: heldId, estimatedTokens: HELD_TOKENS, windowTokens: WINDOW }],
{
model: RECOMMENDED_EMBEDDING_MODEL,
embeddingSignature: chunkHoldExclusion(original)!.embeddingSignature,
diff --git a/tests/core/search/chunk-window-census.test.ts b/tests/core/search/chunk-window-census.test.ts
index e4252d2a..eaedebf8 100644
--- a/tests/core/search/chunk-window-census.test.ts
+++ b/tests/core/search/chunk-window-census.test.ts
@@ -14,12 +14,13 @@
*
* Three properties are pinned here, and each one can fail:
*
- * - the predicate the store evaluates is EXACTLY the estimator the
- * embedding request path sizes batches with, proved by brute force
- * rather than by inspection;
- * - the ONE place the two disagree - SQLite counts code points where
- * JavaScript counts UTF-16 code units - is pinned as a bound and a
- * direction, not left to be discovered;
+ * - the census's over-window arm is EXACTLY the hold bound the
+ * embedding phase acts on, so the report and the phase can never
+ * disagree about which chunks are provably over, and the boundary is
+ * proved by brute force rather than by inspection;
+ * - the ONE place the estimates disagree with the database - SQLite
+ * counts code points where JavaScript counts UTF-16 code units - is
+ * pinned as a bound and a direction, not left to be discovered;
* - a model with no declared window reports that the check did not
* run. Never that it passed.
*
@@ -108,7 +109,12 @@ function oversizeBody(windowTokens: number): string {
// โ”€โ”€ the predicate is the estimator โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
-test("the census predicate equals the request path's estimator at every length", () => {
+test("the request path's spend estimator is chars-over-four at every length", () => {
+ // The average-rate estimate the batches are priced with. The census's
+ // over-window arm is deliberately NOT this predicate - it is the
+ // conservative hold bound - because the average reads whole English
+ // chunks that fit the window as over it; the undecided band between
+ // the two is what the census reports instead of passing them.
for (const windowTokens of [1, 8, 128, RECOMMENDED_WINDOW]) {
const threshold = charLengthOverTokenBudget(windowTokens);
for (let len = 0; len <= threshold + 8; len++) {
@@ -153,9 +159,12 @@ test("the shipped defaults are the ones that overflow each other", async () => {
// โ”€โ”€ the store-side count โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
-test("the store counts exactly the chunks whose estimate exceeds the window", async () => {
+test("the store counts exactly the chunks whose hold bound exceeds the window", async () => {
const window = 16;
- const threshold = charLengthOverTokenBudget(window);
+ // The over-window arm is the HOLD population: past it only when even
+ // the conservative bound - one token per twelve code points - cannot
+ // fit the window.
+ const holdBoundary = window * 12;
const store = await Store.open(cfg(), { mode: "write" });
try {
const docId = store.upsertDocument({
@@ -171,7 +180,7 @@ test("the store counts exactly the chunks whose estimate exceeds the window", as
store.replaceChunks(docId, [
{
chunkIndex: 0,
- content: "b".repeat(threshold),
+ content: "b".repeat(holdBoundary),
contentHash: "h0",
startLine: 1,
endLine: 1,
@@ -179,7 +188,7 @@ test("the store counts exactly the chunks whose estimate exceeds the window", as
},
{
chunkIndex: 1,
- content: "b".repeat(threshold + 1),
+ content: "b".repeat(holdBoundary + 1),
contentHash: "h1",
startLine: 2,
endLine: 2,
@@ -187,7 +196,7 @@ test("the store counts exactly the chunks whose estimate exceeds the window", as
},
{
chunkIndex: 2,
- content: "b".repeat(threshold * 3),
+ content: "b".repeat(holdBoundary * 3),
contentHash: "h2",
startLine: 3,
endLine: 3,
@@ -196,7 +205,8 @@ test("the store counts exactly the chunks whose estimate exceeds the window", as
{ chunkIndex: 3, content: "", contentHash: "h3", startLine: 4, endLine: 4, tokenCount: 0 },
]);
// Exactly the two over the boundary - the one AT it is not over it,
- // which is the same boundary `estimateTokens` draws.
+ // which is the same boundary the phase's hold draws. The first two
+ // chunks sit in the undecided band instead: reported, still sent.
expect(store.chunksOverTokenWindow(window)).toBe(2);
expect(store.chunksOverTokenWindow(window * 4)).toBe(0);
} finally {
@@ -206,13 +216,13 @@ test("the store counts exactly the chunks whose estimate exceeds the window", as
test("the code-point / code-unit divergence is a lower bound, and only that", async () => {
const window = 16;
- const threshold = charLengthOverTokenBudget(window);
+ const holdBoundary = window * 12;
// A supplementary-plane character is ONE code point to SQLite's
// length() and TWO code units to String.length. A chunk built entirely
- // from them sits at the threshold in the database and past it in the
- // estimator, so the census declines to count it.
- const astral = "\u{1F600}".repeat(threshold);
- expect(astral.length).toBe(threshold * 2);
+ // from them sits at the hold boundary in the database and past it in
+ // the estimator, so the census declines to count it as over.
+ const astral = "\u{1F600}".repeat(holdBoundary);
+ expect(astral.length).toBe(holdBoundary * 2);
expect(estimateTokens([astral])).toBeGreaterThan(window);
const store = await Store.open(cfg(), { mode: "write" });
@@ -240,12 +250,12 @@ test("the code-point / code-unit divergence is a lower bound, and only that", as
// The stated bound: never a false alarm, and short by at most one
// code unit per supplementary character.
expect(store.chunksOverTokenWindow(window)).toBe(0);
- // One more astral character past the code-point threshold and the
+ // One more astral character past the code-point boundary and the
// count fires, so the divergence is a bound and not a blind spot.
store.replaceChunks(docId, [
{
chunkIndex: 0,
- content: "\u{1F600}".repeat(threshold + 1),
+ content: "\u{1F600}".repeat(holdBoundary + 1),
contentHash: "h1",
startLine: 1,
endLine: 1,
diff --git a/tests/core/search/chunk-window-estimate.test.ts b/tests/core/search/chunk-window-estimate.test.ts
index 96f01fa6..98fc14d9 100644
--- a/tests/core/search/chunk-window-estimate.test.ts
+++ b/tests/core/search/chunk-window-estimate.test.ts
@@ -19,10 +19,14 @@
* database - a chunk estimated at twice the window was missed.
*
* All three shared one shape: the census said nothing, and nothing was
- * defined to mean "every chunk fits". The fix is a two-sided bound - a
- * chunk over the low estimate is counted, a chunk over only the high
- * estimate is reported as undecided - so silence is earned rather than
- * assumed.
+ * defined to mean "every chunk fits". The fix is two arms with a stated
+ * boundary between them: `over-window` is the HOLD population, decided by
+ * the conservative hold bound (`tokenHoldFloor`, a code point charged at
+ * no more than a twelfth of a token) - the only estimate that reads as
+ * proof, after the chars-over-four average was caught holding chunks that
+ * fit; and `undecided` is everything the proof cannot settle, whose high
+ * estimate still reads over the window and which is reported, still sent.
+ * Silence is earned rather than assumed.
*/
import { afterEach, beforeEach, expect, test } from "bun:test";
@@ -33,7 +37,7 @@ import {
estimateTokens,
textExtent,
tokenEstimateCeiling,
- tokenEstimateFloor,
+ tokenHoldFloor,
utf8ByteFloorUnderTokenBudget,
} from "../../../src/core/search/embeddings/signature.ts";
import {
@@ -144,8 +148,9 @@ test("a chunk pushed over the window BY the instruction prefix is counted", asyn
expect(RESOLVED_PASSAGE_PREFIX).toBe(E5_PASSAGE_PREFIX);
const content = "a".repeat(RECOMMENDED_WINDOW * 4);
- // Without the prefix the estimate lands exactly ON the window; with it,
- // over. The request always carries it.
+ // Without the prefix the average estimate lands exactly ON the window;
+ // with it, over. The request always carries it, so this chunk is in
+ // the census's undecided band - reported, never a pass.
expect(estimateTokens([content])).toBe(RECOMMENDED_WINDOW);
expect(estimateTokens([RESOLVED_PASSAGE_PREFIX + content])).toBeGreaterThan(RECOMMENDED_WINDOW);
@@ -155,6 +160,23 @@ test("a chunk pushed over the window BY the instruction prefix is counted", asyn
expect(warning ?? "(the census said nothing)").toContain(String(RECOMMENDED_WINDOW));
});
+test("a chunk pushed over the hold bound BY the instruction prefix is counted as over", async () => {
+ // The same charge at the hold bound itself: this body fits the window
+ // by exactly one code point, and the prefix the backend sends is what
+ // puts the request over it. The verdict is the over-window arm, the
+ // one the embedding phase holds on.
+ const content = "a".repeat(RECOMMENDED_WINDOW * 12 - RESOLVED_PASSAGE_PREFIX.length + 1);
+ expect(tokenHoldFloor(textExtent(content))).toBe(RECOMMENDED_WINDOW);
+ expect(tokenHoldFloor(textExtent(RESOLVED_PASSAGE_PREFIX + content))).toBeGreaterThan(
+ RECOMMENDED_WINDOW,
+ );
+
+ await seedOneChunk(cfg(), content);
+ const status = await indexStatus(cfg());
+ const warning = status.warnings.find((w) => w.includes(OVERFLOW_COMMAND));
+ expect(warning ?? "(the census said nothing)").toContain("held");
+});
+
test("a provider that sends no prefix has none subtracted", async () => {
// ZeroEntropy embeds the raw text: its `embed` takes no kind and
// prepends nothing, so the same chunk is exactly at the window and the
@@ -212,22 +234,36 @@ test("an all-ASCII chunk comfortably inside the window still says nothing", asyn
expect(surface.undeclared).toBe(false);
});
-// โ”€โ”€ the two bounds, as arithmetic โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+// โ”€โ”€ the two arms, as arithmetic โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
-test("for pure ASCII the two estimates ARE the old one, so the band is empty", () => {
+test("for pure ASCII the high estimate IS the average one", () => {
for (let len = 0; len <= 64; len++) {
const text = "a".repeat(len);
const extent = textExtent(text);
- expect(`len=${len} floor=${tokenEstimateFloor(extent)}`).toBe(
- `len=${len} floor=${estimateTokens([text])}`,
- );
expect(`len=${len} ceiling=${tokenEstimateCeiling(extent)}`).toBe(
`len=${len} ceiling=${estimateTokens([text])}`,
);
}
});
-test("the ceiling brackets what the old estimate under-reported on Han and astral text", () => {
+test("the hold bound clears prose the average estimate would hold", () => {
+ // The defect that moved the hold to a conservative bound: a real
+ // English paragraph of 519 code points costs about 111 tokens under
+ // the shipped preset families' 4.5-9 code points per token, so it fits
+ // the narrowest shipped window (128) - while the chars-over-four
+ // average reads 130 and held it from embedding for nothing.
+ const paragraph =
+ "The survey team recorded every measurement in the ledger before the storm reached " +
+ "the harbour wall, and the notes survived the night in a dry canvas bag. ";
+ const prose = paragraph.repeat(Math.ceil(519 / paragraph.length)).slice(0, 519);
+ const extent = textExtent(prose);
+ expect(extent.codePoints).toBe(519);
+ expect(Math.ceil(extent.codePoints / 4)).toBe(130);
+ expect(Math.ceil(extent.codePoints / 4)).toBeGreaterThan(128);
+ expect(tokenHoldFloor(extent)).toBeLessThanOrEqual(128);
+});
+
+test("the ceiling brackets what the average estimate under-reports on Han and astral text", () => {
// Han: about one token per character under a BERT-family tokenizer.
const han = "ๆ–‡".repeat(400);
expect(estimateTokens([han])).toBe(100);
@@ -235,21 +271,21 @@ test("the ceiling brackets what the old estimate under-reported on Han and astra
// Astral: SQLite reads half of what String.length reads, and the
// ceiling covers both readings.
const astral = "\u{1F600}".repeat(400);
- expect(tokenEstimateFloor(textExtent(astral))).toBe(100);
+ expect(tokenHoldFloor(textExtent(astral))).toBe(34);
expect(estimateTokens([astral])).toBe(200);
expect(tokenEstimateCeiling(textExtent(astral))).toBe(400);
});
-test("the floor never exceeds the ceiling, over every script mixture", () => {
+test("the hold bound never exceeds the ceiling, over every script mixture", () => {
const alphabets = ["a", "รฉ", "ๆ–‡", "\u{1F600}", " ", "ะ–"];
for (const first of alphabets) {
for (const second of alphabets) {
for (const repeats of [0, 1, 3, 17]) {
const text = (first + second).repeat(repeats);
const extent = textExtent(text);
- const floor = tokenEstimateFloor(extent);
+ const bound = tokenHoldFloor(extent);
const ceiling = tokenEstimateCeiling(extent);
- expect(`${JSON.stringify(text)}: ${floor <= ceiling}`).toBe(
+ expect(`${JSON.stringify(text)}: ${bound <= ceiling}`).toBe(
`${JSON.stringify(text)}: true`,
);
}
@@ -257,11 +293,12 @@ test("the floor never exceeds the ceiling, over every script mixture", () => {
}
});
-test("the byte prefilter never skips a text either estimate would flag", () => {
+test("the byte prefilter never skips a text either census arm would flag", () => {
// The query tests the free measurement first and walks only the
// survivors. That is only sound if a text under the byte floor is
- // provably under both estimates, so prove it by brute force over every
- // script mixture and every length either estimate can straddle.
+ // provably under BOTH arms - the hold bound and the high estimate - so
+ // prove it by brute force over every script mixture and every length
+ // either arm can straddle.
const units = ["a", "ab", "รฉ", "ๆ–‡", "\u{1F600}", "ะ–ๆ–‡", "aๆ–‡", "\u{1F600}a"];
for (const windowTokens of [1, 4, 16, 128, 512]) {
const floor = utf8ByteFloorUnderTokenBudget(windowTokens);
@@ -270,7 +307,7 @@ test("the byte prefilter never skips a text either estimate would flag", () => {
const extent = textExtent(unit.repeat(repeats));
if (extent.utf8Bytes > floor) continue;
const flagged =
- tokenEstimateFloor(extent) > windowTokens || tokenEstimateCeiling(extent) > windowTokens;
+ tokenHoldFloor(extent) > windowTokens || tokenEstimateCeiling(extent) > windowTokens;
expect(`w=${windowTokens} ${JSON.stringify(unit)}x${repeats}: ${flagged}`).toBe(
`w=${windowTokens} ${JSON.stringify(unit)}x${repeats}: false`,
);
@@ -281,18 +318,26 @@ test("the byte prefilter never skips a text either estimate would flag", () => {
// โ”€โ”€ the store evaluates exactly those two predicates โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
-test("the SQL tally reproduces the two predicates chunk for chunk", async () => {
+test("the SQL tally reproduces the two census arms chunk for chunk", async () => {
const windowTokens = 16;
const prefix = RESOLVED_PASSAGE_PREFIX;
- const budget = windowTokens * 4;
+ const holdBoundary = windowTokens * 12; // code points past this are held
+ const averageBoundary = windowTokens * 4; // past this the high estimate reads over
const bodies: string[] = [];
- // Straddle the low threshold in ASCII, then the high one in three
- // other scripts, then the empty chunk.
- for (const len of [budget - prefix.length - 1, budget - prefix.length, budget, budget + 1]) {
+ // Straddle the hold bound in ASCII, then the undecided band below it,
+ // then the high estimate in four other scripts, then the empty chunk.
+ for (const len of [
+ holdBoundary - prefix.length - 1,
+ holdBoundary - prefix.length,
+ holdBoundary - prefix.length + 1,
+ ]) {
bodies.push("a".repeat(len));
}
+ for (const len of [averageBoundary - prefix.length - 1, averageBoundary - prefix.length]) {
+ bodies.push("a".repeat(Math.max(0, len)));
+ }
for (const unit of ["ๆ–‡", "\u{1F600}", "ะ–", "รฉ"]) {
- for (const count of [1, windowTokens - 1, windowTokens, windowTokens + 1, budget]) {
+ for (const count of [1, windowTokens, windowTokens + 1, averageBoundary, holdBoundary + 1]) {
bodies.push(unit.repeat(count));
}
}
@@ -326,7 +371,7 @@ test("the SQL tally reproduces the two predicates chunk for chunk", async () =>
let undecided = 0;
for (const body of bodies) {
const extent = textExtent(prefix + body);
- if (tokenEstimateFloor(extent) > windowTokens) overWindow++;
+ if (tokenHoldFloor(extent) > windowTokens) overWindow++;
else if (tokenEstimateCeiling(extent) > windowTokens) undecided++;
}
// The fixture must actually exercise both arms, or this test would
diff --git a/tests/core/search/chunker-dense-ascii.test.ts b/tests/core/search/chunker-dense-ascii.test.ts
new file mode 100644
index 00000000..2dfd1867
--- /dev/null
+++ b/tests/core/search/chunker-dense-ascii.test.ts
@@ -0,0 +1,174 @@
+/**
+ * Long whitespace-free runs (URLs, file paths, hashes, identifiers) are
+ * the Latin-side twin of unspaced scripts: a subword tokenizer spends
+ * roughly one token per two to four characters on them, so counting a
+ * 3,700-character URL as one "word" let a single chunk thousands of real
+ * tokens past the budget, and no `maxTokens` setting could split it,
+ * because the splitter only ever cut between lines, sentences or
+ * whitespace words (issue #272). The run is now charged about one token
+ * per three characters and cut inside itself.
+ *
+ * Every chunk must also stay small enough to plausibly fit a real model
+ * window. Such runs cost about one real tokenizer token per two
+ * characters, so the 1,024-character ceiling below holds a chunk under
+ * roughly 512 real tokens - asserted with character counts, because the
+ * test suite must not download a tokenizer.
+ */
+
+import { test, expect } from "bun:test";
+
+import { chunkMarkdown, countChunkTokens } from "../../../src/core/search/chunker.ts";
+import { extractLinks } from "../../../src/core/search/links.ts";
+
+const URL_LINE = "https://example.invalid/" + "alpha-beta-gamma-delta/".repeat(160);
+const DOC = `# Fictional link\n\n${URL_LINE}`;
+const URL_ESTIMATED_TOKENS = Math.ceil(URL_LINE.length / 3);
+
+/** The issue's reproducer options for one `maxTokens` value. */
+function reproducerOptions(size: number): {
+ maxTokens: number;
+ minTokens: number;
+ overlapTokens: number;
+} {
+ return { maxTokens: size, minTokens: Math.min(20, size), overlapTokens: 5 };
+}
+
+/** Chunk contents with the packer's synthetic line breaks removed. */
+function flatChunks(chunks: ReadonlyArray<{ readonly content: string }>): string[] {
+ return chunks.map((c) => c.content.replace(/\n/g, ""));
+}
+
+for (const size of [80, 40, 20]) {
+ test(`the issue reproducer at maxTokens ${size} splits within the cap and the window`, () => {
+ const r = chunkMarkdown(DOC, "Fictional link", reproducerOptions(size));
+ // The old chunker emitted this document as a single 3,721-character
+ // chunk estimated at 4 tokens.
+ expect(r.chunks.length).toBeGreaterThan(1);
+ for (const c of r.chunks) {
+ expect(c.tokenCount).toBeLessThanOrEqual(size);
+ expect(c.content.length).toBeLessThanOrEqual(1024);
+ }
+ });
+}
+
+test("the split distributes the whole URL, none of it lost", () => {
+ for (const size of [80, 40, 20]) {
+ const parts = flatChunks(chunkMarkdown(DOC, "Fictional link", reproducerOptions(size)).chunks);
+ // Consecutive chunks carry an overlap of at least ten characters, so
+ // every ten-character window of the URL falls inside some single
+ // chunk rather than straddling a seam.
+ for (let i = 0; i < URL_LINE.length; i += 10) {
+ const window = URL_LINE.slice(i, i + 10);
+ expect(parts.some((p) => p.includes(window))).toBe(true);
+ }
+ }
+});
+
+test("with no overlap the chunks rebuild the document exactly", () => {
+ const r = chunkMarkdown(DOC, "doc", { maxTokens: 80, minTokens: 20, overlapTokens: 0 });
+ expect(flatChunks(r.chunks).join("")).toBe(DOC.replace(/\n/g, ""));
+});
+
+test("splitting one run duplicates no more than the overlap budget", () => {
+ const r = chunkMarkdown(DOC, "Fictional link", reproducerOptions(80));
+ const carried = flatChunks(r.chunks).reduce((n, c) => n + c.length, 0);
+ // An overlap is at most `overlapTokens` estimated tokens, and one
+ // estimated token covers at most 15 characters before the dense-run
+ // rule applies.
+ expect(carried).toBeLessThanOrEqual(DOC.length + r.chunks.length * 5 * 15);
+});
+
+// โ”€โ”€ wikilinks survive the split โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+const WIKILINK = "[[pref-imperative-commit-messages|write commits that read well]]";
+
+/**
+ * The indexer extracts links per chunk, so a `[[...]]` span severed at a
+ * chunk boundary vanishes from the link index even though its text is
+ * still indexed - a dense slug is exactly the kind of run the estimator
+ * now charges and cuts inside. Both cut sites are pinned here: the body
+ * pieces of an over-budget line, and the overlap tail of the chunk
+ * before. A chunk that opens a wikilink must also close it, and the
+ * repo's own link extractor must still resolve the target from the
+ * chunks alone.
+ */
+function expectWikilinkIntact(options: {
+ maxTokens: number;
+ minTokens: number;
+ overlapTokens: number;
+ body: string;
+}): void {
+ const r = chunkMarkdown(`# Log\n\n${options.body}`, "log", options);
+ expect(r.chunks.length).toBeGreaterThan(1);
+ for (const c of r.chunks) {
+ expect(c.tokenCount).toBeLessThanOrEqual(options.maxTokens);
+ // No chunk opens or closes a wikilink it never completes.
+ expect(c.content.split("[[").length - 1).toBe(c.content.split("]]").length - 1);
+ }
+ expect(r.chunks.some((c) => c.content.includes(WIKILINK))).toBe(true);
+ const targets = r.chunks.flatMap((c) =>
+ extractLinks(c.content)
+ .filter((l) => l.linkType === "wikilink")
+ .map((l) => l.targetPath),
+ );
+ expect(targets).toContain("pref-imperative-commit-messages");
+}
+
+test("a budget boundary never severs a wikilink while an alternative cut exists", () => {
+ const filler = Array.from({ length: 14 }, (_, i) => `word${i}`).join(" ");
+ expectWikilinkIntact({
+ maxTokens: 40,
+ minTokens: 10,
+ overlapTokens: 10,
+ body: `${filler} ${WIKILINK} and then the sentence carries on with more ordinary prose until the budget boundary moves well past the link again.`,
+ });
+});
+
+test("an overlap tail never opens with a severed wikilink", () => {
+ const filler = Array.from({ length: 6 }, (_, i) => `word${i}`).join(" ");
+ const tail = Array.from({ length: 6 }, (_, i) => `tail${i}`).join(" ");
+ expectWikilinkIntact({
+ maxTokens: 30,
+ minTokens: 8,
+ overlapTokens: 5,
+ body: `${filler} ${WIKILINK} ${tail}`,
+ });
+});
+
+test("the estimator charges long runs per character and prose per word", () => {
+ // Prose keeps the plain word count, punctuation attached.
+ expect(countChunkTokens("The quick brown fox jumps over the lazy dog.")).toBe(9);
+ expect(countChunkTokens("responsibility.")).toBe(1);
+ // Below the threshold a run is one word; from sixteen characters the
+ // run is charged at three characters per token.
+ expect(countChunkTokens("abcdefghijklmno")).toBe(1);
+ expect(countChunkTokens("abcdefghijklmnop")).toBe(6);
+ expect(countChunkTokens(URL_LINE)).toBe(URL_ESTIMATED_TOKENS);
+ // Han characters next to a URL charge the URL densely, not the Han.
+ expect(countChunkTokens(`็ดขๅผ•${URL_LINE}`)).toBe(2 + URL_ESTIMATED_TOKENS);
+});
+
+test("a long hash between short prose lines chunks cleanly", () => {
+ const hash = "a3f9c2e7".repeat(32);
+ const doc =
+ "## Sync state\n\nThe manifest pins " +
+ hash +
+ " as the verified head.\n\nOrdinary prose follows, short words only.";
+ const r = chunkMarkdown(doc, "sync", { maxTokens: 60, minTokens: 10, overlapTokens: 0 });
+ expect(r.chunks.length).toBeGreaterThan(1);
+ for (const c of r.chunks) {
+ expect(c.tokenCount).toBeLessThanOrEqual(60);
+ expect(c.content.length).toBeLessThanOrEqual(1024);
+ }
+ expect(flatChunks(r.chunks).join("")).toBe(doc.replace(/\n/g, ""));
+});
+
+test("pure Han text chunks exactly as before this change", () => {
+ const han = "ๅ‘้‡็ดขๅผ•ๆŠŠๆฏไธ€็ฏ‡็ฌ”่ฎฐๅˆ‡ๆˆ่‹ฅๅนฒ็‰‡ๆฎตๅนถๅˆ†ๅˆซ่ฎก็ฎ—ๅตŒๅ…ฅไปฅไพฟ่ฏญไน‰ๆฃ€็ดข";
+ expect(countChunkTokens(han)).toBe(han.length);
+ const long = han.repeat(100);
+ const r = chunkMarkdown(long, "zh", { maxTokens: 100, minTokens: 10, overlapTokens: 0 });
+ expect(r.chunks.length).toBeGreaterThan(1);
+ for (const c of r.chunks) expect(c.tokenCount).toBeLessThanOrEqual(100);
+ expect(flatChunks(r.chunks).join("")).toBe(long);
+});
diff --git a/tests/core/search/embed-window-hold.test.ts b/tests/core/search/embed-window-hold.test.ts
index f3a2d622..e365555b 100644
--- a/tests/core/search/embed-window-hold.test.ts
+++ b/tests/core/search/embed-window-hold.test.ts
@@ -1,16 +1,19 @@
/**
* F2 chunk skip-and-hold (Lane B, task B1): the embedding phase holds
- * chunks whose token estimate PROVABLY exceeds the model's effective
+ * chunks whose token count PROVABLY exceeds the model's effective
* declared input window.
*
- * "Provably" is the whole unit. The estimator is two-sided (floor
- * `ceil(codePoints/4)`, ceiling charging every non-ASCII code point a
- * full token), and only the floor clearing the window is a fact - the
- * undecided band between the two bounds stays embedded exactly as before
- * this feature existed, because holding there would act on a guess. A
+ * "Provably" is the whole unit. The hold acts on the conservative bound
+ * (`tokenHoldFloor`: a code point charged at no more than a twelfth of a
+ * token), because the chars-over-four average is an average, not a bound
+ * - real tokenizers absorb 4.5-9 code points per token on prose, tables
+ * and code, so the average read whole English chunks that fit the window
+ * as over it and held them from embedding for nothing. Everything between
+ * that bound and the average estimate is the undecided band: reported by
+ * the window census, still embedded, exactly as before this feature. A
* model with no declared window (outside the curated preset table and no
- * operator `embedding_input_window_tokens`) proves nothing, so nothing
- * is held for it; declaring the window is what arms the hold.
+ * operator `embedding_input_window_tokens`) proves nothing, so nothing is
+ * held for it; declaring the window is what arms the hold.
*
* The hold is a FILTER before batching, so whatever the provider does
* receive still answers 1:1 (the batch invariant in `runEmbeddingPhase`
@@ -31,7 +34,11 @@ import {
RECOMMENDED_EMBEDDING_MODEL,
} from "../../../src/core/search/embeddings/presets.ts";
import { chunkHoldExclusion } from "../../../src/core/search/embedding-spend.ts";
-import { signatureIdentityKnown } from "../../../src/core/search/embeddings/signature.ts";
+import {
+ signatureIdentityKnown,
+ textExtent,
+ tokenHoldFloor,
+} from "../../../src/core/search/embeddings/signature.ts";
import {
charLengthOverTokenBudget,
estimateTokens,
@@ -55,14 +62,43 @@ const UNCURATED_MODEL = "acme/custom-embed-9000";
/** Comfortably under any window. */
const FITTING = "small chunk of ordinary prose";
-/** The LOW estimate already exceeds the window: provably over. */
-const OVER_CONTENT = "b".repeat(charLengthOverTokenBudget(WINDOW) + 1);
-const OVER_FLOOR = Math.ceil(OVER_CONTENT.length / 4);
+/**
+ * An ordinary English sentence, assembled into fixtures of an exact
+ * code-point size. Prose, not a padded run: the hold's proof reads code
+ * points, and a fixture shaped like the text the phase actually sees is
+ * what keeps these tests honest about which chunks fit a real tokenizer.
+ */
+const PROSE_SENTENCE =
+ "The survey team recorded every measurement in the ledger before the storm reached the harbour wall. ";
+
+function proseOfCodePoints(codePoints: number): string {
+ return PROSE_SENTENCE.repeat(Math.ceil(codePoints / PROSE_SENTENCE.length)).slice(0, codePoints);
+}
+
+/**
+ * The held fixture: even at the conservative bound - one token per
+ * twelve code points, far below what this prose costs a real tokenizer -
+ * it cannot fit the window. A run of dense ASCII would prove the same
+ * thing about the estimator and nothing about prose; this proves the
+ * hold on text shaped like the corpus.
+ */
+const OVER_CONTENT = proseOfCodePoints(WINDOW * 12 + 1);
+const HELD_TOKENS = tokenHoldFloor(textExtent(OVER_CONTENT));
+
+/**
+ * The undecided band, Latin half: ordinary English prose whose average
+ * estimate (`ceil(codePoints / 4)`) reads over the window while the
+ * conservative bound clears it comfortably. The estimator this feature
+ * shipped with HELD this chunk; real tokenizers cost it far fewer tokens
+ * than the average claims, and it is embedded.
+ */
+const BAND_CONTENT = proseOfCodePoints(WINDOW * 6);
/**
- * The undecided band: floor `ceil(2048/4)` equals the window (fits), the
- * ceiling charges each Cyrillic code point a full token (far over). The
- * census reports this band; the hold must never act on it.
+ * The undecided band, non-Latin half: the conservative bound fits (the
+ * hold never acts on it) while the ceiling charges each Cyrillic code
+ * point a full token (far over). The census reports this band; the hold
+ * must never act on it.
*/
const UNDECIDED_CONTENT = "ะต".repeat(charLengthOverTokenBudget(WINDOW));
@@ -181,7 +217,7 @@ const EMPTY_TALLY = (): EmbeddingPhaseTally => ({ embeddingsComputed: 0, embeddi
test("a provably-over chunk is never sent while fitting and undecided-band chunks are", async () => {
const sent = await captureProvider();
- const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, UNDECIDED_CONTENT]);
+ const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, BAND_CONTENT, UNDECIDED_CONTENT]);
const tally = EMPTY_TALLY();
try {
await runEmbeddingPhase(store, cfg(), tally, {});
@@ -189,18 +225,19 @@ test("a provably-over chunk is never sent while fitting and undecided-band chunk
await store.close();
}
expect(sent).toContain(FITTING);
+ expect(sent).toContain(BAND_CONTENT);
expect(sent).toContain(UNDECIDED_CONTENT);
expect(sent).not.toContain(OVER_CONTENT);
// One vector per input survived the filter: the phase would have thrown
- // on a count mismatch, and the two texts it did send are stored.
- expect(tally.embeddingsComputed).toBe(2);
+ // on a count mismatch, and the three texts it did send are stored.
+ expect(tally.embeddingsComputed).toBe(3);
const heldId = chunkIdByLength(OVER_CONTENT.length);
expect(embeddedChunkIds()).not.toContain(heldId);
});
-test("the held content reaches no vector store: embeddings rows exactly match what was sent", async () => {
+test("held prose reaches no vector store: embeddings rows exactly match what was sent", async () => {
await captureProvider();
- const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, UNDECIDED_CONTENT]);
+ const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, BAND_CONTENT, UNDECIDED_CONTENT]);
try {
await runEmbeddingPhase(store, cfg(), EMPTY_TALLY(), {});
} finally {
@@ -208,15 +245,41 @@ test("the held content reaches no vector store: embeddings rows exactly match wh
}
const ids = embeddedChunkIds().toSorted();
expect(ids).toEqual(
- [chunkIdByLength(FITTING.length), chunkIdByLength(UNDECIDED_CONTENT.length)].toSorted(),
+ [
+ chunkIdByLength(FITTING.length),
+ chunkIdByLength(BAND_CONTENT.length),
+ chunkIdByLength(UNDECIDED_CONTENT.length),
+ ].toSorted(),
);
});
+test("English prose the average estimate would hold fits the window and is embedded", async () => {
+ // The defect that moved the hold to a conservative bound: this chunk's
+ // chars-over-four estimate reads over the window - the old trigger -
+ // while the conservative bound clears it, and real tokenizers cost
+ // prose far fewer tokens than the average claims. Holding it was
+ // semantic recall lost on an arithmetic guess.
+ expect(estimateTokens([BAND_CONTENT])).toBeGreaterThan(WINDOW);
+ expect(tokenHoldFloor(textExtent(BAND_CONTENT))).toBeLessThanOrEqual(WINDOW);
+
+ const sent = await captureProvider();
+ const store = await openSeeded(cfg(), [BAND_CONTENT]);
+ const tally = EMPTY_TALLY();
+ try {
+ await runEmbeddingPhase(store, cfg(), tally, {});
+ } finally {
+ await store.close();
+ }
+ expect(sent).toContain(BAND_CONTENT);
+ expect(tally.embeddingsHeld).toBeUndefined();
+ expect(tally.embeddingsComputed).toBe(1);
+});
+
// โ”€โ”€ the tally โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
-test("the phase records each held chunk with its floor estimate, the window and the reason", async () => {
+test("the phase records each held chunk with its bound, the window and the reason", async () => {
await captureProvider();
- const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, UNDECIDED_CONTENT]);
+ const store = await openSeeded(cfg(), [FITTING, OVER_CONTENT, BAND_CONTENT]);
const tally = EMPTY_TALLY();
try {
await runEmbeddingPhase(store, cfg(), tally, {});
@@ -227,7 +290,7 @@ test("the phase records each held chunk with its floor estimate, the window and
expect(tally.heldChunks).toEqual([
{
chunkId: chunkIdByLength(OVER_CONTENT.length),
- estimatedTokens: OVER_FLOOR,
+ estimatedTokens: HELD_TOKENS,
windowTokens: WINDOW,
reason: CHUNK_HOLD_REASON.overDeclaredWindow,
},
diff --git a/tests/core/search/entity-alias.test.ts b/tests/core/search/entity-alias.test.ts
index 0631fc2c..8e21c73a 100644
--- a/tests/core/search/entity-alias.test.ts
+++ b/tests/core/search/entity-alias.test.ts
@@ -14,9 +14,11 @@ import { join } from "node:path";
import { indexVault, search } from "../../../src/core/search/index.ts";
import { makeConfig, createTempVault } from "../../helpers/search-fixtures.ts";
import {
+ DERIVED_PERSON_AMBIGUITY_CODE,
expandQueryEntities,
PERSON_NICKNAMES_ARTIFACT_REL,
} from "../../../src/core/search/entity-alias.ts";
+import { RETRIEVAL_DETAIL_IDENTIFIER } from "../../../src/core/search/retrieval-trail.ts";
import { archiveEntity, upsertEntity } from "../../../src/core/brain/entities/registry.ts";
let tmp: ReturnType<typeof createTempVault>;
@@ -57,6 +59,15 @@ function writeNicknames(entries: Record<string, string[]>): void {
writeFileSync(join(tmp.vault, PERSON_NICKNAMES_ARTIFACT_REL), JSON.stringify(entries));
}
+/** One note the keyword lane can match, so the pipeline reaches ranking. */
+function writeMatchableNote(): void {
+ mkdirSync(join(tmp.vault, "notes"), { recursive: true });
+ writeFileSync(
+ join(tmp.vault, "notes", "a.md"),
+ "---\ntitle: A\n---\n\nR Smith planned the payment flow for the vault project this week.\n",
+ );
+}
+
describe("expandQueryEntities", () => {
test("query naming an alias gains the canonical name forms", () => {
seedRegistry();
@@ -208,6 +219,79 @@ describe("person-variant expansion", () => {
expect(out.added).toEqual([]);
expect(out.derived).toEqual([]);
});
+
+ test("a form two people share only through derivation expands nothing and is reported", () => {
+ // The design's rule for derived ambiguity: reported, never guessed.
+ // Neither Robert joins the expansion - adding both people's declared
+ // names would be a guess that the query means all of them.
+ const smith = seedPerson("Robert Smith");
+ const jones = seedPerson("Robert Jones");
+ writeNicknames({ bob: ["Robert Smith", "Robert Jones"] });
+ const out = expandQueryEntities(tmp.vault, ["bob"]);
+ expect(out.expanded).toEqual(["bob"]);
+ expect(out.added).toEqual([]);
+ expect(out.derived).toEqual([]);
+ expect(out.sourceIds).toEqual([]);
+ expect(out.derivedAmbiguities).toEqual([{ form: "bob", ids: [smith, jones].toSorted() }]);
+ });
+
+ test("an initial form two people share is not a guess either", () => {
+ // "r smith" is derived by BOTH persons (initial-plus-surname), with
+ // no declared claimant anywhere: ambiguity, not expansion.
+ seedPerson("Robert Smith");
+ seedPerson("Richard Smith");
+ const out = expandQueryEntities(tmp.vault, ["r smith"]);
+ expect(out.added).toEqual([]);
+ expect(out.derived).toEqual([]);
+ expect(out.derivedAmbiguities.map((a) => a.form)).toEqual(["r smith"]);
+ expect(out.derivedAmbiguities[0]!.ids).toHaveLength(2);
+ });
+
+ test("a unique form expands beside a reported ambiguity", () => {
+ // Robert and Richard share "r smith" only through derivation - that
+ // form is reported and expands nothing - while "bob" is uniquely
+ // Robert Smith's: the unique form still expands Robert's declared
+ // name, and Richard (reachable only through the ambiguous form)
+ // contributes nothing.
+ const smith = seedPerson("Robert Smith");
+ seedPerson("Richard Smith");
+ writeNicknames({ bob: ["Robert Smith"] });
+ const out = expandQueryEntities(tmp.vault, ["bob", "r smith"]);
+ expect(out.derivedAmbiguities.map((a) => a.form)).toEqual(["r smith"]);
+ expect(out.derivedAmbiguities[0]!.ids).toHaveLength(2);
+ expect(out.added).toEqual(["robert smith"]);
+ expect(out.sourceIds).toEqual([smith]);
+ });
+
+ test("the dotted-initial spelling expands the plain-named person", () => {
+ const id = seedPerson("Ada Lovelace");
+ const out = expandQueryEntities(tmp.vault, ["A. Lovelace"]);
+ expect(out.added).toEqual(["ada lovelace"]);
+ // The query's dot survives in its own raw form, so the plain
+ // initial spelling is a derived addition here, not a query form.
+ expect(out.derived).toEqual(["a lovelace", "lovelace ada"]);
+ expect(out.sourceIds).toEqual([id]);
+ });
+
+ test("the comma spelling expands through the declared word order", () => {
+ const id = seedPerson("Ada Lovelace");
+ const out = expandQueryEntities(tmp.vault, ["Lovelace, Ada"]);
+ // The canonical word order is new query content (the raw query only
+ // carried the comma spelling), so it joins `added`; the derivation's
+ // structural forms follow in `derived`.
+ expect(out.added).toEqual(["ada lovelace"]);
+ expect(out.derived).toEqual(["a lovelace", "lovelace ada"]);
+ expect(out.sourceIds).toEqual([id]);
+ });
+
+ test("a two-letter operator nickname expands the person it stands for", () => {
+ seedPerson("Alan Lovelace");
+ writeNicknames({ al: ["Alan Lovelace"] });
+ const out = expandQueryEntities(tmp.vault, ["al"]);
+ expect(out.added).toEqual(["alan lovelace"]);
+ expect(out.derived).toEqual(["a lovelace", "lovelace alan"]);
+ expect(out.derivedAmbiguities).toEqual([]);
+ });
});
describe("alias-aware search boost", () => {
@@ -255,3 +339,44 @@ describe("alias-aware search boost", () => {
expect(out.results[0]!.reasons.some((x) => x.includes("ent-"))).toBe(false);
});
});
+
+describe("derived-ambiguity trail reporting", () => {
+ test("an ambiguous derived query names the form and the competing ids on the trail", async () => {
+ const richard = seedPerson("Richard Smith");
+ const robert = seedPerson("Robert Smith");
+ writeMatchableNote();
+ const config = makeConfig({ vault: tmp.vault, dbPath: tmp.dbPath, maxHops: 0 });
+ await indexVault(config, {});
+
+ // "R Smith" is derived by BOTH persons and declared by neither, so
+ // the expansion adds nothing - and the trail says so instead of the
+ // answer reading as a quiet miss.
+ const out = await search(config, { query: "R Smith payment", limit: 10 });
+ const entries = (out.retrievalTrail?.degraded ?? []).filter(
+ (d) => d.code === DERIVED_PERSON_AMBIGUITY_CODE,
+ );
+ expect(entries).toHaveLength(1);
+ const ids = [richard, robert].toSorted().join(".");
+ expect(entries[0]!.detail).toEqual({ "r smith": ids });
+ // The detail values stay identifiers, under the trail's own rule.
+ for (const value of Object.values(entries[0]!.detail ?? {})) {
+ expect(value).toMatch(RETRIEVAL_DETAIL_IDENTIFIER);
+ }
+ });
+
+ test("an unambiguous query leaves no derived-ambiguity entry on the trail", async () => {
+ seedPerson("Richard Smith");
+ seedPerson("Robert Smith");
+ writeMatchableNote();
+ const config = makeConfig({ vault: tmp.vault, dbPath: tmp.dbPath, maxHops: 0 });
+ await indexVault(config, {});
+
+ // "Robert Smith" is a declared form of exactly one person: it expands,
+ // and no ambiguity is reported.
+ const out = await search(config, { query: "Robert Smith payment", limit: 10 });
+ const entries = (out.retrievalTrail?.degraded ?? []).filter(
+ (d) => d.code === DERIVED_PERSON_AMBIGUITY_CODE,
+ );
+ expect(entries).toEqual([]);
+ });
+});
diff --git a/tests/core/search/entity-variant-rank.test.ts b/tests/core/search/entity-variant-rank.test.ts
index 3de6c264..d367c5b0 100644
--- a/tests/core/search/entity-variant-rank.test.ts
+++ b/tests/core/search/entity-variant-rank.test.ts
@@ -76,13 +76,10 @@ describe("derived entity ranking lane (unit)", () => {
});
test("declared, neutral and derived-only candidates sort transitively", () => {
- // A declared chunk with a tiny base score, a derived-only chunk with
- // a large one, and a neutral chunk in between: a comparator that
- // answers declared-before-derived BEFORE comparing scores is not a
- // strict weak ordering (A<B, B<C, C<A), so the sorted output depends
- // on the input order. The ranking must be a total order - declared,
- // then neutral, then derived-only, score within each tier - the same
- // for every input permutation.
+ // Three lanes, three scores: the composite score orders every pair
+ // on its own, so the sort is a total order - the same for every
+ // input permutation - with the declared/derived tier reserved for
+ // exact ties.
const inputsFor = (ids: number[]) => {
const bm25 = new Map([
[1, -1],
@@ -103,7 +100,39 @@ describe("derived entity ranking lane (unit)", () => {
const forward = idsOf(rankResults(inputsFor([1, 2, 3]), RANK_OPTS));
const backward = idsOf(rankResults(inputsFor([3, 2, 1]), RANK_OPTS));
expect(forward).toEqual(backward);
- expect(forward).toEqual([1, 3, 2]);
+ // The derived-only chunk carries the best bm25, the neutral chunk
+ // the middle one, and the declared chunk - whose boost cannot
+ // rescue a much weaker base - the worst.
+ expect(forward).toEqual([2, 3, 1]);
+ });
+
+ test("a derived-only chunk with the highest score stays first, and a low-scoring declared chunk does not jump ahead", () => {
+ // The regression the tier-before-score comparator made inevitable on
+ // both ends: the derived-only hit the feature exists for was sunk
+ // below every neutral chunk (and off the limit cut), while any
+ // single declared match floated above every higher-scoring chunk.
+ const ranked = rankResults(
+ {
+ ...baseInputs(),
+ keyword: [
+ { chunkId: 1, documentId: 10, bm25: -9 },
+ { chunkId: 2, documentId: 11, bm25: -5 },
+ { chunkId: 3, documentId: 12, bm25: -1 },
+ ] as KeywordHit[],
+ hydrated: new Map([
+ [1, hyd(1, 10)],
+ [2, hyd(2, 11)],
+ [3, hyd(3, 12)],
+ ]),
+ entityMatchByChunk: new Map([[3, 1]]),
+ derivedEntityMatchByChunk: new Map([[1, 1]]),
+ },
+ RANK_OPTS,
+ );
+ // Score order end to end: the derived-only best match, then the
+ // neutral middle, then the declared chunk whose weak base its boost
+ // cannot lift.
+ expect(ranked.map((r) => r.chunkId)).toEqual([1, 2, 3]);
});
test("the derived lane is capped strictly below the declared cap", () => {
@@ -168,31 +197,46 @@ describe("derived entity ranking lane (unit)", () => {
expect(withEmpty.map((r) => r.score)).toEqual(without.map((r) => r.score));
});
- test("two otherwise-equal chunks, one with a single declared match, one with three derived matches: declared ranks first for several entityMul values", () => {
- for (const entityMul of [0.5, 1.0, 2.0]) {
- for (const [declaredId, derivedId] of [
- [1, 2],
- [2, 1],
- ] as const) {
- const ranked = rankResults(
- {
- ...baseInputs(),
- entityMatchByChunk: new Map([[declaredId, 1]]),
- derivedEntityMatchByChunk: new Map([[derivedId, 3]]),
- },
- {
- ...RANK_OPTS,
- weightProfile: { keywordMul: 1, semanticMul: 1, recencyMul: 1, entityMul },
- },
- );
- const declaredIndex = ranked.findIndex((r) => r.chunkId === declaredId);
- const derivedIndex = ranked.findIndex((r) => r.chunkId === derivedId);
- expect(declaredIndex).toBeGreaterThanOrEqual(0);
- expect(derivedIndex).toBeGreaterThanOrEqual(0);
- expect(declaredIndex).toBeLessThan(derivedIndex);
- }
+ test("one declared match against three derived matches: the score decides, the tier only breaks an exact tie", () => {
+ // Identical base scores for both chunks. At entityMul 2 the declared
+ // cap (0.04) tops the derived cap (0.02): declared first by score. At
+ // entityMul 1 both lanes land on 0.02 - an exact tie the tier
+ // break resolves toward declared. At entityMul 0.5 the three derived
+ // matches (0.015) outweigh the one declared match (0.01): the
+ // derived chunk is simply the higher-scoring one.
+ for (const [entityMul, firstId] of [
+ [2.0, 1],
+ [1.0, 1],
+ [0.5, 2],
+ ] as const) {
+ const ranked = rankResults(
+ {
+ ...baseInputs(),
+ entityMatchByChunk: new Map([[1, 1]]),
+ derivedEntityMatchByChunk: new Map([[2, 3]]),
+ },
+ {
+ ...RANK_OPTS,
+ weightProfile: { keywordMul: 1, semanticMul: 1, recencyMul: 1, entityMul },
+ },
+ );
+ expect(ranked[0]!.chunkId).toBe(firstId);
}
});
+
+ test("an exact score tie between a declared and a derived-only chunk resolves toward declared", () => {
+ // One declared match and two derived matches both land on 0.02: an
+ // exact tie the tier break is the only thing left to separate, and
+ // it separates declared first.
+ const inputs = {
+ ...baseInputs(),
+ entityMatchByChunk: new Map([[1, 1]]),
+ derivedEntityMatchByChunk: new Map([[2, 2]]),
+ };
+ const ranked = rankResults(inputs, RANK_OPTS);
+ expect(ranked[0]!.chunkId).toBe(1);
+ expect(ranked[0]!.score).toBeCloseTo(ranked[1]!.score, 12);
+ });
});
let tmp: ReturnType<typeof createTempVault>;
diff --git a/tests/core/search/pending-vectors.test.ts b/tests/core/search/pending-vectors.test.ts
index 32878a7b..a5270b04 100644
--- a/tests/core/search/pending-vectors.test.ts
+++ b/tests/core/search/pending-vectors.test.ts
@@ -28,10 +28,13 @@
import { afterEach, beforeEach, expect, test } from "bun:test";
import { Database } from "bun:sqlite";
-import { mkdirSync, writeFileSync } from "node:fs";
-import { dirname } from "node:path";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { tmpdir } from "node:os";
-import { indexCheck, indexVault } from "../../../src/core/search/indexer.ts";
+import { indexCheck, indexVault, runEmbeddingPhase } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import { chunkHoldExclusion } from "../../../src/core/search/embedding-spend.ts";
import {
countChunksWithoutEmbeddings,
findChunksWithoutEmbeddings,
@@ -85,6 +88,38 @@ function cfg(semantic: Partial<ResolvedEmbeddingConfig> = {}) {
});
}
+/**
+ * A temporary directory whose config file records a decline, for the
+ * `search check` consent arm.
+ */
+function fixtureDeclinedConfig(): {
+ readonly configPath: string;
+ readonly cleanup: () => void;
+} {
+ const dir = mkdtempSync(join(tmpdir(), "osb-pending-declined-"));
+ const configPath = join(dir, "config.yaml");
+ writeFileSync(configPath, "search_semantic_enabled: false\n");
+ return { configPath, cleanup: () => rmSync(dir, { recursive: true, force: true }) };
+}
+
+/**
+ * The consent env twins are saved and CLEARED around a test that resolves
+ * the consent from the process environment: env values take precedence
+ * over the config file, so a developer or CI runner exporting either of
+ * them would flip the case no matter what the fixture writes to disk.
+ */
+function saveAndClearConsentEnv(): () => void {
+ const names = ["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC", "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER"];
+ const saved = names.map((name) => [name, process.env[name]] as const);
+ for (const name of names) delete process.env[name];
+ return () => {
+ for (const [name, value] of saved) {
+ if (value === undefined) delete process.env[name];
+ else process.env[name] = value;
+ }
+ };
+}
+
test("the count agrees with the row walk it replaces", async () => {
writeMd(vault, "a.md", "# A\n\nFirst note.");
writeMd(vault, "b.md", "# B\n\nSecond note.");
@@ -194,6 +229,116 @@ test("a partially embedded index names the count and the verb that prices the wo
expect(report.recommendations.some((r) => r.includes("compute the first vectors"))).toBe(false);
});
+test("an index whose vector-less chunks are all held reports none pending and no backfill", async () => {
+ if (!sqliteVecLoadable()) return;
+ // A narrow operator-declared window arms the hold for every chunk of
+ // ordinary prose: the phase records the holds and embeds nothing, so
+ // the index owes no vector for any of them.
+ const PROSE =
+ "The survey team recorded every measurement in the ledger before the storm reached the harbour wall. ";
+ const heldWindow = cfg({ inputWindowTokens: 8 });
+ {
+ const store = await Store.open(heldWindow, { mode: "write" });
+ try {
+ const docId = store.upsertDocument({
+ path: "Brain/held.md",
+ title: "held",
+ contentHash: "h-held",
+ mtime: 1,
+ size: PROSE.length,
+ });
+ store.replaceChunks(docId, [
+ {
+ chunkIndex: 0,
+ content: PROSE,
+ contentHash: "h-held-0",
+ startLine: 1,
+ endLine: 1,
+ tokenCount: 9,
+ },
+ ]);
+ } finally {
+ await store.close();
+ }
+ }
+ {
+ const store = await Store.open(heldWindow, { mode: "write" });
+ try {
+ await runEmbeddingPhase(store, heldWindow, {
+ embeddingsComputed: 0,
+ embeddingsRetries: 0,
+ });
+ } finally {
+ await store.close();
+ }
+ }
+
+ // The key-free diagnostic read still sees the missing vector: the chunk
+ // genuinely has none, which the hold does not erase.
+ const hold = chunkHoldExclusion(heldWindow);
+ expect(hold).not.toBeNull();
+ {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ expect(countChunksWithoutEmbeddings(db)).toBe(1);
+ expect(countChunksWithoutEmbeddings(db, undefined, hold ?? undefined)).toBe(0);
+ } finally {
+ db.close();
+ }
+ }
+ const peeked = peekPendingVectorsSync(dbPath, hold ?? undefined);
+ expect(peeked.kind).toBe("read");
+ if (peeked.kind === "read") {
+ expect(peeked.value.pending).toBe(0);
+ expect(peeked.value.chunks).toBeGreaterThan(0);
+ }
+
+ // Declaring the model's price keeps the price hint out of a report
+ // whose only job here is the backfill recommendation.
+ const priced = cfg({
+ inputWindowTokens: 8,
+ priceOverride: { model: "fake-model", usdPerMtok: 0 },
+ });
+ const report = await indexCheck(priced);
+ expect(report.pendingVectors.verdict).toBe("measured");
+ if (report.pendingVectors.verdict === "measured") {
+ expect(report.pendingVectors.pending).toBe(0);
+ expect(report.pendingVectors.chunks).toBeGreaterThan(0);
+ }
+ // The recommendation never clears when it is driven by the raw count:
+ // vector-backfill excludes the held chunks, finds nothing to do, and
+ // the same state would be re-reported on every check.
+ expect(report.recommendations.some((r) => r.includes("vector-backfill"))).toBe(false);
+ expect(report.recommendations.some((r) => r.includes("compute the first vectors"))).toBe(false);
+});
+
+test("a declined vault reports the probe as skipped with the decline, never as unconfigured", async () => {
+ if (!sqliteVecLoadable()) return;
+ const f = fixtureDeclinedConfig();
+ const restoreConsentEnv = saveAndClearConsentEnv();
+ try {
+ const config = Object.freeze({
+ ...makeConfig({
+ vault,
+ dbPath,
+ semantic: { enabled: false },
+ }),
+ configPath: f.configPath,
+ });
+ const report = await indexCheck(config);
+ // A decline is a decision recorded in the machine configuration, and
+ // the probe names it: skipped, with the consent's own sentence, on
+ // the same state an absent configuration used to read as
+ // not-configured.
+ expect(report.providerProbe).toBe("skipped");
+ expect(report.providerReason ?? "").toContain("provider consent declined");
+ expect(report.providerReason ?? "").toContain("search_semantic_enabled=true");
+ } finally {
+ restoreConsentEnv();
+ f.cleanup();
+ }
+});
+
test("an index that could not be read recommends building it and says so", async () => {
if (!sqliteVecLoadable()) return;
writeMd(vault, "a.md", "# A\n\nA note.");
diff --git a/tests/core/search/provider-consent.test.ts b/tests/core/search/provider-consent.test.ts
index 81618e5e..9682ef06 100644
--- a/tests/core/search/provider-consent.test.ts
+++ b/tests/core/search/provider-consent.test.ts
@@ -259,6 +259,57 @@ describe("the provider-consent resolver", () => {
}
});
+ test("the re-enable advice names only the keys this decline used, with the value that clears each", () => {
+ const f = fixture("consent-reenable-precise");
+ try {
+ // The switch alone: the advice is the switch's clearing value, and
+ // it does NOT name the provider key - the two keys decline
+ // independently, so an operator who followed advice naming both
+ // could clear one and still be refused by the other.
+ writeFileSync(f.configPath, "search_semantic_enabled: false\n");
+ const bySwitch = resolveProviderConsent({ configPath: f.configPath, env: {} });
+ expect(bySwitch.detail).toContain(`re-enable it with ${PROVIDER_CONSENT_KEY.semantic}=true`);
+ expect(bySwitch.detail).not.toContain(`${PROVIDER_CONSENT_KEY.provider}=<provider>`);
+
+ // The provider name alone: mirror image.
+ writeFileSync(f.configPath, "embedding_provider: disabled\n");
+ const byProvider = resolveProviderConsent({ configPath: f.configPath, env: {} });
+ expect(byProvider.detail).toContain(
+ `re-enable it with ${PROVIDER_CONSENT_KEY.provider}=<provider>`,
+ );
+ expect(byProvider.detail).not.toContain(`${PROVIDER_CONSENT_KEY.semantic}=true`);
+
+ // Both at once: one advice line naming both keys and both values.
+ writeFileSync(f.configPath, "search_semantic_enabled: false\nembedding_provider: disabled\n");
+ const byBoth = resolveProviderConsent({ configPath: f.configPath, env: {} });
+ expect(byBoth.detail).toContain(
+ `re-enable it with ${PROVIDER_CONSENT_KEY.semantic}=true or ` +
+ `${PROVIDER_CONSENT_KEY.provider}=<provider>`,
+ );
+ } finally {
+ f.cleanup();
+ }
+ });
+
+ test("an unreadable record advises repairing the file, not flipping a key", () => {
+ // A DIRECTORY in the config's place, the same platform-safe fixture
+ // the unreadable arm above uses: the record named no key, so the
+ // advice cannot be a key to clear - and the reason the record could
+ // not be read rides on the verdict beside the file to repair.
+ const f = fixture("consent-unreadable-advice");
+ try {
+ mkdirSync(f.configPath, { recursive: true });
+ const consent = resolveProviderConsent({ configPath: f.configPath, env: {} });
+ expect(providerConsentIsDeclined(consent)).toBe(true);
+ expect(consent.unreadableReason).not.toBeNull();
+ expect(consent.detail).toContain("repair the configuration file at");
+ expect(consent.detail).toContain(f.configPath);
+ expect(consent.detail).not.toContain("re-enable it with");
+ } finally {
+ f.cleanup();
+ }
+ });
+
test("every verdict carries a non-empty detail", () => {
const f = fixture("consent-detail");
try {
diff --git a/tests/core/search/query-embed.test.ts b/tests/core/search/query-embed.test.ts
index 87252dcd..cc6ce6df 100644
--- a/tests/core/search/query-embed.test.ts
+++ b/tests/core/search/query-embed.test.ts
@@ -43,7 +43,6 @@ import {
LOCAL_EMBEDDING_MODEL,
textExtent,
tokenEstimateCeiling,
- tokenEstimateFloor,
} from "../../../src/core/search/embeddings/signature.ts";
const PREFIX = "query: ";
@@ -91,9 +90,10 @@ test("a Han query is cut under the ceiling bound, never the floor", () => {
expect(fit.truncated).toBe(true);
expect(ceiling(PREFIX + fit.text)).toBeLessThanOrEqual(window);
expect(ceiling(PREFIX + fit.text + nextCodePoint(query, fit.text)!)).toBeGreaterThan(window);
- // The floor would have admitted roughly four times as much Han text.
+ // The average-rate estimate would have admitted roughly four times as
+ // much Han text.
const floorCut = [...query].slice(0, (window - 2) * 4).join("");
- expect(tokenEstimateFloor(textExtent(PREFIX + floorCut))).toBeLessThanOrEqual(window);
+ expect(estimateTokens([PREFIX + floorCut])).toBeLessThanOrEqual(window);
expect([...fit.text].length).toBeLessThan([...floorCut].length);
});
diff --git a/tests/core/search/retrieval-trail.test.ts b/tests/core/search/retrieval-trail.test.ts
index f2d9ed8d..1c434184 100644
--- a/tests/core/search/retrieval-trail.test.ts
+++ b/tests/core/search/retrieval-trail.test.ts
@@ -80,6 +80,21 @@ test("the rerank liveness code is kebab-case and its sentence names no provider"
expect(sentence).not.toMatch(RERANK_VENDOR_NAMES);
});
+test("the derived person ambiguity code narrows and its detail keys carry the form", () => {
+ expect(RETRIEVAL_DEGRADATION.personDerivedAmbiguity).toBe("person-derived-ambiguity");
+ expect(isRetrievalDegradationCode("person-derived-ambiguity")).toBe(true);
+ const sentence = describeRetrievalDegradation(RETRIEVAL_DEGRADATION.personDerivedAmbiguity);
+ expect(sentence.length).toBeGreaterThan(0);
+ // The detail maps the ambiguous form (the key - a name form is not an
+ // identifier) to the competing person ids dot-joined (the value - ids
+ // never carry a dot, so the value stays an identifier under the rule).
+ const ids = ["ent-people-richard-smith", "ent-people-robert-smith"];
+ const detail = { "r smith": ids.join(".") };
+ for (const value of Object.values(detail)) {
+ expect(value).toMatch(RETRIEVAL_DETAIL_IDENTIFIER);
+ }
+});
+
test("the guard rejects a near miss", () => {
expect(isRetrievalDegradationCode(RETRIEVAL_DEGRADATION.hybridDegraded)).toBe(true);
expect(isRetrievalDegradationCode("hybrid_degraded")).toBe(false);
diff --git a/tests/core/search/schema-migration-v10.test.ts b/tests/core/search/schema-migration-v10.test.ts
index 4211ea2c..ac807840 100644
--- a/tests/core/search/schema-migration-v10.test.ts
+++ b/tests/core/search/schema-migration-v10.test.ts
@@ -37,11 +37,12 @@ function hasColumn(db: Database, table: string, name: string): boolean {
.some((c) => c.name === name);
}
-// v15 (write-honesty): the chunk_embedding_holds table joined the latest
-// version (v14 (recall-quality) had added the virtual chunks.chunk_kind
-// column, the taxonomy indexes and the document_attributes side table).
-test("latest schema version is 15", () => {
- expect(LATEST_SCHEMA_VERSION).toBe(15);
+// v16 (privacy-marker visibility restamp): the marker-aware visibility
+// re-measurement joined (v15 (chunk skip-and-hold) had added the
+// chunk_embedding_holds table; v14 the chunk_kind column, taxonomy
+// indexes and document_attributes side table).
+test("latest schema version is 16", () => {
+ expect(LATEST_SCHEMA_VERSION).toBe(16);
});
test("fresh migration reaches latest with the authored_at column", () => {
diff --git a/tests/core/search/schema-v14.test.ts b/tests/core/search/schema-v14.test.ts
index 15d7fbfc..b2dd2687 100644
--- a/tests/core/search/schema-v14.test.ts
+++ b/tests/core/search/schema-v14.test.ts
@@ -214,10 +214,11 @@ function attributesFor(path: string): ReadonlyArray<{ key: string; value: string
}
describe("the v14 entry", () => {
- test("remains in the frozen array once the v15 entry appends after it", () => {
- expect(LATEST_SCHEMA_VERSION).toBe(15);
- expect(MIGRATIONS.length).toBe(15);
- expect(MIGRATIONS[MIGRATIONS.length - 2]!.version).toBe(14);
+ test("remains in the frozen array once later entries append after it", () => {
+ expect(LATEST_SCHEMA_VERSION).toBe(16);
+ expect(MIGRATIONS.length).toBe(16);
+ expect(MIGRATIONS[MIGRATIONS.length - 2]!.version).toBe(15);
+ expect(MIGRATIONS[MIGRATIONS.length - 3]!.version).toBe(14);
});
test("runs from a rewound v13 index and stamps the version", async () => {
@@ -226,7 +227,7 @@ describe("the v14 entry", () => {
expect(migrate()).toBe(LATEST_SCHEMA_VERSION);
const db = new Database(dbPath, { readonly: true });
try {
- expect(readSchemaVersion(db)).toBe(15);
+ expect(readSchemaVersion(db)).toBe(LATEST_SCHEMA_VERSION);
} finally {
db.close();
}
diff --git a/tests/core/search/schema-v15.test.ts b/tests/core/search/schema-v15.test.ts
index 35787cc8..e0bf5d7d 100644
--- a/tests/core/search/schema-v15.test.ts
+++ b/tests/core/search/schema-v15.test.ts
@@ -91,10 +91,11 @@ function tableSql(): string | null {
}
describe("the v15 entry", () => {
- test("appends to MIGRATIONS as the latest version", () => {
- expect(LATEST_SCHEMA_VERSION).toBe(15);
- expect(MIGRATIONS.length).toBe(15);
- expect(MIGRATIONS[MIGRATIONS.length - 1]!.version).toBe(15);
+ test("appends to MIGRATIONS ahead of the later v16 entry", () => {
+ expect(LATEST_SCHEMA_VERSION).toBe(16);
+ expect(MIGRATIONS.length).toBe(16);
+ expect(MIGRATIONS[14]!.version).toBe(15);
+ expect(MIGRATIONS[MIGRATIONS.length - 1]!.version).toBe(16);
});
test("runs from a rewound v14 index and stamps the version", async () => {
@@ -103,7 +104,7 @@ describe("the v15 entry", () => {
expect(migrate()).toBe(LATEST_SCHEMA_VERSION);
const db = new Database(dbPath, { readonly: true });
try {
- expect(readSchemaVersion(db)).toBe(15);
+ expect(readSchemaVersion(db)).toBe(LATEST_SCHEMA_VERSION);
} finally {
db.close();
}
diff --git a/tests/core/search/schema-v16.test.ts b/tests/core/search/schema-v16.test.ts
new file mode 100644
index 00000000..359ca6ca
--- /dev/null
+++ b/tests/core/search/schema-v16.test.ts
@@ -0,0 +1,179 @@
+/**
+ * Schema v16 (privacy-marker visibility restamp): the one-pass
+ * re-measurement of `documents.visibility` for rows whose chunk-zero
+ * frontmatter carries an informal privacy marker key.
+ *
+ * The column has existed since v12, and the v12 backfill ran ONCE, for
+ * `IS NULL` rows. An index migrated to v12-v14 by a pre-marker binary
+ * holds marker pages stamped from the declared `visibility:` alone - a
+ * non-NULL `[]` - and the incremental indexer skips an unchanged file, so
+ * no later run corrects them. Consumers reading the column alone
+ * undercount exactly the pages the column exists to make countable.
+ *
+ * The migration is DDL-free: the schema it produces is v15's, so the
+ * manifest check cannot see it. Its effects are only in row values, and
+ * those are what the tests below pin: a marker row re-stamped with the
+ * reserved token, a row already carrying the marker-aware value
+ * unchanged, a formal row untouched, and a NULL row (no chunk zero)
+ * left NULL.
+ */
+
+import { Database } from "bun:sqlite";
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import {
+ DOCUMENT_VISIBILITY_COLUMN,
+ LATEST_SCHEMA_VERSION,
+ MIGRATIONS,
+ applyMigrations,
+ encodeDocumentVisibility,
+ readSchemaVersion,
+} from "../../../src/core/search/schema.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+const PRE_V16_SCHEMA_VERSION = 15;
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ ({ vault, dbPath, cleanup } = createTempVault("schema-v16"));
+});
+afterEach(() => cleanup());
+
+async function seedVault(): Promise<void> {
+ writeMd(vault, "marked.md", "---\naccess: private\n---\n\nMarked heron counts.\n");
+ writeMd(vault, "formal.md", "---\nvisibility: [team]\n---\n\nTeam runbook.\n");
+ writeMd(vault, "plain.md", "# Plain\n\nNothing declared.\n");
+ await indexVault(makeConfig({ vault, dbPath }));
+}
+
+/** Stamp the column as a pre-marker binary wrote it, then rewind to v15. */
+function rewindWithPreMarkerStamps(stamps: Readonly<Record<string, string>>): void {
+ const db = new Database(dbPath);
+ try {
+ const set = db.query<never, [string, string]>(
+ `UPDATE documents SET ${DOCUMENT_VISIBILITY_COLUMN} = ?1 WHERE path = ?2`,
+ );
+ for (const [path, value] of Object.entries(stamps)) set.run(value, path);
+ db.query("UPDATE index_state SET value = ? WHERE key = 'schema_version'").run(
+ String(PRE_V16_SCHEMA_VERSION),
+ );
+ } finally {
+ db.close();
+ }
+}
+
+function migrate(): number {
+ const db = new Database(dbPath);
+ try {
+ return applyMigrations(db);
+ } finally {
+ db.close();
+ }
+}
+
+/** The visibility column of one document, by its path. */
+function columnOf(path: string): string | null {
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ return (
+ db
+ .query<{ visibility: string | null }, [string]>(
+ `SELECT ${DOCUMENT_VISIBILITY_COLUMN} AS visibility FROM documents WHERE path = ?1`,
+ )
+ .get(path)?.visibility ?? null
+ );
+ } finally {
+ db.close();
+ }
+}
+
+describe("the v16 entry", () => {
+ test("appends to MIGRATIONS as the latest version and adds no DDL", () => {
+ expect(LATEST_SCHEMA_VERSION).toBe(16);
+ expect(MIGRATIONS.length).toBe(16);
+ expect(MIGRATIONS[MIGRATIONS.length - 1]!.version).toBe(16);
+ });
+
+ test("runs from a rewound v15 index and stamps the version", async () => {
+ await seedVault();
+ rewindWithPreMarkerStamps({ "marked.md": "[]" });
+ expect(migrate()).toBe(LATEST_SCHEMA_VERSION);
+ const db = new Database(dbPath, { readonly: true });
+ try {
+ expect(readSchemaVersion(db)).toBe(16);
+ } finally {
+ db.close();
+ }
+ });
+});
+
+describe("the marker restamp", () => {
+ test("a pre-marker-stamped marker row gains the reserved token", async () => {
+ await seedVault();
+ // What a pre-marker binary wrote: the declared `visibility:` alone.
+ rewindWithPreMarkerStamps({ "marked.md": encodeDocumentVisibility([]) });
+ migrate();
+ expect(columnOf("marked.md")).toBe(encodeDocumentVisibility(["private"]));
+ });
+
+ test("a formal row and a plain row are left exactly as they were", async () => {
+ await seedVault();
+ rewindWithPreMarkerStamps({
+ "formal.md": encodeDocumentVisibility(["team"]),
+ "plain.md": encodeDocumentVisibility([]),
+ });
+ migrate();
+ expect(columnOf("formal.md")).toBe(encodeDocumentVisibility(["team"]));
+ expect(columnOf("plain.md")).toBe(encodeDocumentVisibility([]));
+ });
+
+ test("a marker row already carrying the marker-aware value is unchanged", async () => {
+ await seedVault();
+ // The current indexer already writes the marker-aware value; a
+ // re-run over it is a no-op.
+ rewindWithPreMarkerStamps({ "marked.md": encodeDocumentVisibility(["private"]) });
+ migrate();
+ expect(columnOf("marked.md")).toBe(encodeDocumentVisibility(["private"]));
+ });
+
+ test("a capitalized marker key on the page is caught too", async () => {
+ writeMd(vault, "cased.md", "---\nAccess: private\n---\n\nCased claim.\n");
+ writeMd(vault, "plain.md", "# Plain\n\nNothing declared.\n");
+ await indexVault(makeConfig({ vault, dbPath }));
+ rewindWithPreMarkerStamps({ "cased.md": encodeDocumentVisibility([]) });
+ migrate();
+ expect(columnOf("cased.md")).toBe(encodeDocumentVisibility(["private"]));
+ });
+
+ test("a row with no chunk zero stays NULL even when a marker was hand-stamped before", async () => {
+ await seedVault();
+ const db = new Database(dbPath);
+ try {
+ // There is nothing to measure once the chunk rows are gone: the
+ // honest answer stays "the index holds nothing", not a guess.
+ db.query(
+ "DELETE FROM chunks WHERE document_id = (SELECT id FROM documents WHERE path = ?1)",
+ ).run("marked.md");
+ db.exec(`UPDATE documents SET ${DOCUMENT_VISIBILITY_COLUMN} = NULL WHERE path = 'marked.md'`);
+ db.query("UPDATE index_state SET value = ? WHERE key = 'schema_version'").run(
+ String(PRE_V16_SCHEMA_VERSION),
+ );
+ } finally {
+ db.close();
+ }
+ migrate();
+ expect(columnOf("marked.md")).toBeNull();
+ });
+
+ test("idempotent: migrating a current index changes nothing", async () => {
+ await seedVault();
+ expect(migrate()).toBe(LATEST_SCHEMA_VERSION);
+ const before = [columnOf("marked.md"), columnOf("formal.md"), columnOf("plain.md")];
+ expect(migrate()).toBe(LATEST_SCHEMA_VERSION);
+ expect([columnOf("marked.md"), columnOf("formal.md"), columnOf("plain.md")]).toEqual(before);
+ });
+});
diff --git a/tests/scripts/launcher-bootstrap.test.ts b/tests/scripts/launcher-bootstrap.test.ts
new file mode 100644
index 00000000..f9e2b610
--- /dev/null
+++ b/tests/scripts/launcher-bootstrap.test.ts
@@ -0,0 +1,347 @@
+/**
+ * The launchers' dependency bootstrap (issue #260).
+ *
+ * A Codex marketplace add/upgrade replaces the checkout wholesale, so the
+ * tree can arrive with package.json and bun.lock but WITHOUT node_modules.
+ * The CLI then cannot start (`Cannot find package 'proper-lockfile'` from
+ * src/core/config.ts), and `o2b install-cli` cannot repair anything because
+ * it runs through the same import. The fix lives in the shared launcher
+ * preamble: `scripts/_bun-precheck.sh` (bash) and the `.cmd` twins probe for
+ * the one runtime dependency with a file stat and run
+ * `bun install --frozen-lockfile` once when it is missing.
+ *
+ * Three layers are pinned here:
+ *
+ * - POSIX unit tests source `scripts/_bun-precheck.sh` against a stub `bun`
+ * that logs its arguments, so the probe's decision is observable without
+ * importing the CLI at all: a cold tree installs once, a warm tree never,
+ * a failed install degrades to a named manual command, and the
+ * O2B_SKIP_DEP_BOOTSTRAP seam opts out.
+ * - A Windows unit test drives `scripts/o2b.cmd` through cmd.exe with the
+ * same kind of stub, following tests/scripts/windows-launchers.test.ts.
+ * - The end-to-end regression copies the launcher-relevant tree (src, hooks,
+ * scripts, package.json, bun.lock, bunfig.toml) WITHOUT node_modules into
+ * a temp directory, runs the real launcher with an isolated HOME and an
+ * empty bun cache, and asserts the CLI answers `--version` - a real
+ * entry point, no model credentials, and only the network the
+ * `bun install` itself needs. `install-cli` then works from the same
+ * checkout, which is the order the documented Codex flow relies on.
+ */
+import { afterEach, describe, expect, setDefaultTimeout, test } from "bun:test";
+import { spawnSync } from "node:child_process";
+import {
+ chmodSync,
+ cpSync,
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { CLI_SPAWN_BUDGET_MS } from "../helpers/cli-timeout.ts";
+import { homeEnv, IS_WINDOWS } from "../helpers/platform.ts";
+
+setDefaultTimeout(CLI_SPAWN_BUDGET_MS);
+
+const REPO = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..");
+const VERSION = (
+ JSON.parse(readFileSync(join(REPO, "package.json"), "utf8")) as { version: string }
+).version;
+
+/** The failure the launcher must never let surface (issue #260's symptom). */
+const MODULE_NOT_FOUND = "Cannot find package";
+
+const tmps: string[] = [];
+function tempDir(prefix: string): string {
+ const dir = mkdtempSync(join(tmpdir(), prefix));
+ tmps.push(dir);
+ return dir;
+}
+
+afterEach(() => {
+ while (tmps.length) {
+ try {
+ rmSync(tmps.pop()!, { recursive: true, force: true });
+ } catch {
+ // Best-effort cleanup: a temp dir some process still holds must not
+ // fail the suite (Windows refuses to delete open directories).
+ }
+ }
+});
+
+/**
+ * The tree a cold marketplace clone holds: the launcher-relevant sources and
+ * the manifests, and deliberately no node_modules. The full src and hooks
+ * trees are needed because `--version` imports the whole CLI graph, which
+ * reaches hooks/lib from src.
+ */
+function coldTree(): string {
+ const tree = tempDir("o2b-boot-tree-");
+ cpSync(join(REPO, "src"), join(tree, "src"), { recursive: true });
+ cpSync(join(REPO, "hooks"), join(tree, "hooks"), { recursive: true });
+ cpSync(join(REPO, "scripts"), join(tree, "scripts"), { recursive: true });
+ for (const name of ["package.json", "bun.lock", "bunfig.toml"]) {
+ cpSync(join(REPO, name), join(tree, name));
+ }
+ rmSync(join(tree, "node_modules"), { recursive: true, force: true });
+ return tree;
+}
+
+/**
+ * Just the launcher script under test in a scripts/ directory. The unit
+ * layers stub bun out, so no sources and no manifests are needed - the
+ * decision under test is the probe, and a full copy would spend seconds
+ * per test on files the stub never reads.
+ */
+function minimalTree(scriptName: string): string {
+ const tree = tempDir("o2b-boot-mini-");
+ mkdirSync(join(tree, "scripts"), { recursive: true });
+ cpSync(join(REPO, "scripts", scriptName), join(tree, "scripts", scriptName));
+ return tree;
+}
+
+/** Plant the probe's marker: the file the bootstrap stats on the warm path. */
+function plantDependencies(tree: string): void {
+ const marker = join(tree, "node_modules", "proper-lockfile");
+ mkdirSync(marker, { recursive: true });
+ writeFileSync(join(marker, "package.json"), '{"name":"proper-lockfile"}\n');
+}
+
+// โ”€โ”€ POSIX: the shared preamble sourced against a stub bun โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+// Everything `_bun-precheck.sh` reaches for that is not a shell builtin.
+// The child's PATH is built from exactly these plus the stub, so no real
+// Bun on this machine can answer for the tree under test.
+const REQUIRED_UTILITIES = ["bash", "cat", "tr", "head"];
+
+/** A `bun` whose arguments are logged and whose version check succeeds. */
+function stubBun(bin: string, options: { failInstall?: boolean } = {}): void {
+ const install = options.failInstall ? "exit 1" : ":";
+ const stub = join(bin, "bun");
+ writeFileSync(
+ stub,
+ [
+ "#!/bin/sh",
+ `printf '%s\\n' "$*" >> "$BUN_CALL_LOG"`,
+ `if [ "$1" = "--version" ]; then printf '%s\\n' "1.4.3"; else ${install}; fi`,
+ "",
+ ].join("\n"),
+ );
+ chmodSync(stub, 0o755);
+}
+
+/** A PATH holding the preamble's utilities and provably no real bun. */
+function isolatedPath(): string {
+ const bin = tempDir("o2b-boot-bin-");
+ for (const name of REQUIRED_UTILITIES) {
+ const resolved = Bun.which(name);
+ if (!resolved) throw new Error(`test prerequisite missing from this machine: ${name}`);
+ symlinkSync(resolved, join(bin, name));
+ }
+ if (Bun.which("bun", { PATH: bin })) throw new Error("isolated PATH leaked a bun");
+ return bin;
+}
+
+interface PrecheckRun {
+ readonly code: number;
+ /** Both streams: the assertions look for notices wherever they printed. */
+ readonly output: string;
+ /** One entry per stub-bun invocation, in order. */
+ readonly calls: string[];
+}
+
+async function sourcePrecheck(
+ tree: string,
+ options: { failInstall?: boolean; skipBootstrap?: boolean } = {},
+): Promise<PrecheckRun> {
+ const bin = isolatedPath();
+ stubBun(bin, options);
+ const log = join(tempDir("o2b-boot-log-"), "bun-calls.log");
+ const env: Record<string, string> = {
+ HOME: tempDir("o2b-boot-home-"),
+ PATH: bin,
+ REPO_ROOT: tree,
+ BUN_CALL_LOG: log,
+ ...(options.skipBootstrap ? { O2B_SKIP_DEP_BOOTSTRAP: "1" } : {}),
+ };
+ const proc = Bun.spawn(
+ ["bash", "-c", `. "${join(tree, "scripts", "_bun-precheck.sh")}"; echo PRECHECK_PASSED`],
+ { env, stdout: "pipe", stderr: "pipe" },
+ );
+ const [stderr, code, stdout] = await Promise.all([
+ new Response(proc.stderr).text(),
+ proc.exited,
+ new Response(proc.stdout).text(),
+ ]);
+ const calls = existsSync(log) ? readFileSync(log, "utf8").split("\n").filter(Boolean) : [];
+ return { code: code ?? -1, output: stderr + stdout, calls };
+}
+
+describe.skipIf(IS_WINDOWS)("launcher bootstrap decision (scripts/_bun-precheck.sh)", () => {
+ test("a cold checkout runs the frozen install exactly once and then passes", async () => {
+ const tree = minimalTree("_bun-precheck.sh");
+ const run = await sourcePrecheck(tree);
+ expect(run.code).toBe(0);
+ expect(run.output).toContain("bun install --frozen-lockfile");
+ expect(run.calls.filter((call) => call.startsWith("install"))).toHaveLength(1);
+ expect(run.calls.some((call) => call.startsWith("--version"))).toBe(true);
+ });
+
+ test("a checkout with the dependency tree present adds no bootstrap", async () => {
+ const tree = minimalTree("_bun-precheck.sh");
+ plantDependencies(tree);
+ const run = await sourcePrecheck(tree);
+ expect(run.code).toBe(0);
+ expect(run.calls.some((call) => call.startsWith("install"))).toBe(false);
+ // The only Bun work left is the version check the precheck always does.
+ expect(run.calls).toEqual(["--version"]);
+ });
+
+ test("a failed bootstrap exits nonzero and names the manual command", async () => {
+ const tree = minimalTree("_bun-precheck.sh");
+ const run = await sourcePrecheck(tree, { failInstall: true });
+ expect(run.code).not.toBe(0);
+ expect(run.output).toContain("bun install --frozen-lockfile");
+ expect(run.output).toContain("then retry");
+ });
+
+ test("O2B_SKIP_DEP_BOOTSTRAP opts a cold checkout out of the install", async () => {
+ const tree = minimalTree("_bun-precheck.sh");
+ const run = await sourcePrecheck(tree, { skipBootstrap: true });
+ expect(run.code).toBe(0);
+ expect(run.calls.some((call) => call.startsWith("install"))).toBe(false);
+ });
+});
+
+// โ”€โ”€ Windows: the .cmd twin through cmd.exe, with a logging bun stub โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+/** A bun.cmd whose arguments are logged; o2b.cmd finds it first on PATH. */
+function stubBunCmd(bin: string, log: string): void {
+ writeFileSync(join(bin, "bun.cmd"), `@echo off\r\necho %*>>"${log}"\r\n`);
+}
+
+function runO2bCmd(
+ tree: string,
+ args: ReadonlyArray<string>,
+ env: Record<string, string>,
+): { status: number | null; stdout: string; stderr: string } {
+ const comspec = process.env["ComSpec"] || "cmd.exe";
+ const proc = spawnSync(comspec, ["/d", "/c", join(tree, "scripts", "o2b.cmd"), ...args], {
+ env,
+ encoding: "utf8",
+ timeout: CLI_SPAWN_BUDGET_MS,
+ windowsHide: true,
+ });
+ return { status: proc.status, stdout: proc.stdout ?? "", stderr: proc.stderr ?? "" };
+}
+
+function stubEnv(log: string): Record<string, string> {
+ const bin = tempDir("o2b-boot-cmdbin-");
+ stubBunCmd(bin, log);
+ const env: Record<string, string> = {};
+ for (const [name, value] of Object.entries(process.env)) {
+ if (value !== undefined) env[name] = value;
+ }
+ env["BUN_CALL_LOG"] = log;
+ delete env["O2B_SKIP_DEP_BOOTSTRAP"];
+ // The stub dir first, so the launcher's `where bun` and its later bare
+ // `bun` invocations both answer the stub.
+ env["PATH"] = `${bin};${process.env["PATH"] ?? ""}`;
+ return env;
+}
+
+function readCalls(log: string): string[] {
+ return existsSync(log) ? readFileSync(log, "utf8").split("\r\n").filter(Boolean) : [];
+}
+
+describe.skipIf(!IS_WINDOWS)("launcher bootstrap decision (scripts/o2b.cmd)", () => {
+ test("a cold checkout runs the frozen install exactly once and then passes", () => {
+ const tree = minimalTree("o2b.cmd");
+ const log = join(tempDir("o2b-boot-cmdlog-"), "bun-calls.log");
+ const run = runO2bCmd(tree, ["--version"], stubEnv(log));
+ expect(run.status).toBe(0);
+ const installs = readCalls(log).filter((call) => call.startsWith("install"));
+ expect(installs).toHaveLength(1);
+ expect(installs[0]).toContain("--frozen-lockfile");
+ });
+
+ test("a checkout with the dependency tree present adds no bootstrap", () => {
+ const tree = minimalTree("o2b.cmd");
+ plantDependencies(tree);
+ const log = join(tempDir("o2b-boot-cmdlog-"), "bun-calls.log");
+ const run = runO2bCmd(tree, ["--version"], stubEnv(log));
+ expect(run.status).toBe(0);
+ expect(readCalls(log).some((call) => call.startsWith("install"))).toBe(false);
+ });
+});
+
+// โ”€โ”€ End to end: the real launcher, a real CLI start, no node_modules โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
+
+describe("cold marketplace checkout self-heals on the first launcher run", () => {
+ let tree: string;
+
+ /**
+ * The real launcher, isolated HOME, empty bun cache - the reporter's repro.
+ *
+ * The child's TMPDIR is pointed at a temp entry of this file's own: bun's
+ * native-module staging (node-gyp) scratches in the ambient temp directory
+ * during a cold install, and the suite's leftover guard fails the run for
+ * anything a child abandons there (tests/setup.ts). Scratch lives and dies
+ * with the afterEach sweep below.
+ */
+ function runLauncher(args: ReadonlyArray<string>) {
+ const home = tempDir("o2b-boot-e2e-home-");
+ const scratch = tempDir("o2b-boot-e2e-scratch-");
+ const env: Record<string, string> = { ...homeEnv(home) };
+ for (const [name, value] of Object.entries(process.env)) {
+ if (value !== undefined) env[name] = value;
+ }
+ env["BUN_INSTALL_CACHE_DIR"] = tempDir("o2b-boot-e2e-cache-");
+ env["npm_config_devdir"] = join(scratch, ".node-gyp");
+ for (const name of ["TMPDIR", "TMP", "TEMP"] as const) env[name] = scratch;
+ delete env["O2B_SKIP_DEP_BOOTSTRAP"];
+ if (IS_WINDOWS) {
+ const comspec = process.env["ComSpec"] || "cmd.exe";
+ return spawnSync(comspec, ["/d", "/c", join(tree, "scripts", "o2b.cmd"), ...args], {
+ env,
+ encoding: "utf8",
+ timeout: CLI_SPAWN_BUDGET_MS,
+ windowsHide: true,
+ });
+ }
+ return spawnSync("bash", [join(tree, "scripts", "o2b"), ...args], {
+ env,
+ encoding: "utf8",
+ timeout: CLI_SPAWN_BUDGET_MS,
+ });
+ }
+
+ test("a fresh checkout starts the real CLI and then serves install-cli", () => {
+ tree = coldTree();
+
+ // The core regression: the launcher runs, the bootstrap installs once,
+ // and the CLI answers - a real entry point, not manifest validation.
+ const cold = runLauncher(["--version"]);
+ expect(cold.status).toBe(0);
+ expect(cold.stdout).toContain(VERSION);
+ expect(cold.stderr).not.toContain(MODULE_NOT_FOUND);
+ // The self-heal is what made it pass: the notice ran and the tree the
+ // probe wanted now exists.
+ expect(cold.stderr).toContain("bun install --frozen-lockfile");
+ expect(existsSync(join(tree, "node_modules", "proper-lockfile", "package.json"))).toBe(true);
+
+ // The documented Codex flow's next step, from the same now-warm checkout.
+ const bin = tempDir("o2b-boot-e2e-bin-");
+ const install = runLauncher(["install-cli", "--bindir", bin]);
+ expect(install.status).toBe(0);
+ expect(install.stderr ?? "").not.toContain(MODULE_NOT_FOUND);
+ // POSIX writes symlinks, Windows writes .cmd shims; both are exit 0.
+ expect(existsSync(join(bin, IS_WINDOWS ? "o2b.cmd" : "o2b"))).toBe(true);
+ });
+});
diff --git a/uv.lock b/uv.lock
index 6228650a..c5b81737 100644
--- a/uv.lock
+++ b/uv.lock
@@ -4,5 +4,5 @@ requires-python = ">=3.11"
[[package]]
name = "open-second-brain"
-version = "1.92.0"
+version = "1.93.0"
source = { editable = "." }