Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat(store): shard quarantine, partition leave, private reach for log readers and doctor (#294)
Serg Eroshe7187c393a30b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a18389cf..9c164ab5 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [Unreleased]++### Added++- **Brain store hardening.** Readers that show log events, or numbers derived from them, to a caller below local reach now withhold events of the `private` partition, and a census test fails on any new log reader that has not been classified. `brain_doctor` names each shard that reads only in part by kind, offset and repair command, and `o2b store shard quarantine|list|release` skips one bad line on this machine without touching the shard (a line of a newer format is never skipped, a pending blob only after seven days and never in `settings`). `o2b store partition leave <partition>` takes a partition off this machine on purpose, and the error that names a lost partition names the verb. A `lane-entry` record keeps the file's text in an optional `raw` field when its fields and text do not reproduce it, so a lane file written as text reads back byte for byte, also after a rollback; the format version does not change.+## [1.98.0] - 2026-11-02Open Second Brain 1.98.0 is the agent-presence and self-improvement release: the vault states its usage where the agents that read it actually look, and its skills improve only under evidence. The install path closes its own gap - `o2b install --target <host> --apply` now writes the brain-usage block into the agent-instruction file the installed host reads at the vault root, through a shared managed-block writer that preserves every byte outside the block, refuses a hand-edited block by name, and records its ownership so uninstall removes exactly the pair the tool wrote. The dream pass gains an operator-approved exit into those same files: a lesson promotes into a managed lessons block only when the full qualification holds and a fresh approval pins the approved text, every non-member carries one named blocked reason, and the doctor grades every block state it can name. The skill lane gains a benchmark-gated rewrite loop where a skill is rewritten only after a candidate beats the incumbent on scored dev tasks with full coverage and survives the one held-out check - Open Second Brain never calls a model, the calling agent executes the task envelopes, the scorer reads structural predicates and nothing the agent claims about its own work, the accept stays an operator step on the command line, and the incumbent's bytes ride a content-restoring rollback journal until the sequence settles. And ChatGPT desktop is named where its operators look for it: it reads the shared Codex config, and the detect output and install docs say so.diff --git a/docs/brain-store.md b/docs/brain-store.mdindex 21521aa4..5cac87f7 100644--- a/docs/brain-store.md+++ b/docs/brain-store.md@@ -199,6 +199,23 @@ from there to the other trusted machines:o2b store init --partitions shared,private```+### Leaving a partition++A machine that held a partition refuses every store command once that+partition's `store.json` is gone: the held partitions are sticky, and the+refusal names the two remedies. To take the partition off on purpose,+stop syncing its folder, remove the folder, and run+`o2b store partition leave <partition>`; it clears this machine's held+marker for the partition, its seen-shard entries and the digest caches,+and the search index drops its records on its next catch-up. It refuses+`shared` (leaving it is leaving the store), and it refuses, changing+nothing, while `shared/store.json` is missing or unreadable, so restore+that file first. It also refuses while the partition's `store.json` is+still present, and it refuses while the folder still holds files, unless+`--force` is given; the files then stay on disk and this machine ignores+them. A `store.json` that returns by sync or a restore makes+the machine hold the partition again.+## SyncingOpen Second Brain never syncs anything itself. Use any file sync tool:@@ -575,6 +592,76 @@ follows old instructions), doctor reports `legacy-tree-reappeared` with thewriter when a file names it. Nothing reads that folder; stop the writer andmove the folder out of the vault.+## Shards read only in part++A shard of a held partition can read only in part: a line in its middle is+not a valid record, a record is of a format this release does not know, or+a record's body blob has not arrived. A record in such a gap may be the one+that makes a page private or sets a setting, so every read that decides a+protection fails closed on this machine until the gap is gone: capture,+settings reads and writes, lane privacy, exact state, attestations, the OKF+export and the migration. A line cut short by a crash and closed by the+device's next append is not a gap.++The doctor names each gap, with the shard as `<partition>/<shard path>` and+the byte offset of the line:++- `store-shard-corrupt-line` (error): a bad line in the middle. Repair it at+ its source when you can; otherwise the finding names the command that+ quarantines it on this machine.+- `store-shard-line-too-long` (error): a bad line in the middle too long+ for a quarantine to name. Repair or remove it at its source; no command+ is named.+- `store-shard-newer-format` (warning): upgrade Open Second Brain on this+ machine, which then reads the record. Such a line is never quarantined:+ it is a valid record.+- `store-shard-blob-pending` (warning, error once the record is seven days+ old): a sync tool may still deliver the blob, so it is quarantined only+ from seven days on, and never in a `settings` shard, whose record may be+ a setting in force: deliver that blob from the machine that wrote it.+ When the record's clock stamp does not parse as a time, its age is+ unknown: the finding is an error that says so, and the record is never+ quarantined by age.++`o2b store shard quarantine <partition>/<shard> --offset <n>` records, in a+ledger kept in this machine's device-local state, the line's offset and the+hash of its bytes. The shard then reads as whole on this machine; the line+stays in the file and no other machine is affected. The entry matches only+those bytes at that offset: a different line written there, or another bad+line of the same shard, is a gap again. The last line of a shard this+machine writes is refused, because its writer may still be appending. A+quarantined record whose blob arrives later reads with its body, with no+step of yours. The rollback, the late import of the transition, the+migration preflight and the legacy removal still refuse while any record+waits for its blob, quarantined or not. One exception:+`o2b brain migrate-store --remove-legacy --force` goes past a shard read+only in part, a quarantined or pending blob included, as it goes past a+cohort device still on the legacy layout. That risks data loss: a record+in that shard may have the legacy tree as its only readable copy, and the+tree leaves the vault (it is kept in the dated archive described under+"Removing the legacy tree"). A shard that cannot be read at all still+refuses, even with `--force`. Prefer letting the blob arrive or repairing+the line before forcing the removal.++A quarantine cannot name a line longer than the hash cap (sixty-four times+the store line cap, 1 MiB): the verb refuses it as `line-too-long`, so the+doctor reports such a line as `store-shard-line-too-long` (error) and names+no command; repair or remove it at its source, in the shard of the device+that wrote it. The quarantine ledger is read only when it is of the version+this release writes; a ledger of another version, or one that does not+parse, quarantines nothing on this machine, so every line it named is a gap+again until the ledger is repaired or removed.++`o2b store shard list` prints every entry with its state: `held` while the+same bytes still stand there, `resolved` once the line reads as a record,+`stale` once other bytes stand at the offset.+`o2b store shard release <partition>/<shard> --offset <n>` removes the+entry's mark. The line is a gap again only if it still is one: then the+protection reads fail closed again, while a `resolved` entry (its blob has+arrived) leaves the shard whole. When another process holds the ledger's+lock past the wait budget, `quarantine` and `release` refuse with+`ledger-busy` and change nothing; run the command again.+## What the partitions protect, and what they do notThe partitions decide **which machines receive which records**. They are a@@ -621,8 +708,8 @@ that come from the store, the migration and the view:| Code | What it means | What to do || --- | --- | --- || `store-missing` | No store exists at the configured root. | Run `o2b init` (or `o2b store init`) on the first machine; on other machines check `store_root` and wait for sync. |-| `store-root-invalid` | The store root cannot be used: it is inside the vault, or holds partitions of two different stores. | Point `store_root` outside the vault, and keep only one store's partitions under it. |-| `store-partition-lost` | A partition this machine has held (it opened or initialised it here) no longer shows its `store.json`. The store is not opened: every protection fails closed and every store write is refused, because the partition's records cannot be read. | Restore that partition's `store.json` from sync or a backup. No command takes a partition off this machine's list. |+| `store-root-invalid` | The store root cannot be used: it is inside the vault, a partition's `store.json` does not parse, or it holds partitions of two different stores. A caller below local reach (a remote MCP client) sees only that the store could not be opened, without the file or the store ids. | Point `store_root` outside the vault, keep only one store's partitions under it, and restore a `store.json` that does not parse from sync or a backup. |+| `store-partition-lost` | A partition this machine has held (it opened or initialised it here) no longer shows its `store.json`. The store is not opened: every protection fails closed and every store write is refused, because the partition's records cannot be read. | Restore that partition's `store.json` from sync or a backup. To stop holding `private` on this machine on purpose, stop syncing its folder, remove the folder, then run `o2b store partition leave private` (it needs `shared/store.json`, so restore `shared` first when both are lost); `shared` can only be restored. || `store-device-state-foreign` | The device-local state directory holds entries of other devices: it was put into a sync folder. | Remove it from the sync tool, sync only `shared/` and `private/`, then delete the other devices' entries in it. || `machine-state-adopt-conflicts` | When this machine switched, two fresh vaults held a machine-state file with different bytes; the first vault's copy is live and the other is kept in the `adopt-conflicts/` folder the finding names, with the file count. | Compare each kept file with the live copy at the same path, copy over what is still needed, then delete the folder. || `store-mode-loose` | Store files or folders are readable by other accounts on this machine. | Run the `chmod -R go-rwx` command the finding prints. |@@ -635,8 +722,8 @@ that come from the store, the migration and the view:| `store-device-retired-returned` | A device you retired wrote a record after its retirement, so it is on the digest roster again. | Nothing to do if the machine is back on purpose (migrate it if it still keeps the legacy tree). If a new machine reused the id, give it its own `device_id` and retire the old id again. || `store-clock-ahead` | Records carry a clock more than a day ahead of now. | Fix the clock of the device the finding names. || `store-device-dir-reserved` | A peer's device id is a name Windows reserves (such as `con`), so its folder cannot exist on Windows. | Give that peer another `device_id` in its machine config. |-| `store-write-fail-open` | Writes from hooks or telemetry did not land (no store yet, no device id, a missing partition); their callers went on. | Fix the cause the finding names, then delete the counter file `store-write.json` in the `stores` state directory (`~/.local/state/open-second-brain/stores/` on Linux) to reset it. |-| `store-dual-write-failures`, `store-dual-write-skipped`, `store-dual-write-no-store` | Only on a machine running the phase 1 dual-write trial (`store_dual_write`): store copies of legacy writes failed or were skipped. | Fix the cause, or migrate the machine and turn the trial off. |+| `store-write-fail-open` | Writes from hooks or telemetry did not land (no store yet, no device id, a missing partition); their callers went on. A caller below local reach (a remote MCP client) sees only the failed writes known to go to `shared/`: a failure that may concern `private/` shows no count, time, error text or path there. | Fix the cause the finding names, then delete the counter file `store-write.json` in the `stores` state directory (`~/.local/state/open-second-brain/stores/` on Linux) to reset it. |+| `store-dual-write-failures`, `store-dual-write-skipped`, `store-dual-write-no-store` | Only on a machine running the phase 1 dual-write trial (`store_dual_write`): store copies of legacy writes failed or were skipped. Below local reach the failures follow the rule of `store-write-fail-open`, and the skipped writes are reported without their count. | Fix the cause, or migrate the machine and turn the trial off. || `migration-available` | This machine still keeps its brain in the vault's `Brain/` tree, or it switched before this vault was configured and the vault's legacy tree was never imported. | Follow [Moving an existing install](#moving-an-existing-install); on a switched machine run `o2b brain migrate-store`, which imports the tree and binds the vault. || `legacy-devices-pending` | This machine switched, but the named devices still write the legacy tree. | Migrate each named machine, and retire the ones that are gone with `o2b store device retire <device-id>`. || `legacy-tree-reappeared` | A `Brain/` folder is back in the vault after the legacy tree was removed. | Stop the writer the finding names and move the folder out of the vault. |diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex eb599a33..0c33ba2c 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -23,6 +23,9 @@ o2b state status|migrate|rollbacko2b store init Create the brain store once, on the first machine: both partitions with one store id, or --partitions shared for a machine that must not hold private records; refuses an existing store (exit 1), completes a half-created one with its own store id, and refuses a partition folder that holds store files without store.json; --config, --json (see "Brain store dual writes" in how-it-works.md)o2b store path Print where the brain store lives without creating anything: the store root, each partition (held or not), the device-local state directory, and the human view's target folder, each with its origin (env, config or default; `none` for a view target that is not set) and whether it exists; a store that does not exist yet still prints the resolved paths, says so and names `o2b store init` (exit 0, like `o2b search status` on a missing index), a store root that cannot be used (inside the vault, a corrupt store.json, partitions of two stores) prints the paths and the reason (exit 1); --config, --json (`root`, `initialized`, `store_id`, `partitions`, `device_state_dir`, `view_target` (with its own `error` when the folder lies inside the store), `error`, and `next_command` while the store is missing)o2b store device list|retire The devices the brain store knows: `list` prints each device id with its last record in a partition this machine holds, its state (`switched` to the store, `legacy` layout, or `retired`), whether it is on the digest roster (a record in the last 30 days and not retired) and which one is this machine; `retire <device-id> [--reason <text>]` writes a human `device-retire` record in the shared settings, which takes a machine that is gone for good off the digest roster (`store-digest-silent`) and out of the devices `o2b brain migrate-store --remove-legacy` waits for. Retiring this machine's own id is refused, and so is a reason that names a private page and an id the store has no record of (a typo) unless `--force` is given; retiring a retired device writes nothing. A retired device that writes again is on the roster again and doctor reports `store-device-retired-returned`. --config, --json. Exit 0 for an answer, 1 for a refusal (no store, no device id, own id, unknown id, private reason), 2 for a usage error; under --json each refusal and usage error has its own `reason`+o2b store partition leave Take a partition off this machine on purpose, once its folder no longer carries store.json (stop syncing it and remove the folder first): removes this machine's held marker for it, clears its seen-shard entries and the digest caches, and the search index drops its records on its next catch-up. Refuses shared, a partition whose store.json is still there, a folder that still holds files unless --force or that cannot be listed, and a store whose shared/store.json is missing or unreadable (restore it first; nothing is changed); while the partition's store.json is still there the still-held refusal comes first, and only then does leaving a partition this machine does not hold exit 0 and change nothing. A store.json that returns by sync or a restore makes the machine hold the partition again. --config, --json. Exit 0 for an answer, 1 for a refusal (shared, still-held, files-left, no-store-id, clear-failed, a store root inside the vault, an unreadable config), 2 for a usage error+o2b store shard quarantine|list|release+ One bad line of a held shard, skipped on this machine only: `quarantine <partition>/<shard> --offset <n>` records the line's offset and the hash of its bytes in a device-local ledger, so the shard reads as whole here while the line stays in the file and no other machine is affected; a different line at that offset, or another bad line of the shard, is a gap again. It quarantines a corrupt line (doctor `store-shard-corrupt-line`) and a record whose body blob has not arrived for seven days (`store-shard-blob-pending`), and refuses a line of a newer format, a younger pending blob, a pending blob in a `settings` shard, a pending blob whose clock stamp does not parse as a time, the last line of a shard this machine writes, an offset that holds no gap, a line too long to hash (`line-too-long`, permanent until the shard's bytes change), a line whose bytes changed while they were checked, or a shard removed after it was listed (`shard-changed`, run the command again), a quarantine ledger whose lock another process held past the wait budget (`ledger-busy`, run the command again; also for `release`), a partition this machine does not hold, a shard the store does not list and a shard that is there and cannot be read (`shard-unreadable`). A ledger file this release cannot rewrite (it cannot be read, does not parse, has another version, or holds an entry this release does not read) is refused with `ledger-unreadable` and left as it is. `list` prints each entry with its state (`held`, `resolved` once the line reads as a record, `stale` once other bytes stand at the offset); `release <partition>/<shard> --offset <n>` removes the entry's mark, and the line is a gap again only if it still is one (a `resolved` entry leaves none); releasing an offset with no entry exits 0 and changes nothing. --config, --json. Exit 0 for an answer, 1 for a refusal (no-store, partition-lost, store-open-error, store-root-in-vault, config, not-held, unknown-shard, no-gap, newer-format, blob-too-young, settings-blob, blob-clock-unreadable, own-tail, line-too-long, shard-changed, ledger-unreadable, ledger-busy), 2 for a usage error; under --json each refusal and usage error has its own `reason`o2b view writer|render|recoverThe human view (pages written to the `view_target` folder, or OPEN_SECOND_BRAIN_VIEW_TARGET): `writer` shows the one device that renders it, `writer --here` asks the current writer to hand the role to this machine at its next catch-up, `--force` takes it at once when the old writer is unreachable (it may render once more before the change reaches it); `render` runs one render pass (only the view writer writes pages, and edits made to a page become human records first); `recover <page>` lists the replaced versions of a page from the last 30 days, `--print <id>` prints one; --config, --json. The machine that runs `o2b store init` first becomes the view writer; it renders from its second catch-up after the claim. When two machines create a store at the same moment the lower clock wins once both claims sync, and a loser whose sync is slower than one catch-up may render once before it stops. The view writer also renders at the end of every dream pass and, at most once an hour, from the session-start hooko2b tool-call Invoke an MCP tool handler from the CLIdiff --git a/openclaw/index.js b/openclaw/index.jsindex 0220e6de..93a48ab2 100644--- a/openclaw/index.js+++ b/openclaw/index.js@@ -5137,9 +5137,123 @@ var init_fold_settings = __esm(() => {});});+// src/core/store/shard-quarantine.ts+import { createHash as createHash2 } from "node:crypto";+import { closeSync as closeSync6, readFileSync as readFileSync8, readSync as readSync3, statSync as statSync6 } from "node:fs";+import { join as join12, resolve as resolve6 } from "node:path";+function ledgerPath(root) {+ const key = createHash2("sha256").update(resolve6(root)).digest("hex").slice(0, 32);+ return join12(storesStateDir(), SHARD_QUARANTINE_DIR, `${key}.json`);+}+function stampOf(path) {+ const stat = statSync6(path, { throwIfNoEntry: false });+ return stat === undefined ? "absent" : `${stat.size}:${stat.mtimeMs}`;+}+function isPartition(value) {+ return STORE_PARTITIONS.includes(value);+}+function isLine(value) {+ if (typeof value !== "object" || value === null)+ return false;+ const v = value;+ return isPartition(v["partition"]) && typeof v["shard"] === "string" && typeof v["offset"] === "number" && Number.isSafeInteger(v["offset"]) && typeof v["sha256"] === "string" && (v["kind"] === "corrupt-line" || v["kind"] === "blob-pending") && typeof v["reason"] === "string" && typeof v["quarantined_at"] === "string";+}+function appliedLines(parsed) {+ if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed))+ return [];+ const { version, lines } = parsed;+ if (version !== QUARANTINE_LEDGER_VERSION || !Array.isArray(lines))+ return [];+ return lines.filter(isLine);+}+function load2(root) {+ const path = ledgerPath(root);+ let stamp;+ try {+ stamp = stampOf(path);+ } catch {+ return { root: resolve6(root), lines: [] };+ }+ const hit = loaded.get(path);+ if (hit !== undefined && hit.stamp === stamp)+ return hit.ledger;+ let lines = [];+ if (stamp !== "absent") {+ try {+ lines = appliedLines(JSON.parse(readFileSync8(path, "utf8")));+ } catch {+ lines = [];+ }+ }+ const ledger = { root: resolve6(root), lines };+ loaded.set(path, { stamp, ledger });+ return ledger;+}+function quarantineStamp(root) {+ try {+ return stampOf(ledgerPath(root));+ } catch {+ return "unreadable";+ }+}+function quarantinedLines(partitionRoot, rel) {+ const partition = resolve6(partitionRoot).split(/[\\/]/).pop();+ if (!isPartition(partition))+ return new Map;+ const out = new Map;+ for (const line of load2(resolve6(partitionRoot, "..")).lines) {+ if (line.partition === partition && line.shard === rel) {+ out.set(line.offset, { sha256: line.sha256, kind: line.kind });+ }+ }+ return out;+}+function probeLineHashAtFd(fd, offset) {+ try {+ const hash = createHash2("sha256");+ const chunk = Buffer.alloc(HASH_CHUNK_BYTES);+ let position = offset;+ let read = 0;+ for (;; ) {+ const want = Math.min(chunk.length, QUARANTINE_HASH_CAP_BYTES - read + 1);+ const n = readSync3(fd, chunk, 0, want, position);+ if (n === 0)+ break;+ const end = chunk.subarray(0, n).indexOf(10);+ const take = end === -1 ? n : end + 1;+ if (read + take > QUARANTINE_HASH_CAP_BYTES)+ return { failure: "too-long" };+ hash.update(chunk.subarray(0, take));+ read += take;+ if (end !== -1)+ break;+ position += n;+ }+ return read === 0 ? { failure: "unreadable" } : { sha256: hash.digest("hex") };+ } catch {+ return { failure: "unreadable" };+ }+}+function lineSha256AtFd(fd, offset) {+ return hashOf(probeLineHashAtFd(fd, offset));+}+var SHARD_QUARANTINE_DIR = "shard-quarantine", QUARANTINE_LEDGER_VERSION = 1, DAY_MS2, PENDING_BLOB_QUARANTINE_AGE_MS, PENDING_BLOB_QUARANTINE_AGE_TEXT, loaded, HASH_CHUNK_BYTES, QUARANTINE_HASH_CAP_BYTES, hashOf = (probe) => ("sha256" in probe) ? probe.sha256 : null;+var init_shard_quarantine = __esm(() => {+ init_constants();+ init_hlc();+ init_safe_fs();+ init_store_paths();+ DAY_MS2 = 24 * 60 * 60 * 1000;+ PENDING_BLOB_QUARANTINE_AGE_MS = 7 * DAY_MS2;+ PENDING_BLOB_QUARANTINE_AGE_TEXT = `${PENDING_BLOB_QUARANTINE_AGE_MS / DAY_MS2} days`;+ loaded = new Map;+ HASH_CHUNK_BYTES = 64 * 1024;+ QUARANTINE_HASH_CAP_BYTES = 64 * STORE_LINE_CAP_BYTES;+});+// src/core/store/shard-list.tsimport { readdirSync as readdirSync2 } from "node:fs";-import { join as join12, posix as posix5 } from "node:path";+import { join as join13, posix as posix5 } from "node:path";function isShardFileName(name) {return SHARD_FILE_RE.test(name) && !name.includes(".sync-conflict");}@@ -5159,18 +5273,18 @@ function listShards(partitionRoot, ref) {const segments = ref.ledger === undefined ? [ref.stream] : [ref.stream, ref.ledger];if (ref.ledger !== undefined && !LEDGER_NAME_RE.test(ref.ledger))return [];- const streamDir = join12(partitionRoot, ...segments);+ const streamDir = join13(partitionRoot, ...segments);const out = [];for (const deviceEntry of entries(streamDir)) {if (!deviceEntry.isDirectory() || !isReadableDeviceDir(deviceEntry.name))continue;- for (const file of entries(join12(streamDir, deviceEntry.name))) {+ for (const file of entries(join13(streamDir, deviceEntry.name))) {if (!(file.isFile() || file.isSymbolicLink()) || !isShardFileName(file.name))continue;out.push({device: deviceEntry.name,rel: posix5.join(...segments, deviceEntry.name, file.name),- path: join12(streamDir, deviceEntry.name, file.name)+ path: join13(streamDir, deviceEntry.name, file.name)});}}@@ -5328,8 +5442,8 @@ var init_torn_line = __esm(() => {});// src/core/store/reader.ts-import { basename as basename5, dirname as dirname10 } from "node:path";-import { closeSync as closeSync6, fstatSync as fstatSync5, readSync as readSync3 } from "node:fs";+import { basename as basename5, dirname as dirname10, relative as relative3 } from "node:path";+import { closeSync as closeSync7, fstatSync as fstatSync5, readSync as readSync4 } from "node:fs";function observeRead(partitionRoot, records) {let newest = null;for (const { record } of records) {@@ -5442,6 +5556,21 @@ function readLine(text, lineOffset, length, isLast, shardDevice, partitionRoot)const compaction = record.kind === COMPACTED_KIND && bodyState !== "pending" ? asCompaction(record) : null;return { kind: "record", record: { record, bodyState, offset: lineOffset, length }, compaction };}+function markQuarantined(fd, path, partitionRoot, records, diagnostics) {+ const rel = relative3(partitionRoot, path).split("\\").join("/");+ const ledger = quarantinedLines(partitionRoot, rel);+ if (ledger.size === 0)+ return { records, diagnostics };+ const settings = rel.split("/")[0] === "settings";+ const matches = (offset, kind) => {+ const entry = ledger.get(offset);+ return entry !== undefined && entry.kind === kind && lineSha256AtFd(fd, offset) === entry.sha256;+ };+ return {+ records: records.map((r) => r.bodyState === "pending" && !settings && matches(r.offset, "blob-pending") ? { ...r, quarantined: true } : r),+ diagnostics: diagnostics.map((d) => d.kind === "corrupt-line" && d.torn !== true && matches(d.offset, "corrupt-line") ? { ...d, quarantined: true } : d)+ };+}function readShard(path, options = {}) {const start = options.offset ?? 0;const records = [];@@ -5489,7 +5618,7 @@ function readShard(path, options = {}) {};let held = false;while (!held && position < size) {- const n = readSync3(fd, chunk, 0, Math.min(chunk.length, size - position), position);+ const n = readSync4(fd, chunk, 0, Math.min(chunk.length, size - position), position);if (n === 0)break;const base = position;@@ -5543,26 +5672,27 @@ function readShard(path, options = {}) {}if (options.partitionRoot !== undefined)observeRead(options.partitionRoot, records);+ const marked = options.partitionRoot === undefined ? { records, diagnostics } : markQuarantined(fd, path, options.partitionRoot, records, diagnostics);return {- records,- diagnostics,+ records: marked.records,+ diagnostics: marked.diagnostics,compactions,nextOffset: consumed,pendingTailBytes: size - consumed,size};} finally {- closeSync6(fd);+ closeSync7(fd);}}function shardReadGap(read) {const reasons = [];- const pending = read.records.filter((r) => r.bodyState === "pending").length;+ const pending = read.records.filter((r) => r.bodyState === "pending" && r.quarantined !== true).length;if (pending > 0) {reasons.push(pending === 1 ? "a record whose body is a blob that has not arrived yet" : `${pending} records whose bodies are blobs that have not arrived yet`);}for (const d of read.diagnostics) {- if (d.torn === true)+ if (d.torn === true || d.quarantined === true)continue;reasons.push(`${d.kind} at byte ${d.offset}: ${d.reason}`);}@@ -5584,6 +5714,7 @@ var init_reader = __esm(() => {init_hlc();init_envelope();init_safe_fs();+ init_shard_quarantine();init_shard_list();init_torn_line();READ_CHUNK_BYTES = 1024 * 1024;@@ -5608,11 +5739,11 @@ var init_device_retirement = __esm(() => {});// src/core/store/format-doc.ts-import { readFileSync as readFileSync8 } from "node:fs";-import { join as join13 } from "node:path";+import { readFileSync as readFileSync9 } from "node:fs";+import { join as join14 } from "node:path";import { fileURLToPath } from "node:url";function shippedFormatDoc() {- shipped ??= readFileSync8(join13(fileURLToPath(new URL(".", import.meta.url)), STORE_FORMAT_FILE), "utf8");+ shipped ??= readFileSync9(join14(fileURLToPath(new URL(".", import.meta.url)), STORE_FORMAT_FILE), "utf8");return shipped;}function formatDocVersion(text) {@@ -5621,10 +5752,10 @@ function formatDocVersion(text) {}function writeFormatDoc(dir) {const text = shippedFormatDoc();- const target = join13(dir, STORE_FORMAT_FILE);+ const target = join14(dir, STORE_FORMAT_FILE);let existing = null;try {- existing = readFileSync8(target, "utf8");+ existing = readFileSync9(target, "utf8");} catch {existing = null;}@@ -5646,15 +5777,15 @@ var init_format_doc = __esm(() => {});// src/core/store/held-ledger.ts-import { createHash as createHash2 } from "node:crypto";-import { existsSync as existsSync5, mkdirSync as mkdirSync9 } from "node:fs";-import { join as join14, resolve as resolve6 } from "node:path";+import { createHash as createHash3 } from "node:crypto";+import { existsSync as existsSync5, mkdirSync as mkdirSync9, rmSync as rmSync2 } from "node:fs";+import { join as join15, resolve as resolve7 } from "node:path";function ledgerDir(dirs = processDirsEnv()) {- return join14(storesStateDir(dirs), HELD_LEDGER_DIR);+ return join15(storesStateDir(dirs), HELD_LEDGER_DIR);}function markerPath(dir, root, partition) {- const key = createHash2("sha256").update(resolve6(root)).digest("hex").slice(0, 32);- return join14(dir, `${key}.${partition}`);+ const key = createHash3("sha256").update(resolve7(root)).digest("hex").slice(0, 32);+ return join15(dir, `${key}.${partition}`);}function recordHeldPartitions(root, partitions, dirs) {const dir = ledgerDir(dirs);@@ -5664,7 +5795,7 @@ function recordHeldPartitions(root, partitions, dirs) {continue;try {mkdirSync9(dir, { recursive: true });- atomicWriteFileSync(marker, `${resolve6(root)}+ atomicWriteFileSync(marker, `${resolve7(root)}`);} catch {}}@@ -5684,12 +5815,12 @@ var init_held_ledger = __esm(() => {});// src/core/store/store-open.ts-import { readFileSync as readFileSync9, statSync as statSync6 } from "node:fs";-import { join as join15 } from "node:path";+import { readFileSync as readFileSync10, statSync as statSync7 } from "node:fs";+import { join as join16 } from "node:path";function readStoreId(dir) {let text;try {- text = readFileSync9(join15(dir, STORE_IDENTITY_FILE), "utf8");+ text = readFileSync10(join16(dir, STORE_IDENTITY_FILE), "utf8");} catch (err) {if (err.code === "ENOENT")return null;@@ -5699,15 +5830,15 @@ function readStoreId(dir) {try {parsed = JSON.parse(text);} catch {- throw new StoreIdentityError(`${join15(dir, STORE_IDENTITY_FILE)} is not valid JSON`);+ throw new StoreIdentityError(`${join16(dir, STORE_IDENTITY_FILE)} is not valid JSON`);}if (!isUlid(parsed.store_id)) {- throw new StoreIdentityError(`${join15(dir, STORE_IDENTITY_FILE)} carries no valid store_id`);+ throw new StoreIdentityError(`${join16(dir, STORE_IDENTITY_FILE)} carries no valid store_id`);}return parsed.store_id;}function heldPartitions(root) {- if (statSync6(root, { throwIfNoEntry: false })?.isDirectory() === false) {+ if (statSync7(root, { throwIfNoEntry: false })?.isDirectory() === false) {throw new StoreOpenError(`the brain store root ${root} is not a directory; move the file away or point ` + "the store root at a directory");}const ids = new Map;@@ -5785,7 +5916,8 @@ var init_store_open = __esm(() => {root;partitions;constructor(root, partitions) {- super(`the ${partitions.join(" and ")} partition${partitions.length > 1 ? "s" : ""} of the ` + `brain store at ${root}, which this machine has held, no longer show${partitions.length > 1 ? "" : "s"} ${STORE_IDENTITY_FILE}; its records cannot be read, so the store is not opened. ` + `Restore ${partitions.map((p) => `${p}/${STORE_IDENTITY_FILE}`).join(" and ")} ` + "from sync or a backup");+ const leave = partitions.includes("shared") ? "" : `, or run ${partitions.map((p) => `\`o2b store partition leave ${p}\``).join(" and ")} to stop holding ${partitions.length > 1 ? "them" : "it"} on this machine`;+ super(`the ${partitions.join(" and ")} partition${partitions.length > 1 ? "s" : ""} of the ` + `brain store at ${root}, which this machine has held, no longer show${partitions.length > 1 ? "" : "s"} ${STORE_IDENTITY_FILE}; its records cannot be read, so the store is not opened. ` + `Restore ${partitions.map((p) => `${p}/${STORE_IDENTITY_FILE}`).join(" and ")} ` + `from sync or a backup${leave}`);this.root = root;this.partitions = partitions;this.name = "HeldPartitionLostError";@@ -5794,49 +5926,49 @@ var init_store_open = __esm(() => {});// src/core/store/seen-shards.ts-import { mkdirSync as mkdirSync10, readFileSync as readFileSync10, statSync as statSync7 } from "node:fs";-import { createHash as createHash3 } from "node:crypto";-import { basename as basename6, dirname as dirname11, join as join16, resolve as resolve7 } from "node:path";-function isPartition(value) {+import { mkdirSync as mkdirSync10, readFileSync as readFileSync11, statSync as statSync8 } from "node:fs";+import { createHash as createHash4 } from "node:crypto";+import { basename as basename6, dirname as dirname11, join as join17, resolve as resolve8 } from "node:path";+function isPartition2(value) {return STORE_PARTITIONS.includes(value);}-function stampOf(path) {- const stat = statSync7(path, { throwIfNoEntry: false });+function stampOf2(path) {+ const stat = statSync8(path, { throwIfNoEntry: false });return stat === undefined ? "absent" : `${stat.size}:${stat.mtimeMs}`;}function parseLedger(text) {const out = new Map;const parsed = JSON.parse(text);for (const [partition, rels] of Object.entries(parsed.shards ?? {})) {- if (!isPartition(partition) || !Array.isArray(rels))+ if (!isPartition2(partition) || !Array.isArray(rels))continue;out.set(partition, new Set(rels.filter((r) => typeof r === "string")));}return out;}-function load2(path) {+function load3(path) {let stamp;try {- stamp = stampOf(path);+ stamp = stampOf2(path);} catch {return new Map;}- const hit = loaded.get(path);+ const hit = loaded2.get(path);if (hit !== undefined && hit.stamp === stamp)return hit.ledger;let ledger = new Map;if (stamp !== "absent") {try {- ledger = parseLedger(readFileSync10(path, "utf8"));+ ledger = parseLedger(readFileSync11(path, "utf8"));} catch {}}- loaded.set(path, { stamp, ledger });+ loaded2.set(path, { stamp, ledger });return ledger;}function update2(path, change) {try {- loaded.delete(path);- const ledger = load2(path);+ loaded2.delete(path);+ const ledger = load3(path);change(ledger);const shards = {};for (const [partition, rels] of [...ledger].toSorted(([a], [b]) => compareCodeUnits(a, b))) {@@ -5845,14 +5977,19 @@ function update2(path, change) {mkdirSync10(dirname11(path), { recursive: true });atomicWriteText(path, `${JSON.stringify({ version: 1, shards }, null, 2)}`);- loaded.set(path, { stamp: stampOf(path), ledger });+ loaded2.set(path, { stamp: stampOf2(path), ledger });} catch {- loaded.delete(path);+ loaded2.delete(path);}}-function ledgerPath(partitionRoot) {+function seenLedgerFile(storeRoot, storeId) {+ const rootKey = createHash4("sha256").update(resolve8(storeRoot)).digest("hex").slice(0, 16);+ const name = SEEN_SHARDS_FILE.replace(/\.json$/, `-${rootKey}.json`);+ return join17(deviceStateDirFor(storeId, processDirsEnv()), name);+}+function ledgerPath2(partitionRoot) {const partition = basename6(partitionRoot);- if (!isPartition(partition))+ if (!isPartition2(partition))return null;let storeId;try {@@ -5862,9 +5999,7 @@ function ledgerPath(partitionRoot) {}if (storeId === null)return null;- const rootKey = createHash3("sha256").update(resolve7(dirname11(partitionRoot))).digest("hex").slice(0, 16);- const name = SEEN_SHARDS_FILE.replace(/\.json$/, `-${rootKey}.json`);- return { path: join16(deviceStateDirFor(storeId, processDirsEnv()), name), partition };+ return { path: seenLedgerFile(dirname11(partitionRoot), storeId), partition };}function deviceOf(rel) {const parts = rel.split("/");@@ -5891,10 +6026,10 @@ function retiredDevices(storeRoot) {return new Set(latestRetirements(records).keys());}function trackSeenShards(partitionRoot, scope, listed, replaced = new Set) {- const at = ledgerPath(partitionRoot);+ const at = ledgerPath2(partitionRoot);if (at === null)return [];- const seen = load2(at.path).get(at.partition) ?? new Set;+ const seen = load3(at.path).get(at.partition) ?? new Set;const present = new Set(listed);const fresh = listed.filter((rel) => !seen.has(rel));let missing = [...seen].filter((rel) => rel.startsWith(scope.prefix) && !present.has(rel) && (scope.device === undefined || deviceOf(rel) === scope.device) && (scope.month === undefined || scope.month(monthOf(rel))));@@ -5917,7 +6052,7 @@ function trackSeenShards(partitionRoot, scope, listed, replaced = new Set) {}return missing.toSorted();}-var loaded, readOnlyDepth = 0, SEEN_SHARD_MISSING_REASON;+var loaded2, readOnlyDepth = 0, SEEN_SHARD_MISSING_REASON;var init_seen_shards = __esm(() => {init_fs_atomic();init_platform_dirs();@@ -5927,7 +6062,7 @@ var init_seen_shards = __esm(() => {init_shard_list();init_store_open();init_store_paths();- loaded = new Map;+ loaded2 = new Map;SEEN_SHARD_MISSING_REASON = "this machine read the shard before and it is gone now (the store never deletes a shard); " + "its records are unknown until sync delivers it again";});@@ -5969,9 +6104,10 @@ function readStreamRecords(root, partition, ref, options = {}) {try {shards = listShards(partitionRoot, ref);} catch {- return { records, unreadable: [streamPrefix(ref)] };+ return { records, unreadable: [streamPrefix(ref)], skippedPending: false };}const compactions = [];+ let skippedPending = false;for (const shard of shards) {if (options.device !== undefined && shard.device !== options.device)continue;@@ -5994,6 +6130,8 @@ function readStreamRecords(root, partition, ref, options = {}) {for (const entry of read.records) {if (entry.bodyState !== "pending")records.push(entry.record);+ else+ skippedPending = true;}}const gone = trackSeenShards(partitionRoot, {@@ -6001,7 +6139,7 @@ function readStreamRecords(root, partition, ref, options = {}) {...options.device !== undefined ? { device: options.device } : {},...options.month !== undefined ? { month: options.month } : {}}, shards.map((shard) => shard.rel), replacedShards(compactions));- return { records, unreadable: [...unreadable, ...gone] };+ return { records, unreadable: [...unreadable, ...gone], skippedPending };}var init_stream_read = __esm(() => {init_reader();@@ -6074,22 +6212,22 @@ var init_journal = __esm(() => {});// src/core/store/store-only.ts-import { existsSync as existsSync6, mkdirSync as mkdirSync11, readFileSync as readFileSync11 } from "node:fs";-import { dirname as dirname12, join as join17 } from "node:path";+import { existsSync as existsSync6, mkdirSync as mkdirSync11, readFileSync as readFileSync12 } from "node:fs";+import { dirname as dirname12, join as join18 } from "node:path";function hasLegacyTree(vault) {if (vault === null || vault.trim() === "")return false;- return existsSync6(join17(vault, BRAIN_ROOT_REL));+ return existsSync6(join18(vault, BRAIN_ROOT_REL));}function markerPath2() {- return join17(storesStateDir(processDirsEnv()), SWITCH_MARKER_FILE);+ return join18(storesStateDir(processDirsEnv()), SWITCH_MARKER_FILE);}function markerKey(root, device) {return `${root}\x00${device}`;}function readMarker() {try {- const parsed = JSON.parse(readFileSync11(markerPath2(), "utf8"));+ const parsed = JSON.parse(readFileSync12(markerPath2(), "utf8"));if (!Array.isArray(parsed.switched))return new Set;return new Set(parsed.switched.filter((e) => typeof e === "object" && e !== null && typeof e.root === "string" && typeof e.device === "string").map((e) => markerKey(e.root, e.device)));@@ -6205,7 +6343,7 @@ var init_store_only = __esm(() => {// src/core/brain/secrets/owner-acl.tsimport { spawnSync } from "node:child_process";-import { resolve as resolve8, win32 as win323 } from "node:path";+import { resolve as resolve9, win32 as win323 } from "node:path";function system32Tool(name, env = process.env) {const root = env["SystemRoot"] || env["windir"] || "C:\\Windows";return win323.join(root, "System32", name);@@ -6236,7 +6374,7 @@ function ownerOnlyAclArgv(path, sid, kind) {function restrictToOwner(path, kind, platform = process.platform, force = false) {if (platform !== "win32")return true;- const key = `${kind}:${resolve8(path).toLowerCase()}`;+ const key = `${kind}:${resolve9(path).toLowerCase()}`;if (!force && restricted.has(key))return true;let detail;@@ -6275,7 +6413,7 @@ var init_owner_acl = __esm(() => {import { randomBytes as randomBytes3 } from "node:crypto";import {chmodSync,- closeSync as closeSync7,+ closeSync as closeSync8,constants as constants2,copyFileSync,existsSync as existsSync7,@@ -6285,15 +6423,15 @@ import {lstatSync as lstatSync3,mkdirSync as mkdirSync12,readdirSync as readdirSync3,- readFileSync as readFileSync12,+ readFileSync as readFileSync13,rmdirSync,- statSync as statSync8,+ statSync as statSync9,unlinkSync as unlinkSync4,utimesSync,writeFileSync,writeSync as writeSync5} from "node:fs";-import { basename as basename7, dirname as dirname13, join as join18, relative as relative3, sep as sep4 } from "node:path";+import { basename as basename7, dirname as dirname13, join as join19, relative as relative4, sep as sep4 } from "node:path";function filesBelow(path) {let stat;try {@@ -6305,13 +6443,13 @@ function filesBelow(path) {return TEMP_NAME_RE.test(basename7(path)) ? [] : [path];if (!stat.isDirectory())return [];- return readdirSync3(path).toSorted().flatMap((name) => filesBelow(join18(path, name)));+ return readdirSync3(path).toSorted().flatMap((name) => filesBelow(join19(path, name)));}function sameBytes(a, b) {try {- if (statSync8(a).size !== statSync8(b).size)+ if (statSync9(a).size !== statSync9(b).size)return false;- return readFileSync12(a).equals(readFileSync12(b));+ return readFileSync13(a).equals(readFileSync13(b));} catch {return false;}@@ -6327,11 +6465,11 @@ function writeExclusive(path, bytes) {writeSync5(fd, bytes);fsyncSync4(fd);} catch (err) {- closeSync7(fd);+ closeSync8(fd);tryUnlink(path);throw err;}- closeSync7(fd);+ closeSync8(fd);}function tryUnlink(path) {try {@@ -6342,8 +6480,8 @@ function copyNoClobber(file, target) {if (existsSync7(target))return false;mkdirOwnerOnly(dirname13(target));- const temp = join18(dirname13(target), `.${basename7(target)}${TEMP_INFIX}${randomBytes3(6).toString("hex")}`);- const { atime, mtime } = statSync8(file);+ const temp = join19(dirname13(target), `.${basename7(target)}${TEMP_INFIX}${randomBytes3(6).toString("hex")}`);+ const { atime, mtime } = statSync9(file);try {copyFileSync(file, temp, constants2.COPYFILE_EXCL);if (process.platform !== "win32")@@ -6357,7 +6495,7 @@ function copyNoClobber(file, target) {return false;}try {- writeExclusive(target, readFileSync12(temp));+ writeExclusive(target, readFileSync13(temp));} catch (err) {if (err.code === "EEXIST")return false;@@ -6380,7 +6518,7 @@ function removeEmptyDirs(dir) {if (!stat.isDirectory())return;for (const name of readdirSync3(dir))- removeEmptyDirs(join18(dir, name));+ removeEmptyDirs(join19(dir, name));try {rmdirSync(dir);} catch {}@@ -6389,7 +6527,7 @@ function sameLanded(file, target, aside, rel) {if (sameBytes(file, target))return true;for (let n = 0;n < 100; n++) {- const kept = join18(aside, n === 0 ? rel : `${rel}.${n}`);+ const kept = join19(aside, n === 0 ? rel : `${rel}.${n}`);if (!existsSync7(kept))return false;if (sameBytes(file, kept))@@ -6399,7 +6537,7 @@ function sameLanded(file, target, aside, rel) {}function keepAside(file, aside, rel) {for (let n = 0;n < 100; n++) {- const kept = join18(aside, n === 0 ? rel : `${rel}.${n}`);+ const kept = join19(aside, n === 0 ? rel : `${rel}.${n}`);if (sameBytes(file, kept))return true;if (!existsSync7(kept)) {@@ -6416,10 +6554,10 @@ function adoptFolder(from, to) {if (!existsSync7(from))return true;let settled = true;- const aside = join18(dirname13(to), ADOPT_CONFLICTS_DIR, basename7(from));+ const aside = join19(dirname13(to), ADOPT_CONFLICTS_DIR, basename7(from));for (const file of filesBelow(from)) {- const rel = relative3(from, file);- const target = join18(to, rel);+ const rel = relative4(from, file);+ const target = join19(to, rel);try {const landed = copyNoClobber(file, target) || sameBytes(file, target) || keepAside(file, aside, rel);if (!landed || !sameLanded(file, target, aside, rel)) {@@ -6440,8 +6578,8 @@ function seedFolderOnce(from, to, marker) {try {const seeded = [];for (const file of filesBelow(from)) {- const rel = relative3(from, file);- if (copyNoClobber(file, join18(to, rel)))+ const rel = relative4(from, file);+ if (copyNoClobber(file, join19(to, rel)))seeded.push(rel.split(sep4).join("/"));}mkdirSync12(dirname13(marker), { recursive: true });@@ -6456,9 +6594,9 @@ var init_state_moves = __esm(() => {});// src/core/state/machine-state.ts-import { createHash as createHash4 } from "node:crypto";+import { createHash as createHash5 } from "node:crypto";import { cpSync, existsSync as existsSync8, readdirSync as readdirSync4, writeFileSync as writeFileSync2 } from "node:fs";-import { dirname as dirname14, join as join19, relative as relative4, resolve as resolve9, sep as sep5 } from "node:path";+import { dirname as dirname14, join as join20, relative as relative5, resolve as resolve10, sep as sep5 } from "node:path";function storeIdAt(root) {const known = STORE_IDS.get(root);if (known !== undefined)@@ -6485,10 +6623,10 @@ function deviceStateDir(storeId) {return storeId === null ? storesStateDir(source) : deviceStateDirFor(storeId, source);}function vaultKey(vault) {- return createHash4("sha256").update(resolve9(vault)).digest("hex").slice(0, VAULT_KEY_CHARS);+ return createHash5("sha256").update(resolve10(vault)).digest("hex").slice(0, VAULT_KEY_CHARS);}function freshVaultRel(vault) {- return join19(FRESH_STATE_DIR, vaultKey(vault));+ return join20(FRESH_STATE_DIR, vaultKey(vault));}function adoptOnce(from, to) {if (from === to || ADOPTED.has(from) || !existsSync8(from))@@ -6497,19 +6635,19 @@ function adoptOnce(from, to) {ADOPTED.add(from);}function seedFromVaultOnce(vault, root) {- const marker = join19(root, SEEDED_MARKER_DIR, `${vaultKey(vault)}.json`);- seedFolderOnce(join19(vault, DERIVED_STORE_DIR), join19(root, DERIVED_STORE_DIR), marker);+ const marker = join20(root, SEEDED_MARKER_DIR, `${vaultKey(vault)}.json`);+ seedFolderOnce(join20(vault, DERIVED_STORE_DIR), join20(root, DERIVED_STORE_DIR), marker);}function restoreBackupOnce(local) {if (RESTORED.has(local))return;const migration = dirname14(local);- const marker = join19(migration, BACKUP_RESTORED_MARKER);+ const marker = join20(migration, BACKUP_RESTORED_MARKER);if (existsSync8(marker)) {RESTORED.add(local);return;}- const files = join19(migration, MIGRATION_BACKUP_DIR, MIGRATION_BACKUP_FILES_DIR);+ const files = join20(migration, MIGRATION_BACKUP_DIR, MIGRATION_BACKUP_FILES_DIR);if (!existsSync8(files)) {RESTORED.add(local);return;@@ -6517,8 +6655,8 @@ function restoreBackupOnce(local) {try {const restored = [];for (const file of filesBelow(files)) {- const rel = relative4(files, file);- const target = join19(local, rel);+ const rel = relative5(files, file);+ const target = join20(local, rel);if (existsSync8(target))continue;cpSync(file, target, { preserveTimestamps: true });@@ -6540,27 +6678,27 @@ function locate(vault, configPath, settle) {return { relocated: false, root: vault };if (mode.reason === "no-legacy-tree") {const storeId2 = configuredStoreId(configPath);- const root2 = join19(deviceStateDir(storeId2), freshVaultRel(vault));+ const root2 = join20(deviceStateDir(storeId2), freshVaultRel(vault));if (!settle)return { relocated: true, root: root2 };if (storeId2 !== null)- adoptOnce(join19(deviceStateDir(null), freshVaultRel(vault)), root2);+ adoptOnce(join20(deviceStateDir(null), freshVaultRel(vault)), root2);seedFromVaultOnce(vault, root2);return { relocated: true, root: root2 };}}const storeId = configuredStoreId(configPath);- const root = join19(deviceStateDir(storeId), MIGRATION_STATE_DIR, MIGRATION_LOCAL_DIR);+ const root = join20(deviceStateDir(storeId), MIGRATION_STATE_DIR, MIGRATION_LOCAL_DIR);if (!settle)return { relocated: true, root };if (storeId !== null) {- adoptOnce(join19(deviceStateDir(null), MIGRATION_STATE_DIR, MIGRATION_LOCAL_DIR), root);+ adoptOnce(join20(deviceStateDir(null), MIGRATION_STATE_DIR, MIGRATION_LOCAL_DIR), root);}if (hasVault) {- adoptOnce(join19(deviceStateDir(null), freshVaultRel(vault)), root);+ adoptOnce(join20(deviceStateDir(null), freshVaultRel(vault)), root);if (storeId !== null)- adoptOnce(join19(deviceStateDir(storeId), freshVaultRel(vault)), root);- if (!existsSync8(join19(vault, BRAIN_ROOT_REL)))+ adoptOnce(join20(deviceStateDir(storeId), freshVaultRel(vault)), root);+ if (!existsSync8(join20(vault, BRAIN_ROOT_REL)))seedFromVaultOnce(vault, root);}restoreBackupOnce(root);@@ -6573,7 +6711,7 @@ function machineStatePath(vault, ...segments) {return machineStatePathIn(machineStateRoot(vault), ...segments);}function machineStatePathIn(root, ...segments) {- return ensureInsideVault(join19(root, ...segments), root);+ return ensureInsideVault(join20(root, ...segments), root);}var FRESH_STATE_DIR = "fresh", VAULT_KEY_CHARS = 16, BACKUP_RESTORED_MARKER = "local-restored.json", STORE_IDS, ADOPTED, SEEDED_MARKER_DIR, RESTORED;var init_machine_state = __esm(() => {@@ -6587,7 +6725,7 @@ var init_machine_state = __esm(() => {init_state_moves();STORE_IDS = new Map;ADOPTED = new Set;- SEEDED_MARKER_DIR = join19(DERIVED_STORE_DIR, "seeded-from-vault");+ SEEDED_MARKER_DIR = join20(DERIVED_STORE_DIR, "seeded-from-vault");RESTORED = new Set;});@@ -6696,8 +6834,8 @@ var init_stamp = __esm(() => {});// src/core/brain/freeze-marker.ts-import { existsSync as existsSync9, readFileSync as readFileSync13, statSync as statSync9 } from "node:fs";-import { resolve as resolve10 } from "node:path";+import { existsSync as existsSync9, readFileSync as readFileSync14, statSync as statSync10 } from "node:fs";+import { resolve as resolve11 } from "node:path";function markerPathIn(root) {return machineStatePathIn(root, BRAIN_INTERNAL_STATE_REL, FROZEN_MARKER_FILE);}@@ -6705,7 +6843,7 @@ function parseMarker(path) {reloadCount += 1;let parsed;try {- parsed = JSON.parse(readFileSync13(path, "utf8"));+ parsed = JSON.parse(readFileSync14(path, "utf8"));} catch {return UNREADABLE_MARKER;}@@ -6722,7 +6860,7 @@ function parseMarker(path) {});}function readFreezeMarker(vault) {- const root = resolve10(vault);+ const root = resolve11(vault);let path = MARKER_PATHS.get(root);if (path === undefined) {const location = machineStateLocation(root);@@ -6732,7 +6870,7 @@ function readFreezeMarker(vault) {}let stat;try {- stat = statSync9(path, { throwIfNoEntry: false });+ stat = statSync10(path, { throwIfNoEntry: false });} catch {stat = undefined;}@@ -6758,7 +6896,7 @@ function vaultFrozenNotice(vault, marker) {return degradationNotice({code: DEGRADATION_CODE.vaultFrozen,site: SITE,- path: resolve10(vault),+ path: resolve11(vault),detail: `refusing to write: this vault was frozen at ${marker.frozen_at} by ` + `${marker.by === "" ? "an unnamed agent" : marker.by} (${why}). ` + `Run \`${FREEZE_NEXT_COMMAND}\` to lift it`});}@@ -6805,17 +6943,17 @@ var init_freeze_marker = __esm(() => {});// src/core/migration-gate.ts-import { mkdirSync as mkdirSync13, readFileSync as readFileSync14, realpathSync as realpathSync3, unlinkSync as unlinkSync5 } from "node:fs";-import { dirname as dirname15, join as join20, resolve as resolve11 } from "node:path";+import { mkdirSync as mkdirSync13, readFileSync as readFileSync15, realpathSync as realpathSync3, unlinkSync as unlinkSync5 } from "node:fs";+import { dirname as dirname15, join as join21, resolve as resolve12 } from "node:path";function migrationGatePath(source = processDirsEnv()) {- return join20(stateRootDir(source), GATE_FILE);+ return join21(stateRootDir(source), GATE_FILE);}function canonicalVault(vault) {let path;try {path = realpathSync3(vault);} catch {- path = resolve11(vault);+ path = resolve12(vault);}return process.platform === "win32" ? path.toLowerCase() : path;}@@ -6830,7 +6968,7 @@ function readMigrationGateState(source) {if (attempt > 0)sleepSync(READ_RETRY_MS);try {- text = readFileSync14(path, "utf8");+ text = readFileSync15(path, "utf8");} catch (err) {code = String(err.code ?? "EIO");if (code === "ENOENT")@@ -6891,17 +7029,17 @@ var init_migration_gate = __esm(() => {});// src/core/brain/vault-identity.ts-import { existsSync as existsSync10, readFileSync as readFileSync15, statSync as statSync10 } from "node:fs";-import { join as join21, resolve as resolve12 } from "node:path";+import { existsSync as existsSync10, readFileSync as readFileSync16, statSync as statSync11 } from "node:fs";+import { join as join22, resolve as resolve13 } from "node:path";function vaultIdentityPath(vault) {- return ensureInsideVault(join21(vault, BRAIN_ROOT_REL, VAULT_IDENTITY_FILE), vault);+ return ensureInsideVault(join22(vault, BRAIN_ROOT_REL, VAULT_IDENTITY_FILE), vault);}function readVaultIdentity(vault) {const path = vaultIdentityPath(vault);if (!existsSync10(path))return null;try {- const parsed = JSON.parse(readFileSync15(path, "utf8"));+ const parsed = JSON.parse(readFileSync16(path, "utf8"));if (typeof parsed.vault_id !== "string" || parsed.vault_id.length === 0)return null;return Object.freeze({@@ -6921,7 +7059,7 @@ function currentVaultId(root) {}let stat;try {- stat = statSync10(path, { throwIfNoEntry: false });+ stat = statSync11(path, { throwIfNoEntry: false });} catch {stat = undefined;}@@ -6947,7 +7085,7 @@ function currentVaultId(root) {return identity.vault_id;}function vaultMarkerAbsentNotice(vault) {- const root = resolve12(vault);+ const root = resolve13(vault);if (currentVaultId(root) !== null)return null;return degradationNotice({@@ -6958,7 +7096,7 @@ function vaultMarkerAbsentNotice(vault) {});}function assertVaultIdentityForWrite(vault, sink, lane = WRITE_LANE.content) {- const root = resolve12(vault);+ const root = resolve13(vault);if (lane === WRITE_LANE.content) {assertVaultNotFrozen(root);assertNoMigrationInProgress(root);@@ -7013,21 +7151,21 @@ var init_vault_identity = __esm(() => {});// src/core/brain/paths.ts-import { join as join22, posix as posix6 } from "node:path";+import { join as join23, posix as posix6 } from "node:path";function brainDirs(vault) {- const brain = ensureInsideVault(join22(vault, BRAIN_ROOT_REL), vault);+ const brain = ensureInsideVault(join23(vault, BRAIN_ROOT_REL), vault);return {brain,- inbox: ensureInsideVault(join22(vault, BRAIN_INBOX_REL), vault),- processed: ensureInsideVault(join22(vault, BRAIN_PROCESSED_REL), vault),- archived: ensureInsideVault(join22(vault, BRAIN_ARCHIVED_SIGNALS_REL), vault),- pending: ensureInsideVault(join22(vault, BRAIN_PENDING_REL), vault),- preferences: ensureInsideVault(join22(vault, BRAIN_PREFERENCES_REL), vault),- retired: ensureInsideVault(join22(vault, BRAIN_RETIRED_REL), vault),- log: ensureInsideVault(join22(vault, BRAIN_LOG_REL), vault),- entities: ensureInsideVault(join22(vault, BRAIN_ENTITIES_REL), vault),- bases: ensureInsideVault(join22(vault, BRAIN_BASES_REL), vault),- snapshots: ensureInsideVault(join22(vault, BRAIN_SNAPSHOTS_REL), vault)+ inbox: ensureInsideVault(join23(vault, BRAIN_INBOX_REL), vault),+ processed: ensureInsideVault(join23(vault, BRAIN_PROCESSED_REL), vault),+ archived: ensureInsideVault(join23(vault, BRAIN_ARCHIVED_SIGNALS_REL), vault),+ pending: ensureInsideVault(join23(vault, BRAIN_PENDING_REL), vault),+ preferences: ensureInsideVault(join23(vault, BRAIN_PREFERENCES_REL), vault),+ retired: ensureInsideVault(join23(vault, BRAIN_RETIRED_REL), vault),+ log: ensureInsideVault(join23(vault, BRAIN_LOG_REL), vault),+ entities: ensureInsideVault(join23(vault, BRAIN_ENTITIES_REL), vault),+ bases: ensureInsideVault(join23(vault, BRAIN_BASES_REL), vault),+ snapshots: ensureInsideVault(join23(vault, BRAIN_SNAPSHOTS_REL), vault)};}function brainDirsForWrite(vault, notices, lane) {@@ -7036,11 +7174,11 @@ function brainDirsForWrite(vault, notices, lane) {}function preferencePath(vault, slug) {const s = validateSlug(slug);- return ensureInsideVault(join22(brainDirs(vault).preferences, `pref-${s}.md`), vault);+ return ensureInsideVault(join23(brainDirs(vault).preferences, `pref-${s}.md`), vault);}function retiredPath(vault, slug) {const s = validateSlug(slug);- return ensureInsideVault(join22(brainDirs(vault).retired, `ret-${s}.md`), vault);+ return ensureInsideVault(join23(brainDirs(vault).retired, `ret-${s}.md`), vault);}function validateSlug(slug) {const trimmed = slug.trim();@@ -7160,13 +7298,13 @@ var init_privacy_markers = __esm(() => {});// src/core/vault.ts-import { mkdirSync as mkdirSync14, readFileSync as readFileSync16, readdirSync as readdirSync5, writeFileSync as writeFileSync3 } from "node:fs";-import { dirname as dirname16, join as join23, relative as relative5 } from "node:path";+import { mkdirSync as mkdirSync14, readFileSync as readFileSync17, readdirSync as readdirSync5, writeFileSync as writeFileSync3 } from "node:fs";+import { dirname as dirname16, join as join24, relative as relative6 } from "node:path";function parseFrontmatterWithNotices(path, opts = {}) {const site = opts.site ?? FRONTMATTER_SITE;let text;try {- text = readFileSync16(path, "utf8");+ text = readFileSync17(path, "utf8");} catch (err) {const notices = [];emitDegradationNotice(notices, {@@ -7277,7 +7415,7 @@ function walk(root, dir, skipDirs, skipFiles, out, notices) {return;}for (const entry of entries2) {- const full = join23(dir, entry.name);+ const full = join24(dir, entry.name);if (entry.isDirectory()) {if (skipDirs.has(entry.name))continue;@@ -7290,7 +7428,7 @@ function walk(root, dir, skipDirs, skipFiles, out, notices) {continue;if (skipFiles.has(entry.name.toLowerCase()))continue;- const rel = relative5(root, full);+ const rel = relative6(root, full);const parts = rel.split(/[\\/]/);if (parts.some((p) => skipDirs.has(p)))continue;@@ -7584,9 +7722,9 @@ var init_fold_signals = __esm(() => {});// src/core/store/read/store-entities.ts-import { statSync as statSync11 } from "node:fs";+import { statSync as statSync12 } from "node:fs";function fingerprintOf(store) {- const parts = [store.storeId];+ const parts = [store.storeId, `quarantine=${quarantineStamp(store.root)}`];for (const partition of [...store.partitions].toSorted()) {let shards;try {@@ -7598,7 +7736,7 @@ function fingerprintOf(store) {for (const shard of shards) {let stamp = "missing";try {- const stat = statSync11(shard.path);+ const stat = statSync12(shard.path);stamp = `${stat.size}:${stat.mtimeMs}`;} catch {}parts.push(`${partition}/${shard.rel}=${stamp}`);@@ -7617,8 +7755,11 @@ function build(store) {const all = [];const unreadable = [];const pinned = new Map;+ let skippedPending = false;for (const partition of store.partitions) {const read = readStreamRecords(store.root, partition, EVENTS);+ if (read.skippedPending)+ skippedPending = true;unreadable.push(...read.unreadable.map((rel) => `${partition}/${rel}`));for (const record of read.records)all.push({ record, partition });@@ -7649,7 +7790,7 @@ function build(store) {partitions: new Set(entry.history.map((r) => partitionOf.get(r.id) ?? "shared"))});}- return { store, preferences, signals, pinned, unreadable };+ return { store, preferences, signals, pinned, unreadable, skippedPending };}function readEntitiesOf(store) {const fingerprint = fingerprintOf(store);@@ -7657,7 +7798,7 @@ function readEntitiesOf(store) {if (cached !== undefined && cached.fingerprint === fingerprint)return cached.entities;const entities = build(store);- if (entities.unreadable.length === 0)+ if (entities.unreadable.length === 0 && !entities.skippedPending)cache.set(store.root, { fingerprint, entities });elsecache.delete(store.root);@@ -7685,6 +7826,7 @@ var init_store_entities = __esm(() => {init_store_open();init_store_paths();init_stream_read();+ init_shard_quarantine();cache = new Map;EVENTS = { stream: "events" };});@@ -7853,8 +7995,8 @@ var init_memory_ids = __esm(() => {});// src/core/brain/artifact-ref-resolve.ts-import { isAbsolute as isAbsolute3, join as join24 } from "node:path";-import { existsSync as existsSync11, statSync as statSync12 } from "node:fs";+import { isAbsolute as isAbsolute3, join as join25 } from "node:path";+import { existsSync as existsSync11, statSync as statSync13 } from "node:fs";function artifactRefScope(vault, fromStore) {let store = fromStore;let dirs;@@ -7880,11 +8022,11 @@ function artifactPath(vault, id, scope) {const memoryDirs = fromStore ? [] : [dirs.preferences, dirs.retired, dirs.inbox, dirs.processed, dirs.archived, dirs.pending];for (const dir of [...memoryDirs,- join24(vault, BRAIN_SOURCES_REL),+ join25(vault, BRAIN_SOURCES_REL),dirs.entities,- join24(vault, BRAIN_DECISIONS_REL)+ join25(vault, BRAIN_DECISIONS_REL)]) {- const abs = join24(dir, `${id}${MARKDOWN_EXT}`);+ const abs = join25(dir, `${id}${MARKDOWN_EXT}`);if (existsSync11(abs))return vaultRelative(abs, vault);}@@ -7899,10 +8041,10 @@ function pageCandidate(vault, bare) {return null;}function isVaultFile(vault, rel) {- if (!pathIsInside(join24(vault, rel), vault))+ if (!pathIsInside(join25(vault, rel), vault))return false;try {- return statSync12(join24(vault, rel)).isFile();+ return statSync13(join25(vault, rel)).isFile();} catch {return false;}@@ -8091,8 +8233,8 @@ var init_private_region = __esm(() => {});// src/core/store/reference-privacy.ts-import { existsSync as existsSync12, statSync as statSync13 } from "node:fs";-import { resolve as resolve13 } from "node:path";+import { existsSync as existsSync12, statSync as statSync14 } from "node:fs";+import { resolve as resolve14 } from "node:path";function namesAPage(ref) {const trimmed = ref.trim();return ANCHORED_WIKILINK_RE.test(trimmed) || trimmed.endsWith(PAGE_EXTENSION);@@ -8153,7 +8295,7 @@ function refsNamePrivatePage(vault, refs, guessed, fromStore, scope) {const rel = resolveArtifactRef(vault, ref, scope ?? artifactRefScope(vault, fromStore));if (rel === null)continue;- const abs = resolve13(vault, rel);+ const abs = resolve14(vault, rel);if (!pathIsInside(abs, vault) || existsSync12(abs) && isPrivatePageFile(abs))return true;}@@ -8197,7 +8339,7 @@ var init_operation_scope = __esm(() => {});// src/core/store/dual-write.ts-import { readFileSync as readFileSync17 } from "node:fs";+import { readFileSync as readFileSync18 } from "node:fs";function isDualWriteEnabled(configPath) {try {return resolveStoreDualWrite(configPath);@@ -8207,14 +8349,17 @@ function isDualWriteEnabled(configPath) {}function readDualWriteStatus(path = dualWriteStatusPath()) {try {- const parsed = JSON.parse(readFileSync17(path, "utf8"));+ const parsed = JSON.parse(readFileSync18(path, "utf8"));return {failures: typeof parsed.failures === "number" ? parsed.failures : 0,skipped_no_device: typeof parsed.skipped_no_device === "number" ? parsed.skipped_no_device : 0,skipped_no_store: typeof parsed.skipped_no_store === "number" ? parsed.skipped_no_store : 0,last_error: typeof parsed.last_error === "string" ? parsed.last_error : null,last_failure_at: typeof parsed.last_failure_at === "string" ? parsed.last_failure_at : null,- last_skip_at: typeof parsed.last_skip_at === "string" ? parsed.last_skip_at : null+ last_skip_at: typeof parsed.last_skip_at === "string" ? parsed.last_skip_at : null,+ shared_failures: typeof parsed.shared_failures === "number" ? parsed.shared_failures : 0,+ shared_last_error: typeof parsed.shared_last_error === "string" ? parsed.shared_last_error : null,+ shared_last_failure_at: typeof parsed.shared_last_failure_at === "string" ? parsed.shared_last_failure_at : null};} catch {return EMPTY_STATUS;@@ -8243,11 +8388,17 @@ function recordFailure(err) {const message = err instanceof Error ? err.message : String(err);warnOnce("failure", `a brain store dual write failed (${message}); the legacy write is intact. ` + "Later failures in this process are counted, not printed; `o2b brain doctor` reports them.");const at = new Date().toISOString();+ const shared = failedOnlyInShared(err);bumpStatus((s) => ({...s,failures: s.failures + 1,last_error: message.slice(0, 500),- last_failure_at: at+ last_failure_at: at,+ ...shared ? {+ shared_failures: s.shared_failures + 1,+ shared_last_error: message.slice(0, 500),+ shared_last_failure_at: at+ } : {}}));}function recordNoDevice() {@@ -8315,19 +8466,36 @@ function appendStoreRecords(store, records, context) {continue;placed.push([record, raised]);}- for (const [, at] of placed) {- if (!store.partitions.has(at))- throw new PartitionUnavailableError(at, store.root);- }- return placed.map(([record, at]) => appendRecord(store, {- partition: at,- stream: record.stream,- ...record.ledger !== undefined ? { ledger: record.ledger } : {},- kind: record.kind,- agent: record.agent !== undefined && record.agent.trim() !== "" ? record.agent : defaultAgent(),- scope: record.scope ?? null,- body: { ...record.body, [VAULT_ID_FIELD]: vaultId }- }));+ try {+ for (const [, at] of placed) {+ if (!store.partitions.has(at))+ throw new PartitionUnavailableError(at, store.root);+ }+ return placed.map(([record, at]) => appendRecord(store, {+ partition: at,+ stream: record.stream,+ ...record.ledger !== undefined ? { ledger: record.ledger } : {},+ kind: record.kind,+ agent: record.agent !== undefined && record.agent.trim() !== "" ? record.agent : defaultAgent(),+ scope: record.scope ?? null,+ body: { ...record.body, [VAULT_ID_FIELD]: vaultId }+ }));+ } catch (err) {+ if (typeof err === "object" && err !== null) {+ failedPlacement.set(err, new Set(placed.map(([, at]) => at)));+ }+ throw err;+ }+}+function failedOnlyInShared(err) {+ for (const e of [err, err?.cause]) {+ if (typeof e !== "object" || e === null)+ continue;+ const placed = failedPlacement.get(e);+ if (placed !== undefined)+ return placed.size > 0 && [...placed].every((p) => p === "shared");+ }+ return false;}function dualWrite(build2, context = {}) {if (!isDualWriteEnabled(context.configPath))@@ -8355,7 +8523,7 @@ function dualWrite(build2, context = {}) {recordFailure(err);}}-var EMPTY_STATUS, warned, handles;+var EMPTY_STATUS, warned, handles, failedPlacement;var init_dual_write = __esm(() => {init_fs_atomic();init_config();@@ -8375,17 +8543,21 @@ var init_dual_write = __esm(() => {skipped_no_store: 0,last_error: null,last_failure_at: null,- last_skip_at: null+ last_skip_at: null,+ shared_failures: 0,+ shared_last_error: null,+ shared_last_failure_at: null});warned = new Set;handles = new Map;+ failedPlacement = new WeakMap;});// src/core/store/store-identity.ts-import { mkdirSync as mkdirSync15, readFileSync as readFileSync18 } from "node:fs";-import { dirname as dirname17, join as join25 } from "node:path";+import { mkdirSync as mkdirSync15, readFileSync as readFileSync19 } from "node:fs";+import { dirname as dirname17, join as join26 } from "node:path";function storeIdentityPath() {- return join25(storesStateDir(processDirsEnv()), STORE_IDENTITY_FILE2);+ return join26(storesStateDir(processDirsEnv()), STORE_IDENTITY_FILE2);}function isBinding(value) {if (typeof value !== "object" || value === null)@@ -8396,7 +8568,7 @@ function isBinding(value) {function readBindingsStrict(path) {let text;try {- text = readFileSync18(path, "utf8");+ text = readFileSync19(path, "utf8");} catch (err) {return err.code === "ENOENT" ? [] : null;}@@ -8486,8 +8658,8 @@ var init_store_identity = __esm(() => {});// src/core/store/store-write.ts-import { mkdirSync as mkdirSync16, readFileSync as readFileSync19 } from "node:fs";-import { dirname as dirname18, join as join26 } from "node:path";+import { mkdirSync as mkdirSync16, readFileSync as readFileSync20 } from "node:fs";+import { dirname as dirname18, join as join27 } from "node:path";function asStoreWriteError(err) {if (err instanceof StoreWriteError)return err;@@ -8555,16 +8727,19 @@ function routeWrite(spec) {}}function storeWriteStatusPath() {- return join26(storesStateDir(processDirsEnv()), STORE_WRITE_STATUS_FILE);+ return join27(storesStateDir(processDirsEnv()), STORE_WRITE_STATUS_FILE);}function readStoreWriteStatus(path = storeWriteStatusPath()) {try {- const parsed = JSON.parse(readFileSync19(path, "utf8"));+ const parsed = JSON.parse(readFileSync20(path, "utf8"));return {failures: typeof parsed.failures === "number" ? parsed.failures : 0,last_reason: typeof parsed.last_reason === "string" && Object.hasOwn(REMEDIES, parsed.last_reason) ? parsed.last_reason : null,last_error: typeof parsed.last_error === "string" ? parsed.last_error : null,- last_failure_at: typeof parsed.last_failure_at === "string" ? parsed.last_failure_at : null+ last_failure_at: typeof parsed.last_failure_at === "string" ? parsed.last_failure_at : null,+ shared_failures: typeof parsed.shared_failures === "number" ? parsed.shared_failures : 0,+ shared_last_error: typeof parsed.shared_last_error === "string" ? parsed.shared_last_error : null,+ shared_last_failure_at: typeof parsed.shared_last_failure_at === "string" ? parsed.shared_last_failure_at : null};} catch {return EMPTY_STATUS2;@@ -8577,16 +8752,28 @@ function recordFailOpen(error) {process.stderr.write(`Open Second Brain: ${error.message} Later failures in this process are counted, ` + "not printed; `o2b brain doctor` reports them.\n");} catch {}}+ const shared = failedOnlyInShared(error);try {const path = storeWriteStatusPath();mkdirSync16(dirname18(path), { recursive: true });withLocalLock(path, () => {const current = readStoreWriteStatus(path);+ const at = new Date().toISOString();+ const message = error.message.slice(0, 500);const next = {failures: current.failures + 1,last_reason: error.reason,- last_error: error.message.slice(0, 500),- last_failure_at: new Date().toISOString()+ last_error: message,+ last_failure_at: at,+ ...shared ? {+ shared_failures: current.shared_failures + 1,+ shared_last_error: message,+ shared_last_failure_at: at+ } : {+ shared_failures: current.shared_failures,+ shared_last_error: current.shared_last_error,+ shared_last_failure_at: current.shared_last_failure_at+ }};atomicWriteFileSync(path, `${JSON.stringify(next, null, 2)}`);@@ -8628,13 +8815,16 @@ var init_store_write = __esm(() => {failures: 0,last_reason: null,last_error: null,- last_failure_at: null+ last_failure_at: null,+ shared_failures: 0,+ shared_last_error: null,+ shared_last_failure_at: null});});// src/core/reliability/audit.ts-import { closeSync as closeSync8, fsyncSync as fsyncSync5, mkdirSync as mkdirSync17, openSync as openSync7, writeFileSync as writeFileSync4 } from "node:fs";-import { basename as basename8, join as join27 } from "node:path";+import { closeSync as closeSync9, fsyncSync as fsyncSync5, mkdirSync as mkdirSync17, openSync as openSync7, writeFileSync as writeFileSync4 } from "node:fs";+import { basename as basename8, join as join28 } from "node:path";function appendAuditRecord(auditRoot, record, vault, options = {}) {const timestamp = new Date(record.timestamp);if (!Number.isFinite(timestamp.getTime())) {@@ -8674,14 +8864,14 @@ function trailInsideVault(auditRoot, vault) {}function appendLegacyAuditLine(auditRoot, timestamp, line) {mkdirSync17(auditRoot, { recursive: true });- const path = join27(auditRoot, shardedFileName(isoWeekLabel(timestamp), resolveAppendShardId(), JSONL_LEDGER_EXT));+ const path = join28(auditRoot, shardedFileName(isoWeekLabel(timestamp), resolveAppendShardId(), JSONL_LEDGER_EXT));const fileDescriptor = openSync7(path, "a", 384);try {writeFileSync4(fileDescriptor, line + ``, "utf8");fsyncSync5(fileDescriptor);} finally {- closeSync8(fileDescriptor);+ closeSync9(fileDescriptor);}return path;}@@ -8778,14 +8968,14 @@ var init_value_cipher = () => {};import { randomBytes as randomBytes5, scryptSync, timingSafeEqual } from "node:crypto";import {chmodSync as chmodSync2,- closeSync as closeSync9,+ closeSync as closeSync10,fsyncSync as fsyncSync6,openSync as openSync8,- readFileSync as readFileSync20,+ readFileSync as readFileSync21,unlinkSync as unlinkSync6,writeSync as writeSync6} from "node:fs";-import { dirname as dirname19, resolve as resolve14 } from "node:path";+import { dirname as dirname19, resolve as resolve15 } from "node:path";function kdfCostCurveRefusal(kdf) {if (kdf.n > SCRYPT_N_MAX) {return `kdf n ${String(kdf.n)} exceeds this build's ceiling ${String(SCRYPT_N_MAX)}`;@@ -8806,7 +8996,7 @@ function kdfCostCurveRefusal(kdf) {return null;}function holderSlot(keyPath) {- const resolved = resolve14(keyPath);+ const resolved = resolve15(keyPath);return process.platform === "win32" ? resolved.toLowerCase() : resolved;}function heldUnlockedKey(keyPath) {@@ -8865,7 +9055,7 @@ function hasEnvelopeShape(parsed) {function isEnvelopeFile(keyPath) {let bytes;try {- bytes = readFileSync20(keyPath);+ bytes = readFileSync21(keyPath);} catch {return false;}@@ -8881,7 +9071,7 @@ function isEnvelopeBytes(bytes) {function readEnvelope(keyPath) {let parsed;try {- parsed = JSON.parse(readFileSync20(keyPath, "utf8"));+ parsed = JSON.parse(readFileSync21(keyPath, "utf8"));} catch (err) {throw new SecretEnvelopeError(ENVELOPE_REFUSAL_CODES.malformed, keyPath, `not parseable as an envelope: ${err instanceof Error ? err.message : String(err)}`);}@@ -8998,13 +9188,13 @@ function writeRawKeyfileBytes(keyPath, dek) {written += writeSync6(fd, dek, written, dek.byteLength - written);}fsyncSync6(fd);- closeSync9(fd);+ closeSync10(fd);fd = null;renameWithRetry(tmp, keyPath);} catch (err) {if (fd !== null) {try {- closeSync9(fd);+ closeSync10(fd);} catch {}}try {@@ -9017,7 +9207,7 @@ function writeRawKeyfileBytes(keyPath, dek) {try {fsyncSync6(dfd);} finally {- closeSync9(dfd);+ closeSync10(dfd);}} catch {}}@@ -9089,17 +9279,17 @@ var init_envelope2 = __esm(() => {import { randomBytes as randomBytes6 } from "node:crypto";import {chmodSync as chmodSync3,- closeSync as closeSync10,+ closeSync as closeSync11,existsSync as existsSync13,mkdirSync as mkdirSync18,openSync as openSync9,- readFileSync as readFileSync21,+ readFileSync as readFileSync22,writeFileSync as writeFileSync5,writeSync as writeSync7} from "node:fs";-import { dirname as dirname20, join as join28 } from "node:path";+import { dirname as dirname20, join as join29 } from "node:path";function ensureSyncExclusionMarker(dir) {- const marker = join28(dir, ".gitignore");+ const marker = join29(dir, ".gitignore");if (existsSync13(marker))return;try {@@ -9126,7 +9316,7 @@ function loadOrCreateKey(keyPath) {}}ensureSyncExclusionMarker(keyDir);- const key2 = readFileSync21(keyPath);+ const key2 = readFileSync22(keyPath);if (isEnvelopeBytes(key2))return heldKeyOrRefusal(keyPath);if (key2.length !== KEY_BYTES) {@@ -9149,7 +9339,7 @@ function loadOrCreateKey(keyPath) {try {writeSync7(fd, key);} finally {- closeSync10(fd);+ closeSync11(fd);}restrictToOwner(keyPath, "file");return key;@@ -9165,7 +9355,7 @@ var init_crypto = __esm(() => {// src/core/store-migration/owner-only-copy.tsimport { chmodSync as chmodSync4, copyFileSync as copyFileSync2, mkdirSync as mkdirSync19 } from "node:fs";-import { dirname as dirname21, relative as relative6, resolve as resolve15, sep as sep6 } from "node:path";+import { dirname as dirname21, relative as relative7, resolve as resolve16, sep as sep6 } from "node:path";function ensureOwnerOnlyRoot(root) {mkdirSync19(root, { recursive: true, mode: DIR_MODE });if (process.platform === "win32")@@ -9174,10 +9364,10 @@ function ensureOwnerOnlyRoot(root) {chmodSync4(root, DIR_MODE);}function foldersFrom(root, dir) {- const rel = relative6(resolve15(root), resolve15(dir));+ const rel = relative7(resolve16(root), resolve16(dir));if (rel === "" || rel.startsWith(".."))- return [resolve15(root)];- const out = [resolve15(root)];+ return [resolve16(root)];+ const out = [resolve16(root)];for (const segment of rel.split(sep6))out.push(`${out.at(-1)}${sep6}${segment}`);return out;@@ -9225,30 +9415,30 @@ import {chmodSync as chmodSync5,existsSync as existsSync14,readdirSync as readdirSync6,- readFileSync as readFileSync22,+ readFileSync as readFileSync23,renameSync as renameSync2,- rmSync as rmSync2,- statSync as statSync14,+ rmSync as rmSync3,+ statSync as statSync15,writeFileSync as writeFileSync6} from "node:fs";-import { join as join29 } from "node:path";+import { join as join30 } from "node:path";function secretsDir(vault) {const dir = machineStatePath(vault, DERIVED_STORE_DIR, "secrets");- adoptVaultSecrets(join29(vault, DERIVED_STORE_DIR, "secrets"), dir);+ adoptVaultSecrets(join30(vault, DERIVED_STORE_DIR, "secrets"), dir);return dir;}function adoptVaultSecrets(legacy, dir) {if (legacy === dir || existsSync14(dir) || !existsSync14(legacy))return;- const names = readdirSync6(legacy).filter((name) => statSync14(join29(legacy, name)).isFile());+ const names = readdirSync6(legacy).filter((name) => statSync15(join30(legacy, name)).isFile());if (names.length === 0)return;const staging = `${dir}.adopt-${process.pid}`;- rmSync2(staging, { recursive: true, force: true });+ rmSync3(staging, { recursive: true, force: true });try {ensureOwnerOnlyRoot(staging);for (const name of names)- copyOwnerOnly(join29(legacy, name), join29(staging, name), staging);+ copyOwnerOnly(join30(legacy, name), join30(staging, name), staging);try {renameSync2(staging, dir);} catch (err) {@@ -9256,17 +9446,17 @@ function adoptVaultSecrets(legacy, dir) {throw err;}} finally {- rmSync2(staging, { recursive: true, force: true });+ rmSync3(staging, { recursive: true, force: true });}}function storePath(vault) {- return join29(secretsDir(vault), "secrets.json");+ return join30(secretsDir(vault), "secrets.json");}function keyPath(vault) {- return join29(secretsDir(vault), "keyfile");+ return join30(secretsDir(vault), "keyfile");}function tokenStorePath(vault) {- return join29(secretsDir(vault), "mcp-tokens.json");+ return join30(secretsDir(vault), "mcp-tokens.json");}function isValidSecretName(name) {return NAME_RE.test(name);@@ -9472,14 +9662,14 @@ function readStore(vault) {restrictToOwner(path, "file");if (process.platform !== "win32") {try {- if ((statSync14(path).mode & 511) !== 384)+ if ((statSync15(path).mode & 511) !== 384)chmodSync5(path, 384);} catch (err) {process.stderr.write(`warning: could not re-apply owner-only mode to the secrets store: ` + `${path}: ${err instanceof Error ? err.message : String(err)}`);}}- const parsed = JSON.parse(readFileSync22(path, "utf8"));+ const parsed = JSON.parse(readFileSync23(path, "utf8"));if (parsed === null || typeof parsed !== "object" || parsed.version !== SECRETS_SCHEMA_VERSION) {throw new Error(`secrets store is corrupt or from a newer version: ${path}`);}@@ -9513,7 +9703,7 @@ function touchLastUsed(vault, name, now) {});}function audit(vault, ctx, action, name, details) {- appendAuditRecord(join29(brainDirsForWrite(vault).log, SECRET_CUSTODY_AUDIT_DIR), {+ appendAuditRecord(join30(brainDirsForWrite(vault).log, SECRET_CUSTODY_AUDIT_DIR), {timestamp: ctx.now.toISOString(),actor: ctx.agent,action,@@ -9832,18 +10022,18 @@ import {existsSync as existsSync17,mkdirSync as mkdirSync20,openSync as openSync10,- readFileSync as readFileSync23,- rmSync as rmSync3,+ readFileSync as readFileSync24,+ rmSync as rmSync4,writeSync as writeSync8,- closeSync as closeSync11+ closeSync as closeSync12} from "node:fs";-import { dirname as dirname23, join as join31 } from "node:path";+import { dirname as dirname23, join as join32 } from "node:path";// src/core/partner/codegraph.tsinit_config();init_fs_utils();import { existsSync as existsSync16, readdirSync as readdirSync7, realpathSync as realpathSync4 } from "node:fs";-import { dirname as dirname22, join as join30, resolve as resolve16 } from "node:path";+import { dirname as dirname22, join as join31, resolve as resolve17 } from "node:path";// src/core/project-manifests.tsvar MANIFEST_ECOSYSTEM = Object.freeze({@@ -9948,9 +10138,9 @@ function isCodeProject(dir) {try {if (!existsSync16(dir))return false;- if (!isDir(join30(dir, ".git")))+ if (!isDir(join31(dir, ".git")))return false;- return CODE_MANIFEST_FILES.some((m) => existsSync16(join30(dir, m)));+ return CODE_MANIFEST_FILES.some((m) => existsSync16(join31(dir, m)));} catch {return false;}@@ -9963,7 +10153,7 @@ function findCodeProjects(opts) {const consider = (raw) => {if (scanned >= limit)return;- const path = resolve16(raw);+ const path = resolve17(raw);if (seen.has(path))return;seen.add(path);@@ -9974,7 +10164,7 @@ function findCodeProjects(opts) {found.push(path);};consider(opts.cwd);- const vaultParent = dirname22(resolve16(opts.vault));+ const vaultParent = dirname22(resolve17(opts.vault));if (isDir(vaultParent)) {let entries2 = [];try {@@ -9986,7 +10176,7 @@ function findCodeProjects(opts) {for (const name of entries2) {if (scanned >= limit)break;- consider(join30(vaultParent, name));+ consider(join31(vaultParent, name));}}for (const extra of opts.scanExtraPaths ?? []) {@@ -10128,7 +10318,7 @@ function codegraphDisabledResult() {};}function evaluateProjectStatus(project, deps) {- const indexDir = join30(project, ".codegraph");+ const indexDir = join31(project, ".codegraph");let indexed;try {indexed = statOrAbsent(indexDir)?.isDirectory() === true;@@ -10215,11 +10405,11 @@ function checkVaultWriteable(vault) {fix: `mkdir -p "${vault}"`};}- const probe = join31(vault, ".open-second-brain-doctor-test");+ const probe = join32(vault, ".open-second-brain-doctor-test");try {const fd = openSync10(probe, "w");- closeSync11(fd);- rmSync3(probe);+ closeSync12(fd);+ rmSync4(probe);} catch (exc) {return {name: "vault_writeable",@@ -10238,9 +10428,9 @@ function checkConfigWriteable(config) {createdForCheck = true;const fd = openSync10(config, "a");writeSync8(fd, "");- closeSync11(fd);+ closeSync12(fd);if (createdForCheck)- rmSync3(config);+ rmSync4(config);} catch (exc) {return {name: "config_writeable",@@ -10276,7 +10466,7 @@ function loadJsonManifest(path, name) {}let data;try {- data = JSON.parse(readFileSync23(path, "utf8"));+ data = JSON.parse(readFileSync24(path, "utf8"));} catch (exc) {return {result: {@@ -10402,7 +10592,7 @@ function checkHermesManifest(path) {}let text;try {- text = readFileSync23(path, "utf8");+ text = readFileSync24(path, "utf8");} catch (exc) {return {name: "hermes_manifest",@@ -10451,7 +10641,7 @@ function checkOpenclawManifest(path) {}function checkOpenclawInstallability(repoRoot) {const results = [];- const pkgPath = join31(repoRoot, "package.json");+ const pkgPath = join32(repoRoot, "package.json");const { result, data } = loadJsonManifest(pkgPath, "openclaw_package_json");results.push(result);if (!data)@@ -10482,7 +10672,7 @@ function checkOpenclawInstallability(repoRoot) {});continue;}- const entryPath = join31(repoRoot, entry);+ const entryPath = join32(repoRoot, entry);const problem = manifestFileProblem(entryPath);if (problem === null) {results.push({@@ -10518,10 +10708,10 @@ function doctor(opts) {results.push(checkConfigWriteable(opts.config));if (opts.repoRoot) {const root = opts.repoRoot;- results.push(checkClaudeManifest(join31(root, ".claude-plugin", "plugin.json")));- results.push(checkCodexManifest(join31(root, ".codex-plugin", "plugin.json")));- results.push(checkHermesManifest(join31(root, "plugins", "hermes", "plugin.yaml")));- results.push(checkOpenclawManifest(join31(root, "openclaw.plugin.json")));+ results.push(checkClaudeManifest(join32(root, ".claude-plugin", "plugin.json")));+ results.push(checkCodexManifest(join32(root, ".codex-plugin", "plugin.json")));+ results.push(checkHermesManifest(join32(root, "plugins", "hermes", "plugin.yaml")));+ results.push(checkOpenclawManifest(join32(root, "openclaw.plugin.json")));results.push(...checkOpenclawInstallability(root));}const cg = checkCodegraph({@@ -10536,10 +10726,10 @@ function doctor(opts) {}// src/core/identity-reminder.ts-import { readFileSync as readFileSync24 } from "node:fs";-import { dirname as dirname24, resolve as resolve17 } from "node:path";+import { readFileSync as readFileSync25 } from "node:fs";+import { dirname as dirname24, resolve as resolve18 } from "node:path";import { fileURLToPath as fileURLToPath2 } from "node:url";-var TEMPLATE_PATH = resolve17(dirname24(fileURLToPath2(import.meta.url)), "..", "..", "templates", "identity-reminder.txt");+var TEMPLATE_PATH = resolve18(dirname24(fileURLToPath2(import.meta.url)), "..", "..", "templates", "identity-reminder.txt");var RUNTIME_TARGET = Object.freeze({hermes: "hermes",openclaw: "openclaw"@@ -10556,7 +10746,7 @@ function loadReminderTemplate() {if (commonTemplateCache !== undefined)return commonTemplateCache;try {- commonTemplateCache = readFileSync24(TEMPLATE_PATH, "utf8").trimEnd();+ commonTemplateCache = readFileSync25(TEMPLATE_PATH, "utf8").trimEnd();return commonTemplateCache;} catch (err) {const message = err instanceof Error ? err.message : String(err);@@ -10565,8 +10755,8 @@ function loadReminderTemplate() {});}}-var TEMPLATES_DIR = resolve17(dirname24(fileURLToPath2(import.meta.url)), "..", "..", "templates");-var PER_TARGET_PATHS = Object.freeze(Object.fromEntries(KNOWN_RUNTIME_TARGETS.map((t) => [t, resolve17(TEMPLATES_DIR, `identity-reminder.${t}.txt`)])));+var TEMPLATES_DIR = resolve18(dirname24(fileURLToPath2(import.meta.url)), "..", "..", "templates");+var PER_TARGET_PATHS = Object.freeze(Object.fromEntries(KNOWN_RUNTIME_TARGETS.map((t) => [t, resolve18(TEMPLATES_DIR, `identity-reminder.${t}.txt`)])));var TEMPLATE_CACHE = new Map;function tryReadTargetTemplate(target) {const cached = TEMPLATE_CACHE.get(target);@@ -10574,7 +10764,7 @@ function tryReadTargetTemplate(target) {return cached;let body;try {- body = readFileSync24(PER_TARGET_PATHS[target], "utf8").trimEnd();+ body = readFileSync25(PER_TARGET_PATHS[target], "utf8").trimEnd();} catch (err) {if (err.code !== "ENOENT")throw err;diff --git a/openspec/changes/archive/2026-11-13-store-hardening/.openspec.yaml b/openspec/changes/archive/2026-11-13-store-hardening/.openspec.yamlnew file mode 100644index 00000000..4d5ada58--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/.openspec.yaml@@ -0,0 +1,2 @@+schema: osb+created: 2026-11-11diff --git a/openspec/changes/archive/2026-11-13-store-hardening/brainstorm.md b/openspec/changes/archive/2026-11-13-store-hardening/brainstorm.mdnew file mode 100644index 00000000..06db94cd--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/brainstorm.md@@ -0,0 +1,143 @@+# Brainstorm++## Problem++Four follow-ups queued by the phase 2 review of `brain-store-and-topics`+(tasks 3.17, 3.15, 3.16 and 3.8 there), decided there as work to do; this+brainstorm settles how. 3.17: readers that show store log events to a+caller below local reach without the rule of change `lane-pages-split`+(design D3). 3.15: a shard read only in part fails closed with no doctor+finding and no repair. 3.16: no verb takes a held partition off a machine+on purpose. 3.8: a lane page does not read back verbatim from its+`lane-entry` record (OCR F291). Two executors build it in parallel: Part C+(3.17, 3.15) and Part Z (3.16, 3.8).++## Related work++No other change in flight under `openspec/changes/`. One open pull request+touches the search index's Windows rename, not these modules. The three+store pull requests merged last (core, migration, lane split) are the base.++## Doubts++1. 3.17: whether a filter at the reader seam can stay out of computations+ that write (dream, evidence counters).+2. 3.17: the timeline events and the request view carry no private mark+ today; which surfaces depend on them.+3. 3.17: a guard that stops the next new reader from leaking.+4. 3.15: whether a repair may touch another device's shard; how a+ quarantine travels to the other machines; a newer-format record.+5. 3.15: whether rollback and the late import should wait for a pending+ blob instead of refusing.+6. 3.16: what the verb refuses (`shared`, an identity file still present,+ store files left in the folder) and which device-local state it clears.+7. 3.8: the privacy judge over a raw copy (a frontmatter comment), older+ readers, the line cap, moves and the equality check.+8. Files both parts touch; Windows; the OpenClaw bundle.++## Approaches considered++3.17 (author): a. carry the private mark into timeline events and fix the+request view, then fix each reader; b. a reader option that drops private+events at the seam; c. a per-request reach scope the readers consult.+Guard: a census of log reader call sites. Partner: a, amended; the+`brain_query` preference mode also skips the reference rule its sibling+modes apply, so it leaks shared events naming unreadable records and its+telemetry count; counts, timestamps and sizes moved by a withheld event+are leaks too.++3.15 (author): a. a device-local quarantine ledger keyed by shard and line+hash; b. a synced quarantine record in the shard's partition; c. rewriting+the shard. Doctor: one finding per gap kind. Partner: a, with atomic+ledger writes, list and release, a rule for the index's stale+`corrupt_lines` counter, and no quarantine of this device's own live tail.++3.16 (author): a. remove the held marker only while the identity file is+absent; b. also purge device-local state of the partition; c. refuse while+store files remain. Partner: a, b and c, plus a "partitions left" record,+marker cleanup by marker content on Windows, and a lock against a+concurrent re-mark.++3.8 (author): a. an optional `raw` field on every record; b. `raw` only+when the parts do not reproduce the bytes; c. a canonical serialiser.+Partner: b, keeping the parse-back check, stripping regions from `raw` on+delivery, one privacy rule for write and fold.++## Decision++3.17: approach a. Two roots are fixed: `requestRefView` withholds below+local reach, and `TemporalEvent` carries the mark that `eventAtReach`+honours. `brain_query` preference mode filters like its sibling modes.+The per-surface readers (agent provenance, backlinks, morning brief, idea+discovery, note write logs, the three timeline builders that never filter)+and every counting reader with a remote surface (most-applied, status,+the maintenance flag, the cost meter, the token footprint) ignore withheld+events below local reach. Guard: a closed log-reader census plus one+remote-reach test per surface with one private event that names nothing.+The doctor's orphan-session check stays whole (a computation: withholding+there would report a false orphan and its repair would detach a valid+reference); its finding texts quote no event. Dream and evidence keep+every event; their use of private evidence is queued as task 3.18.++3.15: approach a. One finding per gap, with one finding code per gap kind+(corrupt line, newer format, pending blob), naming the shard, the offset+and the repair command. The+ledger is matched by shard, offset and the hash of the line bytes;+written atomically under a local lock; `o2b store shard+quarantine|list|release`. Never a newer-format line; a pending blob only+past seven days and never in `settings` (it may be a deny rule); a record+whose blob arrives later reads normally; the last line of this device's+own shard is refused. Rollback, the late import, migration and legacy+removal keep refusing on any pending blob, quarantined or not (no wait+loop). The index counter keeps counting; nothing reads it as a gap.++3.16: approaches a, b and c. `o2b store partition leave <partition>`+refuses `shared`, refuses while the identity file is present, refuses+leftover files without `--force`, exits 0 with nothing to leave, removes+the marker by the key function that wrote it, clears the partition's+seen-shard entries and the digest caches, and leaves the index to drop+the rows on its next catch-up (proved by a test). An identity file that+returns is held again by the existing rule.++3.8: approach b. `raw` only when the parts do not reproduce the bytes and+it parses back to them; text writes and moves carry it, structured writes+never do; `sameEntry` compares it; the write-time privacy judge reads it;+a reader serves it only when it parses back. No format version bump.++## Rejected++- 3.17 b, a seam filter: it would also filter dream and the evidence+ counters, and the read-pass cache is keyed by store and vault only.+- 3.17 c, an ambient scope: a remote `brain_dream` would compute a+ different dream per caller, and the pass cache would mix reaches.+- 3.17, splitting the four files past 700 lines first: a change of its+ own (task 3.19); here they get call-site edits only.+- 3.15 b, a synced quarantine: one machine's verdict would drop a line on+ machines that read it correctly (a blob that arrived there).+- 3.15 c, rewriting a shard: it races the owner's next append and sync,+ and moves the index watermark.+- 3.15, quarantine under `brain doctor --repair`: that verb needs a vault;+ the store verb works on a vault-free machine.+- 3.16, a "partitions left" record: no unsafe sequence found. Shards that+ return without `store.json` are not held, as on a machine that never+ held the partition, and the verb refuses while files remain.+- 3.16, marker cleanup by content on Windows: the open checks only the key+ of the configured root, which is the key the verb removes.+- 3.16, a lock against a concurrent re-mark: a writing open re-marks only a+ partition whose identity file is present, which the verb refuses.+- 3.8 a: `raw` on every record doubles every body for no gain.+- 3.8 c: a lossless serialiser changes the shared frontmatter writer.+- 3.8, `raw` in the migration: a changed record sum would re-append every+ already imported entry on the next transition run.++## Process++Partner: an agent of a different model family, read-only, one round. Its+summary reached the author whole and its full report in part. Its claims+were checked in the tree: the two roots, the preference mode leak and its+telemetry count, the morning brief notes, the agent provenance and the+counting readers were confirmed; the claim that the operator snapshot+reads no log events was wrong (its stale count builds the timeline), and+so was the claim that `osb://active` passes no reach (it passes+`restricted`, which `renderActiveForReader` then drops). No open questions+remain.diff --git a/openspec/changes/archive/2026-11-13-store-hardening/design.md b/openspec/changes/archive/2026-11-13-store-hardening/design.mdnew file mode 100644index 00000000..29d46079--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/design.md@@ -0,0 +1,614 @@+# Design: harden the brain store++See proposal.md for why and brainstorm.md for the alternatives. Four+independent decisions, built in two parts by two executors in parallel:+Part C (D1 to D6: tasks 3.17 and 3.15) and Part Z (D7, D8: tasks 3.16+and 3.8), then merged by an integrator (D9). A name marked (new) does not+exist yet; every other name exists in the tree this change was written+against.++## Context++- `src/core/store/read/log-events.ts` sets `fromPrivatePartition: true` on+ every `BrainLogEntry` read from the `private` partition (change+ `lane-pages-split`, D3). Every store log reader (`readLogEvents`,+ `readLogEventsReport`, `readStoreLogEvents`, `storeLogDays`,+ `logDayReader`, `logDaysWithDevices`, `readAllLogEntries`) returns+ entries with the mark. `logEntryAtReach` and `logEntriesAtReach`+ (`src/core/brain/log-events-at-reach.ts`) drop marked entries when the+ view's `withholdsPrivate` is set; `reachView` sets it below local reach,+ `readerRefView(vault, readable, withholdsPrivate = false)` only when+ asked.+- `requestRefView` (`src/mcp/brain/reach-events.ts`) calls+ `readerRefView(ctx.vault, readableAtContextReachOrUndefined(ctx))`+ without the third argument, so the analytics and brief tools that build+ on it never withhold. `normaliseLogEntry`+ (`src/core/brain/temporal/build-index.ts`) drops the mark, and+ `TemporalEvent` (`src/core/brain/temporal/types.ts`) has no field for+ it, so `eventAtReach` cannot see it; a `note` names nothing and passes.+- `src/core/store/reader.ts`: `readShard` returns records and+ diagnostics; `shardReadGap(read)` is the one place that decides a shard+ read only in part (a pending blob, any non-torn diagnostic). Its callers:+ `read/stream-read.ts`, `state-digest.ts` (`readPartition`, which feeds+ `store-migration/journal.ts` `readHeldRecords`), `view/catch-up.ts`,+ `brain/attestation/record.ts`.+- `src/core/store/held-ledger.ts` keeps one marker file per store root and+ held partition (`held-partitions/<sha256(resolve(root))[0:32]>.<partition>`+ under `storesStateDir()`); `lostHeldPartitions` makes+ `heldPartitionsChecked` (`store-open.ts`) throw `HeldPartitionLostError`;+ the doctor reports its message as `store-partition-lost`+ (`src/core/brain/doctor/store-check.ts`).+- `src/core/store/lane-pages.ts` (627 lines) `writeLaneText` splits a text+ into fields and text with `laneTextParts`+ (`src/core/store/lane-entry-store.ts`); `readLaneText` renders them back+ with `formatFrontmatter` or `JSON.stringify`, which loses bytes.+- Files already past the 700-line rule that this change must touch:+ `src/core/brain/digest.ts` (1234), `src/mcp/brain/query-tools.ts` (1456),+ `src/mcp/resources.ts` (994), `src/mcp/brain/brief-tools.ts` (818),+ `src/mcp/brain/context-tools.ts` (984), `src/mcp/brain/recall-tools.ts`+ (1766), `src/mcp/brain/knowledge-tools.ts` (1588) and the data table+ `src/cli/command-manifest.ts` (1516).++## Goals / Non-Goals++**Goals:** the four behaviours of the specs; no file under 700 lines+crosses 700 because of this change; the eight files already past it+(Context) grow by call-site edits only (an argument or an option added at+a call, an import line; any logic lives in a helper in another module);+`lane-pages.ts` stays under 700; Part C and Part Z build without each+other's code.++**Non-Goals:**++1. Splitting the files past 700 lines: task 3.19 of+ `brain-store-and-topics` (added to that change's `tasks.md` in this+ change's tree, unchecked; task 14.5 confirms it is there).+2. Dream and evidence computations (`evidence.ts`, `dream-refresh.ts`,+ the review candidates of the operator snapshot) that let private+ apply-evidence move a shared preference's confidence or status: task+ 3.18 (added the same way, unchecked). They keep reading every event.+3. The token footprint's page buckets: they count private pages, a page+ aggregate, not a log reader. Only its log bucket follows D2.+4. Raw text in the migration's lane entries (D8 explains why).+5. Pages the human view rendered from a partition before it was left: the+ view is a user-facing folder, not device-local state; the view writer+ re-renders from the held partitions.+6. Any change to how log events are written, or to the store format+ version.++## Decisions++### D1. The two roots (Part C, task 3.17)++1. `requestRefView(ctx)` passes+ `contextReach(ctx) !== TRANSPORT_REACH.local` as `readerRefView`'s+ third argument. Every caller (`analytics-tools.ts`, `brief-tools.ts`,+ `recall-tools.ts`) then withholds through `eventAtReach`,+ `eventsAtReach` and `logEntriesAtReach` with no edit of its own.+2. `TemporalEvent` gains `readonly fromPrivatePartition?: true`, set by+ `normaliseLogEntry` from the entry. `eventAtReach` returns `null` for a+ marked event when `refs.withholdsPrivate === true`, as its first line,+ the same placement as in `logEntryAtReach`.+3. `BuildTimelineIndexOptions` gains `readonly withholdsPrivate?: boolean`;+ `collectLogEvents` skips marked entries when it is set (store-only+ machines only; day files carry no mark). For the builders whose output+ no view filters (D2 item 9).+4. `src/core/brain/log-events-at-reach.ts` gains (new)+ `withoutPrivateEntries<T extends { readonly fromPrivatePartition?: true }>(entries: ReadonlyArray<T>, withholds: boolean): ReadonlyArray<T>`,+ the one filter every counting reader of D2 calls.++Alternatives: a seam filter or an ambient per-request scope (brainstorm,+Rejected).++### D2. Each reader with a remote surface (Part C, task 3.17)++Each core reader gains an optional `withholdsPrivate?: boolean` (default+false: local callers and the CLI are unchanged) and skips marked entries+through `withoutPrivateEntries`; each MCP call site passes+`contextReach(ctx) !== TRANSPORT_REACH.local`, or the view's+`withholdsPrivate === true` where it already holds a view.++| # | Core reader (option added) | MCP call sites (call-site edit) |+|---|---|---|+| 1 | `brain_query` preference mode: no core change | `query-tools.ts`: `const evidence = logEntriesAtReach(queryView, res.evidence)`; telemetry counts `evidence.length`; return `evidence.map(serializeLogEntry)` |+| 2 | `computeMostApplied` (`most-applied.ts`, `ComputeMostAppliedOptions`); `renderActive` (`active.ts`, `RenderActiveOptions.withholdsPrivate`, passed to `computeMostApplied`); `renderActiveForReader` sets it from `opts.restricted === true` | none for `brain_context`, `osb://active`, the pre-compress pack (they pass `restricted`); `digest.ts` passes `refs.withholdsPrivate === true` to `computeMostApplied` |+| 3 | `buildMorningBrief` (`morning-brief.ts`, `MorningBriefOptions.withholdsPrivate`): its view is `readerRefView(vault, opts.readable, opts.withholdsPrivate === true)`, and `scanRecentLog` skips every entry for which `logEntryAtReach(refs, e) === null` before the reconcile and note branches; `discoverIdeas` (`idea-discovery.ts`, `DiscoverIdeasOptions.withholdsPrivate`) passes it on | `brief-tools.ts` (view=morning); `knowledge-tools.ts` (`discoverIdeas`) |+| 4 | Agent provenance: `AgentSourceContribution` (`agent-source/types.ts`) gains `readonly fromPrivatePartition?: true`, set by `logContribution` and `noteWriteContribution` (`vault-provider.ts`); `queryAgentSources` (`agent-source/query.ts`) drops a marked contribution when `opts.view?.withholdsPrivate === true`, and strips the field from every contribution it returns (no output change) | none (`brain_agent_query` and `brain_agent_diff` already pass `reachView`) |+| 5 | `buildBacklinkIndex(vault, ownerScope?, opts: { withholdsPrivate?: boolean } = {})` (`backlinks.ts`): `collectLog` skips marked entries; `buildConceptCluster` (`link-graph/concept-cluster.ts`, `BuildConceptClusterOptions.withholdsPrivate`) and `auditMoc` (`link-graph/moc-audit.ts`, `AuditMocOptions.withholdsPrivate`) pass it to their index; the doctor's broken-backlinks check (`doctor/link-checks.ts`, `brokenBacklinkCheck`) passes `DoctorCheckContext.withholdsPrivate`, which `runDoctor` copies from the new `RunDoctorOptions.withholdsPrivate` (its `sources` name `log-<date>` and its target is the id a private event named); `collectDoctorIssues` (`diagnostics.ts`, the repair path's own doctor pass) gains a `withholdsPrivate` parameter it passes to `runDoctor`: `applyRepair` passes `view.withholdsPrivate === true` (its view already joins `reachView(vault, opts.reach)` when a transport set `reach`), `planRepair`'s own call passes `false`, so the remote plan's `unfixable` count of `broken-backlinks` is taken over the same entries | `resources.ts` (`osb://backlinks`, `view.refs.withholdsPrivate === true`); `query-tools.ts` (`brain_backlinks`); `digest.ts` (inbound counts, `refs.withholdsPrivate === true`); `analytics-tools.ts` (`brain_analytics` view=concept_synthesis: take `requestRefView(ctx)` before `buildConceptCluster` and pass `refs.withholdsPrivate === true`; its log linkers carry the event's kind and timestamp); `knowledge-tools.ts` (`brain_moc_audit`: `auditMoc` gets `contextReach(ctx) !== TRANSPORT_REACH.local`; its member `backlinkCount` and buckets count log references); `health-tools.ts` (`brain_doctor`: its `runDoctor` call passes `contextReach(ctx) !== TRANSPORT_REACH.local`; the repair branch needs no edit, it already passes `reach: contextReach(ctx)` to `applyRepair`); the operator summary and the operator snapshot pass their own option to `runDoctor` (rows 8 and 9) |+| 6 | `computeBrainStatus` (`ComputeBrainStatusOptions.withholdsPrivate`) and `computeMaintenanceOverdueFlag(vault, opts = {})` (`status.ts`): `statusLog` and `recentStatusLog` filter entries, and a day with no entry left is not a log day | `tools.ts` (`second_brain_status`); `resources.ts` (`osb://status`); `context-tools.ts` (the overdue flag) |+| 7 | `computeMemoryCostMeter` and `summarizeMemoryWrites` (`memory-cost-meter.ts`, `MemoryCostMeterOptions.withholdsPrivate`, `MemoryWriteFilter.withholdsPrivate`) | `recall-tools.ts` (operation `cost`) |+| 8 | `computeTokenFootprint(vault, opts)` (`token-footprint.ts`): `sumStoreLog` skips marked entries; `collectMaintenanceActions` (`maintenance/collect.ts`, `CollectMaintenanceActionsOptions.withholdsPrivate`) passes it; `buildOperatorSummary` (`trust/operator-summary.ts`, `BuildOperatorSummaryOptions.withholdsPrivate`) passes it to `collectMaintenanceActions` in `safeTopActions` and to `runDoctor` in `safeDoctor` (row 5) | `health-tools.ts` (`brain_doctor` suggested actions); `digest.ts` (its `actions` list, which `osb://digest/latest` and `brain_brief` view=digest render: `collectMaintenanceActions` gets `refs.withholdsPrivate === true`); `brief-tools.ts` (`brain_brief` view=operator: `buildOperatorSummary` gets `withholdsPrivate: view.withholdsPrivate === true`, outside the `view.filtersNothing` spread; `everyArtifactRefView` carries the flag from `reachView`) |+| 9 | `buildTimelineIndex` (D1 item 3) for the builders no view filters: `traceIdeaLineage` passes its `withholdPrivate` into `preferenceLineage`; `buildOperatorSnapshot` (`BuildOperatorSnapshotOptions.withholdsPrivate`) passes it to its own `buildTimelineIndex` call for its stale count and to its `runDoctor` call (row 5; its doctor-warnings count) | `review-tools.ts` (`brain_stale_scan`): its own `buildTimelineIndex(ctx.vault, {})` call, feeding `findStaleEntries`, passes `withholdsPrivate: reviewView(ctx).withholdsPrivate === true` (the tool already builds that view from `reachView(ctx.vault, contextReach(ctx))` below the call; take it before the index), with no `buildOperatorSnapshot` involved; `analytics-tools.ts` (`brain_analytics` view=belief_evolution: take `requestRefView(ctx)` before the index and pass `refs.withholdsPrivate === true`; its evidence rows and transitions are derived from the index, carry the event's time, result and running counts, and never pass `eventAtReach`); `brief-tools.ts` (`brain_brief` view=daily and view=weekly: both index calls pass `requestRefView(ctx).withholdsPrivate === true`; their `status_transitions`, `retired` and `contradictions` are derived from the index and the reference view asks them only what they name); `health-tools.ts` (`brain_status`) passes the option to `buildOperatorSnapshot`; idea lineage needs none (`synthesis-tools.ts` already passes `withholdPrivate`) |+| 10 | `listNoteWrites` (`notes/write-log.ts`, `ListNoteWritesFilter.withholdsPrivate`) and `listWriteDestinations` (`notes/write-destinations.ts`, `WriteDestinationsFilter.withholdsPrivate`); `planNoteRevert` (`notes/revert.ts`, `PlanNoteRevertOptions.withholdsPrivate`) passes it to its selected read only: the universe read, which feeds only the interleaving and creation rules, keeps every write, private ones included, so a private write between two selected shared writes still refuses the plan (`interleaved`), and the refusal names only the selected writes; `ApplyNoteRevertOptions` omits the option, so an apply always reads in full | `query-tools.ts` (`brain_writes`, `brain_write_destinations`; `brain_writes` action `plan_revert`: take `reachView(ctx.vault, contextReach(ctx))` before `planNoteRevert` and pass `view.withholdsPrivate === true`. The CLI apply re-plans in full, so a plan that withheld a private write is refused with `digest_mismatch`, as a plan that withheld an entry is today) |++Settled "verify" rows: a note-write event on a shared target is private+when its operation was private; it is withheld (row 10), because the rule+is "whatever it names". The orphan-session check+(`doctor/orphan-session-check.ts`) quotes no event text: it names a signal+path and its `session_ref`, both from a page the path filter already+judges; it keeps reading every event, because withholding a private+lifecycle event would report a false orphan whose repair detaches a valid+reference. The two evidence lints (`doctor/log-checks.ts`,+`orphanEvidenceCheck` and `evidenceRangeCheck`) do quote an+`apply-evidence` event's timestamp and artifact, so below local reach+(`DoctorCheckContext.withholdsPrivate`) they leave out the events of the+`private` partition (`withoutPrivateEntries`); neither has a fixer+(`orphan-evidence` is `autoRepairable: false`, `malformed-evidence-range`+is on the no-exit list), so the withheld finding changes no repair. The+log snapshot itself stays unfiltered. The+broken-backlinks check does withhold (row 5): withholding hides a+dangling reference from the remote caller only, it creates no finding,+and the code has no fixer (`autoRepairable: false`).++Settled callers without a remote surface (no call-site edit): the CLI+verbs; `o2b brain explorer` (`explorer.ts`: `buildBacklinkIndex` and+`buildOperatorSnapshot`; a server bound to `127.0.0.1` that refuses+non-loopback `Host` headers, and its export is a CLI write);+`profile-doc.ts` (`buildMorningBrief`, reached only from+`o2b brain profile`); the `active.md` stamp and write in `active.ts`+(`renderActive`, a vault file). Settled remote callers that need no+edit: `stale-dependency.ts` (`buildBacklinkIndex` inside the doctor) skips+every log-sourced reference already (`LOG_SOURCE_KIND_PREFIX`);+`brain_health` (`health-tools.ts`) and `scanTriggers` (`triggers/scan.ts`,+`brain_trigger`) run the doctor for `semantic_health` only, which reads+no broken-backlinks finding; `today-dashboard.ts` and `monthly-review.ts`+build a timeline index whose events `brief-tools.ts` filters through+`eventAtReach` (D1 item 1) before any row is derived from them (the today+dashboard regroups the index over the visible events; the monthly review+counts transitions and contradictions over them); `brain_analytics`+view=timeline renders the events themselves through `eventsAtReach`.++### D3. The log-reader census (Part C, task 3.17)++(new) `tests/core/architecture/log-reader-census.test.ts`, in the shape of+`write-site-census.test.ts`: walk `src/**/*.ts`, find every call+`\b(logDayReader|storeLogDays|readStoreLogEvents|readStoreLogEventsReport|readLogEvents|readLogEventsReport|logDaysWithDevices|readAllLogEntries|buildTimelineIndex)\(`+outside a `function <name>(` declaration, and require each calling module+to be a key of a closed table `{ class: "at-reach" | "local-only" | "computation"; reason: string }`.+A calling module missing from the table fails, naming it; a table key+that no longer calls fails as stale. The definers+(`src/core/brain/log-days.ts`, `src/core/store/read/log-events.ts`,+`src/core/brain/temporal/build-index.ts` for its own `readLogEvents`+call) are listed as `computation` with "defines the reader". The table at+the time of writing:++| Module | Class | Reason |+|---|---|---|+| `cli/brain/verbs/temporal-{daily,evolution,stale,timeline,weekly}.ts` | local-only | CLI |+| `core/brain/agent-source/vault-provider.ts` | at-reach | D2 row 4 |+| `core/brain/backlinks.ts` | at-reach | D2 row 5 |+| `core/brain/digest-agent-summary.ts`, `core/brain/digest.ts`, `core/brain/lessons.ts`, `core/brain/event-trace.ts` | at-reach | `logEntryAtReach` with the caller's view |+| `core/brain/doctor/orphan-session-check.ts`, `core/brain/doctor/records.ts` | computation | D2, settled rows |+| `core/brain/dream-refresh.ts`, `core/brain/evidence.ts` | computation | dream and evidence (task 3.18) |+| `core/brain/governance/forget-plan.ts`, `core/brain/profile-doc.ts`, `core/brain/temporal/citations.ts`, `core/discipline/log-counts.ts` | local-only | CLI only |+| `core/brain/idea-lineage.ts`, `core/brain/operator-snapshot.ts` | at-reach | D2 row 9 |+| `core/brain/memory-cost-meter.ts`, `core/brain/most-applied.ts`, `core/brain/morning-brief.ts`, `core/brain/status.ts`, `core/brain/token-footprint.ts`, `core/brain/notes/write-log.ts`, `core/brain/notes/write-destinations.ts` | at-reach | D2 rows 2, 3, 6, 7, 8, 10 |+| `core/brain/monthly-review.ts`, `core/brain/today-dashboard.ts` | at-reach | the caller's `eventAtReach` (D1) |+| `core/brain/portability/okf.ts` | at-reach | private events leave `log.md` (spec `okf-export`) |+| `core/brain/query.ts` | at-reach | `brain_query` filters every mode (D2 row 1) |+| `mcp/brain/analytics-tools.ts`, `mcp/brain/brief-tools.ts` | at-reach | `requestRefView` (D1); derived rows: an index built withholding (D2 row 9) |+| `mcp/brain/review-tools.ts` | at-reach | D2 row 9 |+| `mcp/resources.ts` | at-reach | `osb://log` filters (D3 of `lane-pages-split`) |++Behaviour proof: (new) `tests/mcp/log-private-reach-surfaces.test.ts`,+one test per row of D2 and per root of D1, each with one private event+that names nothing, asserting the remote answer and the local one (the+setup of `tests/mcp/log-private-reach.test.ts`: `makeStoreEnv`,+`switchDevice`, `withOperationPrivacy(true, ...)`, `appendLogEvent`; tools+through `new MCPServer({ vault, configPath }, { reach }).callTool(...)`+as in `tests/helpers/reach-log-fixture.ts` `reachServer`).++### D4. The quarantine ledger and the reader (Part C, task 3.15)++(new) `src/core/store/shard-quarantine.ts` (the read side; the two writes live in+`shard-quarantine-write.ts`, so a reader of the ledger loads no lock module):++1. The ledger file is+ `<storesStateDir()>/shard-quarantine/<sha256(resolve(root))[0:32]>.json`+ (keyed by the store root like `held-ledger.ts`; device-local, never+ synced): `{ "version": 1, "root": "<resolved root>", "lines": [ { "partition", "shard": "<partition-relative path>", "offset": <n>, "sha256": "<hex>", "kind": "corrupt-line" | "blob-pending", "reason": "<text>", "quarantined_at": "<ISO>" } ] }`.+ `sha256` is the hash of the line's bytes from `offset` through its+ newline, newline included, read from the file. A line over the line cap+ is a corrupt line too, so it is hashed as well, up to+ `QUARANTINE_HASH_CAP_BYTES` (64 times the line cap, 1 MiB); past that+ the hash is `null`, the line cannot be quarantined and stays a gap. The+ reader hashes from the descriptor its read holds (`lineSha256AtFd`);+ every other caller (`lineSha256At(path, offset, partitionRoot)`) opens+ the shard through the reader's guarded open (`openRegularFileForRead`+ with the partition root: no symbolic link, special file or linked+ folder), and a failure answers `null`, which matches no entry. Writes+ are read-modify-write under `withLocalLock(<ledger path>, ...)`+ (`local-lock.ts`) with `atomicWriteFileSync` (`fs-atomic.ts`); the+ write side parses the ledger strictly and throws+ `QuarantineLedgerUnreadableError` when the file exists but cannot be+ read (its own detail: removed while it was read, or the read error),+ does not parse, has a `version` other than 1, or holds an entry it does+ not read, instead of rewriting it; `addQuarantine` answers `already`+ only for an entry with the same offset, bytes and kind, and replaces an+ entry of the other kind, and it leaves the file untouched when a+ change is a no-op (`already`, `not-quarantined`). Reads are cached by+ the file's size and mtime, as `seen-shards.ts` caches its ledger. The+ read side (`load`, through `appliedLines`) fails closed: it applies+ entries only from a JSON object whose `version` is+ `QUARANTINE_LEDGER_VERSION` (1) and whose `lines` is a list; a ledger+ that cannot be read, does not parse, has another or no `version`, is+ not an object or has no list of lines quarantines nothing. Within an+ accepted ledger each entry stands on its own: one `isLine` rejects (a+ newer build's kind, a missing field) applies nothing and the others+ still apply. Either way a skipped entry only leaves its line a gap, so+ no ledger the release cannot vouch for makes a shard read as whole;+ the write side stays strict and refuses the same file instead.+2. Exports of `shard-quarantine.ts`: `quarantinedLines(partitionRoot, rel)`+ (offset to line hash and kind, for the reader), `lineSha256AtFd(fd,+ offset)`, `lineSha256At(path, offset, partitionRoot)`,+ `QUARANTINE_HASH_CAP_BYTES`, `probeLineHashAtFd` and `probeLineHashAt`+ (the same hash, or why there is none: `too-long` past the cap,+ `unreadable` otherwise; `lineSha256AtFd` and `lineSha256At` answer+ `null` for both), `listQuarantine` (each shard parsed at most once per+ call, and only when one of its entries still matches; the staleness+ probe hashes each entry's line on its own, opening the shard once per+ entry; a shard whose read throws leaves its matching entries `held`),+ `pendingRecordWrittenMs`, `quarantineStamp`,+ `PENDING_BLOB_QUARANTINE_AGE_MS` (7 days) and+ `PENDING_BLOB_QUARANTINE_AGE_TEXT`; of `shard-quarantine-write.ts`:+ `addQuarantine`, `releaseQuarantine` and+ `QuarantineLedgerUnreadableError`.+3. `readShard` (`reader.ts`): when `options.partitionRoot` is set and the+ ledger names the shard, after the read it marks an item only when both+ its byte offset and the hash of its line's bytes match one ledger+ entry of the matching kind. This holds for every mark alike: a+ non-torn `corrupt-line` diagnostic is marked only by a `corrupt-line`+ entry, and a `pending` record only by a `blob-pending` entry and never+ in a `settings` shard, each on an offset-and-hash match, never on the+ shard or the offset alone, so a+ second corrupt line at another offset of the same shard, or other bytes+ at a quarantined offset, stay gaps. `ShardDiagnostic` and `ReadRecord`+ gain `readonly quarantined?: true`. `shardReadGap` skips a quarantined+ diagnostic and does not count a quarantined pending record, exactly as+ it skips `torn`. Everything else is unchanged: a quarantined pending+ record is still left out by every reader (they keep `bodyState !==+ "pending"`), and when its blob arrives it reads as `blob`, so the entry+ no longer matters. An `unknown-version` diagnostic is never marked.+4. Rollback, the late import, the migration preflight and the legacy+ removal all go through `readHeldRecords` (`store-migration/journal.ts`):+ it adds every shard outside the digest stream whose+ `PartitionShard.pending` is not empty to `partial`, so they refuse on+ any such pending blob, quarantined or not, and still never wait (a+ digest-stream shard is left out of `partial` altogether, as it is of+ `unreadable`: a published digest is no record any of them imports or+ restores). The legacy removal keeps its existing `--force`: it goes+ past a `partial` shard, the pending blob included (a bad line may never+ read again), while a shard that cannot be read at all still refuses+ even with `--force`.+5. The index's `corrupt_lines` counter (`search/schema.ts`) keeps counting+ every bad line; no reader treats it as a gap (the executor confirms by+ search before relying on it, and stops if one does).+6. A partition left on this machine (D7) keeps its ledger entries; they+ are harmless by design: an entry is consulted only when that exact+ shard is read, and matches only the same bytes at the same offset, so+ on a rejoin it releases exactly the line the operator released.++### D5. Doctor findings (Part C, task 3.15)++(new) `src/core/brain/doctor/store-gap-check.ts` exports+`storeGapFindings(store, partial, nowMs, out)`. `readHeldPartitions` in+`store-check.ts` collects the shards its `onUnreadable` reports with+`incomplete: true` (today ignored) and `storeFindings` calls the new+function with them (call-site edits). For each such shard it reads the+shard with `readShard(path, { partitionRoot })` and reports one finding+per gap, path `<partition>/<rel>`:++1. `store-shard-corrupt-line`, error: the offset, the reason, what fails+ closed (capture, settings reads and writes, lane privacy, exact state,+ attestations, the OKF export, migration), and+ `o2b store shard quarantine <partition>/<rel> --offset <n>`. The+ command is named only for a line the verb can hash: the doctor probes+ the line with `probeLineHashAt`, and a `too-long` answer (past+ `QUARANTINE_HASH_CAP_BYTES`, which the verb always refuses as+ `line-too-long`) is reported instead as `store-shard-line-too-long`,+ error, with the same offset and reason and repair at the line's source+ (the shard of the device that wrote it). That code registers no+ command (`doctor-exits.ts` gives the reason), so the finding carries+ no `next_command`. An `unreadable` probe keeps `store-shard-corrupt-line`:+ the shard changed under the doctor, and the verb's own check answers+ `shard-changed`.+2. `store-shard-newer-format`, warning: upgrade Open Second Brain on this+ machine; no quarantine command.+3. `store-shard-blob-pending`, warning while the record (by its clock's+ physical time) is younger than `PENDING_BLOB_QUARANTINE_AGE_MS`, error+ from then on, with the quarantine command once it is allowed and never+ for a `settings` shard. A record whose clock stamp does not parse as a+ time (`pendingRecordWrittenMs` in `shard-quarantine.ts` answers `null`:+ no stamp, or a physical part no date can hold) has no age: the finding+ is an error that says the clock stamp does not parse and offers no+ quarantine by age; the current time is never put in its place.++A shard removed since the partition read (`ENOENT` on the second read) is+skipped: no gap is left. Any other failure of the second read is a+`store-shard-unreadable` error naming the shard and the error code. A+diagnostic other than `corrupt-line` and `unknown-version` (a shard+swapped for a link, an offset past its end) is a `store-shard-unreadable`+error too: that code's next step names no command, since a quarantine+cannot release it.++Below local reach (`StoreSideCheckContext.withholdsPrivate` true, or+`reach` set and not `local`; the context carries both, from `runDoctor`'s+vault pass and from `runStoreSideDoctor`, so `brain_doctor`,+`brain_brief view=operator`, `brain_status` and the repair pass's+`collectDoctorIssues` are covered alike), `storeFindings` removes the+`private` partition from the store it inspects once, before any check+runs (`storeAtReach`). Every store finding is then computed without it:+no unreadable shard, gap or held tail, no record id conflict, clock-ahead+or digest finding, no cross-partition id, loose mode or reserved name of+`private`, and the removal backlog counts only `shared`. Its paths,+offsets, record ids and times are private metadata, so the caller's `ok`,+counts and `trust_verdict` read as if this machine did not hold it. A+lost `private` identity file is still an error (the store is not opened,+so `shared` fails closed too), with a message that does not name the+partition (`lostPartitionFinding`). A store that does not open for any+other reason (a partition's `store.json` that does not parse or carries+no valid store id, partitions of two stores) is still reported as+`store-root-invalid` with the store root, but below local reach with a+generic message instead of the error text, which can name+`private/store.json` or the store ids (`storeOpenFailureDetail`). The write failure counters follow+the same rule, though they are read before the store is: the dual-write+and fail-open status files count every failure, with the last error text+and time, and beside them (`shared_failures`, `shared_last_error`,+`shared_last_failure_at`) only the failures known to concern only+`shared`. A failure is known to concern only `shared` when+`appendStoreRecords` threw it for an operation whose records were all+placed in `shared` (`failedOnlyInShared` in `dual-write.ts`, which looks+at the error and the error it wraps); a failure before placement (device+id, store open, identity check, a builder) or in an operation with a+record in `private` may concern `private`. Below local reach+`store-dual-write-failures` and `store-write-fail-open` report only the+`shared_*` counters, so a failed private write shows no count, time,+error text or path (the error text can name a private shard), and the+skipped dual writes are reported without their count. Local reach and+the CLI doctor are unchanged.++### D6. The operator verb (Part C, task 3.15)++(new) `src/cli/store-shard.ts`, on the model of `src/cli/store-device.ts`:++```+o2b store shard quarantine <partition>/<shard> --offset <n> [--config <file>] [--json]+o2b store shard list [--config <file>] [--json]+o2b store shard release <partition>/<shard> --offset <n> [--config <file>] [--json]+```++Exit 0 for an answer, 1 for a refusal, 2 for a usage error; under+`--json` every refusal and usage error is `{ ok: false, reason, error }`.+Refusal reasons: `no-store` (no partition holds a store identity),+`partition-lost` (`HeldPartitionLostError`: a partition this machine has+held lost its identity file), `store-open-error` (any other+`StoreOpenError`, such as a corrupt identity file),+`store-root-in-vault`, `config`, `not-held`, `unknown-shard`, `no-gap`+(nothing at that offset is a gap), `line-too-long` (the line's hash,+taken before and after it is judged, cannot be taken both times because+the line runs past the hash cap; permanent until the bytes change, so the+refusal gives no advice to retry), `shard-changed` (the two hashes differ,+or one cannot be read, or the shard was removed between the listing and+the read: the shard changed while it was checked; running the command+again checks it once more), `ledger-busy` (another process held the+ledger's device-local lock past the wait budget, `ELOCKED`; transient,+run the command again), `newer-format`, `blob-too-young`,+`settings-blob`, `ledger-unreadable` (the ledger file exists but this+release cannot rewrite it, D4 item 1; it is left as it is),+`blob-clock-unreadable` (a pending record whose clock+stamp does not parse as a time, D5 item 3: it is never quarantined by+age), `own-tail` (the last line of a shard in this device's own folder:+its writer may still be appending; decided from the reader's held tail+too, before `no-gap`, so an unverified last line the reader holds back+answers `own-tail` rather than `no-gap`; another device's held last line+stays `no-gap`, as the reader retries it). The store is inspected read-only+(`inspectStore`); the device id is read without minting+(`readDeviceId`). Answers: `quarantine` `{ ok, status: "quarantined" | "already", partition, shard, offset, kind }`;+`release` `{ ok, status: "released" | "not-quarantined", ... }` (a+release removes the mark; the line is a gap again only if it still is+one, so a `resolved` or `stale` entry reopens nothing);+`list` `{ ok, lines: [{ partition, shard, offset, kind, state: "held" | "resolved" | "stale", quarantined_at }] }`.+Part C adds to the shared files only its own lines (D9).++### D7. Leaving a partition (Part Z, task 3.16)++1. `src/core/store/held-ledger.ts` gains `isHeldPartitionMarked(root, partition): boolean`+ (new) and `forgetHeldPartition(root, partition): boolean` (new: removes+ the marker; `true` when one was there). Both use the existing+ `markerPath(ledgerDir(), root, partition)`, the key function that wrote+ it, so a root spelled as configured finds its own marker on every OS.+ The module docblock says the verb is the one way to take a partition+ off.+2. `src/core/store/seen-shards.ts`: the file-name logic of `ledgerPath`+ moves into (new) `seenLedgerFile(storeRoot, storeId)`, which+ `ledgerPath` calls; (new)+ `forgetSeenPartition(storeRoot, storeId, partition): number` removes the+ partition's entries through `update` and returns how many there were.+ It takes the store id from `shared`, because the left partition has no+ `store.json` any more.+3. `src/core/brain/compiled-digest-path.ts` gains (new)+ `digestCacheRoot(storeId): string`, `join(deviceStateDirFor(storeId), DIGESTS_DIR)`.+4. (new) `src/core/store/partition-leave.ts`: `PartitionLeaveRefusal`+ (`reason: "shared" | "still-held" | "files-left" | "no-store-id" | "clear-failed"`), and+ `leavePartition({ root, partition, force }): LeavePartitionResult` with+ `{ left, partition, cleared: { heldMarker, seenShards, digestCaches } }`,+ in this order: refuse `shared`; refuse while+ `<root>/<partition>/store.json` exists (`still-held`); answer+ `left: false` when the marker is absent (nothing to leave, even with+ files left); refuse a folder that still holds any entry unless `force`+ (`files-left`); read the store id from `shared/store.json`+ (`readStoreId`) BEFORE anything is removed, and refuse `no-store-id`+ when it cannot (the file is missing, or `readStoreId` throws on a+ corrupt one), changing nothing, with a message that says to restore the+ shared partition's identity file first; then remove+ `digestCacheRoot(storeId)`, clear the seen-shard entries+ (`forgetSeenPartition` checks that its write landed, because the+ ledger's `update` never throws), and remove the marker LAST: a failure+ before it refuses `clear-failed` and leaves the machine still holding+ the partition. Leaving+ fails closed: without the store id the seen-shard entries and the+ digest caches, which may hold the partition's text, cannot be found,+ so the verb never reports a leave it could not clear. No lock: a+ writing open re-marks only a partition whose `store.json` is present,+ which the verb refuses, so a concurrent open cannot undo it.+5. (new) `src/cli/store-partition.ts`:+ `o2b store partition leave <partition> [--force] [--config <file>] [--json]`.+ Exit 0 left or nothing to leave, 1 refusal (`shared`, `still-held`,+ `files-left`, `no-store-id`, `clear-failed`, `store-root-in-vault`,+ `config`), 2 usage+ (a partition name not in `STORE_PARTITIONS`, a missing or extra+ argument, an unknown verb). Under `--json` a refusal answers+ `{ ok: false, reason, error }` on stdout and a usage error+ `{ ok: false, reason: "usage", error }` on stdout, the shape+ `src/cli/store-device.ts` already uses; the message also goes to+ stderr as `error: <message>`, followed by the usage text for a usage+ error. The text+ answer names what was cleared, says the search index drops the+ partition's records on its next catch-up, and says that a+ `store.json` that returns by sync or a restore makes the machine hold+ the partition again from its next writing open. JSON:+ `{ ok: true, left, partition, root, cleared: { held_marker, seen_shards, digest_caches } }`.+6. `HeldPartitionLostError` (`store-open.ts`) keeps its text and appends+ `, or run \`o2b store partition leave <p>\` to stop holding it on this machine`+ (one command per lost partition; "them" for two). The doctor finding+ `store-partition-lost` prints that message, so it names the verb with no+ edit of `store-check.ts`.+7. The index: nothing in the verb. After the leave the store opens without+ the partition, `ShardScanner` lists only held partitions, the+ partition's watermarks vanish and `settleOrphans` removes their+ records. A test proves it (tasks).+8. No "partitions left" record (brainstorm, Rejected): shards that come+ back without `store.json` are not held, exactly as on a machine that+ never held the partition, and a `store.json` that comes back is held+ again by the existing rule.++### D8. Raw text of a lane entry (Part Z, task 3.8)++1. (new) `src/core/store/lane-raw.ts` holds `laneTextParts` (moved+ verbatim from `lane-entry-store.ts`, which re-exports it so no importer+ changes) and:+ - `renderLaneText(path, fields, text)`: the store branch of+ `readLaneText` today (`.md`: `formatFrontmatter`; `.json` with empty+ text: `JSON.stringify(fields, null, 2) + "\n"`; else `text`);+ - `rawParsesBack(path, raw, fields, text)`: `laneTextParts(path, raw)`+ gives a body equal to `text` and fields whose `canonicalJson` equals+ that of `fields`;+ - `rawToKeep(path, raw, fields, text)`: `raw` when it is defined,+ differs from `renderLaneText(path, fields, text)` and parses back;+ otherwise `undefined`;+ - `lanePrivacyRow(fields, text, raw?)`: `{ fields, text }`, plus `raw`+ when defined: the one row the write-time privacy judge reads.+ It imports only from `node:*`, `../vault.ts` (`parseFrontmatterText`,+ `formatFrontmatter`), `../integrity/digest.ts` (`canonicalJson`) and+ `../types.ts` (the `FrontmatterMap` type), never `lane-records.ts`,+ `lane-paths.ts`, `lane-entry-store.ts` or `lane-pages.ts` (no import+ cycle).+2. `lane-records.ts`: `laneEntryRecord` input gains `raw?: string`, written+ as `body.raw` after `text` only when defined; `LaneEntryValue` gains+ `readonly raw?: string`; `foldLaneEntries` copies `body.raw` into the+ value when it is a string and ignores it otherwise; `rejection` is+ unchanged, so a record with `raw` folds under the old rules too.+3. `lane-entry-store.ts`: `sameEntry(value, fields, text, raw?)` also+ requires `value.raw === raw`; `isPrivateEntry(vault, fields, text, raw?)`+ judges `lanePrivacyRow(fields, text, raw)`; `entryRecord` gains a last+ parameter `raw?: string` and writes `rawToKeep(path, raw, fields, text)`.+ The dual write already sends a body with a private region anywhere,+ `raw` included, to `private` (`valueCarriesPrivateRegion(record.body)`).+4. `lane-pages.ts`: `readLaneText` returns `value.raw` when it is defined+ and `rawParsesBack(path, value.raw, value.fields, value.text)`, else+ `renderLaneText(path, value.fields, value.text)`. `writeLaneText`+ computes `raw = rawToKeep(path, text, fields, body)` where it computes+ the parts, passes it to `sameEntry`, `isPrivateEntry` and+ `entryRecord`. `moveLanePage` passes `value.raw` when there is no+ rewrite; `moveLaneTree` passes `value.raw`. `writeLanePage` and+ `exact-state.ts` pass none: a structured write replaces an entry that+ had `raw`, since `sameEntry` sees the difference.+5. Privacy at fold: `storedEntryPrivate` judges the entry's partition and+ its fields' visibility markers; a served `raw` parses back to those+ same fields, so its markers are the fields' markers, and references in+ `raw` were judged at write by the same row (item 1).+ The trust boundary, stated: the writer's privacy judge is the only+ judge of `raw`, exactly as it is the only judge of `text`. The fold+ and the readers re-judge neither: a peer that appends records by hand+ can already put any text into `text`, and can put the same into `raw`.+ The read-side guard of item 4 (`rawParsesBack`) does not judge+ privacy; it only stops `raw` from saying something other than the+ entry's own fields and text. Parts of `raw` that the parse drops (a+ frontmatter comment line: `parseFrontmatterText` skips every line that+ starts with `#`) pass that guard, so a hand-appended shared record+ whose `raw` comment names a private page is served as written; the+ spec `lane-entry-fidelity` states this and task 3.4 case 11 pins it.+6. Readers that strip private regions for a caller do it on the string+ `readLaneText` or `readLaneFile` returns, so they strip `raw` as they+ strip `text`; no lane reader strips the fold's `text` itself.+7. A `raw` makes a body larger, so more entries spill into a blob: a+ peer then reads them as pending until the blob syncs, the gap D4+ reports and can release.+8. `FORMAT.md`, lane-entry bullet: the optional `"raw"`, when it is+ written, that a reader serves it only when it parses back and ignores+ it otherwise, that an older reader ignores it, and that it counts for+ privacy and spills into a blob with the body. The format version+ marker stays.+9. Not in the migration (`store-migration/lane-entry-records.ts`,+ `legacy-scan.ts`): a record that gained `raw` would change its sum, and+ the next transition run would re-append every lane entry already+ imported. The rollback export serves `raw` under the reader's rule+ (`exporters.ts` `laneEntryText`): it writes `raw` when it parses back+ to the record's own fields and text, and the rendering otherwise, so+ the byte loss D8 fixes does not come back on rollback.++### D9. Parts, ownership, overlaps, integration++Part Z owns: `src/core/store/held-ledger.ts`, `store-open.ts`,+`seen-shards.ts`, `partition-leave.ts` (new), `lane-raw.ts` (new),+`lane-records.ts`, `lane-entry-store.ts`, `lane-pages.ts`, `FORMAT.md`;+`src/core/brain/compiled-digest-path.ts`; `src/cli/store-partition.ts`+(new); and its tests.++Part C owns: every file of D1 to D6 (`src/mcp/brain/*`, `src/mcp/resources.ts`,+`src/mcp/tools.ts`, the core readers of D2, `src/core/store/reader.ts`,+`shard-quarantine.ts` (new), `store-migration/journal.ts`,+`src/core/brain/doctor/store-check.ts`, `store-gap-check.ts` (new),+`src/cli/store-shard.ts` (new)); and its tests.++Overlaps, each part adding only its own lines at the named place, the+integrator keeping both when they meet:++1. `src/cli/store.ts`: an import, the usage constant joined into `USAGE`,+ one `case` after `case "device"` (Part Z `partition`, Part C `shard`).+2. `src/cli/command-manifest.ts`: one `command(...)` after the `device`+ entry of `store` (Part Z `partition`, Part C `shard`).+3. `docs/cli-reference.md`: one line after `o2b store device list|retire`.+4. `docs/brain-store.md`: Part Z a section "Leaving a partition" at the+ end of "## Setup"; Part C a section "Shards read only in part" before+ "## What the partitions protect, and what they do not".+5. Integrator only: `openclaw/index.js` (it carries `reader.ts`,+ `held-ledger.ts`, `store-open.ts`), `CHANGELOG.md`, the parent+ `tasks.md`. No version bump: this pull request ships no release.++## Risks / Trade-offs++- [A caller of a changed core reader is missed] → the option defaults to+ the old behaviour, so a missed caller stays as leaky as today, never+ breaks; the census names every reader and the surface tests catch the+ known ones.+- [A shared note naming an unreadable page now drops from the remote+ morning brief] → intended: the reference rule every sibling reader+ applies (brainstorm, partner finding).+- [A quarantined pending blob hides a record on this machine for good]+ → only past seven days, never in `settings`, reversible with `release`,+ and the record returns by itself when the blob arrives.+- [The ledger is device-local, so every machine releases its own line]+ → by design: a synced verdict would drop a line another machine reads.+- [Stores keep their copy of `FORMAT.md` without the `raw` paragraph,+ because the version marker does not move] → the copy only describes;+ readers do not parse it.+- [Leaving with `--force` while shards remain] → the shards are ignored+ on this machine as on one that never held the partition; the operator+ asked for it.++## Migration Plan++None on disk: the ledger and the markers are device-local, `raw` is an+optional field. Rollback of the release leaves `raw` ignored and the+quarantine ledger unread (a shard gap fails closed again).diff --git a/openspec/changes/archive/2026-11-13-store-hardening/proposal.md b/openspec/changes/archive/2026-11-13-store-hardening/proposal.mdnew file mode 100644index 00000000..32a48f5d--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/proposal.md@@ -0,0 +1,79 @@+# Proposal: harden the brain store (private log reads, partial shards, leaving a partition, lane page bytes)++## Why++Four follow-ups from the phase 2 review of `brain-store-and-topics`+(tasks 3.17, 3.15, 3.16 and 3.8 there) each leave a store-only machine+either leaking or stuck:++1. A remote MCP caller still reads private log events, their counts and+ their timestamps through readers that bypass the rule of change+ `lane-pages-split` (design D3): analytics, briefs, the preference mode+ of `brain_query`, agent provenance, backlinks, status and others.+2. A shard with one bad line in the middle, a record of a newer format or+ a blob that never arrives makes every protection read fail closed, with+ no doctor finding and no repair short of hand-editing a shard another+ device owns.+3. A machine that stops syncing `private/` refuses every store operation,+ because held partitions are sticky, and no verb clears that on purpose.+4. A lane file written whole on a store-only machine reads back changed:+ a page without frontmatter gains an empty one, `'123'` turns into a+ number, CRLF and JSON formatting are lost (OCR F291).++## What Changes++- Every reader that shows log events, or numbers derived from them, to a+ caller below local reach withholds events of the `private` partition;+ `brain_query` preference mode also applies the reference rule its+ sibling modes apply. A census test fails on any new log reader that has+ not been classified.+- `brain_doctor` names each shard read only in part, by kind, offset and+ repair command. A new verb `o2b store shard quarantine|list|release`+ releases one bad line on this machine without touching the shard;+ a newer-format line is never released, a pending blob only after seven+ days and never in `settings`.+- A new verb `o2b store partition leave <partition>` takes a partition off+ this machine on purpose; the error that names a lost partition and the+ doctor finding name the verb.+- A `lane-entry` record keeps the file's text in an optional `raw` field+ when its fields and text do not reproduce it, so a text-written lane+ file reads back byte for byte. `FORMAT.md` documents the field; the+ format version does not change.++## Capabilities++### New Capabilities++- `store-shard-gaps`: how the store reports a shard it reads only in part+ and how an operator releases one bad line on one machine.+- `held-partitions`: which partitions a machine holds, how a lost one fails+ closed, and how the operator takes one off the machine on purpose.+- `lane-entry-fidelity`: what a lane file written whole on a store-only+ machine reads back as.++### Modified Capabilities++- `remote-log-reads`: adds requirements; every reader with a remote surface+ withholds private events, counts and timestamps ignore them, and+ `brain_query` preference mode applies the reference rule.++## Impact++- Code: `src/mcp/brain/reach-events.ts`, `src/core/brain/log-events-at-reach.ts`,+ the timeline (`src/core/brain/temporal/`), the readers listed in+ design D1 and D2, `src/core/store/reader.ts`, a new+ `src/core/store/shard-quarantine.ts`, the doctor's store section, the+ held-partition ledger and its open, the lane page store+ (`src/core/store/lane-*.ts`), and two new CLI files under `src/cli/`+ (`store-shard.ts`, `store-partition.ts`).+- CLI: two new `o2b store` verbs; `src/cli/command-manifest.ts`,+ `docs/cli-reference.md` and `docs/brain-store.md`.+- Format: `src/core/store/FORMAT.md` gains the optional `raw` field of+ `lane-entry`; no format version bump, older readers ignore the field.+- The OpenClaw bundle (`openclaw/index.js`) is rebuilt: it carries the+ shard reader and the held-partition ledger.+- `openspec/changes/brain-store-and-topics/tasks.md`: tasks 3.8, 3.15,+ 3.16 and 3.17 point here; tasks 3.18 (dream and evidence over+ private apply-evidence) and 3.19 (split the files past 700 lines) are+ added to that change's `tasks.md` in this change's tree.+- No new dependency, no migration. No version bump in this pull request.diff --git a/openspec/changes/archive/2026-11-13-store-hardening/review.md b/openspec/changes/archive/2026-11-13-store-hardening/review.mdnew file mode 100644index 00000000..2f46b77c--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/review.md@@ -0,0 +1,34 @@+# Independent review++## Reviewer++An agent of a different model family than the author, read-only: it wrote+no artifact of this change and checked every claim against the tree this+change was written against. Round 1: 11 findings, verdict "no". The+author checked each finding against the code and updated the artifacts.+Round 2 (the same kind of reviewer): four findings (13 to 16 below),+none blocking, verdict "yes"; all four are fixed.++## Findings++1. design D7 item 4, tasks 1.1, 1.5, 2.1, 2.2, 2.4, spec `held-partitions`: `store partition leave` reported a clean leave while the shared store id was unreadable, so the seen-shard entries and the digest caches (which may hold the partition's text) stayed. Fixed: the verb fails closed; `leavePartition` reads the store id from `shared/store.json` before removing anything and refuses with the new reason `no-store-id` (exit 1), changing nothing, with a message to restore the shared partition's identity file first. D7 items 4 and 5, a new requirement "Leaving fails closed without the shared store id" with its scenario in `held-partitions` (the refusal requirement now says every refusal changes nothing), task 1.1 case 10, task 1.5 steps 5 to 9 and the refusal type, task 2.1 case 9, task 2.2 refusal list, and both docs texts of task 2.4 updated.+2. design D4 item 3, task 9.1, spec `store-shard-gaps`: the quarantine mark could be read as needing only the shard or the offset for a corrupt-line diagnostic. Fixed: D4 item 3 now states that every mark, diagnostic and pending record alike, needs the offset and the line hash to match one ledger entry; the spec adds that rule and the scenario "A second corrupt line in the same shard"; task 9.1 adds "a second corrupt line at another offset in the same shard is still a gap" and a pending record with a ledger offset but another hash.+3. design D8, spec `lane-entry-fidelity`, task 3.4: the trust boundary of `raw` was unstated, and the parse-back guard passes a frontmatter comment. Fixed: D8 item 5 states that the writer's judge is the only judge of `raw`, exactly as of `text`, and that the read-side guard only stops `raw` from saying something other than the entry's own fields and text; the spec adds the requirement "The writer's judge is the only privacy judge of the raw text" with the scenario "A hand-appended shared record with a raw comment" (served as written); task 3.4 case 11 asserts it.+4. proposal, design Non-Goals, task 14.5: tasks 3.18 and 3.19 were claimed to be added by this change with no task adding them. Rejected: both are already in `openspec/changes/brain-store-and-topics/tasks.md` in this change's tree, added with it. Task 14.5 now confirms they are present and open, and stops when either is missing; the Non-Goals wording names where they live.+5. task 6.1, spec `remote-log-reads`: the weekly brief was named but not verified. Fixed: task 6.1 adds `brain_brief` view weekly, and the spec scenario names it.+6. task 7.6, spec `remote-log-reads`: the last dream time had no verifying case. Fixed: the code reads the last dream and the last apply-evidence times from the same `statusLog` scan, so the spec keeps both; its status scenario adds the dream time and task 7.6 adds a private dream event newer than a shared one.+7. spec `lane-entry-fidelity` "Older readers keep working": not testable in one tree. Fixed: the requirement is now "The raw field is optional and never rejects a record", with one scenario for a record without `raw` and one for a non-string `raw`; task 3.3 names its three cases as that requirement's verification.+8. design D2 row 9, task 7.9: `brain_stale_scan` does not go through `buildOperatorSnapshot`. Fixed: row 9 names its own `buildTimelineIndex(ctx.vault, {})` call in `review-tools.ts`, which gets `withholdsPrivate` from the tool's `reviewView(ctx)`; `buildOperatorSnapshot` stays for `brain_status`; task 7.9 names both paths.+9. design D8 item 1, task 3.2: the import list of `lane-raw.ts` disagreed. Fixed: D8 item 1 now allows `node:*`, `../vault.ts`, `../integrity/digest.ts` and `../types.ts`, as task 3.2 does.+10. tasks 0.4, 6.2, 7.8, 3.3, 2.2: paths left to discovery and an unstated `--json` shape. Fixed: task 0.4 lists the existing test files of every D2 reader and of the timeline builders, and 6.2 names the temporal and analytics tests; task 7.8 names the threshold path (`collectMaintenanceActions` calls `computeTokenFootprint(vault)` at the 200000 default, no override) and the test that reaches it with a filler page; task 3.3 names the record builders (`rec`, `authored`, `laneEntryRecord`) and the assertions; task 2.2 states the refusal and usage output on stdout and stderr, and task 2.1 case 8 asserts it.+11. design Goals: the 700-line sentence contradicted itself. Fixed: no file under 700 lines crosses 700 because of this change, and the eight files already past it grow by call-site edits only.+12. design D2 rows 5, 8, 9 and 10, tasks 0.4, 7.5, 7.8, 7.9 and 7.10, spec `remote-log-reads` (found by the author while answering the review): the call-site column missed callers of the D2 core readers that have a remote surface. Fixed: `collectMaintenanceActions` gets the option from `digest.ts` (`osb://digest/latest`, `brain_brief` view=digest) and from `buildOperatorSummary` (`brain_brief` view=operator, through `brief-tools.ts`); `buildBacklinkIndex` gets it from concept synthesis (`brain_analytics`), `brain_moc_audit` and the doctor's broken-backlinks check (`brain_doctor`, its repair preview, the operator summary and the operator snapshot); `planNoteRevert` gets it for `brain_writes` action `plan_revert`. D2 lists the callers left unedited and why (local only, or no log-derived output), the tasks add a remote and a local case for each new call site, and the spec adds the scenarios "Link-derived answers from a private event" and "Maintenance actions from the log footprint" and a revert-plan clause to the note-write scenario.++13. design D7 item 4, task 1.3, task 1.5, spec `held-partitions` (round 2): `leavePartition` removed the held marker before clearing, so a failed clearing left the machine no longer holding the partition with caches that may hold its text, and the seen-shard clear reported a count over a best-effort write. Fixed: the digest caches and the seen-shard entries are cleared first and the marker is removed last; `forgetSeenPartition` checks that its write landed; a failure refuses with the new reason `clear-failed` and the partition stays held.+14. task 2.4 (round 2): the docs line listed four of the six refusal reasons. Fixed: it names all of them, `clear-failed` included.+15. design D2 row 9 (round 2): "two lines below" mis-cited the position of the view. Fixed: "below the call".+16. proposal, Impact (round 2): tasks 3.18 and 3.19 read as added by this change's own list. Fixed: the line says they are added to the parent change's `tasks.md` in this change's tree.++## Verdict++Ready for implementation: yesdiff --git a/openspec/changes/archive/2026-11-13-store-hardening/specs/held-partitions/spec.md b/openspec/changes/archive/2026-11-13-store-hardening/specs/held-partitions/spec.mdnew file mode 100644index 00000000..6d1329e3--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/specs/held-partitions/spec.md@@ -0,0 +1,167 @@+# Spec Delta++## Purpose++Which partitions of the brain store a machine holds, how a partition it+held and lost makes the store fail closed, and how the operator takes a+partition off a machine on purpose.++## ADDED Requirements++### Requirement: A lost held partition fails closed and names the remedies++A machine that held a partition whose `store.json` is now missing SHALL+refuse to open the store, and the refusal and the doctor finding+`store-partition-lost` SHALL name the remedies: restore the partition's+`store.json` from sync or a backup, or, for a partition other than+`shared`, run `o2b store partition leave <partition>`. For a lost+`shared` partition the refusal SHALL name only the restore, because+leaving `shared` is refused.++#### Scenario: private/store.json lost without the verb++- **GIVEN** a machine that held both partitions+- **WHEN** `private/store.json` disappears and the operator runs any store+ command+- **THEN** the command refuses and names+ `o2b store partition leave private`++#### Scenario: shared/store.json lost++- **GIVEN** a machine that held both partitions+- **WHEN** `shared/store.json` disappears and the operator runs any store+ command+- **THEN** the command refuses and names the restore of+ `shared/store.json`, but names no `o2b store partition leave` command++#### Scenario: shared/store.json and private/store.json both lost++- **GIVEN** a machine that held both partitions+- **WHEN** `shared/store.json` and `private/store.json` both disappear and+ the operator runs any store command+- **THEN** the command refuses and names the restores of+ `shared/store.json` and `private/store.json`, but names no+ `o2b store partition leave` command: with `shared/store.json` gone, a+ leave cannot read the store id and would refuse without changing+ anything++### Requirement: Leaving a partition makes the machine hold the rest++`o2b store partition leave <partition>` SHALL take the partition off this+machine's held list once its folder carries no `store.json`; from then on+the store SHALL open with the remaining partitions, with no refusal.++#### Scenario: Leaving private++- **GIVEN** a machine that held both partitions, and the `private` folder+ removed+- **WHEN** the operator runs `o2b store partition leave private`+- **THEN** the verb exits 0+- **AND** the next store command treats the machine as shared-only++### Requirement: Leaving refuses what it cannot do safely++The verb SHALL refuse with exit code 1 and a named reason: `shared`+(leaving it is leaving the store); `still-held` while the partition's+`store.json` is present; `files-left` while the partition's folder still+holds anything, unless `--force` is given, and always when the folder+cannot be listed, so what is left cannot be checked. A partition name+that is not a store partition SHALL be a usage error (exit 2).++#### Scenario: Leaving shared++- **WHEN** the operator runs `o2b store partition leave shared`+- **THEN** the verb exits 1 with reason `shared`++#### Scenario: Files left in the folder++- **GIVEN** a `private` folder with shards and no `store.json`+- **WHEN** the operator runs the verb without `--force`+- **THEN** it exits 1 with reason `files-left` and the machine still+ refuses to open the store+- **AND** with `--force` the verb leaves the partition++#### Scenario: The folder cannot be listed++- **GIVEN** the place of the `private` folder taken by a regular file, so+ the folder cannot be listed+- **WHEN** the operator runs the verb+- **THEN** it exits 1 with reason `files-left`, with or without `--force`++### Requirement: What each refusal changes++On a `shared`, `still-held`, `files-left` or `no-store-id` refusal the+verb SHALL change nothing. On a `clear-failed` refusal it MAY already+have cleared device-local caches or seen-shard entries, but SHALL keep+the held marker, so the machine still fails closed.++#### Scenario: A failed clearing keeps the marker++- **GIVEN** a machine that holds `private` and whose device-local state+ of it cannot be cleared+- **WHEN** the operator runs the verb+- **THEN** it exits 1 with reason `clear-failed`, and the machine still+ holds `private` (the store still refuses to open)++### Requirement: Leaving fails closed without the shared store id++The verb SHALL read the store id from the shared partition's+`store.json` before it removes anything, and SHALL refuse with exit code+1 and reason `no-store-id`, changing nothing, when that file is missing+or unreadable: without the store id it cannot find the partition's+device-local state, so it SHALL NOT report a leave it could not clear.+It SHALL remove the held marker last, so a failed clearing (reason+`clear-failed`) leaves the machine still holding the partition.++#### Scenario: The shared identity file cannot be read++- **GIVEN** a machine that held both partitions, the `private` folder+ removed, and `shared/store.json` missing or not valid JSON+- **WHEN** the operator runs `o2b store partition leave private`+- **THEN** it exits 1 with reason `no-store-id` and a message that says to+ restore the shared partition's identity file first+- **AND** the machine still holds `private` (the store still refuses to+ open), and no seen-shard entry or digest cache is removed++### Requirement: Leaving is idempotent++The `still-held` check (whether the partition's `store.json` is present)+comes before the not-held answer. When that file is absent and this+machine does not hold the partition, the verb SHALL exit 0, say there is+nothing to leave and change nothing; under `--json` it SHALL answer+`{ "ok": true, "left": false }`. So a folder that was synced in without+a writing open recording it is refused as `still-held`, not reported as+nothing to leave.++#### Scenario: Leaving twice++- **WHEN** the operator runs the verb a second time+- **THEN** it exits 0 and reports `left: false`++### Requirement: Leaving clears what this machine kept of the partition++Leaving SHALL clear this machine's device-local state of the partition:+its record of shards seen and the cached digests that may hold its text.+The search index SHALL hold no record of the partition after its next+catch-up.++#### Scenario: The index forgets the partition++- **GIVEN** a private record that the search index returns+- **WHEN** the operator leaves `private` and the index catches up+- **THEN** the index no longer returns the record++### Requirement: A partition that comes back is held again++A left partition whose `store.json` later arrives again (by sync or a+restore) SHALL be held again from the next writing open, under the same+rule as any partition; the verb's output and the documentation SHALL say+so.++#### Scenario: private returns++- **GIVEN** a machine that left `private`+- **WHEN** `private/store.json` and its shards arrive again and a store+ write runs+- **THEN** the machine holds `private` again, and losing its `store.json`+ later fails closed againdiff --git a/openspec/changes/archive/2026-11-13-store-hardening/specs/lane-entry-fidelity/spec.md b/openspec/changes/archive/2026-11-13-store-hardening/specs/lane-entry-fidelity/spec.mdnew file mode 100644index 00000000..f3289fd5--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/specs/lane-entry-fidelity/spec.md@@ -0,0 +1,127 @@+# Spec Delta++## Purpose++What a lane file written whole on a store-only machine reads back as, so+staged bytes survive the trip through a `lane-entry` record.++## ADDED Requirements++### Requirement: A lane file written whole reads back byte for byte++On a store-only machine a lane file written from its whole text SHALL read+back as exactly that text: a page without frontmatter, a quoted value that+would parse as a number or a boolean, CRLF line ends, a missing final+newline and the formatting of a JSON file are all kept.++#### Scenario: A page without frontmatter++- **WHEN** a lane page is written whole as `Just text.\n`+- **THEN** it reads back as `Just text.\n`, with no frontmatter added++#### Scenario: A quoted value++- **WHEN** a page is written whole with the frontmatter line `id: '123'`+- **THEN** it reads back with `id: '123'`, and its parsed `id` is the+ string `123`++#### Scenario: JSON formatting++- **WHEN** a JSON lane file is written whole as `{"a":1}\n`+- **THEN** it reads back as `{"a":1}\n`++### Requirement: The raw text is stored only when the parts lose it++A `lane-entry` record SHALL carry the file's text only when its fields and+text do not reproduce it, and only when that text parses back to them; a+file the parts reproduce SHALL be stored as before.++#### Scenario: A canonical page++- **WHEN** a page is written whole in the form the frontmatter writer+ itself produces+- **THEN** its record carries no raw text++### Requirement: The raw text counts for the entry's privacy++A reference to a private page, or a private region, that appears only in+the raw text (for example in a frontmatter comment) SHALL send the entry+to the `private` partition, as the same reference in its fields or text+does.++#### Scenario: A private reference in a frontmatter comment++- **WHEN** a page is written whole whose frontmatter comment names a page+ reserved as private+- **THEN** its record lands in the `private` partition++### Requirement: The writer's judge is the only privacy judge of the raw text++The writer's privacy judge SHALL be the only judge of the raw text,+exactly as it is the only judge of the text: the fold and the readers+SHALL NOT re-judge either. The read-side check that the raw text parses+back is not a privacy judge; it only stops the raw text from saying+something other than the record's own fields and text.++#### Scenario: A hand-appended shared record with a raw comment++- **GIVEN** a `lane-entry` record appended by hand to the `shared`+ partition, bypassing the writer's judge, whose raw text carries a+ frontmatter comment that names a private page and otherwise parses back+ to the record's fields and text+- **WHEN** the lane file is read+- **THEN** the reader serves the raw text, comment included, as written+ (the same trust a hand-appended `text` already gets)++### Requirement: A raw text that does not match its record is not served++A reader SHALL serve a record's raw text only when it parses back to the+record's own fields and text; otherwise it SHALL render the fields and+text as before, so a peer cannot serve content that differs from the+record's parts.++#### Scenario: A mismatched raw text++- **GIVEN** a synced record whose raw text parses to other fields than+ the record carries+- **WHEN** the lane file is read+- **THEN** the reader returns the rendering of the record's fields and+ text++### Requirement: Structured writes and moves++A structured write (fields and body) SHALL store no raw text and replace+an entry that had one; a move SHALL keep the entry's raw text; a whole-text+write that changes only bytes the parts do not carry SHALL be written, not+skipped as unchanged.++#### Scenario: A quoting-only change++- **GIVEN** an entry written whole with `id: '123'`+- **WHEN** it is written whole again with `id: "123"`+- **THEN** a new record is written and the file reads back with the+ double quotes++#### Scenario: A move++- **WHEN** an entry with raw text is moved+- **THEN** the moved file reads back byte for byte++### Requirement: The raw field is optional and never rejects a record++The raw text SHALL be an optional field of the existing record kind, with+no format version change and no change to the fold's rejection rules: a+record without it SHALL fold exactly as before, and a record whose raw+field is not a string SHALL fold without rejection, with no raw text in+its value.++#### Scenario: A record without raw text++- **WHEN** a `lane-entry` record without the field is folded+- **THEN** the entry's value has its fields and text and no raw text++#### Scenario: A raw field that is not a string++- **WHEN** a `lane-entry` record whose raw field is the number `5` is+ folded+- **THEN** the record is accepted and the entry's value has no raw textdiff --git a/openspec/changes/archive/2026-11-13-store-hardening/specs/remote-log-reads/spec.md b/openspec/changes/archive/2026-11-13-store-hardening/specs/remote-log-reads/spec.mdnew file mode 100644index 00000000..e76cf06d--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/specs/remote-log-reads/spec.md@@ -0,0 +1,133 @@+# Spec Delta++## ADDED Requirements++### Requirement: Every reader with a remote surface withholds private log events++Every MCP tool and resource that renders log events, or text derived from+them, to a caller below local reach SHALL withhold the events the store+holds in its `private` partition, whatever they name: the timeline+analytics, the daily, weekly, monthly, today and morning briefs, idea+discovery, idea lineage, agent provenance, backlinks and the note write+logs. A local caller SHALL see them as before.++#### Scenario: A private note in the remote timeline and briefs++- **GIVEN** a switched machine with one shared note event and one private+ note event that names nothing, both inside the window+- **WHEN** a remote caller reads `brain_analytics` (timeline) and+ `brain_brief` (daily, weekly, morning)+- **THEN** each answer carries the shared note's text and not the private+ note's text+- **AND** a local caller's answers carry both++#### Scenario: Agent provenance below local reach++- **GIVEN** a switched machine where agent `secret-agent` wrote only one+ private log event+- **WHEN** a remote caller runs `brain_agent_query` or `brain_agent_diff`+- **THEN** neither the contribution nor the agent appears, and the counts+ read as if the event did not exist++#### Scenario: Backlinks from a private event++- **GIVEN** a switched machine where the only reference to a shared+ preference is a private log event+- **WHEN** a remote caller reads `osb://backlinks/<that preference>` or+ `brain_backlinks`+- **THEN** the answer lists no backlink from the log and its count is 0++#### Scenario: Link-derived answers from a private event++- **GIVEN** the same machine, a shared MOC note that links that+ preference, and a second private log event that links an artifact id+ with no file+- **WHEN** a remote caller reads `brain_analytics` (concept synthesis) of+ the preference, `brain_moc_audit` of the MOC note, `brain_doctor`, or+ the `brain_doctor` repair preview+- **THEN** no answer lists or counts a reference from either private+ event, and no broken-backlink finding names the missing id+- **AND** a local caller's answers list and count both++#### Scenario: A note write in a private operation++- **GIVEN** a switched machine where a shared note was written inside a+ private operation, so its note-write event is private+- **WHEN** a remote caller runs `brain_writes` or `brain_write_destinations`+- **THEN** the write is not listed and not counted+- **AND** the revert plan `brain_writes` returns for that note's path+ lists no write from the private operation++### Requirement: Counts and timestamps below local reach ignore private log events++A number or timestamp derived from log events and shown to a caller below+local reach SHALL be computed as if the withheld events did not exist:+most-applied counts and order, the last dream and last apply-evidence+times, the count of log days, the maintenance overdue flag, the memory+cost meter, the token footprint of the log, and the stale counts. Every+private event, counted or rendered, stays visible to a local caller.++#### Scenario: Most-applied counts++- **GIVEN** a switched machine with a shared confirmed preference that has+ one shared and two private `applied` evidence events in the window+- **WHEN** a remote caller reads `brain_context`, `osb://active` or+ `osb://digest/latest`+- **THEN** the preference's applied count reads 1+- **AND** a local caller reads 3++#### Scenario: Status timestamps++- **GIVEN** a switched machine whose newest apply-evidence event and+ newest dream event are private+- **WHEN** a remote caller reads `second_brain_status` or `osb://status`+- **THEN** the last apply-evidence time and the last dream time are the+ newest shared ones, or absent when there is none++#### Scenario: Maintenance actions from the log footprint++- **GIVEN** a switched machine whose token footprint crosses the warning+ threshold only with a private event's text counted+- **WHEN** a remote caller reads `brain_doctor`, `osb://digest/latest`,+ or `brain_brief` (digest, operator)+- **THEN** no answer carries the token-footprint maintenance action+- **AND** a local caller's answers carry it++#### Scenario: Cost meter++- **GIVEN** a switched machine with one shared and one private memory+ write in the window+- **WHEN** a remote caller asks `brain_recall_telemetry` for the cost+ meter+- **THEN** the meter counts one write++### Requirement: Preference mode of brain_query applies the reference rule++`brain_query` in preference mode SHALL return, to a caller below local+reach, only the evidence events its sibling modes would return: no+private event, and no event naming a record the caller cannot read. The+result count it records in recall telemetry SHALL be the count it+returned.++#### Scenario: Private and unreadable evidence++- **GIVEN** a switched machine with a shared preference whose evidence is+ one shared event, one private event, and one shared event that names a+ page reserved against remote reads+- **WHEN** a remote caller runs `brain_query` with that preference+- **THEN** the evidence holds only the first event+- **AND** the recall telemetry row of the call counts one result++### Requirement: A new log reader is classified before it ships++Every module that reads store log events SHALL be classified as one that+filters for a remote caller, one that serves only local callers, or a+computation that must see every event, with its reason; a module that+reads log events and carries no classification SHALL fail the build's+test suite, naming the module.++#### Scenario: An unclassified reader++- **WHEN** a module starts calling a store log reader and no+ classification names it+- **THEN** the test suite fails and names the modulediff --git a/openspec/changes/archive/2026-11-13-store-hardening/specs/store-shard-gaps/spec.md b/openspec/changes/archive/2026-11-13-store-hardening/specs/store-shard-gaps/spec.mdnew file mode 100644index 00000000..151a5fb1--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/specs/store-shard-gaps/spec.md@@ -0,0 +1,382 @@+# Spec Delta++## Purpose++How the brain store reports a shard that a machine reads only in part,+and how an operator releases one bad line on one machine without editing+a shard another device owns.++## ADDED Requirements++### Requirement: Doctor names each shard read only in part++`brain_doctor` SHALL report every shard of a held partition that reads+only in part, one finding per gap, naming the shard as+`<partition>/<shard path>`, the byte offset of the line, what fails closed+on it, and the command that repairs it. A line cut short by a crash and+closed by the next append is not a gap and SHALL NOT be reported.++#### Scenario: A corrupt line in the middle++- **GIVEN** a shard whose middle line is not a valid record+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-corrupt-line` naming the shard,+ the offset and `o2b store shard quarantine <partition>/<shard> --offset <n>`++#### Scenario: A record of a newer format++- **GIVEN** a shard with a record whose format version this release does+ not know+- **WHEN** the operator runs the doctor+- **THEN** it reports a warning `store-shard-newer-format` naming the shard+ and the offset and saying to upgrade Open Second Brain, with no+ quarantine command++#### Scenario: A blob that has not arrived++- **GIVEN** a shard with a record whose body blob is absent+- **WHEN** the operator runs the doctor+- **THEN** it reports `store-shard-blob-pending`, a warning while the+ record is younger than seven days and an error from then on, naming the+ quarantine command once it is allowed++#### Scenario: A pending record whose clock stamp does not parse++- **GIVEN** a shard with a record whose body blob is absent and whose+ clock stamp does not parse as a time+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-blob-pending` saying the clock+ stamp does not parse, with no quarantine command and no write time++### Requirement: A corrupt line too long to quarantine names no quarantine command++The doctor SHALL name the quarantine command for a corrupt line only when+the verb can hash that line; a corrupt line past the quarantine hash cap,+which the verb always refuses as `line-too-long`, SHALL be reported as an+error `store-shard-line-too-long` whose message names repair at the+line's source and whose next step names no command.++#### Scenario: A corrupt line too long to quarantine++- **GIVEN** a shard whose middle line is not a valid record and runs past+ the quarantine hash cap+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-line-too-long` naming the+ shard, the offset and repair at the line's source+- **AND** neither the finding nor its next step names a quarantine command++### Requirement: A gap no quarantine can release names no quarantine command++A gap no quarantine can release (the shard was swapped for a link, or cut+below the offset, since the partition read) and a shard read only in part+that cannot be read again for any reason other than its removal SHALL be+reported as an error `store-shard-unreadable` naming the shard, whose next+step names no command; the second case SHALL name the error code. A shard+removed since the partition read SHALL give no finding.++#### Scenario: A shard swapped for a link++- **GIVEN** a shard read only in part that is replaced by a symbolic link+ before the doctor reads it again+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-unreadable` naming the shard+- **AND** neither the finding nor its next step names a quarantine command++#### Scenario: A shard that cannot be read again++- **GIVEN** a shard read only in part that the doctor cannot open again+ (a permission, a lock)+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-unreadable` naming the shard+ and the error code++### Requirement: The doctor withholds private store findings below local reach++Below local reach (a caller whose reach is not local, or one marked as+withheld from private data), every surface that runs the doctor+(`brain_doctor`, `brain_brief view=operator`, `brain_status` and the+repair pass) SHALL compute the store findings as if this machine did not+hold the `private` partition, and that partition SHALL NOT change the+caller's `ok`, counts or `trust_verdict`. Local reach and the CLI doctor+SHALL report everything as before.++#### Scenario: A gap in a private shard below local reach++- **GIVEN** a shard of the `private` partition with a corrupt middle line+- **WHEN** a remote caller runs `brain_doctor`, `brain_brief view=operator`+ or `brain_status`+- **THEN** no finding names the shard, its offset or a record's time+- **AND** `ok`, the error and warning counts and `trust_verdict` are what+ they would be without the gap+- **AND** a local caller's doctor reports the gap++### Requirement: Every store finding about the private partition is withheld below local reach++The findings withheld below local reach SHALL be every store finding that+names the `private` partition or a shard, offset, record id, record time+or device clock of it: a gap, a held tail, an unreadable shard, a record+id conflict, a clock ahead, a digest divergence or silence, a record id+found in both partitions, a loose permission and a reserved device+folder; the removal backlog SHALL count only `shared`. Failed writes+follow the next requirement.++#### Scenario: A private record whose clock is ahead++- **GIVEN** records of the `private` partition whose clock is days ahead+ of now+- **WHEN** a remote caller runs the doctor+- **THEN** no `store-clock-ahead` finding names the `private` partition+- **AND** a local caller's doctor reports it++### Requirement: A failed private write is withheld below local reach++Below local reach the write failure findings (`store-dual-write-failures`,+`store-write-fail-open`) SHALL show only the failures known to concern+only `shared`: every record of the operation was placed in `shared` when+its append failed. For any other failure they SHALL NOT show its count,+time, error text or path, since the error text can name a private shard.+The skipped dual writes SHALL be reported below local reach without their+count.++#### Scenario: A failed private write below local reach++- **GIVEN** a dual write and a fail-open store-only write into the+ `private` partition that failed with an error naming the private path+- **WHEN** a remote caller runs `brain_doctor`+- **THEN** no `store-dual-write-failures` or `store-write-fail-open`+ finding is reported, and nothing names the path+- **AND** a local caller's doctor reports both, with the error text+- **AND** once a write into `shared` fails too, the remote caller sees+ that one failure, with its own error text and time only++### Requirement: A lost private identity file is reported below local reach without its name++A lost `private` identity file follows the same rule: the store is not+opened and every protection fails closed for `shared` too, so below local+reach the `store-partition-lost` error SHALL still be reported, but+without naming the `private` partition. With `shared` lost as well it+SHALL name `shared` and "another partition".++#### Scenario: A lost private identity file below local reach++- **GIVEN** a machine that held the `private` partition whose+ `private/store.json` is gone+- **WHEN** a remote caller runs the doctor+- **THEN** it reports an error `store-partition-lost` that does not name+ the `private` partition++### Requirement: A quarantined line releases the protection reads on this machine++`o2b store shard quarantine <partition>/<shard> --offset <n>` SHALL+record, on this machine only, that the line at that offset with exactly+its current bytes is skipped; the shard SHALL then read as whole on this+machine, the line SHALL stay in the file, and no other machine SHALL be+affected.++#### Scenario: Capture reopens++- **GIVEN** a store-only machine whose capture is closed by a corrupt line+ in a shared shard+- **WHEN** the operator quarantines that line+- **THEN** capture writes again on this machine+- **AND** the shard's bytes are unchanged++Every quarantine mark, of a corrupt line and of a pending blob alike,+SHALL require both the byte offset and the hash of the line's bytes to+match one ledger entry of the matching kind: a `corrupt-line` entry marks+only a corrupt line, a `blob-pending` entry only a pending record, and a+pending record of a `settings` shard SHALL never be marked. The shard or+the offset alone SHALL never mark a line.++#### Scenario: An entry of another kind++- **GIVEN** a `corrupt-line` entry whose offset and line hash match a+ record whose blob is pending+- **WHEN** the shard is read+- **THEN** the pending record is a gap++### Requirement: Quarantining the same bytes as the other kind replaces the entry++Quarantining a line SHALL answer `already` only when the ledger holds an+entry for that offset with the same bytes and the same kind; an entry of+the other kind SHALL be replaced, so the line is marked by its current+kind.++#### Scenario: The same bytes quarantined as the other kind++- **GIVEN** a `corrupt-line` entry whose offset and line hash match a+ record whose blob is pending+- **WHEN** the operator quarantines that record+- **THEN** the verb answers `quarantined`, the entry's kind becomes+ `blob-pending`, and the record no longer counts as a gap++#### Scenario: The line changes++- **GIVEN** a quarantined line+- **WHEN** a different line appears at the same offset (the shard was+ replaced)+- **THEN** the new line is a gap again and the doctor reports it++#### Scenario: A second corrupt line in the same shard++- **GIVEN** a shard with one quarantined corrupt line+- **WHEN** another corrupt line appears at a different offset of the same+ shard+- **THEN** that line is a gap: the protection reads fail closed again and+ the doctor reports it++### Requirement: Some gaps are never quarantined++The verb SHALL refuse, with exit code 1 and a named reason: a line of a+newer format; a pending blob younger than seven days; a pending blob in a+`settings` shard; a pending blob whose record's clock stamp does not parse+as a time; the last line of a shard of this machine's own device, whether+the reader reports it or holds it back for a retry; an offset that holds+no gap.++#### Scenario: A newer-format line++- **WHEN** the operator quarantines a newer-format line+- **THEN** the verb exits 1 with reason `newer-format` and records nothing++#### Scenario: A young pending blob++- **WHEN** the operator quarantines a pending blob written one day ago+- **THEN** the verb exits 1 with reason `blob-too-young`++#### Scenario: A pending blob with an unreadable clock stamp++- **WHEN** the operator quarantines a pending blob whose record's clock+ stamp does not parse as a time+- **THEN** the verb exits 1 with reason `blob-clock-unreadable` and records+ nothing++#### Scenario: A held corrupt last line of this machine's own shard++- **WHEN** the operator quarantines the last line of a shard this machine+ writes, which does not verify and which the reader holds back+- **THEN** the verb exits 1 with reason `own-tail`++### Requirement: A line the verb cannot hash the same way twice is refused by cause++The verb SHALL hash the line before and after it is judged. A line that+runs past the hash cap both times SHALL be refused as `line-too-long`,+which stays so until the shard's bytes change, so the refusal SHALL NOT+say to run the command again. Hashes that differ, or a line that could not+be read through to its end once, SHALL be refused as `shard-changed`,+which is transient: running the command again checks it once more.++#### Scenario: A corrupt line too long to hash++- **WHEN** the operator quarantines a corrupt line longer than the hash cap+- **THEN** the verb exits 1 with reason `line-too-long`, records nothing,+ and does not say to run the command again++### Requirement: Every refusal of the verb has a named reason++The JSON reasons of the verb's refusals SHALL be exactly: `not-held`,+`unknown-shard`, `no-gap`, `newer-format`, `blob-too-young`,+`settings-blob`, `blob-clock-unreadable`, `own-tail`, `line-too-long`,+`shard-changed`, `ledger-unreadable`, and the store open errors+`no-store`, `partition-lost`, `store-open-error`, `store-root-in-vault` and+`config`.++#### Scenario: A refusal under --json++- **WHEN** the operator quarantines an offset that holds no gap with+ `--json`+- **THEN** the verb exits 1 and answers `{ ok: false, reason: "no-gap", error }`++### Requirement: A ledger this release cannot rewrite is left as it is++A ledger file that cannot be read, does not parse, has another version,+or holds an entry this release does not read SHALL be refused with+`ledger-unreadable` and left as it is; a file that cannot be read SHALL+be named as unreadable, not as one that does not parse.++#### Scenario: A ledger this release cannot rewrite++- **GIVEN** a ledger file of another version+- **WHEN** the operator quarantines or releases a line+- **THEN** the verb exits 1 with reason `ledger-unreadable` and the ledger+ file is unchanged++### Requirement: The reader applies entries only from a ledger of the supported version++The reader SHALL apply quarantine entries only from a ledger file that is+a JSON object of the version this release writes with a list of lines. A+ledger of another version, one with no version, one that is not an+object, has no list of lines, cannot be read or does not parse SHALL+quarantine nothing, so every line it names stays a gap.++#### Scenario: A ledger of another version++- **GIVEN** a ledger holding a valid entry for a corrupt line whose+ version is changed to 2+- **WHEN** the shard is read+- **THEN** the line is a gap again, and `o2b store shard list` lists no+ entry++### Requirement: Each ledger entry stands on its own++Within a ledger of the supported version an entry this release does not+read SHALL apply nothing, and the other entries SHALL still apply. A+skipped entry only leaves its line a gap, so it never makes a shard read+as whole.++#### Scenario: An entry this release does not read++- **GIVEN** a ledger of the supported version holding a valid entry and+ an entry of a kind this release does not know+- **WHEN** the shard is read+- **THEN** the valid entry still marks its line, and the unknown entry+ marks nothing++### Requirement: A quarantined pending blob reads normally once it arrives++A record whose pending blob was quarantined SHALL be left out of every+read while the blob is absent, and SHALL read as any other record as soon+as the blob arrives, with no operator action.++#### Scenario: The blob arrives late++- **GIVEN** a quarantined pending blob+- **WHEN** the blob file arrives+- **THEN** readers return the record with its body++### Requirement: Rollback and the late import still refuse on a pending blob++The legacy rollback, the late import of the transition, the migration+preflight and the legacy removal SHALL keep refusing while any held shard+outside the digest stream holds a record whose blob has not arrived,+whether or not it was quarantined; none of them waits for it. A shard of+the digest stream holds published digests, no record any of them imports+or restores, and is not counted. The legacy removal's existing `--force`+goes past such a shard, as it goes past any shard read only in part.++#### Scenario: Rollback with a quarantined pending blob++- **GIVEN** a quarantined pending blob+- **WHEN** the operator runs the rollback+- **THEN** it refuses and names the shard++### Requirement: Quarantines can be listed and released++`o2b store shard list` SHALL print every quarantined line of this+machine's store with its state (`held`, `resolved` when the line now reads+as a record, `stale` when other bytes stand at the offset);+`o2b store shard release <partition>/<shard> --offset <n>` SHALL remove+the entry. Both, and `quarantine`, SHALL take `--config` and `--json`,+exit 0 for an answer, 1 for a refusal and 2 for a usage error, and under+`--json` answer `{ ok: false, reason, error }` for each refusal.++#### Scenario: Releasing an entry++- **GIVEN** a quarantined corrupt line+- **WHEN** the operator releases it+- **THEN** the shard is a gap again and capture closes again++#### Scenario: Releasing what is not quarantined++- **WHEN** the operator releases an offset with no entry+- **THEN** the verb exits 0 and says nothing was quarantined therediff --git a/openspec/changes/archive/2026-11-13-store-hardening/tasks.md b/openspec/changes/archive/2026-11-13-store-hardening/tasks.mdnew file mode 100644index 00000000..e2bb6c0e--- /dev/null+++ b/openspec/changes/archive/2026-11-13-store-hardening/tasks.md@@ -0,0 +1,913 @@+# Tasks: harden the brain store++First read `openspec/project.md`, then this change folder: `proposal.md`,+`design.md`, and the spec deltas under `specs/`; `brainstorm.md` says why+the design is what it is. This list has three parts: Part Z (tasks 3.16+and 3.8 of `brain-store-and-topics`), Part C (tasks 3.17 and 3.15) and+Part I (integration). Two executors build Part Z and Part C at the same+time, each in its own checkout of this branch; an integrator then does+Part I. You were started with one line that names your part: do that part+only, and do not touch a box of another part.++Work in order, one section at a time. Before a section, read the design+decision and the spec delta it names. Tick each box (`- [x]`) as soon as+it is done, so an interrupted run resumes from the first open box.+Partial edits from an earlier attempt may already be in the tree: review+them, keep what is right, finish the rest.++For every behaviour: write the failing test first, run it and see it+fail, then write the code. Change nothing this list does not ask for.+When the code does not match what `design.md` or this list says (a name,+a line, a signature, a test that should fail and passes, or one that+should pass and fails for a reason this list does not name), stop at that+task, write what you found into your report file, and end your run.++## Environment and rules++These rules bind both executors.++1. Work only in the checkout you were started in. Set this in every shell+ before any `bun` command (`z` for Part Z, `c` for Part C):++ ```sh+ export TMPDIR=$HOME/.cache/osb-store-hardening-z-tmp # Part C: ...-c-tmp+ mkdir -p "$TMPDIR"+ ```++ Delete that folder when your part is done.+2. Never read or write the real `~/.config/open-second-brain`, any real+ vault, or the real `~/.local/state/open-second-brain`. Tests get temp+ directories (`mkdtempSync(join(tmpdir(), "osb-<name>-"))`) and remove+ them in `afterEach`.+3. Never stop a process you did not start. Do not commit, push or open a+ pull request. Do not start subagents.+4. English only in code, comments, docs and tests; no exclamation marks;+ the product is written "Open Second Brain" in full. No machine paths,+ host names or account names anywhere in the repository.+5. During a section run only the test files that section names, with+ `bun test <file> [<file> ...]` (add `-t "<name>"` for one case). Never+ run the whole suite (`bun run test`): the integrator runs it once.+6. A test that runs the CLI goes through `runCli` from+ `tests/helpers/run-cli.ts` and passes an explicit empty stdin, as+ `tests/cli/brain-attest.test.ts` does:+ `runCli(args, { env: env(), stdin: "" })`.+7. Do not add `<!-- doc-test: ... -->` markers to any doc.+8. Never rebuild `openclaw/index.js`, never edit `CHANGELOG.md`,+ `package.json` or `openspec/changes/brain-store-and-topics/tasks.md`:+ those belong to Part I.+9. A file that is already past 700 lines (`design.md`, Goals) gets+ call-site edits only: an argument, an option property or an import+ line. Put any logic into a helper in another module.++# Part Z: leaving a partition (3.16) and raw lane text (3.8)++Files Part Z owns: `src/core/store/held-ledger.ts`,+`src/core/store/store-open.ts`, `src/core/store/seen-shards.ts`,+`src/core/brain/compiled-digest-path.ts`,+`src/core/store/partition-leave.ts` (new), `src/cli/store-partition.ts`+(new), `src/core/store/lane-raw.ts` (new), `src/core/store/lane-records.ts`,+`src/core/store/lane-entry-store.ts`, `src/core/store/lane-pages.ts`,+`src/core/store/FORMAT.md`, and the test files this part names.+Files shared with Part C, where Part Z adds only the lines named here:+`src/cli/store.ts`, `src/cli/command-manifest.ts`,+`docs/cli-reference.md`, `docs/brain-store.md`.+Part Z must not touch any other file, in particular `src/core/store/reader.ts`,+`src/core/store-migration/journal.ts`, `src/core/brain/doctor/store-check.ts`,+anything under `src/mcp/`, and the core log readers of design D2.++Your report file is `REPORT-part-z.md` at the checkout root.++## 0. Preconditions (Part Z)++- [x] 0.1 Confirm the tree. `git log --oneline -1` shows the commit+ "feat(store): split the lane page store, keep private log events off+ exports and remote readers (#293)" or a later main. `wc -l+ src/core/store/lane-pages.ts` prints 627, and `wc -l+ src/core/store/held-ledger.ts` prints 91. These names exist (check each+ with `grep -n`): `markerPath` and `ledgerDir` in+ `src/core/store/held-ledger.ts`; `ledgerPath` and `update` in+ `src/core/store/seen-shards.ts`; `const DIGESTS_DIR = "digests"` in+ `src/core/brain/compiled-digest-path.ts`; `export function laneTextParts`+ and `export function sameEntry` in `src/core/store/lane-entry-store.ts`;+ `export function readLaneText` and `export function writeLaneText` in+ `src/core/store/lane-pages.ts`; `case "device":` in `src/cli/store.ts`.+ On any mismatch, stop and report.+- [x] 0.2 Take the baseline. Run+ `bun test tests/core/store/held-ledger.test.ts tests/core/store/seen-shards.test.ts tests/core/store/store-open.test.ts tests/core/store/protection-store.test.ts tests/core/store/lane-records.test.ts tests/core/store/format-doc.test.ts tests/core/lanes/lane-pages.test.ts tests/cli/store-device.test.ts tests/core/search/brain-store-catch-up.test.ts tests/cli/manifest-completeness.test.ts tests/cli/help-surface-parity.test.ts tests/cli/completions.test.ts tests/core/architecture/import-cycles.test.ts`+ and write the pass and fail counts into the report. Everything should+ pass; if anything fails, stop and report.++## 1. Leaving a partition: the core (design D7 items 1 to 4, 6 and 7; spec `held-partitions`)++- [x] 1.1 Test first: create `tests/core/store/partition-leave.test.ts`.+ Setup per test: `parent = mkdtempSync(join(tmpdir(), "osb-partition-leave-"))`,+ `root = join(parent, "store")`, `initStore({ root })` (both partitions),+ then `openStore({ root, device: "dev-a" })` (records both held+ markers in this process's state directory; the test preload isolates+ it); `afterEach` removes `parent`. Import `leavePartition` and+ `PartitionLeaveRefusal` from `src/core/store/partition-leave.ts`,+ `HeldPartitionLostError` and `openStore` from+ `src/core/store/store-open.ts`. Write these cases:+ 1. "refuses the shared partition": `leavePartition({ root, partition: "shared" })`+ throws a `PartitionLeaveRefusal` with `reason` `"shared"`.+ 2. "refuses while private/store.json is present": with nothing removed,+ `leavePartition({ root, partition: "private" })` throws with reason+ `"still-held"`.+ 3. "a lost private/store.json without the verb still fails closed":+ `rmSync(join(root, "private"), { recursive: true, force: true })`;+ `openStore({ root, device: "dev-a" })` throws `HeldPartitionLostError`+ whose message contains `o2b store partition leave private` and still+ contains `from sync or a backup`.+ 4. "leaving private turns the machine shared-only": remove the+ `private` folder as in 3; `leavePartition({ root, partition: "private" })`+ returns `left: true` and `cleared.heldMarker: true`; then+ `openStore({ root, device: "dev-a" }).partitions` has `shared` and not+ `private`, with no throw.+ 5. "nothing to leave changes nothing": after case 4's+ leave, a second `leavePartition` returns `left: false` and+ `cleared.heldMarker: false`.+ 6. "files left refuse without force": remove only+ `join(root, "private", "store.json")`; `leavePartition` throws reason+ `"files-left"`; `openStore` still throws `HeldPartitionLostError`;+ `leavePartition({ root, partition: "private", force: true })` returns+ `left: true`, and `openStore` then succeeds with `shared` only.+ 7. "clears the seen shards and the digest caches": before removing+ anything, append one record with `appendRecord(openStore({ root, device: "dev-a" }), { partition: "private", stream: "events", kind: "decision", agent: "t", body: { text: "x" } })`+ and read it with `readStreamRecords(root, "private", { stream: "events" })`+ (`src/core/store/read/stream-read.ts`; this records the shard as+ seen); take the store id with `readStoreId(join(root, "shared"))`+ (`store-open.ts`); create `join(digestCacheRoot(storeId), "k", "active.md")`+ with `mkdirSync` and `writeFileSync`. Then remove the `private`+ folder and leave: `cleared.seenShards` is at least 1 and+ `cleared.digestCaches` is `true`, and `digestCacheRoot(storeId)` no+ longer exists.+ 8. "a store.json that returns is held again": after case 4's leave, run+ `initStore({ root, partitions: ["shared", "private"] })` (it creates+ `private/store.json` with the shared store id), then+ `openStore({ root, device: "dev-a" })` (records the marker again),+ then remove `join(root, "private", "store.json")`: `openStore` throws+ `HeldPartitionLostError` again.+ 9. "the search index forgets the partition on its next catch-up": use+ `makeStoreIndexFixture` and `LOCAL_DEVICE` from+ `tests/core/search/fixtures/brain-store-index.ts` instead of the+ setup above (call `fx.cleanup()` at the end), and set+ `process.env[STORE_CATCH_UP_INTERVAL_ENV] = "0"` for the test,+ restoring it after, as `tests/core/search/brain-store-catch-up.test.ts`+ does. Steps: `const store = fx.open(LOCAL_DEVICE)`; if+ `store.partitions` lacks `private`, stop and report. Append+ `appendRecord(store, { partition: "private", stream: "events", kind: "decision", agent: "agent-a", body: { text: "the private heron sighting" } })`;+ `fx.writeNote("notes/a.md", "# A\n\nplain note\n")`;+ `await indexVault(fx.config())`; `await catchUpBrainStore(fx.config(), { force: true })`;+ a local `search(fx.config(), { query: "heron sighting", limit: 10, transportReach: TRANSPORT_REACH.local })`+ returns at least one result. Then `recordHeldPartitions(fx.root, ["shared", "private"])`+ (process state), remove `join(fx.root, "private")`,+ `leavePartition({ root: fx.root, partition: "private" })`,+ `await catchUpBrainStore(fx.config(), { force: true })`, and the+ same search returns no result. Imports as in+ `tests/core/search/brain-store-catch-up.test.ts`.+ 10. "an unreadable shared store id refuses and changes nothing": set up+ as in case 7 (append, read, take `storeId`, create the digest cache+ file), keep `const sharedId = join(root, "shared", "store.json")` and+ its text (`readFileSync(sharedId, "utf8")`), remove the `private`+ folder, then write `"{"` to `sharedId`. `leavePartition({ root, partition: "private" })`+ throws a `PartitionLeaveRefusal` with reason `"no-store-id"` whose+ message contains `shared` and `store.json`; `isHeldPartitionMarked(root, "private")`+ (`src/core/store/held-ledger.ts`) is still `true`, and+ `join(digestCacheRoot(storeId), "k", "active.md")` still exists. The+ same holds with `sharedId` removed (`rmSync(sharedId)`) instead of+ corrupted. Then write the saved text back to `sharedId`: the leave+ returns `left: true`, `cleared.seenShards` at least 1 and+ `cleared.digestCaches` `true`.+ Run `bun test tests/core/store/partition-leave.test.ts`: it fails+ (the module does not exist). Note the failure in the report.+- [x] 1.2 In `src/core/store/held-ledger.ts` add, below+ `lostHeldPartitions`:+ `export function isHeldPartitionMarked(root: string, partition: StorePartition): boolean`+ returning `existsSync(markerPath(ledgerDir(), root, partition))`, and+ `export function forgetHeldPartition(root: string, partition: StorePartition): boolean`+ that returns `false` when the marker does not exist, otherwise removes+ it with `rmSync(marker, { force: true })` and returns `true` (import+ `rmSync` from `node:fs`). Replace the docblock paragraph that starts+ "No command takes a partition off this record" with: "One command takes+ a partition off this record: `o2b store partition leave <partition>`+ (`partition-leave.ts`), once the partition's identity file is gone. The+ other remedy is to restore the identity file (sync, or a backup). The+ list is keyed by the store root as configured, like the switch marker."+- [x] 1.3 In `src/core/store/seen-shards.ts`: add+ `function seenLedgerFile(storeRoot: string, storeId: string): string`+ holding the file-name lines of `ledgerPath` (the `rootKey` hash of+ `resolve(storeRoot)`, the `SEEN_SHARDS_FILE` rename, the `join` with+ `deviceStateDirFor(storeId, processDirsEnv())`), and make `ledgerPath`+ call it with `dirname(partitionRoot)` (behaviour unchanged). Add+ `export function forgetSeenPartition(storeRoot: string, storeId: string, partition: StorePartition): number`:+ `path = seenLedgerFile(storeRoot, storeId)`; `count = load(path).get(partition)?.size ?? 0`;+ when `count > 0`, `update(path, (ledger) => { ledger.delete(partition); })`+ and then, because `update` never throws, check that the write landed:+ when `(load(path).get(partition)?.size ?? 0) > 0`, throw+ `new Error(\`could not clear the seen-shard entries of the ${partition} partition in ${path}\`)`;+ return `count`. Extend the module docblock with one sentence: leaving a+ partition (`partition-leave.ts`) clears its entries.+- [x] 1.4 In `src/core/brain/compiled-digest-path.ts` add+ `export function digestCacheRoot(storeId: string): string` returning+ `join(deviceStateDirFor(storeId), DIGESTS_DIR)`, with a one-line doc+ comment ("The folder of every digest cache of the store `storeId`").+- [x] 1.5 Create `src/core/store/partition-leave.ts` with a header+ comment (what it does, design D7 of change `store-hardening`, and that+ a `store.json` that returns is held again by `recordHeldPartitions`).+ Contents:+ - `export class PartitionLeaveRefusal extends Error` with+ `constructor(readonly reason: "shared" | "still-held" | "files-left" | "no-store-id" | "clear-failed", message: string)`+ and `this.name = "PartitionLeaveRefusal"`.+ - `export interface LeavePartitionResult { readonly left: boolean; readonly partition: StorePartition; readonly cleared: { readonly heldMarker: boolean; readonly seenShards: number; readonly digestCaches: boolean } }`.+ - `export function leavePartition(options: { readonly root: string; readonly partition: StorePartition; readonly force?: boolean }): LeavePartitionResult`,+ in exactly this order:+ 1. `partition === "shared"`: throw reason `"shared"`, message+ `the shared partition holds every rule of the brain store; leaving it is leaving the store, which this command does not do`.+ 2. `dir = partitionDir(root, partition)`; when+ `existsSync(join(dir, STORE_IDENTITY_FILE))` (constant from+ `./constants.ts`): throw reason `"still-held"`, message+ `` `${join(dir, STORE_IDENTITY_FILE)} is still here, so this machine still holds the ${partition} partition; stop syncing ${dir}, remove the folder, then run the command again` ``.+ 3. When `!isHeldPartitionMarked(root, partition)`: return+ `{ left: false, partition, cleared: { heldMarker: false, seenShards: 0, digestCaches: false } }`.+ 4. `left = existsSync(dir) ? readdirSync(dir) : []`; when+ `left.length > 0 && options.force !== true`: throw reason+ `"files-left"`, message+ `` `${dir} still holds ${left.length} item(s) of the ${partition} partition without its ${STORE_IDENTITY_FILE}; remove the folder, or pass --force to leave the partition anyway (its files are then ignored on this machine)` ``.+ 5. Before anything is removed: `storeId = readStoreId(partitionDir(root, "shared"))`,+ treating a `null` return and any thrown error alike (catch every+ error). When there is no store id: throw reason `"no-store-id"`,+ message+ `` `${join(partitionDir(root, "shared"), STORE_IDENTITY_FILE)} is missing or unreadable, so the device-local state of the ${partition} partition cannot be found; restore the shared partition's identity file first (sync, or a backup), then run the command again` ``.+ Nothing has changed at this point.+ 6. Clear the device-local state first and the marker last, so a+ failure leaves the machine still holding the partition (it fails+ closed, and `HeldPartitionLostError` still names the verb). Inside+ one `try`: `cacheDir = digestCacheRoot(storeId)`,+ `existed = existsSync(cacheDir)`,+ `rmSync(cacheDir, { recursive: true, force: true })`,+ `digestCaches = existed`; then+ `seenShards = forgetSeenPartition(root, storeId, partition)`. In its+ `catch (err)`: throw reason `"clear-failed"`, message+ `` `could not clear this machine's state of the ${partition} partition (${err instanceof Error ? err.message : String(err)}); the partition is still held here; fix the cause, then run the command again` ``.+ 7. Only now `heldMarker = forgetHeldPartition(root, partition)`.+ 8. Return `{ left: true, partition, cleared: { heldMarker, seenShards, digestCaches } }`.+ It must not import `src/core/store/reader.ts` directly, nor anything under+ `src/mcp/`.+- [x] 1.6 In `src/core/store/store-open.ts`, `HeldPartitionLostError`:+ keep the message as it is and replace its last string piece+ `"from sync or a backup"` by+ `` `from sync or a backup, or run ${partitions.map((p) => `\`o2b store partition leave ${p}\``).join(" and ")} to stop holding ${partitions.length > 1 ? "them" : "it"} on this machine` ``.+ Run `bun test tests/core/store/partition-leave.test.ts tests/core/store/held-ledger.test.ts tests/core/store/seen-shards.test.ts tests/core/store/store-open.test.ts tests/core/store/protection-store.test.ts`:+ everything passes. If case 9 of 1.1 fails because the index still+ returns the record after the catch-up, stop and report (do not change+ any search module).++## 2. Leaving a partition: the verb and the docs (design D7 item 5, D9; spec `held-partitions`)++- [x] 2.1 Test first: create `tests/cli/store-partition.test.ts`, with+ the setup of `tests/cli/store-device.test.ts` (`parent`, `root`) and+ this `env()`:+ `{ [STORE_ROOT_ENV]: root, HOME: join(parent, "home"), XDG_STATE_HOME: join(parent, "state"), LOCALAPPDATA: join(parent, "state"), OPEN_SECOND_BRAIN_CONFIG: join(parent, "config.yaml"), O2B_DEVICE_ID: "dev-a" }`.+ Every test first runs `runCli(["store", "init"], { env: env(), stdin: "" })`+ and expects exit 0 (this records both held markers in the test's state+ folder). Cases:+ 1. "shared is refused": `store partition leave shared` exits 1 and+ stderr contains `shared partition`; with `--json`, stdout parses to+ `{ ok: false, reason: "shared" }` (plus `error`).+ 2. "a present store.json is refused": `store partition leave private`+ exits 1; `--json` reason `"still-held"`.+ 3. "a lost private partition fails closed and names the verb": remove+ `join(root, "private")`; `store device list` exits 1 and stderr+ contains `o2b store partition leave private`.+ 4. "leaving private": after removing the folder,+ `store partition leave private --json` exits 0 and stdout parses to+ an object with `ok: true`, `left: true`, `partition: "private"` and+ `cleared.held_marker: true`; then `store device list` exits 0.+ 5. "the text answer": the same without `--json` exits 0 and stdout+ contains `Left the private partition`, `next catch-up` and+ `holds the private partition again`.+ 6. "leaving twice": a second `--json` run exits 0 with `left: false`;+ its text form says `nothing to leave`.+ 7. "files left": remove only `join(root, "private", "store.json")`;+ the verb exits 1 with `--json` reason `"files-left"`; with `--force`+ it exits 0 and `left: true`.+ 8. "usage errors": `store partition leave` (no partition),+ `store partition leave bogus`, `store partition leave private extra`+ and `store partition` (no verb) each exit 2 and stderr starts with+ `error: ` and contains `usage: o2b store partition leave`; with+ `--json` each prints on stdout an object that parses to+ `{ ok: false, reason: "usage" }` with a string `error`.+ 9. "an unreadable shared store id is refused": remove `join(root, "private")`+ and write `"{"` to `join(root, "shared", "store.json")`;+ `store partition leave private --json` exits 1 and stdout parses to+ `{ ok: false, reason: "no-store-id" }` with an `error` that contains+ `restore the shared partition's identity file first`; without+ `--json` stderr starts with `error: ` and stdout is empty.+ Run `bun test tests/cli/store-partition.test.ts`: it fails. Note it.+- [x] 2.2 Create `src/cli/store-partition.ts` on the model of+ `src/cli/store-device.ts` (its `EXIT`, `parse`, `usage`, `refuse`+ helpers copied and adapted; header comment with the exit codes):+ - `export const STORE_PARTITION_USAGE = "usage: o2b store partition leave <partition> [--force] [--config <file>] [--json]"`.+ - `export function cmdStorePartition(argv: ReadonlyArray<string>): number`:+ `-h`/`--help` prints the usage and returns 0; a first argument other+ than `leave` (or none) is a usage error (exit 2, message+ `missing store partition verb` or `unknown store partition verb: <verb>`).+ - `leave`: flags `force` (boolean), `config` (string), `json`+ (boolean); exactly one positional, which must be one of+ `STORE_PARTITIONS` (`src/core/store/constants.ts`), else a usage error+ `name the partition to leave (shared or private)` or+ `unknown partition: <name>` or `unexpected argument: <arg>`.+ - Resolve `root` with `resolveConfiguredStoreRoot(configPath).root`+ (`src/core/store/store-paths.ts`), call `leavePartition`. Refusals+ (exit 1): `PartitionLeaveRefusal` (its `reason`: `shared`,+ `still-held`, `files-left`, `no-store-id` or `clear-failed`),+ `StoreRootInsideVaultError` (`"store-root-in-vault"`),+ `ConfigReadError` (`"config"`, from `src/core/config.ts`). Anything+ else propagates.+ - Output shapes, exactly those of `store-device.ts`'s `refuse` and+ `usage` helpers: a refusal writes `error: <message>\n` to stderr and,+ under `--json`, `{ ok: false, reason, error }` (with `sortedReplacer`,+ two-space indent, a final newline) to stdout; a usage error writes+ `error: <message>\n<STORE_PARTITION_USAGE>\n` to stderr and, when the+ arguments contain `--json`, `{ ok: false, reason: "usage", error }` to+ stdout. Nothing else is written on a refusal or a usage error.+ - Text answer when `left`:+ `` `Left the ${partition} partition of the brain store at ${root} on this machine.\n cleared: held marker, ${seenShards} seen shard entr${seenShards === 1 ? "y" : "ies"}, ${digestCaches ? "digest caches" : "no digest cache"}\nThe search index drops the ${partition} partition's records on its next catch-up. If ${join(root, partition, "store.json")} returns, by sync or a restore, this machine holds the ${partition} partition again from its next store write.\n` ``.+ When not `left`:+ `` `This machine does not hold the ${partition} partition of the brain store at ${root}; nothing to leave. If ${join(root, partition, "store.json")} arrives, by sync or a restore, this machine holds the ${partition} partition again from its next store write.\n` ``.+ - JSON answer (with `sortedReplacer` from `./helpers.ts`, two-space+ indent): `{ ok: true, left, partition, root, cleared: { held_marker, seen_shards, digest_caches } }`.+- [x] 2.3 In `src/cli/store.ts` add exactly: the import+ `import { cmdStorePartition, STORE_PARTITION_USAGE } from "./store-partition.ts";`+ after the `store-device.ts` import; `\n${STORE_PARTITION_USAGE}` at the+ end of the `USAGE` template; and, right after the `case "device":`+ return line, `case "partition":` returning `cmdStorePartition(rest)`.+ Add one sentence to the file's header comment: "`partition leave`+ takes a partition off this machine (`store-partition.ts`)." In+ `src/cli/command-manifest.ts`, inside the `store` command, right after+ the `device` entry, add:+ `command("partition", "Take a partition off this machine on purpose", [flag("config", "string"), flag("json", "boolean")], [command("leave", "Stop holding a partition whose folder is gone (never shared)", [flag("force", "boolean")])])`.+ Run `bun test tests/cli/store-partition.test.ts tests/cli/store-device.test.ts tests/cli/manifest-completeness.test.ts tests/cli/help-surface-parity.test.ts tests/cli/completions.test.ts`:+ everything passes. If a manifest or parity test fails, fix only the+ manifest entry or the usage text; if it needs anything else, stop and+ report.+- [x] 2.4 Docs. In `docs/cli-reference.md`, right after the line that+ starts `o2b store device list|retire`, add one line in the same+ two-column style:+ `o2b store partition leave Take a partition off this machine on purpose, once its folder no longer carries store.json (stop syncing it and remove the folder first): removes this machine's held marker for it, clears its seen-shard entries and the digest caches, and the search index drops its records on its next catch-up. Refuses shared, a partition whose store.json is still there, a folder that still holds files unless --force, and a store whose shared/store.json is missing or unreadable (restore it first; nothing is changed); leaving a partition this machine does not hold exits 0 and changes nothing. A store.json that returns by sync or a restore makes the machine hold the partition again. --config, --json. Exit 0 for an answer, 1 for a refusal (shared, still-held, files-left, no-store-id, clear-failed, a store root inside the vault, an unreadable config), 2 for a usage error`.+ In `docs/brain-store.md`, at the end of the section "## Setup" (right+ before the heading `## Syncing`), add a subsection `### Leaving a partition`+ of one or two short paragraphs and one fenced `sh` block+ (`o2b store partition leave private`) that says: a machine that held a+ partition refuses every store command once that partition's+ `store.json` is gone; to take the partition off on purpose, stop+ syncing its folder, remove the folder, and run the command; what the+ command clears; that it refuses `shared`, and refuses (changing nothing)+ while `shared/store.json` is missing or unreadable, so restore that+ file first; that a returning `store.json`+ makes the machine hold the partition again. Run+ `bun test tests/docs` and expect the same result as before your edit+ (run it once before editing to compare); a new failure that names+ your text: fix your text.++## 3. Raw lane text (design D8; spec `lane-entry-fidelity`)++- [x] 3.1 Test first: create `tests/core/store/lane-raw.test.ts` for the+ new module `src/core/store/lane-raw.ts` (functions `renderLaneText`,+ `rawParsesBack`, `rawToKeep`, `lanePrivacyRow`, `laneTextParts`). For+ each case, compute `{ fields, body } = laneTextParts(path, raw)` first.+ Cases:+ 1. "a page the parts reproduce keeps no raw": `path = "x.md"`, `raw` is+ `renderLaneText("x.md", { kind: "k" }, "Body.\n")`; `rawToKeep`+ returns `undefined`.+ 2. "a page without frontmatter keeps its raw": `raw = "Just text.\n"`;+ `rawToKeep("x.md", raw, fields, body)` returns `raw`.+ 3. "a quoted value keeps its raw": `raw = "---\nid: '123'\n---\nBody.\n"`;+ `fields.id` is the string `"123"` and `rawToKeep` returns `raw`+ (if `renderLaneText` already writes `id: '123'` byte for byte, use+ `id: "123"` instead and say so in the report).+ 4. "CRLF keeps its raw": `raw = "---\r\nkind: k\r\n---\r\nBody.\r\n"`;+ `rawToKeep` returns `raw`.+ 5. "a missing final newline keeps its raw": `raw = "---\nkind: k\n---\nBody."`.+ 6. "JSON formatting keeps its raw": `path = "x.json"`, `raw = '{"a":1}\n'`.+ 7. "plain text never needs raw": `path = "x.txt"`, `raw = "a\nb"`:+ `rawToKeep` returns `undefined`.+ 8. "a raw that does not parse back is not kept": `rawToKeep("x.md", "---\nkind: other\n---\nBody.\n", { kind: "k" }, "Body.\n")`+ returns `undefined`, and `rawParsesBack` with the same arguments is+ `false`.+ 9. "renderLaneText is the reader's rendering": `.md` gives+ `formatFrontmatter(fields, text)` (`src/core/vault.ts`); `.json` with+ empty text gives `JSON.stringify(fields, null, 2) + "\n"`; `.json`+ with text gives the text; another extension gives the text.+ 10. "the privacy row": `lanePrivacyRow({ a: 1 }, "t")` deep-equals+ `{ fields: { a: 1 }, text: "t" }`, and with a third argument `"r"`+ deep-equals `{ fields: { a: 1 }, text: "t", raw: "r" }`.+ Run `bun test tests/core/store/lane-raw.test.ts`: it fails. Note it.+- [x] 3.2 Create `src/core/store/lane-raw.ts` (header comment: what it+ holds, design D8 of change `store-hardening`). Move+ `laneTextParts` there verbatim from `src/core/store/lane-entry-store.ts`+ with its doc comment, and in `lane-entry-store.ts` replace it with+ `export { laneTextParts } from "./lane-raw.ts";` plus an+ `import { laneTextParts } from "./lane-raw.ts";` for its own use+ (remove only imports the compiler then reports unused). Add:+ - `export function renderLaneText(path: string, fields: Readonly<Record<string, unknown>>, text: string): string`:+ `path.endsWith(".md")` gives `formatFrontmatter(fields as FrontmatterMap, text)`;+ `path.endsWith(".json") && text === ""` gives+ `` `${JSON.stringify(fields, null, 2)}\n` ``; otherwise `text`.+ - `export function rawParsesBack(path: string, raw: string, fields: Readonly<Record<string, unknown>>, text: string): boolean`:+ `const parts = laneTextParts(path, raw); return parts.body === text && canonicalJson(parts.fields) === canonicalJson(fields);`+ (`canonicalJson` from `../integrity/digest.ts`).+ - `export function rawToKeep(path: string, raw: string | undefined, fields: Readonly<Record<string, unknown>>, text: string): string | undefined`:+ `undefined` when `raw === undefined`, when+ `renderLaneText(path, fields, text) === raw`, or when+ `!rawParsesBack(path, raw, fields, text)`; otherwise `raw`.+ - `export function lanePrivacyRow(fields: Readonly<Record<string, unknown>>, text: string, raw?: string): Record<string, unknown>`:+ `raw === undefined ? { fields, text } : { fields, text, raw }`.+ `lane-raw.ts` imports only from `node:*`, `../vault.ts`,+ `../integrity/digest.ts` and `../types.ts`. Run+ `bun test tests/core/store/lane-raw.test.ts tests/core/architecture/import-cycles.test.ts`:+ both pass.+- [x] 3.3 Test first, in `tests/core/store/lane-records.test.ts`, three+ cases: "laneEntryRecord writes raw only when given" (`body.raw` is+ absent without `raw` and equals it with `raw: "x"`); "the fold carries a+ string raw" (`foldLaneEntries` over a record built with `raw: "x"`+ gives an entry whose `value.raw` is `"x"`); "the fold ignores a raw that+ is not a string" (a record whose `body.raw` is `5` folds with no+ rejection and `value.raw` is `undefined`). Build the records as that+ file's `entry` helper does: `rec({ physical: 1, kind: LANE_ENTRY_KIND, body: authored("agent", { lane: "obligations", entry: "obl-a", fields: { s: 1 }, text: "t", raw: "x" }) })`+ (`rec` and `authored` from `tests/core/store/fixtures/fold-records.ts`;+ `raw: 5` for the third case), fold with `foldLaneEntries([record])`,+ read the value with `.entries.get(laneEntryKey("obligations", "obl-a"))!.value`,+ and assert `fold.rejected` is empty for the third case. For the first+ case call `laneEntryRecord` with the input of the file's first test+ ("the builder writes the generic events record"), once without and once+ with `raw: "x"`. These three cases verify the requirement "The raw field+ is optional and never rejects a record" of spec `lane-entry-fidelity`.+ Run the file: the new cases+ fail. Then in `src/core/store/lane-records.ts`: `laneEntryRecord`'s+ input gains `readonly raw?: string`, and its `body` gains+ `...(input.raw !== undefined ? { raw: input.raw } : {})` after `text`;+ `LaneEntryValue` gains `readonly raw?: string` (doc comment: "The+ file's text byte for byte, when the fields and text do not reproduce+ it; served only when it parses back to them (`lane-raw.ts`)"); in+ `foldLaneEntries` the value gains+ `...(typeof record.body["raw"] === "string" ? { raw: record.body["raw"] } : {})`.+ `rejection` is not changed. Run the file again: everything passes.+- [x] 3.4 Test first, in `tests/core/lanes/lane-pages.test.ts`, a new+ `describe("raw lane text on a store-only machine", ...)` that uses+ `makeLaneEnv("store")` only, with these cases:+ 1. "a page without frontmatter reads back verbatim": `writeLaneText`+ of `Just text.\n` to `join(lane.vault, "Brain", "obligations", "obl-raw.md")`;+ `readLaneText` returns `Just text.\n`.+ 2. "a quoted value reads back verbatim": text+ `---\nid: '123'\n---\nBody.\n` (or the variant 3.1 case 3 chose)+ reads back verbatim, and `readLanePage(...)!.frontmatter.id` is the+ string `"123"`.+ 3. "CRLF and a missing final newline read back verbatim".+ 4. "JSON formatting reads back verbatim": `{"a":1}\n` to+ `.../obligations/raw.json`.+ 5. "a rewrite that changes only quoting is written": write+ `---\nid: '123'\n---\nBody.\n`, then `---\nid: "123"\n---\nBody.\n`:+ `lane.laneRecords().filter((r) => r.kind === "lane-entry")` grows by+ one and the read returns the second text; writing the second text+ again adds no record.+ 6. "a structured write replaces the raw": after case 1's write,+ `writeLanePage(vault, page, { kind: "k" }, "New.\n", { overwrite: true })`+ then `readLaneText` returns `formatFrontmatter({ kind: "k" }, "New.\n")`.+ 7. "a move keeps the raw": after case 1's write,+ `moveLanePage(vault, page, join(dir, "obl-moved.md"))` and the new+ path reads back `Just text.\n`; `moveLaneTree` of the folder to+ another folder keeps it too.+ 8. "a private region in a frontmatter comment sends the entry to+ private": text `---\n# <private>secret</private>\nkind: k\n---\nBody.\n`;+ `lane.laneRecords("shared")` is empty and `lane.laneRecords("private")`+ has the entry.+ 9. "a reference only in the raw text is judged": text+ `---\n# see [[pref-nowhere-raw]]\nkind: k\n---\nBody.\n` lands in+ `private` (a preference id with no record cannot be judged and+ counts as private); the same page without the comment line lands+ in `shared`. If the second half lands in `private` too, or the first+ in `shared`, stop and report (do not change the reference judge).+ 10. "a raw that does not parse back is not served": append a+ `lane-entry` record by hand whose `body` has `fields: { kind: "k" }`,+ `text: "Body.\n"` and `raw: "---\nkind: other\n---\nBody.\n"`, as+ the test "stays in the shared partition it lives in, tombstone and+ copy alike" of this file appends one:+ `appendRecord(openStore({ root: lane.env.storeRoot, device: "dev-b" }), { partition: "shared", stream: "events", kind: "lane-entry", agent: "t", body: { authority: "agent", lane: "obligations", entry: "obl-hand", fields: { kind: "k" }, text: "Body.\n", raw: "---\nkind: other\n---\nBody.\n" } })`;+ `readLaneText` of `join(lane.vault, "Brain", "obligations", "obl-hand.md")`+ returns `formatFrontmatter({ kind: "k" }, "Body.\n")`.+ 11. "a hand-appended shared raw with a frontmatter comment is served as+ written": append as in case 10 (entry `obl-hand-comment`) a record+ whose `raw` is `---\n# see [[pref-nowhere-raw]]\nkind: k\n---\nBody.\n`,+ with `fields: { kind: "k" }` and `text: "Body.\n"`; `readLaneText`+ returns that `raw` byte for byte, comment included. This pins the+ trust boundary of design D8 item 5: the writer's judge is the only+ judge of `raw`, as of `text`, and the read-side check only proves the+ raw parses back. If the read returns the rendering instead, stop and+ report.+ Run `bun test tests/core/lanes/lane-pages.test.ts`: the new cases fail+ (except 8, which may already pass, and 10, whose record already reads+ as the rendering: say so in the report).+- [x] 3.5 Implement D8 items 3 and 4.+ In `src/core/store/lane-entry-store.ts`:+ `sameEntry(value, fields, text, raw?: string)` adds+ `&& value.raw === raw` to its condition; `isPrivateEntry(vault, fields, text, raw?: string)`+ passes `lanePrivacyRow(fields, text, raw)` to `rowNamesPrivatePage`+ instead of `{ fields, text }`; `entryRecord` gains a last parameter+ `raw?: string` (after `placement`), computes+ `const kept = rawToKeep(path, raw, fields, text)` and passes+ `...(kept !== undefined ? { raw: kept } : {})` to `laneEntryRecord`.+ In `src/core/store/lane-pages.ts`:+ - `readLaneText`, store branch: after `if (value === null) return null;`+ use+ `if (value.raw !== undefined && rawParsesBack(path, value.raw, value.fields, value.text)) return value.raw;`+ and `return renderLaneText(path, value.fields, value.text);`+ (replacing the two former return lines).+ - `writeLaneText`: in the held-in-store check compute+ `const raw = rawToKeep(path, text, fields, body);` after+ `laneTextParts` and call `sameEntry(storeEntry(vault, path), fields, body, raw)`;+ in the `store` closure compute `raw` the same way after+ `laneTextParts`, pass it as the fourth argument of `isPrivateEntry`+ and as the last argument of `entryRecord` (with `"judged"` as the+ `placement` argument before it).+ - `moveLanePage`: the `entryRecord` call gets the last argument+ `rewrite === undefined ? value.raw : undefined`.+ - `moveLaneTree`: the `entryRecord` call gets the last argument+ `value.raw`.+ - `writeLanePage`: no change.+ Remove only imports the compiler or the linter reports unused. Run+ `bun test tests/core/lanes/lane-pages.test.ts tests/core/store/lane-records.test.ts tests/core/store/lane-raw.test.ts tests/core/lanes`+ : everything passes. `wc -l src/core/store/lane-pages.ts` stays under+ 700 (expected about 630).+- [x] 3.6 `src/core/store/FORMAT.md`: in the `lane-entry` bullet, write+ the shape as+ `{ "authority", "lane": "<lane>", "entry": "<entry id>", "fields": { ... }, "text": "<Markdown>", "raw": "<file text>" (optional) }`+ and add, after the sentence about `text`, these sentences: "`raw`, when+ present, is the file's text byte for byte, written only when `fields`+ and `text` do not reproduce it (a page without frontmatter, a quoted+ value, CRLF line ends, the formatting of a JSON file). A reader serves+ `raw` only when it parses back to the record's own `fields` and `text`,+ and renders `fields` and `text` otherwise; a reader that does not know+ `raw` ignores it. `raw` counts for the entry's privacy like `fields` and+ `text`, and spills into a blob with the rest of the body." Do not touch+ the first line (the format version marker). Run+ `bun test tests/core/store/format-doc.test.ts`: it passes.++## 4. Closing checks (Part Z)++- [x] 4.1 Run each, one at a time, and fix what they report in your own+ files: `bunx tsc --noEmit -p .`, `bun run lint`, `bun run fmt:check`+ (run `bun run fmt` to fix formatting, then check again),+ `bun run check:paths`.+- [x] 4.2 Run every test file Part Z touched or created once more, in one+ command:+ `bun test tests/core/store/partition-leave.test.ts tests/cli/store-partition.test.ts tests/core/store/lane-raw.test.ts tests/core/store/lane-records.test.ts tests/core/lanes/lane-pages.test.ts tests/core/store/held-ledger.test.ts tests/core/store/seen-shards.test.ts tests/core/store/store-open.test.ts tests/core/store/protection-store.test.ts tests/core/store/format-doc.test.ts tests/cli/store-device.test.ts tests/core/search/brain-store-catch-up.test.ts tests/cli/manifest-completeness.test.ts tests/cli/help-surface-parity.test.ts tests/cli/completions.test.ts tests/core/architecture/import-cycles.test.ts tests/docs`.+ All pass. `wc -l` of every source file Part Z changed or created is+ under 700 (`command-manifest.ts` excepted: it was past 700 before).+ `git status --short` lists only Part Z's files (plus `.apb/.gitignore`,+ which is not yours: leave it).++## 5. Report (Part Z)++- [x] 5.1 Write `REPORT-part-z.md` at the checkout root (never commit it;+ at most 60 lines): every Part Z box ticked or why not; the baseline and+ final test counts; the line counts of `lane-pages.ts`,+ `lane-entry-store.ts`, `lane-raw.ts`, `partition-leave.ts`,+ `store-partition.ts`; every place where the code did not match this+ brief and what you did; the choices this brief left to you (3.1 case 3,+ 3.4 case 8); `git status --short`. Then delete your `TMPDIR` folder.++# Part C: private log reads (3.17) and partial shards (3.15)++Files Part C owns: every file design D1 to D6 names (D2's core readers+and MCP call sites, `src/mcp/brain/reach-events.ts`,+`src/core/brain/log-events-at-reach.ts`, `src/core/brain/temporal/types.ts`,+`src/core/brain/temporal/build-index.ts`, `src/core/store/reader.ts`,+`src/core/store/shard-quarantine.ts` (new),+`src/core/store-migration/journal.ts`,+`src/core/brain/doctor/store-check.ts`,+`src/core/brain/doctor/store-gap-check.ts` (new), `src/cli/store-shard.ts`+(new)), and the test files this part names. Files shared with Part Z,+where Part C adds only the lines named here: `src/cli/store.ts`,+`src/cli/command-manifest.ts`, `docs/cli-reference.md`,+`docs/brain-store.md`. Part C must not touch Part Z's files (Part Z's+list above), in particular `src/core/store/held-ledger.ts`,+`src/core/store/store-open.ts`, `src/core/store/seen-shards.ts`, the+lane modules and `FORMAT.md`.++Your report file is `REPORT-part-c.md` at the checkout root.++## 0. Preconditions (Part C)++- [x] 0.3 Confirm the tree: the commit of 0.1; `wc -l` of the files+ design "Context" lists past 700 lines matches it (record each count in+ the report: these are the counts the file may not grow past beyond+ call-site edits); `grep -n` finds `export function requestRefView` in+ `src/mcp/brain/reach-events.ts`, `function normaliseLogEntry` in+ `src/core/brain/temporal/build-index.ts`, `export function eventAtReach`+ in `src/core/brain/log-events-at-reach.ts`, `res.evidence.map(serializeLogEntry)`+ in `src/mcp/brain/query-tools.ts`, `export function shardReadGap` in+ `src/core/store/reader.ts`, `export function readHeldRecords` in+ `src/core/store-migration/journal.ts`, and+ `if (s.incomplete !== true) unreadable.push(s);` in+ `src/core/brain/doctor/store-check.ts`. Search `src/` for readers of the+ index's `corrupt_lines` column (design D4 item 5): if any code treats it+ as a gap, stop and report. On any mismatch, stop and report.+- [x] 0.4 Take the baseline:+ `bun test tests/mcp/log-private-reach.test.ts tests/mcp/log-readers-reach.test.ts tests/core/store/reader.test.ts tests/core/store/read-gaps.test.ts tests/core/store/state-digest.test.ts tests/core/brain/doctor-store.test.ts tests/cli/store-device.test.ts tests/cli/manifest-completeness.test.ts tests/cli/help-surface-parity.test.ts tests/cli/completions.test.ts tests/core/architecture`+ plus the existing test files of the core readers of design D2 and of+ the timeline builders:+ `tests/core/brain/most-applied.test.ts tests/core/brain/store-readers/reader-parity.test.ts tests/core/brain/owner-scope-delivery.test.ts tests/core/brain/morning-brief.test.ts tests/core/brain/idea-discovery.test.ts tests/core/brain.agent-source.query.test.ts tests/core/brain.backlinks.test.ts tests/core/brain/backlinks-owner-scope.test.ts tests/core/brain/status-maintenance-debt.test.ts tests/core/brain/store-only-operator-readers.test.ts tests/core/brain/memory-cost-meter.test.ts tests/core/brain/token-footprint.test.ts tests/core/brain/maintenance-collect.test.ts tests/core/brain.operator-snapshot.test.ts tests/core/brain.idea-lineage.test.ts tests/core/brain/notes/write-log.test.ts tests/core/brain/notes/write-destinations.test.ts tests/core/brain/notes/revert.test.ts tests/core/brain/link-graph/concept-cluster.test.ts tests/core/brain/link-graph/moc-audit.test.ts tests/core/brain.doctor.test.ts tests/core/brain/digest-actions.test.ts tests/core/brain/trust/operator-summary.test.ts tests/core/brain/temporal tests/mcp/temporal-mcp-tools.test.ts tests/mcp/consolidated-tools.test.ts`+ (this is "the reader's existing tests" every task of sections 6 and 7+ runs: the files of that list whose name matches the reader). Record+ the counts; on a failure, stop and report.++## 6. The two roots (design D1; spec `remote-log-reads`)++- [x] 6.1 Test first: create `tests/mcp/log-private-reach-surfaces.test.ts`+ with the setup of `tests/mcp/log-private-reach.test.ts` (`makeStoreEnv`,+ `switchDevice`, `appendLogEvent`, `withOperationPrivacy(true, ...)`),+ calling tools through `new MCPServer({ vault: env.vault, configPath: env.configPath }, { reach })`+ and `callTool` (see `reachServer` in `tests/helpers/reach-log-fixture.ts`).+ Cases, each with one shared note and one private note that names+ nothing, in the window: `brain_analytics` timeline view, `brain_brief`+ daily, weekly, monthly and today views: the remote answer carries the shared+ text and not the private text, the local one both. Add a unit case for+ `eventAtReach` with a marked event and a view whose `withholdsPrivate`+ is set (`null`) and not set (the event). Run the file: the cases fail.+- [x] 6.2 Implement D1 items 1, 2 and 4 (`requestRefView`,+ `TemporalEvent.fromPrivatePartition`, `normaliseLogEntry`,+ `eventAtReach`, `withoutPrivateEntries`) and item 3+ (`BuildTimelineIndexOptions.withholdsPrivate`, `collectLogEvents`), with+ a unit case for item 3 in the same test file. Run it and+ `tests/mcp/log-private-reach.test.ts tests/mcp/log-readers-reach.test.ts`+ and the existing temporal and analytics tests+ (`tests/core/brain/temporal tests/mcp/temporal-mcp-tools.test.ts tests/mcp/consolidated-tools.test.ts`): all pass.++## 7. The readers with a remote surface (design D2; spec `remote-log-reads`)++For each row: test first in `tests/mcp/log-private-reach-surfaces.test.ts`+(one `describe` per row, a remote and a local assertion, one private+event that names nothing unless the row says otherwise), see it fail,+then the code, then that file and the reader's existing tests pass.++- [x] 7.1 Row 1, `brain_query` preference mode: evidence with one shared+ event, one private event and one shared event naming a page reserved+ against remote reads (as `tests/helpers/reach-log-fixture.ts`+ reserves one); the remote evidence is the first only, and the recall+ telemetry row the call wrote counts 1 (read it the way the existing+ recall-telemetry tests do).+- [x] 7.2 Row 2, most-applied: a shared confirmed preference with one+ shared and two private `applied` evidence events: remote `brain_context`,+ `osb://active` and `osb://digest/latest` count 1, local ones 3.+- [x] 7.3 Row 3, morning brief and idea discovery: a private note and a+ private open reconcile question are absent from remote+ `brain_brief` view=morning and `brain_idea_discovery`, present locally.+- [x] 7.4 Row 4, agent provenance: an agent whose only event is private+ is absent from remote `brain_agent_query` and `brain_agent_diff`+ (contributions, roster and counts), present locally; no answer, local+ or remote, carries a `fromPrivatePartition` field.+- [x] 7.5 Row 5, backlinks: a shared preference referenced only by a+ private event: remote `osb://backlinks/<id>` and `brain_backlinks` list+ no log backlink (count 0), the remote digest's inbound count ignores it;+ local ones list it. On the same fixture: remote `brain_analytics`+ view=concept_synthesis for that preference lists no linker whose+ `sourceKind` starts with `log-` (local lists one); a shared hub note+ (a MOC body over the `auditMoc` thresholds, as+ `tests/core/brain/link-graph/moc-audit.test.ts` builds one) links that+ preference, and remote `brain_moc_audit` of the hub reports its member+ with `backlinkCount` 0, local with 1. A second private event wikilinks+ `pref-missing-target` (no such file): remote `brain_doctor` has no+ `broken-backlinks` warning for it and the remote repair preview+ (`brain_doctor` with `repair: true` and no `apply`) counts no `broken-backlinks` in+ `unfixable`; local ones have the warning and the count. Code:+ `buildConceptCluster`, `auditMoc`, `RunDoctorOptions` and+ `DoctorCheckContext` gain the option; `brokenBacklinkCheck` and+ `collectDoctorIssues` pass it (`applyRepair` gives the latter+ `view.withholdsPrivate === true`); then the call-site edits in+ `analytics-tools.ts`, `knowledge-tools.ts` and `health-tools.ts`+ (design D2 row 5).+- [x] 7.6 Row 6, status: the newest apply-evidence, the newest `dream`+ event (`BRAIN_LOG_EVENT_KIND.dream`, appended with `appendLogEvent`+ inside `withOperationPrivacy(true, ...)` like any other private event)+ and the only event of the newest day are private, with an older shared+ apply-evidence and an older shared dream: remote `second_brain_status`+ and `osb://status` show the shared apply-evidence time, the shared+ dream time (`last_dream_at`; `scanLogTimestamps` reads both from+ `statusLog`) and one log day fewer, local ones the private times; the+ `brain_context` maintenance flag is computed without the private day.+- [x] 7.7 Row 7, cost meter: remote `brain_recall_telemetry` operation+ `cost` counts one write of one shared and one private.+- [x] 7.8 Row 8, token footprint: the log bucket of+ `computeTokenFootprint` with `withholdsPrivate: true` leaves out the+ private event's tokens and with `false` counts them. The `brain_doctor` path+ offers no way to set the threshold: `collectMaintenanceActions` calls+ `computeTokenFootprint(vault)` with no options, so the threshold is+ `TOKEN_WARN_THRESHOLD_DEFAULT` (200000 tokens; the+ `BRAIN_TOKEN_WARN_THRESHOLD` variable is read only by the CLI verb).+ So test `collectMaintenanceActions` directly (in the row's `describe`+ of `tests/mcp/log-private-reach-surfaces.test.ts`, with its store+ setup) at that default: give+ the private event a text of 8000 characters (about 2000 tokens,+ `estimateTokens` is bytes / 4), then fill a bucket the store counts.+ A store-only machine does not count files under `Brain/` (the `other`+ bucket is skipped when `memoryFromStore(vault)` is true), so the filler+ is shared log events: append shared `note` events (no+ `withOperationPrivacy`), each with a text of at most 60000 characters,+ until+ `computeTokenFootprint(vault, { withholdsPrivate: true }).total` is+ between `TOKEN_WARN_THRESHOLD_DEFAULT - 1000` and+ `TOKEN_WARN_THRESHOLD_DEFAULT` (measure the total after each append and+ size the last event from the difference; assert both bounds before+ going on). If the log bucket does not grow with a shared event's size+ either (a cap, or the body spilling into a blob that is not counted),+ use whichever store-counted bucket of `computeTokenFootprint` does grow+ (read the function), and say which in the report. Then `collectMaintenanceActions(vault, { withholdsPrivate: true })`+ has no action with category `token-footprint` and+ `collectMaintenanceActions(vault)` has one. Then+ the call-site edit in `health-tools.ts` passes the option. On the same+ fixture (filler sized the same way), the two other remote surfaces of+ the collector: remote `osb://digest/latest` and `brain_brief`+ view=digest have no `token-footprint` action, local ones have it (the+ edit: `digest.ts` passes `refs.withholdsPrivate === true` to+ `collectMaintenanceActions`); remote `brain_brief` view=operator has no+ `token-footprint` entry in `top_actions`, local has it (the edit:+ `BuildOperatorSummaryOptions.withholdsPrivate`, passed by+ `safeTopActions` to `collectMaintenanceActions` and by `safeDoctor` to+ `runDoctor`, and `brief-tools.ts` passes+ `withholdsPrivate: view.withholdsPrivate === true` outside the+ `view.filtersNothing` spread). Call `brain_brief` view=operator with+ `include_dream: false` so no dry-run dream runs, and `top_actions`+ large enough to hold every action of the fixture.+- [x] 7.9 Row 9, the unfiltered timeline builders: a private+ apply-evidence on a shared preference is absent from remote+ `brain_idea_lineage` of that preference, and does not change the+ remote `brain_stale_scan` or `brain_status` stale counts. The two stale+ counts take different paths (design D2 row 9): `brain_stale_scan`+ (`src/mcp/brain/review-tools.ts`, `toolBrainStaleScan`) builds its own+ index with `buildTimelineIndex(ctx.vault, {})` and `findStaleEntries`,+ and gets the option at that call (build `reviewView(ctx)` before it and+ pass `withholdsPrivate: view.withholdsPrivate === true`);+ `brain_status` (`health-tools.ts`) passes it to `buildOperatorSnapshot`,+ which passes it to its own `buildTimelineIndex` call and to its+ `runDoctor` call (design D2 rows 5 and 9; the doctor-warnings count of+ remote `brain_status` ignores a `broken-backlinks` finding raised only+ by a private event, as in task 7.5).+- [x] 7.10 Row 10, note writes: a shared note written inside a private+ operation is absent from remote `brain_writes` and+ `brain_write_destinations` (lists and counts), present locally. Remote+ `brain_writes` action `plan_revert` with a selector on that note's path+ lists no write from the private operation and its digest differs from+ the local plan's digest; the local plan lists it (the edit:+ `PlanNoteRevertOptions.withholdsPrivate`, passed to both+ `listNoteWrites` reads, and `query-tools.ts` takes the `reachView`+ before `planNoteRevert`).++## 8. The census (design D3; spec `remote-log-reads`)++- [x] 8.1 Create `tests/core/architecture/log-reader-census.test.ts` as+ design D3 describes, with the table of D3 (one row per module). Add a+ case that proves the census fails on an unlisted module (feed the+ checker a fake module list, as `write-site-census.test.ts` proves its+ own rule). Run it: it passes; if the walk finds a module the D3 table+ does not list, stop and report it (do not classify it yourself unless+ it is plainly CLI-only, and then say so in the report).++## 9. The quarantine ledger and the reader (design D4; spec `store-shard-gaps`)++- [x] 9.1 Test first, in a new `tests/core/store/shard-quarantine.test.ts`+ (store set up as in `tests/core/store/read-gaps.test.ts`): a corrupt+ middle line makes `shardReadGap` non-null; after `addQuarantine` for its+ partition, shard, offset and line hash, the shard reads whole+ (`shardReadGap` null) and the bytes on disk are unchanged; a different+ line written at the same offset (rewrite the shard) is a gap again; a+ second corrupt line at another offset in the same shard is still a gap+ while the first stays quarantined (the mark needs the offset and the+ line hash both, for a diagnostic and a pending record alike, design D4+ item 3); a pending record whose offset is in the ledger with another+ line hash still counts; an `unknown-version` line is never marked; a pending blob entry stops+ counting, its record is still left out of `readStreamRecords`, and once+ the blob file is written the record reads with its body; two+ concurrent `addQuarantine` calls from two processes both land (spawn as+ `tests/core/store/held-ledger.test.ts` does); `releaseQuarantine`+ brings the gap back; `listQuarantine` reports `held`, `resolved` and+ `stale`. A capture case: a corrupt middle line in a `shared` settings+ shard closes `buildCaptureBoundary` (`src/core/brain/capture-boundary.ts`),+ and the quarantine opens it again. A migration case: a quarantined+ pending blob still puts its shard into `readHeldRecords(...).unseen.partial`.+ Run: the cases fail.+- [x] 9.2 Implement `src/core/store/shard-quarantine.ts` (D4 items 1 and+ 2), the marks in `readShard` and the skip in `shardReadGap` (item 3),+ and the pending rule in `readHeldRecords` (item 4). Run the new file+ and `tests/core/store/reader.test.ts tests/core/store/read-gaps.test.ts tests/core/store/state-digest.test.ts tests/core/store/seen-shards.test.ts`+ and the store-migration tests (`tests/core/store-migration`): all pass.++## 10. Doctor findings (design D5; spec `store-shard-gaps`)++- [x] 10.1 Test first, in `tests/core/brain/doctor-store.test.ts`: one+ case per finding (`store-shard-corrupt-line` error with the offset and+ the quarantine command; `store-shard-newer-format` warning with no+ command; `store-shard-blob-pending` warning when young, error when+ older than seven days, no command for a `settings` shard; a torn line+ and a quarantined line give no finding). Run: they fail.+- [x] 10.2 Implement `src/core/brain/doctor/store-gap-check.ts` and the+ call-site edits in `store-check.ts` (D5). Update `store-check.ts`'s+ header list with one item for the new findings. Run the doctor tests:+ all pass.++## 11. The verb and the docs (design D6, D9; spec `store-shard-gaps`)++- [x] 11.1 Test first: create `tests/cli/store-shard.test.ts` with the+ `env()` of `tests/cli/store-partition.test.ts` (task 2.1; write the+ same object), `runCli(..., { env: env(), stdin: "" })`: quarantine,+ `already`, list, release, `not-quarantined`, each refusal reason of D6+ under `--json`, and usage errors (exit 2). Run: it fails.+- [x] 11.2 Create `src/cli/store-shard.ts` (D6). In `src/cli/store.ts`+ add only: the import of `cmdStoreShard` and `STORE_SHARD_USAGE`, the+ usage joined at the end of `USAGE`, and `case "shard":` right after+ `case "device":`'s return line, plus one header comment sentence. In+ `src/cli/command-manifest.ts` add only a `command("shard", ...)` entry+ right after the `device` entry of `store`, with `quarantine`, `list`+ and `release` and their flags. Run the new test and+ `tests/cli/manifest-completeness.test.ts tests/cli/help-surface-parity.test.ts tests/cli/completions.test.ts tests/cli/store-device.test.ts`:+ all pass.+- [x] 11.3 Docs: one line in `docs/cli-reference.md` right after the line+ that starts `o2b store device list|retire` (the verbs, refusals, exit+ codes, `--json`), and in `docs/brain-store.md` a section+ `## Shards read only in part` right before+ `## What the partitions protect, and what they do not`: the three+ doctor findings, what fails closed, the quarantine on one machine, why+ never a newer-format line or a young or settings blob, that rollback+ and the late import still refuse on a pending blob, list and release.+ Run `bun test tests/docs` before and after: no new failure.++## 12. Closing checks (Part C)++- [x] 12.1 Run each, one at a time, and fix what they report in your own+ files: `bunx tsc --noEmit -p .`, `bun run lint`, `bun run fmt:check`+ (`bun run fmt` to fix), `bun run check:paths`.+- [x] 12.2 Run every test file Part C touched or created, and the+ baseline of 0.4, once more in one command; all pass. `wc -l`: every+ file Part C created or changed that was under 700 lines stays under+ 700; every file past 700 grew only by its call-site edits (list the+ growth per file in the report). `git status --short` lists only Part+ C's files (and `.apb/.gitignore`, which is not yours).++## 13. Report (Part C)++- [x] 13.1 Write `REPORT-part-c.md` at the checkout root (never commit+ it; at most 80 lines): every Part C box ticked or why not; the baseline+ and final counts; the line growth of each file past 700; every+ deviation from design.md and why; every module the census found that+ D3 did not list; anything in this brief that did not match the code;+ `git status --short`. Then delete your `TMPDIR` folder.++# Part I: integration (the integrator)++- [x] 14.1 Merge Part Z's and Part C's working trees into one checkout of+ this branch, and read both reports first. Resolve the overlaps of design+ D9 by keeping both parts' lines: `src/cli/store.ts` (both imports, both+ usages, `case "partition"` then `case "shard"` after `case "device"`),+ `src/cli/command-manifest.ts` (`partition` then `shard` after+ `device`), `docs/cli-reference.md` (the `partition` line then the+ `shard` line), `docs/brain-store.md` (each part's own section). Delete+ `REPORT-part-z.md` and `REPORT-part-c.md`. Verify:+ `git status --short` shows no report file and no conflict marker+ (`git diff | grep -c '^[<>=]\{7\}'` prints 0).+- [x] 14.2 Rebuild the OpenClaw bundle once:+ `bunx [email protected] run build:openclaw`; keep `openclaw/index.js`. Verify:+ `git diff --stat openclaw/index.js` shows it changed (it carries+ `reader.ts`, `held-ledger.ts` and `store-open.ts`).+- [x] 14.3 `CHANGELOG.md`: one entry naming the four behaviours of the+ proposal under a `## [Unreleased]` heading placed above the newest+ release heading (Keep a Changelog; create it when absent). No version+ heading and no version bump: this pull request ships no release.+ Verify: `bun run scripts/sync-version.ts --check` passes, `package.json`+ is unchanged, and `bun test tests/docs tests/scripts` passes; if a test+ rejects the `Unreleased` heading, remove the entry and record that the+ store work ships its notes with the phase release (task 3.7 of+ `brain-store-and-topics`), as the three store pull requests before it+ did.+- [x] 14.4 Run the full local gate of `openspec/project.md`, each command+ on its own; all must pass:+ `bun run test`+ `bunx tsc --noEmit -p .`+ `bun run lint`+ `bun run fmt:check`+ `bun run check:paths`+ `bun run sync-plugin-mirrors:check`+ `bun run scripts/sync-version.ts --check`+- [x] 14.5 In `openspec/changes/brain-store-and-topics/tasks.md`, append+ "Done in change `store-hardening`." to tasks 3.8, 3.15, 3.16 and 3.17+ and tick them. Tasks 3.18 (dream and evidence over private+ apply-evidence) and 3.19 (split the files past 700 lines) are already+ in that file, added with this change: confirm both are present and+ open (`grep -n "^- \[ \] 3.1[89] " openspec/changes/brain-store-and-topics/tasks.md`+ prints two lines) and leave them open; if either is missing, stop and+ report. Verify: `openspec validate store-hardening --strict` passes.++## Workflow follow-up++- Independent code review rounds of the merged result, fixes re-reviewed.+- Archive the change (`openspec archive store-hardening --yes`) inside its+ pull request, after the whole cycle is green.diff --git a/openspec/changes/brain-store-and-topics/tasks.md b/openspec/changes/brain-store-and-topics/tasks.mdindex d79612a7..c6987f3d 100644--- a/openspec/changes/brain-store-and-topics/tasks.md+++ b/openspec/changes/brain-store-and-topics/tasks.md@@ -82,16 +82,21 @@ Phases follow design.md D13. Each phase ends in its own release PR, and the vers- a concurrent dream on two machines, and two dream processes on the view writer, each yield exactly one topic per cluster.- [ ] 3.5 Add the `brain_topic` MCP tool and the `o2b topic` CLI verbs. Rename the preference-topic internals to `ruleTopic` and the resource to `osb://rule-topic/{slug}`, with an alias for one minor version, and add `osb://topic/{id}`. Verify: MCP contract tests, an alias test and a CHANGELOG note.- [ ] 3.6 Write the topics guide in docs: what a topic is, the page schema, how to ask an agent, and undo. Verify: a doc test runs the listed CLI verbs.-- [ ] 3.8 Keep a lane page's raw text in its `lane-entry` record (an optional `raw` body field, FORMAT.md), so a store-only machine reads back staged bytes verbatim (OCR F291). Verify: a round-trip test of text without frontmatter and of quoted values.+- [x] 3.8 Keep a lane page's raw text in its `lane-entry` record (an optional `raw` body field, FORMAT.md), so a store-only machine reads back staged bytes verbatim (OCR F291). Verify: a round-trip test of text without frontmatter and of quoted values. Done in change `store-hardening`.- [ ] 3.9 Make lane entry ids a bijection with legacy paths (every extension, `x.json` vs `x.json.md`, folders that are not lane names), or refuse the names that do not round-trip (OCR F295). Verify: a round-trip test over the lane path table.- [ ] 3.10 Keep the procedural graph and hint counters as device-local state instead of re-appending whole `lane-entry` records on every usage tick (OCR F360). Verify: a usage tick appends no store record, and a migration test moves the counters into the device-local state.- [ ] 3.11 Write the schema-apply audit row in the partition of the pack records it attests (`shared`), with one privacy verdict over the batch and its reason, or have the schema integrity check accept the newest settings records as attested (OCR F394). Verify: a machine that holds only `shared` reports a schema apply from another device as attested, not `modified`.- [ ] 3.12 Settle the cross-device merge of the schema field lists (a set merge rather than last writer wins on the whole list), personas as files or records, and whether the shared namespace gets a store of its own; the mirror into this machine's own store refuses until then (OCR F397, F464, F398). Verify: a concurrent edit test on two devices for the field lists, and a test per decision taken.- [x] 3.13 Keep private-partition log events out of the OKF `log.md`, or carry a privacy marker for them: a log entry has no frontmatter for the `visibility: [private]` stamp pages from the private partition now get (OCR F333, log side). Verify: an export test on a switched machine with one private and one shared log event. Done in change `lane-pages-split`.- [x] 3.14 Split the store side of `src/core/store/lane-pages.ts` (fold accessors, partition keep-rule, lane-entry writers) into its own module: the file is over the 700-line limit after the replace-keeps-private fix. Verify: no behaviour change, the lanes and store suites pass unchanged. Done in change `lane-pages-split`.-- [ ] 3.15 Name a shard that reads only in part (a corrupt middle line, a record of a newer format, a blob that never arrives) in a doctor finding with its path, and add a repair that quarantines the bad line, so a protection read that fails closed on it can be released without a hand edit. Also decide whether rollback and the late import wait for a pending blob rather than leave its record out. Verify: a doctor test per gap kind, a quarantine test that reopens capture.-- [ ] 3.16 Add an operator verb that takes a partition off this machine on purpose (for example `o2b store partition leave private`): held partitions are sticky per machine since the phase 2 review, so a machine that stops syncing `private/` refuses every store operation until the verb clears it from the held list. Verify: leaving `private` turns the machine shared-only with no refusal, and a lost `private/store.json` without the verb still fails closed.-- [ ] 3.17 Audit every reader that renders store log events to a caller below local reach without `logEntryAtReach` or `logEntriesAtReach` (the active-preferences digest's most-applied counts, the morning brief, temporal citations, status, backlinks, analytics, anything else built on `logDayReader`, `storeLogDays` or `readStoreLogEvents`), and make each withhold events marked `fromPrivatePartition` (change `lane-pages-split`, design D3). Verify: a remote-reach test per reader with one private event that names nothing.+- [x] 3.15 Name a shard that reads only in part (a corrupt middle line, a record of a newer format, a blob that never arrives) in a doctor finding with its path, and add a repair that quarantines the bad line, so a protection read that fails closed on it can be released without a hand edit. Also decide whether rollback and the late import wait for a pending blob rather than leave its record out. Verify: a doctor test per gap kind, a quarantine test that reopens capture. Done in change `store-hardening`.+- [x] 3.16 Add an operator verb that takes a partition off this machine on purpose (for example `o2b store partition leave private`): held partitions are sticky per machine since the phase 2 review, so a machine that stops syncing `private/` refuses every store operation until the verb clears it from the held list. Verify: leaving `private` turns the machine shared-only with no refusal, and a lost `private/store.json` without the verb still fails closed. Done in change `store-hardening`.+- [x] 3.17 Audit every reader that renders store log events to a caller below local reach without `logEntryAtReach` or `logEntriesAtReach` (the active-preferences digest's most-applied counts, the morning brief, temporal citations, status, backlinks, analytics, anything else built on `logDayReader`, `storeLogDays` or `readStoreLogEvents`), and make each withhold events marked `fromPrivatePartition` (change `lane-pages-split`, design D3). Verify: a remote-reach test per reader with one private event that names nothing. Done in change `store-hardening`.+- [ ] 3.18 Decide and apply how private `apply-evidence` events take part in the dream and evidence computations (`evidence.ts`, `dream-refresh.ts`, the review candidates of the operator snapshot): today a private event moves a shared preference's confidence and status, which every machine and remote caller then sees (out of scope of change `store-hardening`). Verify: a dream test on a switched machine with private evidence on a shared preference, asserting the decided outcome.+- [ ] 3.19 Split the source files past the 700-line limit that the store readers touch (`src/core/brain/digest.ts`, `src/mcp/brain/query-tools.ts`, `src/mcp/resources.ts`, `src/mcp/brain/brief-tools.ts`, and the other MCP tool files past it), with no behaviour change. Verify: `wc -l` shows each under 700, and the suites of those modules pass unchanged.+- [ ] 3.20 Keep the name of a lost private partition out of fail-closed reader errors served below local reach: `src/core/store/read/protection-store.ts` copies the store-open error, which names `private/store.json`, into findings such as `permissions-unreadable` that a remote caller receives (found in the second repair round of change `store-hardening`; the store check's own finding is already redacted). Verify: a remote-reach doctor test with a lost `private/store.json` whose whole answer does not contain `private`.+- [ ] 3.21 Serialize the seen-shards ledger's read-modify-write (`update` in `src/core/store/seen-shards.ts`) under the device-local lock: two processes on one machine can interleave, so a stale write from `trackSeenShards` can bring back entries `o2b store partition leave` just cleared, or a leave can drop a shard another process just recorded (pull-request review of change `store-hardening`). Verify: a two-process test in which a tracked shard and a partition removal both survive.+- [ ] 3.22 Cache a read that skipped a quarantined pending blob: the entity read and the lane fold do not cache such a read (change `store-hardening`), so a blob that never arrives makes them rescan their shards on every call. Put the availability of the pending blobs into the cache fingerprint instead. Verify: the read is cached while the blob is absent, and the record appears on the first read after the blob arrives.- [ ] 3.7 Integration. Run the full gate on Linux and Windows. Then the owner checks topics by hand on two machines and in a phone view, including a phone edit. Then open the release PR.## 4. Phase 4: decision-model uses and agent contextdiff --git a/openspec/specs/held-partitions/spec.md b/openspec/specs/held-partitions/spec.mdnew file mode 100644index 00000000..88c4bd76--- /dev/null+++ b/openspec/specs/held-partitions/spec.md@@ -0,0 +1,166 @@+# held-partitions Specification++## Purpose+Which partitions of the brain store a machine holds, how a partition it+held and lost makes the store fail closed, and how the operator takes a+partition off a machine on purpose.++## Requirements++### Requirement: A lost held partition fails closed and names the remedies++A machine that held a partition whose `store.json` is now missing SHALL+refuse to open the store, and the refusal and the doctor finding+`store-partition-lost` SHALL name the remedies: restore the partition's+`store.json` from sync or a backup, or, for a partition other than+`shared`, run `o2b store partition leave <partition>`. For a lost+`shared` partition the refusal SHALL name only the restore, because+leaving `shared` is refused.++#### Scenario: private/store.json lost without the verb++- **GIVEN** a machine that held both partitions+- **WHEN** `private/store.json` disappears and the operator runs any store+ command+- **THEN** the command refuses and names+ `o2b store partition leave private`++#### Scenario: shared/store.json lost++- **GIVEN** a machine that held both partitions+- **WHEN** `shared/store.json` disappears and the operator runs any store+ command+- **THEN** the command refuses and names the restore of+ `shared/store.json`, but names no `o2b store partition leave` command++#### Scenario: shared/store.json and private/store.json both lost++- **GIVEN** a machine that held both partitions+- **WHEN** `shared/store.json` and `private/store.json` both disappear and+ the operator runs any store command+- **THEN** the command refuses and names the restores of+ `shared/store.json` and `private/store.json`, but names no+ `o2b store partition leave` command: with `shared/store.json` gone, a+ leave cannot read the store id and would refuse without changing+ anything++### Requirement: Leaving a partition makes the machine hold the rest++`o2b store partition leave <partition>` SHALL take the partition off this+machine's held list once its folder carries no `store.json`; from then on+the store SHALL open with the remaining partitions, with no refusal.++#### Scenario: Leaving private++- **GIVEN** a machine that held both partitions, and the `private` folder+ removed+- **WHEN** the operator runs `o2b store partition leave private`+- **THEN** the verb exits 0+- **AND** the next store command treats the machine as shared-only++### Requirement: Leaving refuses what it cannot do safely++The verb SHALL refuse with exit code 1 and a named reason: `shared`+(leaving it is leaving the store); `still-held` while the partition's+`store.json` is present; `files-left` while the partition's folder still+holds anything, unless `--force` is given, and always when the folder+cannot be listed, so what is left cannot be checked. A partition name+that is not a store partition SHALL be a usage error (exit 2).++#### Scenario: Leaving shared++- **WHEN** the operator runs `o2b store partition leave shared`+- **THEN** the verb exits 1 with reason `shared`++#### Scenario: Files left in the folder++- **GIVEN** a `private` folder with shards and no `store.json`+- **WHEN** the operator runs the verb without `--force`+- **THEN** it exits 1 with reason `files-left` and the machine still+ refuses to open the store+- **AND** with `--force` the verb leaves the partition++#### Scenario: The folder cannot be listed++- **GIVEN** the place of the `private` folder taken by a regular file, so+ the folder cannot be listed+- **WHEN** the operator runs the verb+- **THEN** it exits 1 with reason `files-left`, with or without `--force`++### Requirement: What each refusal changes++On a `shared`, `still-held`, `files-left` or `no-store-id` refusal the+verb SHALL change nothing. On a `clear-failed` refusal it MAY already+have cleared device-local caches or seen-shard entries, but SHALL keep+the held marker, so the machine still fails closed.++#### Scenario: A failed clearing keeps the marker++- **GIVEN** a machine that holds `private` and whose device-local state+ of it cannot be cleared+- **WHEN** the operator runs the verb+- **THEN** it exits 1 with reason `clear-failed`, and the machine still+ holds `private` (the store still refuses to open)++### Requirement: Leaving fails closed without the shared store id++The verb SHALL read the store id from the shared partition's+`store.json` before it removes anything, and SHALL refuse with exit code+1 and reason `no-store-id`, changing nothing, when that file is missing+or unreadable: without the store id it cannot find the partition's+device-local state, so it SHALL NOT report a leave it could not clear.+It SHALL remove the held marker last, so a failed clearing (reason+`clear-failed`) leaves the machine still holding the partition.++#### Scenario: The shared identity file cannot be read++- **GIVEN** a machine that held both partitions, the `private` folder+ removed, and `shared/store.json` missing or not valid JSON+- **WHEN** the operator runs `o2b store partition leave private`+- **THEN** it exits 1 with reason `no-store-id` and a message that says to+ restore the shared partition's identity file first+- **AND** the machine still holds `private` (the store still refuses to+ open), and no seen-shard entry or digest cache is removed++### Requirement: Leaving is idempotent++The `still-held` check (whether the partition's `store.json` is present)+comes before the not-held answer. When that file is absent and this+machine does not hold the partition, the verb SHALL exit 0, say there is+nothing to leave and change nothing; under `--json` it SHALL answer+`{ "ok": true, "left": false }`. So a folder that was synced in without+a writing open recording it is refused as `still-held`, not reported as+nothing to leave.++#### Scenario: Leaving twice++- **WHEN** the operator runs the verb a second time+- **THEN** it exits 0 and reports `left: false`++### Requirement: Leaving clears what this machine kept of the partition++Leaving SHALL clear this machine's device-local state of the partition:+its record of shards seen and the cached digests that may hold its text.+The search index SHALL hold no record of the partition after its next+catch-up.++#### Scenario: The index forgets the partition++- **GIVEN** a private record that the search index returns+- **WHEN** the operator leaves `private` and the index catches up+- **THEN** the index no longer returns the record++### Requirement: A partition that comes back is held again++A left partition whose `store.json` later arrives again (by sync or a+restore) SHALL be held again from the next writing open, under the same+rule as any partition; the verb's output and the documentation SHALL say+so.++#### Scenario: private returns++- **GIVEN** a machine that left `private`+- **WHEN** `private/store.json` and its shards arrive again and a store+ write runs+- **THEN** the machine holds `private` again, and losing its `store.json`+ later fails closed againdiff --git a/openspec/specs/lane-entry-fidelity/spec.md b/openspec/specs/lane-entry-fidelity/spec.mdnew file mode 100644index 00000000..f948b634--- /dev/null+++ b/openspec/specs/lane-entry-fidelity/spec.md@@ -0,0 +1,126 @@+# lane-entry-fidelity Specification++## Purpose+What a lane file written whole on a store-only machine reads back as, so+staged bytes survive the trip through a `lane-entry` record.++## Requirements++### Requirement: A lane file written whole reads back byte for byte++On a store-only machine a lane file written from its whole text SHALL read+back as exactly that text: a page without frontmatter, a quoted value that+would parse as a number or a boolean, CRLF line ends, a missing final+newline and the formatting of a JSON file are all kept.++#### Scenario: A page without frontmatter++- **WHEN** a lane page is written whole as `Just text.\n`+- **THEN** it reads back as `Just text.\n`, with no frontmatter added++#### Scenario: A quoted value++- **WHEN** a page is written whole with the frontmatter line `id: '123'`+- **THEN** it reads back with `id: '123'`, and its parsed `id` is the+ string `123`++#### Scenario: JSON formatting++- **WHEN** a JSON lane file is written whole as `{"a":1}\n`+- **THEN** it reads back as `{"a":1}\n`++### Requirement: The raw text is stored only when the parts lose it++A `lane-entry` record SHALL carry the file's text only when its fields and+text do not reproduce it, and only when that text parses back to them; a+file the parts reproduce SHALL be stored as before.++#### Scenario: A canonical page++- **WHEN** a page is written whole in the form the frontmatter writer+ itself produces+- **THEN** its record carries no raw text++### Requirement: The raw text counts for the entry's privacy++A reference to a private page, or a private region, that appears only in+the raw text (for example in a frontmatter comment) SHALL send the entry+to the `private` partition, as the same reference in its fields or text+does.++#### Scenario: A private reference in a frontmatter comment++- **WHEN** a page is written whole whose frontmatter comment names a page+ reserved as private+- **THEN** its record lands in the `private` partition++### Requirement: The writer's judge is the only privacy judge of the raw text++The writer's privacy judge SHALL be the only judge of the raw text,+exactly as it is the only judge of the text: the fold and the readers+SHALL NOT re-judge either. The read-side check that the raw text parses+back is not a privacy judge; it only stops the raw text from saying+something other than the record's own fields and text.++#### Scenario: A hand-appended shared record with a raw comment++- **GIVEN** a `lane-entry` record appended by hand to the `shared`+ partition, bypassing the writer's judge, whose raw text carries a+ frontmatter comment that names a private page and otherwise parses back+ to the record's fields and text+- **WHEN** the lane file is read+- **THEN** the reader serves the raw text, comment included, as written+ (the same trust a hand-appended `text` already gets)++### Requirement: A raw text that does not match its record is not served++A reader SHALL serve a record's raw text only when it parses back to the+record's own fields and text; otherwise it SHALL render the fields and+text as before, so a peer cannot serve content that differs from the+record's parts.++#### Scenario: A mismatched raw text++- **GIVEN** a synced record whose raw text parses to other fields than+ the record carries+- **WHEN** the lane file is read+- **THEN** the reader returns the rendering of the record's fields and+ text++### Requirement: Structured writes and moves++A structured write (fields and body) SHALL store no raw text and replace+an entry that had one; a move SHALL keep the entry's raw text; a whole-text+write that changes only bytes the parts do not carry SHALL be written, not+skipped as unchanged.++#### Scenario: A quoting-only change++- **GIVEN** an entry written whole with `id: '123'`+- **WHEN** it is written whole again with `id: "123"`+- **THEN** a new record is written and the file reads back with the+ double quotes++#### Scenario: A move++- **WHEN** an entry with raw text is moved+- **THEN** the moved file reads back byte for byte++### Requirement: The raw field is optional and never rejects a record++The raw text SHALL be an optional field of the existing record kind, with+no format version change and no change to the fold's rejection rules: a+record without it SHALL fold exactly as before, and a record whose raw+field is not a string SHALL fold without rejection, with no raw text in+its value.++#### Scenario: A record without raw text++- **WHEN** a `lane-entry` record without the field is folded+- **THEN** the entry's value has its fields and text and no raw text++#### Scenario: A raw field that is not a string++- **WHEN** a `lane-entry` record whose raw field is the number `5` is+ folded+- **THEN** the record is accepted and the entry's value has no raw textdiff --git a/openspec/specs/remote-log-reads/spec.md b/openspec/specs/remote-log-reads/spec.mdindex c6f4ecff..b7a5ed45 100644--- a/openspec/specs/remote-log-reads/spec.md+++ b/openspec/specs/remote-log-reads/spec.md@@ -27,3 +27,143 @@ day that was never logged gives.- **GIVEN** a switched machine whose only log event of a day is private- **WHEN** a remote caller reads `osb://log/<that day>`- **THEN** the read fails exactly as for a day with no log events++### Requirement: Every reader with a remote surface withholds private log events++Every MCP tool and resource that renders log events, or text derived from+them, to a caller below local reach SHALL withhold the events the store+holds in its `private` partition, whatever they name: the timeline+analytics, the daily, weekly, monthly, today and morning briefs, idea+discovery, idea lineage, agent provenance, backlinks and the note write+logs. A local caller SHALL see them as before.++#### Scenario: A private note in the remote timeline and briefs++- **GIVEN** a switched machine with one shared note event and one private+ note event that names nothing, both inside the window+- **WHEN** a remote caller reads `brain_analytics` (timeline) and+ `brain_brief` (daily, weekly, morning)+- **THEN** each answer carries the shared note's text and not the private+ note's text+- **AND** a local caller's answers carry both++#### Scenario: Agent provenance below local reach++- **GIVEN** a switched machine where agent `secret-agent` wrote only one+ private log event+- **WHEN** a remote caller runs `brain_agent_query` or `brain_agent_diff`+- **THEN** neither the contribution nor the agent appears, and the counts+ read as if the event did not exist++#### Scenario: Backlinks from a private event++- **GIVEN** a switched machine where the only reference to a shared+ preference is a private log event+- **WHEN** a remote caller reads `osb://backlinks/<that preference>` or+ `brain_backlinks`+- **THEN** the answer lists no backlink from the log and its count is 0++#### Scenario: Link-derived answers from a private event++- **GIVEN** the same machine, a shared MOC note that links that+ preference, and a second private log event that links an artifact id+ with no file+- **WHEN** a remote caller reads `brain_analytics` (concept synthesis) of+ the preference, `brain_moc_audit` of the MOC note, `brain_doctor`, or+ the `brain_doctor` repair preview+- **THEN** no answer lists or counts a reference from either private+ event, and no broken-backlink finding names the missing id+- **AND** a local caller's answers list and count both++#### Scenario: Doctor evidence lints from a private event++- **GIVEN** a switched machine with a private `apply-evidence` event whose+ artifact names no file and carries a range that does not parse+- **WHEN** a remote caller reads `brain_doctor`+- **THEN** no `orphan-evidence` or `malformed-evidence-range` finding+ quotes the event's time or artifact+- **AND** a local caller's doctor reports both findings for it++#### Scenario: A note write in a private operation++- **GIVEN** a switched machine where a shared note was written inside a+ private operation, so its note-write event is private+- **WHEN** a remote caller runs `brain_writes` or `brain_write_destinations`+- **THEN** the write is not listed and not counted+- **AND** the revert plan `brain_writes` returns for that note's path+ lists no write from the private operation++### Requirement: Counts and timestamps below local reach ignore private log events++These numbers and timestamps, derived from log events and shown to a+caller below local reach, SHALL be computed as if the withheld events did+not exist (a preference's stored confidence and status are not among+them): most-applied counts and order, the last dream and last apply-evidence+times, the count of log days, the maintenance overdue flag, the memory+cost meter, the token footprint of the log, and the stale counts. Every+private event, counted or rendered, stays visible to a local caller.++#### Scenario: Most-applied counts++- **GIVEN** a switched machine with a shared confirmed preference that has+ one shared and two private `applied` evidence events in the window+- **WHEN** a remote caller reads `brain_context`, `osb://active` or+ `osb://digest/latest`+- **THEN** the preference's applied count reads 1+- **AND** a local caller reads 3++#### Scenario: Status timestamps++- **GIVEN** a switched machine whose newest apply-evidence event and+ newest dream event are private+- **WHEN** a remote caller reads `second_brain_status` or `osb://status`+- **THEN** the last apply-evidence time and the last dream time are the+ newest shared ones, or absent when there is none++#### Scenario: Maintenance actions from the log footprint++- **GIVEN** a switched machine whose token footprint crosses the warning+ threshold only with a private event's text counted+- **WHEN** a remote caller reads `brain_doctor`, `osb://digest/latest`,+ or `brain_brief` (digest, operator)+- **THEN** no answer carries the token-footprint maintenance action+- **AND** a local caller's answers carry it++#### Scenario: Cost meter++- **GIVEN** a switched machine with one shared and one private memory+ write in the window+- **WHEN** a remote caller asks `brain_recall_telemetry` for the cost+ meter+- **THEN** the meter counts one write++### Requirement: Preference mode of brain_query applies the reference rule++`brain_query` in preference mode SHALL return, to a caller below local+reach, only the evidence events its sibling modes would return: no+private event, and no event naming a record the caller cannot read. The+result count it records in recall telemetry SHALL be the count it+returned.++#### Scenario: Private and unreadable evidence++- **GIVEN** a switched machine with a shared preference whose evidence is+ one shared event, one private event, and one shared event that names a+ page reserved against remote reads+- **WHEN** a remote caller runs `brain_query` with that preference+- **THEN** the evidence holds only the first event+- **AND** the recall telemetry row of the call counts one result++### Requirement: A new log reader is classified before it ships++Every module that reads store log events SHALL be classified as one that+filters for a remote caller, one that serves only local callers, or a+computation that must see every event, with its reason; a module that+reads log events and carries no classification SHALL fail the build's+test suite, naming the module.++#### Scenario: An unclassified reader++- **WHEN** a module starts calling a store log reader and no+ classification names it+- **THEN** the test suite fails and names the modulediff --git a/openspec/specs/store-shard-gaps/spec.md b/openspec/specs/store-shard-gaps/spec.mdnew file mode 100644index 00000000..5afbd3b0--- /dev/null+++ b/openspec/specs/store-shard-gaps/spec.md@@ -0,0 +1,456 @@+# store-shard-gaps Specification++## Purpose+How the brain store reports a shard that a machine reads only in part,+and how an operator releases one bad line on one machine without editing+a shard another device owns.++## Requirements++### Requirement: Doctor names each shard read only in part++`brain_doctor` SHALL report every shard of a held partition that reads+only in part, one finding per gap, naming the shard as+`<partition>/<shard path>`, the byte offset of the line, what fails closed+on it, and the command that repairs it. A line cut short by a crash and+closed by the next append is not a gap and SHALL NOT be reported.++#### Scenario: A corrupt line in the middle++- **GIVEN** a shard whose middle line is not a valid record+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-corrupt-line` naming the shard,+ the offset and `o2b store shard quarantine <partition>/<shard> --offset <n>`++#### Scenario: A record of a newer format++- **GIVEN** a shard with a record whose format version this release does+ not know+- **WHEN** the operator runs the doctor+- **THEN** it reports a warning `store-shard-newer-format` naming the shard+ and the offset and saying to upgrade Open Second Brain, with no+ quarantine command++#### Scenario: A blob that has not arrived++- **GIVEN** a shard with a record whose body blob is absent+- **WHEN** the operator runs the doctor+- **THEN** it reports `store-shard-blob-pending`, a warning while the+ record is younger than seven days and an error from then on, naming the+ quarantine command once it is allowed++#### Scenario: A pending record whose clock stamp does not parse++- **GIVEN** a shard with a record whose body blob is absent and whose+ clock stamp does not parse as a time+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-blob-pending` saying the clock+ stamp does not parse, with no quarantine command and no write time++### Requirement: A corrupt line too long to quarantine names no quarantine command++The doctor SHALL name the quarantine command for a corrupt line only when+the verb can hash that line; a corrupt line past the quarantine hash cap,+which the verb always refuses as `line-too-long`, SHALL be reported as an+error `store-shard-line-too-long` whose message names repair at the+line's source and whose next step names no command.++#### Scenario: A corrupt line too long to quarantine++- **GIVEN** a shard whose middle line is not a valid record and runs past+ the quarantine hash cap+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-line-too-long` naming the+ shard, the offset and repair at the line's source+- **AND** neither the finding nor its next step names a quarantine command++### Requirement: A gap no quarantine can release names no quarantine command++A gap no quarantine can release (the shard was swapped for a link, or cut+below the offset, since the partition read) and a shard read only in part+that cannot be read again for any reason other than its removal SHALL be+reported as an error `store-shard-unreadable` naming the shard, whose next+step names no command; the second case SHALL name the error code. A shard+removed since the partition read SHALL give no finding.++#### Scenario: A shard swapped for a link++- **GIVEN** a shard read only in part that is replaced by a symbolic link+ before the doctor reads it again+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-unreadable` naming the shard+- **AND** neither the finding nor its next step names a quarantine command++#### Scenario: A shard that cannot be read again++- **GIVEN** a shard read only in part that the doctor cannot open again+ (a permission, a lock)+- **WHEN** the operator runs the doctor+- **THEN** it reports an error `store-shard-unreadable` naming the shard+ and the error code++### Requirement: The doctor withholds private store findings below local reach++Below local reach (a caller whose reach is not local, or one marked as+withheld from private data), every surface that runs the doctor+(`brain_doctor`, `brain_brief view=operator`, `brain_status` and the+repair pass) SHALL compute the store findings as if this machine did not+hold the `private` partition, and that partition SHALL NOT change the+caller's `ok`, counts or `trust_verdict` while the store opens. Local+reach and the CLI doctor SHALL report everything as before.++#### Scenario: A gap in a private shard below local reach++- **GIVEN** a shard of the `private` partition with a corrupt middle line+- **WHEN** a remote caller runs `brain_doctor`, `brain_brief view=operator`+ or `brain_status`+- **THEN** no finding names the shard, its offset or a record's time+- **AND** `ok`, the error and warning counts and `trust_verdict` are what+ they would be without the gap+- **AND** a local caller's doctor reports the gap++### Requirement: Every store finding about the private partition is withheld below local reach++The findings withheld below local reach SHALL be every store finding that+names the `private` partition or a shard, offset, record id, record time+or device clock of it: a gap, a held tail, an unreadable shard, a record+id conflict, a clock ahead, a digest divergence or silence, a record id+found in both partitions, a loose permission and a reserved device+folder; the removal backlog SHALL count only `shared`. Failed writes+follow the next requirement.++#### Scenario: A private record whose clock is ahead++- **GIVEN** records of the `private` partition whose clock is days ahead+ of now+- **WHEN** a remote caller runs the doctor+- **THEN** no `store-clock-ahead` finding names the `private` partition+- **AND** a local caller's doctor reports it++### Requirement: A failed private write is withheld below local reach++Below local reach the write failure findings (`store-dual-write-failures`,+`store-write-fail-open`) SHALL show only the failures known to concern+only `shared`: every record of the operation was placed in `shared` when+its append failed. For any other failure they SHALL NOT show its count,+time, error text or path, since the error text can name a private shard.+The skipped dual writes SHALL be reported below local reach without their+count.++#### Scenario: A failed private write below local reach++- **GIVEN** a dual write and a fail-open store-only write into the+ `private` partition that failed with an error naming the private path+- **WHEN** a remote caller runs `brain_doctor`+- **THEN** no `store-dual-write-failures` or `store-write-fail-open`+ finding is reported, and nothing names the path+- **AND** a local caller's doctor reports both, with the error text+- **AND** once a write into `shared` fails too, the remote caller sees+ that one failure, with its own error text and time only++### Requirement: A lost private identity file is reported below local reach without its name++A lost `private` identity file follows the same rule: the store is not+opened and every protection fails closed for `shared` too, so below local+reach the `store-partition-lost` error SHALL still be reported, but+without naming the `private` partition. With `shared` lost as well it+SHALL name `shared` and "another partition".++#### Scenario: A lost private identity file below local reach++- **GIVEN** a machine that held the `private` partition whose+ `private/store.json` is gone+- **WHEN** a remote caller runs the doctor+- **THEN** it reports an error `store-partition-lost` that does not name+ the `private` partition++### Requirement: A store that does not open is reported below local reach without its cause++When the store does not open for a reason other than a lost partition,+the error text can name a partition's file or the store ids. Below local+reach the `store-root-invalid` finding SHALL still be reported, with the+store root as its path, but with a generic message that names no+partition, file of a partition or store id. A local caller's doctor SHALL+report the error text as it is.++#### Scenario: A malformed private identity file below local reach++- **GIVEN** a machine whose `private/store.json` does not parse, or+ carries the id of another store+- **WHEN** a remote caller runs the doctor+- **THEN** it reports `store-root-invalid` with the store root, and the+ message names neither the `private` partition nor a store id+- **AND** a local caller's doctor reports the error text that names+ `private/store.json` or both store ids++### Requirement: A quarantined line releases the protection reads on this machine++`o2b store shard quarantine <partition>/<shard> --offset <n>` SHALL+record, on this machine only, that the line at that offset with exactly+its current bytes is skipped; the shard SHALL then read as whole on this+machine, the line SHALL stay in the file, and no other machine SHALL be+affected.++#### Scenario: Capture reopens++- **GIVEN** a store-only machine whose capture is closed by a corrupt line+ in a shared shard+- **WHEN** the operator quarantines that line+- **THEN** capture writes again on this machine+- **AND** the shard's bytes are unchanged++Every quarantine mark, of a corrupt line and of a pending blob alike,+SHALL require both the byte offset and the hash of the line's bytes to+match one ledger entry of the matching kind: a `corrupt-line` entry marks+only a corrupt line, a `blob-pending` entry only a pending record, and a+pending record of a `settings` shard SHALL never be marked. The shard or+the offset alone SHALL never mark a line.++#### Scenario: An entry of another kind++- **GIVEN** a `corrupt-line` entry whose offset and line hash match a+ record whose blob is pending+- **WHEN** the shard is read+- **THEN** the pending record is a gap++### Requirement: Quarantining the same bytes as the other kind replaces the entry++Quarantining a line SHALL answer `already` only when the ledger holds an+entry for that offset with the same bytes and the same kind; an entry of+the other kind SHALL be replaced, so the line is marked by its current+kind.++#### Scenario: The same bytes quarantined as the other kind++- **GIVEN** a `corrupt-line` entry whose offset and line hash match a+ record whose blob is pending+- **WHEN** the operator quarantines that record+- **THEN** the verb answers `quarantined`, the entry's kind becomes+ `blob-pending`, and the record no longer counts as a gap++#### Scenario: The line changes++- **GIVEN** a quarantined line+- **WHEN** a different line appears at the same offset (the shard was+ replaced)+- **THEN** the new line is a gap again and the doctor reports it++#### Scenario: A second corrupt line in the same shard++- **GIVEN** a shard with one quarantined corrupt line+- **WHEN** another corrupt line appears at a different offset of the same+ shard+- **THEN** that line is a gap: the protection reads fail closed again and+ the doctor reports it++### Requirement: Some gaps are never quarantined++The verb SHALL refuse, with exit code 1 and a named reason: a line of a+newer format; a pending blob younger than seven days; a pending blob in a+`settings` shard; a pending blob whose record's clock stamp does not parse+as a time; the last line of a shard of this machine's own device, whether+the reader reports it or holds it back for a retry; an offset that holds+no gap.++#### Scenario: A newer-format line++- **WHEN** the operator quarantines a newer-format line+- **THEN** the verb exits 1 with reason `newer-format` and records nothing++#### Scenario: A young pending blob++- **WHEN** the operator quarantines a pending blob written one day ago+- **THEN** the verb exits 1 with reason `blob-too-young`++#### Scenario: A pending blob with an unreadable clock stamp++- **WHEN** the operator quarantines a pending blob whose record's clock+ stamp does not parse as a time+- **THEN** the verb exits 1 with reason `blob-clock-unreadable` and records+ nothing++#### Scenario: A held corrupt last line of this machine's own shard++- **WHEN** the operator quarantines the last line of a shard this machine+ writes, which does not verify and which the reader holds back+- **THEN** the verb exits 1 with reason `own-tail`++### Requirement: A line the verb cannot hash the same way twice is refused by cause++The verb SHALL hash the line before and after it is judged. A line that+runs past the hash cap both times SHALL be refused as `line-too-long`,+which stays so until the shard's bytes change, so the refusal SHALL NOT+say to run the command again. Hashes that differ, a line that could not+be read through to its end once, or a shard removed between the listing+and the read SHALL be refused as `shard-changed`, which is transient:+running the command again checks it once more.++#### Scenario: A corrupt line too long to hash++- **WHEN** the operator quarantines a corrupt line longer than the hash cap+- **THEN** the verb exits 1 with reason `line-too-long`, records nothing,+ and does not say to run the command again++#### Scenario: A shard removed after the listing++- **WHEN** a sync tool removes the shard after the verb listed it and+ before it read the shard+- **THEN** the verb exits 1 with reason `shard-changed` and says to run+ the command again++### Requirement: A shard the verb cannot read is refused by name++When the shard the operator named, or the folder of its stream, is there+and cannot be read (a permission or input error), `quarantine` SHALL+refuse with `shard-unreadable`, change nothing, and name the read error;+it SHALL NOT answer `unknown-shard`, which is kept for a shard a+successful listing does not hold.++#### Scenario: A shard this account cannot read++- **GIVEN** a held shard whose file this account may not read+- **WHEN** the operator quarantines a line of it with `--json`+- **THEN** the verb exits 1 and answers+ `{ ok: false, reason: "shard-unreadable", error }`, and no ledger entry+ is written++### Requirement: A ledger lock held too long is refused as transient++`quarantine` and `release` change the ledger under its device-local lock.+When another process holds that lock past the wait budget, the verb SHALL+refuse with `ledger-busy`, change nothing, and say to run the command+again; under `--json` the answer SHALL be `{ ok: false, reason, error }`.++#### Scenario: Another process holds the ledger lock++- **GIVEN** another process that holds the quarantine ledger's lock past+ the wait budget+- **WHEN** the operator quarantines or releases a line with `--json`+- **THEN** the verb exits 1 and answers+ `{ ok: false, reason: "ledger-busy", error }`, and the error says to run+ the command again++### Requirement: Every refusal of the verb has a named reason++The JSON reasons of the verb's refusals SHALL be exactly: `not-held`,+`unknown-shard`, `no-gap`, `newer-format`, `blob-too-young`,+`settings-blob`, `blob-clock-unreadable`, `own-tail`, `line-too-long`,+`shard-changed`, `shard-unreadable`, `ledger-unreadable`, `ledger-busy`, and+the store open errors+`no-store`, `partition-lost`, `store-open-error`, `store-root-in-vault` and+`config`.++#### Scenario: A refusal under --json++- **WHEN** the operator quarantines an offset that holds no gap with+ `--json`+- **THEN** the verb exits 1 and answers `{ ok: false, reason: "no-gap", error }`++### Requirement: A ledger this release cannot rewrite is left as it is++A ledger file that cannot be read, does not parse, has another version,+or holds an entry this release does not read SHALL be refused with+`ledger-unreadable` and left as it is; a file that cannot be read SHALL+be named as unreadable, not as one that does not parse.++#### Scenario: A ledger this release cannot rewrite++- **GIVEN** a ledger file of another version+- **WHEN** the operator quarantines or releases a line+- **THEN** the verb exits 1 with reason `ledger-unreadable` and the ledger+ file is unchanged++### Requirement: The reader applies entries only from a ledger of the supported version++The reader SHALL apply quarantine entries only from a ledger file that is+a JSON object of the version this release writes with a list of lines. A+ledger of another version, one with no version, one that is not an+object, has no list of lines, cannot be read or does not parse SHALL+quarantine nothing, so every line it names stays a gap.++#### Scenario: A ledger of another version++- **GIVEN** a ledger holding a valid entry for a corrupt line whose+ version is changed to 2+- **WHEN** the shard is read+- **THEN** the line is a gap again, and `o2b store shard list` lists no+ entry++### Requirement: Each ledger entry stands on its own++Within a ledger of the supported version an entry this release does not+read SHALL apply nothing, and the other entries SHALL still apply. A+skipped entry only leaves its line a gap, so it never makes a shard read+as whole.++#### Scenario: An entry this release does not read++- **GIVEN** a ledger of the supported version holding a valid entry and+ an entry of a kind this release does not know+- **WHEN** the shard is read+- **THEN** the valid entry still marks its line, and the unknown entry+ marks nothing++### Requirement: A quarantined pending blob reads normally once it arrives++A record whose pending blob was quarantined SHALL be left out of every+read while the blob is absent, and SHALL read as any other record as soon+as the blob arrives, with no operator action.++#### Scenario: The blob arrives late++- **GIVEN** a quarantined pending blob+- **WHEN** the blob file arrives+- **THEN** readers return the record with its body++### Requirement: Rollback and the late import still refuse on a pending blob++The legacy rollback, the late import of the transition, the migration+preflight and the legacy removal SHALL keep refusing while any held shard+outside the digest stream holds a record whose blob has not arrived,+whether or not it was quarantined; none of them waits for it. A shard of+the digest stream holds published digests, no record any of them imports+or restores, and is not counted. The legacy removal's existing `--force`+goes past such a shard, as it goes past any shard read only in part.++#### Scenario: Rollback with a quarantined pending blob++- **GIVEN** a quarantined pending blob+- **WHEN** the operator runs the rollback+- **THEN** it refuses and names the shard++### Requirement: Quarantines can be listed and released++`o2b store shard list` SHALL print every quarantined line of this+machine's store with its state (`held`, `resolved` when the line now reads+as a record, `stale` when other bytes stand at the offset);+`o2b store shard release <partition>/<shard> --offset <n>` SHALL remove+the entry. Both, and `quarantine`, SHALL take `--config` and `--json`,+exit 0 for an answer, 1 for a refusal and 2 for a usage error, and under+`--json` answer `{ ok: false, reason, error }` for each refusal.++#### Scenario: Releasing an entry++- **GIVEN** a quarantined corrupt line+- **WHEN** the operator releases it+- **THEN** the shard is a gap again and capture closes again++#### Scenario: Releasing what is not quarantined++- **WHEN** the operator releases an offset with no entry+- **THEN** the verb exits 0 and says nothing was quarantined there++### Requirement: A release does not claim that a gap reopens++A release removes the entry's mark; the line is a gap again only if it+still is one. The release message SHALL say so and SHALL NOT say that the+shard is read only in part again, since a `resolved` entry, whose blob has+arrived, and a `stale` one leave no gap of their own.++#### Scenario: Releasing a resolved entry++- **GIVEN** a quarantined pending blob whose blob has arrived since, so+ `o2b store shard list` shows it `resolved`+- **WHEN** the operator releases it+- **THEN** the verb says the mark is removed and the line is a gap again+ only if it still is one, and the shard still reads wholediff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex 9658eee6..e1a7c084 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -1342,6 +1342,28 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({]),],),+ command(+ "partition",+ "Take a partition off this machine on purpose",+ [flag("config", "string"), flag("json", "boolean")],+ [+ command("leave", "Stop holding a partition whose folder is gone (never shared)", [+ flag("force", "boolean"),+ ]),+ ],+ ),+ command(+ "shard",+ "Skip one bad line of a held shard on this machine, list the skipped lines, or release one",+ [flag("config", "string"), flag("json", "boolean")],+ [+ command("quarantine", "Skip one bad line of a held shard on this machine", [+ flag("offset", "string"),+ ]),+ command("list", "List the quarantined lines with their state"),+ command("release", "Read a quarantined line again", [flag("offset", "string")]),+ ],+ ),],),command(diff --git a/src/cli/store-partition.ts b/src/cli/store-partition.tsnew file mode 100644index 00000000..91d7dfc0--- /dev/null+++ b/src/cli/store-partition.ts@@ -0,0 +1,161 @@+/**+ * `o2b store partition leave` (design D7 item 5 of change `store-hardening`):+ * take a partition off this machine on purpose, once its folder no longer+ * carries the store identity file. The work lives in+ * `core/store/partition-leave.ts`; this file parses flags and renders.+ *+ * Exit codes: 0 for an answer (left, or nothing to leave), 1 for a refusal+ * (`shared`, `still-held`, `files-left`, `no-store-id`, `clear-failed`,+ * `store-root-in-vault`, `config`), 2 for a broken argv. Under `--json`+ * every refusal and usage error is a `{ ok: false, reason, error }` answer+ * on stdout.+ */++import { ConfigReadError } from "../core/config.ts";+import { join } from "node:path";+import {+ leavePartition,+ PartitionLeaveRefusal,+ type LeavePartitionResult,+} from "../core/store/partition-leave.ts";+import {+ STORE_IDENTITY_FILE,+ STORE_PARTITIONS,+ type StorePartition,+} from "../core/store/constants.ts";+import {+ partitionDir,+ resolveConfiguredStoreRoot,+ StoreRootInsideVaultError,+} from "../core/store/store-paths.ts";+import { CliError, parseFlags } from "./argparse.ts";+import { sortedReplacer } from "./helpers.ts";++export const STORE_PARTITION_USAGE =+ "usage: o2b store partition leave <partition> [--force] [--config <file>] [--json]";++const EXIT = Object.freeze({ ok: 0, refused: 1, usage: 2 } as const);++export function cmdStorePartition(argv: ReadonlyArray<string>): number {+ const verb = argv[0];+ if (verb === "-h" || verb === "--help") {+ process.stdout.write(`${STORE_PARTITION_USAGE}\n`);+ return EXIT.ok;+ }+ if (verb === "leave") return cmdLeave(argv.slice(1));+ const message =+ verb === undefined ? "missing store partition verb" : `unknown store partition verb: ${verb}`;+ return usage(argv, message);+}++/** A usage error: the usage on stderr, and the JSON failure on stdout under `--json`. */+function usage(argv: ReadonlyArray<string>, message: string): number {+ if (argv.includes("--json")) {+ process.stdout.write(+ `${JSON.stringify({ ok: false, reason: "usage", error: message }, sortedReplacer, 2)}\n`,+ );+ }+ process.stderr.write(`error: ${message}\n${STORE_PARTITION_USAGE}\n`);+ return EXIT.usage;+}++/** A refusal: the message on stderr and, under `--json`, the failure on stdout. */+function refuse(err: Error, reason: string, json: boolean): number {+ if (json) {+ process.stdout.write(+ `${JSON.stringify({ ok: false, reason, error: err.message }, sortedReplacer, 2)}\n`,+ );+ }+ process.stderr.write(`error: ${err.message}\n`);+ return EXIT.refused;+}++/** Parse leave's flags; a `CliError` is a usage error with the usage text. */+function parse(+ argv: ReadonlyArray<string>,+): { flags: Record<string, unknown>; positional: string[] } | number {+ try {+ const { flags, positional } = parseFlags(argv, {+ force: { type: "boolean" },+ config: { type: "string" },+ json: { type: "boolean" },+ });+ return { flags, positional };+ } catch (err) {+ if (!(err instanceof CliError)) throw err;+ return usage(argv, err.message);+ }+}++/** The text answer of a leave: what was cleared, or that there is nothing to leave. */+function renderText(result: LeavePartitionResult, root: string): string {+ const cleared = result.cleared;+ const identityPath = join(partitionDir(root, result.partition), STORE_IDENTITY_FILE);+ if (result.left) {+ return (+ `Left the ${result.partition} partition of the brain store at ${root} on this machine.\n` ++ ` cleared: held marker, ${cleared.seenShards} seen shard entr` ++ `${cleared.seenShards === 1 ? "y" : "ies"}, ` ++ `${cleared.digestCaches ? "digest caches" : "no digest cache"}\n` ++ `The search index drops the ${result.partition} partition's records on its next catch-up. ` ++ `If ${identityPath} returns, by sync or a restore, this ` ++ `machine holds the ${result.partition} partition again from its next store write.\n`+ );+ }+ return (+ `This machine does not hold the ${result.partition} partition of the brain store at ${root}; ` ++ `nothing to leave. If ${identityPath} arrives, by sync or a ` ++ `restore, this machine holds the ${result.partition} partition again from its next store ` ++ "write.\n"+ );+}++function cmdLeave(argv: ReadonlyArray<string>): number {+ const parsed = parse(argv);+ if (typeof parsed === "number") return parsed;+ const json = parsed.flags["json"] === true;+ if (parsed.positional.length === 0) {+ return usage(argv, "name the partition to leave (shared or private)");+ }+ if (parsed.positional.length > 1) {+ return usage(argv, `unexpected argument: ${parsed.positional[1]}`);+ }+ const partition = parsed.positional[0]!;+ if (!(STORE_PARTITIONS as ReadonlyArray<string>).includes(partition)) {+ return usage(argv, `unknown partition: ${partition}`);+ }+ const configPath = parsed.flags["config"] as string | undefined;+ try {+ const { root } = resolveConfiguredStoreRoot(configPath);+ const result = leavePartition({+ root,+ partition: partition as StorePartition,+ ...(parsed.flags["force"] === true ? { force: true } : {}),+ });+ process.stdout.write(+ json+ ? `${JSON.stringify(+ {+ ok: true,+ left: result.left,+ partition: result.partition,+ root,+ cleared: {+ held_marker: result.cleared.heldMarker,+ seen_shards: result.cleared.seenShards,+ digest_caches: result.cleared.digestCaches,+ },+ },+ sortedReplacer,+ 2,+ )}\n`+ : renderText(result, root),+ );+ return EXIT.ok;+ } catch (err) {+ if (err instanceof PartitionLeaveRefusal) return refuse(err, err.reason, json);+ if (err instanceof StoreRootInsideVaultError) return refuse(err, "store-root-in-vault", json);+ if (err instanceof ConfigReadError) return refuse(err, "config", json);+ throw err;+ }+}diff --git a/src/cli/store-shard.ts b/src/cli/store-shard.tsnew file mode 100644index 00000000..d45cb45a--- /dev/null+++ b/src/cli/store-shard.ts@@ -0,0 +1,535 @@+/**+ * `o2b store shard quarantine|list|release` (change store-hardening,+ * design D6; spec `store-shard-gaps`): record on this machine that one bad+ * line of a held shard is skipped, list what is recorded, and release an+ * entry. The ledger lives in `core/store/shard-quarantine.ts`; this file+ * checks the line, parses flags and renders. The store is only inspected+ * (`inspectStore`), and the device id is read without minting+ * (`readDeviceId`).+ *+ * A line is quarantined only when it is a gap of the shard right now: a+ * corrupt line in the middle, or a record whose body blob has not arrived+ * for seven days (`PENDING_BLOB_QUARANTINE_AGE_MS`) in a shard other than+ * `settings`. Never a newer-format+ * line (upgrading reads it), never the last line of a shard of this+ * machine's own device (its writer may still be appending).+ *+ * Exit codes: 0 for an answer (`quarantined`, `already`, a list,+ * `released`, `not-quarantined`), 1 for a refusal (`no-store`,+ * `partition-lost`, `store-open-error`, `store-root-in-vault`, `config`,+ * `not-held`, `unknown-shard`, `no-gap`, `newer-format`, `blob-too-young`,+ * `settings-blob`, `blob-clock-unreadable`, `own-tail`, `line-too-long`,+ * `shard-changed`, `shard-unreadable`, `ledger-unreadable`, `ledger-busy`),+ * 2 for a broken argv.+ * `line-too-long` (the line runs past the hash cap) is permanent until the+ * shard's bytes change; `shard-changed` (the line's bytes differ, or could+ * not be read, between the hashes taken before and after it is judged, or+ * the shard was removed after it was listed) is transient, and running the+ * command again checks it once more. `shard-unreadable` (the shard or its+ * stream folder is there and cannot be read) stays until the operator makes+ * it readable. `ledger-busy` (another process held the+ * ledger's device-local lock past the wait budget) is transient too: run+ * the command again. Under `--json` every refusal and usage error is a+ * `{ ok: false, reason, error }` answer on stdout.+ */++import { ConfigReadError, readDeviceId } from "../core/config.ts";+import {+ LEDGER_STREAM,+ STORE_PARTITIONS,+ STORE_STREAMS,+ type StorePartition,+ type StoreStream,+} from "../core/store/constants.ts";+import { readShard } from "../core/store/reader.ts";+import { listShards, type ShardEntry } from "../core/store/shard-list.ts";+import {+ addQuarantine,+ QuarantineLedgerUnreadableError,+ releaseQuarantine,+} from "../core/store/shard-quarantine-write.ts";+import {+ listQuarantine,+ probeLineHashAt,+ QUARANTINE_HASH_CAP_BYTES,+ PENDING_BLOB_QUARANTINE_AGE_MS,+ PENDING_BLOB_QUARANTINE_AGE_TEXT,+ pendingRecordWrittenMs,+ type QuarantineKind,+} from "../core/store/shard-quarantine.ts";+import {+ HeldPartitionLostError,+ inspectStore,+ NoStoreError,+ StoreOpenError,+ type StoreInspection,+} from "../core/store/store-open.ts";+import {+ partitionDir,+ resolveConfiguredStoreRoot,+ StoreRootInsideVaultError,+} from "../core/store/store-paths.ts";+import { CliError, parseFlags } from "./argparse.ts";+import { sortedReplacer } from "./helpers.ts";++const QUARANTINE_USAGE =+ "usage: o2b store shard quarantine <partition>/<shard> --offset <n> [--config <file>] [--json]";+const LIST_USAGE = "usage: o2b store shard list [--config <file>] [--json]";+const RELEASE_USAGE =+ "usage: o2b store shard release <partition>/<shard> --offset <n> [--config <file>] [--json]";+export const STORE_SHARD_USAGE = `${QUARANTINE_USAGE}\n${LIST_USAGE}\n${RELEASE_USAGE}`;++const EXIT = Object.freeze({ ok: 0, refused: 1, usage: 2 } as const);++type RefusalReason =+ | "not-held"+ | "unknown-shard"+ | "no-gap"+ | "newer-format"+ | "blob-too-young"+ | "settings-blob"+ | "blob-clock-unreadable"+ | "own-tail"+ | "line-too-long"+ | "shard-changed"+ | "shard-unreadable"+ | "ledger-busy";++/** A refusal of this verb itself, with its JSON reason. */+class ShardVerbRefusal extends Error {+ constructor(+ readonly reason: RefusalReason,+ message: string,+ ) {+ super(message);+ this.name = "ShardVerbRefusal";+ }+}++export function cmdStoreShard(argv: ReadonlyArray<string>): number {+ const verb = argv[0];+ if (verb === "-h" || verb === "--help") {+ process.stdout.write(`${STORE_SHARD_USAGE}\n`);+ return EXIT.ok;+ }+ if (verb === "quarantine") return cmdQuarantine(argv.slice(1));+ if (verb === "list") return cmdList(argv.slice(1));+ if (verb === "release") return cmdRelease(argv.slice(1));+ return usage(+ argv,+ verb === undefined ? "missing store shard verb" : `unknown store shard verb: ${verb}`,+ STORE_SHARD_USAGE,+ );+}++function isRefusal(err: unknown): err is Error {+ return (+ err instanceof ShardVerbRefusal ||+ err instanceof QuarantineLedgerUnreadableError ||+ err instanceof StoreOpenError ||+ err instanceof StoreRootInsideVaultError ||+ err instanceof ConfigReadError+ );+}++/** The JSON reason of a refusal: one per error class, so a caller can act on it. */+function reasonOf(err: Error): string {+ if (err instanceof ShardVerbRefusal) return err.reason;+ if (err instanceof QuarantineLedgerUnreadableError) return "ledger-unreadable";+ if (err instanceof NoStoreError) return "no-store";+ if (err instanceof HeldPartitionLostError) return "partition-lost";+ if (err instanceof StoreOpenError) return "store-open-error";+ if (err instanceof StoreRootInsideVaultError) return "store-root-in-vault";+ return "config";+}++function refuse(err: Error, json: boolean): number {+ if (json) {+ process.stdout.write(+ `${JSON.stringify({ ok: false, reason: reasonOf(err), error: err.message }, sortedReplacer, 2)}\n`,+ );+ }+ process.stderr.write(`error: ${err.message}\n`);+ return EXIT.refused;+}++/** A usage error: the usage on stderr, and the JSON failure on stdout under `--json`. */+function usage(argv: ReadonlyArray<string>, message: string, text: string): number {+ if (argv.includes("--json")) {+ process.stdout.write(+ `${JSON.stringify({ ok: false, reason: "usage", error: message }, sortedReplacer, 2)}\n`,+ );+ }+ process.stderr.write(`error: ${message}\n${text}\n`);+ return EXIT.usage;+}++function answer(json: boolean, body: Record<string, unknown>, text: string): number {+ process.stdout.write(json ? `${JSON.stringify(body, sortedReplacer, 2)}\n` : text);+ return EXIT.ok;+}++/** The line a verb names: its partition, its partition-relative shard path, its offset. */+interface LineArg {+ readonly partition: StorePartition;+ readonly shard: string;+ readonly offset: number;+ readonly configPath: string | undefined;+ readonly json: boolean;+}++/** Parse `<partition>/<shard> --offset <n>`; a number is the usage error's exit code. */+function parseLine(argv: ReadonlyArray<string>, usageText: string): LineArg | number {+ let flags: Record<string, unknown>;+ let positional: string[];+ try {+ ({ flags, positional } = parseFlags(argv, {+ offset: { type: "string" },+ config: { type: "string" },+ json: { type: "boolean" },+ }));+ } catch (err) {+ if (!(err instanceof CliError)) throw err;+ return usage(argv, err.message, usageText);+ }+ if (positional.length === 0) {+ return usage(argv, "name the shard as <partition>/<shard>", usageText);+ }+ if (positional.length > 1) {+ return usage(argv, `unexpected argument: ${positional[1]}`, usageText);+ }+ const target = positional[0]!;+ const slash = target.indexOf("/");+ const partition = slash === -1 ? target : target.slice(0, slash);+ if (!(STORE_PARTITIONS as ReadonlyArray<string>).includes(partition)) {+ return usage(argv, `unknown partition: ${partition}`, usageText);+ }+ const shard = slash === -1 ? "" : target.slice(slash + 1);+ if (shard === "") return usage(argv, "name the shard as <partition>/<shard>", usageText);+ const offsetText = flags["offset"];+ if (typeof offsetText !== "string") return usage(argv, "--offset <n> is required", usageText);+ const offset = /^\d+$/.test(offsetText) ? Number(offsetText) : Number.NaN;+ if (!Number.isSafeInteger(offset)) {+ return usage(argv, `--offset takes a byte offset, got ${offsetText}`, usageText);+ }+ return {+ partition: partition as StorePartition,+ shard,+ offset,+ configPath: flags["config"] as string | undefined,+ json: flags["json"] === true,+ };+}++/** The store this machine reads, inspected only. */+function storeHere(configPath: string | undefined): StoreInspection {+ const { root } = resolveConfiguredStoreRoot(configPath);+ const store = inspectStore(root);+ if (store === null) throw new NoStoreError(root);+ return store;+}++/** The listed shard `rel` of `partition`, matched against the store's own listing. */+function findShard(store: StoreInspection, partition: StorePartition, rel: string): ShardEntry {+ const unknown = new ShardVerbRefusal(+ "unknown-shard",+ `the ${partition} partition of the brain store at ${store.root} holds no shard ${rel}`,+ );+ const segments = rel.split("/");+ if (segments.some((s) => s === "" || s === "." || s === "..")) throw unknown;+ const stream = segments[0]!;+ if (!(STORE_STREAMS as ReadonlyArray<string>).includes(stream)) throw unknown;+ if (segments.length !== (stream === LEDGER_STREAM ? 4 : 3)) throw unknown;+ const ref =+ segments.length === 3+ ? { stream: stream as StoreStream }+ : { stream: stream as StoreStream, ledger: segments[1]! };+ let shards: ShardEntry[];+ try {+ shards = listShards(partitionDir(store.root, partition), ref);+ } catch (err) {+ // A stream folder that is not there lists no shard; one that is there+ // and cannot be listed is not a proof that the shard is absent.+ if ((err as NodeJS.ErrnoException).code === "ENOENT") throw unknown;+ throw unreadableShard(`${partition}/${rel}`, err);+ }+ const found = shards.find((s) => s.rel === rel);+ if (found === undefined) throw unknown;+ return found;+}++/** The refusal for a shard, or its stream folder, that is there and cannot be read. */+function unreadableShard(named: string, err: unknown): ShardVerbRefusal {+ return new ShardVerbRefusal(+ "shard-unreadable",+ `the shard ${named} cannot be read (${err instanceof Error ? err.message : String(err)}); ` ++ "nothing was changed. Make it readable for this account, then run the command again",+ );+}++/**+ * Read the shard the listing named. A sync tool may remove or replace it+ * between the listing and this read; the reader then throws `ENOENT`,+ * which is the same transient change the two line hashes guard against,+ * so it answers `shard-changed` rather than an unnamed error. Any other+ * read failure answers `shard-unreadable`.+ */+export function readListedShard(+ path: string,+ partitionRoot: string,+ named: string,+): ReturnType<typeof readShard> {+ try {+ return readShard(path, { partitionRoot });+ } catch (err) {+ if ((err as NodeJS.ErrnoException).code !== "ENOENT") throw unreadableShard(named, err);+ throw new ShardVerbRefusal(+ "shard-changed",+ `the shard ${named} was removed after it was listed (a sync tool may be replacing it); ` ++ "run the command again",+ );+ }+}++/**+ * Run a change of the quarantine ledger. Its device-local lock is held+ * while another `o2b` process changes the ledger; when it stays held past+ * the wait budget, the lock answers `ELOCKED`, which is transient and+ * answers `ledger-busy` so a `--json` caller gets a named refusal.+ */+function underLedgerLock<T>(change: () => T): T {+ try {+ return change();+ } catch (err) {+ if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;+ throw new ShardVerbRefusal(+ "ledger-busy",+ "another o2b process on this machine holds the quarantine ledger's lock and did not " ++ `release it in time (${(err as Error).message}); run the command again`,+ );+ }+}++/** What the line at `offset` is: the kind to record and its reason, or a refusal. */+function gapAt(+ line: LineArg,+ entry: ShardEntry,+ store: StoreInspection,+): { kind: QuarantineKind; reason: string } {+ const partitionRoot = partitionDir(store.root, line.partition);+ const named = `${line.partition}/${line.shard}`;+ const read = readListedShard(entry.path, partitionRoot, named);+ const diagnostic = read.diagnostics.find(+ (d) =>+ d.offset === line.offset &&+ d.torn !== true &&+ (d.kind === "corrupt-line" || d.kind === "unknown-version"),+ );+ // A shard that is no regular file (a link, a folder) was not read at+ // all, so no line of it can be judged.+ const notRead = read.diagnostics.find((d) => d.kind === "not-a-file");+ if (notRead !== undefined) throw unreadableShard(named, new Error(notRead.reason));+ const pending = read.records.find((r) => r.offset === line.offset && r.bodyState === "pending");+ const ownTail = (): ShardVerbRefusal | null =>+ entry.device === readDeviceId(line.configPath ?? null)+ ? new ShardVerbRefusal(+ "own-tail",+ `the line at byte ${line.offset} is the last line of the shard ${named}, which this ` ++ "machine writes; its writer may still be appending, so it is not quarantined",+ )+ : null;+ // The reader holds back an unverified last line for a retry: it is no+ // diagnostic yet, but in this machine's own shard it is the tail its+ // writer may still be appending.+ if (+ diagnostic === undefined &&+ pending === undefined &&+ read.pendingTailBytes > 0 &&+ line.offset === read.nextOffset+ ) {+ const refusal = ownTail();+ if (refusal !== null) throw refusal;+ }+ if (diagnostic === undefined && pending === undefined) {+ throw new ShardVerbRefusal(+ "no-gap",+ `nothing at byte ${line.offset} of the shard ${named} is a gap; the doctor names each gap ` ++ "with its offset",+ );+ }+ if (diagnostic?.kind === "unknown-version") {+ throw new ShardVerbRefusal(+ "newer-format",+ `the record at byte ${line.offset} of the shard ${named} is of a newer format ` ++ `(${diagnostic.reason}); upgrade Open Second Brain on this machine to read it`,+ );+ }+ const last =+ read.pendingTailBytes === 0 &&+ !read.records.some((r) => r.offset > line.offset) &&+ !read.diagnostics.some((d) => d.offset > line.offset);+ if (last) {+ const refusal = ownTail();+ if (refusal !== null) throw refusal;+ }+ if (diagnostic !== undefined) return { kind: "corrupt-line", reason: diagnostic.reason };+ if (line.shard.split("/")[0] === "settings") {+ throw new ShardVerbRefusal(+ "settings-blob",+ `the settings record at byte ${line.offset} of the shard ${named} waits for its body ` ++ "blob; a settings record is never quarantined: deliver the blob from the machine that " ++ "wrote it",+ );+ }+ const written = pendingRecordWrittenMs(pending!.record.hlc);+ if (written === null) {+ throw new ShardVerbRefusal(+ "blob-clock-unreadable",+ `the record at byte ${line.offset} of the shard ${named} waits for a body blob, and its ` ++ "clock stamp does not parse as a time, so its age is unknown; it is never quarantined " ++ "by age: deliver the blob, or repair the record, from the machine that wrote it",+ );+ }+ if (Date.now() - written < PENDING_BLOB_QUARANTINE_AGE_MS) {+ throw new ShardVerbRefusal(+ "blob-too-young",+ `the record at byte ${line.offset} of the shard ${named} waits for a body blob written ` ++ `${new Date(written).toISOString()}; a sync tool may still deliver it, so it is not ` ++ `quarantined before it is ${PENDING_BLOB_QUARANTINE_AGE_TEXT} old`,+ );+ }+ return { kind: "blob-pending", reason: "its body blob has not arrived" };+}++function cmdQuarantine(argv: ReadonlyArray<string>): number {+ const line = parseLine(argv, QUARANTINE_USAGE);+ if (typeof line === "number") return line;+ const named = `${line.partition}/${line.shard}`;+ try {+ const store = storeHere(line.configPath);+ if (!store.partitions.has(line.partition)) {+ throw new ShardVerbRefusal(+ "not-held",+ `this machine does not hold the ${line.partition} partition of the brain store at ${store.root}`,+ );+ }+ const entry = findShard(store, line.partition, line.shard);+ const partitionRoot = partitionDir(store.root, line.partition);+ // Hashed before and after the line is judged: a sync tool may rewrite+ // the shard between the two, and the ledger must name the bytes judged.+ const before = probeLineHashAt(entry.path, line.offset, partitionRoot);+ const { kind, reason } = gapAt(line, entry, store);+ const after = probeLineHashAt(entry.path, line.offset, partitionRoot);+ if (+ "failure" in before &&+ "failure" in after &&+ before.failure === "too-long" &&+ after.failure === "too-long"+ ) {+ throw new ShardVerbRefusal(+ "line-too-long",+ `the line at byte ${line.offset} of the shard ${named} runs past ` ++ `${QUARANTINE_HASH_CAP_BYTES} bytes with no newline, the longest line a quarantine ` ++ "can name, so it cannot be quarantined and stays a gap. Repair the shard at its " ++ "source, or deliver a whole copy from the machine that wrote it",+ );+ }+ if (!("sha256" in before) || !("sha256" in after) || after.sha256 !== before.sha256) {+ throw new ShardVerbRefusal(+ "shard-changed",+ `the bytes at byte ${line.offset} of the shard ${named} changed, or could not be read ` ++ "through to their end, while they were checked (a sync tool may be writing the " ++ "shard); run the command again",+ );+ }+ const sha256 = after.sha256;+ const status = underLedgerLock(() =>+ addQuarantine(store.root, {+ partition: line.partition,+ shard: line.shard,+ offset: line.offset,+ sha256,+ kind,+ reason,+ }),+ );+ return answer(+ line.json,+ { ok: true, status, partition: line.partition, shard: line.shard, offset: line.offset, kind },+ status === "already"+ ? `The line at byte ${line.offset} of ${named} is already quarantined on this machine; nothing changed.\n`+ : `Quarantined the line at byte ${line.offset} of ${named} on this machine (${kind}). ` ++ "It is skipped here and stays in the file; no other machine is affected. If other " ++ "bytes come to stand at that offset, they are read again.\n",+ );+ } catch (err) {+ if (!isRefusal(err)) throw err;+ return refuse(err, line.json);+ }+}++function cmdList(argv: ReadonlyArray<string>): number {+ let flags: Record<string, unknown>;+ let positional: string[];+ try {+ ({ flags, positional } = parseFlags(argv, {+ config: { type: "string" },+ json: { type: "boolean" },+ }));+ } catch (err) {+ if (!(err instanceof CliError)) throw err;+ return usage(argv, err.message, LIST_USAGE);+ }+ if (positional.length > 0)+ return usage(argv, `unexpected argument: ${positional[0]}`, LIST_USAGE);+ const json = flags["json"] === true;+ try {+ const store = storeHere(flags["config"] as string | undefined);+ const lines = listQuarantine(store.root, readShard).map((l) => ({+ partition: l.partition,+ shard: l.shard,+ offset: l.offset,+ kind: l.kind,+ state: l.state,+ quarantined_at: l.quarantined_at,+ }));+ const text =+ lines.length === 0+ ? "No shard line is quarantined on this machine.\n"+ : lines+ .map(+ (l) =>+ `${l.partition}/${l.shard} at byte ${l.offset}: ${l.kind}, ${l.state}, ` ++ `since ${l.quarantined_at}\n`,+ )+ .join("");+ return answer(json, { ok: true, lines }, text);+ } catch (err) {+ if (!isRefusal(err)) throw err;+ return refuse(err, json);+ }+}++function cmdRelease(argv: ReadonlyArray<string>): number {+ const line = parseLine(argv, RELEASE_USAGE);+ if (typeof line === "number") return line;+ const named = `${line.partition}/${line.shard}`;+ try {+ const store = storeHere(line.configPath);+ const status = underLedgerLock(() =>+ releaseQuarantine(store.root, line.partition, line.shard, line.offset),+ );+ return answer(+ line.json,+ { ok: true, status, partition: line.partition, shard: line.shard, offset: line.offset },+ status === "released"+ ? `Released the line at byte ${line.offset} of ${named} on this machine: its mark is ` ++ "removed, and the line is a gap again only if it still is one (a body blob that " ++ "has arrived since leaves none).\n"+ : `Nothing was quarantined at byte ${line.offset} of ${named} on this machine; nothing changed.\n`,+ );+ } catch (err) {+ if (!isRefusal(err)) throw err;+ return refuse(err, line.json);+ }+}diff --git a/src/cli/store.ts b/src/cli/store.tsindex 3a839575..b3e6ba9f 100644--- a/src/cli/store.ts+++ b/src/cli/store.ts@@ -10,7 +10,9 @@* in `core/store/store-init.ts`; this file parses flags and renders.* `path` reports where the store and its device-local state are* (`store-path.ts`); `device` lists the devices the store knows and- * retires one that is gone (`store-device.ts`).+ * retires one that is gone (`store-device.ts`); `partition leave` takes a+ * partition off this machine (`store-partition.ts`). `shard` quarantines one bad+ * line of a held shard on this machine, lists and releases it (`store-shard.ts`).*/import {@@ -32,11 +34,13 @@ import { claimViewWriter } from "../core/view/writer.ts";import { CliError, parseFlags } from "./argparse.ts";import { sortedReplacer } from "./helpers.ts";import { cmdStoreDevice, STORE_DEVICE_USAGE } from "./store-device.ts";+import { cmdStorePartition, STORE_PARTITION_USAGE } from "./store-partition.ts";import { cmdStorePath, STORE_PATH_USAGE } from "./store-path.ts";+import { cmdStoreShard, STORE_SHARD_USAGE } from "./store-shard.ts";const INIT_USAGE ="usage: o2b store init [--partitions shared|shared,private] [--config <file>] [--json]";-const USAGE = `${INIT_USAGE}\n${STORE_PATH_USAGE}\n${STORE_DEVICE_USAGE}`;+const USAGE = `${INIT_USAGE}\n${STORE_PATH_USAGE}\n${STORE_DEVICE_USAGE}\n${STORE_PARTITION_USAGE}\n${STORE_SHARD_USAGE}`;/** `refused` is an answer (the store exists, or is partial), not a broken argv. */const STORE_EXIT = Object.freeze({ ok: 0, refused: 1, usage: 2 } as const);@@ -55,6 +59,10 @@ export async function handleStoreSubcommand(argv: ReadonlyArray<string>): Promisreturn cmdStorePath(rest);case "device":return cmdStoreDevice(rest);+ case "partition":+ return cmdStorePartition(rest);+ case "shard":+ return cmdStoreShard(rest);default:process.stderr.write(`error: unknown store verb: ${verb}\n${USAGE}\n`);return STORE_EXIT.usage;diff --git a/src/core/brain/active.ts b/src/core/brain/active.tsindex e723b4e2..d6dad89c 100644--- a/src/core/brain/active.ts+++ b/src/core/brain/active.ts@@ -108,6 +108,11 @@ export interface RenderActiveOptions extends RegenerateActiveOptions {* reaches the shared write.*/readonly readable?: (rel: string) => boolean;+ /**+ * The reader is below local reach: the most-applied counts leave out+ * events of the store's `private` partition. Never reaches the write.+ */+ readonly withholdsPrivate?: boolean;}/** Who is reading the shared digest: the two read-time narrowings. */@@ -225,6 +230,7 @@ export function renderActive(vault: string | null, opts: RenderActiveOptions = {now,windowDays,limit,+ withholdsPrivate: opts.withholdsPrivate === true,});const body = renderBody({@@ -276,6 +282,7 @@ export function renderActiveForReader(...(opts.agentScope !== undefined ? { agentScope: opts.agentScope } : {}),...(opts.readable !== undefined ? { readable: opts.readable } : {}),...(generatedAt !== null ? { generatedAt } : {}),+ withholdsPrivate: opts.restricted === true,});}diff --git a/src/core/brain/agent-source/query.ts b/src/core/brain/agent-source/query.tsindex 519f932b..5af8df57 100644--- a/src/core/brain/agent-source/query.ts+++ b/src/core/brain/agent-source/query.ts@@ -66,11 +66,17 @@ export function queryAgentSources(// correct `total_matched` sat next to a roster that named exactly what// it withheld (context-integrity-gates, A5). An unscoped call filters// nothing, so the roster is the same array it always was.- const visible = collectAgentSourceContributions(vault).filter(- (contribution) =>- (ownerScope === null || isOwnerVisible(contribution.owner ?? null, ownerScope)) &&- (opts.view === undefined || opts.view.row(...contributionRefs(contribution))),- );+ // A contribution read from the store's `private` partition is withheld+ // below local reach whatever it names, and the mark itself never leaves.+ const withholdsPrivate = opts.view?.withholdsPrivate === true;+ const visible = collectAgentSourceContributions(vault)+ .filter(+ (contribution) =>+ !(withholdsPrivate && contribution.fromPrivatePartition === true) &&+ (ownerScope === null || isOwnerVisible(contribution.owner ?? null, ownerScope)) &&+ (opts.view === undefined || opts.view.row(...contributionRefs(contribution))),+ )+ .map(withoutPrivateMark);const availableAgents = summarizeAgentSources(visible);const availableIds = new Set(availableAgents.map((a) => a.id));const selectedAgents = normalizeAgents(@@ -172,3 +178,10 @@ function contributionRefs(}return refs;}++/** The contribution as it leaves this module: without the private mark. */+function withoutPrivateMark(contribution: AgentSourceContribution): AgentSourceContribution {+ if (contribution.fromPrivatePartition === undefined) return contribution;+ const { fromPrivatePartition: _mark, ...rest } = contribution;+ return Object.freeze(rest);+}diff --git a/src/core/brain/agent-source/types.ts b/src/core/brain/agent-source/types.tsindex 1028ea2c..37a5ad08 100644--- a/src/core/brain/agent-source/types.ts+++ b/src/core/brain/agent-source/types.ts@@ -52,6 +52,12 @@ export interface AgentSourceContribution {*/readonly origin_vault?: string;readonly data: Readonly<Record<string, unknown>>;+ /**+ * Set on a log contribution read from the store's `private` partition.+ * Internal: `queryAgentSources` withholds such a contribution below+ * local reach and strips the field from every contribution it returns.+ */+ readonly fromPrivatePartition?: true;}/**diff --git a/src/core/brain/agent-source/vault-provider.ts b/src/core/brain/agent-source/vault-provider.tsindex 720e6aab..357167f4 100644--- a/src/core/brain/agent-source/vault-provider.ts+++ b/src/core/brain/agent-source/vault-provider.ts@@ -194,6 +194,7 @@ function logContribution(entry: BrainLogEntry): AgentSourceContribution | null {event_type: entry.eventType,body: cloneLogBody(entry.body),},+ ...privateMark(entry),});}@@ -234,9 +235,15 @@ function noteWriteContribution(event_type: entry.eventType,body: cloneLogBody(entry.body),},+ ...privateMark(entry),});}+/** The mark of an event read from the store's `private` partition, or nothing. */+function privateMark(entry: BrainLogEntry): { readonly fromPrivatePartition?: true } {+ return entry.fromPrivatePartition === true ? { fromPrivatePartition: true } : {};+}+function logTopic(entry: BrainLogEntry): string | undefined {const topic = entry.body["topic"];return typeof topic === "string" && topic.trim().length > 0 ? topic : undefined;diff --git a/src/core/brain/backlinks.ts b/src/core/brain/backlinks.tsindex ed5a832e..03e6c001 100644--- a/src/core/brain/backlinks.ts+++ b/src/core/brain/backlinks.ts@@ -40,6 +40,7 @@ import { normalizeRelationTarget } from "../graph/frontmatter-relations.ts";import { buildAliasIndex } from "./link-graph/alias-index.ts";import { extractWikilinkRichBodies, parseWikilinkRich } from "./link-graph/parse-wikilink.ts";import { storeLogDays } from "./log-days.ts";+import { withoutPrivateEntries } from "./log-events-at-reach.ts";import { listLogShardFiles, readLogDay } from "./log-jsonl.ts";import { brainDirs } from "./paths.ts";import { normalizeDerivedKeys } from "./preference.ts";@@ -156,7 +157,11 @@ export interface BacklinkIndex extends ReadonlyMap<string, ReadonlyArray<Backlin* report would leak exactly what the filter withheld. Absent (every* existing call site) means no ownership filtering at all.*/-export function buildBacklinkIndex(vault: string, ownerScope?: string | null): BacklinkIndex {+export function buildBacklinkIndex(+ vault: string,+ ownerScope?: string | null,+ opts: { readonly withholdsPrivate?: boolean } = {},+): BacklinkIndex {const dirs = brainDirs(vault);const view = ownerScopeView(vault, ownerScope ?? null);const aliasIndex = buildAliasIndex(vault);@@ -221,7 +226,7 @@ export function buildBacklinkIndex(vault: string, ownerScope?: string | null): BcollectSignals(pagesOf(dirs.inbox), push, fail);collectSignals(pagesOf(dirs.processed), push, fail);collectSignals(pagesOf(dirs.archived), push, fail);- collectLog(vault, dirs.log, push, fail);+ collectLog(vault, dirs.log, push, fail, opts.withholdsPrivate === true);// Freeze each entry's array so downstream callers can't mutate the// shared index.@@ -379,6 +384,7 @@ function collectLog(dir: string,push: (target: string, ref: BacklinkRef) => void,fail: (row: BacklinkParseFailure) => void,+ withholdsPrivate: boolean,): void {// Store-only (task 2.10): the log is the store's `log-event` records.const stored = storeLogDays(vault);@@ -402,7 +408,9 @@ function collectLog(fail({ source, sourceKind: LOG_READ_KIND, reason: failureReason(err, dir) });continue;}- for (const e of entries) {+ // Below local reach an event of the store's `private` partition+ // references nothing (`withholdsPrivate`).+ for (const e of withoutPrivateEntries(entries, withholdsPrivate)) {// Walk every payload value, pulling wikilink targets out of// scalar fields and string arrays alike. We don't whitelist// which fields can carry references — the payload key becomesdiff --git a/src/core/brain/compiled-digest-path.ts b/src/core/brain/compiled-digest-path.tsindex 7b56f0c1..9b8ac00e 100644--- a/src/core/brain/compiled-digest-path.ts+++ b/src/core/brain/compiled-digest-path.ts@@ -37,6 +37,13 @@ import { noVaultRoot } from "../platform-dirs.ts";export type CompiledDigest = "active" | "lessons";const DIGESTS_DIR = "digests";++/**+ * The folder of every digest cache of the store `storeId`.+ */+export function digestCacheRoot(storeId: string): string {+ return join(deviceStateDirFor(storeId), DIGESTS_DIR);+}const VAULT_KEY_CHARS = 16;/** The cache directory of the digests of a machine with no vault. */const NO_VAULT_DIGEST_KEY = "no-vault";diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.tsindex f4c23e5f..539f2ddb 100644--- a/src/core/brain/diagnostics.ts+++ b/src/core/brain/diagnostics.ts@@ -64,6 +64,7 @@ import {import { ATTESTATION_CHECK_CODE } from "./doctor/attestation-check.ts";import { RECEIPT_CONTENT_DRIFT_CODE } from "./doctor/receipt-content-drift-check.ts";import { GIT_TRACKED_DERIVED_STATE_CODE } from "./doctor/git-tracked-state-check.ts";+import { STORE_SHARD_CORRUPT_LINE_CODE } from "./doctor/store-gap-check.ts";import { scanDanglingWorkruns, WORKRUN_PHASE } from "./dream-workrun.ts";import { extractWikilinkRichBodies, parseWikilinkRich } from "./link-graph/parse-wikilink.ts";import { LINT_CONSOLIDATE_KIND } from "./lint-consolidate.ts";@@ -1050,6 +1051,14 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b brain doctor",autoRepairable: false,},+ {+ // The structural form; the finding's message carries the concrete+ // command with the shard as `<partition>/<shard>` and its offset.+ code: STORE_SHARD_CORRUPT_LINE_CODE,+ issueClass: "a bad line in the middle of a held store shard",+ nextCommand: "o2b store shard quarantine <partition-shard> --offset <n>",+ autoRepairable: false,+ },// --- Skill-accept transaction refusals (no-dead-ends, phase 3) ---// Both are states the accept transaction itself can leave an// operator in, and both used to refuse without naming a way out.@@ -1553,7 +1562,7 @@ export interface PlanRepairOptions {* the same findings, each with its next-command hint.*/export function planRepair(vault: string, opts: PlanRepairOptions = {}): RepairPlan {- const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable, opts.dbPath);+ const issues = opts.issues ?? collectDoctorIssues(vault, opts.readable, opts.dbPath, false);const fixes: RepairItem[] = [];for (const fixer of FIXERS) {@@ -1590,10 +1599,12 @@ function collectDoctorIssues(vault: string,readable: ((rel: string) => boolean) | undefined,dbPath: string | undefined,+ withholdsPrivate: boolean,): ReadonlyArray<DoctorIssue> {const doctor = runDoctor(vault, {...(readable !== undefined ? { readable } : {}),...(dbPath !== undefined ? { dbPath } : {}),+ withholdsPrivate,});return [...doctor.errors, ...doctor.warnings];}@@ -1810,8 +1821,9 @@ export function applyRepair(vault: string, opts: ApplyRepairOptions): RepairOutc...(opts.dbPath !== undefined ? { dbPath: opts.dbPath } : {}),}): planRepair(vault, {- issues: view.keep(collectDoctorIssues(vault, opts.readable, opts.dbPath), (i) =>- doctorIssueRefs(vault, i),+ issues: view.keep(+ collectDoctorIssues(vault, opts.readable, opts.dbPath, view.withholdsPrivate === true),+ (i) => doctorIssueRefs(vault, i),),});const fixes = view.keep(plan.fixes, repairRefs);diff --git a/src/core/brain/digest.ts b/src/core/brain/digest.tsindex 5a29712d..6882ce31 100644--- a/src/core/brain/digest.ts+++ b/src/core/brain/digest.ts@@ -583,7 +583,9 @@ function collectDigestData(// Built once and shared with `computeConnectionHealth` below: both// read the same vault state within this digest run, so one// full-history backlink scan replaces two.- const backlinkIndex = buildBacklinkIndex(vault);+ const backlinkIndex = buildBacklinkIndex(vault, undefined, {+ withholdsPrivate: refs.withholdsPrivate === true,+ });const inbound = (id: string): number => visibleBacklinkCount(backlinkIndex, id, refs);const top_referenced = pickTopReferenced(inbound, preferences);// `merge_suggestions` reflects current vault state, not windowed@@ -637,6 +639,7 @@ function collectDigestData(now: until,windowDays: mostAppliedWindowDays,limit: mostAppliedLimit,+ withholdsPrivate: refs.withholdsPrivate === true,}).map((e) => ({id: e.preference.id,principle: e.preference.principle,@@ -684,6 +687,7 @@ function collectDigestData(collectMaintenanceActions(vault, {onError: (e) => collectErrors.push(e),visible: scopedRefs.visible,+ withholdsPrivate: refs.withholdsPrivate === true,}),(action) => [action.target],),diff --git a/src/core/brain/doctor-exits.ts b/src/core/brain/doctor-exits.tsindex 4b55ae5a..6c40764b 100644--- a/src/core/brain/doctor-exits.ts+++ b/src/core/brain/doctor-exits.ts@@ -158,6 +158,26 @@ const EXCLUSIONS: ReadonlyArray<DoctorExitExclusion> = ["the held bytes are the unfinished last line of another device's shard; only that device's " +"next append or a completed sync closes it, and no command on this machine can",},+ {+ code: "store-shard-newer-format",+ reason:+ "the shard holds a record of a format this release does not know; upgrading Open Second " ++ "Brain on this machine reads it, and setting the line aside would hide a valid record",+ },+ {+ code: "store-shard-blob-pending",+ reason:+ "a record's body blob has not arrived yet; a sync tool may still deliver it, and setting " ++ "the record aside is allowed only once it is old enough and never for a settings shard, " ++ "which the finding itself says per record",+ },+ {+ code: "store-shard-line-too-long",+ reason:+ "the bad line runs past the longest line a quarantine can name, so it cannot be set aside " ++ "on this machine; only repairing or removing it in the shard of the device that wrote it " ++ "resolves it",+ },{code: "store-shard-unreadable",reason:diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.tsindex 3be30986..5f2f91df 100644--- a/src/core/brain/doctor.ts+++ b/src/core/brain/doctor.ts@@ -612,6 +612,7 @@ function resolveContext(configPath: opts.configPath,...(opts.readable !== undefined ? { readable: opts.readable } : {}),...(opts.reach !== undefined ? { reach: opts.reach } : {}),+ ...(opts.withholdsPrivate === true ? { withholdsPrivate: true } : {}),knownBasenames: collectAllBasenames(vault, {site: CONTEXT_SITE,consequence:diff --git a/src/core/brain/doctor/check.ts b/src/core/brain/doctor/check.tsindex 9ac6f331..43c4a8fa 100644--- a/src/core/brain/doctor/check.ts+++ b/src/core/brain/doctor/check.ts@@ -59,6 +59,13 @@ export interface DoctorCheckContext {* here; it never leaves this absent.*/readonly reach?: TransportReach;+ /**+ * The caller is below local reach: the broken-backlinks check ignores+ * references made only by log events of the store's `private` partition,+ * and the store check reports nothing about that partition. Absent reads+ * as false (the operator's own shell).+ */+ readonly withholdsPrivate?: boolean;/** Every valid wikilink target inside `Brain/`, keyed by basename. */readonly knownBasenames: ReadonlySet<string>;/**@@ -95,7 +102,7 @@ export interface DoctorFindings {*/export interface StoreSideCheckContext extends Pick<DoctorCheckContext,- "now" | "dbPath" | "configPath" | "readable" | "reach"+ "now" | "dbPath" | "configPath" | "readable" | "reach" | "withholdsPrivate"> {readonly vault: string | null;}diff --git a/src/core/brain/doctor/link-checks.ts b/src/core/brain/doctor/link-checks.tsindex a98967e7..bb625fde 100644--- a/src/core/brain/doctor/link-checks.ts+++ b/src/core/brain/doctor/link-checks.ts@@ -23,12 +23,12 @@ import { reportSweptFailure, sweptFailurePath, SWEEP_ORIGIN } from "./unreadable*/export const brokenBacklinkCheck: DoctorCheck = {failSoft: false,- run({ vault, knownBasenames }, { issues, uncertain }) {+ run({ vault, knownBasenames, withholdsPrivate }, { issues, uncertain }) {// Only attempt the check when there's something to scan — an empty// Brain layer naturally has no backlinks, and `buildBacklinkIndex`// would already return an empty map, but we save the parse pass.if (knownBasenames.size === 0) return;- const index = buildIndex(vault, uncertain);+ const index = buildIndex(vault, uncertain, withholdsPrivate === true);if (index === null) return;for (const [target, refs] of index) {// We only flag references whose target *should* live in this@@ -72,9 +72,10 @@ const BACKLINK_SITE = "brain.doctor.brokenBacklinks";function buildIndex(vault: string,uncertain: DoctorUncertainEntry[],+ withholdsPrivate: boolean,): ReturnType<typeof buildBacklinkIndex> | null {try {- return buildBacklinkIndex(vault);+ return buildBacklinkIndex(vault, undefined, { withholdsPrivate });} catch (err) {reportSweptFailure(sweptFailurePath(err, brainDirs(vault).brain),diff --git a/src/core/brain/doctor/log-checks.ts b/src/core/brain/doctor/log-checks.tsindex eb324f4c..8e8dacf3 100644--- a/src/core/brain/doctor/log-checks.ts+++ b/src/core/brain/doctor/log-checks.ts@@ -9,12 +9,13 @@import type { DegradationNotice } from "../../integrity/degradation.ts";import { listVaultBasenames } from "../../vault.ts";+import { withoutPrivateEntries } from "../log-events-at-reach.ts";import { listLogMarkdownFiles } from "../log-jsonl.ts";import { parseLogDayFile, type BrainLogEntry } from "../log.ts";import { brainDirs } from "../paths.ts";import { BRAIN_LOG_EVENT_KIND } from "../types.ts";import { parseArtifactRef } from "../wikilink.ts";-import type { DoctorCheck } from "./check.ts";+import type { DoctorCheck, DoctorCheckContext } from "./check.ts";import type { DoctorUncertainEntry } from "./report.ts";import { forwardSweepNotices, reportSweptFailure, SWEEP_ORIGIN } from "./unreadable-path.ts";@@ -89,6 +90,28 @@ function evidencePreferenceSources(e: BrainLogEntry): { readonly sources?: Readoreturn target ? { sources: [target] } : {};}+/**+ * The `apply-evidence` events an evidence lint may quote: every one for+ * the operator's own shell, and none of the store's `private` partition+ * for a caller that withholds it. Both lints below put the event's+ * timestamp and artifact name into their message, so a private event+ * quoted to a remote caller would disclose private activity. Neither+ * lint has a fixer, so leaving a private event out hides a finding from+ * that caller only and changes no repair.+ */+function quotableEvidence(+ logs: DoctorCheckContext["logs"],+ withholdsPrivate: boolean | undefined,+): BrainLogEntry[] {+ const out: BrainLogEntry[] = [];+ for (const { entries } of logs) {+ for (const e of withoutPrivateEntries(entries, withholdsPrivate === true)) {+ if (e.eventType === BRAIN_LOG_EVENT_KIND.applyEvidence) out.push(e);+ }+ }+ return out;+}+/*** `malformed-evidence-range`: walks every `apply-evidence` event,* runs the artifact wikilink through {@link parseArtifactRef}, and@@ -97,24 +120,21 @@ function evidencePreferenceSources(e: BrainLogEntry): { readonly sources?: Reado*/export const evidenceRangeCheck: DoctorCheck = {failSoft: true,- run({ logs }, { issues }) {- for (const { entries } of logs) {- for (const e of entries) {- if (e.eventType !== BRAIN_LOG_EVENT_KIND.applyEvidence) continue;- const artifact = e.body["artifact"];- if (typeof artifact !== "string") continue;- const parsed = parseArtifactRef(artifact);- if (parsed.malformedRange) {- issues.push({- severity: "warning",- code: "malformed-evidence-range",- ...evidencePreferenceSources(e),- message:- `apply-evidence at ${e.timestamp} references artifact ${parsed.raw}` +- ` with malformed range '${parsed.rangeText}'.` +- " Use `[[file:N-N]]` (inclusive, 1-based) or `[[file:N]]`.",- });- }+ run({ logs, withholdsPrivate }, { issues }) {+ for (const e of quotableEvidence(logs, withholdsPrivate)) {+ const artifact = e.body["artifact"];+ if (typeof artifact !== "string") continue;+ const parsed = parseArtifactRef(artifact);+ if (parsed.malformedRange) {+ issues.push({+ severity: "warning",+ code: "malformed-evidence-range",+ ...evidencePreferenceSources(e),+ message:+ `apply-evidence at ${e.timestamp} references artifact ${parsed.raw}` ++ ` with malformed range '${parsed.rangeText}'.` ++ " Use `[[file:N-N]]` (inclusive, 1-based) or `[[file:N]]`.",+ });}}},@@ -140,7 +160,7 @@ const ORPHAN_EVIDENCE_SITE = "brain.doctor.orphanEvidence";*/export const orphanEvidenceCheck: DoctorCheck = {failSoft: true,- run({ vault, logs }, { issues, uncertain }) {+ run({ vault, logs, withholdsPrivate }, { issues, uncertain }) {const notices: DegradationNotice[] = [];const basenames: ReadonlySet<string> = listVaultBasenames(vault, {notices,@@ -153,23 +173,20 @@ export const orphanEvidenceCheck: DoctorCheck = {"references against, so a reference that resolves under it reads as orphaned here",uncertain,});- for (const { entries } of logs) {- for (const e of entries) {- if (e.eventType !== BRAIN_LOG_EVENT_KIND.applyEvidence) continue;- const artifact = e.body["artifact"];- if (typeof artifact !== "string") continue;- const target = parseArtifactRef(artifact).target;- if (!target) continue;- if (basenames.has(target)) continue;- issues.push({- severity: "warning",- code: "orphan-evidence",- ...evidencePreferenceSources(e),- message:- `apply-evidence at ${e.timestamp} references artifact [[${target}]]` +- " but no file with that basename exists in the vault.",- });- }+ for (const e of quotableEvidence(logs, withholdsPrivate)) {+ const artifact = e.body["artifact"];+ if (typeof artifact !== "string") continue;+ const target = parseArtifactRef(artifact).target;+ if (!target) continue;+ if (basenames.has(target)) continue;+ issues.push({+ severity: "warning",+ code: "orphan-evidence",+ ...evidencePreferenceSources(e),+ message:+ `apply-evidence at ${e.timestamp} references artifact [[${target}]]` ++ " but no file with that basename exists in the vault.",+ });}},};diff --git a/src/core/brain/doctor/report.ts b/src/core/brain/doctor/report.tsindex cc2d9d4c..c20b08be 100644--- a/src/core/brain/doctor/report.ts+++ b/src/core/brain/doctor/report.ts@@ -53,6 +53,11 @@ export interface RunDoctorOptions {* (the operator's own shell).*/readonly reach?: TransportReach;+ /**+ * The caller is below local reach. Passed through to+ * {@link DoctorCheckContext.withholdsPrivate}. Omitted reads as false.+ */+ readonly withholdsPrivate?: boolean;/*** Optional precomputed dream summary (v0.10.16). When supplied,* the doctor runs the verification-delta helper and folds thediff --git a/src/core/brain/doctor/store-check.ts b/src/core/brain/doctor/store-check.tsindex 62dd087c..92cb0801 100644--- a/src/core/brain/doctor/store-check.ts+++ b/src/core/brain/doctor/store-check.ts@@ -33,16 +33,34 @@* the device retirements may be in it;* - a shard whose last bytes have not verified for an hour: the reader* holds that tail back, so the record in it is read nowhere;+ * - a shard read only in part, one finding per gap (`store-gap-check.ts`):+ * a corrupt middle line and an old pending blob as errors, a+ * newer-format line and a young pending blob as warnings;* - what a synced peer can deliver (`store-trust-check.ts`): a record id* in both partitions, and store paths other local accounts can read.*+ * Below local reach (a caller with `withholdsPrivate`, or a `reach` other+ * than local) every finding is computed without the `private` partition,+ * so none names a private shard, offset, record id, device time or count,+ * and a lost `private` identity file is reported without naming the+ * partition (`storeAtReach`, `lostPartitionFinding`); a store that does not+ * open for another reason is reported without the error text, which can+ * name a partition's file or the store ids (`storeOpenFailureDetail`). The write counters+ * follow the same rule: only failures known to concern only `shared` are+ * shown, and the skipped writes without their count (`dualWriteFindings`).+ ** Read-only: it never creates the store or refreshes its format document.*/import { statSync } from "node:fs";import { join } from "node:path";-import { STORE_PARTITIONS, type StorePartition } from "../../store/constants.ts";+import { TRANSPORT_REACH } from "../../graph/transport-reach.ts";+import {+ STORE_IDENTITY_FILE,+ STORE_PARTITIONS,+ type StorePartition,+} from "../../store/constants.ts";import type { StoreRecord } from "../../store/envelope.ts";import { compareHlc, formatHlc, HLC_MAX_DRIFT_MS, parseHlc, type Hlc } from "../../store/hlc.ts";import { compareCodeUnits } from "../../store/compare.ts";@@ -79,6 +97,13 @@ import {} from "./store-trust-check.ts";import type { DoctorFindings, StoreSideCheckContext, StoreSideDoctorCheck } from "./check.ts";import { returnedRetiredIssues } from "./device-retire-check.ts";+import {+ STORE_SHARD_UNREADABLE_CODE,+ storeGapFindings,+ type PartialShard,+} from "./store-gap-check.ts";++export { STORE_SHARD_UNREADABLE_CODE };/** Removed content still awaits compaction in the shards. */export const STORE_REMOVED_AWAITING_COMPACTION_CODE = "store-removed-awaiting-compaction";@@ -101,8 +126,6 @@ export const STORE_DIGEST_SILENT_CODE = "store-digest-silent";export const STORE_RECORD_ID_CONFLICT_CODE = "store-record-id-conflict";/** A partition this machine holds could not be read. */export const STORE_PARTITION_UNREADABLE_CODE = "store-partition-unreadable";-/** One shard of a held partition could not be read; the rest were checked. */-export const STORE_SHARD_UNREADABLE_CODE = "store-shard-unreadable";/** A shard's last bytes have not verified for a long time, so its last record is read nowhere. */export const STORE_SHARD_TAIL_HELD_CODE = "store-shard-tail-held";/** No store exists at the root; `o2b store init` creates one. */@@ -132,26 +155,62 @@ function warn(out: DoctorFindings, code: string, message: string, path?: string)} satisfies DoctorIssue);}-function dualWriteFindings(out: DoctorFindings): void {+/**+ * The failure part of a write status: below local reach only the+ * failures known to concern only `shared` (the `shared_*` counters);+ * otherwise every failure.+ */+function failuresAtReach(+ status: {+ readonly failures: number;+ readonly last_error: string | null;+ readonly last_failure_at: string | null;+ readonly shared_failures: number;+ readonly shared_last_error: string | null;+ readonly shared_last_failure_at: string | null;+ },+ atReach: boolean,+): { count: number; error: string | null; at: string | null } {+ return atReach+ ? {+ count: status.shared_failures,+ error: status.shared_last_error,+ at: status.shared_last_failure_at,+ }+ : { count: status.failures, error: status.last_error, at: status.last_failure_at };+}++/**+ * The dual-write counters. Below local reach (`atReach`) a failure that+ * may concern the `private` partition is not shown at all (no count, time+ * or error text, which can name a private shard): only the failures known+ * to concern only `shared` are. The skipped writes are reported without+ * their count, since the skipped writes include private ones.+ */+function dualWriteFindings(out: DoctorFindings, atReach: boolean): void {const path = dualWriteStatusPath();const status = readDualWriteStatus(path);- if (status.failures > 0) {+ const failed = failuresAtReach(status, atReach);+ if (failed.count > 0) {warn(out,STORE_DUAL_WRITE_FAILURES_CODE,- `${status.failures} brain store dual write(s) failed on this machine` +- (status.last_failure_at !== null ? `, the last at ${status.last_failure_at}` : "") +- (status.last_error !== null ? ` (${status.last_error})` : "") ++ `${failed.count} brain store dual write(s) ${atReach ? "to the shared partition " : ""}` ++ "failed on this machine" ++ (failed.at !== null ? `, the last at ${failed.at}` : "") ++ (failed.error !== null ? ` (${failed.error})` : "") +"; the legacy writes were kept, so the store is missing those records. " +"Fix the cause, then delete the counter file to reset it",path,);}+ const counted = (n: number): string =>+ atReach ? "brain store dual writes were" : `${n} brain store dual write(s) were`;if (status.skipped_no_device > 0) {warn(out,STORE_DUAL_WRITE_SKIPPED_CODE,- `${status.skipped_no_device} brain store dual write(s) were skipped because this machine ` ++ `${counted(status.skipped_no_device)} skipped because this machine ` +"has no device id; set device_id in the machine config (and leave O2B_DEVICE_ID unset " +"or set it to a real id) so the store can name this device's shards",path,@@ -161,7 +220,7 @@ function dualWriteFindings(out: DoctorFindings): void {warn(out,STORE_DUAL_WRITE_NO_STORE_CODE,- `${status.skipped_no_store} brain store dual write(s) were skipped because no store ` ++ `${counted(status.skipped_no_store)} skipped because no store ` +"existed; the legacy writes were kept, so the store is missing those records. " +"Run `o2b store init` if it has not been run, then delete the counter file to reset it",path,@@ -169,16 +228,18 @@ function dualWriteFindings(out: DoctorFindings): void {}}-function failOpenWriteFindings(out: DoctorFindings): void {+/** The fail-open counters, under the same rule at reach as {@link dualWriteFindings}. */+function failOpenWriteFindings(out: DoctorFindings, atReach: boolean): void {const path = storeWriteStatusPath();- const status = readStoreWriteStatus(path);- if (status.failures === 0) return;+ const failed = failuresAtReach(readStoreWriteStatus(path), atReach);+ if (failed.count === 0) return;warn(out,STORE_WRITE_FAIL_OPEN_CODE,- `${status.failures} fail-open brain store write(s) on this machine did not land` +- (status.last_failure_at !== null ? `, the last at ${status.last_failure_at}` : "") +- (status.last_error !== null ? ` (${status.last_error})` : "") ++ `${failed.count} fail-open brain store write(s) ${atReach ? "to the shared partition " : ""}` ++ "on this machine did not land" ++ (failed.at !== null ? `, the last at ${failed.at}` : "") ++ (failed.error !== null ? ` (${failed.error})` : "") +"; the hooks and telemetry that made them went on, so the store is missing those " +"records. Fix the cause, then delete the counter file to reset it",path,@@ -260,9 +321,14 @@ function digestMessage(finding: DigestFinding): string {function readHeldPartitions(store: StoreInspection,out: DoctorFindings,-): { shards: Map<StorePartition, PartitionShard[]>; complete: Set<StorePartition> } {+): {+ shards: Map<StorePartition, PartitionShard[]>;+ complete: Set<StorePartition>;+ partial: PartialShard[];+} {const shards = new Map<StorePartition, PartitionShard[]>();const complete = new Set<StorePartition>();+ const partial: PartialShard[] = [];for (const partition of STORE_PARTITIONS) {if (!store.partitions.has(partition)) continue;const dir = partitionDir(store.root, partition);@@ -281,6 +347,7 @@ function readHeldPartitions(readPartition(dir, {onUnreadable: (s) => {if (s.incomplete !== true) unreadable.push(s);+ else partial.push({ partition, rel: s.rel });},}),);@@ -308,7 +375,7 @@ function readHeldPartitions(}if (unreadable.length === 0) complete.add(partition);}- return { shards, complete };+ return { shards, complete, partial };}/**@@ -348,13 +415,45 @@ function heldTailFindings(}}+/**+ * Whether the caller is below local reach: a wire surface that names its+ * reach (`brain_doctor`) or one that only marks the caller as withheld from+ * private data (`brain_brief view=operator`, `brain_status`, the repair+ * pass). Neither set reads as the operator's own shell.+ */+function withholdsPrivate(ctx: StoreSideCheckContext): boolean {+ return (+ ctx.withholdsPrivate === true ||+ (ctx.reach !== undefined && ctx.reach !== TRANSPORT_REACH.local)+ );+}++/**+ * The store as a caller below local reach may see it: without the `private`+ * partition. Every finding below is computed from this one view, so none of+ * them names a private shard, offset, record id, device time or count: no+ * unreadable shard, gap or held tail, no record id conflict, clock-ahead or+ * digest finding, no cross-partition id, loose mode or reserved name, and+ * the removal backlog counts only `shared`. The caller's `ok`, counts and+ * trust verdict read as if the `private` partition were not held here.+ */+function storeAtReach(store: StoreInspection, ctx: StoreSideCheckContext): StoreInspection {+ if (!withholdsPrivate(ctx) || !store.partitions.has("private")) return store;+ return {+ ...store,+ partitions: new Set([...store.partitions].filter((partition) => partition !== "private")),+ };+}+function storeFindings(- store: StoreInspection,+ held: StoreInspection,ctx: StoreSideCheckContext,out: DoctorFindings,): void {const nowMs = ctx.now.getTime();- const { shards, complete } = readHeldPartitions(store, out);+ const store = storeAtReach(held, ctx);+ const { shards, complete, partial } = readHeldPartitions(store, out);+ storeGapFindings(store, partial, nowMs, out);// Digests are compared only over partitions read in full: a cut with a// shard missing would read as a divergence that is not there.const digestShards = new Map([...shards].filter(([partition]) => complete.has(partition)));@@ -431,6 +530,51 @@ function removalBacklogFinding();}+/**+ * A held partition lost its identity file, so the store is not opened.+ * Below local reach the finding follows the same rule as every other store+ * finding and never names the `private` partition. It is still reported,+ * with the partitions it may name: the store that does not open fails+ * every protection closed for `shared` as well, so `ok` and the trust+ * verdict must not read as healthy.+ */+function lostPartitionFinding(+ err: HeldPartitionLostError,+ ctx: StoreSideCheckContext,+ out: DoctorFindings,+): void {+ let message = err.message;+ if (withholdsPrivate(ctx) && err.partitions.includes("private")) {+ const both = err.partitions.includes("shared");+ const which = both ? "the shared partition and another partition" : "a partition";+ message =+ `${which} of the brain store at ${err.root}, which this machine has held, no longer ` ++ `${both ? "show" : "shows"} ${STORE_IDENTITY_FILE}, so the store is not opened and every protection fails ` ++ "closed. Run `o2b brain doctor` on this machine for the partitions and the remedy";+ }+ out.issues.push({+ severity: "error",+ code: STORE_PARTITION_LOST_CODE,+ path: err.root,+ message,+ } satisfies DoctorIssue);+}++/**+ * The detail of a store that did not open for a reason other than a lost+ * partition. The error text can name a partition's file (a malformed+ * `private/store.json`) or the store ids that disagree, so below local+ * reach the finding keeps its code and the store root and says only that+ * the store did not open; the operator's own doctor reads the cause.+ */+function storeOpenFailureDetail(err: unknown, root: string, ctx: StoreSideCheckContext): string {+ if (!withholdsPrivate(ctx)) return err instanceof Error ? err.message : String(err);+ return (+ `the brain store at ${root} could not be opened, so every protection fails closed. ` ++ "Run `o2b brain doctor` on this machine for the cause and the remedy"+ );+}+/** Whether this machine uses the store: the flag is on or a root is configured. */function storeInUse(ctx: StoreSideCheckContext,@@ -467,8 +611,10 @@ export const storeCheck: StoreSideDoctorCheck = {}if (!storeInUse(ctx, root, rootError)) return;- dualWriteFindings(out);- failOpenWriteFindings(out);+ // Read before the store is: a failed write into `private` is withheld+ // below local reach as every other private finding is.+ dualWriteFindings(out, withholdsPrivate(ctx));+ failOpenWriteFindings(out, withholdsPrivate(ctx));if (rootError !== null || root === null) {warn(out,@@ -482,20 +628,10 @@ export const storeCheck: StoreSideDoctorCheck = {store = inspectStore(root.root);} catch (err) {if (err instanceof HeldPartitionLostError) {- out.issues.push({- severity: "error",- code: STORE_PARTITION_LOST_CODE,- path: root.root,- message: err.message,- } satisfies DoctorIssue);+ lostPartitionFinding(err, ctx, out);return;}- warn(- out,- STORE_ROOT_INVALID_CODE,- err instanceof Error ? err.message : String(err),- root.root,- );+ warn(out, STORE_ROOT_INVALID_CODE, storeOpenFailureDetail(err, root.root, ctx), root.root);return;}if (store === null) {diff --git a/src/core/brain/doctor/store-gap-check.ts b/src/core/brain/doctor/store-gap-check.tsnew file mode 100644index 00000000..10005c44--- /dev/null+++ b/src/core/brain/doctor/store-gap-check.ts@@ -0,0 +1,222 @@+/**+ * The doctor's findings for a shard read only in part (change+ * store-hardening, design D5; spec `store-shard-gaps`).+ *+ * A held shard with a gap fails every protection read closed on this+ * machine. For each such shard (the ones the partition read reports with+ * `incomplete`), the shard is read again and one finding is reported per+ * gap, path `<partition>/<shard>`:+ *+ * - `store-shard-corrupt-line`, error: a bad line in the middle, with the+ * offset, what fails closed, and the command that quarantines it on this+ * machine;+ * - `store-shard-line-too-long`, error: a bad line that runs past the+ * quarantine hash cap (the probe `probeLineHashAt` answers `too-long`),+ * so the quarantine command would always refuse: no command is named,+ * only repair at the line's source;+ * - `store-shard-newer-format`, warning: a record of a format this release+ * does not know; upgrading reads it, so no quarantine is offered;+ * - `store-shard-blob-pending`, warning while the record is younger than+ * {@link PENDING_BLOB_QUARANTINE_AGE_MS} (a sync tool may still deliver+ * the blob), error from then on, with the quarantine command once it is+ * allowed and never for a `settings` shard; an error with no command when+ * the record's clock stamp does not parse as a time (its age is unknown,+ * and the current time is never put in its place).+ *+ * A line torn by a crash and closed by the next append, and a line this+ * machine quarantined, are no gap and give no finding. A shard removed+ * since the partition read (`ENOENT` on the second read) is skipped: no gap+ * is left. Any other failure of the second read (a lock, a permission, a+ * link) is a `store-shard-unreadable` error naming the shard and the error+ * code. Any other diagnostic (a shard swapped for a link, an offset past+ * its end) is a `store-shard-unreadable` error too, a code whose next step+ * names no command, since a quarantine cannot release it.+ *+ * Below local reach the store check hands this module no shard of the+ * `private` partition (`store-check.ts`, `storeAtReach`): its path,+ * offsets and record times are private metadata, and a remote caller's+ * `ok`, counts and trust verdict read as if the gap were not there.+ */++import { join } from "node:path";++import type { StorePartition } from "../../store/constants.ts";+import { readShard } from "../../store/reader.ts";+import {+ PENDING_BLOB_QUARANTINE_AGE_MS,+ PENDING_BLOB_QUARANTINE_AGE_TEXT,+ pendingRecordWrittenMs,+ probeLineHashAt,+} from "../../store/shard-quarantine.ts";+import { partitionDir } from "../../store/store-paths.ts";+import type { StoreInspection } from "../../store/store-open.ts";+import type { DoctorIssue } from "../types.ts";+import type { DoctorFindings } from "./check.ts";++/** One shard of a held partition could not be read; the rest were checked. */+export const STORE_SHARD_UNREADABLE_CODE = "store-shard-unreadable";+/** A bad line in the middle of a held shard. */+export const STORE_SHARD_CORRUPT_LINE_CODE = "store-shard-corrupt-line";+/** A record of a newer format in a held shard. */+export const STORE_SHARD_NEWER_FORMAT_CODE = "store-shard-newer-format";+/** A record whose body blob has not arrived. */+export const STORE_SHARD_BLOB_PENDING_CODE = "store-shard-blob-pending";+/**+ * A bad line too long to hash (past `QUARANTINE_HASH_CAP_BYTES`): no+ * quarantine can name it, so the code registers no quarantine command.+ */+export const STORE_SHARD_LINE_TOO_LONG_CODE = "store-shard-line-too-long";++/** The error code of a failed read, or the error's name when it carries none. */+function errorCode(err: unknown): string {+ if (typeof err === "object" && err !== null) {+ const code = (err as { code?: unknown }).code;+ if (typeof code === "string" && code !== "") return code;+ if (err instanceof Error) return err.name;+ }+ return String(err);+}++/** A held shard the partition read reported as read only in part. */+export interface PartialShard {+ readonly partition: StorePartition;+ /** Partition-relative path, `/`-separated. */+ readonly rel: string;+}++const FAILS_CLOSED =+ "Until it is resolved, every read that decides a protection fails closed on this machine: " ++ "capture, settings reads and writes, lane privacy, exact state, attestations, the OKF " ++ "export and the migration";++const quarantineCommand = (shard: string, offset: number): string =>+ `o2b store shard quarantine ${shard} --offset ${offset}`;++/** What to do about a pending blob of the given age and shard. */+function pendingRemedy(old: boolean, settings: boolean, shard: string, offset: number): string {+ if (!old) {+ return (+ "A sync tool may still deliver it; it is not quarantined before it is " ++ `${PENDING_BLOB_QUARANTINE_AGE_TEXT} old`+ );+ }+ if (settings) {+ return "A settings record is never quarantined: deliver the blob from the machine that wrote it";+ }+ return `When the blob is lost, skip the record on this machine with \`${quarantineCommand(shard, offset)}\``;+}++/**+ * Report one finding per gap of each of `partial`'s shards. Below local+ * reach the caller passes no shard of the `private` partition+ * (`store-check.ts`, `storeAtReach`).+ */+export function storeGapFindings(+ store: StoreInspection,+ partial: ReadonlyArray<PartialShard>,+ nowMs: number,+ out: DoctorFindings,+): void {+ for (const { partition, rel } of partial) {+ const partitionRoot = partitionDir(store.root, partition);+ const shard = `${partition}/${rel}`;+ const shardPath = join(partitionRoot, ...rel.split("/"));+ let read: ReturnType<typeof readShard>;+ try {+ read = readShard(shardPath, { partitionRoot });+ } catch (err) {+ // Removed since the partition read: there is no gap left to name.+ if (errorCode(err) === "ENOENT") continue;+ out.issues.push({+ severity: "error",+ code: STORE_SHARD_UNREADABLE_CODE,+ path: shard,+ message:+ `the shard ${shard}, read only in part, could not be read again (${errorCode(err)}), ` ++ `for example while a sync client holds it, so its gaps are not named here. ${FAILS_CLOSED}`,+ } satisfies DoctorIssue);+ continue;+ }+ for (const d of read.diagnostics) {+ if (d.torn === true || d.quarantined === true) continue;+ if (d.kind === "unknown-version") {+ out.issues.push({+ severity: "warning",+ code: STORE_SHARD_NEWER_FORMAT_CODE,+ path: shard,+ message:+ `the record at byte ${d.offset} of the shard ${shard} is of a newer format ` ++ `(${d.reason}); upgrade Open Second Brain on this machine to read it. ${FAILS_CLOSED}`,+ } satisfies DoctorIssue);+ continue;+ }+ if (d.kind !== "corrupt-line") {+ // A shard swapped for a link, or cut below the offset, since the+ // partition read: no line a quarantine could release, so the code+ // is one whose next step names no quarantine command.+ out.issues.push({+ severity: "error",+ code: STORE_SHARD_UNREADABLE_CODE,+ path: shard,+ message:+ `the shard ${shard} could not be read from byte ${d.offset} (${d.reason}). ` ++ `${FAILS_CLOSED}`,+ } satisfies DoctorIssue);+ continue;+ }+ const probe = probeLineHashAt(shardPath, d.offset, partitionRoot);+ if ("failure" in probe && probe.failure === "too-long") {+ // The verb hashes the line before it records it and refuses one past+ // the hash cap (`line-too-long`): naming it would be a dead end.+ out.issues.push({+ severity: "error",+ code: STORE_SHARD_LINE_TOO_LONG_CODE,+ path: shard,+ message:+ `the line at byte ${d.offset} of the shard ${shard} is not a valid record ` ++ `(${d.reason}). ${FAILS_CLOSED}. The line runs past the longest line a quarantine ` ++ "can name, so it cannot be skipped on this machine: repair or remove it at its " ++ "source, in the shard of the device that wrote it",+ } satisfies DoctorIssue);+ continue;+ }+ out.issues.push({+ severity: "error",+ code: STORE_SHARD_CORRUPT_LINE_CODE,+ path: shard,+ message:+ `the line at byte ${d.offset} of the shard ${shard} is not a valid record ` ++ `(${d.reason}). ${FAILS_CLOSED}. When the line cannot be repaired at its source, ` ++ `skip it on this machine with \`${quarantineCommand(shard, d.offset)}\``,+ } satisfies DoctorIssue);+ }+ const settings = rel.split("/")[0] === "settings";+ for (const r of read.records) {+ if (r.bodyState !== "pending" || r.quarantined === true) continue;+ const written = pendingRecordWrittenMs(r.record.hlc);+ if (written === null) {+ out.issues.push({+ severity: "error",+ code: STORE_SHARD_BLOB_PENDING_CODE,+ path: shard,+ message:+ `the record at byte ${r.offset} of the shard ${shard} has a body blob that has not ` ++ `arrived, and its clock stamp does not parse as a time, so its age is unknown. ` ++ `${FAILS_CLOSED}. It is never quarantined by age: deliver the blob, or repair the ` ++ "record, from the machine that wrote it",+ } satisfies DoctorIssue);+ continue;+ }+ const old = nowMs - written >= PENDING_BLOB_QUARANTINE_AGE_MS;+ const remedy = pendingRemedy(old, settings, shard, r.offset);+ out.issues.push({+ severity: old ? "error" : "warning",+ code: STORE_SHARD_BLOB_PENDING_CODE,+ path: shard,+ message:+ `the record at byte ${r.offset} of the shard ${shard} has a body blob that has not ` ++ `arrived since ${new Date(written).toISOString()}. ${FAILS_CLOSED}. ${remedy}`,+ } satisfies DoctorIssue);+ }+ }+}diff --git a/src/core/brain/doctor/store-side-pass.ts b/src/core/brain/doctor/store-side-pass.tsindex c88cdbbb..43de7ea7 100644--- a/src/core/brain/doctor/store-side-pass.ts+++ b/src/core/brain/doctor/store-side-pass.ts@@ -46,6 +46,7 @@ export function runStoreSideDoctor(configPath: opts.configPath,...(opts.readable !== undefined ? { readable: opts.readable } : {}),...(opts.reach !== undefined ? { reach: opts.reach } : {}),+ ...(opts.withholdsPrivate === true ? { withholdsPrivate: true } : {}),};for (const check of STORE_SIDE_CHECKS) {if (!check.failSoft) {diff --git a/src/core/brain/idea-discovery.ts b/src/core/brain/idea-discovery.tsindex bc5c6d4b..58170762 100644--- a/src/core/brain/idea-discovery.ts+++ b/src/core/brain/idea-discovery.ts@@ -56,6 +56,12 @@ export interface DiscoverIdeasOptions {* vault without that page would give. Absent, every page counts.*/readonly include?: (rel: string) => boolean;+ /**+ * The caller is below local reach: the open questions leave out log+ * events of the store's `private` partition (passed to the morning+ * brief's scan).+ */+ readonly withholdsPrivate?: boolean;}interface NoteFile {@@ -204,6 +210,7 @@ export function discoverIdeas(now: opts.now,topK: 0,lookbackDays: opts.lookbackDays ?? 30,+ withholdsPrivate: opts.withholdsPrivate === true,}).openQuestions;for (const question of openQuestions) {candidates.push(diff --git a/src/core/brain/idea-lineage.ts b/src/core/brain/idea-lineage.tsindex 270c174a..0287ecf3 100644--- a/src/core/brain/idea-lineage.ts+++ b/src/core/brain/idea-lineage.ts@@ -71,7 +71,8 @@ export interface TraceIdeaLineageOptions {/** Maximum backward hops from the queried artifact. Default 8. */readonly maxDepth?: number;/**- * Leave out continuity rows flagged `private`, as if they did not exist.+ * Leave out continuity rows flagged `private`, and a preference's log+ * events of the store's `private` partition, as if they did not exist.* Set for a caller at remote reach.*/readonly withholdPrivate?: boolean;@@ -88,7 +89,7 @@ export function traceIdeaLineage(const id = input.id.trim();if (id.length === 0) throw new IdeaLineageError("idea lineage requires a non-empty id");if (id.startsWith("pref-") || id.startsWith("ret-")) {- return preferenceLineage(vault, id);+ return preferenceLineage(vault, id, opts.withholdPrivate === true);}return continuityLineage(vault,@@ -276,8 +277,12 @@ function snippet(text: string): string {// ----- preference belief evolution ------------------------------------------function preferenceLineage(vault: string, prefId: string): IdeaLineageResult {- const index = buildTimelineIndex(vault, {});+function preferenceLineage(+ vault: string,+ prefId: string,+ withholdPrivate: boolean,+): IdeaLineageResult {+ const index = buildTimelineIndex(vault, { withholdsPrivate: withholdPrivate });const evo = buildBeliefEvolution(index, vault, { prefId });if (evo.transitions.length === 0 && evo.evidence.length === 0 && evo.retirements.length === 0) {throw new IdeaLineageError(`no belief lineage for ${prefId}`);diff --git a/src/core/brain/link-graph/concept-cluster.ts b/src/core/brain/link-graph/concept-cluster.tsindex 53638bc0..8468e243 100644--- a/src/core/brain/link-graph/concept-cluster.ts+++ b/src/core/brain/link-graph/concept-cluster.ts@@ -61,6 +61,11 @@ export interface BuildConceptClusterOptions {* other would leave the more disclosive half open.*/readonly ownerScope?: string | null;+ /**+ * The caller is below local reach: log events of the store's `private`+ * partition link nothing (passed to the backlink index).+ */+ readonly withholdsPrivate?: boolean;}/**@@ -80,7 +85,9 @@ export function buildConceptCluster(const scope = opts.ownerScope ?? null;const targetTitle = resolveTitle(vault, normalised);- const index = buildBacklinkIndex(vault, scope);+ const index = buildBacklinkIndex(vault, scope, {+ withholdsPrivate: opts.withholdsPrivate === true,+ });const refs = index.get(normalised) ?? [];const linkers: ConceptLinker[] = refs.map((r) =>diff --git a/src/core/brain/link-graph/moc-audit.ts b/src/core/brain/link-graph/moc-audit.tsindex 4bca8cf7..16dfedbe 100644--- a/src/core/brain/link-graph/moc-audit.ts+++ b/src/core/brain/link-graph/moc-audit.ts@@ -97,6 +97,12 @@ export interface AuditMocOptions {* target's `referenceCount`.*/readonly readable?: (rel: string) => boolean;+ /**+ * The caller is below local reach: log events of the store's `private`+ * partition count toward no member's `backlinkCount` (passed to the+ * backlink index).+ */+ readonly withholdsPrivate?: boolean;}/** The structural hub thresholds (from `link_graph` config unless overridden). */@@ -231,7 +237,9 @@ export function auditMoc(const outboundTargets = verdict.outboundTargets;// Backlink index + cluster member metadata.- const index = buildBacklinkIndex(vault, view.scope);+ const index = buildBacklinkIndex(vault, view.scope, {+ withholdsPrivate: opts.withholdsPrivate === true,+ });const wellCovered: MocClusterMember[] = [];const fragile: MocClusterMember[] = [];const candidateMissing: { id: string; referenceCount: number }[] = [];diff --git a/src/core/brain/log-events-at-reach.ts b/src/core/brain/log-events-at-reach.tsindex cefe8b80..2daed937 100644--- a/src/core/brain/log-events-at-reach.ts+++ b/src/core/brain/log-events-at-reach.ts@@ -91,6 +91,9 @@ function trimTransitions<K extends string>(* is the reason this per-event rule exists.*/export function eventAtReach(refs: ArtifactRefView, ev: TemporalEvent): TemporalEvent | null {+ // An event of the store's `private` partition is withheld below local+ // reach whatever it names, as in `logEntryAtReach`.+ if (refs.withholdsPrivate === true && ev.fromPrivatePartition === true) return null;if (refs.filtersNothing) return ev;if (!isBrainLogRel(ev.source.path) && !refs.visible(ev.source.path)) return null;if (ev.kind === BRAIN_LOG_EVENT_KIND.applyEvidence) {@@ -175,10 +178,21 @@ export function logEntriesAtReach(): ReadonlyArray<BrainLogEntry> {// An event of the store's `private` partition is withheld below local// reach whatever it names, as a private page is (review finding F333).- const shown =- refs.withholdsPrivate === true- ? entries.filter((e) => e.fromPrivatePartition !== true)- : entries;+ const shown = withoutPrivateEntries(entries, refs.withholdsPrivate === true);if (refs.filtersNothing) return shown;return shown.flatMap((entry) => logEntryAtReach(refs, entry) ?? []);}++/**+ * The entries without those of the store's `private` partition when+ * `withholds` is set (a caller below local reach), all of them otherwise:+ * the one filter every counting reader applies before it counts, so a+ * number or timestamp shown to a remote caller reads as if the withheld+ * events did not exist.+ */+export function withoutPrivateEntries<T extends { readonly fromPrivatePartition?: true }>(+ entries: ReadonlyArray<T>,+ withholds: boolean,+): ReadonlyArray<T> {+ return withholds ? entries.filter((e) => e.fromPrivatePartition !== true) : entries;+}diff --git a/src/core/brain/maintenance/collect.ts b/src/core/brain/maintenance/collect.tsindex e9cdf314..72f4c933 100644--- a/src/core/brain/maintenance/collect.ts+++ b/src/core/brain/maintenance/collect.ts@@ -73,6 +73,11 @@ export interface CollectMaintenanceActionsOptions {* call.*/readonly visible?: (ref: string | null | undefined) => boolean;+ /**+ * The caller is below local reach: the token footprint leaves out the+ * log events of the store's `private` partition. Default false.+ */+ readonly withholdsPrivate?: boolean;}/**@@ -139,7 +144,9 @@ export function collectMaintenanceActions(.filter((f) => visible(f.path) && visible(f.from)).map((f) => ({ path: f.path, from: f.from }));}- const footprint = collectSignal("token-footprint", onError, () => computeTokenFootprint(vault));+ const footprint = collectSignal("token-footprint", onError, () =>+ computeTokenFootprint(vault, { withholdsPrivate: opts.withholdsPrivate === true }),+ );if (footprint !== undefined) {tokenFootprint = {total: footprint.total,diff --git a/src/core/brain/memory-cost-meter.ts b/src/core/brain/memory-cost-meter.tsindex 6faa69b3..66e20e05 100644--- a/src/core/brain/memory-cost-meter.ts+++ b/src/core/brain/memory-cost-meter.ts@@ -32,6 +32,7 @@ import { listContinuityRecords } from "./continuity/store.ts";import { HOST_MEMORY_WRITE_KIND } from "./host-memory-write.ts";import { summarizeRecallTelemetry, type RecallTelemetryMode } from "./recall-telemetry.ts";import { logDayReader } from "./log-days.ts";+import { withoutPrivateEntries } from "./log-events-at-reach.ts";/** Countable memory-write categories the meter folds against reads. */export type MemoryWriteKind = "feedback" | "apply_evidence" | "note" | "host_memory_write";@@ -53,6 +54,11 @@ export interface MemoryWriteFilter {readonly since?: string;/** Inclusive upper ISO-8601 timestamp bound. */readonly until?: string;+ /**+ * The caller is below local reach: a write logged by an event of the+ * store's `private` partition is not counted. Default false.+ */+ readonly withholdsPrivate?: boolean;}export interface MemoryWriteSummary {@@ -86,7 +92,10 @@ export function summarizeMemoryWrites(for (const date of log.dates) {if (sinceDay !== undefined && date < sinceDay) continue;if (untilDay !== undefined && date > untilDay) continue;- for (const entry of log.day(date).entries) {+ for (const entry of withoutPrivateEntries(+ log.day(date).entries,+ filter.withholdsPrivate === true,+ )) {const kind = LOG_EVENT_TO_WRITE_KIND[entry.eventType];if (kind === undefined) continue;if (!withinBounds(entry.timestamp, filter)) continue;@@ -129,6 +138,8 @@ export interface MemoryCostMeterOptions {readonly weights?: Partial<MemoryCostWeights>;/** Ratio above which the period is flagged write-heavy. Default 1. */readonly writeHeavyRatio?: number;+ /** See {@link MemoryWriteFilter.withholdsPrivate}. Default false. */+ readonly withholdsPrivate?: boolean;}export interface MemoryCostMeter {@@ -167,7 +178,10 @@ export function computeMemoryCostMeter(...(options.since !== undefined ? { since: options.since } : {}),...(options.until !== undefined ? { until: options.until } : {}),};- const writes = summarizeMemoryWrites(vault, filter);+ const writes = summarizeMemoryWrites(vault, {+ ...filter,+ withholdsPrivate: options.withholdsPrivate === true,+ });const readSummary = summarizeRecallTelemetry(vault, filter);const reads = { total: readSummary.total, by_mode: readSummary.by_mode };diff --git a/src/core/brain/morning-brief.ts b/src/core/brain/morning-brief.tsindex 816b18f0..47db0a9e 100644--- a/src/core/brain/morning-brief.ts+++ b/src/core/brain/morning-brief.ts@@ -21,6 +21,7 @@ import { brainDirs } from "./paths.ts";import { collectPreferences, resolveOwnerScopeDelivery } from "./preferences-collect.ts";import { applyCharBudget } from "./recall-budget.ts";import { logDayReader } from "./log-days.ts";+import { logEntryAtReach } from "./log-events-at-reach.ts";import { reconcileTopicPreferenceId } from "./log.ts";import { renderActivityTimeline, type ActivityItem } from "./render/activity-line.ts";import { isoDate, relativeAge } from "./time.ts";@@ -73,6 +74,11 @@ export interface MorningBriefOptions {* every preference is a candidate.*/readonly readable?: (rel: string) => boolean;+ /**+ * The caller is below local reach: log events of the store's `private`+ * partition are left out of the open questions and the notes.+ */+ readonly withholdsPrivate?: boolean;}interface ConfirmedPref {@@ -144,6 +150,10 @@ function scanRecentLog(const entries = log.day(date).entries;for (let j = entries.length - 1; j >= 0; j--) {const e = entries[j]!;+ // The reference rule every sibling reader applies: an event this+ // caller may not see (a private one, or one naming an unreadable+ // record) is neither an open question nor a note.+ if (logEntryAtReach(refs, e) === null) continue;const ts = typeof e.timestamp === "string" ? e.timestamp : "";if (e.eventType === BRAIN_LOG_EVENT_KIND.reconcile) {// Auto-resolutions carry a `resolution` field; only open@@ -198,7 +208,7 @@ export function buildMorningBrief(vault: string, opts: MorningBriefOptions): Morvault,opts.now,lookbackDays,- readerRefView(vault, opts.readable),+ readerRefView(vault, opts.readable, opts.withholdsPrivate === true),);// Budget all variable-length entries together so one oversized entrydiff --git a/src/core/brain/most-applied.ts b/src/core/brain/most-applied.tsindex 1cae3715..f1ad221c 100644--- a/src/core/brain/most-applied.ts+++ b/src/core/brain/most-applied.ts@@ -8,6 +8,7 @@*/import { logDayReader } from "./log-days.ts";+import { withoutPrivateEntries } from "./log-events-at-reach.ts";import { MOST_APPLIED_LIMIT_DEFAULT, MOST_APPLIED_WINDOW_DAYS_DEFAULT } from "./policy.ts";import { isoDate } from "./time.ts";import { normaliseWikilinkTarget } from "./wikilink.ts";@@ -31,6 +32,12 @@ export interface ComputeMostAppliedOptions {readonly windowDays?: number;/** Max entries returned. Defaults to 10. */readonly limit?: number;+ /**+ * The caller is below local reach: events of the store's `private`+ * partition are not counted, so the counts and the order read as if+ * they did not exist. Default false (local callers and the CLI).+ */+ readonly withholdsPrivate?: boolean;}/**@@ -90,7 +97,7 @@ export function computeMostApplied();continue;}- for (const e of parsed.entries) {+ for (const e of withoutPrivateEntries(parsed.entries, opts.withholdsPrivate === true)) {if (e.eventType !== BRAIN_LOG_EVENT_KIND.applyEvidence) continue;if (e.body["result"] !== BRAIN_APPLY_RESULT.applied) continue;const eventMs = Date.parse(e.timestamp);diff --git a/src/core/brain/notes/revert.ts b/src/core/brain/notes/revert.tsindex 428a2673..49066d2d 100644--- a/src/core/brain/notes/revert.ts+++ b/src/core/brain/notes/revert.ts@@ -302,10 +302,12 @@ export interface NoteRevertApplyResult {/** Injected clock, shared by both entry points. */export interface PlanNoteRevertOptions {readonly now?: Date;+ /** Select as a caller below local reach; the safety rules still see every write. */+ readonly withholdsPrivate?: boolean;}/** How an apply attributes the `revert` writes it records. */-export interface ApplyNoteRevertOptions extends PlanNoteRevertOptions {+export interface ApplyNoteRevertOptions extends Omit<PlanNoteRevertOptions, "withholdsPrivate"> {/** Caller-asserted identity; defaults to the resolved agent name. */readonly agent?: string;/** Config file that names the writing agent, threaded to the record. */@@ -562,7 +564,9 @@ export function planNoteRevert(const normalised = normaliseSelector(selector);// `listNoteWrites` returns newest-first; every rule here reads the// history forwards, so both lists are reversed once, here.- const selectedRead = listNoteWrites(vault, normalised);+ const withholdsPrivate = opts.withholdsPrivate === true;+ const selectedRead = listNoteWrites(vault, { ...normalised, withholdsPrivate });+ // Only the safety rules read the universe: it keeps private writes.const universeRead = listNoteWrites(vault,normalised.path !== undefined ? { path: normalised.path } : {},diff --git a/src/core/brain/notes/write-destinations.ts b/src/core/brain/notes/write-destinations.tsindex b0af907a..8789298b 100644--- a/src/core/brain/notes/write-destinations.ts+++ b/src/core/brain/notes/write-destinations.ts@@ -113,6 +113,11 @@ export interface WriteDestinationsFilter {* bound measured from the present would answer a clean, wrong empty.*/readonly now?: Date;+ /**+ * The caller is below local reach: a write logged by an event of the+ * store's `private` partition is not counted. Default false.+ */+ readonly withholdsPrivate?: boolean;}export interface WriteDestinationsResult {@@ -252,6 +257,7 @@ export function listWriteDestinations()) {warnings.push(...day.warnings);day.entries.forEach((entry, index) => {+ if (filter.withholdsPrivate === true && entry.fromPrivatePartition === true) return;const device = day.entryShardIds[index] ?? "";if (filter.device !== undefined && device !== filter.device) return;const touch = projectTouch(entry);diff --git a/src/core/brain/notes/write-log.ts b/src/core/brain/notes/write-log.tsindex 2606e648..bbbecafb 100644--- a/src/core/brain/notes/write-log.ts+++ b/src/core/brain/notes/write-log.ts@@ -86,6 +86,11 @@ export interface ListNoteWritesFilter {/** Inclusive upper bound, same two spellings. */readonly until?: string;readonly op?: NoteWriteOp;+ /**+ * The caller is below local reach: a write logged by an event of the+ * store's `private` partition is not listed. Default false.+ */+ readonly withholdsPrivate?: boolean;}export interface ListNoteWritesResult {@@ -197,6 +202,7 @@ export function listNoteWrites()) {warnings.push(...day.warnings);day.entries.forEach((entry, index) => {+ if (filter.withholdsPrivate === true && entry.fromPrivatePartition === true) return;const record = projectRecord(entry, day.entryShardIds[index] ?? "");if (record === null) return;if (filter.agent !== undefined && record.agent !== filter.agent) return;diff --git a/src/core/brain/operator-snapshot.ts b/src/core/brain/operator-snapshot.tsindex 68795dc4..641358ed 100644--- a/src/core/brain/operator-snapshot.ts+++ b/src/core/brain/operator-snapshot.ts@@ -207,6 +207,12 @@ export interface BuildOperatorSnapshotOptions {* so the two seams read as one pattern.*/readonly writerLockHeldAt?: (dbPath: string) => boolean;+ /**+ * The caller is below local reach: the stale scan's timeline and the+ * doctor pass leave out the log events of the store's `private`+ * partition. Default false.+ */+ readonly withholdsPrivate?: boolean;}/**@@ -265,6 +271,7 @@ export async function buildOperatorSnapshot(const doctor = runDoctor(vault, {now,...(searchDbPath !== null ? { dbPath: searchDbPath } : {}),+ ...(opts.withholdsPrivate === true ? { withholdsPrivate: true } : {}),});if (doctor.errors.length > 0) {problem("doctor-errors", `${doctor.errors.length} invariant error(s)`);@@ -308,7 +315,7 @@ export async function buildOperatorSnapshot(if (vault !== null) {try {const cfg = loadTemporalConfigSafe(vault);- const index = buildTimelineIndex(vault, {});+ const index = buildTimelineIndex(vault, { withholdsPrivate: opts.withholdsPrivate === true });const stale = findStaleEntries(index, vault, cfg, { now });staleTotal =stale.stalePreferences.length + stale.staleSignals.length + stale.staleLogFiles.length;diff --git a/src/core/brain/policy/settings-source.ts b/src/core/brain/policy/settings-source.tsindex 77a5a3b1..7237cde1 100644--- a/src/core/brain/policy/settings-source.ts+++ b/src/core/brain/policy/settings-source.ts@@ -36,6 +36,7 @@ import { defaultConfigPath, discoverConfig } from "../../config.ts";import { foldSettings } from "../../store/fold-settings.ts";import { protectionStore, StoreUnavailableError } from "../../store/read/protection-store.ts";import { readStreamRecords } from "../../store/read/stream-read.ts";+import { quarantineStamp } from "../../store/shard-quarantine.ts";import { listShards } from "../../store/shard-list.ts";import { partitionDir } from "../../store/store-paths.ts";import type { StoreInspection } from "../../store/store-open.ts";@@ -217,11 +218,14 @@ function statStamp(path: string): string {}}-/** What a read depends on: the settings shards and the machine config file. */+/**+ * What a read depends on: the settings shards, the quarantine ledger+ * (which decides whether a shard reads whole) and the machine config file.+ */function fingerprintOf(store: StoreInspection | null, configPath: string): string {const parts = [configPath, statStamp(configPath)];if (store !== null && store.partitions.has("shared")) {- parts.push(store.storeId);+ parts.push(store.storeId, `quarantine=${quarantineStamp(store.root)}`);for (const shard of listShards(partitionDir(store.root, "shared"), SETTINGS)) {parts.push(`${shard.rel}=${statStamp(shard.path)}`);}diff --git a/src/core/brain/status.ts b/src/core/brain/status.tsindex 0dc5af84..54ab7f78 100644--- a/src/core/brain/status.ts+++ b/src/core/brain/status.ts@@ -68,6 +68,7 @@ import { parseFrontmatter } from "../vault.ts";import { brainDirs, snapshotsDir } from "./paths.ts";import type { BrainLogEntry } from "./log.ts";import { storeLogDays, withLogReadPass } from "./log-days.ts";+import { withoutPrivateEntries } from "./log-events-at-reach.ts";import { readStoreLogEvents } from "../store/read/log-events.ts";import { listLogDates, readLogDay } from "./log-jsonl.ts";import { loadBrainConfig } from "./policy.ts";@@ -188,6 +189,12 @@ function lessonsCountsSafe(vault: string): LessonApprovalCounts & {export interface ComputeBrainStatusOptions {/** Wall clock for staleness math. Defaults to `new Date()`. */readonly now?: Date;+ /**+ * The caller is below local reach: the log days, the last dream and+ * apply-evidence times and the maintenance debt are computed as if the+ * events of the store's `private` partition did not exist. Default false.+ */+ readonly withholdsPrivate?: boolean;}// ----- Public API ----------------------------------------------------------@@ -231,14 +238,15 @@ function computeBrainStatusInPass(});}- const counts = source === "store" ? countStoreArtifacts(vault) : countArtifacts(vault);- const { lastDreamAt, lastApplyEvidenceAt } = scanLogTimestamps(vault);+ const withholds = opts.withholdsPrivate === true;+ const counts = source === "store" ? countStoreArtifacts(vault, withholds) : countArtifacts(vault);+ const { lastDreamAt, lastApplyEvidenceAt } = scanLogTimestamps(vault, withholds);const signalsAwaitingDream = countSignalsAwaitingDream(vault,opts.now ?? new Date(),source === "store",);- const maintenanceDebt = computeMaintenanceDebt(vault, lastDreamAt);+ const maintenanceDebt = computeMaintenanceDebt(vault, lastDreamAt, withholds);return Object.freeze({present: true,@@ -276,8 +284,12 @@ function computeBrainStatusInPass(* `maintenance_debt.log_events_since_dream` from `computeBrainStatus`* instead (`second_brain_status` / `osb://status`).*/-export function computeMaintenanceOverdueFlag(vault: string | null): boolean | null {- const log = recentStatusLog(vault) ?? statusLog(vault);+export function computeMaintenanceOverdueFlag(+ vault: string | null,+ opts: { readonly withholdsPrivate?: boolean } = {},+): boolean | null {+ const withholds = opts.withholdsPrivate === true;+ const log = recentStatusLog(vault, withholds) ?? statusLog(vault, withholds);if (log === null) return null;const newest = log.dates.at(-1);if (newest === undefined) return null;@@ -349,7 +361,7 @@ function countArtifacts(vault: string): BrainStatusCounts {* their status. The log days and the snapshots are not memory records* and are counted where they are.*/-function countStoreArtifacts(vault: string): BrainStatusCounts {+function countStoreArtifacts(vault: string, withholdsPrivate: boolean): BrainStatusCounts {const dirs = brainDirs(vault);const inFolder = (dir: string): number => storePagesIn(vault, dir).length;const preferences_by_status: Record<string, number> = {};@@ -371,7 +383,7 @@ function countStoreArtifacts(vault: string): BrainStatusCounts {preferences,preferences_by_status: Object.freeze(preferences_by_status),retired: inFolder(dirs.retired),- log_days: statusLog(vault)?.dates.length ?? 0,+ log_days: statusLog(vault, withholdsPrivate)?.dates.length ?? 0,// The snapshot module's folder: the device-local state once switched.snapshots: countZst(snapshotsDir(vault)),};@@ -398,7 +410,7 @@ const RECENT_LOG_DAYS = 31;* own timestamp spells, and one a device whose clock runs ahead wrote can* already name tomorrow; the whole-log read has no upper bound either.*/-function recentStatusLog(vault: string | null): StatusLog | null {+function recentStatusLog(vault: string | null, withholdsPrivate: boolean): StatusLog | null {const now = new Date();const from = new Date(now.getTime() - RECENT_LOG_DAYS * 86_400_000);const to = new Date(now.getTime() + 86_400_000);@@ -406,9 +418,16 @@ function recentStatusLog(vault: string | null): StatusLog | null {{ from: from.toISOString().slice(0, 10), to: to.toISOString().slice(0, 10) },{ vault },);- if (stored === null || stored.length === 0) return null;+ if (stored === null) return null;+ // Below local reach a private event is not there, and a day it alone+ // filled is not a log day.+ const shown = withoutPrivateEntries(+ stored.map((s) => s.entry),+ withholdsPrivate,+ );+ if (shown.length === 0) return null;const byDate = new Map<string, BrainLogEntry[]>();- for (const { entry } of stored) {+ for (const entry of shown) {const date = entry.timestamp.slice(0, 10);const day = byDate.get(date);if (day === undefined) byDate.set(date, [entry]);@@ -425,12 +444,17 @@ function recentStatusLog(vault: string | null): StatusLog | null {* `Brain/log/` is absent), the store's `log-event` records on a store-only* one (task 2.10), where a day carries no read warning.*/-function statusLog(vault: string | null): StatusLog | null {+function statusLog(vault: string | null, withholdsPrivate = false): StatusLog | null {const stored = storeLogDays(vault);if (stored !== null) {- const byDate = new Map(stored.map((d) => [d.date, d.entries] as const));+ // Below local reach a private event is not there, and a day it alone+ // filled is not a log day. Day files carry no mark.+ const days = stored+ .map((d) => ({ date: d.date, entries: withoutPrivateEntries(d.entries, withholdsPrivate) }))+ .filter((d) => d.entries.length > 0 || !withholdsPrivate);+ const byDate = new Map(days.map((d) => [d.date, d.entries] as const));return {- dates: stored.map((d) => d.date),+ dates: days.map((d) => d.date),day: (date) => ({ entries: byDate.get(date) ?? [], warnings: [] }),};}@@ -473,11 +497,14 @@ function readFrontmatterStatus(path: string): string {}}-function scanLogTimestamps(vault: string): {+function scanLogTimestamps(+ vault: string,+ withholdsPrivate: boolean,+): {lastDreamAt: string | null;lastApplyEvidenceAt: string | null;} {- const log = statusLog(vault);+ const log = statusLog(vault, withholdsPrivate);if (log === null) return { lastDreamAt: null, lastApplyEvidenceAt: null };const days = log.dates.toReversed(); // newest day first@@ -525,8 +552,12 @@ function scanLogTimestamps(vault: string): {* dream's own day, so the scan is bounded to "since the last dream",* not the whole log history, on every vault that HAS dreamed.*/-function computeMaintenanceDebt(vault: string, lastDreamAt: string | null): MaintenanceDebt {- const log = statusLog(vault);+function computeMaintenanceDebt(+ vault: string,+ lastDreamAt: string | null,+ withholdsPrivate: boolean,+): MaintenanceDebt {+ const log = statusLog(vault, withholdsPrivate);if (log === null) return NEVER_DREAMED_DEBT;const cutoffMs = lastDreamAt !== null ? Date.parse(lastDreamAt) : null;diff --git a/src/core/brain/temporal/build-index.ts b/src/core/brain/temporal/build-index.tsindex 1b04eb4a..7539c260 100644--- a/src/core/brain/temporal/build-index.ts+++ b/src/core/brain/temporal/build-index.ts@@ -58,6 +58,12 @@ export interface BuildTimelineIndexOptions {readonly until?: string;/** Wall clock; defaults to `new Date()`. Used to derive the default `until`. */readonly now?: Date;+ /**+ * Skip every log event of the store's `private` partition: for the+ * builders whose output no reference view filters, below local reach.+ * Only a store-only machine marks events; day files carry no mark.+ */+ readonly withholdsPrivate?: boolean;}const ISO_DATE_ONLY_RE = /^\d{4}-\d{2}-\d{2}$/;@@ -78,7 +84,7 @@ export function buildTimelineIndex(const window = resolveWindow(opts);const events: TemporalEvent[] = [];- collectLogEvents(vault, window, events);+ collectLogEvents(vault, window, events, opts.withholdsPrivate === true);collectRetiredEvents(vault, window, events);events.sort(compareEvents);@@ -170,7 +176,12 @@ function addDays(day: string, delta: number): string {return new Date(t + delta * ONE_DAY_MS).toISOString().slice(0, 10);}-function collectLogEvents(vault: string, window: TimelineWindow, out: TemporalEvent[]): void {+function collectLogEvents(+ vault: string,+ window: TimelineWindow,+ out: TemporalEvent[],+ withholdsPrivate: boolean,+): void {const sinceDay = dateKeyFromIso(window.since);const untilDay = dateKeyFromIso(window.until);// Err permissive by one day on each side (timezone safety - the@@ -185,6 +196,7 @@ function collectLogEvents(vault: string, window: TimelineWindow, out: TemporalEvfor (const entry of entries) {if (entry.timestamp < window.since) continue;if (entry.timestamp >= window.until) continue;+ if (withholdsPrivate && entry.fromPrivatePartition === true) continue;out.push(normaliseLogEntry(entry, {path: `${STORE_LOG_POINTER_PREFIX}${dateKeyFromIso(entry.timestamp)}`,@@ -265,6 +277,7 @@ function normaliseLogEntry(entry: BrainLogEntry, source: TemporalEventSource): T...(artifact !== undefined ? { artifact } : {}),...(text !== undefined ? { text } : {}),...(dreamSummary !== undefined ? { dreamSummary } : {}),+ ...(entry.fromPrivatePartition === true ? { fromPrivatePartition: true as const } : {}),});}diff --git a/src/core/brain/temporal/types.ts b/src/core/brain/temporal/types.tsindex a5dbe569..53a98c23 100644--- a/src/core/brain/temporal/types.ts+++ b/src/core/brain/temporal/types.ts@@ -83,6 +83,11 @@ export interface TemporalEvent {* belief-evolution to derive per-preference status transitions.*/readonly dreamSummary?: DreamSummarySlots;+ /**+ * Set when the event was read from the store's `private` partition.+ * A reader below local reach withholds it, whatever it names.+ */+ readonly fromPrivatePartition?: true;}/** Window that produced a {@link TimelineIndex}. */diff --git a/src/core/brain/token-footprint.ts b/src/core/brain/token-footprint.tsindex 1896dc13..75606f2a 100644--- a/src/core/brain/token-footprint.ts+++ b/src/core/brain/token-footprint.ts@@ -13,6 +13,7 @@ import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";import { join } from "node:path";import { storeLogDays } from "./log-days.ts";+import { withoutPrivateEntries } from "./log-events-at-reach.ts";import { memoryFromStore, storePagesIn } from "./memory-pages.ts";import { BRAIN_ROOT_REL, brainDirs } from "./paths.ts";import { estimateTokens } from "./text/tokenizer.ts";@@ -81,15 +82,21 @@ function sumStorePages(vault: string, dir: string): { tokens: number; files: num/*** The store's log on a store-only machine, measured as its days: one* "file" per day with an event, the events of the day as JSON lines (the- * shape a day file holds).+ * shape a day file holds). With `withholdsPrivate`, an event of the+ * store's `private` partition is not measured, and a day left with no+ * event is not a file.*/-function sumStoreLog(vault: string): { tokens: number; files: number } {+function sumStoreLog(vault: string, withholdsPrivate: boolean): { tokens: number; files: number } {const days = storeLogDays(vault) ?? [];let tokens = 0;+ let files = 0;for (const day of days) {- tokens += estimateTokens(day.entries.map((entry) => JSON.stringify(entry)).join("\n"));+ const entries = withoutPrivateEntries(day.entries, withholdsPrivate);+ if (entries.length === 0) continue;+ tokens += estimateTokens(entries.map((entry) => JSON.stringify(entry)).join("\n"));+ files += 1;}- return { tokens, files: days.length };+ return { tokens, files };}/**@@ -105,9 +112,20 @@ function resolveWarnThreshold(envValue: string | undefined): number {return TOKEN_WARN_THRESHOLD_DEFAULT;}+export interface ComputeTokenFootprintOptions {+ readonly warnThreshold?: number;+ readonly envWarnThreshold?: string;+ /**+ * The caller is below local reach: the log bucket leaves out the events+ * of the store's `private` partition. The page buckets are page+ * aggregates and count every page. Default false.+ */+ readonly withholdsPrivate?: boolean;+}+export function computeTokenFootprint(vault: string,- opts: { warnThreshold?: number; envWarnThreshold?: string } = {},+ opts: ComputeTokenFootprintOptions = {},): TokenFootprintReport {const dirs = brainDirs(vault);const tracked: Array<[name: string, path: string]> = [@@ -128,7 +146,7 @@ export function computeTokenFootprint(for (const [name, path] of tracked) {let measured: { tokens: number; files: number };if (!fromStore) measured = sumDir(path);- else if (name === "log") measured = sumStoreLog(vault);+ else if (name === "log") measured = sumStoreLog(vault, opts.withholdsPrivate === true);else measured = sumStorePages(vault, path);const { tokens, files } = measured;byCategory.push({ name, tokens, files });diff --git a/src/core/brain/trust/operator-summary.ts b/src/core/brain/trust/operator-summary.tsindex 014df7ff..44050c11 100644--- a/src/core/brain/trust/operator-summary.ts+++ b/src/core/brain/trust/operator-summary.ts@@ -137,6 +137,13 @@ export interface BuildOperatorSummaryOptions {* taken over the entries it keeps. Omitted, every entry counts.*/readonly keepVerification?: (entry: VerificationDeltaEntry) => boolean;+ /**+ * The caller is below local reach: the maintenance actions and the+ * doctor pass leave out the log events of the store's `private`+ * partition (the token footprint's log bucket, the broken-backlink+ * check). Default false.+ */+ readonly withholdsPrivate?: boolean;}const DEFAULT_TOP_ACTIONS_N = 5;@@ -149,7 +156,7 @@ export function buildOperatorSummary(const dreamSummary = opts.dreamSummary;const doctorResult = keptDoctorResult(- safeDoctor(vault, guardrails, dreamSummary, opts.readable),+ safeDoctor(vault, guardrails, dreamSummary, opts.readable, opts.withholdsPrivate === true),opts.keepIssue,);const dreamCounts = summariseDream(dreamSummary);@@ -169,6 +176,7 @@ export function buildOperatorSummary(collectErrors,opts.readable !== undefined,opts.visibleRef,+ opts.withholdsPrivate === true,);const trust = computeTrustVerdict({@@ -257,12 +265,14 @@ function safeDoctor(guardrails: ResolvedBrainGuardrailConfig,dreamSummary: DreamRunSummary | undefined,readable: BuildOperatorSummaryOptions["readable"],+ withholdsPrivate: boolean,): RunDoctorResult {try {return runDoctor(vault, {guardrails,...(dreamSummary ? { dreamSummary } : {}),...(readable !== undefined ? { readable } : {}),+ ...(withholdsPrivate ? { withholdsPrivate: true } : {}),});} catch {return Object.freeze({ warnings: [], errors: [] });@@ -373,6 +383,7 @@ function safeTopActions(collectErrors: { scan: MaintenanceCollectSignal; message: string }[],restricted: boolean,visibleRef?: BuildOperatorSummaryOptions["visibleRef"],+ withholdsPrivate = false,): ReadonlyArray<ActionItem> {try {const all = collectMaintenanceActions(vault, {@@ -387,6 +398,7 @@ function safeTopActions(// actions ask, so the sinks cannot disagree about what a scoped// caller is shown....(visibleRef !== undefined ? { visible: visibleRef } : {}),+ withholdsPrivate,});const kept = keepAction === undefined ? all : all.filter(keepAction);return Object.freeze(kept.slice(0, Math.max(0, topN)));diff --git a/src/core/search/brain-store/catch-up.ts b/src/core/search/brain-store/catch-up.tsindex 6ee1a771..1b0f1613 100644--- a/src/core/search/brain-store/catch-up.ts+++ b/src/core/search/brain-store/catch-up.ts@@ -93,6 +93,8 @@ type FileStat = Pick<ShardStat, "identity" | "size" | "mtimeMs">;interface ProcessState {readonly scanner: ShardScanner;readonly storeRoot: string;+ /** The partitions the scanner lists: a machine that leaves one gets a new state. */+ readonly partitions: string;lastCheckMs: number;indexIdentity: string | null;/** Shard stats as of the watermarks last seen, or null when unknown. */@@ -129,10 +131,18 @@ export function catchUpIntervalMs(env: NodeJS.ProcessEnv = process.env): numberfunction stateFor(config: ResolvedSearchConfig, store: IndexedStore): ProcessState {const existing = states.get(config.dbPath);- if (existing !== undefined && existing.storeRoot === store.root) return existing;+ const partitions = store.partitions.join(",");+ if (+ existing !== undefined &&+ existing.storeRoot === store.root &&+ existing.partitions === partitions+ ) {+ return existing;+ }const fresh: ProcessState = {scanner: new ShardScanner(store),storeRoot: store.root,+ partitions,lastCheckMs: Number.NEGATIVE_INFINITY,indexIdentity: null,known: null,diff --git a/src/core/store-migration/exporters.ts b/src/core/store-migration/exporters.tsindex 69ac929b..af5edf71 100644--- a/src/core/store-migration/exporters.ts+++ b/src/core/store-migration/exporters.ts@@ -59,6 +59,7 @@ import { preferencePath, retiredPath, validateSlug } from "../brain/paths.ts";import { preferencePageBody } from "../brain/preference.ts";import { PREFERENCE_TEXT_FIELD } from "../brain/preference-store.ts";import { brainRelOfEntry } from "../store/lane-paths.ts";+import { rawParsesBack, renderLaneText } from "../store/lane-raw.ts";import { signalPagePath } from "../store/read/signals.ts";import {LANE_ENTRY_KIND,@@ -409,14 +410,20 @@ function laneEntryPath(vault: string, key: string, fold: LaneFold): string | nul);}-/** A lane entry's file text: JSON, a page, or plain text, by its extension. */+/** A lane entry's file text: its `raw` when it parses back, else by its extension. */function laneEntryText(path: string,- value: { readonly fields: Readonly<Record<string, unknown>>; readonly text: string },+ value: {+ readonly fields: Readonly<Record<string, unknown>>;+ readonly text: string;+ readonly raw?: string;+ },): string {- if (path.endsWith(".json")) return `${JSON.stringify(value.fields, null, 2)}\n`;+ if (value.raw !== undefined && rawParsesBack(path, value.raw, value.fields, value.text)) {+ return value.raw;+ }if (path.endsWith(".md")) return formatFrontmatter(asFrontmatter(value.fields), value.text);- return value.text;+ return renderLaneText(path, value.fields, value.text);}/** Every touched lane entry, written back from the lane fold at its path. */diff --git a/src/core/store-migration/journal.ts b/src/core/store-migration/journal.tsindex cb66e5a6..e1e411f3 100644--- a/src/core/store-migration/journal.ts+++ b/src/core/store-migration/journal.ts@@ -250,6 +250,17 @@ export function readHeldRecords(store: Pick<StoreHandle, "root" | "partitions">)},})) {records.push(...shard.all);+ // A pending blob is a gap here even when this machine quarantined its+ // line: rollback, the late import, the preflight and the legacy+ // removal refuse on it, and none of them waits for it.+ const rel = `${partition}/${shard.rel}`;+ if (+ shard.pending.length > 0 &&+ shard.rel.split("/")[0] !== DIGEST_STREAM &&+ !partial.includes(rel)+ ) {+ partial.push(rel);+ }}}return {diff --git a/src/core/store/FORMAT.md b/src/core/store/FORMAT.mdindex 9dc65902..092d4e1a 100644--- a/src/core/store/FORMAT.md+++ b/src/core/store/FORMAT.md@@ -298,7 +298,7 @@ Kinds of attestations (a signed statement of the state one machine measured, madKinds of record lanes (the Brain record lanes that have no kind of their own: decisions, obligations, pledges, theses, tensions, gap tasks, attention flows, entities, procedures, procedural memory, skill proposals and benchmarks, captures, write sessions, exact-state pages, standing rules and the rest):-- `lane-entry`: `{ "authority", "lane": "<lane>", "entry": "<entry id>", "fields": { ... }, "text": "<Markdown>" }`, written to `events`. The whole current entry: `fields` are the structured fields the entry carried (a page's frontmatter, or the object a JSON file held), `text` its Markdown body. A lane name is 1 to 64 lowercase letters, digits and hyphens, starting with a letter or digit; an entry id is a non-empty single line of at most 512 characters, unique within its lane. Standing rules are entries of the lane `standing-rule` and carry `authority` `human`. An entry goes to `private` when its page is private by its visibility markers, or when its fields or text name a private page or one that cannot be judged; the text of a large entry spills into a blob like any other body.+- `lane-entry`: `{ "authority", "lane": "<lane>", "entry": "<entry id>", "fields": { ... }, "text": "<Markdown>", "raw": "<file text>" (optional) }`, written to `events`. The whole current entry: `fields` are the structured fields the entry carried (a page's frontmatter, or the object a JSON file held), `text` its Markdown body. `raw`, when present, is the file's text byte for byte, written only when `fields` and `text` do not reproduce it (a page without frontmatter, a quoted value, CRLF line ends, the formatting of a JSON file). A reader serves `raw` only when it parses back to the record's own `fields` and `text`, and renders `fields` and `text` otherwise; a reader that does not know `raw` ignores it. The rollback export serves `raw` under the same rule: it writes `raw` into the restored file when it parses back, and the rendering of `fields` and `text` otherwise. `raw` counts for the entry's privacy like `fields` and `text`, and spills into a blob with the rest of the body. A lane name is 1 to 64 lowercase letters, digits and hyphens, starting with a letter or digit; an entry id is a non-empty single line of at most 512 characters, unique within its lane. Standing rules are entries of the lane `standing-rule` and carry `authority` `human`. An entry goes to `private` when its page is private by its visibility markers, or when its fields or text name a private page or one that cannot be judged; the text of a large entry spills into a blob like any other body.- `lane-entry-remove`: `{ "authority", "lane": "<lane>", "entry": "<entry id>" }`, written to `events`. The tombstone of an entry.Entries fold by lane and entry id: each entry is one register, resolved by authority, then clock.diff --git a/src/core/store/dual-write.ts b/src/core/store/dual-write.tsindex 6e516f0b..ef1ca5f5 100644--- a/src/core/store/dual-write.ts+++ b/src/core/store/dual-write.ts@@ -142,6 +142,15 @@ export interface DualWriteStatus {readonly last_error: string | null;readonly last_failure_at: string | null;readonly last_skip_at: string | null;+ /**+ * The failures known to concern only the `shared` partition: the+ * operation's records were all placed in `shared` when its append failed+ * ({@link failedOnlyInShared}). The only failure counters a caller below+ * local reach may see; `failures` counts these and every other one.+ */+ readonly shared_failures: number;+ readonly shared_last_error: string | null;+ readonly shared_last_failure_at: string | null;}const EMPTY_STATUS: DualWriteStatus = Object.freeze({@@ -151,6 +160,9 @@ const EMPTY_STATUS: DualWriteStatus = Object.freeze({last_error: null,last_failure_at: null,last_skip_at: null,+ shared_failures: 0,+ shared_last_error: null,+ shared_last_failure_at: null,});type WarningKind = "failure" | "device" | "store";@@ -194,6 +206,11 @@ export function readDualWriteStatus(path: string = dualWriteStatusPath()): DualWlast_error: typeof parsed.last_error === "string" ? parsed.last_error : null,last_failure_at: typeof parsed.last_failure_at === "string" ? parsed.last_failure_at : null,last_skip_at: typeof parsed.last_skip_at === "string" ? parsed.last_skip_at : null,+ shared_failures: typeof parsed.shared_failures === "number" ? parsed.shared_failures : 0,+ shared_last_error:+ typeof parsed.shared_last_error === "string" ? parsed.shared_last_error : null,+ shared_last_failure_at:+ typeof parsed.shared_last_failure_at === "string" ? parsed.shared_last_failure_at : null,};} catch {return EMPTY_STATUS;@@ -230,11 +247,19 @@ function recordFailure(err: unknown): void {"Later failures in this process are counted, not printed; `o2b brain doctor` reports them.",);const at = new Date().toISOString();+ const shared = failedOnlyInShared(err);bumpStatus((s) => ({...s,failures: s.failures + 1,last_error: message.slice(0, 500),last_failure_at: at,+ ...(shared+ ? {+ shared_failures: s.shared_failures + 1,+ shared_last_error: message.slice(0, 500),+ shared_last_failure_at: at,+ }+ : {}),}));}@@ -353,23 +378,51 @@ export function appendStoreRecords(if (at.onlyIfApart === true && (joined.length === 0 || raised === partition)) continue;placed.push([record, raised]);}- // Every partition is checked before the first append, so a machine- // without one writes none of the records rather than half the operation.- for (const [, at] of placed) {- if (!store.partitions.has(at)) throw new PartitionUnavailableError(at, store.root);+ try {+ // Every partition is checked before the first append, so a machine+ // without one writes none of the records rather than half the operation.+ for (const [, at] of placed) {+ if (!store.partitions.has(at)) throw new PartitionUnavailableError(at, store.root);+ }+ return placed.map(([record, at]) =>+ appendRecord(store, {+ partition: at,+ stream: record.stream,+ ...(record.ledger !== undefined ? { ledger: record.ledger } : {}),+ kind: record.kind,+ agent:+ record.agent !== undefined && record.agent.trim() !== "" ? record.agent : defaultAgent(),+ scope: record.scope ?? null,+ body: { ...record.body, [VAULT_ID_FIELD]: vaultId },+ }),+ );+ } catch (err) {+ if (typeof err === "object" && err !== null) {+ failedPlacement.set(err, new Set(placed.map(([, at]) => at)));+ }+ throw err;+ }+}++/** The partitions of the operation whose append threw this error (see {@link appendStoreRecords}). */+const failedPlacement = new WeakMap<object, ReadonlySet<StorePartition>>();++/**+ * Whether the failed write `err` (or the error it wraps, `cause`) is known+ * to concern only the `shared` partition: it was thrown by+ * {@link appendStoreRecords} for an operation whose records were all placed+ * in `shared`. Any other failure (one before the records were placed, a+ * device id or store that cannot be resolved, a builder that threw, or an+ * operation with a record in `private`) may concern `private`, so it+ * answers `false`: doctor never shows it to a caller below local reach.+ */+export function failedOnlyInShared(err: unknown): boolean {+ for (const e of [err, (err as { cause?: unknown } | null)?.cause]) {+ if (typeof e !== "object" || e === null) continue;+ const placed = failedPlacement.get(e);+ if (placed !== undefined) return placed.size > 0 && [...placed].every((p) => p === "shared");}- return placed.map(([record, at]) =>- appendRecord(store, {- partition: at,- stream: record.stream,- ...(record.ledger !== undefined ? { ledger: record.ledger } : {}),- kind: record.kind,- agent:- record.agent !== undefined && record.agent.trim() !== "" ? record.agent : defaultAgent(),- scope: record.scope ?? null,- body: { ...record.body, [VAULT_ID_FIELD]: vaultId },- }),- );+ return false;}/**diff --git a/src/core/store/held-ledger.ts b/src/core/store/held-ledger.tsindex 8b65d18a..cf0ea8f2 100644--- a/src/core/store/held-ledger.ts+++ b/src/core/store/held-ledger.ts@@ -17,13 +17,14 @@* protection reader fails closed exactly as for an unopenable store, and* every write refuses. A machine that never held a partition is unaffected.*- * No command takes a partition off this record: the remedy is to restore the- * partition's identity file (sync, or a backup). The list is keyed by the- * store root as configured, like the switch marker.+ * One command takes a partition off this record: `o2b store partition leave+ * <partition>` (`partition-leave.ts`), once the partition's identity file is+ * gone. The other remedy is to restore the identity file (sync, or a backup).+ * The list is keyed by the store root as configured, like the switch marker.*/import { createHash } from "node:crypto";-import { existsSync, mkdirSync } from "node:fs";+import { existsSync, mkdirSync, rmSync } from "node:fs";import { join, resolve } from "node:path";import { atomicWriteFileSync } from "../fs-atomic.ts";@@ -89,3 +90,23 @@ export function lostHeldPartitions(const dir = ledgerDir();return STORE_PARTITIONS.filter((p) => !held.has(p) && existsSync(markerPath(dir, root, p)));}++/**+ * Whether this machine holds a marker for the partition at `root`: the+ * fact {@link recordHeldPartitions} wrote and {@link HeldPartitionLostError}+ * reads.+ */+export function isHeldPartitionMarked(root: string, partition: StorePartition): boolean {+ return existsSync(markerPath(ledgerDir(), root, partition));+}++/**+ * Take the partition off this machine's held list: remove the marker, and+ * return `true` when one was there. `false` changes nothing.+ */+export function forgetHeldPartition(root: string, partition: StorePartition): boolean {+ const marker = markerPath(ledgerDir(), root, partition);+ if (!existsSync(marker)) return false;+ rmSync(marker, { force: true });+ return true;+}diff --git a/src/core/store/lane-entry-store.ts b/src/core/store/lane-entry-store.tsindex 7aa0027a..1cb3c14c 100644--- a/src/core/store/lane-entry-store.ts+++ b/src/core/store/lane-entry-store.ts@@ -14,7 +14,6 @@ import { isAbsolute, join, relative, sep } from "node:path";import { BRAIN_ROOT_REL } from "../brain/path-constants.ts";import { canonicalJson } from "../integrity/digest.ts";import type { FrontmatterMap } from "../types.ts";-import { parseFrontmatterText } from "../vault.ts";import { laneStoreFold } from "./lane-fold-cache.ts";import { lanePlaceOf } from "./lane-paths.ts";import {@@ -24,6 +23,7 @@ import {STANDING_RULE_LANE,type LaneEntryValue,} from "./lane-records.ts";+import { lanePrivacyRow, laneTextParts, rawToKeep } from "./lane-raw.ts";import { opaqueFields } from "./operation-privacy.ts";import { isPrivateFrontmatter } from "./private-page.ts";import { rowNamesPrivatePage } from "./reference-privacy.ts";@@ -175,9 +175,13 @@ export function sameEntry(value: LaneEntryValue | null,fields: Readonly<Record<string, unknown>>,text: string,+ raw?: string,): boolean {return (- value !== null && value.text === text && canonicalJson(value.fields) === canonicalJson(fields)+ value !== null &&+ value.text === text &&+ canonicalJson(value.fields) === canonicalJson(fields) &&+ value.raw === raw);}@@ -189,10 +193,15 @@ export function isPrivateEntry(vault: string,fields: Readonly<Record<string, unknown>>,text: string,+ raw?: string,): boolean {return (isPrivateFrontmatter(fields) ||- rowNamesPrivatePage(existsSync(vault) ? vault : null, { fields, text }, PAGE_OPAQUE_FIELDS)+ rowNamesPrivatePage(+ existsSync(vault) ? vault : null,+ lanePrivacyRow(fields, text, raw),+ PAGE_OPAQUE_FIELDS,+ ));}@@ -209,17 +218,20 @@ export function entryRecord(* device without that partition never sees the old entry removed.*/placement: "judged" | "located" = "judged",+ raw?: string,): DualWriteRecord {const { lane, entry } = placeOf(vault, path);const agent =typeof fields["agent"] === "string" && fields["agent"].trim() !== ""? fields["agent"].trim(): undefined;+ const kept = rawToKeep(path, raw, fields, text);return laneEntryRecord({lane,entry,fields,text,+ ...(kept !== undefined ? { raw: kept } : {}),authority: authorityFor(lane),...(agent !== undefined ? { agent } : {}),private: isPrivate,@@ -276,24 +288,4 @@ export function storedEntryPrivate(vault: string, path: string): boolean {* and body, a JSON object as the fields, anything else (malformed JSON* included, which the legacy writer also writes as it is) as text.*/-export function laneTextParts(- path: string,- text: string,-): { fields: Record<string, unknown>; body: string } {- if (path.endsWith(".md")) {- const [frontmatter, rest] = parseFrontmatterText(text);- return { fields: frontmatter, body: rest };- }- if (path.endsWith(".json")) {- let parsed: unknown = null;- try {- parsed = JSON.parse(text);- } catch {- return { fields: {}, body: text };- }- if (parsed !== null && typeof parsed === "object" && !Array.isArray(parsed)) {- return { fields: parsed as Record<string, unknown>, body: "" };- }- }- return { fields: {}, body: text };-}+export { laneTextParts } from "./lane-raw.ts";diff --git a/src/core/store/lane-fold-cache.ts b/src/core/store/lane-fold-cache.tsindex 0b14ed0e..b5db1b2c 100644--- a/src/core/store/lane-fold-cache.ts+++ b/src/core/store/lane-fold-cache.ts@@ -13,7 +13,8 @@* machine read before and that are gone now: an entry's records in one* are unknown, so a privacy decision over the fold fails closed while any* is listed. Such a fold is never cached, so a shard that reads again is- * folded at once. A switched machine whose store cannot be opened lists+ * folded at once; nor is a fold that left out a record whose body is a+ * pending blob, so the record is folded as soon as its blob arrives. A switched machine whose store cannot be opened lists* that reason the same way (`read/protection-store.ts`).*/@@ -32,6 +33,7 @@ import {type LaneEntryValue,} from "./lane-records.ts";import { readStreamRecords } from "./read/stream-read.ts";+import { quarantineStamp } from "./shard-quarantine.ts";import { protectionStore, StoreUnavailableError } from "./read/protection-store.ts";import { listShards } from "./shard-list.ts";import type { StoreInspection } from "./store-open.ts";@@ -61,9 +63,13 @@ const EMPTY_FOLD: LaneFolded = {const cache = new Map<string, LaneFolded>();-/** A stamp of every `events` shard of `store`; it moves whenever a record lands. */+/**+ * A stamp of every `events` shard of `store`; it moves whenever a record+ * lands, and with the quarantine ledger, which decides whether a shard+ * reads whole without changing its bytes.+ */export function laneFingerprintOf(store: StoreInspection): string {- const parts: string[] = [store.storeId];+ const parts: string[] = [store.storeId, `quarantine=${quarantineStamp(store.root)}`];for (const partition of [...store.partitions].toSorted()) {for (const shard of listShards(partitionDir(store.root, partition), { stream: "events" })) {let stamp = "missing";@@ -101,9 +107,11 @@ export function laneStoreFold(): LaneFolded {const records: StoreRecord[] = [];const inPrivate = new Set<string>();const unreadable: string[] = [];+ let skippedPending = false;for (const partition of STORE_PARTITIONS) {if (!store.partitions.has(partition)) continue;const read = readStreamRecords(store.root, partition, { stream: "events" });+ if (read.skippedPending) skippedPending = true;for (const rel of read.unreadable) unreadable.push(`${partition}/${rel}`);for (const record of read.records) {if (record.kind !== LANE_ENTRY_KIND && record.kind !== LANE_ENTRY_REMOVE_KIND) continue;@@ -125,8 +133,9 @@ export function laneStoreFold(): LaneFolded {const folded = { fingerprint, entries, clocks, privateKeys, unreadable };// A shard that reads again after a permission change keeps its size and// mtime, so the fingerprint would not move: an incomplete fold is never- // reused.- if (unreadable.length === 0) cache.set(store.root, folded);+ // reused. Nor is one that left out a record whose body is a pending blob:+ // the blob's arrival moves neither a shard stamp nor the quarantine stamp.+ if (unreadable.length === 0 && !skippedPending) cache.set(store.root, folded);else cache.delete(store.root);return folded;}diff --git a/src/core/store/lane-pages.ts b/src/core/store/lane-pages.tsindex 5160c143..3cc5850e 100644--- a/src/core/store/lane-pages.ts+++ b/src/core/store/lane-pages.ts@@ -44,7 +44,6 @@ import { noVaultRoot } from "../platform-dirs.ts";import { containedRecordPath } from "../record-path.ts";import type { FrontmatterMap } from "../types.ts";import {- formatFrontmatter,parseFrontmatter,parseFrontmatterText,writeFrontmatterAtomic,@@ -72,6 +71,7 @@ import {storedEntryPrivate,tombstoneOf,} from "./lane-entry-store.ts";+import { rawParsesBack, rawToKeep, renderLaneText } from "./lane-raw.ts";import { brainRelOfEntry, lanePlaceOf } from "./lane-paths.ts";import { configuredStore } from "./read/store-entities.ts";import { acquireDirLock, STORE_LOCK_STALE_MS, withLocalLock } from "./local-lock.ts";@@ -263,10 +263,10 @@ export function readLaneText(vault: string, path: string): string | null {if (heldInStore(vault, path)) {const value = storeEntry(vault, path);if (value === null) return null;- if (path.endsWith(".md")) return formatFrontmatter(value.fields as FrontmatterMap, value.text);- return path.endsWith(".json") && value.text === ""- ? `${JSON.stringify(value.fields, null, 2)}\n`- : value.text;+ if (value.raw !== undefined && rawParsesBack(path, value.raw, value.fields, value.text)) {+ return value.raw;+ }+ return renderLaneText(path, value.fields, value.text);}return existsSync(path) ? readFileSync(path, "utf8") : null;}@@ -399,7 +399,8 @@ export function writeLaneText(): void {if (opts.exclusive !== true && heldInStore(vault, path)) {const { fields, body } = laneTextParts(path, text);- if (sameEntry(storeEntry(vault, path), fields, body)) return;+ const raw = rawToKeep(path, text, fields, body);+ if (sameEntry(storeEntry(vault, path), fields, body, raw)) return;}const replaced = opts.exclusive !== true && legacyPagePrivate(vault, path);withCreateLock(vault, path, opts.exclusive === true, undefined, () =>@@ -411,6 +412,7 @@ export function writeLaneText(},store: () => {const { fields, body } = laneTextParts(path, text);+ const raw = rawToKeep(path, text, fields, body);return [...replacedShared(vault, path),entryRecord(@@ -418,7 +420,9 @@ export function writeLaneText(path,fields,body,- replaced || storedEntryPrivate(vault, path) || isPrivateEntry(vault, fields, body),+ replaced || storedEntryPrivate(vault, path) || isPrivateEntry(vault, fields, body, raw),+ "judged",+ raw,),];},@@ -536,6 +540,7 @@ export function moveLanePage(rewrite?.body ?? value.text,isPrivate,"located",+ rewrite === undefined ? value.raw : undefined,),],context(vault),@@ -601,6 +606,7 @@ export function moveLaneTree(vault: string, from: string, to: string): void {value.text,partition === "private","located",+ value.raw,),located: { partition, content: true },},diff --git a/src/core/store/lane-raw.ts b/src/core/store/lane-raw.tsnew file mode 100644index 00000000..098048e9--- /dev/null+++ b/src/core/store/lane-raw.ts@@ -0,0 +1,105 @@+/**+ * The raw text of a lane entry (design D8 of change `store-hardening`).+ *+ * A lane file written whole on a store-only machine read back changed: a+ * page without frontmatter gained an empty one, `'123'` turned into a+ * number, CRLF line ends and the formatting of a JSON file were lost. A+ * `lane-entry` record now keeps the file's text in an optional `raw`+ * field when its fields and text do not reproduce it, and a reader+ * serves it only when it parses back to the record's own fields and+ * text. The writer's privacy judge is the only judge of `raw`, exactly+ * as it is the only judge of `text`; the read-side check only proves the+ * raw parses back.+ */++import { parseFrontmatterText, formatFrontmatter } from "../vault.ts";+import { canonicalJson } from "../integrity/digest.ts";+import type { FrontmatterMap } from "../types.ts";++/**+ * A lane file's text as an entry's fields and text: a page's frontmatter+ * and body, a JSON object as the fields, anything else (malformed JSON+ * included, which the legacy writer also writes as it is) as text.+ */+export function laneTextParts(+ path: string,+ text: string,+): { fields: Record<string, unknown>; body: string } {+ if (path.endsWith(".md")) {+ const [frontmatter, rest] = parseFrontmatterText(text);+ return { fields: frontmatter, body: rest };+ }+ if (path.endsWith(".json")) {+ let parsed: unknown = null;+ try {+ parsed = JSON.parse(text);+ } catch {+ return { fields: {}, body: text };+ }+ if (parsed !== null && typeof parsed === "object" && !Array.isArray(parsed)) {+ return { fields: parsed as Record<string, unknown>, body: "" };+ }+ }+ return { fields: {}, body: text };+}++/**+ * The store branch of `readLaneText` today: `.md` renders the frontmatter,+ * a `.json` with empty text renders the fields, anything else is the text.+ */+export function renderLaneText(+ path: string,+ fields: Readonly<Record<string, unknown>>,+ text: string,+): string {+ if (path.endsWith(".md")) {+ return formatFrontmatter(fields as FrontmatterMap, text);+ }+ if (path.endsWith(".json") && text === "") {+ return `${JSON.stringify(fields, null, 2)}\n`;+ }+ return text;+}++/**+ * Whether `raw` parses back to these fields and this text: the parts+ * {@link laneTextParts} makes of it equal them.+ */+export function rawParsesBack(+ path: string,+ raw: string,+ fields: Readonly<Record<string, unknown>>,+ text: string,+): boolean {+ const parts = laneTextParts(path, raw);+ return parts.body === text && canonicalJson(parts.fields) === canonicalJson(fields);+}++/**+ * The file's text byte for byte to write into the record: `raw` when it is+ * defined, differs from the rendering, and parses back; otherwise+ * `undefined`.+ */+export function rawToKeep(+ path: string,+ raw: string | undefined,+ fields: Readonly<Record<string, unknown>>,+ text: string,+): string | undefined {+ if (raw === undefined) return undefined;+ if (renderLaneText(path, fields, text) === raw) return undefined;+ if (!rawParsesBack(path, raw, fields, text)) return undefined;+ return raw;+}++/**+ * The one row the write-time privacy judge reads: the fields and the text,+ * plus `raw` when defined.+ */+export function lanePrivacyRow(+ fields: Readonly<Record<string, unknown>>,+ text: string,+ raw?: string,+): Record<string, unknown> {+ return raw === undefined ? { fields, text } : { fields, text, raw };+}diff --git a/src/core/store/lane-records.ts b/src/core/store/lane-records.tsindex 5eaea665..ca23e1f6 100644--- a/src/core/store/lane-records.ts+++ b/src/core/store/lane-records.ts@@ -83,6 +83,8 @@ export function laneEntryRecord(input: {readonly entry: string;readonly fields: Readonly<Record<string, unknown>>;readonly text: string;+ /** The file's text byte for byte, when the fields and text do not reproduce it. */+ readonly raw?: string;readonly authority: Authority;readonly agent?: string;readonly private: boolean;@@ -105,6 +107,7 @@ export function laneEntryRecord(input: {entry: input.entry,fields: plainJson(input.fields),text: input.text,+ ...(input.raw !== undefined ? { raw: input.raw } : {}),},};}@@ -136,6 +139,11 @@ export interface LaneEntryValue {readonly entry: string;readonly fields: Readonly<Record<string, unknown>>;readonly text: string;+ /**+ * The file's text byte for byte, when the fields and text do not+ * reproduce it; served only when it parses back to them (`lane-raw.ts`).+ */+ readonly raw?: string;}export interface LaneEntryState {@@ -192,6 +200,7 @@ export function foldLaneEntries(input: Iterable<StoreRecord>): LaneFold {entry,fields: record.body["fields"] as Record<string, unknown>,text: record.body["text"] as string,+ ...(typeof record.body["raw"] === "string" ? { raw: record.body["raw"] } : {}),};register.offer(laneEntryKey(lane, entry), {record,diff --git a/src/core/store/partition-leave.ts b/src/core/store/partition-leave.tsnew file mode 100644index 00000000..a84b8683--- /dev/null+++ b/src/core/store/partition-leave.ts@@ -0,0 +1,157 @@+/**+ * Take a partition off this machine on purpose (design D7 of change+ * `store-hardening`).+ *+ * The held partitions are sticky (`held-ledger.ts`): a machine that held a+ * partition whose `store.json` is gone refuses every store operation. This+ * module clears what this machine kept of a partition once its folder no+ * longer carries the identity file: the held marker, the seen-shard+ * entries and the digest caches, marker last, so a failed clearing leaves+ * the machine still holding the partition (it fails closed, and+ * `HeldPartitionLostError` still names the verb). A `store.json` that+ * returns is held again by `recordHeldPartitions`.+ */++import { existsSync, readdirSync, rmSync } from "node:fs";+import { join } from "node:path";++import { digestCacheRoot } from "../brain/compiled-digest-path.ts";+import { STORE_IDENTITY_FILE, type StorePartition } from "./constants.ts";+import { forgetHeldPartition, isHeldPartitionMarked } from "./held-ledger.ts";+import { readStoreId } from "./store-open.ts";+import { forgetSeenPartition } from "./seen-shards.ts";+import { partitionDir } from "./store-paths.ts";++/** Why {@link leavePartition} refuses. */+export class PartitionLeaveRefusal extends Error {+ constructor(+ readonly reason: "shared" | "still-held" | "files-left" | "no-store-id" | "clear-failed",+ message: string,+ ) {+ super(message);+ this.name = "PartitionLeaveRefusal";+ }+}++/** What a leave reported: whether it left, and what it cleared. */+export interface LeavePartitionResult {+ readonly left: boolean;+ readonly partition: StorePartition;+ readonly cleared: {+ readonly heldMarker: boolean;+ readonly seenShards: number;+ readonly digestCaches: boolean;+ };+}++/**+ * Take `partition` off this machine's held list of the store at `root`:+ * refuse what it cannot do safely, then clear this machine's device-local+ * state of it, marker last.+ */+export function leavePartition(options: {+ readonly root: string;+ readonly partition: StorePartition;+ readonly force?: boolean;+}): LeavePartitionResult {+ const { root, partition } = options;+ if (partition === "shared") {+ throw new PartitionLeaveRefusal(+ "shared",+ "the shared partition holds every rule of the brain store; leaving it is leaving " ++ "the store, which this command does not do",+ );+ }+ const dir = partitionDir(root, partition);+ const refuseStillHeld = (): void => {+ if (!existsSync(join(dir, STORE_IDENTITY_FILE))) return;+ throw new PartitionLeaveRefusal(+ "still-held",+ `${join(dir, STORE_IDENTITY_FILE)} is still here, so the ${partition} partition is not ` ++ `left; stop syncing ${dir}, remove the folder, then run the command again`,+ );+ };+ refuseStillHeld();+ if (!isHeldPartitionMarked(root, partition)) {+ return {+ left: false,+ partition,+ cleared: { heldMarker: false, seenShards: 0, digestCaches: false },+ };+ }+ let entries: string[];+ try {+ entries = existsSync(dir) ? readdirSync(dir) : [];+ } catch (err) {+ const code = (err as { code?: string } | null)?.code ?? "unknown";+ throw new PartitionLeaveRefusal(+ "files-left",+ `could not list ${dir} (${code}), so this machine's state of the ${partition} partition ` ++ "cannot be checked; fix the cause, then run the command again",+ );+ }+ if (entries.length > 0 && options.force !== true) {+ throw new PartitionLeaveRefusal(+ "files-left",+ `${dir} still holds ${entries.length} item(s) of the ${partition} partition without its ` ++ `${STORE_IDENTITY_FILE}; remove the folder, or pass --force to leave the partition ` ++ "anyway (its files are then ignored on this machine)",+ );+ }+ let storeId: string | null;+ let idProblem: string | null = null;+ try {+ storeId = readStoreId(partitionDir(root, "shared"));+ } catch (err) {+ storeId = null;+ idProblem = err instanceof Error ? err.message : String(err);+ }+ if (storeId === null) {+ // `idProblem` is null for a missing file and otherwise the read's own+ // error, which already carries the full path, so the wording below+ // names the path once either way.+ const what =+ idProblem ?? `${join(partitionDir(root, "shared"), STORE_IDENTITY_FILE)} is missing`;+ throw new PartitionLeaveRefusal(+ "no-store-id",+ `${what}, so the device-local state of the ${partition} partition cannot be found; ` ++ "restore the shared partition's identity file first (sync, or a backup), then run the " ++ "command again",+ );+ }+ let digestCaches = false;+ let seenShards = 0;+ try {+ const cacheDir = digestCacheRoot(storeId);+ const existed = existsSync(cacheDir);+ rmSync(cacheDir, { recursive: true, force: true });+ digestCaches = existed;+ seenShards = forgetSeenPartition(root, storeId, partition);+ } catch (err) {+ throw new PartitionLeaveRefusal(+ "clear-failed",+ `could not clear this machine's state of the ${partition} partition ` ++ `(${err instanceof Error ? err.message : String(err)}); the partition is still held ` ++ "here; fix the cause, then run the command again",+ );+ }+ // A sync tool may have brought the identity file back while the caches+ // were cleared: the marker goes only while the file is still absent.+ refuseStillHeld();+ let heldMarker: boolean;+ try {+ heldMarker = forgetHeldPartition(root, partition);+ } catch (err) {+ throw new PartitionLeaveRefusal(+ "clear-failed",+ `could not remove the held marker of the ${partition} partition ` ++ `(${err instanceof Error ? err.message : String(err)}); the partition is still held ` ++ "here; fix the cause, then run the command again",+ );+ }+ return {+ left: true,+ partition,+ cleared: { heldMarker, seenShards, digestCaches },+ };+}diff --git a/src/core/store/read/store-entities.ts b/src/core/store/read/store-entities.tsindex 64048064..70ba4fd3 100644--- a/src/core/store/read/store-entities.ts+++ b/src/core/store/read/store-entities.ts@@ -10,6 +10,8 @@* (`../store-privacy.ts`). It is cached per store root and re-read only* when an `events` shard of a held partition appears, grows or changes,* so a writer operation that judges many references reads the store once.+ * A read that left out a record whose body is a pending blob is not+ * cached, because the blob's arrival moves no stamp the cache compares.** Signals fold their `signal` and `signal-state` records* (`../fold-signals.ts`), so a reader tells an inbox signal from a@@ -44,6 +46,7 @@ import { listShards } from "../shard-list.ts";import { inspectStore, type StoreInspection } from "../store-open.ts";import { partitionDir, resolveConfiguredStoreRoot } from "../store-paths.ts";import { readStreamRecords } from "./stream-read.ts";+import { quarantineStamp } from "../shard-quarantine.ts";/** One preference of the fold, with its privacy. */export interface StorePreference {@@ -77,6 +80,8 @@ export interface StoreEntities {readonly pinned: ReadonlyMap<StorePartition, PinnedFold>;/** Partition-relative paths of `events` shards that could not be read. */readonly unreadable: ReadonlyArray<string>;+ /** Whether a record was left out because its body is a pending blob. */+ readonly skippedPending: boolean;}interface Cached {@@ -88,8 +93,12 @@ const cache = new Map<string, Cached>();const EVENTS = { stream: "events" } as const;+/**+ * What a read depends on: every `events` shard, and the quarantine ledger,+ * which decides whether a shard reads whole without changing its bytes.+ */function fingerprintOf(store: StoreInspection): string {- const parts: string[] = [store.storeId];+ const parts: string[] = [store.storeId, `quarantine=${quarantineStamp(store.root)}`];for (const partition of [...store.partitions].toSorted()) {let shards: ReturnType<typeof listShards>;try {@@ -129,8 +138,10 @@ function build(store: StoreInspection): StoreEntities {const all: Array<{ record: StoreRecord; partition: StorePartition }> = [];const unreadable: string[] = [];const pinned = new Map<StorePartition, PinnedFold>();+ let skippedPending = false;for (const partition of store.partitions) {const read = readStreamRecords(store.root, partition, EVENTS);+ if (read.skippedPending) skippedPending = true;unreadable.push(...read.unreadable.map((rel) => `${partition}/${rel}`));for (const record of read.records) all.push({ record, partition });pinned.set(@@ -192,7 +203,7 @@ function build(store: StoreInspection): StoreEntities {partitions: new Set(entry.history.map((r) => partitionOf.get(r.id) ?? "shared")),});}- return { store, preferences, signals, pinned, unreadable };+ return { store, preferences, signals, pinned, unreadable, skippedPending };}/** The entities of the store at `store.root`, cached until an `events` shard changes. */@@ -202,8 +213,12 @@ export function readEntitiesOf(store: StoreInspection): StoreEntities {if (cached !== undefined && cached.fingerprint === fingerprint) return cached.entities;const entities = build(store);// A shard unreadable for a moment (a Windows sharing violation) keeps its- // size and mtime, so a cached incomplete read would never be retried.- if (entities.unreadable.length === 0) cache.set(store.root, { fingerprint, entities });+ // size and mtime, so a cached incomplete read would never be retried. A+ // read that left out a pending record is not cached either: when its blob+ // arrives no stamp of the fingerprint moves (a quarantined pending blob+ // opens no gap, so it is not in `unreadable`).+ if (entities.unreadable.length === 0 && !entities.skippedPending)+ cache.set(store.root, { fingerprint, entities });else cache.delete(store.root);return entities;}diff --git a/src/core/store/read/stream-read.ts b/src/core/store/read/stream-read.tsindex ac2a5ac3..516549a4 100644--- a/src/core/store/read/stream-read.ts+++ b/src/core/store/read/stream-read.ts@@ -35,6 +35,15 @@ export interface StreamRead {* this machine read before and that are gone now.*/readonly unreadable: ReadonlyArray<string>;+ /**+ * Whether a record was left out because its body is still a pending+ * blob. A quarantined pending blob opens no gap, so `unreadable` does+ * not name its shard; and when the blob arrives, neither the shard's+ * bytes nor the quarantine ledger change. A caller that caches a read+ * on those stamps must not cache one that skipped a pending record, or+ * it keeps answering without that record after the blob is delivered.+ */+ readonly skippedPending: boolean;}export interface StreamReadOptions {@@ -72,9 +81,10 @@ export function readStreamRecords(// A stream that cannot be listed (its folder, or any device folder// below it: EACCES, EBUSY) is named whole, as an unreadable shard is,// and none of its shards is read; the other streams still read.- return { records, unreadable: [streamPrefix(ref)] };+ return { records, unreadable: [streamPrefix(ref)], skippedPending: false };}const compactions: Compaction[] = [];+ let skippedPending = false;for (const shard of shards) {if (options.device !== undefined && shard.device !== options.device) continue;if (options.month !== undefined && !options.month(monthOf(shard.rel))) continue;@@ -93,6 +103,7 @@ export function readStreamRecords(compactions.push(...read.compactions);for (const entry of read.records) {if (entry.bodyState !== "pending") records.push(entry.record);+ else skippedPending = true;}}const gone = trackSeenShards(@@ -105,5 +116,5 @@ export function readStreamRecords(shards.map((shard) => shard.rel),replacedShards(compactions),);- return { records, unreadable: [...unreadable, ...gone] };+ return { records, unreadable: [...unreadable, ...gone], skippedPending };}diff --git a/src/core/store/reader.ts b/src/core/store/reader.tsindex b57c6984..da87e584 100644--- a/src/core/store/reader.ts+++ b/src/core/store/reader.ts@@ -30,7 +30,7 @@* consumed, never held back as a pending tail.*/-import { basename, dirname } from "node:path";+import { basename, dirname, relative } from "node:path";import { closeSync, fstatSync, readSync } from "node:fs";import { isValidDeviceId } from "../config.ts";@@ -53,6 +53,7 @@ import {type StoreRecord,} from "./envelope.ts";import { openRegularFileForRead } from "./safe-fs.ts";+import { lineSha256AtFd, quarantinedLines, type QuarantineKind } from "./shard-quarantine.ts";import { isShardFileName } from "./shard-list.ts";import { isTornObjectPrefix } from "./torn-line.ts";@@ -67,6 +68,12 @@ export interface ReadRecord {readonly offset: number;/** Byte length of the line, newline included. */readonly length: number;+ /**+ * A pending record whose line this machine quarantined (offset and line+ * hash both match, `shard-quarantine.ts`): still left out of every read,+ * but no gap.+ */+ readonly quarantined?: true;}export type ShardDiagnosticKind =@@ -86,6 +93,11 @@ export interface ShardDiagnostic {* machine read, so it hides nothing.*/readonly torn?: true;+ /**+ * A non-torn `corrupt-line` whose line this machine quarantined (offset+ * and line hash both match, `shard-quarantine.ts`): no gap.+ */+ readonly quarantined?: true;}/** One compaction a shard announced: partition-relative shard paths. */@@ -281,6 +293,47 @@ function readLine(return { kind: "record", record: { record, bodyState, offset: lineOffset, length }, compaction };}+/**+ * Mark the diagnostics and pending records this machine quarantined: only+ * on a match of both the offset and the hash of the line's bytes with one+ * ledger entry of the matching kind, never on the shard or the offset+ * alone. A `corrupt-line` entry marks only a corrupt line, a `blob-pending`+ * entry only a pending record, and a pending record of a `settings` shard+ * is never marked. The hash is taken from `fd`, the descriptor the read+ * parsed, so the bytes hashed are the bytes judged. A torn line needs no+ * mark and a newer-format line is never marked.+ */+function markQuarantined(+ fd: number,+ path: string,+ partitionRoot: string,+ records: ReadRecord[],+ diagnostics: ShardDiagnostic[],+): { records: ReadRecord[]; diagnostics: ShardDiagnostic[] } {+ const rel = relative(partitionRoot, path).split("\\").join("/");+ const ledger = quarantinedLines(partitionRoot, rel);+ if (ledger.size === 0) return { records, diagnostics };+ const settings = rel.split("/")[0] === "settings";+ const matches = (offset: number, kind: QuarantineKind): boolean => {+ const entry = ledger.get(offset);+ return (+ entry !== undefined && entry.kind === kind && lineSha256AtFd(fd, offset) === entry.sha256+ );+ };+ return {+ records: records.map((r) =>+ r.bodyState === "pending" && !settings && matches(r.offset, "blob-pending")+ ? { ...r, quarantined: true }+ : r,+ ),+ diagnostics: diagnostics.map((d) =>+ d.kind === "corrupt-line" && d.torn !== true && matches(d.offset, "corrupt-line")+ ? { ...d, quarantined: true }+ : d,+ ),+ };+}+/** Read and verify the shard at `path` from `options.offset`. */export function readShard(path: string, options: ReadShardOptions = {}): ShardRead {const start = options.offset ?? 0;@@ -399,9 +452,13 @@ export function readShard(path: string, options: ReadShardOptions = {}): ShardReconsumed = position;}if (options.partitionRoot !== undefined) observeRead(options.partitionRoot, records);+ const marked =+ options.partitionRoot === undefined+ ? { records, diagnostics }+ : markQuarantined(fd, path, options.partitionRoot, records, diagnostics);return {- records,- diagnostics,+ records: marked.records,+ diagnostics: marked.diagnostics,compactions,nextOffset: consumed,pendingTailBytes: size - consumed,@@ -423,7 +480,9 @@ export function readShard(path: string, options: ReadShardOptions = {}): ShardRe*/export function shardReadGap(read: ShardRead): string | null {const reasons: string[] = [];- const pending = read.records.filter((r) => r.bodyState === "pending").length;+ const pending = read.records.filter(+ (r) => r.bodyState === "pending" && r.quarantined !== true,+ ).length;if (pending > 0) {reasons.push(pending === 1@@ -432,7 +491,7 @@ export function shardReadGap(read: ShardRead): string | null {);}for (const d of read.diagnostics) {- if (d.torn === true) continue;+ if (d.torn === true || d.quarantined === true) continue;reasons.push(`${d.kind} at byte ${d.offset}: ${d.reason}`);}return reasons.length === 0 ? null : reasons.join("; ");diff --git a/src/core/store/seen-shards.ts b/src/core/store/seen-shards.tsindex c42b9c1f..236948c2 100644--- a/src/core/store/seen-shards.ts+++ b/src/core/store/seen-shards.ts@@ -25,7 +25,8 @@* between two processes only delays tracking a shard to the next read.* A surface that promises to write nothing (doctor, the status block)* runs inside {@link withoutSeenRecording}: it still reports a shard the- * ledger names that is gone, and records nothing.+ * ledger names that is gone, and records nothing. Leaving a partition+ * (`partition-leave.ts`) clears its entries.*/import { mkdirSync, readFileSync, statSync } from "node:fs";@@ -119,6 +120,51 @@ function load(path: string): Ledger {return ledger;}+/**+ * The ledger at `path`, read strictly: a file that exists but cannot be+ * read or parsed, or is not a version 1 ledger of the shape this release+ * writes, throws, so a caller that must clear the file's entries+ * (`forgetSeenPartition`) refuses instead of seeing zero entries and+ * reporting a clearing that did not happen. An absent file is zero+ * entries.+ */+function loadStrict(path: string): Ledger {+ let text: string;+ try {+ text = readFileSync(path, "utf8");+ } catch (err) {+ const code = (err as { code?: string } | null)?.code;+ if (code === "ENOENT") return new Map();+ throw err;+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(text);+ } catch (err) {+ throw new Error(+ `the seen-shards ledger ${path} cannot be parsed ` ++ `(${err instanceof Error ? err.message : String(err)})`,+ { cause: err },+ );+ }+ if (!isStrictLedger(parsed)) {+ throw new Error(`the seen-shards ledger ${path} is not a version 1 ledger this release reads`);+ }+ return parseLedger(text);+}++/** A version 1 ledger whose `shards` maps each partition to a list of shard names. */+function isStrictLedger(value: unknown): boolean {+ if (typeof value !== "object" || value === null || Array.isArray(value)) return false;+ const { version, shards } = value as { version?: unknown; shards?: unknown };+ if (version !== 1) return false;+ if (typeof shards !== "object" || shards === null || Array.isArray(shards)) return false;+ return Object.entries(shards).every(+ ([partition, rels]) =>+ isPartition(partition) && Array.isArray(rels) && rels.every((rel) => typeof rel === "string"),+ );+}+/** Apply `change` to the ledger on disk and write it back; never throws. */function update(path: string, change: (ledger: Ledger) => void): void {try {@@ -138,6 +184,17 @@ function update(path: string, change: (ledger: Ledger) => void): void {}}+/**+ * The device-local ledger file of the store `storeId`, keyed by the root:+ * two copies of one store on one machine (a moved root, a test bench of two+ * machines) never share a ledger.+ */+function seenLedgerFile(storeRoot: string, storeId: string): string {+ const rootKey = createHash("sha256").update(resolve(storeRoot)).digest("hex").slice(0, 16);+ const name = SEEN_SHARDS_FILE.replace(/\.json$/, `-${rootKey}.json`);+ return join(deviceStateDirFor(storeId, processDirsEnv()), name);+}+function ledgerPath(partitionRoot: string): { path: string; partition: StorePartition } | null {const partition = basename(partitionRoot);if (!isPartition(partition)) return null;@@ -148,14 +205,36 @@ function ledgerPath(partitionRoot: string): { path: string; partition: StorePartreturn null;}if (storeId === null) return null;- // Keyed by the root as well: two copies of one store on one machine (a- // moved root, a test bench of two machines) never share a ledger.- const rootKey = createHash("sha256")- .update(resolve(dirname(partitionRoot)))- .digest("hex")- .slice(0, 16);- const name = SEEN_SHARDS_FILE.replace(/\.json$/, `-${rootKey}.json`);- return { path: join(deviceStateDirFor(storeId, processDirsEnv()), name), partition };+ return { path: seenLedgerFile(dirname(partitionRoot), storeId), partition };+}++/**+ * Remove the partition's entries from this machine's seen-shards ledger of+ * the store `storeId`, and return how many there were. Checked, not+ * best-effort: the ledger is read strictly, so a file that exists but+ * cannot be read or parsed throws (the lenient `load` would answer an+ * empty map and report a clearing that did not happen), and the ledger's+ * `update` never throws, so this verifies the write landed before the+ * caller reports the partition cleared.+ */+export function forgetSeenPartition(+ storeRoot: string,+ storeId: string,+ partition: StorePartition,+): number {+ const path = seenLedgerFile(storeRoot, storeId);+ const count = loadStrict(path).get(partition)?.size ?? 0;+ if (count > 0) {+ update(path, (ledger) => {+ ledger.delete(partition);+ });+ if ((loadStrict(path).get(partition)?.size ?? 0) > 0) {+ throw new Error(+ `could not clear the seen-shard entries of the ${partition} partition in ${path}`,+ );+ }+ }+ return count;}function deviceOf(rel: string): string {diff --git a/src/core/store/shard-quarantine-write.ts b/src/core/store/shard-quarantine-write.tsnew file mode 100644index 00000000..3571aa4c--- /dev/null+++ b/src/core/store/shard-quarantine-write.ts@@ -0,0 +1,175 @@+/**+ * The write side of the shard quarantine ledger (change store-hardening,+ * design D4): record and release one line under the local lock. Kept apart+ * from `shard-quarantine.ts` so a reader of the ledger loads no lock module.+ */++import { existsSync, mkdirSync, readFileSync } from "node:fs";+import { join, resolve } from "node:path";++import { atomicWriteFileSync } from "../fs-atomic.ts";+import type { StorePartition } from "./constants.ts";+import { withLocalLock } from "./local-lock.ts";+import {+ compareLines,+ dropLoadedLedger,+ isLine,+ ledgerPath,+ QUARANTINE_LEDGER_VERSION,+ sameLine,+ SHARD_QUARANTINE_DIR,+ type QuarantineLine,+} from "./shard-quarantine.ts";+import { storesStateDir } from "./store-paths.ts";++/**+ * The ledger file exists but this release cannot rewrite it without losing+ * entries: it cannot be read, does not parse, has another `version`, or+ * holds an entry this release does not read (a newer build's kind, for+ * example). The write refuses instead of overwriting it. `remedy` replaces+ * the default advice when repairing the file is not the way out.+ */+export class QuarantineLedgerUnreadableError extends Error {+ constructor(+ readonly path: string,+ detail: string,+ remedy = "it is left as it is. Repair or remove it, then run the command again",+ ) {+ super(`the shard quarantine ledger ${path} ${detail}; ${remedy}`);+ this.name = "QuarantineLedgerUnreadableError";+ }+}++/** The error's code (`EACCES`, `EIO`, ...), or its message when it carries none. */+function readFailure(err: unknown): { code: string | null; text: string } {+ const code = (err as { code?: unknown } | null)?.code;+ return {+ code: typeof code === "string" ? code : null,+ text: err instanceof Error ? err.message : String(err),+ };+}++/** The ledger's entries, parsed strictly: absent is empty, anything unreadable throws. */+function loadStrict(path: string): QuarantineLine[] {+ if (!existsSync(path)) return [];+ let text: string;+ try {+ text = readFileSync(path, "utf8");+ } catch (err) {+ const failure = readFailure(err);+ if (failure.code === "ENOENT") {+ // Removed by something outside the lock between the check and the read.+ throw new QuarantineLedgerUnreadableError(+ path,+ "was removed while it was being read",+ "nothing was changed. Run the command again",+ );+ }+ throw new QuarantineLedgerUnreadableError(+ path,+ `could not be read (${failure.text})`,+ "it is left as it is. Check that this account can read it, then run the command again",+ );+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(text);+ } catch (err) {+ throw new QuarantineLedgerUnreadableError(+ path,+ `does not parse as JSON (${err instanceof Error ? err.message : String(err)})`,+ );+ }+ if (typeof parsed !== "object" || parsed === null) {+ throw new QuarantineLedgerUnreadableError(path, "is not a JSON object");+ }+ const { version, lines } = parsed as { version?: unknown; lines?: unknown };+ if (version !== QUARANTINE_LEDGER_VERSION) {+ throw new QuarantineLedgerUnreadableError(+ path,+ `has version ${String(version)}, not ${QUARANTINE_LEDGER_VERSION}`,+ );+ }+ if (!Array.isArray(lines)) {+ throw new QuarantineLedgerUnreadableError(path, "holds no list of lines");+ }+ if (!lines.every(isLine)) {+ throw new QuarantineLedgerUnreadableError(path, "holds an entry this release does not read");+ }+ return [...lines];+}++/** What a change did: `write` false leaves the file untouched. */+interface Change<T> {+ readonly result: T;+ readonly write: boolean;+}++/**+ * Apply `change` to the ledger of `root` under the local lock and write it+ * back when it changed something. Throws+ * {@link QuarantineLedgerUnreadableError} for a ledger it cannot rewrite.+ */+function update<T>(root: string, change: (lines: QuarantineLine[]) => Change<T>): T {+ const path = ledgerPath(root);+ mkdirSync(join(storesStateDir(), SHARD_QUARANTINE_DIR), { recursive: true });+ return withLocalLock(path, () => {+ const lines = loadStrict(path);+ const { result, write } = change(lines);+ // A no-op leaves the mtime alone, so no cache keyed on the stamp drops.+ if (!write) return result;+ const sorted = lines.toSorted(compareLines);+ atomicWriteFileSync(+ path,+ `${JSON.stringify({ version: QUARANTINE_LEDGER_VERSION, root: resolve(root), lines: sorted }, null, 2)}\n`,+ );+ dropLoadedLedger(path);+ return result;+ });+}++/**+ * Record `line` in the ledger of the store at `root`. `already` when the+ * ledger names the same line with the same bytes and the same kind; an+ * entry for the same offset with other bytes or another kind is replaced,+ * since the reader marks a line only for an entry of its own kind (a+ * `corrupt-line` entry never skips a pending record, nor the reverse).+ */+export function addQuarantine(+ root: string,+ line: Omit<QuarantineLine, "quarantined_at">,+ now: Date = new Date(),+): "quarantined" | "already" {+ return update(root, (lines) => {+ const at = lines.findIndex((l) => sameLine(l, line.partition, line.shard, line.offset));+ const same = at !== -1 && lines[at]!.sha256 === line.sha256 && lines[at]!.kind === line.kind;+ if (same) return { result: "already", write: false };+ const entry: QuarantineLine = {+ partition: line.partition,+ shard: line.shard,+ offset: line.offset,+ sha256: line.sha256,+ kind: line.kind,+ reason: line.reason,+ quarantined_at: now.toISOString(),+ };+ if (at === -1) lines.push(entry);+ else lines[at] = entry;+ return { result: "quarantined", write: true };+ });+}++/** Remove the entry of the line at `offset` of a shard; `not-quarantined` when there is none. */+export function releaseQuarantine(+ root: string,+ partition: StorePartition,+ shard: string,+ offset: number,+): "released" | "not-quarantined" {+ return update(root, (lines) => {+ const at = lines.findIndex((l) => sameLine(l, partition, shard, offset));+ if (at === -1) return { result: "not-quarantined", write: false };+ lines.splice(at, 1);+ return { result: "released", write: true };+ });+}diff --git a/src/core/store/shard-quarantine.ts b/src/core/store/shard-quarantine.tsnew file mode 100644index 00000000..60683a06--- /dev/null+++ b/src/core/store/shard-quarantine.ts@@ -0,0 +1,388 @@+/**+ * The device-local quarantine of shard lines (change store-hardening,+ * design D4; spec `store-shard-gaps`).+ *+ * A shard read only in part fails every protection read closed (capture,+ * settings, lane privacy, exact state, attestations, the OKF export, the+ * migration). When the gap is one bad line in another device's shard, the+ * operator can record on this machine that exactly that line is skipped:+ * the shard then reads as whole here, the line stays in the file, and no+ * other machine is affected.+ *+ * An entry is the line's byte offset and the hash of its bytes, from the+ * offset through its newline (newline included, read from the file). A+ * line over the store's line cap is a corrupt line too, so it is hashed as+ * well, up to {@link QUARANTINE_HASH_CAP_BYTES}: a longer run has no hash+ * (`null`), cannot be quarantined and stays a gap, which fails closed. The+ * reader hashes from the descriptor it already holds; every other caller+ * opens the shard through the reader's guarded open (no symbolic link, no+ * special file, no linked folder inside the partition).+ *+ * The reader marks a diagnostic or a pending record only when both match+ * one entry of the matching kind: a `corrupt-line` entry marks only a+ * corrupt line, a `blob-pending` entry only a pending record, and never in+ * a `settings` shard. The shard or the offset alone never marks a line, so+ * another bad line of the same shard, or other bytes at a quarantined+ * offset, stay gaps.+ *+ * The ledger is one file per store root under the device-local stores+ * state folder, never synced:+ * `<storesStateDir()>/shard-quarantine/<sha256(resolve(root))[0:32]>.json`.+ * The reader applies entries only from a ledger of the supported version+ * and fails closed on anything else ({@link appliedLines}).+ * Writes are read-modify-write under the local lock, so two processes+ * quarantining at once both land (`shard-quarantine-write.ts`, kept apart+ * so the readers that only ask for the ledger's stamp or lines load no+ * lock module); reads are cached by the file's size and mtime.+ *+ * This module parses no shard itself (the reader imports it), so+ * {@link listQuarantine} takes the shard reader as an argument.+ */++import { createHash } from "node:crypto";+import { closeSync, readFileSync, readSync, statSync } from "node:fs";+import { join, resolve } from "node:path";++import { STORE_LINE_CAP_BYTES, STORE_PARTITIONS, type StorePartition } from "./constants.ts";+import { parseHlc } from "./hlc.ts";+import { openRegularFileForRead } from "./safe-fs.ts";+import { storesStateDir } from "./store-paths.ts";++/** Folder of the ledgers under the stores state folder. */+export const SHARD_QUARANTINE_DIR = "shard-quarantine";++/** The ledger `version` this release reads and writes. */+export const QUARANTINE_LEDGER_VERSION = 1;++const DAY_MS = 24 * 60 * 60 * 1000;++/** Age from which a pending blob may be quarantined: seven days. */+export const PENDING_BLOB_QUARANTINE_AGE_MS = 7 * DAY_MS;++/** {@link PENDING_BLOB_QUARANTINE_AGE_MS} as message text, e.g. `7 days`. */+export const PENDING_BLOB_QUARANTINE_AGE_TEXT = `${PENDING_BLOB_QUARANTINE_AGE_MS / DAY_MS} days`;++/**+ * The instant a pending record was written, from its clock stamp, or `null`+ * when the stamp does not parse as a time (no stamp, or a physical part no+ * date can hold). Such a record has no age: it is never quarantined by age,+ * and nothing substitutes the current time for it.+ */+export function pendingRecordWrittenMs(hlc: string): number | null {+ const physical = parseHlc(hlc)?.physical;+ if (physical === undefined) return null;+ return Number.isNaN(new Date(physical).getTime()) ? null : physical;+}++export type QuarantineKind = "corrupt-line" | "blob-pending";++export interface QuarantineLine {+ readonly partition: StorePartition;+ /** Partition-relative shard path, `/`-separated. */+ readonly shard: string;+ readonly offset: number;+ /** Hash of the line's bytes from `offset` through its newline. */+ readonly sha256: string;+ readonly kind: QuarantineKind;+ readonly reason: string;+ readonly quarantined_at: string;+}++/** `held`: the same bytes still stand there and are still a gap; `resolved`: they now read as a record; `stale`: other bytes stand at the offset. */+export type QuarantineState = "held" | "resolved" | "stale";++export interface QuarantineListEntry extends QuarantineLine {+ readonly state: QuarantineState;+}++interface Ledger {+ readonly root: string;+ readonly lines: QuarantineLine[];+}++/** The ledger file of the store at `root`. */+export function ledgerPath(root: string): string {+ const key = createHash("sha256").update(resolve(root)).digest("hex").slice(0, 32);+ return join(storesStateDir(), SHARD_QUARANTINE_DIR, `${key}.json`);+}++function stampOf(path: string): string {+ const stat = statSync(path, { throwIfNoEntry: false });+ return stat === undefined ? "absent" : `${stat.size}:${stat.mtimeMs}`;+}++function isPartition(value: unknown): value is StorePartition {+ return (STORE_PARTITIONS as ReadonlyArray<unknown>).includes(value);+}++/** Whether `value` is a ledger entry this release reads. */+export function isLine(value: unknown): value is QuarantineLine {+ if (typeof value !== "object" || value === null) return false;+ const v = value as Record<string, unknown>;+ return (+ isPartition(v["partition"]) &&+ typeof v["shard"] === "string" &&+ typeof v["offset"] === "number" &&+ Number.isSafeInteger(v["offset"]) &&+ typeof v["sha256"] === "string" &&+ (v["kind"] === "corrupt-line" || v["kind"] === "blob-pending") &&+ typeof v["reason"] === "string" &&+ typeof v["quarantined_at"] === "string"+ );+}++const loaded = new Map<string, { readonly stamp: string; readonly ledger: Ledger }>();++/**+ * The entries the reader applies from a parsed ledger file. It fails+ * closed: anything but an object of version {@link QUARANTINE_LEDGER_VERSION}+ * with a `lines` list (another version, a non-object, no list) applies no+ * entry, so every line it names stays a gap. Within such a ledger each+ * entry stands on its own: one this release does not read (a newer+ * build's kind, a missing field) applies nothing, and the others still+ * apply. Skipping an entry only leaves its line a gap, so neither rule+ * ever makes a shard read as whole.+ */+export function appliedLines(parsed: unknown): QuarantineLine[] {+ if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) return [];+ const { version, lines } = parsed as { version?: unknown; lines?: unknown };+ if (version !== QUARANTINE_LEDGER_VERSION || !Array.isArray(lines)) return [];+ return lines.filter(isLine);+}++/**+ * The ledger of `root`, read again only when its file changed. An+ * unreadable ledger, or one {@link appliedLines} does not accept,+ * quarantines nothing.+ */+export function load(root: string): Ledger {+ const path = ledgerPath(root);+ let stamp: string;+ try {+ stamp = stampOf(path);+ } catch {+ return { root: resolve(root), lines: [] };+ }+ const hit = loaded.get(path);+ if (hit !== undefined && hit.stamp === stamp) return hit.ledger;+ let lines: QuarantineLine[] = [];+ if (stamp !== "absent") {+ try {+ lines = appliedLines(JSON.parse(readFileSync(path, "utf8")));+ } catch {+ lines = [];+ }+ }+ const ledger = { root: resolve(root), lines };+ loaded.set(path, { stamp, ledger });+ return ledger;+}++/** Forget the cached ledger at `path`, so the next read takes the file. */+export function dropLoadedLedger(path: string): void {+ loaded.delete(path);+}++/**+ * The size and mtime of the ledger of the store at `root` (`absent` when+ * there is none). A cache of a whole read keys on it beside the shards'+ * stamps: a quarantine or a release changes no shard byte, yet changes+ * whether a shard reads whole.+ */+export function quarantineStamp(root: string): string {+ try {+ return stampOf(ledgerPath(root));+ } catch {+ return "unreadable";+ }+}++/** The ledger's line order: partition, shard, offset. */+export function compareLines(a: QuarantineLine, b: QuarantineLine): number {+ if (a.partition !== b.partition) return a.partition < b.partition ? -1 : 1;+ if (a.shard !== b.shard) return a.shard < b.shard ? -1 : 1;+ return a.offset - b.offset;+}++/** Whether `line` is the entry of that offset of that shard. */+export const sameLine = (+ line: QuarantineLine,+ partition: StorePartition,+ shard: string,+ offset: number,+): boolean => line.partition === partition && line.shard === shard && line.offset === offset;++/** What the reader matches a line against: the entry's line hash and kind. */+export interface QuarantinedLine {+ readonly sha256: string;+ readonly kind: QuarantineKind;+}++/**+ * The quarantined lines of one shard, offset to line hash and kind, for+ * the reader. `partitionRoot` is `<store root>/<partition>`; `rel` the+ * shard's partition-relative path. Empty when the ledger names none.+ */+export function quarantinedLines(+ partitionRoot: string,+ rel: string,+): ReadonlyMap<number, QuarantinedLine> {+ const partition = resolve(partitionRoot).split(/[\\/]/).pop();+ if (!isPartition(partition)) return new Map();+ const out = new Map<number, QuarantinedLine>();+ for (const line of load(resolve(partitionRoot, "..")).lines) {+ if (line.partition === partition && line.shard === rel) {+ out.set(line.offset, { sha256: line.sha256, kind: line.kind });+ }+ }+ return out;+}++const HASH_CHUNK_BYTES = 64 * 1024;++/**+ * The longest line a quarantine entry can name: sixty-four times the store+ * line cap (1 MiB). The reader reports a run past the line cap as a corrupt+ * line, so such a line stays quarantinable up to this bound; past it the+ * hash is `null` and the line stays a gap. The bound keeps a hash from+ * reading without end (a file with no newline).+ */+export const QUARANTINE_HASH_CAP_BYTES = 64 * STORE_LINE_CAP_BYTES;++/**+ * What hashing one line found: its hash, or why there is none. `too-long`:+ * the line runs past {@link QUARANTINE_HASH_CAP_BYTES} with no newline, which+ * stays so until the bytes change. `unreadable`: the file holds nothing at+ * the offset, cannot be opened through the guarded open, or the read failed.+ */+export type LineHashProbe =+ | { readonly sha256: string }+ | { readonly failure: "too-long" | "unreadable" };++/**+ * Hash the bytes of the open file `fd` from `offset` through the next+ * newline (or the end of the file), newline included.+ */+export function probeLineHashAtFd(fd: number, offset: number): LineHashProbe {+ try {+ const hash = createHash("sha256");+ const chunk = Buffer.alloc(HASH_CHUNK_BYTES);+ let position = offset;+ let read = 0;+ for (;;) {+ const want = Math.min(chunk.length, QUARANTINE_HASH_CAP_BYTES - read + 1);+ const n = readSync(fd, chunk, 0, want, position);+ if (n === 0) break;+ const end = chunk.subarray(0, n).indexOf(0x0a);+ const take = end === -1 ? n : end + 1;+ if (read + take > QUARANTINE_HASH_CAP_BYTES) return { failure: "too-long" };+ hash.update(chunk.subarray(0, take));+ read += take;+ if (end !== -1) break;+ position += n;+ }+ return read === 0 ? { failure: "unreadable" } : { sha256: hash.digest("hex") };+ } catch {+ return { failure: "unreadable" };+ }+}++/**+ * {@link probeLineHashAtFd} of the shard at `path`, opened through the+ * reader's guarded open with its partition root: `unreadable` when the name+ * is a symbolic link, a special file, reached through a linked folder+ * inside the partition, or cannot be opened.+ */+export function probeLineHashAt(+ path: string,+ offset: number,+ partitionRoot: string,+): LineHashProbe {+ let fd: number | null;+ try {+ fd = openRegularFileForRead(path, partitionRoot);+ } catch {+ return { failure: "unreadable" };+ }+ if (fd === null) return { failure: "unreadable" };+ try {+ return probeLineHashAtFd(fd, offset);+ } finally {+ closeSync(fd);+ }+}++const hashOf = (probe: LineHashProbe): string | null => ("sha256" in probe ? probe.sha256 : null);++/**+ * The hash of the bytes of the open file `fd` from `offset` through the+ * next newline (or the end of the file), newline included; `null` when the+ * file holds nothing at `offset`, cannot be read, or the line runs past+ * {@link QUARANTINE_HASH_CAP_BYTES}.+ */+export function lineSha256AtFd(fd: number, offset: number): string | null {+ return hashOf(probeLineHashAtFd(fd, offset));+}++/**+ * {@link lineSha256AtFd} of the shard at `path`, opened through the+ * reader's guarded open with its partition root: `null` when the name is a+ * symbolic link, a special file, reached through a linked folder inside the+ * partition, or cannot be opened. `null` fails closed: it matches no entry.+ */+export function lineSha256At(path: string, offset: number, partitionRoot: string): string | null {+ return hashOf(probeLineHashAt(path, offset, partitionRoot));+}++/** What {@link listQuarantine} needs of a shard read: where its records are and which wait for a blob. */+export interface QuarantineShardRead {+ readonly records: ReadonlyArray<{ readonly offset: number; readonly bodyState: string }>;+}++/** The shard reader {@link listQuarantine} asks (`readShard` of `reader.ts`). */+export type QuarantineShardReader = (+ path: string,+ options: { readonly partitionRoot: string },+) => QuarantineShardRead;++/**+ * Every quarantined line of the store at `root`, with its state, in ledger+ * order. Each shard is parsed at most once per call (`read` runs once per+ * shard, and only for a shard with an entry whose bytes still match); the+ * staleness probe hashes each entry's line on its own, so it opens the+ * shard once per entry. A shard whose read throws+ * affects only its own entries: an entry whose bytes still match is then+ * reported `held` (the entry is still in force and the reader still skips+ * the line), never `resolved`.+ */+export function listQuarantine(+ root: string,+ read: QuarantineShardReader,+): ReadonlyArray<QuarantineListEntry> {+ const out: QuarantineListEntry[] = [];+ const reads = new Map<string, QuarantineShardRead | null>();+ const readOnce = (path: string, partitionRoot: string): QuarantineShardRead | null => {+ if (!reads.has(path)) {+ let result: QuarantineShardRead | null;+ try {+ result = read(path, { partitionRoot });+ } catch {+ result = null;+ }+ reads.set(path, result);+ }+ return reads.get(path) ?? null;+ };+ for (const line of load(root).lines.toSorted(compareLines)) {+ const partitionRoot = join(resolve(root), line.partition);+ const path = join(partitionRoot, ...line.shard.split("/"));+ let state: QuarantineState;+ if (lineSha256At(path, line.offset, partitionRoot) !== line.sha256) state = "stale";+ else {+ const record = readOnce(path, partitionRoot)?.records.find((r) => r.offset === line.offset);+ state = record !== undefined && record.bodyState !== "pending" ? "resolved" : "held";+ }+ out.push({ ...line, state });+ }+ return out;+}diff --git a/src/core/store/store-open.ts b/src/core/store/store-open.tsindex a90ee873..222ccb60 100644--- a/src/core/store/store-open.ts+++ b/src/core/store/store-open.ts@@ -89,13 +89,24 @@ export class HeldPartitionLostError extends StoreOpenError {readonly root: string,readonly partitions: ReadonlyArray<StorePartition>,) {+ // Leaving `shared` is leaving the store, so the verb refuses it: for a+ // lost `shared` the only remedy is the restore. With `shared` among the+ // lost partitions, leaving anything else cannot work either: the leave+ // reads the store id from `shared/store.json` before it clears anything,+ // so it refuses with `no-store-id` and changes nothing. Name only the+ // restores.+ const leave = partitions.includes("shared")+ ? ""+ : `, or run ${partitions+ .map((p) => `\`o2b store partition leave ${p}\``)+ .join(" and ")} to stop holding ${partitions.length > 1 ? "them" : "it"} on this machine`;super(`the ${partitions.join(" and ")} partition${partitions.length > 1 ? "s" : ""} of the ` +`brain store at ${root}, which this machine has held, no longer show${partitions.length > 1 ? "" : "s"} ${STORE_IDENTITY_FILE}; its records cannot be read, so the store is not opened. ` +`Restore ${partitions.map((p) => `${p}/${STORE_IDENTITY_FILE}`).join(" and ")} ` +- "from sync or a backup",+ `from sync or a backup${leave}`,);this.name = "HeldPartitionLostError";}diff --git a/src/core/store/store-write.ts b/src/core/store/store-write.tsindex d577d3c4..931b445d 100644--- a/src/core/store/store-write.ts+++ b/src/core/store/store-write.ts@@ -35,6 +35,7 @@ import {appendStoreRecords,builtRecords,dualWrite,+ failedOnlyInShared,storeFor,type DualWriteBuild,type DualWriteContext,@@ -212,6 +213,15 @@ export interface StoreWriteStatus {readonly last_reason: StoreWriteFailure | null;readonly last_error: string | null;readonly last_failure_at: string | null;+ /**+ * The failures known to concern only the `shared` partition+ * (`failedOnlyInShared` in `dual-write.ts`): the only failure counters a+ * caller below local reach may see; `failures` counts these and every+ * other one.+ */+ readonly shared_failures: number;+ readonly shared_last_error: string | null;+ readonly shared_last_failure_at: string | null;}const EMPTY_STATUS: StoreWriteStatus = Object.freeze({@@ -219,6 +229,9 @@ const EMPTY_STATUS: StoreWriteStatus = Object.freeze({last_reason: null,last_error: null,last_failure_at: null,+ shared_failures: 0,+ shared_last_error: null,+ shared_last_failure_at: null,});/** Path of the fail-open status file: beside the per-store state directories. */@@ -238,6 +251,11 @@ export function readStoreWriteStatus(path: string = storeWriteStatusPath()): Sto: null,last_error: typeof parsed.last_error === "string" ? parsed.last_error : null,last_failure_at: typeof parsed.last_failure_at === "string" ? parsed.last_failure_at : null,+ shared_failures: typeof parsed.shared_failures === "number" ? parsed.shared_failures : 0,+ shared_last_error:+ typeof parsed.shared_last_error === "string" ? parsed.shared_last_error : null,+ shared_last_failure_at:+ typeof parsed.shared_last_failure_at === "string" ? parsed.shared_last_failure_at : null,};} catch {return EMPTY_STATUS;@@ -258,16 +276,30 @@ function recordFailOpen(error: StoreWriteError): void {// stderr closed: nothing left to tell}}+ const shared = failedOnlyInShared(error);try {const path = storeWriteStatusPath();mkdirSync(dirname(path), { recursive: true });withLocalLock(path, () => {const current = readStoreWriteStatus(path);+ const at = new Date().toISOString();+ const message = error.message.slice(0, 500);const next: StoreWriteStatus = {failures: current.failures + 1,last_reason: error.reason,- last_error: error.message.slice(0, 500),- last_failure_at: new Date().toISOString(),+ last_error: message,+ last_failure_at: at,+ ...(shared+ ? {+ shared_failures: current.shared_failures + 1,+ shared_last_error: message,+ shared_last_failure_at: at,+ }+ : {+ shared_failures: current.shared_failures,+ shared_last_error: current.shared_last_error,+ shared_last_failure_at: current.shared_last_failure_at,+ }),};atomicWriteFileSync(path, `${JSON.stringify(next, null, 2)}\n`);});diff --git a/src/mcp/brain/analytics-tools.ts b/src/mcp/brain/analytics-tools.tsindex 874bddf6..a7c75ba6 100644--- a/src/mcp/brain/analytics-tools.ts+++ b/src/mcp/brain/analytics-tools.ts@@ -142,7 +142,11 @@ async function toolBrainBeliefEvolution(const target = hasPref? { prefId: (prefIdRaw as string).trim() }: { topic: (topicRaw as string).trim() };- const index = buildTimelineIndex(ctx.vault, {});+ // Every row is derived from the index and never passes `eventAtReach`,+ // so below local reach a private event must not enter the index at all:+ // its evidence row or transition would name only readable records.+ const refs = requestRefView(ctx);+ const index = buildTimelineIndex(ctx.vault, { withholdsPrivate: refs.withholdsPrivate === true });const evo = buildBeliefEvolution(index, ctx.vault, target);// A topic target fans out to every pref/ret carrying that topic, so// one `topic` argument reaches artifacts the caller never named@@ -155,7 +159,6 @@ async function toolBrainBeliefEvolution(// records answer to, and a pref_id target the caller cannot read is// answered as an absent one: no row at all.const view = gatedOwnerScopeView(ctx.vault, ctx.agentName);- const refs = requestRefView(ctx);const targetShown = !hasPref || recordReadable(refs, (prefIdRaw as string).trim());const shown = <T>(rows: ReadonlyArray<T>,@@ -220,15 +223,16 @@ async function toolBrainConceptSynthesis(// `unlinkedMentions[]` carries a VERBATIM line of the mentioning// page's body (a-label-is-not-a-boundary, U3). Both producers already// take the scope; passing it is the whole fix.- const cluster = buildConceptCluster(ctx.vault, targetId, {- includeUnlinked,- ownerScope: gatedOwnerScopeView(ctx.vault, ctx.agentName).scope,- });// At the caller's reach a linker or a mention whose source page the// caller cannot read is dropped, and a target it cannot read (under its// pref- or ret- spelling) is answered as an absent one: its own id as- // the title and an empty cluster.+ // the title and an empty cluster. A private log event links nothing.const refs = requestRefView(ctx);+ const cluster = buildConceptCluster(ctx.vault, targetId, {+ includeUnlinked,+ ownerScope: gatedOwnerScopeView(ctx.vault, ctx.agentName).scope,+ withholdsPrivate: refs.withholdsPrivate === true,+ });const targetShown = recordReadable(refs, cluster.targetId);return {vault_path: vaultPathField(ctx),diff --git a/src/mcp/brain/brief-tools.ts b/src/mcp/brain/brief-tools.tsindex de67fe9b..629329ff 100644--- a/src/mcp/brain/brief-tools.ts+++ b/src/mcp/brain/brief-tools.ts@@ -106,6 +106,7 @@ async function toolBrainMorningBrief(topK,...(agentScope !== undefined ? { agentScope } : {}),...(readable !== undefined ? { readable } : {}),+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,lookbackDays,...(maxCharsPerMemory !== undefined ? { maxCharsPerMemory } : {}),...(maxTotalChars !== undefined ? { maxTotalChars } : {}),@@ -361,7 +362,9 @@ async function toolBrainDailyBrief(const dateCoerced = coerceIsoTimestampOrDate("brain_daily_brief", "date", dateRaw, "date-only");const date = dateCoerced ?? new Date().toISOString().slice(0, 10);const cfg = loadTemporalConfigSafe(ctx.vault);- const index = buildTimelineIndex(ctx.vault, {});+ const index = buildTimelineIndex(ctx.vault, {+ withholdsPrivate: requestRefView(ctx).withholdsPrivate === true,+ });const brief = buildDailyBrief(index, ctx.vault, date, {offsetHours: cfg.daily_window_offset_hours,});@@ -411,7 +414,9 @@ async function toolBrainWeeklySynthesis();const weekEnd = weekEndCoerced ?? new Date().toISOString().slice(0, 10);const cfg = loadTemporalConfigSafe(ctx.vault);- const index = buildTimelineIndex(ctx.vault, {});+ const index = buildTimelineIndex(ctx.vault, {+ withholdsPrivate: requestRefView(ctx).withholdsPrivate === true,+ });const synth = buildWeeklySynthesis(index, ctx.vault, weekEnd, cfg);const atReach = periodRowsAtReach(ctx, index, {since: synth.windowStart,@@ -573,6 +578,7 @@ async function toolBrainOperatorSummary(...(dreamSummary ? { dreamSummary } : {}),...(topActionsN !== undefined ? { topActionsN } : {}),...(readable !== undefined ? { readable } : {}),+ withholdsPrivate: view.withholdsPrivate === true,...(view.filtersNothing? {}: {diff --git a/src/mcp/brain/context-tools.ts b/src/mcp/brain/context-tools.tsindex 75431dda..b42cd033 100644--- a/src/mcp/brain/context-tools.ts+++ b/src/mcp/brain/context-tools.ts@@ -436,7 +436,9 @@ async function toolBrainContext(ctx: ServerContext): Promise<Record<string, unkn// `computeMaintenanceOverdueFlag`'s docblock for the cost bound. This// is the always-loaded path, so the full `maintenance_debt` derivation// on `computeBrainStatus` never runs here.- const maintenanceOverdue = computeMaintenanceOverdueFlag(vault);+ const maintenanceOverdue = computeMaintenanceOverdueFlag(vault, {+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,+ });// The operator's standing rules, resolved before anything else this// surface assembles - INCLUDING the no-Brain return below. `brain_context`diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.tsindex 28324b24..1efb9187 100644--- a/src/mcp/brain/health-tools.ts+++ b/src/mcp/brain/health-tools.ts@@ -147,6 +147,7 @@ async function toolBrainDoctor(// local reach in the uncertain stream, by name (`doctor/check.ts`).// Resolved fail-closed: a context that proved nothing is remote.reach: contextReach(ctx),+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,});// Every issue names the artifact it is about - `path`, `target`, the@@ -241,6 +242,7 @@ async function toolBrainDoctor(: collectMaintenanceActions(vault, {onError: (e) => collectErrors.push(e),visible: view.visible,+ withholdsPrivate: view.withholdsPrivate === true,}),(a) => [a.target],)@@ -407,6 +409,7 @@ async function toolBrainStatus(...(ctx.configPath !== null ? { configPath: ctx.configPath } : {}),// A cited page the caller may not read at its reach answers as an absent one.readable: readableAtContextReach(ctx),+ withholdsPrivate: findingView.withholdsPrivate === true,...(findingView.filtersNothing? {}: { keepFinding: (finding) => findingView.row(...findingRefs(ctx.vault, finding)) }),diff --git a/src/mcp/brain/knowledge-tools.ts b/src/mcp/brain/knowledge-tools.tsindex 6ab37319..23b786c8 100644--- a/src/mcp/brain/knowledge-tools.ts+++ b/src/mcp/brain/knowledge-tools.ts@@ -454,6 +454,7 @@ async function toolBrainMocAudit(const report = auditMoc(ctx.vault, targetId, {ownerScope: gatedOwnerScopeView(ctx.vault, ctx.agentName).scope,readable: readableAtContextReach(ctx),+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,});return {vault_path: vaultPathField(ctx),@@ -600,7 +601,12 @@ function toolBrainIdeaDiscovery(// itself, before the ranking and the cap, so the list is the one a vault// without that page would give.const ideas = view.keep(- discoverIdeas(ctx.vault, { now, cap, include: readableAtContextReach(ctx) }),+ discoverIdeas(ctx.vault, {+ now,+ cap,+ include: readableAtContextReach(ctx),+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,+ }),(i) => i.sourceArtifacts,);let triggersCreated: number | undefined;diff --git a/src/mcp/brain/query-tools.ts b/src/mcp/brain/query-tools.tsindex 102609e3..9169c3c5 100644--- a/src/mcp/brain/query-tools.ts+++ b/src/mcp/brain/query-tools.ts@@ -340,11 +340,12 @@ async function toolBrainQuery(if (!queryView.visible(res.preference.id)) {throw new BrainNotFoundError(preference);}- emitQueryTelemetry(res.evidence.length > 0 ? "ok" : "empty", res.evidence.length);+ const evidence = logEntriesAtReach(queryView, res.evidence);+ emitQueryTelemetry(evidence.length > 0 ? "ok" : "empty", evidence.length);return {mode: "preference",preference: shapePreferencePayload(delivery, res.preference),- evidence: res.evidence.map(serializeLogEntry),+ evidence: evidence.map(serializeLogEntry),};} catch (exc) {if (exc instanceof BrainNotFoundError) {@@ -539,6 +540,12 @@ async function toolBrainWrites(const limit = coerceInt(args, "limit", BRAIN_WRITES_DEFAULT_LIMIT, 1, BRAIN_WRITES_MAX_LIMIT);const offset = cursorOffsetArg("brain_writes", args);+ // The log never opens the notes it names, so the reach rule is asked+ // per target here: at remote reach a write to a page reserved against+ // remote reads (or one whose file is gone - an unreadable claim fails+ // closed) is dropped before counting, so neither its path nor the+ // total betrays it. At local reach the view is the shared no-op.+ const view = reachView(ctx.vault, contextReach(ctx));const result = ledgerRead("brain_writes", () =>listNoteWrites(ctx.vault, {...(agent !== null ? { agent } : {}),@@ -547,14 +554,9 @@ async function toolBrainWrites(...(since !== null ? { since } : {}),...(until !== null ? { until } : {}),...(rawOp !== null ? { op: rawOp } : {}),+ withholdsPrivate: view.withholdsPrivate === true,}),);- // The log never opens the notes it names, so the reach rule is asked- // per target here: at remote reach a write to a page reserved against- // remote reads (or one whose file is gone - an unreadable claim fails- // closed) is dropped before counting, so neither its path nor the- // total betrays it. At local reach the view is the shared no-op.- const view = reachView(ctx.vault, contextReach(ctx));const writes = view.filtersNothing? result.writes: result.writes.filter((w) => view.visible(w.target));@@ -600,9 +602,12 @@ function planRevert(ctx: ServerContext, args: Record<string, unknown>): Record<s...selectorArg(args, "since"),...selectorArg(args, "until"),};+ const view = reachView(ctx.vault, contextReach(ctx));let plan;try {- plan = planNoteRevert(ctx.vault, selector);+ plan = planNoteRevert(ctx.vault, selector, {+ withholdsPrivate: view.withholdsPrivate === true,+ });} catch (err) {if (err instanceof NoteRevertError) {throw new MCPError(INVALID_PARAMS, `brain_writes: ${err.code}: ${err.message}`);@@ -621,7 +626,6 @@ function planRevert(ctx: ServerContext, args: Record<string, unknown>): Record<s// whenever the withheld page did. The CLI apply re-plans in full and// refuses this digest with `digest_mismatch`, the safe outcome for a// plan the caller could not fully see.- const view = reachView(ctx.vault, contextReach(ctx));const entries = view.filtersNothing? plan.entries: plan.entries.filter((entry) => view.visible(entry.target));@@ -677,6 +681,7 @@ async function toolBrainWriteDestinations();const offset = cursorOffsetArg("brain_write_destinations", args);+ const view = reachView(ctx.vault, contextReach(ctx));const result = ledgerRead("brain_write_destinations", () =>listWriteDestinations(ctx.vault, {...(agent !== null ? { agent } : {}),@@ -684,9 +689,9 @@ async function toolBrainWriteDestinations(...(path !== null ? { path } : {}),...(since !== null ? { since } : {}),...(until !== null ? { until } : {}),+ withholdsPrivate: view.withholdsPrivate === true,}),);- const view = reachView(ctx.vault, contextReach(ctx));const rows = view.filtersNothing? result.destinations: result.destinations.filter((d) => view.visible(d.target));@@ -827,7 +832,9 @@ async function toolBrainBacklinks(// published another owner's preference and retired ids to any caller// (a-label-is-not-a-boundary, U3). The scope is the gated one: no// argument exists on this tool, and under `off` the view hides nothing.- const index = buildBacklinkIndex(ctx.vault, gatedOwnerScopeView(ctx.vault, ctx.agentName).scope);+ const index = buildBacklinkIndex(ctx.vault, gatedOwnerScopeView(ctx.vault, ctx.agentName).scope, {+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,+ });// The same reasoning one rule over: a ref names the artifact that WROTE// it, so an index unfiltered by reach publishes the id of a page// reserved against remote reads. A withheld TARGET answers as an absentdiff --git a/src/mcp/brain/reach-events.ts b/src/mcp/brain/reach-events.tsindex 1d94992d..da3e3ad0 100644--- a/src/mcp/brain/reach-events.ts+++ b/src/mcp/brain/reach-events.ts@@ -44,9 +44,15 @@ export { eventAtReach, recordRefs };/*** The reference view for one request: unfiltered at local reach with the* ownership gate off, the request's reach and owner predicate otherwise.+ * Below local reach it also withholds every log event of the store's+ * `private` partition, whatever it names.*/export function requestRefView(ctx: ServerContext): ArtifactRefView {- return readerRefView(ctx.vault, readableAtContextReachOrUndefined(ctx));+ return readerRefView(+ ctx.vault,+ readableAtContextReachOrUndefined(ctx),+ contextReach(ctx) !== TRANSPORT_REACH.local,+ );}/**diff --git a/src/mcp/brain/recall-tools.ts b/src/mcp/brain/recall-tools.tsindex 308a74b8..61ba8d0f 100644--- a/src/mcp/brain/recall-tools.ts+++ b/src/mcp/brain/recall-tools.ts@@ -292,6 +292,7 @@ async function toolBrainRecallTelemetry(const readCost = coerceNonNegativeNumber("read_cost", args["read_cost"]);const writeHeavyRatio = coerceNonNegativeNumber("write_heavy_ratio", args["write_heavy_ratio"]);const meter = computeMemoryCostMeter(ctx.vault, {+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,...(filter.since !== undefined ? { since: filter.since } : {}),...(filter.until !== undefined ? { until: filter.until } : {}),...(writeCost !== undefined || readCost !== undefineddiff --git a/src/mcp/brain/review-tools.ts b/src/mcp/brain/review-tools.tsindex 9705d6a9..ecae3558 100644--- a/src/mcp/brain/review-tools.ts+++ b/src/mcp/brain/review-tools.ts@@ -231,12 +231,15 @@ async function toolBrainStaleScan(): Promise<Record<string, unknown>> {void _args;const cfg = loadTemporalConfigSafe(ctx.vault);- const index = buildTimelineIndex(ctx.vault, {});- const report = findStaleEntries(index, ctx.vault, cfg);// Every stale row names its artifact by id, topic and vault-relative// path (a-label-is-not-a-boundary, U3). Log shards are named by date// and shared by construction, so they carry no ownership to read.+ // Below local reach a private log event anchors no preference's age.const view = reviewView(ctx);+ const index = buildTimelineIndex(ctx.vault, {+ withholdsPrivate: view.withholdsPrivate === true,+ });+ const report = findStaleEntries(index, ctx.vault, cfg);return {vault_path: vaultPathField(ctx),thresholds: report.thresholds,diff --git a/src/mcp/resources.ts b/src/mcp/resources.tsindex 4186f85b..56c4c31a 100644--- a/src/mcp/resources.ts+++ b/src/mcp/resources.ts@@ -495,7 +495,9 @@ function readDigestLatest(ctx: ResourceContext, uri: string, view: RequestView):}function readStatus(ctx: ResourceContext, uri: string): ResourceContent {- const snapshot = computeBrainStatus(ctx.vault);+ const snapshot = computeBrainStatus(ctx.vault, {+ withholdsPrivate: resolvedTransportReach(ctx.reach) !== TRANSPORT_REACH.local,+ });return {uri,mimeType: MIME_MARKDOWN,@@ -748,7 +750,9 @@ function readBacklinks(// not-found path at all (an unknown target is a legitimate zero), so a// refusal here would be the one response shape that proves the page// exists. The echoed target is the caller's own argument.- const index = buildBacklinkIndex(ctx.vault, view.ownerScope);+ const index = buildBacklinkIndex(ctx.vault, view.ownerScope, {+ withholdsPrivate: view.refs.withholdsPrivate === true,+ });// Every spelling the target answers to across its retirement: a// retired reserved record is otherwise asked as `pref-x`, which names// no page once `ret-x` is the one on disk.diff --git a/src/mcp/tools.ts b/src/mcp/tools.tsindex 5431ce52..028e31cd 100644--- a/src/mcp/tools.ts+++ b/src/mcp/tools.ts@@ -27,6 +27,7 @@ import { resolvedSecretLiterals } from "../core/secret-resolver.ts";import type { ConfigDiscovery } from "../core/types.ts";import { REMOVED_TOOLS } from "../core/removed-surfaces.ts";import { computeBrainStatus } from "../core/brain/status.ts";+import { TRANSPORT_REACH } from "../core/graph/transport-reach.ts";import { doctor } from "../core/doctor.ts";import { nextCommandField } from "../core/brain/next-step.ts";import { collectRuntimeNotices } from "../core/brain/runtime-notices.ts";@@ -140,7 +141,11 @@ async function toolStatus(ctx: ServerContext): Promise<Record<string, unknown>>broken === null ? Object.keys(discovery.data).toSorted() : unresolvedField(broken);// Safe to call on a vault that has no Brain layer yet — returns// `present: false` with zero counts.- const brain = vaultExists ? computeBrainStatus(ctx.vault) : unexaminable;+ const brain = vaultExists+ ? computeBrainStatus(ctx.vault, {+ withholdsPrivate: contextReach(ctx) !== TRANSPORT_REACH.local,+ })+ : unexaminable;// On the broken branch this gate is unresolvable, so the block is built// rather than skipped: `buildSearchStatusBlock` resolves its own config// and degrades to `{ exists: false, error }` naming the same file. A skipdiff --git a/tests/cli/brain-attest.test.ts b/tests/cli/brain-attest.test.tsindex 70f3990e..6d56ab58 100644--- a/tests/cli/brain-attest.test.ts+++ b/tests/cli/brain-attest.test.ts@@ -22,7 +22,7 @@* the next does not.*/-import { afterEach, beforeEach, expect, test } from "bun:test";+import { afterEach, beforeEach, expect, setDefaultTimeout, test } from "bun:test";import {existsSync,mkdirSync,@@ -50,8 +50,12 @@ import { runCli as runCliRaw, type RunCliOptions } from "../helpers/run-cli.ts";*/const runCli = (args: ReadonlyArray<string>, opts: RunCliOptions = {}) =>runCliRaw(args, { stdin: "", ...opts });+import { CLI_SPAWN_BUDGET_MS } from "../helpers/cli-timeout.ts";import { fakeCredential } from "../helpers/fake-credentials.ts";+// Every test here runs the CLI, most of them several times.+setDefaultTimeout(CLI_SPAWN_BUDGET_MS);+/*** Passphrases assembled at runtime so no source line carries a literal* shaped like a real credential (the fake-credentials doctrine); thediff --git a/tests/cli/export-redaction.test.ts b/tests/cli/export-redaction.test.tsindex d231a4a9..dc6144b8 100644--- a/tests/cli/export-redaction.test.ts+++ b/tests/cli/export-redaction.test.ts@@ -262,8 +262,12 @@ describe("an export with nothing to redact is byte-identical", () => {for (const file of expected.files) {// `okf.json` carries `generated_at`, which moves between runs.if (file.path === "okf.json") continue;- expect(`${file.path}: ${readFileSync(join(out, file.path), "utf8")}`).toBe(- `${file.path}: ${file.contents}`,+ // A page's `generated.at` is stamped to the second, and the CLI run+ // and this in-process build may fall on either side of one.+ const unstamped = (text: string): string =>+ text.replace(/^(generated: \{.*\bat: ")[^"]+(")/m, "$1<at>$2");+ expect(`${file.path}: ${unstamped(readFileSync(join(out, file.path), "utf8"))}`).toBe(+ `${file.path}: ${unstamped(file.contents)}`,);}});diff --git a/tests/cli/store-partition.test.ts b/tests/cli/store-partition.test.tsnew file mode 100644index 00000000..74d2c0d7--- /dev/null+++ b/tests/cli/store-partition.test.ts@@ -0,0 +1,216 @@+/**+ * `o2b store partition leave` (design D7 item 5 of change `store-hardening`):+ * the CLI contract only. Each refusal reason exits 1 and names itself in+ * the `--json` envelope, answers exit 0, and a lost partition that names+ * the verb is what `store device list` prints. The state effects behind+ * the reasons are proven in `tests/core/store/partition-leave.test.ts`.+ */+import { afterEach, beforeEach, describe, expect, setDefaultTimeout, test } from "bun:test";+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { STORE_ROOT_ENV } from "../../src/core/store/constants.ts";+import { CLI_SPAWN_BUDGET_MS } from "../helpers/cli-timeout.ts";+import { runCli } from "../helpers/run-cli.ts";++// Every test here runs the CLI, most of them several times.+setDefaultTimeout(CLI_SPAWN_BUDGET_MS);++let parent: string;+let root: string;++beforeEach(() => {+ parent = mkdtempSync(join(tmpdir(), "osb-store-partition-"));+ root = join(parent, "store");+});+afterEach(() => rmSync(parent, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }));++function env(): Record<string, string> {+ return {+ [STORE_ROOT_ENV]: root,+ HOME: join(parent, "home"),+ XDG_STATE_HOME: join(parent, "state"),+ LOCALAPPDATA: join(parent, "state"),+ OPEN_SECOND_BRAIN_CONFIG: join(parent, "config.yaml"),+ O2B_DEVICE_ID: "dev-a",+ };+}++/** Every test starts from a store this machine initialized and opened. */+async function initedStore(): Promise<void> {+ const r = await runCli(["store", "init"], { env: env(), stdin: "" });+ expect(r.returncode).toBe(0);+}++describe("o2b store partition leave", () => {+ test("a lost private partition fails closed and names the verb", async () => {+ await initedStore();+ rmSync(join(root, "private"), { recursive: true, force: true });+ const r = await runCli(["store", "device", "list"], { env: env(), stdin: "" });+ expect(r.returncode).toBe(1);+ expect(r.stderr).toContain("o2b store partition leave private");+ });++ test("refusals exit 1 and name their reason", async () => {+ await initedStore();+ // name, setup, argv+ const cases: Array<{+ name: string;+ setup?: () => void;+ argv: Array<string>;+ }> = [+ { name: "shared", argv: ["store", "partition", "leave", "shared"] },+ { name: "still-held", argv: ["store", "partition", "leave", "private"] },+ {+ name: "files-left",+ setup: () => rmSync(join(root, "private", "store.json")),+ argv: ["store", "partition", "leave", "private"],+ },+ {+ name: "files-left on an unlistable folder",+ setup: () => {+ rmSync(join(root, "private"), { recursive: true, force: true });+ writeFileSync(join(root, "private"), "a sync conflict landed here\n");+ },+ argv: ["store", "partition", "leave", "private"],+ },+ {+ name: "no-store-id",+ setup: () => {+ rmSync(join(root, "private"), { recursive: true, force: true });+ writeFileSync(join(root, "shared", "store.json"), "{");+ },+ argv: ["store", "partition", "leave", "private"],+ },+ ];+ for (const { name, setup, argv } of cases) {+ setup?.();+ // eslint-disable-next-line no-await-in-loop -- one refusal at a time+ const text = await runCli(argv, { env: env(), stdin: "" });+ expect(text.returncode, name).toBe(1);+ expect(text.stderr.startsWith("error: "), name).toBe(true);+ expect(text.stdout, name).toBe("");+ // eslint-disable-next-line no-await-in-loop -- one refusal at a time+ const j = await runCli([...argv, "--json"], { env: env(), stdin: "" });+ expect(j.returncode, name).toBe(1);+ const body = JSON.parse(j.stdout) as { ok: boolean; reason: string };+ expect(body.ok, name).toBe(false);+ expect(body.reason, name).toBe(name.split(" ")[0] ?? name);+ }+ });++ test("--force turns the files-left refusal into a leave", async () => {+ await initedStore();+ rmSync(join(root, "private", "store.json"));+ const j = await runCli(["store", "partition", "leave", "private", "--force", "--json"], {+ env: env(),+ stdin: "",+ });+ expect(j.returncode).toBe(0);+ const body = JSON.parse(j.stdout) as { ok: boolean; left: boolean };+ expect(body.ok).toBe(true);+ expect(body.left).toBe(true);+ });++ test("a store root inside the vault is refused", async () => {+ await initedStore();+ const vault = join(parent, "vault");+ mkdirSync(vault, { recursive: true });+ writeFileSync(join(parent, "config.yaml"), `vault: ${vault}\n`);+ const e = env();+ e[STORE_ROOT_ENV] = join(vault, "store");+ const j = await runCli(["store", "partition", "leave", "private", "--json"], {+ env: e,+ stdin: "",+ });+ expect(j.returncode).toBe(1);+ const body = JSON.parse(j.stdout) as { ok: boolean; reason: string };+ expect(body.ok).toBe(false);+ expect(body.reason).toBe("store-root-in-vault");+ });++ test("a config that cannot be read is refused", async () => {+ await initedStore();+ mkdirSync(join(parent, "config.yaml"), { recursive: true });+ const j = await runCli(["store", "partition", "leave", "private", "--json"], {+ env: env(),+ stdin: "",+ });+ expect(j.returncode).toBe(1);+ const body = JSON.parse(j.stdout) as { ok: boolean; reason: string };+ expect(body.ok).toBe(false);+ expect(body.reason).toBe("config");+ });++ test("leaving private", async () => {+ await initedStore();+ rmSync(join(root, "private"), { recursive: true, force: true });+ const j = await runCli(["store", "partition", "leave", "private", "--json"], {+ env: env(),+ stdin: "",+ });+ expect(j.returncode).toBe(0);+ const body = JSON.parse(j.stdout) as {+ ok: boolean;+ left: boolean;+ partition: string;+ cleared: { held_marker: boolean };+ };+ expect(body.ok).toBe(true);+ expect(body.left).toBe(true);+ expect(body.partition).toBe("private");+ expect(body.cleared.held_marker).toBe(true);+ const list = await runCli(["store", "device", "list"], { env: env(), stdin: "" });+ expect(list.returncode).toBe(0);+ });++ test("the text answer", async () => {+ await initedStore();+ rmSync(join(root, "private"), { recursive: true, force: true });+ const r = await runCli(["store", "partition", "leave", "private"], { env: env(), stdin: "" });+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("Left the private partition");+ expect(r.stdout).toContain("next catch-up");+ expect(r.stdout).toContain("holds the private partition again");+ });++ test("leaving twice", async () => {+ await initedStore();+ rmSync(join(root, "private"), { recursive: true, force: true });+ await runCli(["store", "partition", "leave", "private", "--json"], { env: env(), stdin: "" });+ const second = await runCli(["store", "partition", "leave", "private", "--json"], {+ env: env(),+ stdin: "",+ });+ expect(second.returncode).toBe(0);+ const body = JSON.parse(second.stdout) as { left: boolean };+ expect(body.left).toBe(false);+ const text = await runCli(["store", "partition", "leave", "private"], {+ env: env(),+ stdin: "",+ });+ expect(text.stdout).toContain("nothing to leave");+ });++ test("usage errors", async () => {+ await initedStore();+ const cases: Array<Array<string>> = [+ ["store", "partition", "leave"],+ ["store", "partition", "leave", "bogus"],+ ["store", "partition", "leave", "private", "extra"],+ ["store", "partition"],+ ];+ for (const argv of cases) {+ const r = await runCli(argv, { env: env(), stdin: "" });+ expect(r.returncode).toBe(2);+ expect(r.stderr.startsWith("error: ")).toBe(true);+ expect(r.stderr).toContain("usage: o2b store partition leave");+ const j = await runCli([...argv, "--json"], { env: env(), stdin: "" });+ const body = JSON.parse(j.stdout) as { ok: boolean; reason: string; error: string };+ expect(body.ok).toBe(false);+ expect(body.reason).toBe("usage");+ expect(typeof body.error).toBe("string");+ }+ });+});diff --git a/tests/cli/store-shard.test.ts b/tests/cli/store-shard.test.tsnew file mode 100644index 00000000..a75466a7--- /dev/null+++ b/tests/cli/store-shard.test.ts@@ -0,0 +1,490 @@+/**+ * `o2b store shard quarantine|list|release` (change store-hardening,+ * design D6; spec `store-shard-gaps`): the device-local quarantine of one+ * bad line of a shard another device owns, its listing and its release,+ * every refusal reason under `--json`, and the usage errors.+ */+import { afterEach, beforeEach, describe, expect, setDefaultTimeout, test } from "bun:test";+import {+ chmodSync,+ mkdirSync,+ mkdtempSync,+ readdirSync,+ readFileSync,+ renameSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join, relative } from "node:path";++import { readListedShard } from "../../src/cli/store-shard.ts";+import { brainSettingKey } from "../../src/core/store-migration/brain-settings.ts";+import { appendRecord } from "../../src/core/store/appender.ts";+import {+ SETTING_SET_KIND,+ STORE_IDENTITY_FILE,+ STORE_ROOT_ENV,+} from "../../src/core/store/constants.ts";+import { parseHlc } from "../../src/core/store/hlc.ts";+import {+ QUARANTINE_HASH_CAP_BYTES,+ SHARD_QUARANTINE_DIR,+} from "../../src/core/store/shard-quarantine.ts";+import { openStore } from "../../src/core/store/store-open.ts";+import { partitionDir, storesStateDir } from "../../src/core/store/store-paths.ts";+import { CLI_SPAWN_BUDGET_MS } from "../helpers/cli-timeout.ts";+import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";+import { runCli } from "../helpers/run-cli.ts";++// Every test here runs the CLI, most of them several times.+setDefaultTimeout(CLI_SPAWN_BUDGET_MS);++let parent: string;+let root: string;++beforeEach(() => {+ parent = mkdtempSync(join(tmpdir(), "osb-store-shard-"));+ root = join(parent, "store");+});+afterEach(() => rmSync(parent, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }));++function env(): Record<string, string> {+ return {+ [STORE_ROOT_ENV]: root,+ HOME: join(parent, "home"),+ XDG_STATE_HOME: join(parent, "state"),+ LOCALAPPDATA: join(parent, "state"),+ OPEN_SECOND_BRAIN_CONFIG: join(parent, "config.yaml"),+ O2B_DEVICE_ID: "dev-a",+ };+}++const DAY_MS = 24 * 60 * 60 * 1000;++const cli = (args: ReadonlyArray<string>) =>+ runCli(["store", "shard", ...args], { env: env(), stdin: "" });++async function init(partitions?: string): Promise<void> {+ const args = ["store", "init", ...(partitions === undefined ? [] : ["--partitions", partitions])];+ const r = await runCli(args, { env: env(), stdin: "" });+ expect(r.returncode).toBe(0);+}++/** Append one record as `device`, with that device's own local state, `ageMs` ago. */+function write(+ device: string,+ stream: "events" | "settings",+ body: Record<string, unknown>,+ ageMs = 0,+): string {+ const store = openStore({+ root,+ device,+ dirs: {+ platform: process.platform,+ home: join(parent, "home", device),+ env: {+ XDG_STATE_HOME: join(parent, "state", device),+ LOCALAPPDATA: join(parent, "state", device),+ },+ },+ });+ const kind = stream === "settings" ? SETTING_SET_KIND : "note";+ return appendRecord(+ store,+ { partition: "shared", stream, kind, agent: "t", body },+ { now: () => Date.now() - ageMs },+ ).path;+}++/** Three notes of `device` in one shard. */+function threeNotes(device = "dev-b"): string {+ let path = "";+ for (let n = 1; n <= 3; n += 1) path = write(device, "events", { n });+ return path;+}++const shardArg = (path: string): string =>+ `shared/${relative(partitionDir(root, "shared"), path).split("\\").join("/")}`;++function offsetOfLine(path: string, index: number): number {+ const lines = readFileSync(path, "utf8").split("\n");+ return lines.slice(0, index).reduce((sum, line) => sum + Buffer.byteLength(line) + 1, 0);+}++function editLine(path: string, index: number, edit: (line: string) => string): void {+ const lines = readFileSync(path, "utf8").split("\n");+ lines[index] = edit(lines[index]!);+ writeFileSync(path, lines.join("\n"));+}++const breakLine = (line: string): string => line.replace(/"n":(\d+)/, '"n":9$1');++/** Move the shared blobs aside, so every blob body is pending. */+function hideBlobs(): void {+ renameSync(join(partitionDir(root, "shared"), "blobs"), join(parent, "blobs-aside"));+}++/** The store identity file of `partition`. */+const identity = (partition: "shared" | "private"): string =>+ join(partitionDir(root, partition), STORE_IDENTITY_FILE);++async function refusal(args: ReadonlyArray<string>): Promise<string> {+ const r = await cli([...args, "--json"]);+ expect(r.returncode).toBe(1);+ const body = JSON.parse(r.stdout) as { ok: boolean; reason: string; error: unknown };+ expect(body.ok).toBe(false);+ expect(typeof body.error).toBe("string");+ return body.reason;+}++describe("o2b store shard", () => {+ test("quarantine, already, list, release and not-quarantined", async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, breakLine);+ const shard = shardArg(path);+ const offset = String(offsetOfLine(path, 1));++ const first = await cli(["quarantine", shard, "--offset", offset, "--json"]);+ expect(first.returncode).toBe(0);+ expect(JSON.parse(first.stdout)).toEqual({+ ok: true,+ status: "quarantined",+ partition: "shared",+ shard: shard.slice("shared/".length),+ offset: Number(offset),+ kind: "corrupt-line",+ });++ const again = await cli(["quarantine", shard, "--offset", offset]);+ expect(again.returncode).toBe(0);+ expect(again.stdout).toContain("already quarantined");++ const listed = await cli(["list", "--json"]);+ expect(listed.returncode).toBe(0);+ const body = JSON.parse(listed.stdout) as {+ ok: boolean;+ lines: Array<Record<string, unknown>>;+ };+ expect(body.ok).toBe(true);+ expect(body.lines).toHaveLength(1);+ expect(body.lines[0]).toMatchObject({+ partition: "shared",+ shard: shard.slice("shared/".length),+ offset: Number(offset),+ kind: "corrupt-line",+ state: "held",+ });+ expect(typeof body.lines[0]!["quarantined_at"]).toBe("string");+ const text = await cli(["list"]);+ expect(text.stdout).toContain(`${shard} at byte ${offset}`);++ const released = await cli(["release", shard, "--offset", offset, "--json"]);+ expect(released.returncode).toBe(0);+ expect(JSON.parse(released.stdout)).toMatchObject({ ok: true, status: "released" });++ const none = await cli(["release", shard, "--offset", offset]);+ expect(none.returncode).toBe(0);+ expect(none.stdout).toContain("Nothing was quarantined");+ const noneJson = await cli(["release", shard, "--offset", offset, "--json"]);+ expect(JSON.parse(noneJson.stdout)).toMatchObject({ ok: true, status: "not-quarantined" });++ const empty = await cli(["list", "--json"]);+ expect(JSON.parse(empty.stdout)).toEqual({ ok: true, lines: [] });+ }, 60_000);++ test("an old pending blob is quarantined as blob-pending", async () => {+ await init();+ // A device's clock never runs back, so the old record is its first.+ const path = write("dev-b", "events", { n: 1, text: "x".repeat(20_000) }, 8 * DAY_MS);+ write("dev-b", "events", { n: 2 });+ hideBlobs();+ const r = await cli(["quarantine", shardArg(path), "--offset", "0", "--json"]);+ expect(r.returncode).toBe(0);+ expect(JSON.parse(r.stdout)).toMatchObject({ status: "quarantined", kind: "blob-pending" });+ // The blob arrives: the entry is resolved, and its release reopens nothing.+ renameSync(join(parent, "blobs-aside"), join(partitionDir(root, "shared"), "blobs"));+ const listed = JSON.parse((await cli(["list", "--json"])).stdout) as {+ lines: Array<{ state: string }>;+ };+ expect(listed.lines.map((l) => l.state)).toEqual(["resolved"]);+ const released = await cli(["release", shardArg(path), "--offset", "0"]);+ expect(released.returncode).toBe(0);+ expect(released.stdout).toContain("its mark is removed");+ expect(released.stdout).toContain("a gap again only if it still is one");+ expect(released.stdout).not.toContain("read only in part again");+ }, 60_000);++ test("each refusal has its own reason", async () => {+ const missing = "shared/events/dev-b/2026-11.jsonl";+ expect(await refusal(["quarantine", missing, "--offset", "0"])).toBe("no-store");+ expect(await refusal(["list"])).toBe("no-store");++ await init("shared");+ expect(+ await refusal(["quarantine", "private/events/dev-b/2026-11.jsonl", "--offset", "0"]),+ ).toBe("not-held");+ expect(+ await refusal(["quarantine", "shared/events/dev-b/1999-01.jsonl", "--offset", "0"]),+ ).toBe("unknown-shard");+ expect(await refusal(["quarantine", "shared/../store.json", "--offset", "0"])).toBe(+ "unknown-shard",+ );++ const notes = threeNotes();+ expect(+ await refusal(["quarantine", shardArg(notes), "--offset", String(offsetOfLine(notes, 1))]),+ ).toBe("no-gap");+ editLine(notes, 1, (line) => line.replace('"v":1', '"v":99'));+ expect(+ await refusal(["quarantine", shardArg(notes), "--offset", String(offsetOfLine(notes, 1))]),+ ).toBe("newer-format");++ write("dev-c", "events", { n: 1 });+ const young = write("dev-c", "events", { n: 2, text: "y".repeat(20_000) }, DAY_MS);+ // A device's clock never runs back, so the old setting is its device's+ // first record: the settings rule is then the only barrier, not its age.+ const setting = write(+ "dev-d",+ "settings",+ {+ authority: "human",+ key: brainSettingKey("dream", "candidate_threshold"),+ value: "z".repeat(20_000),+ },+ 8 * DAY_MS,+ );+ const settingHlc = (+ JSON.parse(readFileSync(setting, "utf8").split("\n")[0]!) as { hlc: string }+ ).hlc;+ expect(Date.now() - parseHlc(settingHlc)!.physical).toBeGreaterThanOrEqual(7 * DAY_MS);+ // The last line of this machine's own shard: an old pending blob.+ const own = write("dev-a", "events", { n: 1, text: "o".repeat(20_000) }, 8 * DAY_MS);+ hideBlobs();+ expect(await refusal(["quarantine", shardArg(own), "--offset", "0"])).toBe("own-tail");+ expect(+ await refusal(["quarantine", shardArg(young), "--offset", String(offsetOfLine(young, 1))]),+ ).toBe("blob-too-young");+ expect(await refusal(["quarantine", shardArg(setting), "--offset", "0"])).toBe("settings-blob");+ }, 60_000);++ test("a ledger this release cannot rewrite answers ledger-unreadable and is kept", async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, breakLine);+ const shard = shardArg(path);+ const offset = String(offsetOfLine(path, 1));+ expect((await cli(["quarantine", shard, "--offset", offset])).returncode).toBe(0);+ const ledgerDir = join(+ storesStateDir({ platform: process.platform, home: join(parent, "home"), env: env() }),+ SHARD_QUARANTINE_DIR,+ );+ const files = readdirSync(ledgerDir).filter((f) => f.endsWith(".json"));+ expect(files).toHaveLength(1);+ const ledger = join(ledgerDir, files[0]!);+ const broken = JSON.stringify({ version: 2, lines: [] });+ writeFileSync(ledger, broken);+ expect(await refusal(["release", shard, "--offset", offset])).toBe("ledger-unreadable");+ expect(await refusal(["quarantine", shard, "--offset", offset])).toBe("ledger-unreadable");+ expect(readFileSync(ledger, "utf8")).toBe(broken);+ }, 60_000);++ test("a ledger lock held past the wait answers ledger-busy; the release message claims no gap", async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, breakLine);+ const shard = shardArg(path);+ const offset = String(offsetOfLine(path, 1));+ expect((await cli(["quarantine", shard, "--offset", offset])).returncode).toBe(0);+ const ledgerDir = join(+ storesStateDir({ platform: process.platform, home: join(parent, "home"), env: env() }),+ SHARD_QUARANTINE_DIR,+ );+ const ledger = join(+ ledgerDir,+ readdirSync(ledgerDir).find((f) => f.endsWith(".json"))!,+ );+ // Another process holds the ledger's lock: a fresh lock directory, far+ // from stale, and a wait budget much shorter than the stale age.+ mkdirSync(`${ledger}.lock`);+ const busy = async (args: ReadonlyArray<string>) => {+ const r = await runCli(["store", "shard", ...args, "--json"], {+ env: { ...env(), OPEN_SECOND_BRAIN_LOCK_WAIT_MS: "100" },+ stdin: "",+ });+ expect(r.returncode).toBe(1);+ return JSON.parse(r.stdout) as { ok: boolean; reason: string; error: string };+ };+ for (const verb of ["release", "quarantine"]) {+ const body = await busy([verb, shard, "--offset", offset]);+ expect(body).toMatchObject({ ok: false, reason: "ledger-busy" });+ expect(body.error).toContain("run the command again");+ }+ rmSync(`${ledger}.lock`, { recursive: true });+ const released = await cli(["release", shard, "--offset", offset]);+ expect(released.returncode).toBe(0);+ expect(released.stdout).toContain("its mark is removed");+ expect(released.stdout).toContain("a gap again only if it still is one");+ expect(released.stdout).not.toContain("read only in part again");+ }, 60_000);++ test("a shard removed between the listing and the read answers shard-changed", async () => {+ await init();+ const path = threeNotes();+ rmSync(path);+ let caught: unknown;+ try {+ readListedShard(path, partitionDir(root, "shared"), shardArg(path));+ } catch (err) {+ caught = err;+ }+ expect(caught).toBeInstanceOf(Error);+ expect((caught as { reason?: unknown }).reason).toBe("shard-changed");+ expect((caught as Error).message).toContain("run the command again");+ });++ test.skipIf(process.platform === "win32")(+ "a shard that is a link answers shard-unreadable, not no-gap",+ async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, breakLine);+ const offset = String(offsetOfLine(path, 1));+ const moved = join(parent, "moved.jsonl");+ renameSync(path, moved);+ symlinkSync(moved, path);+ const r = await cli(["quarantine", shardArg(path), "--offset", offset, "--json"]);+ expect(r.returncode).toBe(1);+ expect((JSON.parse(r.stdout) as { reason: string }).reason).toBe("shard-unreadable");+ },+ 30_000,+ );++ test.skipIf(CHMOD_CANNOT_DENY)(+ "a shard this account cannot read answers shard-unreadable and records nothing",+ async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, () => "{ not json");+ const offset = String(offsetOfLine(path, 1));+ chmodSync(path, 0o000);+ try {+ const r = await cli(["quarantine", shardArg(path), "--offset", offset, "--json"]);+ expect(r.returncode).toBe(1);+ const body = JSON.parse(r.stdout) as { ok: boolean; reason: string; error: string };+ expect(body.ok).toBe(false);+ expect(body.reason).toBe("shard-unreadable");+ } finally {+ chmodSync(path, 0o644);+ }+ const listed = await cli(["list", "--json"]);+ expect((JSON.parse(listed.stdout) as { lines: unknown[] }).lines).toEqual([]);+ },+ 30_000,+ );++ test("a corrupt line too long to hash is refused as line-too-long, with no advice to retry", async () => {+ await init();+ const path = threeNotes();+ editLine(path, 1, () => "x".repeat(QUARANTINE_HASH_CAP_BYTES + 1));+ const r = await cli([+ "quarantine",+ shardArg(path),+ "--offset",+ String(offsetOfLine(path, 1)),+ "--json",+ ]);+ expect(r.returncode).toBe(1);+ const body = JSON.parse(r.stdout) as { reason: string; error: string };+ expect(body.reason).toBe("line-too-long");+ expect(body.error).toContain(`runs past ${QUARANTINE_HASH_CAP_BYTES} bytes`);+ expect(body.error).not.toContain("run the command again");+ });++ test("a held corrupt last line of this machine's own shard answers own-tail", async () => {+ await init();+ write("dev-a", "events", { n: 1 });+ const own = write("dev-a", "events", { n: 2 });+ editLine(own, 1, breakLine);+ expect(+ await refusal(["quarantine", shardArg(own), "--offset", String(offsetOfLine(own, 1))]),+ ).toBe("own-tail");+ // Another device's held last line is no gap yet: the reader retries it.+ const theirs = threeNotes();+ editLine(theirs, 2, breakLine);+ expect(+ await refusal(["quarantine", shardArg(theirs), "--offset", String(offsetOfLine(theirs, 2))]),+ ).toBe("no-gap");+ });++ test("a pending record whose clock stamp does not parse answers blob-clock-unreadable", async () => {+ await init();+ const path = write("dev-b", "events", { n: 1, text: "x".repeat(20_000) }, 8 * DAY_MS);+ write("dev-b", "events", { n: 2 });+ editLine(path, 0, (line) =>+ line.replace(/"hlc":"[^"]*"/, `"hlc":"${Number.MAX_SAFE_INTEGER}.0"`),+ );+ hideBlobs();+ const r = await cli(["quarantine", shardArg(path), "--offset", "0", "--json"]);+ expect(r.returncode).toBe(1);+ const body = JSON.parse(r.stdout) as { reason: string; error: string };+ expect(body.reason).toBe("blob-clock-unreadable");+ expect(body.error).toContain("clock stamp does not parse");+ });++ test("store open errors answer partition-lost or store-open-error", async () => {+ await init();+ const kept = readFileSync(identity("private"), "utf8");+ rmSync(identity("private"));+ expect(await refusal(["list"])).toBe("partition-lost");+ writeFileSync(identity("private"), kept);+ writeFileSync(identity("shared"), "not a store identity");+ expect(await refusal(["list"])).toBe("store-open-error");+ });++ test("a store root inside the vault and an unreadable config are refused", async () => {+ const vault = join(parent, "vault");+ mkdirSync(vault, { recursive: true });+ writeFileSync(join(parent, "config.yaml"), `vault: ${vault}\n`);+ const inside = await runCli(["store", "shard", "list", "--json"], {+ env: { ...env(), [STORE_ROOT_ENV]: join(vault, "store") },+ stdin: "",+ });+ expect(inside.returncode).toBe(1);+ expect(JSON.parse(inside.stdout)).toMatchObject({ ok: false, reason: "store-root-in-vault" });++ rmSync(join(parent, "config.yaml"));+ mkdirSync(join(parent, "config.yaml"));+ expect(await refusal(["list"])).toBe("config");+ });++ test("usage errors exit 2, with a JSON answer under --json", async () => {+ const cases: ReadonlyArray<ReadonlyArray<string>> = [+ [],+ ["bogus"],+ ["quarantine"],+ ["quarantine", "shared/events/dev-b/2026-11.jsonl"],+ ["quarantine", "shared/events/dev-b/2026-11.jsonl", "--offset", "x"],+ ["quarantine", "shared/events/dev-b/2026-11.jsonl", "--offset", "-1"],+ ["quarantine", "nowhere/events/dev-b/2026-11.jsonl", "--offset", "0"],+ ["quarantine", "shared", "--offset", "0"],+ ["quarantine", "shared/events/dev-b/2026-11.jsonl", "extra", "--offset", "0"],+ ["release", "shared/events/dev-b/2026-11.jsonl"],+ ["list", "extra"],+ ];+ for (const args of cases) {+ const text = await cli(args);+ expect(`${args.join(" ")}: ${text.returncode}`).toBe(`${args.join(" ")}: 2`);+ expect(text.stderr.startsWith("error: ")).toBe(true);+ expect(text.stderr).toContain("usage: o2b store shard");+ const json = await cli([...args, "--json"]);+ expect(json.returncode).toBe(2);+ const body = JSON.parse(json.stdout) as { ok: boolean; reason: string; error: unknown };+ expect(body).toMatchObject({ ok: false, reason: "usage" });+ expect(typeof body.error).toBe("string");+ }+ }, 60_000);+});diff --git a/tests/core/architecture/log-reader-census.test.ts b/tests/core/architecture/log-reader-census.test.tsnew file mode 100644index 00000000..bf121775--- /dev/null+++ b/tests/core/architecture/log-reader-census.test.ts@@ -0,0 +1,294 @@+/**+ * The log-reader census (change store-hardening, design D3; spec+ * `remote-log-reads`): every module that calls a log reader is classified,+ * so a new reader of the log cannot reach a remote caller without saying+ * whether it filters the events of the store's `private` partition.+ *+ * The walk finds every call of a log reader in `src/**\/*.ts` (read off the+ * code view, so a name inside a comment or a string is not a call, and a+ * `function <name>(` declaration is not one either), including a call+ * through a name a named import binds the reader to+ * (`import { storeLogDays as days }`) and requires each+ * calling module to be a key of {@link LOG_READERS}. A calling module the+ * table does not list fails, naming it; a table key that no longer calls a+ * reader fails as stale.+ *+ * The classes:+ * - `at-reach`: a surface below local reach reads it, and it leaves out+ * the private events there. For the surfaces of design D1 and the D2+ * rows, the behaviour is proved per surface in+ * `tests/mcp/log-private-reach-surfaces.test.ts`; this census proves only+ * that every other `at-reach` module is classified.+ * - `local-only`: only the operator's own shell reads it.+ * - `computation`: it defines a reader, or computes a result that is not+ * an answer to a remote caller's read.+ */++import { describe, expect, test } from "bun:test";+import { readdirSync, readFileSync } from "node:fs";+import { join, relative } from "node:path";++import { lexCode } from "../../helpers/source-lexer.ts";++const REPO_ROOT = join(import.meta.dir, "..", "..", "..");+const SRC_ROOT = join(REPO_ROOT, "src");++/** The log readers a call to which enrols a module. */+const LOG_READER_NAMES: ReadonlyArray<string> = Object.freeze([+ "logDayReader",+ "storeLogDays",+ "readStoreLogEvents",+ "readStoreLogEventsReport",+ "readLogEvents",+ "readLogEventsReport",+ "logDaysWithDevices",+ "readAllLogEntries",+ "buildTimelineIndex",+]);++/** A call of one of `names`, not its `function <name>(` declaration. */+function callPattern(names: ReadonlyArray<string>): RegExp {+ return new RegExp(`(?<!\\bfunction\\s+)\\b(?:${names.join("|")})\\s*\\(`);+}++/** A call of a reader under its own name. */+const LOG_READER_CALL_RE = callPattern(LOG_READER_NAMES);++/** The braces of a named import or re-export: `import { a as b, c } from`. */+const NAMED_IMPORT_RE = /\b(?:import|export)\s+(?:type\s+)?\{([^}]*)\}\s*from\b/g;++/**+ * The local names `code` binds a reader to under another name+ * (`import { storeLogDays as days }`), so a call through the alias is+ * still a call of the reader.+ */+function readerAliases(code: string): string[] {+ const readers: ReadonlySet<string> = new Set(LOG_READER_NAMES);+ const aliases: string[] = [];+ for (const match of code.matchAll(NAMED_IMPORT_RE)) {+ for (const specifier of match[1]!.split(",")) {+ const parts = specifier+ .trim()+ .replace(/^type\s+/, "")+ .split(/\s+as\s+/);+ if (parts.length === 2 && readers.has(parts[0]!) && /^[A-Za-z_$][\w$]*$/.test(parts[1]!)) {+ aliases.push(parts[1]!);+ }+ }+ }+ return aliases;+}++/** Whether `code` calls a reader, under its own name or an imported alias. */+function callsLogReader(code: string): boolean {+ if (LOG_READER_CALL_RE.test(code)) return true;+ const aliases = readerAliases(code);+ return aliases.length > 0 && callPattern(aliases).test(code);+}++type LogReaderClass = "at-reach" | "local-only" | "computation";++interface LogReaderRow {+ readonly class: LogReaderClass;+ readonly reason: string;+}++const DEFINES = "defines the reader";++function rows(+ modules: ReadonlyArray<string>,+ cls: LogReaderClass,+ reason: string,+): ReadonlyArray<readonly [string, LogReaderRow]> {+ return modules.map((m) => [`src/${m}`, { class: cls, reason }] as const);+}++/** Every module that calls a log reader, keyed by its `src/` path. */+const LOG_READERS: Readonly<Record<string, LogReaderRow>> = Object.freeze(+ Object.fromEntries([+ ...rows(+ [+ "cli/brain/verbs/temporal-daily.ts",+ "cli/brain/verbs/temporal-evolution.ts",+ "cli/brain/verbs/temporal-stale.ts",+ "cli/brain/verbs/temporal-timeline.ts",+ "cli/brain/verbs/temporal-weekly.ts",+ ],+ "local-only",+ "CLI",+ ),+ ...rows(["core/brain/agent-source/vault-provider.ts"], "at-reach", "D2 row 4"),+ ...rows(["core/brain/backlinks.ts"], "at-reach", "D2 row 5"),+ ...rows(+ [+ "core/brain/digest-agent-summary.ts",+ "core/brain/digest.ts",+ "core/brain/lessons.ts",+ "core/brain/event-trace.ts",+ ],+ "at-reach",+ "`logEntryAtReach` with the caller's view",+ ),+ ...rows(+ ["core/brain/doctor/orphan-session-check.ts", "core/brain/doctor/records.ts"],+ "computation",+ "D2, settled rows",+ ),+ ...rows(+ ["core/brain/dream-refresh.ts", "core/brain/evidence.ts"],+ "computation",+ "dream and evidence (task 3.18)",+ ),+ ...rows(+ [+ "core/brain/governance/forget-plan.ts",+ "core/brain/profile-doc.ts",+ "core/brain/temporal/citations.ts",+ "core/discipline/log-counts.ts",+ ],+ "local-only",+ "CLI only",+ ),+ ...rows(+ ["core/brain/idea-lineage.ts", "core/brain/operator-snapshot.ts"],+ "at-reach",+ "D2 row 9",+ ),+ ...rows(+ [+ "core/brain/memory-cost-meter.ts",+ "core/brain/most-applied.ts",+ "core/brain/morning-brief.ts",+ "core/brain/status.ts",+ "core/brain/token-footprint.ts",+ "core/brain/notes/write-log.ts",+ "core/brain/notes/write-destinations.ts",+ ],+ "at-reach",+ "D2 rows 2, 3, 6, 7, 8, 10",+ ),+ ...rows(+ ["core/brain/monthly-review.ts", "core/brain/today-dashboard.ts"],+ "at-reach",+ "the caller's `eventAtReach` (D1)",+ ),+ ...rows(+ ["core/brain/portability/okf.ts"],+ "at-reach",+ "private events leave `log.md` (spec `okf-export`)",+ ),+ ...rows(["core/brain/query.ts"], "at-reach", "`brain_query` filters every mode (D2 row 1)"),+ ...rows(+ ["mcp/brain/analytics-tools.ts", "mcp/brain/brief-tools.ts"],+ "at-reach",+ "`requestRefView` (D1); derived rows: an index built withholding (D2 row 9)",+ ),+ ...rows(["mcp/brain/review-tools.ts"], "at-reach", "D2 row 9"),+ ...rows(["mcp/resources.ts"], "at-reach", "`osb://log` filters (D3 of `lane-pages-split`)"),+ ...rows(+ [+ "core/brain/log-days.ts",+ "core/store/read/log-events.ts",+ "core/brain/temporal/build-index.ts",+ ],+ "computation",+ DEFINES,+ ),+ ]),+);++interface SourceFile {+ readonly path: string;+ readonly text: string;+}++/** Every `.ts` file under `src/`, as repo-relative path and text. */+function readSourceTree(): SourceFile[] {+ const files: SourceFile[] = [];+ const walk = (dir: string): void => {+ for (const entry of readdirSync(dir, { withFileTypes: true })) {+ const abs = join(dir, entry.name);+ if (entry.isDirectory()) walk(abs);+ else if (entry.name.endsWith(".ts")) {+ files.push({+ path: relative(REPO_ROOT, abs).split("\\").join("/"),+ text: readFileSync(abs, "utf8"),+ });+ }+ }+ };+ walk(SRC_ROOT);+ return files.toSorted((a, b) => a.path.localeCompare(b.path));+}++/** The modules of `files` that call a log reader. */+function logReaderModules(files: ReadonlyArray<SourceFile>): string[] {+ return files.filter((f) => callsLogReader(lexCode(f.text))).map((f) => f.path);+}++/** The calling modules the table does not list. */+function unlisted(files: ReadonlyArray<SourceFile>): string[] {+ return logReaderModules(files).filter((path) => !(path in LOG_READERS));+}++const SOURCE_TREE = readSourceTree();++describe("log-reader census", () => {+ test("every module that calls a log reader is classified", () => {+ expect(unlisted(SOURCE_TREE)).toEqual([]);+ });++ test("no table key outlives the call it classifies", () => {+ const calling = new Set(logReaderModules(SOURCE_TREE));+ expect(Object.keys(LOG_READERS).filter((path) => !calling.has(path))).toEqual([]);+ });++ test("every row carries a class of the closed vocabulary and a reason", () => {+ const classes: ReadonlySet<string> = new Set(["at-reach", "local-only", "computation"]);+ for (const [path, row] of Object.entries(LOG_READERS)) {+ expect({ path, known: classes.has(row.class) }).toEqual({ path, known: true });+ expect(row.reason.length).toBeGreaterThan(0);+ }+ });+});++describe("the census can fail", () => {+ const INTRUDER = "src/core/brain/synthetic-log-reader.ts";++ test("an unlisted module calling a log reader is reported", () => {+ const intruder = {+ path: INTRUDER,+ text: 'import { storeLogDays } from "./log-days.ts";\nexport const days = storeLogDays("v");\n',+ };+ expect(unlisted([...SOURCE_TREE, intruder])).toEqual([INTRUDER]);+ });++ test("a call through an aliased import of a reader is reported", () => {+ const intruder = {+ path: INTRUDER,+ text:+ 'import { storeLogDays as days, type LogDay } from "./log-days.ts";\n' ++ 'export const read: LogDay[] = days("v");\n',+ };+ expect(unlisted([...SOURCE_TREE, intruder])).toEqual([INTRUDER]);+ });++ test("an aliased import that is never called is not a call", () => {+ const intruder = {+ path: INTRUDER,+ text: 'import { storeLogDays as days } from "./log-days.ts";\nexport { days };\n',+ };+ expect(unlisted([intruder])).toEqual([]);+ });++ test("a declaration, a comment or a string naming a reader is not a call", () => {+ const intruder = {+ path: INTRUDER,+ text:+ "// storeLogDays(vault) in prose\n" ++ 'const s = "buildTimelineIndex(vault)";\n' ++ "export function logDayReader(v: string): string {\n return v + s;\n}\n",+ };+ expect(unlisted([intruder])).toEqual([]);+ });+});diff --git a/tests/core/brain/doctor-exit-census.test.ts b/tests/core/brain/doctor-exit-census.test.tsindex e8eca948..e8fa6e65 100644--- a/tests/core/brain/doctor-exit-census.test.ts+++ b/tests/core/brain/doctor-exit-census.test.ts@@ -217,6 +217,7 @@ const DOCTOR_REGISTERED_CODES: ReadonlyArray<string> = ["storage-capacity-critical","storage-capacity-warn","store-missing",+ "store-shard-corrupt-line","sync-conflict-log","tier-drift","topic-key-collision",diff --git a/tests/core/brain/doctor-store.test.ts b/tests/core/brain/doctor-store.test.tsindex a36fc2ad..dee03ee5 100644--- a/tests/core/brain/doctor-store.test.ts+++ b/tests/core/brain/doctor-store.test.ts@@ -3,13 +3,14 @@* not in use, and when it is, the dual-write counters, state-digest* findings and same-id conflicts.*/-import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test";import {appendFileSync,chmodSync,existsSync,mkdirSync,mkdtempSync,+ readFileSync,rmSync,symlinkSync,utimesSync,@@ -19,6 +20,7 @@ import { tmpdir } from "node:os";import { dirname, join } from "node:path";import { runDoctor } from "../../../src/core/brain/doctor.ts";+import { nextCommandField } from "../../../src/core/brain/next-step.ts";import {STORE_CLOCK_AHEAD_CODE,STORE_DIGEST_DIVERGENT_CODE,@@ -33,6 +35,7 @@ import {STORE_SHARD_TAIL_HELD_CODE,STORE_SHARD_UNREADABLE_CODE,STORE_REMOVED_AWAITING_COMPACTION_CODE,+ STORE_WRITE_FAIL_OPEN_CODE,} from "../../../src/core/brain/doctor/store-check.ts";import {reservedDeviceDirIssues,@@ -50,12 +53,29 @@ import { publishStateDigest } from "../../../src/core/store/state-digest.ts";import { inspectStore, openStore, type StoreHandle } from "../../../src/core/store/store-open.ts";import { initStore } from "../../../src/core/store/store-init.ts";import { ensureStore } from "../store/fixtures/temp-store.ts";+import {+ STORE_SHARD_BLOB_PENDING_CODE,+ STORE_SHARD_CORRUPT_LINE_CODE,+ STORE_SHARD_LINE_TOO_LONG_CODE,+ STORE_SHARD_NEWER_FORMAT_CODE,+ storeGapFindings,+} from "../../../src/core/brain/doctor/store-gap-check.ts";+import type { DoctorFindings } from "../../../src/core/brain/doctor/check.ts";+import { addQuarantine } from "../../../src/core/store/shard-quarantine-write.ts";+import {+ lineSha256At,+ QUARANTINE_HASH_CAP_BYTES,+} from "../../../src/core/store/shard-quarantine.ts";+import { ledgerRowRecord } from "../../../src/core/store/dual-write-records.ts";+import { routeWrite } from "../../../src/core/store/store-write.ts";+import { SETTING_SET_KIND } from "../../../src/core/store/constants.ts";import { dualWriteStatusPath, shardPath } from "../../../src/core/store/store-paths.ts";import { CHMOD_CANNOT_DENY, IS_WINDOWS } from "../../helpers/platform.ts";const ENV_KEYS = [STORE_DUAL_WRITE_ENV,STORE_ROOT_ENV,+ "O2B_DEVICE_ID","HOME","XDG_STATE_HOME","XDG_DATA_HOME",@@ -128,6 +148,13 @@ function machine(device: string): StoreHandle {const at = (ms: number) => () => ms;+/** Run the gap findings over one shard named as read only in part. */+function gapFindingsOf(rel: string): DoctorFindings {+ const out: DoctorFindings = { issues: [], uncertain: [] };+ storeGapFindings(inspectStore(storeRoot)!, [{ partition: "shared", rel }], NOW + 10, out);+ return out;+}+describe("store not in use", () => {test("no flag and no store root: the section says nothing, even about old counters", () => {writeFailureCounter(4);@@ -189,9 +216,66 @@ describe("store in use", () => {const finding = report.errors.find((i) => i.code === STORE_PARTITION_LOST_CODE);expect(finding?.message).toContain("private partition");expect(finding?.message).toContain("private/store.json");+ expect(finding?.message).toContain("o2b store partition leave private");expect(storeCodes()).toEqual([STORE_PARTITION_LOST_CODE]);});+ test("below local reach a lost partition is still an error but never names private", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ ensureStore(storeRoot);+ openStore({ root: storeRoot, device: "dev-a" });+ rmSync(join(storeRoot, "private", "store.json"));+ const lostOf = (opts: { withholdsPrivate?: true; reach?: "remote" }) =>+ runDoctor(vault, { now: new Date(NOW), ...opts }).errors.filter(+ (i) => i.code === STORE_PARTITION_LOST_CODE,+ );+ for (const opts of [{ withholdsPrivate: true } as const, { reach: "remote" } as const]) {+ const lost = lostOf(opts);+ expect(lost).toHaveLength(1);+ expect(lost[0]!.message).toContain("a partition of the brain store");+ expect(lost[0]!.message).not.toContain("private");+ }+ rmSync(join(storeRoot, "shared", "store.json"));+ expect(lostOf({})[0]!.message).toContain("private/store.json");+ const both = lostOf({ withholdsPrivate: true });+ expect(both).toHaveLength(1);+ expect(both[0]!.message).toContain("the shared partition and another partition");+ expect(both[0]!.message).not.toContain("private");+ });++ test("below local reach a store that does not open keeps its finding without the cause", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ ensureStore(storeRoot);+ openStore({ root: storeRoot, device: "dev-a" });+ const privateIdentity = join(storeRoot, "private", "store.json");+ const sharedId = (+ JSON.parse(readFileSync(join(storeRoot, "shared", "store.json"), "utf8")) as {+ store_id: string;+ }+ ).store_id;+ const otherId = "01ARZ3NDEKTSV4RRFFQ69G5FAV";+ const invalidOf = (opts: { withholdsPrivate?: true; reach?: "remote" }) =>+ runDoctor(vault, { now: new Date(NOW), ...opts }).warnings.filter(+ (i) => i.code === STORE_ROOT_INVALID_CODE,+ );+ const original = readFileSync(privateIdentity, "utf8");+ for (const broken of ["{ not json", original.replace(sharedId, otherId)]) {+ writeFileSync(privateIdentity, broken);+ const local = invalidOf({});+ expect(local).toHaveLength(1);+ expect(local[0]!.message).toContain("private");+ for (const opts of [{ withholdsPrivate: true } as const, { reach: "remote" } as const]) {+ const remote = invalidOf(opts);+ expect(remote).toHaveLength(1);+ expect(remote[0]!.path).toBe(local[0]!.path);+ expect(remote[0]!.message).toContain("could not be opened");+ expect(remote[0]!.message).not.toContain("private");+ expect(remote[0]!.message).not.toContain(sharedId);+ expect(remote[0]!.message).not.toContain(otherId);+ }+ }+ });+test("a store that never held private on this machine is no finding", () => {process.env[STORE_ROOT_ENV] = storeRoot;initStore({ root: storeRoot, partitions: ["shared"] });@@ -623,3 +707,311 @@ describe("removed content awaiting compaction", () => {expect(issue?.message).toContain("1 of forgotten signals");});});++describe("shards read only in part", () => {+ const DAY = 24 * 60 * 60 * 1000;++ /** Three notes of dev-b in one shard, written at `when`; returns its path. */+ function threeNotes(when = NOW): string {+ machine("dev-a");+ const b = machine("dev-b");+ let path = "";+ for (const n of [1, 2, 3]) {+ path = appendRecord(+ b,+ { partition: "shared", stream: "events", kind: "note", agent: "t", body: { n } },+ { now: at(when) },+ ).path;+ }+ return path;+ }++ /** Rewrite the middle line of the shard at `path`; returns its offset. */+ function editMiddle(path: string, edit: (line: string) => string): number {+ const lines = readFileSync(path, "utf8").split("\n");+ lines[1] = edit(lines[1]!);+ writeFileSync(path, lines.join("\n"));+ return Buffer.byteLength(lines[0]!) + 1;+ }++ const relOf = (path: string): string =>+ path+ .slice(join(storeRoot, "shared").length + 1)+ .split("\\")+ .join("/");++ function findings(code: string, now = NOW + 10) {+ const report = runDoctor(vault, { now: new Date(now) });+ return {+ errors: report.errors.filter((i) => i.code === code),+ warnings: report.warnings.filter((i) => i.code === code),+ };+ }++ test("a corrupt middle line is an error naming the offset and the quarantine command", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ const offset = editMiddle(path, (line) => line.replace('"n":2', '"n":20'));+ const { errors } = findings(STORE_SHARD_CORRUPT_LINE_CODE);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.path).toBe(`shared/${relOf(path)}`);+ expect(errors[0]!.message).toContain(`byte ${offset}`);+ expect(errors[0]!.message).toContain(+ `o2b store shard quarantine shared/${relOf(path)} --offset ${offset}`,+ );+ });++ test("a corrupt line too long to hash names source repair, no quarantine command", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ // Past the hash cap the verb always refuses (`line-too-long`).+ const offset = editMiddle(path, () => "x".repeat(QUARANTINE_HASH_CAP_BYTES + 10));+ expect(findings(STORE_SHARD_CORRUPT_LINE_CODE).errors).toEqual([]);+ const { errors } = findings(STORE_SHARD_LINE_TOO_LONG_CODE);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.path).toBe(`shared/${relOf(path)}`);+ expect(errors[0]!.message).toContain(`byte ${offset}`);+ expect(errors[0]!.message).toContain("repair or remove it at its source");+ expect(errors[0]!.message).not.toContain("o2b store shard quarantine");+ expect(nextCommandField(errors[0]!.code)).toEqual({});+ });++ test("a newer-format line is a warning to upgrade, with no command", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ editMiddle(path, (line) => line.replace('"v":1', '"v":99'));+ const { warnings, errors } = findings(STORE_SHARD_NEWER_FORMAT_CODE);+ expect(errors).toEqual([]);+ expect(warnings).toHaveLength(1);+ expect(warnings[0]!.message).toContain("upgrade Open Second Brain");+ expect(warnings[0]!.message).not.toContain("o2b store shard quarantine");+ });++ /** A dev-b record in `stream` whose blob has not arrived, written at `when`. */+ function pendingBlob(stream: "events" | "settings", when: number): string {+ machine("dev-a");+ const path = appendRecord(+ machine("dev-b"),+ stream === "events"+ ? {+ partition: "shared",+ stream,+ kind: "note",+ agent: "t",+ body: { n: 1, text: "x".repeat(20_000) },+ }+ : {+ partition: "shared",+ stream,+ kind: SETTING_SET_KIND,+ agent: "t",+ body: { authority: "human", key: "brain.x.y", value: "x".repeat(20_000) },+ },+ { now: at(when) },+ ).path;+ rmSync(join(storeRoot, "shared", "blobs"), { recursive: true, force: true });+ return path;+ }++ test("a young pending blob is a warning with no command, an old one an error with it", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = pendingBlob("events", NOW - DAY);+ const young = findings(STORE_SHARD_BLOB_PENDING_CODE);+ expect(young.errors).toEqual([]);+ expect(young.warnings).toHaveLength(1);+ expect(young.warnings[0]!.message).not.toContain("o2b store shard quarantine");+ const old = findings(STORE_SHARD_BLOB_PENDING_CODE, NOW + 8 * DAY);+ expect(old.warnings).toEqual([]);+ expect(old.errors).toHaveLength(1);+ expect(old.errors[0]!.message).toContain(+ `o2b store shard quarantine shared/${relOf(path)} --offset 0`,+ );+ });++ test("an old pending blob in a settings shard names no command", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ pendingBlob("settings", NOW - DAY);+ const { errors } = findings(STORE_SHARD_BLOB_PENDING_CODE, NOW + 8 * DAY);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.message).not.toContain("o2b store shard quarantine");+ });++ // A clock stamp that verifies but names no instant a date can hold.+ const UNREADABLE_HLC = `"hlc":"${Number.MAX_SAFE_INTEGER}.0"`;++ test("a pending blob whose clock stamp does not parse says so and offers no quarantine by age", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = pendingBlob("events", NOW - 8 * DAY);+ const lines = readFileSync(path, "utf8").split("\n");+ lines[0] = lines[0]!.replace(/"hlc":"[^"]*"/, UNREADABLE_HLC);+ writeFileSync(path, lines.join("\n"));+ for (const now of [NOW + 10, NOW + 30 * DAY]) {+ const { errors, warnings } = findings(STORE_SHARD_BLOB_PENDING_CODE, now);+ expect(warnings).toEqual([]);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.message).toContain("its clock stamp does not parse");+ expect(errors[0]!.message).not.toContain("has not arrived since");+ expect(errors[0]!.message).not.toContain("o2b store shard quarantine");+ }+ });++ test("a shard gone since the partition read is skipped, not thrown", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ const rel = relOf(path);+ rmSync(path);+ expect(gapFindingsOf(rel).issues).toEqual([]);+ });++ test.skipIf(IS_WINDOWS)(+ "a shard swapped for a link is an error with no quarantine command",+ () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ const aside = join(parent, "aside.jsonl");+ writeFileSync(aside, readFileSync(path));+ rmSync(path);+ symlinkSync(aside, path);+ const { issues } = gapFindingsOf(relOf(path));+ expect(issues).toHaveLength(1);+ expect(issues[0]!.code).toBe(STORE_SHARD_UNREADABLE_CODE);+ expect(issues[0]!.message).toContain("could not be read");+ expect(issues[0]!.message).not.toContain("o2b store shard quarantine");+ // The next step the doctor attaches by code names no command either.+ expect(nextCommandField(issues[0]!.code)).toEqual({});+ },+ );++ test.skipIf(CHMOD_CANNOT_DENY)(+ "a shard that fails the second read for another reason than its removal is named with the error code",+ () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const path = threeNotes();+ chmodSync(path, 0o000);+ try {+ const { issues } = gapFindingsOf(relOf(path));+ expect(issues).toHaveLength(1);+ expect(issues[0]!.severity).toBe("error");+ expect(issues[0]!.code).toBe(STORE_SHARD_UNREADABLE_CODE);+ expect(issues[0]!.path).toBe(`shared/${relOf(path)}`);+ expect(issues[0]!.message).toContain("EACCES");+ expect(nextCommandField(issues[0]!.code)).toEqual({});+ } finally {+ chmodSync(path, 0o600);+ }+ },+ );++ test("below local reach no store finding names the private partition, and the counts are those of a store without it", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ threeNotes();+ const doctor = (opts: { withholdsPrivate?: true; reach?: "remote" } = {}) =>+ runDoctor(vault, { now: new Date(NOW + 10), ...opts });+ const before = doctor();+ // A private shard with a clock far ahead and a corrupt middle line: a+ // clock-ahead warning and a corrupt-line error, both naming `private`.+ const b = machine("dev-b");+ let path = "";+ for (const n of [1, 2, 3]) {+ path = appendRecord(+ b,+ { partition: "private", stream: "events", kind: "note", agent: "t", body: { n } },+ { now: at(NOW + 30 * DAY) },+ ).path;+ }+ editMiddle(path, (line) => line.replace('"n":2', '"n":20'));+ const local = doctor();+ const named = (r: typeof local) =>+ [...r.errors, ...r.warnings].filter(+ (i) => (i.path ?? "").includes("private") || i.message.includes("private"),+ );+ expect(named(local).map((i) => i.code)).toEqual(+ expect.arrayContaining([STORE_SHARD_CORRUPT_LINE_CODE, STORE_CLOCK_AHEAD_CODE]),+ );+ for (const remote of [doctor({ withholdsPrivate: true }), doctor({ reach: "remote" })]) {+ expect(named(remote)).toEqual([]);+ expect(remote.errors.length).toBe(before.errors.length);+ expect(remote.warnings.length).toBe(before.warnings.length);+ expect(remote.trust_verdict).toEqual(before.trust_verdict);+ }+ });++ test("a torn line and a quarantined line give no finding", () => {+ process.env[STORE_ROOT_ENV] = storeRoot;+ const torn = threeNotes();+ editMiddle(torn, (line) => line.slice(0, Math.floor(line.length / 2)));+ expect(findings(STORE_SHARD_CORRUPT_LINE_CODE).errors).toEqual([]);+ const offset = editMiddle(torn, () =>+ readFileSync(torn, "utf8").split("\n")[0]!.replace('"n":1', '"n":10'),+ );+ expect(findings(STORE_SHARD_CORRUPT_LINE_CODE).errors).toHaveLength(1);+ addQuarantine(storeRoot, {+ partition: "shared",+ shard: relOf(torn),+ offset,+ sha256: lineSha256At(torn, offset, join(storeRoot, "shared"))!,+ kind: "corrupt-line",+ reason: "test",+ });+ expect(findings(STORE_SHARD_CORRUPT_LINE_CODE).errors).toEqual([]);+ });+});++describe("failed writes below local reach", () => {+ /** Make `<partition>/ledgers` a file, so every ledger-row append there fails naming its path. */+ function blockAppends(partition: "shared" | "private"): string {+ const blocked = join(storeRoot, partition, "ledgers");+ writeFileSync(blocked, "not a folder\n");+ return blocked;+ }++ /** One dual write (the vault keeps its legacy tree) and one fail-open store-only write. */+ function writeRow(partition: "shared" | "private"): void {+ const store = () => ledgerRowRecord("metrics", { n: 1 }, partition);+ const quiet = spyOn(process.stderr, "write").mockImplementation(() => true);+ try {+ routeWrite({ legacy: () => null, store, context: { vault } });+ routeWrite({ legacy: () => null, store, context: { vault: null }, failOpen: true });+ } finally {+ quiet.mockRestore();+ }+ }++ const CODES = [STORE_DUAL_WRITE_FAILURES_CODE, STORE_WRITE_FAIL_OPEN_CODE];+ const failuresOf = (opts: { withholdsPrivate?: true; reach?: "remote" } = {}) =>+ runDoctor(vault, { now: new Date(NOW), ...opts }).warnings.filter((i) =>+ CODES.includes(i.code),+ );++ test("a failed private write shows nothing below local reach; a failed shared one still shows", () => {+ process.env[STORE_DUAL_WRITE_ENV] = "1";+ process.env[STORE_ROOT_ENV] = storeRoot;+ process.env["O2B_DEVICE_ID"] = "dev-a";+ ensureStore(storeRoot);+ const blocked = blockAppends("private");+ writeRow("private");+ const local = failuresOf();+ expect(local.map((i) => i.code).toSorted()).toEqual(CODES.toSorted());+ // The error text names the private folder the append could not use.+ expect(local.every((i) => i.message.includes(blocked))).toBe(true);+ for (const opts of [{ withholdsPrivate: true } as const, { reach: "remote" } as const]) {+ expect(failuresOf(opts)).toEqual([]);+ }++ const sharedBlocked = blockAppends("shared");+ writeRow("shared");+ for (const opts of [{ withholdsPrivate: true } as const, { reach: "remote" } as const]) {+ const remote = failuresOf(opts);+ expect(remote.map((i) => i.code).toSorted()).toEqual(CODES.toSorted());+ for (const issue of remote) {+ expect(issue.message.startsWith("1 ")).toBe(true);+ expect(issue.message).toContain("to the shared partition");+ expect(issue.message).toContain(sharedBlocked);+ expect(issue.message).not.toContain(blocked);+ expect(issue.message).not.toContain("private");+ }+ }+ // The operator's own doctor still counts both.+ expect(failuresOf().every((i) => i.message.startsWith("2 "))).toBe(true);+ });+});diff --git a/tests/core/lanes/lane-pages.test.ts b/tests/core/lanes/lane-pages.test.tsindex 79e5bb08..055ed504 100644--- a/tests/core/lanes/lane-pages.test.ts+++ b/tests/core/lanes/lane-pages.test.ts@@ -37,6 +37,7 @@ import {writeLaneText,} from "../../../src/core/store/lane-pages.ts";import { foldLaneEntries } from "../../../src/core/store/lane-records.ts";+import { formatFrontmatter } from "../../../src/core/vault.ts";import { VaultAdapterRequiredError } from "../../../src/core/vault-adapter/index.ts";import {bindVaultToStore,@@ -299,6 +300,26 @@ describe.each(LANE_MODES.filter((m) => m !== "legacy"))("lane pages in the storeexpect(lane.laneRecords().filter((r) => r.kind === "lane-entry")).toHaveLength(2);});+ test("a structured write over a raw entry drops the raw once and then churns nothing", () => {+ lane = makeLaneEnv(mode);+ const page = join(lane.vault, "Brain", "digests", "churn.md");+ // No blank line after the closing fence: the whole write keeps a raw.+ writeLaneText(lane.vault, page, "---\nkind: digest\n---\nBody.\n");+ const entries = () => lane!.laneRecords().filter((r) => r.kind === "lane-entry");+ expect(entries()).toHaveLength(1);+ const written = writeLanePage(lane.vault, page, { kind: "digest" }, "Body.\n", {+ overwrite: true,+ });+ expect(written).toBe(true);+ expect(entries()).toHaveLength(2);+ // The same parts again match the entry (its raw is gone) and are skipped.+ const again = writeLanePage(lane.vault, page, { kind: "digest" }, "Body.\n", {+ overwrite: true,+ });+ expect(again).toBe(false);+ expect(entries()).toHaveLength(2);+ });+test("a caller editing the frontmatter it read does not change the next read", () => {lane = makeLaneEnv(mode);const page = join(lane.vault, "Brain", "pledges", "p.md");@@ -600,3 +621,181 @@ describe.each(LANE_MODES.filter((m) => m !== "legacy"))(});},);++describe("raw lane text on a store-only machine", () => {+ test("a canonical page stores a record with no raw", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-canonical.md");+ writeLaneText(lane.vault, page, formatFrontmatter({ kind: "k" }, "Body."));+ const body = lane.laneRecords().find((r) => r.kind === "lane-entry")!.body as Record<+ string,+ unknown+ >;+ expect("raw" in body).toBe(false);+ expect(readLaneText(lane.vault, page)).toBe(formatFrontmatter({ kind: "k" }, "Body."));+ });++ test("a page without frontmatter reads back verbatim", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-raw.md");+ writeLaneText(lane.vault, page, "Just text.\n");+ expect(readLaneText(lane.vault, page)).toBe("Just text.\n");+ });++ test("a quoted value reads back verbatim", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-quote.md");+ writeLaneText(lane.vault, page, "---\nid: '123'\n---\nBody.\n");+ expect(readLaneText(lane.vault, page)).toBe("---\nid: '123'\n---\nBody.\n");+ expect(readLanePage(lane.vault, page)?.frontmatter.id).toBe("123");+ });++ test("CRLF and a missing final newline read back verbatim", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-crlf.md");+ writeLaneText(lane.vault, page, "---\r\nkind: k\r\n---\r\nBody.\r\n");+ expect(readLaneText(lane.vault, page)).toBe("---\r\nkind: k\r\n---\r\nBody.\r\n");+ const noFinal = join(lane.vault, "Brain", "obligations", "obl-no-final.md");+ writeLaneText(lane.vault, noFinal, "---\nkind: k\n---\nBody.");+ expect(readLaneText(lane.vault, noFinal)).toBe("---\nkind: k\n---\nBody.");+ });++ test("JSON formatting reads back verbatim", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "raw.json");+ writeLaneText(lane.vault, page, '{"a":1}\n');+ expect(readLaneText(lane.vault, page)).toBe('{"a":1}\n');+ });++ test("a quoting-only rewrite is written", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-requote.md");+ writeLaneText(lane.vault, page, "---\nid: '123'\n---\nBody.\n");+ const afterFirst = lane.laneRecords().filter((r) => r.kind === "lane-entry").length;+ writeLaneText(lane.vault, page, '---\nid: "123"\n---\nBody.\n');+ const grown = lane.laneRecords().filter((r) => r.kind === "lane-entry").length;+ expect(grown).toBe(afterFirst + 1);+ expect(readLaneText(lane.vault, page)).toBe('---\nid: "123"\n---\nBody.\n');+ writeLaneText(lane.vault, page, '---\nid: "123"\n---\nBody.\n');+ const stable = lane.laneRecords().filter((r) => r.kind === "lane-entry").length;+ expect(stable).toBe(grown);+ });++ test("a structured write replaces the raw", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-struct.md");+ writeLaneText(lane.vault, page, "Just text.\n");+ writeLanePage(lane.vault, page, { kind: "k" }, "New.\n", { overwrite: true });+ expect(readLaneText(lane.vault, page)).toBe(formatFrontmatter({ kind: "k" }, "New.\n"));+ });++ test("a move keeps the raw", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-mv.md");+ writeLaneText(lane.vault, page, "Just text.\n");+ const dir = join(lane.vault, "Brain", "obligations");+ moveLanePage(lane.vault, page, join(dir, "obl-moved.md"));+ expect(readLaneText(lane.vault, join(dir, "obl-moved.md"))).toBe("Just text.\n");+ const tree = join(lane.vault, "Brain", "obligations", "obl-tree");+ mkdirSync(tree, { recursive: true });+ const treePage = join(tree, "obl-in-tree.md");+ writeLaneText(lane.vault, treePage, "Just text.\n");+ const other = join(lane.vault, "Brain", "obligations", "obl-tree-2");+ moveLaneTree(lane.vault, tree, other);+ expect(readLaneText(lane.vault, join(other, "obl-in-tree.md"))).toBe("Just text.\n");+ });++ test("a private region in a frontmatter comment sends the entry to private", () => {+ lane = makeLaneEnv("store");+ const page = join(lane.vault, "Brain", "obligations", "obl-comment.md");+ writeLaneText(lane.vault, page, "---\n# <private>secret</private>\nkind: k\n---\nBody.\n");+ expect(lane.laneRecords("shared")).toEqual([]);+ expect(lane.laneRecords("private")).toHaveLength(1);+ });++ test("a private reference in a frontmatter comment", () => {+ lane = makeLaneEnv("store");+ // The page the comment names exists on disk and carries no `pref-`/+ // `ret-` prefix: without one, the entry is judged by the reserved-page+ // rule (the page's own `visibility`), not by the missing-preference+ // rule. The page is written as a file directly, since a lane write on+ // a store-only machine lands in the store, not under the vault.+ const notes = join(lane.vault, "Brain", "notes");+ mkdirSync(notes, { recursive: true });+ const reserved = join(notes, "private-note.md");+ writeFileSync(reserved, "Plain.\n");+ const page = join(lane.vault, "Brain", "obligations", "obl-rawref-open.md");+ writeLaneText(+ lane.vault,+ page,+ "---\n# see [[Brain/notes/private-note]]\nkind: k\n---\nBody.\n",+ );+ // Control: the page is not reserved, so the entry naming it stays in+ // shared.+ const openEntry = lane+ .laneRecords("shared")+ .filter((r) => r.kind === "lane-entry" && r.body["lane"] === "obligations");+ expect(openEntry).toHaveLength(1);+ expect(lane.laneRecords("private").filter((r) => r.body["lane"] === "obligations")).toEqual([]);+ // Now reserve the same page as private: a further entry naming it must+ // go to private, and the earlier one stays where it was judged.+ writeFileSync(reserved, "---\nvisibility: private\n---\nReserved.\n");+ const judgedPage = join(lane.vault, "Brain", "obligations", "obl-rawref.md");+ writeLaneText(+ lane.vault,+ judgedPage,+ "---\n# see [[Brain/notes/private-note]]\nkind: k\n---\nBody.\n",+ );+ const judged = lane+ .laneRecords("private")+ .filter((r) => r.kind === "lane-entry" && r.body["lane"] === "obligations");+ expect(judged).toHaveLength(1);+ const shared = lane+ .laneRecords("shared")+ .filter((r) => r.kind === "lane-entry" && r.body["lane"] === "obligations");+ expect(shared).toHaveLength(1);+ expect(shared[0]!.body["entry"]).toBe("obl-rawref-open");+ });++ test("a raw that does not parse back is not served", () => {+ lane = makeLaneEnv("store");+ appendRecord(openStore({ root: lane.env.storeRoot, device: "dev-b" }), {+ partition: "shared",+ stream: "events",+ kind: "lane-entry",+ agent: "t",+ body: {+ authority: "agent",+ lane: "obligations",+ entry: "obl-hand",+ fields: { kind: "k" },+ text: "Body.",+ raw: "---\nkind: other\n---\nBody.\n",+ },+ });+ const page = join(lane.vault, "Brain", "obligations", "obl-hand.md");+ expect(readLaneText(lane.vault, page)).toBe(formatFrontmatter({ kind: "k" }, "Body."));+ });++ test("a hand-appended shared raw with a frontmatter comment is served as written", () => {+ lane = makeLaneEnv("store");+ appendRecord(openStore({ root: lane.env.storeRoot, device: "dev-b" }), {+ partition: "shared",+ stream: "events",+ kind: "lane-entry",+ agent: "t",+ body: {+ authority: "agent",+ lane: "obligations",+ entry: "obl-hand-comment",+ fields: { kind: "k" },+ text: "Body.",+ raw: "---\n# see [[pref-nowhere-raw]]\nkind: k\n---\nBody.\n",+ },+ });+ const page = join(lane.vault, "Brain", "obligations", "obl-hand-comment.md");+ expect(readLaneText(lane.vault, page)).toBe(+ "---\n# see [[pref-nowhere-raw]]\nkind: k\n---\nBody.\n",+ );+ });+});diff --git a/tests/core/store-migration/rollback-quarantine.test.ts b/tests/core/store-migration/rollback-quarantine.test.tsnew file mode 100644index 00000000..58fb0ab7--- /dev/null+++ b/tests/core/store-migration/rollback-quarantine.test.ts@@ -0,0 +1,66 @@+/**+ * A rollback with a quarantined pending blob (change store-hardening,+ * spec `store-shard-gaps`, "Rollback with a quarantined pending blob"): the+ * quarantine skips the record on this machine, yet the rollback still+ * refuses while its blob is pending, names the shard, and restores nothing.+ */+import { afterEach, describe, expect, test } from "bun:test";+import { readdirSync, rmSync } from "node:fs";+import { join, relative } from "node:path";++import { readShard, shardReadGap } from "../../../src/core/store/reader.ts";+import { appendRecord } from "../../../src/core/store/appender.ts";+import { addQuarantine } from "../../../src/core/store/shard-quarantine-write.ts";+import { lineSha256At } from "../../../src/core/store/shard-quarantine.ts";+import { openStore } from "../../../src/core/store/store-open.ts";+import { partitionDir } from "../../../src/core/store/store-paths.ts";+import { runMigration } from "../../../src/core/store-migration/migrate.ts";+import { isDeviceSwitched } from "../../../src/core/store-migration/migration-records.ts";+import { rollback } from "../../../src/core/store-migration/outcome.ts";+import { RollbackRefusedError } from "../../../src/core/store-migration/rollback.ts";+import { makeFleet, type Fleet } from "./fixtures/fleet.ts";++const FAST = { settleMs: 0 } as const;+let fleet: Fleet | null = null;+afterEach(() => {+ fleet?.cleanup();+ fleet = null;+});++describe("rollback with a quarantined pending blob", () => {+ test("refuses and names the shard", () => {+ fleet = makeFleet(["dev-a"]);+ const a = fleet.machine("dev-a");+ a.use();+ expect(runMigration(a.vault, FAST).ok).toBe(true);+ const shared = partitionDir(a.storeRoot, "shared");+ // Another device's record whose body is a blob, then the blob is lost.+ const path = appendRecord(openStore({ root: a.storeRoot, device: "dev-b" }), {+ partition: "shared",+ stream: "events",+ kind: "note",+ agent: "t",+ body: { text: "x".repeat(20_000) },+ }).path;+ rmSync(join(shared, "blobs"), { recursive: true, force: true });+ const rel = relative(shared, path).split("\\").join("/");+ const offset = readShard(path, { partitionRoot: shared }).records.find(+ (r) => r.bodyState === "pending",+ )!.offset;+ addQuarantine(a.storeRoot, {+ partition: "shared",+ shard: rel,+ offset,+ sha256: lineSha256At(path, offset, shared)!,+ kind: "blob-pending",+ reason: "test",+ });+ // The quarantine holds: the shard reads as whole on this machine.+ expect(shardReadGap(readShard(path, { partitionRoot: shared }))).toBeNull();+ const before = readdirSync(join(a.vault, "Brain")).toSorted();+ expect(() => rollback(a.vault)).toThrow(RollbackRefusedError);+ expect(() => rollback(a.vault)).toThrow(`shared/${rel}`);+ expect(isDeviceSwitched(a.store(), "dev-a")).toBe(true);+ expect(readdirSync(join(a.vault, "Brain")).toSorted()).toEqual(before);+ });+});diff --git a/tests/core/store-migration/rollback.test.ts b/tests/core/store-migration/rollback.test.tsindex 3136dadb..0691a555 100644--- a/tests/core/store-migration/rollback.test.ts+++ b/tests/core/store-migration/rollback.test.ts@@ -21,7 +21,7 @@ import { join, sep } from "node:path";import { appendRecord } from "../../../src/core/store/appender.ts";import { openStore, type StoreHandle } from "../../../src/core/store/store-open.ts";import { compareRecords } from "../../../src/core/store/envelope.ts";-import { parseFrontmatterText } from "../../../src/core/vault.ts";+import { formatFrontmatter, parseFrontmatterText } from "../../../src/core/vault.ts";import { migrationAvailability } from "../../../src/core/store/migration-availability.ts";import { migrationDir } from "../../../src/core/store-migration/journal.ts";import { runMigration } from "../../../src/core/store-migration/migrate.ts";@@ -33,6 +33,7 @@ import {} from "../../../src/core/store-migration/migration-records.ts";import { rollback } from "../../../src/core/store-migration/outcome.ts";import { RollbackRefusedError } from "../../../src/core/store-migration/rollback.ts";+import { writeLaneText } from "../../../src/core/store/lane-pages.ts";import { readVaultIdentity } from "../../../src/core/brain/vault-identity.ts";import { readVaultBindings } from "../../../src/core/vault-adapter/vault-binding.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";@@ -211,6 +212,57 @@ describe("rollback", () => {expect(meta["principle"]).toBe("Open rule, edited");});+ test("a lane page written whole on the store rolls back byte for byte, stale raw renders", () => {+ fleet = makeFleet(["dev-a"]);+ const a = fleet.machine("dev-a");+ a.use();+ expect(runMigration(a.vault, FAST).ok).toBe(true);+ // A page with no frontmatter, written whole on the switched machine:+ // the store keeps its `raw`, and the export serves it back unchanged.+ const whole = join(a.vault, "Brain", "notes", "whole.md");+ writeLaneText(a.vault, whole, "Just text.\n");+ // An entry whose `raw` does not parse back to its fields and text is+ // exported as the rendering, never as the stale bytes.+ appendRecord(+ a.store(),+ {+ partition: "shared",+ stream: "events",+ kind: "lane-entry",+ agent: "t",+ body: {+ authority: "agent",+ lane: "notes",+ entry: "stale",+ fields: { title: "Stale" },+ text: "Body.\n",+ raw: "Stale bytes that do not parse back.\n",+ },+ },+ { now: () => afterTheSwitch() },+ );+ expect(existsSync(whole)).toBe(false);+ rollback(a.vault);+ expect(readFileSync(whole, "utf8")).toBe("Just text.\n");+ expect(readFileSync(join(a.vault, "Brain", "notes", "stale.md"), "utf8")).toBe(+ formatFrontmatter({ title: "Stale" }, "Body.\n"),+ );+ });++ test("a malformed JSON lane file written whole on the store rolls back byte for byte", () => {+ fleet = makeFleet(["dev-a"]);+ const a = fleet.machine("dev-a");+ a.use();+ expect(runMigration(a.vault, FAST).ok).toBe(true);+ // Malformed JSON is stored as text with no `raw` (the rendering already+ // equals it), so only the shared fallback can put the bytes back.+ const bad = join(a.vault, "Brain", "notes", "raw-bad.json");+ writeLaneText(a.vault, bad, "{bad");+ expect(existsSync(bad)).toBe(false);+ rollback(a.vault);+ expect(readFileSync(bad, "utf8")).toBe("{bad");+ });+test("puts post-switch and late remote records back once, in fold order", () => {fleet = makeFleet(["dev-a", "dev-b"]);const [a, b] = [fleet.machine("dev-a"), fleet.machine("dev-b")];diff --git a/tests/core/store/lane-raw.test.ts b/tests/core/store/lane-raw.test.tsnew file mode 100644index 00000000..6e986276--- /dev/null+++ b/tests/core/store/lane-raw.test.ts@@ -0,0 +1,23 @@+/**+ * The raw lane text helper cases no boundary test owns (design D8 of+ * change `store-hardening`): the verbatim contracts of a page without a+ * blank line after its closing fence, and of plain text, are proven+ * byte for byte in `tests/core/lanes/lane-pages.test.ts`; these cover the+ * shapes that file does not write.+ */+import { describe, expect, test } from "bun:test";+import { laneTextParts, rawToKeep } from "../../../src/core/store/lane-raw.ts";++describe("lane raw text", () => {+ test("a page with no blank line after the closing fence keeps its raw", () => {+ const raw = "---\nkind: k\n---\nBody.\n";+ const { fields, body } = laneTextParts("x.md", raw);+ expect(rawToKeep("x.md", raw, fields, body)).toBe(raw);+ });++ test("plain text never needs raw", () => {+ const raw = "a\nb";+ const { fields, body } = laneTextParts("x.txt", raw);+ expect(rawToKeep("x.txt", raw, fields, body)).toBeUndefined();+ });+});diff --git a/tests/core/store/lane-records.test.ts b/tests/core/store/lane-records.test.tsindex d4b60c82..207f304e 100644--- a/tests/core/store/lane-records.test.ts+++ b/tests/core/store/lane-records.test.ts@@ -110,4 +110,21 @@ describe("lane entries", () => {expect(laneNameOf(".sessions")).toBe("sessions");expect(laneNameOf("Procedural Memory")).toBe("procedural-memory");});++ test("the fold ignores a raw that is not a string", () => {+ const record = rec({+ physical: 1,+ kind: LANE_ENTRY_KIND,+ body: authored("agent", {+ lane: "obligations",+ entry: "obl-a",+ fields: { s: 1 },+ text: "t",+ raw: 5 as never,+ }),+ });+ const fold = foldLaneEntries([record]);+ expect(fold.rejected).toEqual([]);+ expect(fold.entries.get(laneEntryKey("obligations", "obl-a"))!.value?.raw).toBeUndefined();+ });});diff --git a/tests/core/store/partition-leave.test.ts b/tests/core/store/partition-leave.test.tsnew file mode 100644index 00000000..350abc7d--- /dev/null+++ b/tests/core/store/partition-leave.test.ts@@ -0,0 +1,396 @@+/**+ * Leaving a partition (design D7 of change `store-hardening`): the core+ * refuses what it cannot do safely, clears this machine's device-local+ * state of the partition, and a lost identity file fails closed until the+ * verb runs.+ */+import { afterEach, describe, expect, test } from "bun:test";+import {+ chmodSync,+ existsSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ readdirSync,+ rmSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { processDirsEnv } from "../../../src/core/platform-dirs.ts";+import { appendRecord } from "../../../src/core/store/appender.ts";+import { digestCacheRoot } from "../../../src/core/brain/compiled-digest-path.ts";+import { deviceStateDirFor } from "../../../src/core/store/store-paths.ts";+import {+ isHeldPartitionMarked,+ recordHeldPartitions,+} from "../../../src/core/store/held-ledger.ts";+import {+ HeldPartitionLostError,+ openStore,+ readStoreId,+} from "../../../src/core/store/store-open.ts";+import { leavePartition, PartitionLeaveRefusal } from "../../../src/core/store/partition-leave.ts";+import { initStore } from "../../../src/core/store/store-init.ts";+import { readStreamRecords } from "../../../src/core/store/read/stream-read.ts";+import { STORE_CATCH_UP_INTERVAL_ENV } from "../../../src/core/search/brain-store/constants.ts";+import { catchUpBrainStore } from "../../../src/core/search/brain-store/catch-up.ts";+import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { TRANSPORT_REACH } from "../../../src/core/graph/transport-reach.ts";+import { LOCAL_DEVICE, makeStoreIndexFixture } from "../search/fixtures/brain-store-index.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++let parent: string;+afterEach(() => {+ if (parent !== undefined) rmSync(parent, { recursive: true, force: true });+});++/** A store with both partitions, opened by dev-a (its markers recorded device-locally). */+function storeUnder(): string {+ parent = mkdtempSync(join(tmpdir(), "osb-partition-leave-"));+ const root = join(parent, "store");+ initStore({ root });+ openStore({ root, device: "dev-a" });+ return root;+}++/** Run `fn`, expect a `PartitionLeaveRefusal`, return it. */+function refusalOf(fn: () => unknown): PartitionLeaveRefusal {+ try {+ fn();+ } catch (err) {+ expect(err).toBeInstanceOf(PartitionLeaveRefusal);+ return err as PartitionLeaveRefusal;+ }+ throw new Error("expected a PartitionLeaveRefusal, none was thrown");+}++describe("leaving a partition, the core", () => {+ test("refuses the shared partition", () => {+ const root = storeUnder();+ const err = refusalOf(() => leavePartition({ root, partition: "shared" }));+ expect(err.reason).toBe("shared");+ });++ test("a lost shared/store.json names only the restore, not the verb", () => {+ const root = storeUnder();+ rmSync(join(root, "shared"), { recursive: true, force: true });+ try {+ openStore({ root, device: "dev-a" });+ throw new Error("expected a HeldPartitionLostError, none was thrown");+ } catch (err) {+ expect(err).toBeInstanceOf(HeldPartitionLostError);+ const message = String((err as Error).message);+ expect(message).not.toContain("partition leave");+ expect(message).toContain("shared/store.json");+ expect(message).toContain("from sync or a backup");+ }+ });++ test("losing both partitions names only the restores, not the verb", () => {+ const root = storeUnder();+ rmSync(join(root, "shared"), { recursive: true, force: true });+ rmSync(join(root, "private"), { recursive: true, force: true });+ try {+ openStore({ root, device: "dev-a" });+ throw new Error("expected a HeldPartitionLostError, none was thrown");+ } catch (err) {+ expect(err).toBeInstanceOf(HeldPartitionLostError);+ const message = String((err as Error).message);+ // The leave verb reads the store id from shared/store.json before it+ // clears anything, so with shared lost it refuses with no-store-id+ // and changes nothing: only the restores can work.+ expect(message).not.toContain("partition leave");+ expect(message).toContain("shared/store.json");+ expect(message).toContain("private/store.json");+ expect(message).toContain("from sync or a backup");+ }+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ });++ test("refuses while private/store.json is present", () => {+ const root = storeUnder();+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("still-held");+ // The file may have arrived by sync alone: the message must not claim+ // this machine holds the partition.+ expect(err.message).not.toContain("still holds");+ });++ test("a lost private/store.json without the verb still fails closed", () => {+ const root = storeUnder();+ rmSync(join(root, "private"), { recursive: true, force: true });+ expect(() => openStore({ root, device: "dev-a" })).toThrow(HeldPartitionLostError);+ try {+ openStore({ root, device: "dev-a" });+ } catch (err) {+ expect(String((err as Error).message)).toContain("o2b store partition leave private");+ expect(String((err as Error).message)).toContain("from sync or a backup");+ }+ });++ test("leaving private turns the machine shared-only", () => {+ const root = storeUnder();+ rmSync(join(root, "private"), { recursive: true, force: true });+ const result = leavePartition({ root, partition: "private" });+ expect(result.left).toBe(true);+ expect(result.cleared.heldMarker).toBe(true);+ const handle = openStore({ root, device: "dev-a" });+ expect(handle.partitions.has("shared")).toBe(true);+ expect(handle.partitions.has("private")).toBe(false);+ });++ test("nothing to leave changes nothing", () => {+ const root = storeUnder();+ rmSync(join(root, "private"), { recursive: true, force: true });+ leavePartition({ root, partition: "private" });+ const second = leavePartition({ root, partition: "private" });+ expect(second.left).toBe(false);+ expect(second.cleared.heldMarker).toBe(false);+ });++ test("files left refuse without force", () => {+ const root = storeUnder();+ rmSync(join(root, "private", "store.json"));+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("files-left");+ expect(() => openStore({ root, device: "dev-a" })).toThrow(HeldPartitionLostError);+ const forced = leavePartition({ root, partition: "private", force: true });+ expect(forced.left).toBe(true);+ const handle = openStore({ root, device: "dev-a" });+ expect(handle.partitions.has("shared")).toBe(true);+ expect(handle.partitions.has("private")).toBe(false);+ });++ test("a partition folder that is a regular file refuses files-left, marker kept", () => {+ const root = storeUnder();+ rmSync(join(root, "private"), { recursive: true, force: true });+ writeFileSync(join(root, "private"), "a sync conflict landed here\n");+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("files-left");+ expect(err.message).toContain(join(root, "private"));+ expect(err.message).toContain("ENOTDIR");+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "a seen-shards ledger that cannot be written refuses clear-failed, marker last",+ () => {+ const root = storeUnder();+ const store = openStore({ root, device: "dev-a" });+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "t",+ body: { text: "x" },+ });+ readStreamRecords(root, "private", { stream: "events" });+ const storeId = readStoreId(join(root, "shared")) as string;+ const ledgerDir = deviceStateDirFor(storeId, processDirsEnv());+ const ledger = readdirSync(ledgerDir).find(+ (name) => name.startsWith("seen-shards-") && name.endsWith(".json"),+ );+ expect(ledger).toBeDefined();+ rmSync(join(root, "private"), { recursive: true, force: true });+ chmodSync(ledgerDir, 0o500);+ try {+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("clear-failed");+ // The marker goes last: a leave it could not clear is still held,+ // and the store still fails closed.+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ } finally {+ chmodSync(ledgerDir, 0o700);+ }+ expect(() => openStore({ root, device: "dev-a" })).toThrow(HeldPartitionLostError);+ },+ );++ test.each([+ ["unparseable", "{ not json"],+ ["of another version", '{"version":2,"shards":{"private":["events/dev-a/2026-01.jsonl"]}}'],+ ["with no shards map", '{"version":1}'],+ ["with a partition that is no list", '{"version":1,"shards":{"private":{}}}'],+ ["that is a list", "[]"],+ ["with an entry that is no shard name", '{"version":1,"shards":{"private":[42]}}'],+ ["with an unknown partition", '{"version":1,"shards":{"other":[]}}'],+ ])("a seen-shards ledger %s refuses clear-failed, marker kept, bytes unchanged", (_, bytes) => {+ const root = storeUnder();+ const store = openStore({ root, device: "dev-a" });+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "t",+ body: { text: "x" },+ });+ readStreamRecords(root, "private", { stream: "events" });+ const storeId = readStoreId(join(root, "shared")) as string;+ const ledgerDir = deviceStateDirFor(storeId, processDirsEnv());+ const ledger = readdirSync(ledgerDir).find(+ (name) => name.startsWith("seen-shards-") && name.endsWith(".json"),+ );+ expect(ledger).toBeDefined();+ const ledgerFile = join(ledgerDir, ledger as string);+ rmSync(join(root, "private"), { recursive: true, force: true });+ writeFileSync(ledgerFile, bytes);+ const before = readFileSync(ledgerFile);+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("clear-failed");+ expect(err.message).toContain("seen-shards ledger");+ expect(err.message).toContain(ledgerFile);+ // The entries stay on disk and the marker is kept: the store still+ // fails closed, so the machine still holds the partition.+ expect(readFileSync(ledgerFile).equals(before)).toBe(true);+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ expect(() => openStore({ root, device: "dev-a" })).toThrow(HeldPartitionLostError);+ });++ test("a missing seen-shards ledger still leaves cleanly", () => {+ const root = storeUnder();+ const store = openStore({ root, device: "dev-a" });+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "t",+ body: { text: "x" },+ });+ const storeId = readStoreId(join(root, "shared")) as string;+ const ledgerDir = deviceStateDirFor(storeId, processDirsEnv());+ rmSync(ledgerDir, { recursive: true, force: true });+ rmSync(join(root, "private"), { recursive: true, force: true });+ const result = leavePartition({ root, partition: "private" });+ expect(result.left).toBe(true);+ expect(result.cleared.heldMarker).toBe(true);+ expect(result.cleared.seenShards).toBe(0);+ expect(isHeldPartitionMarked(root, "private")).toBe(false);+ });++ test("clears the seen shards and the digest caches", () => {+ const root = storeUnder();+ const store = openStore({ root, device: "dev-a" });+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "t",+ body: { text: "x" },+ });+ readStreamRecords(root, "private", { stream: "events" });+ const storeId = readStoreId(join(root, "shared")) as string;+ const cacheFile = join(digestCacheRoot(storeId), "k", "active.md");+ mkdirSync(dirname(cacheFile), { recursive: true });+ writeFileSync(cacheFile, "cache text\n");+ rmSync(join(root, "private"), { recursive: true, force: true });+ const result = leavePartition({ root, partition: "private" });+ expect(result.cleared.seenShards).toBeGreaterThanOrEqual(1);+ expect(result.cleared.digestCaches).toBe(true);+ expect(existsSync(cacheFile)).toBe(false);+ });++ test("a store.json that returns is held again", () => {+ const root = storeUnder();+ rmSync(join(root, "private"), { recursive: true, force: true });+ leavePartition({ root, partition: "private" });+ initStore({ root, partitions: ["shared", "private"] });+ openStore({ root, device: "dev-a" });+ rmSync(join(root, "private", "store.json"));+ expect(() => openStore({ root, device: "dev-a" })).toThrow(HeldPartitionLostError);+ });++ test.each([+ ["its folder removed", false],+ ["its files left and --force", true],+ ])("the search index forgets the partition on its next catch-up, %s", async (_, force) => {+ const fx = makeStoreIndexFixture("osb-partition-leave-index-");+ const saved = process.env[STORE_CATCH_UP_INTERVAL_ENV];+ process.env[STORE_CATCH_UP_INTERVAL_ENV] = "0";+ try {+ const store = fx.open(LOCAL_DEVICE);+ expect(store.partitions.has("private")).toBe(true);+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "agent-a",+ body: { text: "the private heron sighting" },+ });+ fx.writeNote("notes/a.md", "# A\n\nplain note\n");+ await indexVault(fx.config());+ await catchUpBrainStore(fx.config(), { force: true });+ const local = await search(fx.config(), {+ query: "heron sighting",+ limit: 10,+ transportReach: TRANSPORT_REACH.local,+ });+ expect(local.results.length).toBeGreaterThanOrEqual(1);+ recordHeldPartitions(fx.root, ["shared", "private"]);+ // With --force the shards stay on disk unchanged: the catch-up of+ // this same process must still stop listing them.+ if (force) rmSync(join(fx.root, "private", "store.json"));+ else rmSync(join(fx.root, "private"), { recursive: true, force: true });+ leavePartition({ root: fx.root, partition: "private", force });+ await catchUpBrainStore(fx.config(), { force: true });+ const after = await search(fx.config(), {+ query: "heron sighting",+ limit: 10,+ transportReach: TRANSPORT_REACH.local,+ });+ expect(after.results.length).toBe(0);+ } finally {+ if (saved === undefined) delete process.env[STORE_CATCH_UP_INTERVAL_ENV];+ else process.env[STORE_CATCH_UP_INTERVAL_ENV] = saved;+ fx.cleanup();+ }+ });++ test("an unreadable shared store id refuses and changes nothing", () => {+ const root = storeUnder();+ const store = openStore({ root, device: "dev-a" });+ appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "decision",+ agent: "t",+ body: { text: "x" },+ });+ readStreamRecords(root, "private", { stream: "events" });+ const storeId = readStoreId(join(root, "shared")) as string;+ const cacheFile = join(digestCacheRoot(storeId), "k", "active.md");+ mkdirSync(dirname(cacheFile), { recursive: true });+ writeFileSync(cacheFile, "cache text\n");+ const sharedId = join(root, "shared", "store.json");+ const sharedText = readFileSync(sharedId, "utf8");+ rmSync(join(root, "private"), { recursive: true, force: true });+ writeFileSync(sharedId, "{");+ const err = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(err.reason).toBe("no-store-id");+ expect(err.message).toContain("shared");+ expect(err.message).toContain("store.json");+ // The read's own error is carried and names the file once, so a+ // corrupt file is told apart from a missing one and the path is not+ // repeated; a file known to be present is not called missing.+ expect(err.message).toContain("is not valid JSON");+ expect(err.message).not.toContain("missing or unreadable");+ expect(err.message.split(join(root, "shared", "store.json")).length - 1).toBe(1);+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ expect(existsSync(cacheFile)).toBe(true);+ rmSync(sharedId);+ const again = refusalOf(() => leavePartition({ root, partition: "private" }));+ expect(again.reason).toBe("no-store-id");+ // A missing file is said to be missing, with the path once.+ expect(again.message).toContain("store.json is missing");+ expect(again.message.split(join(root, "shared", "store.json")).length - 1).toBe(1);+ expect(isHeldPartitionMarked(root, "private")).toBe(true);+ expect(existsSync(cacheFile)).toBe(true);+ writeFileSync(sharedId, sharedText);+ const result = leavePartition({ root, partition: "private" });+ expect(result.left).toBe(true);+ expect(result.cleared.seenShards).toBeGreaterThanOrEqual(1);+ expect(result.cleared.digestCaches).toBe(true);+ });+});diff --git a/tests/core/store/shard-quarantine.test.ts b/tests/core/store/shard-quarantine.test.tsnew file mode 100644index 00000000..257a5ae1--- /dev/null+++ b/tests/core/store/shard-quarantine.test.ts@@ -0,0 +1,712 @@+/**+ * The device-local quarantine of a shard line (change store-hardening,+ * design D4; spec `store-shard-gaps`): a line recorded by its byte offset+ * and the hash of its bytes reads as no gap on this machine, the shard's+ * bytes stay as they are, and any other line, or other bytes at the same+ * offset, stays a gap. A pending blob stops counting as a gap but its+ * record is still left out until the blob arrives, and the migration's+ * refusals still see it.+ */+import { afterEach, describe, expect, test } from "bun:test";+import {+ existsSync,+ mkdirSync,+ readFileSync,+ renameSync,+ rmSync,+ statSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";+import { dirname, join, relative } from "node:path";++import { buildCaptureBoundary } from "../../../src/core/brain/capture-boundary.ts";+import { BrainSettingsUnreadableError } from "../../../src/core/brain/policy/errors.ts";+import { readBrainSettingsFromStore } from "../../../src/core/brain/policy/settings-source.ts";+import { brainSettingKey } from "../../../src/core/store-migration/brain-settings.ts";+import { readHeldRecords } from "../../../src/core/store-migration/journal.ts";+import { appendRecord } from "../../../src/core/store/appender.ts";+import { laneStoreFold } from "../../../src/core/store/lane-fold-cache.ts";+import {+ PINNED_SET_KIND,+ SETTING_SET_KIND,+ STORE_LINE_CAP_BYTES,+} from "../../../src/core/store/constants.ts";+import { LANE_ENTRY_KIND } from "../../../src/core/store/lane-records.ts";+import { readEntitiesOf } from "../../../src/core/store/read/store-entities.ts";+import { readStreamRecords } from "../../../src/core/store/read/stream-read.ts";+import { readShard, shardReadGap } from "../../../src/core/store/reader.ts";+import {+ addQuarantine,+ QuarantineLedgerUnreadableError,+ releaseQuarantine,+} from "../../../src/core/store/shard-quarantine-write.ts";+import {+ dropLoadedLedger,+ ledgerPath,+ lineSha256At,+ listQuarantine,+ QUARANTINE_HASH_CAP_BYTES,+ type QuarantineShardReader,+} from "../../../src/core/store/shard-quarantine.ts";+import { inspectStore, openStore } from "../../../src/core/store/store-open.ts";+import { partitionDir } from "../../../src/core/store/store-paths.ts";+import { switchDevice } from "../brain/store-only/fixtures.ts";+import { makeStoreEnv, type StoreEnv } from "./fixtures/store-env.ts";++let env: StoreEnv | null = null;+afterEach(() => {+ env?.cleanup();+ env = null;+});++const NOV = Date.UTC(2026, 10, 6, 9);+const QUARANTINE = join(import.meta.dir, "../../../src/core/store/shard-quarantine.ts");+const QUARANTINE_WRITE = join(import.meta.dir, "../../../src/core/store/shard-quarantine-write.ts");++function note(e: StoreEnv, body: Record<string, unknown>): string {+ return appendRecord(+ openStore({ root: e.storeRoot, device: "dev-b" }),+ { partition: "shared", stream: "events", kind: "note", agent: "t", body },+ { now: () => NOV },+ ).path;+}++const shared = (e: StoreEnv): string => partitionDir(e.storeRoot, "shared");+const relOf = (e: StoreEnv, path: string): string =>+ relative(shared(e), path).split("\\").join("/");+const gapOf = (e: StoreEnv, path: string): string | null =>+ shardReadGap(readShard(path, { partitionRoot: shared(e) }));+const events = (e: StoreEnv) => readStreamRecords(e.storeRoot, "shared", { stream: "events" });++/** The byte offset of line `index` (0-based) of the shard at `path`. */+function offsetOfLine(path: string, index: number): number {+ const lines = readFileSync(path, "utf8").split("\n");+ return lines.slice(0, index).reduce((sum, line) => sum + Buffer.byteLength(line) + 1, 0);+}++/** Rewrite line `index` of the shard at `path` with `edit`. */+function editLine(path: string, index: number, edit: (line: string) => string): void {+ const lines = readFileSync(path, "utf8").split("\n");+ lines[index] = edit(lines[index]!);+ writeFileSync(path, lines.join("\n"));+}++/** `count` notes (default three) in one shard; returns the shard's path. */+function shardOf(e: StoreEnv, count = 3): string {+ let path = "";+ for (let n = 1; n <= count; n += 1) path = note(e, { n });+ return path;+}++function quarantine(+ e: StoreEnv,+ path: string,+ offset: number,+ kind: "corrupt-line" | "blob-pending" = "corrupt-line",+ sha256 = lineSha256At(path, offset, shared(e))!,+): "quarantined" | "already" {+ return addQuarantine(e.storeRoot, {+ partition: "shared",+ shard: relOf(e, path),+ offset,+ sha256,+ kind,+ reason: "test",+ });+}++const breakLine = (line: string): string => line.replace(/"n":(\d+)/, '"n":9$1');++/** A shard whose second record's body is a blob that has not arrived; returns the blob dir mover. */+function pendingBlob(e: StoreEnv): { path: string; restore: () => void } {+ note(e, { n: 1 });+ const path = note(e, { n: 2, text: "x".repeat(20_000) });+ const blobs = join(shared(e), "blobs");+ const kept = join(e.storeRoot, "..", "blobs-aside");+ rmSync(kept, { recursive: true, force: true });+ renameSync(blobs, kept);+ return { path, restore: () => renameSync(kept, blobs) };+}++describe("a quarantined corrupt line", () => {+ test("reads as no gap, and the shard's bytes are unchanged", () => {+ env = makeStoreEnv("osb-quarantine-corrupt-", { legacyTree: false });+ const path = shardOf(env);+ editLine(path, 1, breakLine);+ expect(gapOf(env, path)).not.toBeNull();+ const before = readFileSync(path);+ expect(quarantine(env, path, offsetOfLine(path, 1))).toBe("quarantined");+ expect(quarantine(env, path, offsetOfLine(path, 1))).toBe("already");+ expect(gapOf(env, path)).toBeNull();+ expect(readFileSync(path).equals(before)).toBe(true);+ });++ test("other bytes at the same offset are a gap again", () => {+ env = makeStoreEnv("osb-quarantine-replaced-", { legacyTree: false });+ const path = shardOf(env);+ editLine(path, 1, breakLine);+ quarantine(env, path, offsetOfLine(path, 1));+ editLine(path, 1, (line) => line.replace('"n":92', '"n":93'));+ expect(gapOf(env, path)).not.toBeNull();+ });++ test("a second corrupt line in the same shard is still a gap", () => {+ env = makeStoreEnv("osb-quarantine-second-", { legacyTree: false });+ const path = shardOf(env, 4);+ editLine(path, 1, breakLine);+ quarantine(env, path, offsetOfLine(path, 1));+ expect(gapOf(env, path)).toBeNull();+ editLine(path, 2, breakLine);+ const gap = gapOf(env, path);+ expect(gap).toContain(`at byte ${offsetOfLine(path, 2)}`);+ expect(gap).not.toContain(`at byte ${offsetOfLine(path, 1)}:`);+ });++ test("a newer-format line is never marked", () => {+ env = makeStoreEnv("osb-quarantine-newer-", { legacyTree: false });+ const path = shardOf(env);+ editLine(path, 1, (line) => line.replace('"v":1', '"v":99'));+ expect(gapOf(env, path)).toContain("unknown-version");+ quarantine(env, path, offsetOfLine(path, 1));+ expect(gapOf(env, path)).toContain("unknown-version");+ });++ test("a release brings the gap back", () => {+ env = makeStoreEnv("osb-quarantine-release-", { legacyTree: false });+ const path = shardOf(env);+ editLine(path, 1, breakLine);+ const offset = offsetOfLine(path, 1);+ quarantine(env, path, offset);+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset)).toBe("released");+ expect(gapOf(env, path)).not.toBeNull();+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset)).toBe(+ "not-quarantined",+ );+ });++ test("two processes quarantining at once both land", async () => {+ env = makeStoreEnv("osb-quarantine-race-", { legacyTree: false });+ const path = shardOf(env, 4);+ editLine(path, 1, breakLine);+ editLine(path, 2, breakLine);+ const dir = join(env.storeRoot, "..");+ const go = join(dir, "go");+ const script = join(dir, "quarantine.ts");+ // Each child hashes its line, says it is ready, and waits for the+ // parent's go: both writes then start together, inside the lock's reach.+ writeFileSync(+ script,+ `import { existsSync, writeFileSync } from "node:fs";+import { addQuarantine } from ${JSON.stringify(QUARANTINE_WRITE)};+import { lineSha256At } from ${JSON.stringify(QUARANTINE)};+const [root, shard, path, partitionRoot, offset, ready, go] = process.argv.slice(2);+const sha256 = lineSha256At(path, Number(offset), partitionRoot);+writeFileSync(ready, "");+while (!existsSync(go)) Bun.sleepSync(1);+addQuarantine(root, { partition: "shared", shard, offset: Number(offset),+ sha256, kind: "corrupt-line", reason: "race" });+`,+ );+ const ready = [1, 2].map((line) => join(dir, `ready-${line}`));+ const procs = [1, 2].map((line, i) =>+ Bun.spawn(+ [+ process.execPath,+ script,+ env!.storeRoot,+ relOf(env!, path),+ path,+ shared(env!),+ String(offsetOfLine(path, line)),+ ready[i]!,+ go,+ ],+ { env: process.env, stdout: "ignore", stderr: "inherit" },+ ),+ );+ const deadline = Date.now() + 20_000;+ while (!ready.every((r) => existsSync(r)) && Date.now() < deadline) Bun.sleepSync(5);+ writeFileSync(go, "");+ for (const p of procs) expect(await p.exited).toBe(0);+ expect(gapOf(env, path)).toBeNull();+ }, 30_000);++ test("an oversized line is quarantinable up to the hash cap", () => {+ env = makeStoreEnv("osb-quarantine-oversized-", { legacyTree: false });+ const path = shardOf(env);+ editLine(path, 1, () => "x".repeat(2 * STORE_LINE_CAP_BYTES));+ expect(gapOf(env, path)).toContain("line-size cap");+ quarantine(env, path, offsetOfLine(path, 1));+ expect(gapOf(env, path)).toBeNull();+ });+});++describe("lineSha256At", () => {+ test("a shard name that is a symbolic link gives null", () => {+ env = makeStoreEnv("osb-quarantine-hash-link-", { legacyTree: false });+ const path = shardOf(env);+ const link = join(dirname(path), "2020-01.jsonl");+ symlinkSync(path, link);+ expect(lineSha256At(path, 0, shared(env))).not.toBeNull();+ expect(lineSha256At(link, 0, shared(env))).toBeNull();+ });++ test("a line past the hash cap gives null; one at the cap is hashed", () => {+ env = makeStoreEnv("osb-quarantine-hash-cap-", { legacyTree: false });+ const dir = join(shared(env), "events", "dev-c");+ mkdirSync(dir, { recursive: true });+ const atCap = join(dir, "2026-01.jsonl");+ writeFileSync(atCap, `${"x".repeat(QUARANTINE_HASH_CAP_BYTES - 1)}\n`);+ expect(lineSha256At(atCap, 0, shared(env))).not.toBeNull();+ const past = join(dir, "2026-02.jsonl");+ writeFileSync(past, `${"x".repeat(QUARANTINE_HASH_CAP_BYTES)}\n`);+ expect(lineSha256At(past, 0, shared(env))).toBeNull();+ const noNewline = join(dir, "2026-03.jsonl");+ writeFileSync(noNewline, "x".repeat(QUARANTINE_HASH_CAP_BYTES + 1));+ expect(lineSha256At(noNewline, 0, shared(env))).toBeNull();+ });+});++const ledgerOf = (e: StoreEnv): string => ledgerPath(e.storeRoot);++function brokenLine(e: StoreEnv): { path: string; offset: number } {+ const path = shardOf(e);+ editLine(path, 1, breakLine);+ return { path, offset: offsetOfLine(path, 1) };+}++describe("the ledger write", () => {+ for (const [name, text] of [+ ["does not parse", "{not json"],+ ["has another version", JSON.stringify({ version: 2, lines: [] })],+ [+ "holds an entry this release does not read",+ JSON.stringify({+ version: 1,+ lines: [+ {+ partition: "shared",+ shard: "events/dev-z/2026-01.jsonl",+ offset: 0,+ sha256: "0".repeat(64),+ kind: "future-kind",+ reason: "newer build",+ quarantined_at: "2026-01-01T00:00:00.000Z",+ },+ ],+ }),+ ],+ ] as const) {+ test(`a ledger that ${name} is refused, not rewritten`, () => {+ env = makeStoreEnv("osb-quarantine-ledger-strict-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ mkdirSync(dirname(ledgerOf(env)), { recursive: true });+ writeFileSync(ledgerOf(env), text);+ expect(() => quarantine(env!, path, offset)).toThrow(QuarantineLedgerUnreadableError);+ expect(() => releaseQuarantine(env!.storeRoot, "shared", relOf(env!, path), offset)).toThrow(+ QuarantineLedgerUnreadableError,+ );+ expect(readFileSync(ledgerOf(env), "utf8")).toBe(text);+ });+ }++ test("a ledger that cannot be read is refused with its own detail, not as JSON", () => {+ env = makeStoreEnv("osb-quarantine-ledger-unreadable-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ // A folder where the file belongs: it exists, and reading it fails.+ mkdirSync(ledgerOf(env), { recursive: true });+ let thrown: unknown = null;+ try {+ quarantine(env, path, offset);+ } catch (err) {+ thrown = err;+ }+ expect(thrown).toBeInstanceOf(QuarantineLedgerUnreadableError);+ const message = (thrown as Error).message;+ expect(message).toContain("could not be read (");+ expect(message).toContain("EISDIR");+ expect(message).not.toContain("JSON");+ expect(message).toContain("Check that this account can read it");+ expect(statSync(ledgerOf(env)).isDirectory()).toBe(true);+ });++ test("a ledger that does not parse names the parse, not the read", () => {+ env = makeStoreEnv("osb-quarantine-ledger-parse-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ mkdirSync(dirname(ledgerOf(env)), { recursive: true });+ writeFileSync(ledgerOf(env), "{not json");+ expect(() => quarantine(env!, path, offset)).toThrow("does not parse as JSON");+ });++ test("a no-op leaves the ledger file untouched", () => {+ env = makeStoreEnv("osb-quarantine-ledger-noop-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ quarantine(env, path, offset);+ const stamp = (): string => {+ const stat = statSync(ledgerOf(env!));+ return `${stat.ino}:${stat.mtimeMs}`;+ };+ const before = stamp();+ expect(quarantine(env, path, offset)).toBe("already");+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset + 1)).toBe(+ "not-quarantined",+ );+ expect(stamp()).toBe(before);+ });+});++describe("the ledger read", () => {+ /** A quarantined corrupt line whose ledger is then rewritten by `rewrite`. */+ function rewritten(rewrite: (ledger: Record<string, unknown>) => unknown): string {+ env = makeStoreEnv("osb-quarantine-ledger-read-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ quarantine(env, path, offset);+ expect(gapOf(env, path)).toBeNull();+ const ledger = JSON.parse(readFileSync(ledgerOf(env), "utf8")) as Record<string, unknown>;+ writeFileSync(ledgerOf(env), JSON.stringify(rewrite(ledger)));+ // The rewrite may keep the size and land within the cached mtime.+ dropLoadedLedger(ledgerOf(env));+ return path;+ }++ for (const [name, rewrite] of [+ ["another version", (l: Record<string, unknown>) => ({ ...l, version: 2 })],+ ["no version", (l: Record<string, unknown>) => ({ root: l["root"], lines: l["lines"] })],+ ["no list of lines", (l: Record<string, unknown>) => ({ ...l, lines: { 0: l["lines"] } })],+ ["no object", (l: Record<string, unknown>) => l["lines"]],+ ] as const) {+ test(`a ledger with ${name} applies no entry, so the gap stays`, () => {+ const path = rewritten(rewrite);+ expect(gapOf(env!, path)).not.toBeNull();+ expect(listQuarantine(env!.storeRoot, readShard)).toEqual([]);+ });+ }++ test("an entry this release does not read applies nothing; the others still apply", () => {+ const path = rewritten((l) => ({+ ...l,+ lines: [+ ...(l["lines"] as unknown[]),+ { ...(l["lines"] as Record<string, unknown>[])[0]!, offset: 0, kind: "future-kind" },+ ],+ }));+ expect(gapOf(env!, path)).toBeNull();+ expect(listQuarantine(env!.storeRoot, readShard)).toHaveLength(1);+ });+});++describe("a quarantined pending blob", () => {+ test("stops counting, stays out of the read, and reads with its body once it arrives", () => {+ env = makeStoreEnv("osb-quarantine-blob-", { legacyTree: false });+ const { path, restore } = pendingBlob(env);+ expect(gapOf(env, path)).toContain("has not arrived");+ quarantine(env, path, offsetOfLine(path, 1), "blob-pending");+ expect(gapOf(env, path)).toBeNull();+ expect(events(env).unreadable).toEqual([]);+ expect(events(env).records.map((r) => r.body["n"])).toEqual([1]);+ restore();+ const read = events(env);+ expect(read.records.map((r) => r.body["n"])).toEqual([1, 2]);+ expect(read.records[1]!.body["text"]).toBe("x".repeat(20_000));+ });++ test("an entry with another line hash at the same offset does not mark it", () => {+ env = makeStoreEnv("osb-quarantine-blob-hash-", { legacyTree: false });+ const { path } = pendingBlob(env);+ quarantine(env, path, offsetOfLine(path, 1), "blob-pending", "0".repeat(64));+ expect(gapOf(env, path)).toContain("has not arrived");+ });++ test("a corrupt-line entry at the same offset and bytes does not mark it", () => {+ env = makeStoreEnv("osb-quarantine-blob-kind-", { legacyTree: false });+ const { path } = pendingBlob(env);+ quarantine(env, path, offsetOfLine(path, 1), "corrupt-line");+ expect(gapOf(env, path)).toContain("has not arrived");+ });++ test("quarantining the same bytes as the other kind replaces the entry and marks the record", () => {+ env = makeStoreEnv("osb-quarantine-blob-rekind-", { legacyTree: false });+ const { path } = pendingBlob(env);+ const offset = offsetOfLine(path, 1);+ expect(quarantine(env, path, offset, "corrupt-line")).toBe("quarantined");+ expect(gapOf(env, path)).toContain("has not arrived");+ expect(quarantine(env, path, offset, "blob-pending")).toBe("quarantined");+ expect(gapOf(env, path)).toBeNull();+ expect(listQuarantine(env.storeRoot, readShard).map((e) => e.kind)).toEqual(["blob-pending"]);+ expect(quarantine(env, path, offset, "blob-pending")).toBe("already");+ });++ test("a blob-pending entry over a corrupt line is replaced by a corrupt-line entry", () => {+ env = makeStoreEnv("osb-quarantine-line-rekind-", { legacyTree: false });+ const { path, offset } = brokenLine(env);+ expect(quarantine(env, path, offset, "blob-pending")).toBe("quarantined");+ expect(gapOf(env, path)).not.toBeNull();+ expect(quarantine(env, path, offset, "corrupt-line")).toBe("quarantined");+ expect(gapOf(env, path)).toBeNull();+ expect(listQuarantine(env.storeRoot, readShard).map((e) => e.kind)).toEqual(["corrupt-line"]);+ });++ test("a blob-pending entry never marks a record of a settings shard", () => {+ env = makeStoreEnv("osb-quarantine-blob-settings-", { legacyTree: false });+ const path = appendRecord(+ openStore({ root: env.storeRoot, device: "dev-b" }),+ {+ partition: "shared",+ stream: "settings",+ kind: SETTING_SET_KIND,+ agent: "t",+ body: { authority: "human", key: "brain.x.y", value: "x".repeat(20_000) },+ },+ { now: () => NOV },+ ).path;+ rmSync(join(shared(env), "blobs"), { recursive: true, force: true });+ quarantine(env, path, 0, "blob-pending");+ expect(gapOf(env, path)).toContain("has not arrived");+ });++ test("the migration still counts its shard as read only in part", () => {+ env = makeStoreEnv("osb-quarantine-blob-migration-", { legacyTree: false });+ const { path } = pendingBlob(env);+ quarantine(env, path, offsetOfLine(path, 1), "blob-pending");+ const store = inspectStore(env.storeRoot)!;+ expect(readHeldRecords(store).unseen.partial).toEqual([`shared/${relOf(env, path)}`]);+ });+});++describe("listQuarantine", () => {+ test("reads each shard once; a read that throws makes only its own entries held", () => {+ env = makeStoreEnv("osb-quarantine-list-once-", { legacyTree: false });+ const path = shardOf(env, 4);+ editLine(path, 1, breakLine);+ editLine(path, 2, breakLine);+ quarantine(env, path, offsetOfLine(path, 1));+ quarantine(env, path, offsetOfLine(path, 2));+ const { path: blobShard, restore } = pendingBlob(env);+ expect(blobShard).toBe(path); // same device, same month+ restore();+ const devC = appendRecord(+ openStore({ root: env.storeRoot, device: "dev-c" }),+ { partition: "shared", stream: "events", kind: "note", agent: "t", body: { n: 1 } },+ { now: () => NOV },+ ).path;+ appendRecord(+ openStore({ root: env.storeRoot, device: "dev-c" }),+ { partition: "shared", stream: "events", kind: "note", agent: "t", body: { n: 2 } },+ { now: () => NOV },+ );+ // dev-c's first line, read as a record, quarantined as if it were one.+ quarantine(env, devC, 0, "blob-pending");+ const calls: string[] = [];+ const reader: QuarantineShardReader = (shard, options) => {+ calls.push(shard);+ if (shard === devC) throw new Error("locked");+ return readShard(shard, options);+ };+ const states = listQuarantine(env.storeRoot, reader).map((l) => [l.shard, l.state]);+ expect(calls.toSorted()).toEqual([path, devC].toSorted());+ expect(states).toEqual([+ [relOf(env, path), "held"],+ [relOf(env, path), "held"],+ [relOf(env, devC), "held"],+ ]);+ expect(listQuarantine(env.storeRoot, readShard).map((l) => l.state)).toEqual([+ "held",+ "held",+ "resolved",+ ]);+ });++ test("reports held, resolved and stale", () => {+ env = makeStoreEnv("osb-quarantine-list-", { legacyTree: false });+ const { path: blobShard, restore } = pendingBlob(env);+ quarantine(env, blobShard, offsetOfLine(blobShard, 1), "blob-pending");+ restore();+ const other = note(env, { n: 3 }); // same shard: one month, one device+ expect(other).toBe(blobShard);+ // A second shard of another device, with two corrupt lines.+ const devC = (n: number): string =>+ appendRecord(+ openStore({ root: env!.storeRoot, device: "dev-c" }),+ { partition: "shared", stream: "events", kind: "note", agent: "t", body: { n } },+ { now: () => NOV },+ ).path;+ devC(1);+ devC(2);+ const held = devC(3);+ devC(4);+ editLine(held, 1, breakLine);+ editLine(held, 2, breakLine);+ quarantine(env, held, offsetOfLine(held, 1));+ quarantine(env, held, offsetOfLine(held, 2));+ editLine(held, 2, (line) => line.replace('"n":93', '"n":94'));+ const states = listQuarantine(env.storeRoot, readShard).map((l) => [+ l.shard,+ l.offset,+ l.state,+ ]);+ expect(states).toEqual([+ [relOf(env, blobShard), offsetOfLine(blobShard, 1), "resolved"],+ [relOf(env, held), offsetOfLine(held, 1), "held"],+ [relOf(env, held), offsetOfLine(held, 2), "stale"],+ ]);+ });+});++describe("capture", () => {+ test("a corrupt line in a shared settings shard closes capture; the quarantine opens it", () => {+ env = makeStoreEnv("osb-quarantine-capture-", { legacyTree: false });+ switchDevice(env.storeRoot);+ const store = openStore({ root: env.storeRoot, device: "dev-b" });+ const setting = (value: number): string =>+ appendRecord(store, {+ partition: "shared",+ stream: "settings",+ kind: SETTING_SET_KIND,+ agent: "test",+ body: {+ authority: "human",+ key: brainSettingKey("dream", "candidate_threshold"),+ value,+ },+ }).path;+ setting(4);+ setting(5);+ const path = setting(6);+ expect(buildCaptureBoundary(env.vault).sessionDecision("any-1")).toBe("capture");+ editLine(path, 1, (line) => line.replace('"value":5', '"value":7'));+ expect(buildCaptureBoundary(env.vault).sessionDecision("any-1")).toBe("ignore");+ quarantine(env, path, offsetOfLine(path, 1));+ expect(buildCaptureBoundary(env.vault).sessionDecision("any-1")).toBe("capture");+ });+});++describe("the cached settings read", () => {+ test("a release after a whole read fails closed again; a quarantine after a failed read passes", () => {+ env = makeStoreEnv("osb-quarantine-settings-cache-", { legacyTree: false });+ switchDevice(env.storeRoot);+ const store = openStore({ root: env.storeRoot, device: "dev-b" });+ const setting = (value: number): string =>+ appendRecord(store, {+ partition: "shared",+ stream: "settings",+ kind: SETTING_SET_KIND,+ agent: "test",+ body: {+ authority: "human",+ key: brainSettingKey("dream", "candidate_threshold"),+ value,+ },+ }).path;+ setting(4);+ setting(5);+ const path = setting(6);+ editLine(path, 1, (line) => line.replace('"value":5', '"value":7'));+ const read = () => readBrainSettingsFromStore({ configPath: env!.configPath });+ expect(read).toThrow(BrainSettingsUnreadableError);+ const offset = offsetOfLine(path, 1);+ quarantine(env, path, offset);+ expect(read().config.dream.candidate_threshold).toBe(6);+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset)).toBe("released");+ expect(read).toThrow(BrainSettingsUnreadableError);+ quarantine(env, path, offset);+ expect(read().config.dream.candidate_threshold).toBe(6);+ });+});++describe("the cached lane fold", () => {+ test("a release after a whole fold lists the shard again; a quarantine after a failed fold clears it", () => {+ env = makeStoreEnv("osb-quarantine-lane-cache-", { legacyTree: false });+ switchDevice(env.storeRoot);+ const path = shardOf(env);+ expect(laneStoreFold().unreadable).toEqual([]);+ editLine(path, 1, breakLine);+ const listed = [`shared/${relOf(env, path)}`];+ expect(laneStoreFold().unreadable).toEqual(listed);+ const offset = offsetOfLine(path, 1);+ quarantine(env, path, offset);+ expect(laneStoreFold().unreadable).toEqual([]);+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset)).toBe("released");+ expect(laneStoreFold().unreadable).toEqual(listed);+ quarantine(env, path, offset);+ expect(laneStoreFold().unreadable).toEqual([]);+ });+});++describe("the cached entity read", () => {+ test("a release after a whole read lists the shard again; a quarantine after a failed read clears it", () => {+ env = makeStoreEnv("osb-quarantine-entities-cache-", { legacyTree: false });+ const path = shardOf(env);+ const unreadable = () => readEntitiesOf(inspectStore(env!.storeRoot)!).unreadable;+ expect(unreadable()).toEqual([]);+ editLine(path, 1, breakLine);+ const listed = [`shared/${relOf(env, path)}`];+ expect(unreadable()).toEqual(listed);+ const offset = offsetOfLine(path, 1);+ quarantine(env, path, offset);+ expect(unreadable()).toEqual([]);+ expect(releaseQuarantine(env.storeRoot, "shared", relOf(env, path), offset)).toBe("released");+ expect(unreadable()).toEqual(listed);+ quarantine(env, path, offset);+ expect(unreadable()).toEqual([]);+ });+});++/**+ * An `events` record of `kind` whose body is a blob that has not arrived,+ * quarantined on this machine (so its shard reads as no gap); returns the+ * mover that delivers the blob.+ */+function quarantinedPending(e: StoreEnv, kind: string, body: Record<string, unknown>): () => void {+ note(e, { n: 1 });+ const path = appendRecord(+ openStore({ root: e.storeRoot, device: "dev-b" }),+ { partition: "shared", stream: "events", kind, agent: "t", body },+ { now: () => NOV },+ ).path;+ const blobs = join(shared(e), "blobs");+ const kept = join(e.storeRoot, "..", "blobs-aside");+ rmSync(kept, { recursive: true, force: true });+ renameSync(blobs, kept);+ quarantine(e, path, offsetOfLine(path, 1), "blob-pending");+ expect(gapOf(e, path)).toBeNull();+ return () => renameSync(kept, blobs);+}++const BIG = "x".repeat(20_000);++describe("a cached read that skipped a quarantined pending record", () => {+ test("the entity read shows the record once its blob arrives, in the same process", () => {+ env = makeStoreEnv("osb-quarantine-entities-blob-", { legacyTree: false });+ const deliver = quarantinedPending(env, PINNED_SET_KIND, {+ authority: "human",+ content: BIG,+ });+ const pinned = () =>+ readEntitiesOf(inspectStore(env!.storeRoot)!).pinned.get("shared")?.content ?? null;+ expect(pinned()).toBeNull();+ // Read twice before the blob arrives, so a cache would be warm.+ expect(pinned()).toBeNull();+ deliver();+ expect(pinned()).toBe(BIG);+ });++ test("the lane fold shows the entry once its blob arrives, in the same process", () => {+ env = makeStoreEnv("osb-quarantine-lane-blob-", { legacyTree: false });+ switchDevice(env.storeRoot);+ const deliver = quarantinedPending(env, LANE_ENTRY_KIND, {+ authority: "human",+ lane: "standing-rule",+ entry: "pending-entry",+ fields: {},+ text: BIG,+ });+ const text = () =>+ [...laneStoreFold().entries.values()].find((v) => v.text === BIG)?.text ?? null;+ expect(laneStoreFold().unreadable).toEqual([]);+ expect(text()).toBeNull();+ expect(text()).toBeNull();+ deliver();+ expect(text()).toBe(BIG);+ });+});diff --git a/tests/mcp/log-private-reach-surfaces.test.ts b/tests/mcp/log-private-reach-surfaces.test.tsnew file mode 100644index 00000000..a6fb280d--- /dev/null+++ b/tests/mcp/log-private-reach-surfaces.test.ts@@ -0,0 +1,969 @@+/**+ * Every reader with a remote surface withholds the log events of the+ * store's `private` partition below local reach (change store-hardening,+ * design D1 and D2; spec `remote-log-reads`). Each case has one private+ * event that names nothing unless it says otherwise, and asserts the+ * remote answer and the local one.+ */+import { afterEach, beforeAll, beforeEach, describe, expect, spyOn, test } from "bun:test";+import { appendFileSync, mkdirSync, readFileSync, utimesSync, writeFileSync } from "node:fs";+import { dirname, join } from "node:path";++import { appendLogEvent } from "../../src/core/brain/log.ts";+import { writePreference } from "../../src/core/brain/preference.ts";+import {+ existingPreferencePageText,+ writePreferencePageText,+} from "../../src/core/brain/preference-page-write.ts";+import { listRecallTelemetry } from "../../src/core/brain/recall-telemetry.ts";+import {+ BRAIN_CONFIDENCE,+ BRAIN_LOG_EVENT_KIND,+ BRAIN_PREFERENCE_STATUS,+} from "../../src/core/brain/types.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";+import { REMOTE_DENY_VISIBILITY_TOKEN } from "../../src/core/graph/visibility.ts";+import { appendRecord } from "../../src/core/store/appender.ts";+import { ledgerRowRecord } from "../../src/core/store/dual-write-records.ts";+import { withOperationPrivacy } from "../../src/core/store/operation-scope.ts";+import { routeWrite } from "../../src/core/store/store-write.ts";+import { openStore } from "../../src/core/store/store-open.ts";+import { recordNoteWrite, storeBeforeImage } from "../../src/core/brain/notes/write-record.ts";+import {+ computeTokenFootprint,+ TOKEN_WARN_THRESHOLD_DEFAULT,+} from "../../src/core/brain/token-footprint.ts";+import { readResource } from "../../src/mcp/resources.ts";+import { formatFrontmatter, parseFrontmatterText } from "../../src/core/vault.ts";+import { MCPServer } from "../../src/mcp/server.ts";+import { switchDevice } from "../core/brain/store-only/fixtures.ts";+import { makeStoreEnv, type StoreEnv } from "../core/store/fixtures/store-env.ts";++const SHARED_TEXT = "a shared note of the surfaces";+const PRIVATE_TEXT = "a private note that names nothing";++let env: StoreEnv;+beforeEach(() => {+ env = makeStoreEnv("osb-log-private-surfaces-");+ switchDevice(env.storeRoot);+});+afterEach(() => env.cleanup());++const REMOTE = TRANSPORT_REACH.remote;+const LOCAL = TRANSPORT_REACH.local;++async function call(+ tool: string,+ args: Record<string, unknown>,+ reach: TransportReach,+): Promise<Record<string, unknown>> {+ const server = new MCPServer({ vault: env.vault, configPath: env.configPath }, { reach });+ const result = (await server.callTool(tool, args)) as Record<string, unknown>;+ if (result["isError"] === true) throw new Error(`${tool} failed: ${JSON.stringify(result)}`);+ return (result["structuredContent"] ?? result) as Record<string, unknown>;+}++function json(value: unknown): string {+ return JSON.stringify(value);+}++/** The widest offsets {@link recentStamp} takes: up to this much later than its base... */+const RECENT_LEAD_MS = 1_000;+/** ...and up to this much earlier. */+const RECENT_LAG_MS = 5_000;++const DAY_MS = 24 * 60 * 60 * 1000;++/**+ * Wait out the first seconds of a UTC day, in which {@link recentStamp}'s+ * base plus its lead would still be in the future. Run once before the+ * file's tests with a timeout longer than the wait, so no test's own+ * timeout pays for it; a test that crosses midnight still waits here.+ */+function awaitRecentWindow(): void {+ const now = Date.now();+ const earliest = now - (now % DAY_MS) + RECENT_LAG_MS + RECENT_LEAD_MS;+ if (now < earliest) Bun.sleepSync(earliest - now);+}++beforeAll(awaitRecentWindow, 3 * (RECENT_LAG_MS + RECENT_LEAD_MS));++/**+ * An instant about one minute before now, `offsetMs` earlier still (a+ * negative offset is later, up to {@link RECENT_LEAD_MS}), inside the+ * daily, weekly, monthly, today and timeline windows, and never after now.+ * In the first minute of a UTC day the base is clamped to+ * {@link RECENT_LAG_MS} past the day's start, so no window boundary falls+ * between the events; in the first seconds of the day, while that base plus+ * the lead would still be in the future, it first waits until it is not.+ */+function recentStamp(offsetMs = 0): string {+ if (offsetMs < -RECENT_LEAD_MS || offsetMs >= RECENT_LAG_MS) {+ throw new Error(`recentStamp offset ${offsetMs} is outside its window`);+ }+ awaitRecentWindow();+ const now = Date.now();+ const dayStart = now - (now % DAY_MS);+ const stamp = Math.max(dayStart + RECENT_LAG_MS, now - 60_000) - offsetMs;+ return new Date(Math.min(stamp, now)).toISOString();+}++function privately<T>(fn: () => T): T {+ return withOperationPrivacy(true, fn);+}++/** One shared and one private note inside the window. */+function twoNotes(): string {+ const at = recentStamp();+ appendLogEvent(env.vault, { timestamp: at, eventType: "note", body: { text: SHARED_TEXT } });+ privately(() =>+ appendLogEvent(env.vault, {+ timestamp: recentStamp(-1000),+ eventType: "note",+ body: { text: PRIVATE_TEXT },+ }),+ );+ return at.slice(0, 10);+}++describe("D1: the timeline and the briefs built on requestRefView", () => {+ test("brain_analytics view=timeline", async () => {+ twoNotes();+ const remote = json(await call("brain_analytics", { view: "timeline" }, REMOTE));+ expect(remote).toContain(SHARED_TEXT);+ expect(remote).not.toContain(PRIVATE_TEXT);+ const local = json(await call("brain_analytics", { view: "timeline" }, LOCAL));+ expect(local).toContain(SHARED_TEXT);+ expect(local).toContain(PRIVATE_TEXT);+ });++ // The daily, weekly and monthly briefs render counts, not note text:+ // the remote count leaves the private note out, the local one counts it.+ test("brain_brief view=daily counts the shared note only", async () => {+ const date = twoNotes();+ const args = { view: "daily", date };+ expect(noteCount(await call("brain_brief", args, REMOTE))).toBe(1);+ expect(noteCount(await call("brain_brief", args, LOCAL))).toBe(2);+ });++ test("brain_brief view=weekly counts the shared note only", async () => {+ twoNotes();+ const weekEnd = new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString().slice(0, 10);+ const args = { view: "weekly", week_end: weekEnd };+ expect(noteCount(await call("brain_brief", args, REMOTE))).toBe(1);+ expect(noteCount(await call("brain_brief", args, LOCAL))).toBe(2);+ });++ test("brain_brief view=monthly counts the shared note only", async () => {+ const date = twoNotes();+ const args = { view: "monthly", month: date.slice(0, 7) };+ const events = async (reach: TransportReach): Promise<unknown> =>+ ((await call("brain_brief", args, reach))["summary"] as Record<string, unknown>)["events"];+ expect(await events(REMOTE)).toBe(1);+ expect(await events(LOCAL)).toBe(2);+ });++ test("brain_brief view=today", async () => {+ twoNotes();+ const remote = json(await call("brain_brief", { view: "today" }, REMOTE));+ expect(remote).toContain(SHARED_TEXT);+ expect(remote).not.toContain(PRIVATE_TEXT);+ const local = json(await call("brain_brief", { view: "today" }, LOCAL));+ expect(local).toContain(SHARED_TEXT);+ expect(local).toContain(PRIVATE_TEXT);+ });+});++/** A daily or weekly brief's count of note events. */+function noteCount(answer: Record<string, unknown>): number {+ return (answer["events_by_kind"] as Record<string, number>)["note"] ?? 0;+}++// ----- D2: the readers with a remote surface -------------------------------++const PREF_SLUG = "surfaced";+const PREF_ID = `pref-${PREF_SLUG}`;++/** A shared confirmed preference, as the reach fixture writes one. */+function confirmedPreference(slug = PREF_SLUG): void {+ writePreference(env.vault, {+ slug,+ topic: slug,+ principle: `Rule ${slug}.`,+ created_at: "2026-05-01T00:00:00Z",+ unconfirmed_until: "2026-05-08T00:00:00Z",+ status: BRAIN_PREFERENCE_STATUS.confirmed,+ evidenced_by: [],+ confirmed_at: "2026-05-02T00:00:00Z",+ applied_count: 1,+ violated_count: 0,+ last_evidence_at: "2026-05-02T00:00:00Z",+ confidence: BRAIN_CONFIDENCE.high,+ confidence_value: 0.8,+ });+}++function applied(at: string, artifact: string, slug = PREF_SLUG): void {+ appendLogEvent(env.vault, {+ timestamp: at,+ eventType: BRAIN_LOG_EVENT_KIND.applyEvidence,+ body: { preference: `[[pref-${slug}]]`, artifact: `[[${artifact}]]`, result: "applied" },+ } as Parameters<typeof appendLogEvent>[1]);+}++/** A vault page reserved against remote reads, as the reach fixture reserves one. */+function reservedPage(rel: string): void {+ mkdirSync(dirname(join(env.vault, rel)), { recursive: true });+ writeFileSync(+ join(env.vault, rel),+ `---\nvisibility: [${REMOTE_DENY_VISIBILITY_TOKEN}]\n---\n\nReserved.\n`,+ );+}++describe("D2 row 1: brain_query preference mode", () => {+ test("the remote evidence is the shared event only, and telemetry counts it", async () => {+ confirmedPreference();+ reservedPage("Notes/reserved.md");+ applied(recentStamp(3000), "Notes/shared-artifact");+ privately(() => applied(recentStamp(2000), "Notes/private-artifact"));+ applied(recentStamp(1000), "Notes/reserved");+ const args = { preference: PREF_ID, telemetry: true };+ const artifacts = (answer: Record<string, unknown>): string[] =>+ (answer["evidence"] as Array<Record<string, unknown>>).map((e) => json(e));+ const remote = artifacts(await call("brain_query", args, REMOTE));+ expect(remote).toHaveLength(1);+ expect(remote[0]).toContain("Notes/shared-artifact");+ const rows = listRecallTelemetry(env.vault, { mode: "query" });+ expect(rows).toHaveLength(1);+ expect(rows[0]!.payload["result_count"]).toBe(1);+ expect(artifacts(await call("brain_query", args, LOCAL))).toHaveLength(3);+ });+});++describe("D2 row 2: most-applied", () => {+ test("remote brain_context, osb://preferences/active and osb://digest/latest count the shared event", async () => {+ confirmedPreference();+ applied(recentStamp(3000), "Notes/one");+ privately(() => {+ applied(recentStamp(2000), "Notes/two");+ applied(recentStamp(1000), "Notes/three");+ });+ expect(json(await call("brain_context", {}, REMOTE))).toContain("(applied_in_window: 1)");+ expect(json(await call("brain_context", {}, LOCAL))).toContain("(applied_in_window: 3)");+ const active = (reach: TransportReach): string =>+ readResource({ vault: env.vault, reach }, "osb://preferences/active").text;+ expect(active(REMOTE)).toContain("(applied_in_window: 1)");+ expect(active(LOCAL)).toContain("(applied_in_window: 3)");+ const digest = (reach: TransportReach): string =>+ readResource({ vault: env.vault, reach }, "osb://digest/latest").text;+ expect(digest(REMOTE)).toContain(`[[${PREF_ID}|Rule ${PREF_SLUG}.]] — —, 1 applied`);+ expect(digest(LOCAL)).toContain(`[[${PREF_ID}|Rule ${PREF_SLUG}.]] — —, 3 applied`);+ });+});++describe("D2 row 3: morning brief and idea discovery", () => {+ test("a private note and a private open question stay off remote answers", async () => {+ privately(() => {+ appendLogEvent(env.vault, {+ timestamp: recentStamp(2000),+ eventType: "note",+ body: { text: PRIVATE_TEXT },+ });+ appendLogEvent(env.vault, {+ timestamp: recentStamp(1000),+ eventType: BRAIN_LOG_EVENT_KIND.reconcile,+ body: { topic: "private-open-question", domain: "coding" },+ } as Parameters<typeof appendLogEvent>[1]);+ });+ const morning = { view: "morning" };+ const remote = json(await call("brain_brief", morning, REMOTE));+ expect(remote).not.toContain(PRIVATE_TEXT);+ expect(remote).not.toContain("private-open-question");+ const local = json(await call("brain_brief", morning, LOCAL));+ expect(local).toContain(PRIVATE_TEXT);+ expect(local).toContain("private-open-question");+ expect(json(await call("brain_idea_discovery", {}, REMOTE))).not.toContain(+ "private-open-question",+ );+ expect(json(await call("brain_idea_discovery", {}, LOCAL))).toContain("private-open-question");+ });+});++describe("D2 row 4: agent provenance", () => {+ test("an agent whose only event is private is absent from remote answers", async () => {+ appendLogEvent(env.vault, {+ timestamp: recentStamp(2000),+ eventType: "note",+ agent: "@open-agent",+ body: { text: SHARED_TEXT },+ });+ privately(() =>+ appendLogEvent(env.vault, {+ timestamp: recentStamp(1000),+ eventType: "note",+ agent: "@secret-agent",+ body: { text: PRIVATE_TEXT },+ }),+ );+ for (const tool of ["brain_agent_query", "brain_agent_diff"]) {+ const remote = json(await call(tool, {}, REMOTE));+ expect(remote).toContain("open-agent");+ expect(remote).not.toContain("secret-agent");+ expect(remote).not.toContain(PRIVATE_TEXT);+ expect(remote).not.toContain("fromPrivatePartition");+ const local = json(await call(tool, {}, LOCAL));+ expect(local).toContain("secret-agent");+ expect(local).not.toContain("fromPrivatePartition");+ }+ const counts = await Promise.all(+ ["brain_agent_query", "brain_agent_diff"].flatMap((tool) =>+ [REMOTE, LOCAL].map(async (reach) => (await call(tool, {}, reach))["total_matched"]),+ ),+ );+ expect(counts).toEqual([1, 2, 1, 2]);+ });+});++const HUB_SLUG = "hub";+const HUB_MEMBERS = ["member-a", "member-b", "member-c", "member-d"];++/** A shared preference whose body is a MOC over the default hub thresholds. */+function hubPreference(): void {+ for (const slug of HUB_MEMBERS) confirmedPreference(slug);+ confirmedPreference(HUB_SLUG);+ const existing = existingPreferencePageText(env.vault, HUB_SLUG, (p) => readFileSync(p, "utf8"));+ if (existing === null) throw new Error("fixture: hub preference missing");+ const [meta] = parseFrontmatterText(existing);+ const links = [PREF_ID, ...HUB_MEMBERS.map((s) => `pref-${s}`)].map((id) => `[[${id}]]`);+ writePreferencePageText(env.vault, HUB_SLUG, formatFrontmatter(meta, `${links.join(" ")}\n`));+}++describe("D2 row 5: backlinks and the link-derived answers", () => {+ beforeEach(() => {+ confirmedPreference();+ hubPreference();+ privately(() => {+ applied(recentStamp(2000), "Notes/private-artifact");+ appendLogEvent(env.vault, {+ timestamp: recentStamp(1000),+ eventType: "note",+ body: { text: "names a target", link: "[[pref-missing-target]]" },+ } as Parameters<typeof appendLogEvent>[1]);+ });+ });++ const logSources = (refs: unknown): string[] =>+ (refs as Array<Record<string, unknown>>)+ .map((r) => String(r["source"]))+ .filter((s) => s.startsWith("log-"));++ test("osb://backlinks and brain_backlinks list no log backlink remotely", async () => {+ const resource = (reach: TransportReach): string =>+ readResource({ vault: env.vault, reach }, `osb://backlinks/${PREF_ID}`).text;+ expect(resource(REMOTE)).not.toContain("## log-");+ expect(resource(REMOTE)).toContain("Total inbound references: 1");+ expect(resource(LOCAL)).toContain("## log-apply-evidence (1)");+ expect(resource(LOCAL)).toContain("Total inbound references: 2");+ const tool = async (reach: TransportReach): Promise<Record<string, unknown>> =>+ call("brain_backlinks", { id: PREF_ID }, reach);+ expect(logSources((await tool(REMOTE))["refs"])).toEqual([]);+ expect((await tool(REMOTE))["count"]).toBe(1);+ expect(logSources((await tool(LOCAL))["refs"])).toHaveLength(1);+ });++ test("the remote digest's inbound count ignores the private reference", () => {+ // A shared event in the window, so the remote digest is not "no changes".+ appendLogEvent(env.vault, {+ timestamp: recentStamp(),+ eventType: "note",+ body: { text: SHARED_TEXT },+ });+ const inbound = (reach: TransportReach): number => {+ const text = readResource({ vault: env.vault, reach }, "osb://digest/latest").text;+ const line = text+ .split("\n")+ .find((l) => l.includes(`[[${PREF_ID}|`) && l.includes("inbound"));+ return line === undefined ? 0 : Number(/(\d+) inbound/.exec(line)![1]);+ };+ expect(inbound(REMOTE)).toBe(1);+ expect(inbound(LOCAL)).toBe(2);+ });++ test("concept synthesis lists no log linker remotely", async () => {+ const linkers = async (reach: TransportReach): Promise<string[]> => {+ const answer = await call(+ "brain_analytics",+ { view: "concept_synthesis", id: PREF_ID },+ reach,+ );+ const cluster = (answer["cluster"] ?? answer) as Record<string, unknown>;+ return (cluster["linkers"] as Array<Record<string, unknown>>)+ .map((l) => String(l["sourceKind"] ?? l["source_kind"]))+ .filter((k) => k.startsWith("log-"));+ };+ expect(await linkers(REMOTE)).toEqual([]);+ expect(await linkers(LOCAL)).toHaveLength(1);+ });++ test("brain_moc_audit counts no private backlink remotely", async () => {+ const count = async (reach: TransportReach): Promise<number> => {+ const answer = await call("brain_moc_audit", { id: `pref-${HUB_SLUG}` }, reach);+ const members = [+ ...((answer["wellCovered"] ?? answer["well_covered"]) as Array<Record<string, unknown>>),+ ...(answer["fragile"] as Array<Record<string, unknown>>),+ ];+ const member = members.find((m) => m["id"] === PREF_ID);+ if (member === undefined) throw new Error(`no member in ${json(answer)}`);+ return Number(member["backlinkCount"] ?? member["backlink_count"]);+ };+ expect(await count(REMOTE)).toBe(0);+ expect(await count(LOCAL)).toBe(1);+ });++ test("brain_doctor and its repair preview ignore a private broken backlink remotely", async () => {+ const warned = async (reach: TransportReach): Promise<boolean> =>+ json((await call("brain_doctor", {}, reach))["warnings"]).includes("pref-missing-target");+ expect(await warned(REMOTE)).toBe(false);+ expect(await warned(LOCAL)).toBe(true);+ const unfixable = async (reach: TransportReach): Promise<number> => {+ const answer = await call("brain_doctor", { repair: true }, reach);+ const repair = answer["repair"] as Record<string, unknown>;+ const rows = repair["unfixable"] as Array<Record<string, unknown>>;+ return Number(rows.find((r) => r["code"] === "broken-backlinks")?.["count"] ?? 0);+ };+ expect(await unfixable(REMOTE)).toBe(0);+ expect(await unfixable(LOCAL)).toBe(1);+ });+});++describe("D2 row 6: status", () => {+ test("remote status shows the shared times and one log day fewer", async () => {+ const dayMs = 24 * 60 * 60 * 1000;+ const at = (daysAgo: number): string =>+ new Date(Date.now() - daysAgo * dayMs).toISOString().slice(0, 10) + "T10:00:00.000Z";+ const sharedDream = at(3);+ const sharedEvidence = at(2);+ const privateEvidence = at(1);+ const privateDream = at(1).replace("T10:", "T11:");+ confirmedPreference();+ appendLogEvent(env.vault, {+ timestamp: sharedDream,+ eventType: BRAIN_LOG_EVENT_KIND.dream,+ body: { run_id: "run-shared" },+ } as Parameters<typeof appendLogEvent>[1]);+ applied(sharedEvidence, "Notes/shared-artifact");+ privately(() => {+ applied(privateEvidence, "Notes/private-artifact");+ appendLogEvent(env.vault, {+ timestamp: privateDream,+ eventType: BRAIN_LOG_EVENT_KIND.dream,+ body: { run_id: "run-private" },+ } as Parameters<typeof appendLogEvent>[1]);+ });+ const tool = async (reach: TransportReach): Promise<Record<string, unknown>> =>+ (await call("second_brain_status", {}, reach))["brain"] as Record<string, unknown>;+ const resource = (reach: TransportReach): string =>+ readResource({ vault: env.vault, reach }, "osb://status").text;+ const remote = await tool(REMOTE);+ expect(remote["last_apply_evidence_at"]).toBe(sharedEvidence);+ expect(remote["last_dream_at"]).toBe(sharedDream);+ expect((remote["counts"] as Record<string, unknown>)["log_days"]).toBe(2);+ expect(resource(REMOTE)).toContain(`- last dream: ${sharedDream}`);+ expect(resource(REMOTE)).toContain(`- last apply-evidence: ${sharedEvidence}`);+ expect(resource(REMOTE)).toContain("- log days: 2");+ const local = await tool(LOCAL);+ expect(local["last_apply_evidence_at"]).toBe(privateEvidence);+ expect(local["last_dream_at"]).toBe(privateDream);+ expect((local["counts"] as Record<string, unknown>)["log_days"]).toBe(3);+ expect(resource(LOCAL)).toContain(`- last dream: ${privateDream}`);+ expect(resource(LOCAL)).toContain(`- last apply-evidence: ${privateEvidence}`);+ expect(resource(LOCAL)).toContain("- log days: 3");+ // The newest shared day holds no dream; the newest day of all holds a private one.+ expect((await call("brain_context", {}, REMOTE))["maintenance_overdue"]).toBe(true);+ expect((await call("brain_context", {}, LOCAL))["maintenance_overdue"]).toBe(false);+ });+});++describe("D2 row 7: the cost meter", () => {+ test("remote brain_recall_telemetry operation cost counts the shared write only", async () => {+ twoNotes();+ const writes = async (reach: TransportReach): Promise<unknown> => {+ const answer = await call("brain_recall_telemetry", { operation: "cost" }, reach);+ const meter = (answer["meter"] ?? answer) as Record<string, unknown>;+ return (meter["writes"] as Record<string, unknown>)["total"];+ };+ expect(await writes(REMOTE)).toBe(1);+ expect(await writes(LOCAL)).toBe(2);+ });+});++/** The brain_doctor suggested actions a caller at `reach` sees. */+async function doctorActions(reach: TransportReach): Promise<ReadonlyArray<{ category: string }>> {+ return (await call("brain_doctor", {}, reach))["suggested_actions"] as ReadonlyArray<{+ category: string;+ }>;+}++/** The revert plan of every write of the agent `claude`, as a caller at `reach` sees it. */+function claudeRevertPlan(reach: TransportReach): Promise<Record<string, unknown>> {+ return call("brain_writes", { action: "plan_revert", agent: "claude" }, reach);+}++/** The first entry of a revert plan. */+function firstEntry(answer: Record<string, unknown>): Record<string, unknown> {+ return (answer["entries"] as ReadonlyArray<Record<string, unknown>>)[0]!;+}++/** Three private notes of another device in one shard; returns its path. */+function privateShard(): string {+ const store = openStore({ root: env.storeRoot, device: "dev-b" });+ let path = "";+ for (const n of [1, 2, 3]) {+ path = appendRecord(store, {+ partition: "private",+ stream: "events",+ kind: "note",+ agent: "t",+ body: { n },+ }).path;+ }+ return path;+}++const storeShardIssues = (answer: Record<string, unknown>): string[] =>+ [+ ...(answer["errors"] as ReadonlyArray<Record<string, unknown>>),+ ...(answer["warnings"] as ReadonlyArray<Record<string, unknown>>),+ ]+ .filter((i) => String(i["code"]).startsWith("store-shard-"))+ .map((i) => json(i));++describe("D2 row 8: the token footprint", () => {+ const FOOTPRINT_TITLE = "Vault exceeds warn threshold";++ function logTokens(withholdsPrivate: boolean): number {+ const report = computeTokenFootprint(env.vault, { withholdsPrivate });+ return report.byCategory.find((c) => c.name === "log")?.tokens ?? 0;+ }++ /** One private note of about 2000 tokens. */+ function privateBulk(): void {+ privately(() =>+ appendLogEvent(env.vault, {+ timestamp: recentStamp(),+ eventType: "note",+ body: { text: "p".repeat(8000) },+ }),+ );+ }++ /**+ * Shared notes, each at most 60000 characters, until the footprint+ * without the private event sits just under the default threshold.+ */+ function fillToThreshold(): void {+ const total = (): number => computeTokenFootprint(env.vault, { withholdsPrivate: true }).total;+ const target = TOKEN_WARN_THRESHOLD_DEFAULT - 500;+ let now = total();+ while (now < TOKEN_WARN_THRESHOLD_DEFAULT - 1000) {+ const chars = Math.max(1, Math.min(60_000, (target - now) * 4 - 400));+ appendLogEvent(env.vault, {+ timestamp: recentStamp(),+ eventType: "note",+ body: { text: "s".repeat(chars) },+ });+ const next = total();+ expect(next).toBeGreaterThan(now);+ now = next;+ }+ expect(now).toBeGreaterThanOrEqual(TOKEN_WARN_THRESHOLD_DEFAULT - 1000);+ expect(now).toBeLessThanOrEqual(TOKEN_WARN_THRESHOLD_DEFAULT);+ }++ const hasFootprint = (actions: ReadonlyArray<{ category: string }>): boolean =>+ actions.some((a) => a.category === "token-footprint");++ test("the log bucket leaves out the private event's tokens only when withholding", () => {+ appendLogEvent(env.vault, {+ timestamp: recentStamp(),+ eventType: "note",+ body: { text: SHARED_TEXT },+ });+ const before = logTokens(false);+ privateBulk();+ expect(logTokens(true)).toBe(before);+ expect(logTokens(false)).toBeGreaterThan(before + 1900);+ });++ test("remote doctor, digest and operator brief carry no footprint action, local ones do", async () => {+ privateBulk();+ fillToThreshold();+ const digest = (reach: TransportReach): string =>+ readResource({ vault: env.vault, reach }, "osb://digest/latest").text;+ expect(digest(REMOTE)).not.toContain(FOOTPRINT_TITLE);+ expect(digest(LOCAL)).toContain(FOOTPRINT_TITLE);+ const brief = async (reach: TransportReach): Promise<string> =>+ json(await call("brain_brief", { view: "digest" }, reach));+ expect(await brief(REMOTE)).not.toContain(FOOTPRINT_TITLE);+ expect(await brief(LOCAL)).toContain(FOOTPRINT_TITLE);+ const topActions = async (+ reach: TransportReach,+ ): Promise<ReadonlyArray<{ category: string }>> =>+ (+ await call(+ "brain_brief",+ { view: "operator", include_dream: false, top_actions: 50 },+ reach,+ )+ )["top_actions"] as ReadonlyArray<{ category: string }>;+ expect(hasFootprint(await topActions(REMOTE))).toBe(false);+ expect(hasFootprint(await topActions(LOCAL))).toBe(true);+ expect(hasFootprint(await doctorActions(REMOTE))).toBe(false);+ expect(hasFootprint(await doctorActions(LOCAL))).toBe(true);+ });+});++describe("D2 row 9: the unfiltered timeline builders", () => {+ beforeEach(() => {+ const dayMs = 24 * 60 * 60 * 1000;+ confirmedPreference();+ // The shared evidence is past the 90-day stale threshold; only the+ // private one keeps the preference fresh.+ applied(new Date(Date.now() - 100 * dayMs).toISOString(), "Notes/shared-artifact");+ privately(() => {+ applied(recentStamp(2000), "Notes/private-artifact");+ appendLogEvent(env.vault, {+ timestamp: recentStamp(1000),+ eventType: "note",+ body: { text: "names a target", link: "[[pref-missing-target]]" },+ } as Parameters<typeof appendLogEvent>[1]);+ });+ });++ // A lineage answer lists transitions, not evidence: the evidence only+ // decides whether the preference has a lineage at all. A preference whose+ // only event is private has none remotely and one locally.+ test("remote brain_idea_lineage leaves out the private apply-evidence", async () => {+ confirmedPreference("lineage-only");+ privately(() => applied(recentStamp(3000), "Notes/private-artifact", "lineage-only"));+ const args = { id: "pref-lineage-only" };+ await expect(call("brain_idea_lineage", args, REMOTE)).rejects.toThrow("no belief lineage");+ expect(json(await call("brain_idea_lineage", args, LOCAL))).toContain("pref-lineage-only");+ });++ test("remote brain_stale_scan counts the preference the private event keeps fresh", async () => {+ const staleIds = async (reach: TransportReach): Promise<string[]> =>+ (+ (await call("brain_stale_scan", {}, reach))["stale_preferences"] as ReadonlyArray<{+ prefId: string;+ }>+ ).map((r) => r.prefId);+ expect(await staleIds(REMOTE)).toContain(PREF_ID);+ expect(await staleIds(LOCAL)).not.toContain(PREF_ID);+ });++ test("remote brain_status counts stale and doctor warnings without the private events", async () => {+ const status = (reach: TransportReach): Promise<Record<string, unknown>> =>+ call("brain_status", {}, reach);+ const warnings = (answer: Record<string, unknown>): number => {+ const row = (answer["problems"] as ReadonlyArray<{ code: string; detail: string }>).find(+ (p) => p.code === "doctor-warnings",+ );+ return row === undefined ? 0 : Number.parseInt(row.detail, 10);+ };+ const remote = await status(REMOTE);+ const local = await status(LOCAL);+ expect(remote["staleTotal"]).toBe((local["staleTotal"] as number) + 1);+ // Two private findings are withheld remotely: the broken backlink of the+ // private note, and the orphan evidence of the private apply-evidence.+ expect(warnings(local)).toBe(warnings(remote) + 2);+ });+});++/** Do what the write seams do: keep the before image, write, record. */+function writeNote(note: string, bytes: string, at: string, agent = "claude"): string {+ const abs = join(env.vault, note);+ let before: string | null = null;+ try {+ before = readFileSync(abs, "utf8");+ } catch {+ before = null;+ }+ mkdirSync(dirname(abs), { recursive: true });+ if (before !== null) storeBeforeImage(env.vault, before);+ writeFileSync(abs, bytes);+ const receipt = recordNoteWrite(env.vault, {+ op: before === null ? "create" : "update",+ target: note,+ before: before === null ? null : { bytes: before },+ after: { bytes },+ timestamp: at,+ agent,+ });+ return (receipt as { write_id: string }).write_id;+}++/** {@link recentStamp} to the second, the precision of a write record. */+const writeStamp = (offsetMs: number): string => recentStamp(offsetMs).replace(/\.\d+Z$/, "Z");++describe("D2 row 10: note writes", () => {+ const NOTE = "Notes/surfaced-note.md";++ let sharedWrite: string;+ let privateWrite: string;+ beforeEach(() => {+ sharedWrite = writeNote(NOTE, "first body\n", writeStamp(3000));+ privateWrite = privately(() => writeNote(NOTE, "second body\n", writeStamp(1000)));+ });++ test("remote brain_writes and brain_write_destinations leave out the private write", async () => {+ const writes = async (reach: TransportReach): Promise<Record<string, unknown>> =>+ call("brain_writes", { path: NOTE }, reach);+ const ids = (answer: Record<string, unknown>): string[] =>+ (answer["writes"] as ReadonlyArray<{ write_id: string }>).map((w) => w.write_id);+ const remote = await writes(REMOTE);+ expect(remote["total_matched"]).toBe(1);+ expect(ids(remote)).toEqual([sharedWrite]);+ const local = await writes(LOCAL);+ expect(local["total_matched"]).toBe(2);+ expect(ids(local)).toContain(privateWrite);+ const row = async (reach: TransportReach): Promise<Record<string, unknown>> => {+ const answer = await call("brain_write_destinations", { path: NOTE }, reach);+ return (answer["destinations"] as ReadonlyArray<Record<string, unknown>>)[0]!;+ };+ expect((await row(REMOTE))["note_writes"]).toBe(1);+ expect((await row(LOCAL))["note_writes"]).toBe(2);+ });++ test("the remote revert plan lists no write of the private operation", async () => {+ const plan = (reach: TransportReach): Promise<Record<string, unknown>> =>+ call("brain_writes", { action: "plan_revert", path: NOTE }, reach);+ const planned = (answer: Record<string, unknown>): string[] =>+ (answer["entries"] as ReadonlyArray<{ writes: ReadonlyArray<string> }>).flatMap(+ (e) => e.writes,+ );+ const remote = await plan(REMOTE);+ const local = await plan(LOCAL);+ expect(planned(remote)).not.toContain(privateWrite);+ expect(planned(local)).toContain(privateWrite);+ expect(remote["digest"]).not.toBe(local["digest"]);+ });+});++describe("D2 row 10: the revert plan's safety rules see every write", () => {+ const NOTE = "Notes/interleaved-note.md";+ const PRIVATE_BODY = "a private body between two shared ones\n";++ test("a private write between two selected shared writes refuses the remote plan", async () => {+ writeNote(NOTE, "origin\n", writeStamp(4000), "origin-agent");+ const first = writeNote(NOTE, "first\n", writeStamp(3000));+ const hidden = privately(() => writeNote(NOTE, PRIVATE_BODY, writeStamp(2000)));+ const last = writeNote(NOTE, "last\n", writeStamp(1000));+ // Locally all three writes are selected and nothing interleaves.+ const local = firstEntry(await claudeRevertPlan(LOCAL));+ expect(local["action"]).toBe("restore");+ expect(local["writes"]).toEqual([first, hidden, last]);+ // Remotely only the shared two are selected, and the private one+ // between them still refuses the plan; the refusal names neither it+ // nor its text.+ const remoteAnswer = await claudeRevertPlan(REMOTE);+ const remote = firstEntry(remoteAnswer);+ expect(remote["action"]).toBe("refuse");+ expect(remote["reason"]).toBe("interleaved");+ expect(remote["writes"]).toEqual([first, last]);+ expect(json(remoteAnswer)).not.toContain(hidden);+ expect(json(remoteAnswer)).not.toContain(PRIVATE_BODY.trim());+ });+});++describe("doctor: a failed private write", () => {+ test("remote brain_doctor shows no trace of it, local names it", async () => {+ // The path as either platform spells it, also escaped inside the JSON answer.+ const PRIVATE_LEDGERS = /private[\\/]+ledgers/;+ // A file where the private ledgers folder belongs: the append fails+ // with an error text that names the private path.+ writeFileSync(join(env.storeRoot, "private", "ledgers"), "not a folder\n");+ const quiet = spyOn(process.stderr, "write").mockImplementation(() => true);+ try {+ const routed = routeWrite({+ legacy: () => null,+ store: () => ledgerRowRecord("metrics", { n: 1 }, "private"),+ context: { vault: env.vault, configPath: env.configPath },+ failOpen: true,+ });+ expect(routed.mode === "store" && routed.failed !== undefined).toBe(true);+ } finally {+ quiet.mockRestore();+ }+ const remote = json(await call("brain_doctor", {}, REMOTE));+ expect(remote).not.toContain("store-write-fail-open");+ expect(remote).not.toMatch(PRIVATE_LEDGERS);+ const local = json(await call("brain_doctor", {}, LOCAL));+ expect(local).toContain("store-write-fail-open");+ // The answer masks the store root, so the path shows from the partition.+ expect(local).toMatch(PRIVATE_LEDGERS);+ });+});++describe("doctor: the evidence lints quote no private event", () => {+ const SHARED_ARTIFACT = "shared-orphan-artifact";+ const PRIVATE_ARTIFACT = "private-orphan-artifact";++ test("remote brain_doctor names neither the private artifact nor its time; local names both", async () => {+ confirmedPreference();+ const sharedAt = recentStamp(2000);+ const privateAt = recentStamp(1000);+ // A range that does not parse and a basename no file has: the private+ // event would raise both `malformed-evidence-range` and `orphan-evidence`.+ applied(sharedAt, `${SHARED_ARTIFACT}:9-3`);+ privately(() => applied(privateAt, `${PRIVATE_ARTIFACT}:9-3`));+ const lintRows = (answer: Record<string, unknown>): string[] =>+ (answer["warnings"] as ReadonlyArray<unknown>)+ .map((w) => json(w))+ .filter((w) => w.includes("orphan-evidence") || w.includes("malformed-evidence-range"));+ const remote = lintRows(await call("brain_doctor", {}, REMOTE));+ // The lints still run remotely: the shared event raises both findings.+ expect(remote.filter((w) => w.includes(SHARED_ARTIFACT))).toHaveLength(2);+ expect(remote.filter((w) => w.includes(PRIVATE_ARTIFACT) || w.includes(privateAt))).toEqual([]);+ const local = lintRows(await call("brain_doctor", {}, LOCAL));+ expect(local.filter((w) => w.includes(PRIVATE_ARTIFACT) && w.includes(privateAt))).toHaveLength(+ 2,+ );+ expect(local.filter((w) => w.includes(SHARED_ARTIFACT))).toHaveLength(2);+ });+});++describe("doctor: a gap in a private shard", () => {+ const HOUR = 60 * 60 * 1000;++ /** A corrupt middle line, and a tail that has not verified for two hours. */+ function breakShard(path: string): void {+ const lines = readFileSync(path, "utf8").split("\n");+ lines[1] = lines[1]!.replace('"n":2', '"n":20');+ writeFileSync(path, lines.join("\n"));+ appendFileSync(path, '{"partial');+ const old = new Date(Date.now() - 2 * HOUR);+ utimesSync(path, old, old);+ }++ /** The shard's partition-relative folder, as every finding about it names it. */+ const PRIVATE_SHARD_DIR = "private/events/dev-b";++ test("remotely no finding names the private shard and ok and the counts are unchanged; locally it is named", async () => {+ const path = privateShard();+ const before = await call("brain_doctor", {}, REMOTE);+ breakShard(path);+ const remote = await call("brain_doctor", {}, REMOTE);+ expect(storeShardIssues(remote).filter((i) => i.includes("private"))).toEqual([]);+ expect(remote["ok"]).toBe(before["ok"]);+ expect(remote["trust_verdict"]).toEqual(before["trust_verdict"]);+ const counts = (answer: Record<string, unknown>) => [+ (answer["errors"] as ReadonlyArray<unknown>).length,+ (answer["warnings"] as ReadonlyArray<unknown>).length,+ ];+ expect(counts(remote)).toEqual(counts(before));+ const local = storeShardIssues(await call("brain_doctor", {}, LOCAL));+ expect(+ local.some((i) => i.includes("store-shard-corrupt-line") && i.includes("private/")),+ ).toBe(true);+ expect(local.some((i) => i.includes("store-shard-tail-held") && i.includes("private"))).toBe(+ true,+ );+ });++ test("remote brain_brief view=operator keeps its counts and trust verdict; local counts the gap", async () => {+ const path = privateShard();+ const brief = (reach: TransportReach) => call("brain_brief", { view: "operator" }, reach);+ const before = await brief(REMOTE);+ const localBefore = await brief(LOCAL);+ breakShard(path);+ const remote = await brief(REMOTE);+ expect(remote["doctor_summary"]).toEqual(before["doctor_summary"]);+ expect(remote["trust_verdict"]).toEqual(before["trust_verdict"]);+ expect(json(remote)).not.toContain(PRIVATE_SHARD_DIR);+ const local = await brief(LOCAL);+ const errorsOf = (answer: Record<string, unknown>) =>+ (answer["doctor_summary"] as { error_count: number }).error_count;+ expect(errorsOf(local)).toBeGreaterThan(errorsOf(localBefore));+ });++ test("remote brain_status keeps its doctor counts; local counts the gap", async () => {+ const path = privateShard();+ const doctorRows = async (reach: TransportReach) => {+ const answer = await call("brain_status", {}, reach);+ return {+ answer,+ rows: (answer["problems"] as ReadonlyArray<{ code: string; detail: string }>).filter((p) =>+ p.code.startsWith("doctor-"),+ ),+ };+ };+ const before = await doctorRows(REMOTE);+ const localBefore = await doctorRows(LOCAL);+ breakShard(path);+ const remote = await doctorRows(REMOTE);+ expect(remote.rows).toEqual(before.rows);+ expect(json(remote.answer)).not.toContain(PRIVATE_SHARD_DIR);+ const local = await doctorRows(LOCAL);+ expect(local.rows).not.toEqual(localBefore.rows);+ expect(local.rows.some((r) => r.code === "doctor-errors")).toBe(true);+ });+});++// ----- Timeline-derived rows: rows that are not events ----------------------+//+// These readers turn the timeline index into rows of their own (evidence+// with running counts, status transitions, retirements), which never pass+// `eventAtReach`. Every row names only the shared preference, so the+// reference view keeps it: the private event must not reach the index.++/** A private dream that confirmed the shared preference. */+function privateDreamConfirming(at: string): void {+ privately(() =>+ appendLogEvent(env.vault, {+ timestamp: at,+ eventType: BRAIN_LOG_EVENT_KIND.dream,+ body: { run_id: "run-private", confirmed: [`[[${PREF_ID}]]`] },+ } as Parameters<typeof appendLogEvent>[1]),+ );+}++describe("timeline-derived rows withhold a private event naming a shared preference", () => {+ test("brain_analytics view=belief_evolution", async () => {+ confirmedPreference();+ applied(recentStamp(3000), "Notes/shared-artifact");+ const privateAt = recentStamp(2000);+ privately(() => applied(privateAt, "Notes/shared-artifact"));+ privateDreamConfirming(recentStamp(1000));+ const args = { view: "belief_evolution", pref_id: PREF_ID };+ const rows = (answer: Record<string, unknown>, key: string): ReadonlyArray<unknown> =>+ answer[key] as ReadonlyArray<unknown>;+ const remote = await call("brain_analytics", args, REMOTE);+ expect(rows(remote, "evidence")).toHaveLength(1);+ expect(rows(remote, "transitions")).toHaveLength(0);+ expect(json(remote)).not.toContain(privateAt);+ const local = await call("brain_analytics", args, LOCAL);+ expect(rows(local, "evidence")).toHaveLength(2);+ expect(rows(local, "transitions")).toHaveLength(1);+ expect(json(local)).toContain(privateAt);+ });++ test("brain_brief view=daily and view=weekly status transitions", async () => {+ confirmedPreference();+ const at = recentStamp();+ privateDreamConfirming(at);+ const weekEnd = new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString().slice(0, 10);+ const transitions = async (+ args: Record<string, unknown>,+ reach: TransportReach,+ ): Promise<ReadonlyArray<unknown>> =>+ (await call("brain_brief", args, reach))["status_transitions"] as ReadonlyArray<unknown>;+ for (const args of [+ { view: "daily", date: at.slice(0, 10) },+ { view: "weekly", week_end: weekEnd },+ ]) {+ expect(await transitions(args, REMOTE)).toHaveLength(0);+ expect(await transitions(args, LOCAL)).toHaveLength(1);+ }+ });+});