Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
fix(snapshot): make snapshot restore all-or-nothing (#302)
Serg Eroshe4a2b6514567d
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex de6ef8d1..3d3b77dd 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **A failed snapshot restore no longer leaves `Brain/` replaced beside the old derived store.** `restoreSnapshot` replaced the live `Brain/` tree before it decompressed the archived derived store, so a `zstd` failure or timeout, a corrupt payload or a full disk at that point stopped the rollback with the tree already restored and the store not. Every step that can fail now runs before anything live changes: the archive is extracted and the `Brain/` entries staged and verified on the live tree's filesystem (under `.snapshots/` when it shares a device with `Brain/`), the store archive is checked against the digest its manifest recorded, decompressed beside the live store and integrity-checked, and only then is the store's writer lock acquired and the recovery point taken, so lock contention also refuses with nothing touched. The lock is held from before the recovery point through the swap, and the recovery point's store archive is written under it, so no write can land between the two and the recovery point holds every commit the swap replaces. The recovery point skips its own lock only for an opaque token minted where the restore acquires the lock, while that lock is held and on the same store; a string naming the store's path, an object that merely looks like a token or a token whose lock was already released takes the lock as usual. The recovery point archives exactly the store the swap replaces, under the restore's coverage policy, also with a `derivedStorePath` override, and the restore refuses before the swap when the recovery point's manifest records a different store. A lock release that fails after the swap committed is reported as a committed restore that names the recovery point, and one that fails after an earlier error is added to that error instead of replacing it. The swap itself is a run of renames: live entries are moved aside, staged ones moved in, then the staged store renamed over the live one, and a rename that fails is undone in reverse. If the undo fails too, the error says how many entries were swapped, where the pre-restore entries were kept and which recovery point puts the tree back. Excluded entries, the preserved `.snapshots/`, manifest-driven store inclusion and the reported restore states are unchanged. A rollback journal (`-journal`, the DELETE journal mode the store uses on remote filesystems) that the old store left beside it, hot after a crash, moves with the outgoing store to `.bak-journal` and back on every undo, for a store inside `Brain/` or outside it, so SQLite never replays it over the replacement; a journal that cannot be moved undoes the whole restore. The old store's `-wal`/`-shm` are removed after the store swap with the Windows retry, and a removal that still fails undoes the whole restore instead of reporting success beside an orphan write-ahead log; when no store was live before the restore, the error says so instead of claiming a pre-restore store was put back. A restore staging directory left directly under `Brain/` (a restore killed mid-swap, when `.snapshots/` is on another device) is skipped by the archiver, the manifest walker and the restore, so it is never archived or swapped back as Brain content. A sound store the restore replaces is always archived in its recovery point, whatever the coverage policy or size cap, and the restore refuses when the recovery point records it as left out; an absent store or one that fails its integrity scan does not block the restore that replaces it, also when the vault's policy turns store coverage on: the recovery point then leaves that store out explicitly instead of refusing it, so the restore recreates or heals it. A derived store that lives directly under `Brain/` (a `search_db_path` or `derivedStorePath` such as `Brain/index.sqlite`) is kept out of the `Brain/` tree swap together with its journals, writer lock, `.bak` and the restore's staged file, so the store swap finds its staged file and the restore completes; the placement is judged for the directory entry the store swap renames, not the file a symlink there points to, so a `Brain/index.sqlite` that links to a database outside `Brain/` is restored at the link instead of having its staged replacement moved away by the tree swap; the raw copy of the store that the `Brain/` archive carries as a top-level entry is not restored over it, so the store is written once, from the store archive, or left untouched when the snapshot did not cover it. A store nested deeper inside a `Brain/` directory the swap replaces as a whole is refused before anything is extracted or touched, with an error that names the directory.+- **The concurrent appenders test no longer fails on a slow runner with `store lock busy`.** Six workers appending back to back can outlast the store lock's wait budget on a loaded Windows runner. That refusal is documented as retryable and arrives before the record is written, so each worker now retries a refused append with a short randomized backoff within a bound; the assertions on record count, unique ids and clocks, the single blob and interleaving are unchanged.- **A snapshot or restore can no longer hang forever on a stuck `tar` or `zstd`.** The snapshot module is synchronous, so a child that never exited froze the hook, CLI verb or MCP call that took the recovery point, with nothing able to interrupt it. Every `tar` and `zstd` run of a snapshot, a derived-store archive and a restore now has a timeout (fifteen minutes by default, `OPEN_SECOND_BRAIN_SNAPSHOT_TIMEOUT_MS` to change it); a run that hits it is killed and the snapshot fails with an error that names the tool and the bound, leaving no partial archive.- **The Hermes bridge keeps the host's parent-death protection on Hermes 0.23+.** Hermes removed `tools.mcp_stdio_watchdog` (hermes-agent `2d783a15`, "one parent-death supervisor per process, not one per stdio server") in favour of `hermes_cli.process_identity.spawn_owned`, so the bridge's `from tools import mcp_stdio_watchdog` stopped resolving: every provider load logged "parent-death watchdog module unavailable" and the `o2b mcp` child sat outside the host's reaping - the one case the guarantee exists for, a host killed by SIGKILL, OOM or a crash. The spawn now prefers `spawn_owned`, recording the child in the host's spawn ledger under `open-second-brain-mcp-bridge` and handing its process group to the per-process supervisor, and returns that group through `release_process_group_once_empty` once the bridge stops the child itself, so a stopped child's pgid cannot be reaped as a stranger's after the kernel recycles it. A host that only has the legacy wrapper still gets it unchanged, a host with neither (a standalone checkout, or any non-Hermes harness) still falls back to a plain spawn with the existing once-per-process warning, and an import that raises anything rather than `ImportError` degrades the same way instead of failing the bridge. Claude Code, Codex and the other harnesses never had a watchdog seam and are untouched; Windows keeps its plain `Popen`, since the layer-3 job object already owns the child there.- **Token callers are attributed to their token on `brain_note` and `brain_expire` even without an `agent` argument.** Both handlers refused a cross-agent claim but then let an omitted agent resolve through the process config, so a token-minted caller's log line and expiration audit stamp could name the config identity instead of the credential's agent; both now fall back to the request identity after the claim check only when the caller arrived by token, and a shared-key caller keeps the config-path resolution it always had.diff --git a/src/core/brain/manifest.ts b/src/core/brain/manifest.tsindex f71685b6..53c1dfb3 100644--- a/src/core/brain/manifest.ts+++ b/src/core/brain/manifest.ts@@ -17,6 +17,7 @@ import { atomicWriteFileSync } from "../fs-atomic.ts";import { sha256Hex } from "../integrity/digest.ts";import { pathCovers } from "../vault-scope/defaults.ts";import {+ BRAIN_RESTORE_STAGING_PREFIX,BRAIN_ROOT_REL,BRAIN_SNAPSHOT_EXCLUDED_ENTRIES,snapshotsDir,@@ -261,6 +262,9 @@ export function buildManifest(brainRoot: string, opts: BuildManifestOptions = {}if (BRAIN_SNAPSHOT_EXCLUDED_ENTRIES.some((entry) => pathCovers(entry, rel))) {continue;}+ // A restore's leftover staging directory, which the archive never+ // holds either (see BRAIN_RESTORE_STAGING_PREFIX).+ if (dir === brainRoot && name.startsWith(BRAIN_RESTORE_STAGING_PREFIX)) continue;// Defense-in-depth: a `..` path *segment* cannot legitimately// appear inside a sane Brain tree. `pathCovers` anchors on the// segment boundary, so an otherwise-valid filename likediff --git a/src/core/brain/path-constants.ts b/src/core/brain/path-constants.tsindex 437d495b..ee32b9a3 100644--- a/src/core/brain/path-constants.ts+++ b/src/core/brain/path-constants.ts@@ -199,6 +199,16 @@ export const BRAIN_SNAPSHOT_EXCLUDED_ENTRIES: ReadonlyArray<string> = Object.freBRAIN_ARTIFACTS_DIR,]);+/**+ * Name prefix of a restore's staging directory. It normally lives under+ * `.snapshots/`, but sits directly under `Brain/` when `.snapshots/` is on+ * another device, and a restore killed mid-swap leaves it there. Such a+ * top-level entry is restore plumbing, never Brain content: the archiver,+ * the manifest walker and the restore all skip it, like+ * {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES}.+ */+export const BRAIN_RESTORE_STAGING_PREFIX = ".restore-staging-";+/*** The DERIVED store: `<vault>/.open-second-brain/brain.sqlite`.*diff --git a/src/core/brain/snapshot-gate.ts b/src/core/brain/snapshot-gate.tsindex ba4973cf..2a12126d 100644--- a/src/core/brain/snapshot-gate.ts+++ b/src/core/brain/snapshot-gate.ts@@ -41,7 +41,7 @@import { existsSync } from "node:fs";-import { loadSnapshotRetentionSafe } from "./policy.ts";+import { loadSnapshotRetentionSafe, type BrainDerivedStorePolicy } from "./policy.ts";import { isFileAlreadyExistsAt } from "../fs-atomic.ts";import {classifyRecoverability,@@ -54,11 +54,13 @@ import {BrainSnapshotError,createSnapshot,pruneSnapshots,+ liveStoreWorthPreserving,restoreSnapshot,type PruneSnapshotsResult,type RecoveryPointEvidence,type RestoreSnapshotOptions,type RestoreSnapshotResult,+ type StoreLockToken,} from "./snapshot.ts";import type { WriteLane } from "./freeze-marker.ts";import { compactRunStamp } from "./time.ts";@@ -136,6 +138,12 @@ export interface WithDestructiveSnapshotOptions {* name.*/readonly available?: (runId: string) => boolean;+ /** Forwarded to {@link CreateSnapshotOptions.heldStoreLock}. */+ readonly heldStoreLock?: StoreLockToken;+ /** Forwarded to {@link SnapshotStoreOptions.derivedStorePath}. */+ readonly derivedStorePath?: string;+ /** Forwarded to {@link SnapshotStoreOptions.derivedStore}. */+ readonly derivedStore?: BrainDerivedStorePolicy;/*** Archives the retention pass behind this snapshot must not remove, on* top of the snapshot itself (which is always protected).@@ -300,7 +308,14 @@ export function takeSnapshot(vault,baseRunId,(runId) =>- createSnapshot(vault, runId, { reason, now, ...(opts.lane ? { lane: opts.lane } : {}) }).path,+ createSnapshot(vault, runId, {+ reason,+ now,+ ...(opts.lane ? { lane: opts.lane } : {}),+ ...(opts.heldStoreLock !== undefined ? { heldStoreLock: opts.heldStoreLock } : {}),+ ...(opts.derivedStorePath !== undefined ? { derivedStorePath: opts.derivedStorePath } : {}),+ ...(opts.derivedStore !== undefined ? { derivedStore: opts.derivedStore } : {}),+ }).path,MAX_SNAPSHOT_ID_ATTEMPTS,opts.available,);@@ -459,9 +474,33 @@ export function restoreSnapshotWithRecoveryPoint(let recoveryPoint: DestructiveSnapshot | null = null;const result = restoreSnapshot(vault, runId, {...opts,- beforeDiscard: (): RecoveryPointEvidence => {+ beforeDiscard: ({ storeTarget, heldStoreLock }): RecoveryPointEvidence => {const point = takeSnapshot(vault, BRAIN_SNAPSHOT_REASON.manual, {...opts,+ // The recovery point archives exactly the store the swap replaces,+ // under the restore's coverage policy (spread from `opts` above):+ // left to resolve its own source, it would archive the configured+ // store while a `derivedStorePath` override names another.+ ...(storeTarget !== null ? { derivedStorePath: storeTarget } : {}),+ // A sound store the swap replaces is always archived, whatever the+ // vault's coverage policy or size cap: its only copy is about to be+ // discarded. An absent store has nothing to keep, and a condemned+ // one is what the restore heals, so for either the recovery point+ // explicitly leaves the store out: left to the vault's policy, a+ // policy that asks for coverage would refuse the absent store and+ // fail the integrity scan on the condemned one, and the restore+ // could never recreate or heal it. The restore refuses when a+ // sound replaced store was left out.+ ...(storeTarget !== null+ ? {+ derivedStore: liveStoreWorthPreserving(storeTarget)+ ? { include: true, maxBytes: Number.MAX_SAFE_INTEGER }+ : { include: false, maxBytes: 0 },+ }+ : {}),+ // The restore holds the store writer lock from before this point+ // through the swap; the snapshot writes its store archive under it.+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),protectRunIds: [runId, ...(opts.protectRunIds ?? [])],});recoveryPoint = point;diff --git a/src/core/brain/snapshot.ts b/src/core/brain/snapshot.tsindex ca642928..094ae001 100644--- a/src/core/brain/snapshot.ts+++ b/src/core/brain/snapshot.ts@@ -109,14 +109,14 @@ import {type Dirent,} from "node:fs";import { tmpdir } from "node:os";-import { basename, dirname, join, relative, resolve, sep } from "node:path";+import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";import { FileAlreadyExistsError, renameWithRetry, unlinkWithRetry } from "../fs-atomic.ts";import { classifyRecoverability, type RecoverabilityVerdict } from "./gates/recoverability.ts";import { sha256Hex } from "../integrity/digest.ts";import { resolveConfiguredIndexPath } from "../search/paths.ts";import { runIntegrityCheck } from "../search/store/lifecycle.ts";-import { acquireWriterLockSync } from "../search/store/writer-lock.ts";+import { acquireWriterLockSync, WRITER_LOCK_STALE_MS } from "../search/store/writer-lock.ts";import { appendLogEvent } from "./log.ts";import {buildManifest,@@ -128,6 +128,7 @@ import {type SnapshotStoreExclusionReason,} from "./manifest.ts";import {+ BRAIN_RESTORE_STAGING_PREFIX,BRAIN_ROOT_REL,BRAIN_SNAPSHOT_EXCLUDED_ENTRIES,BRAIN_SNAPSHOTS_REL,@@ -287,6 +288,59 @@ export interface CreateSnapshotOptions extends SnapshotStoreOptions {* by recording that it took a recovery point.*/readonly now?: Date;+ /**+ * The token of a derived-store writer lock the CALLER already holds.+ * When it is a live token on exactly the store this snapshot archives,+ * the archive is written under that lock instead of acquiring it again+ * (the lock is not re-entrant, so a second acquire would refuse).+ * Anything else - a released token, a token on another path, a string+ * naming the path, an object that merely looks like a token - takes+ * the lock as usual. A token is minted only where the restore acquires+ * the lock, so naming the store is not enough to skip it.+ *+ * One caller needs it: the gated restore takes the lock before its+ * recovery point and keeps it through the swap, so no write lands+ * between the two.+ */+ readonly heldStoreLock?: StoreLockToken;+}++/** Mints {@link StoreLockToken}s; never leaves this module. */+const STORE_LOCK_TOKEN_MINT: unique symbol = Symbol("store-lock-token-mint");++/**+ * The tokens whose lock is still held. A token leaves the set when its+ * lock is released, so a kept reference stops counting at that moment.+ */+// Live token -> the store path its lock is held on. The path is read from+// here, never from the token, so a caller cannot redirect a genuine token.+const liveStoreLockTokens = new WeakMap<StoreLockToken, string>();++/**+ * Proof that this module holds the derived store's writer lock on+ * {@link StoreLockToken.path}. Constructed only with the module-private+ * mint symbol, by the restore's lock acquisition, and honoured only+ * while the lock is held (see {@link isLiveStoreLockToken}).+ */+export class StoreLockToken {+ readonly #path: string;++ constructor(mint: symbol, path: string) {+ if (mint !== STORE_LOCK_TOKEN_MINT) {+ throw new TypeError("a store lock token is minted only by the restore's lock acquisition");+ }+ this.#path = path;+ }++ /** The live store path the lock is held on. */+ get path(): string {+ return this.#path;+ }+}++/** True only for a minted, unreleased token on exactly `path`. */+function isLiveStoreLockToken(token: unknown, path: string): boolean {+ return token instanceof StoreLockToken && liveStoreLockTokens.get(token) === path;}export interface SnapshotInfo {@@ -568,6 +622,30 @@ export interface RecoveryPointEvidence {readonly path: string;}+/**+ * What {@link restoreSnapshot} tells its+ * {@link RestoreSnapshotOptions.beforeDiscard} callback.+ */+export interface BeforeDiscardContext {+ /**+ * The live derived store the restore will replace, or null when it+ * does not replace the store. A recovery point taken here archives+ * exactly this path (`derivedStorePath`), and the restore refuses when+ * the recovery point's manifest records another source.+ */+ readonly storeTarget: string | null;+ /**+ * The token of the writer lock the restore holds on+ * {@link storeTarget}, or null when it holds none. The lock is taken+ * before the callback and kept through the swap, so a recovery point+ * taken here sees every commit the swap replaces. A snapshot taken in+ * the callback passes it as `heldStoreLock`, or its own acquire of the+ * same lock refuses. The token stops counting once the restore+ * releases the lock.+ */+ readonly heldStoreLock: StoreLockToken | null;+}+export interface RestoreSnapshotOptions extends SnapshotStoreOptions {/*** The write lane the restore is guarded on (default: content), as the@@ -598,7 +676,7 @@ export interface RestoreSnapshotOptions extends SnapshotStoreOptions {* minted by `snapshot-gate.ts`, which composes THIS module - taking* `takeSnapshot` from there would make the pair a cycle.*/- readonly beforeDiscard?: () => RecoveryPointEvidence | void;+ readonly beforeDiscard?: (ctx: BeforeDiscardContext) => RecoveryPointEvidence | void;}// ----- Tooling detection ---------------------------------------------------@@ -970,9 +1048,16 @@ export function bufferEntryName(e: Uint8Array | Buffer): string {return Buffer.from(e).toString("utf8");}-/** True for a top-level `Brain/` entry the snapshot family never touches. */+/**+ * True for a top-level `Brain/` entry the snapshot family never touches:+ * the excluded entries, and a restore's staging directory left directly+ * under `Brain/` (see {@link BRAIN_RESTORE_STAGING_PREFIX}), which must+ * never be archived, nor moved aside and put back as Brain content.+ */export function isSnapshotExcludedEntry(name: string): boolean {- return BRAIN_SNAPSHOT_EXCLUDED_ENTRIES.includes(name);+ return (+ BRAIN_SNAPSHOT_EXCLUDED_ENTRIES.includes(name) || name.startsWith(BRAIN_RESTORE_STAGING_PREFIX)+ );}// ----- Derived-store coverage ----------------------------------------------@@ -1023,7 +1108,7 @@ function coverDerivedStore(liveVault: string,runId: string,tools: ToolAvailability,- opts: SnapshotStoreOptions,+ opts: SnapshotStoreOptions & { readonly heldStoreLock?: StoreLockToken },): BrainManifestDerivedStore {// The policy and the live store are the caller's vault's (the search// index is keyed by the real vault), never the mapped state root's; the@@ -1065,8 +1150,13 @@ function coverDerivedStore(// The writer lock on the LIVE store path, held across the integrity// scan and the copy, so an index run cannot mutate the file underneath// the snapshot. It is the same lock every other writer serialises on,- // taken synchronously because this whole module is synchronous.- const release = acquireWriterLockSync(sourcePath);+ // taken synchronously because this whole module is synchronous. A+ // caller that holds it on this exact path (the gated restore) hands+ // down the live token the acquisition minted, and the archive is+ // written under the caller's lock.+ const release = isLiveStoreLockToken(opts.heldStoreLock, sourcePath)+ ? (): void => {}+ : acquireWriterLockSync(sourcePath);try {assertStoreIsSound(sourcePath, runId);@@ -1106,6 +1196,23 @@ function coverDerivedStore(});}+/**+ * Whether the live store at `path` holds data a recovery point must keep:+ * the file exists and passes the same integrity scan a snapshot runs. An+ * absent store has nothing to lose, and a condemned one is what a restore+ * replaces to heal it, so neither forces store coverage on a recovery+ * point.+ */+export function liveStoreWorthPreserving(path: string): boolean {+ if (!existsSync(path)) return false;+ try {+ assertStoreIsSound(path, "recovery-point-probe");+ return true;+ } catch {+ return false;+ }+}+/*** Refuse a condemned store rather than archiving it over a good* snapshot.@@ -2142,27 +2249,48 @@ function refuseEscapingSymlinks(root: string, runId: string): void {* `.snapshots/` directory is preserved verbatim so older rollbacks still* have a path back.*- * Steps:- *- * 1. Locate the archive.- * 2. Extract into a sibling temp dir.- * 3. Verify the extracted tree contains a `Brain/` root.- * 4. Run {@link RestoreSnapshotOptions.beforeDiscard}, the seam where- * a recovery point of the tree about to be discarded is taken.- * 5. For each top-level entry under the extracted `Brain/` (which- * excludes {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES} by virtue of how- * the archive was written), remove the corresponding live entry and- * copy the extracted one into place.- * 6. Swap the derived store, when and only when the manifest says the- * snapshot included one.- * 7. Clean up the temp dir.- *- * Step 5 is a data-loss operation with the strongest confirmation ladder- * in the codebase in front of it and, until step 4 existed, nothing at+ * Every step that can fail for a reason other than the swap itself runs+ * BEFORE anything live changes, so a failed restore leaves `Brain/` and+ * the derived store exactly as they were:+ *+ * 1. Pre-flight the derived-store records: refuse an unreadable record+ * or a recorded archive that is not on disk, and refuse a live store+ * nested inside a top-level `Brain/` directory the swap replaces.+ * 2. Extract the archive into a private temp dir and verify it has a+ * `Brain/` root with no symlink escaping it.+ * 3. Stage the extracted top-level entries (minus+ * {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES}, and minus the store's file+ * family when the store sits directly under `Brain/`) on the live+ * tree's own filesystem, and verify the staged copy holds every file.+ * 4. When the manifest says the snapshot included the derived store:+ * check the archive against the recorded digest, decompress it+ * beside the live store, and run the integrity check on the result.+ * 5. Take the derived store's writer lock (when it is being replaced),+ * so lock contention also refuses before anything is touched. It is+ * held from here through the swap.+ * 6. Run {@link RestoreSnapshotOptions.beforeDiscard}, the seam where a+ * recovery point of the state about to be discarded is taken, then+ * re-check that the derived-store records did not change under it.+ * The lock taken in step 5 means no write lands between the recovery+ * point and the swap, so the recovery point holds every commit the+ * swap replaces.+ * 7. Swap: move each live top-level entry aside and rename the staged+ * one into place, then rename the staged store over the live one.+ * Renames only, on one filesystem, so the window is short. If any+ * rename fails, the renames already done are undone in reverse.+ * 8. Remove the outgoing entries and every staging directory.+ *+ * Step 7 is a data-loss operation with the strongest confirmation ladder+ * in the codebase in front of it and, until step 5 existed, nothing at* all behind it: an operator who rolled back to the wrong run id had no* way back. Callers that want one use `restoreSnapshotWithRecoveryPoint`* in `snapshot-gate.ts`; callers that do not get a `recoverability`- * verdict saying so rather than silence.+ * verdict saying so rather than silence. When the swap fails AND its undo+ * fails too, the error names what was swapped, where the outgoing entries+ * were left, and the recovery point when one was taken. When the lock+ * cannot be released after the swap committed, the error says the restore+ * committed and names the recovery point; a release failure never+ * replaces an earlier error, it is added to it.*/export function restoreSnapshot(vault: string,@@ -2187,7 +2315,29 @@ export function restoreSnapshot(// The records the pre-flight just validated, as one comparable value.const storeStateBefore = derivedStoreRestoreState(vault, runId);+ // The live store target, resolved once: the store step writes to it,+ // and the Brain swap must keep out of its way when it sits inside+ // `Brain/`. The manifest's `source_path` is where the store WAS; the+ // live target is where it is now. They differ when the operator moved+ // the vault or set a `search_db_path` override after the snapshot, and+ // the live answer is the one a restore must write to.+ const storeTarget = opts.derivedStorePath ?? liveDerivedStorePath(liveVault);+ const placement = storePlacement(dirs.brain, storeTarget);+ refuseStoreTheSwapCannotWriteAround(placement, storeTarget, runId);+ // Names the Brain swap leaves alone in both directions, beyond the+ // excluded entries: with the store directly under `Brain/`, its file+ // family. The live store, its journals and lock, and the staged+ // replacement stay where the store swap expects them, and the raw copy+ // of the store the Brain archive carries (it archives every top-level+ // entry) is not restored over it, so the store is written once, by the+ // store swap, from the store archive - or not at all when the snapshot+ // did not cover it.+ const keptOutOfSwap = (name: string, isDirectory: boolean): boolean =>+ placement.kind === "top" && isStoreFamilyEntry(name, placement.base, isDirectory);+const ext = extractSnapshotToTemp(vault, runId);+ let staged: StagedBrainTree | null = null;+ let store: StagedDerivedStore | null = null;try {// Replace every top-level entry under Brain/, except the entries the// snapshot family never touches. The `.snapshots/` exclusion is the@@ -2196,72 +2346,106 @@ export function restoreSnapshot(// away from losing their forward path. `.artifacts/` is excluded for// the opposite reason - it is not in the archive at all, so deleting// the live copy would restore nothing over it.- const replacementEntries = readdirSync(ext.brainRoot).filter(- (e) => !isSnapshotExcludedEntry(e),- );+ //+ // `.snapshots/` is created first, as `createSnapshot` does: the+ // staging directory goes there when it shares `Brain/`'s device, and+ // a missing one would otherwise read as "another device" and send the+ // staging directory into `Brain/` itself.+ ensureSnapshotsDirectory(dirs.snapshots, runId);+ staged = stageBrainTree(ext.brainRoot, dirs.brain, dirs.snapshots, runId, keptOutOfSwap);++ // The derived store is decompressed and verified here, before the+ // recovery point and before the Brain tree is touched. It used to be+ // decompressed only after the tree had been replaced, so a `zstd`+ // timeout, a corrupt payload or a full disk left a rolled-back tree+ // beside the store it was meant to replace.+ const plan = stageDerivedStore(vault, storeTarget, runId);+ store = plan.staged;// The correct semantics are "live tree == snapshot tree minus the- // excluded entries". Delete every live top-level entry except those,- // then copy in from the extracted Brain/. This makes restore+ // excluded entries". Every live top-level entry except those is moved+ // out, and every staged one moved in. This makes restore// deterministic.const liveEntries = existsSync(dirs.brain)- ? readdirSync(dirs.brain).filter((e) => !isSnapshotExcludedEntry(e))+ ? readdirSync(dirs.brain, { withFileTypes: true })+ .filter(+ (e) =>+ !isSnapshotExcludedEntry(e.name) &&+ e.name !== staged!.name &&+ !keptOutOfSwap(e.name, e.isDirectory()),+ )+ .map((e) => e.name): [];- // The last moment at which the live tree still exists. A throw here- // propagates and nothing has been removed yet, which is the same- // abort-before-the-operation rule the destructive gate applies.- const evidence = opts.beforeDiscard?.() ?? null;-- // The callback is the one step that can change `.snapshots/` between- // the pre-flight and the store swap - it takes a recovery point, and- // a retention pass rides behind that. `assertDerivedStoreRestorable`- // above validated a state this callback may have invalidated, so the- // pre-flight is re-run against the state that actually reaches the- // rest of the restore, and the RECORD is compared rather than only- // re-checked: a manifest that has been removed outright would pass a- // fresh pre-flight while turning a covered store into "coverage- // unknown, nothing restored".- if (opts.beforeDiscard !== undefined) {- assertDerivedStoreRestorable(vault, runId);- const after = derivedStoreRestoreState(vault, runId);- if (after !== storeStateBefore) {- throw new BrainSnapshotError(- "the derived-store records of this snapshot changed while the recovery point was " +- `being taken (${storeStateBefore} -> ${after}); refusing before the Brain tree is ` +- "touched, so nothing is half-restored",- runId,- );+ // The writer lock is taken before the recovery point, not inside the+ // store swap, and held through the swap. Taken before the recovery+ // point because a writer committing between the recovery point and+ // the swap would land a change the swap discards and the recovery+ // point never saw. Taken before the first rename because an index run+ // holding it is a refusal like any other, and it must arrive while+ // nothing has been replaced yet.+ const lock = store === null ? null : acquireStoreLockForRestore(store.target, runId);+ let evidence: RecoveryPointEvidence | null = null;+ let committed = false;+ let primary: { readonly err: unknown } | null = null;+ try {+ // The last moment at which the live state still exists. A throw+ // here propagates and nothing has been replaced yet, which is the+ // same abort-before-the-operation rule the destructive gate applies.+ evidence =+ opts.beforeDiscard?.({+ storeTarget: store?.target ?? null,+ heldStoreLock: lock?.token ?? null,+ }) ?? null;+ if (store !== null && evidence !== null) {+ assertRecoveryPointCoversTarget(vault, evidence, store.target, runId);}- }- for (const name of liveEntries) {- const target = join(dirs.brain, name);- try {- rmSync(target, { recursive: true, force: true });- } catch (err) {- throw new BrainSnapshotError(- `failed to remove live entry ${name}: ${(err as Error).message ?? String(err)}`,- runId,- );+ // The callback is the one step that can change `.snapshots/` between+ // the pre-flight and the store swap - it takes a recovery point, and+ // a retention pass rides behind that. `assertDerivedStoreRestorable`+ // above validated a state this callback may have invalidated, so the+ // pre-flight is re-run against the state that actually reaches the+ // rest of the restore, and the RECORD is compared rather than only+ // re-checked: a manifest that has been removed outright would pass a+ // fresh pre-flight while turning a covered store into "coverage+ // unknown, nothing restored".+ if (opts.beforeDiscard !== undefined) {+ assertDerivedStoreRestorable(vault, runId);+ const after = derivedStoreRestoreState(vault, runId);+ if (after !== storeStateBefore) {+ throw new BrainSnapshotError(+ "the derived-store records of this snapshot changed while the recovery point was " ++ `being taken (${storeStateBefore} -> ${after}); refusing before the Brain tree is ` ++ "touched, so nothing is half-restored",+ runId,+ );+ }}++ const swap = swapBrainTree(staged, dirs.brain, liveEntries, runId, evidence);+ if (store !== null) {+ try {+ commitDerivedStoreSwap(store);+ } catch (err) {+ throw swap.fail(`the derived store swap failed (${errorText(err)})`, err, {+ storeIntact: !(err instanceof StoreSwapTornError),+ });+ }+ }+ committed = true;+ swap.finish();+ } catch (err) {+ primary = { err };+ throw err;+ } finally {+ if (lock !== null)+ releaseStoreLockAfterRestore(lock.release, runId, committed, primary, evidence);}- // Copy in each extracted entry. `verbatimSymlinks` is load-bearing:- // without it `cpSync` rewrites every relative link to the ABSOLUTE- // path of its target inside the temp extraction, which the cleanup- // below deletes - the in-tree links the containment sweep kept would- // all restore dangling, pointing outside the vault.- let restoredFiles = 0;- mkdirSync(dirs.brain, { recursive: true });- for (const name of replacementEntries) {- const from = join(ext.brainRoot, name);- const to = join(dirs.brain, name);- cpSync(from, to, { recursive: true, verbatimSymlinks: true });- restoredFiles += countFiles(to);- }- const derivedStore = restoreDerivedStore(vault, liveVault, runId, opts);++ const derivedStore = plan.result;return {- restored_files: restoredFiles,+ restored_files: staged.files,derived_store: derivedStore,// What the DISCARDED tree was worth, not the archive's. The live// derived store is replaced only when the archive carried one, so@@ -2278,28 +2462,266 @@ export function restoreSnapshot(}),};} finally {+ store?.discard();+ staged?.cleanup();ext.cleanup();}}/**- * Put the archived derived store back, but ONLY when the manifest says- * this snapshot carried one.+ * The extracted `Brain/` entries, copied onto the live tree's filesystem+ * so the swap is a run of renames rather than a delete-and-copy.*- * Three answers, and the caller reports whichever it gets rather than- * leaving any of them implicit:+ * `root` holds `incoming/` (the staged entries) and, once the swap has+ * started, `outgoing/` (the live entries moved aside). {@link cleanup}+ * removes it best-effort, and keeps it when {@link keep} has been called+ * because the outgoing entries are the only copy left.+ */+interface StagedBrainTree {+ /** Top-level name of {@link root} inside its parent directory. */+ readonly name: string;+ readonly root: string;+ readonly incoming: string;+ readonly entries: ReadonlyArray<string>;+ /** Regular files under {@link incoming}. */+ readonly files: number;+ keep(): void;+ cleanup(): void;+}++/**+ * Copy the extracted tree to a staging directory on the live tree's+ * filesystem and verify the copy.+ *+ * The staging directory goes under `.snapshots/` - excluded from every+ * archive, manifest walk and restore, and never listed as a snapshot+ * since it carries no archive suffix - when that directory shares a+ * device with `Brain/`. When it does not (a separately mounted snapshot+ * directory), a rename from it into `Brain/` would fail mid-swap with+ * EXDEV, so the staging directory goes directly under `Brain/` instead+ * and the swap leaves it alone.+ *+ * Entries `keptOut` names (the store's file family, when the store sits+ * directly under `Brain/`) are neither staged nor counted: the swap leaves+ * the live ones in place, and the store is restored by the store swap.+ */+function stageBrainTree(+ extractedBrain: string,+ liveBrain: string,+ snapshotsDirPath: string,+ runId: string,+ keptOut: (name: string, isDirectory: boolean) => boolean,+): StagedBrainTree {+ mkdirSync(liveBrain, { recursive: true });+ const parent = sameDevice(snapshotsDirPath, liveBrain) ? snapshotsDirPath : liveBrain;+ let root: string;+ try {+ root = mkdtempSync(join(parent, `${BRAIN_RESTORE_STAGING_PREFIX}${runId}-`));+ } catch (err) {+ throw new BrainSnapshotError(+ `cannot create a restore staging directory under ${parent}: ${errorText(err)}`,+ runId,+ { cause: err },+ );+ }+ let kept = false;+ const cleanup = (): void => {+ if (kept) return;+ try {+ rmSync(root, { recursive: true, force: true });+ } catch (err) {+ console.error(+ `warning: could not remove the restore staging directory ${root}: ${errorText(err)}`,+ );+ }+ };+ try {+ const incoming = join(root, "incoming");+ mkdirSync(incoming);+ const entries = readdirSync(extractedBrain, { withFileTypes: true })+ .filter((e) => !isSnapshotExcludedEntry(e.name) && !keptOut(e.name, e.isDirectory()))+ .map((e) => e.name);+ let files = 0;+ let expected = 0;+ for (const name of entries) {+ // `verbatimSymlinks` is load-bearing: without it `cpSync` rewrites+ // every relative link to the ABSOLUTE path of its target inside the+ // temp extraction, which the cleanup deletes - the in-tree links the+ // containment sweep kept would all restore dangling, pointing outside+ // the vault. A relative link survives the later rename unchanged,+ // because the staged entry sits at the same depth as the live one.+ const from = join(extractedBrain, name);+ const to = join(incoming, name);+ cpSync(from, to, { recursive: true, verbatimSymlinks: true });+ }+ // Counted only once every entry is staged: a top-level link to another+ // top-level entry resolves in the staged copy only after its target is.+ for (const name of entries) {+ expected += countFiles(join(extractedBrain, name));+ files += countFiles(join(incoming, name));+ }+ if (files !== expected) {+ throw new BrainSnapshotError(+ `the staged Brain tree holds ${files} file(s) but the archive extracted ${expected}; ` ++ "refusing before the Brain tree is touched",+ runId,+ );+ }+ return {+ name: basename(root),+ root,+ incoming,+ entries,+ files,+ keep: () => {+ kept = true;+ },+ cleanup,+ };+ } catch (err) {+ cleanup();+ if (err instanceof BrainSnapshotError) throw err;+ throw new BrainSnapshotError(+ `failed to stage the extracted Brain tree under ${root}: ${errorText(err)}`,+ runId,+ { cause: err },+ );+ }+}++/** Whether two existing paths live on the same device; false when unsure. */+function sameDevice(a: string, b: string): boolean {+ try {+ return statSync(a).dev === statSync(b).dev;+ } catch {+ return false;+ }+}++/** An in-progress Brain swap: {@link finish} it, or {@link fail} it. */+interface BrainSwap {+ /** Remove the outgoing entries; the swap is complete. */+ finish(): void;+ /**+ * Undo the swap and return the error to throw, naming what failed,+ * whether the undo succeeded and how to recover when it did not.+ */+ fail(what: string, cause: unknown, opts?: { readonly storeIntact?: boolean }): BrainSnapshotError;+}++/**+ * Move every live top-level entry into the staging directory's+ * `outgoing/`, then rename every staged entry into `Brain/`.*- * - the manifest records inclusion: decompress and swap;- * - the manifest records an exclusion: nothing is touched, and the- * named reason travels back;- * - there is no record at all: coverage is UNKNOWN, and whether a- * sibling archive is on disk travels back with it. A restore must not- * guess in either direction here - replacing a live store on a hunch- * destroys embeddings the archive never held, and reporting- * "excluded" would claim a check that never ran. The archive probe is- * what lets the caller say WHY the record is missing instead of- * asserting the snapshot is older than the feature.+ * Renames on one filesystem: each one either happens or does not, and+ * the outgoing entries stay on disk until {@link BrainSwap.finish}, so a+ * failure at any point - here, or in the store swap that follows - is+ * undone by renaming everything back in reverse order.*/+function swapBrainTree(+ staged: StagedBrainTree,+ liveBrain: string,+ liveEntries: ReadonlyArray<string>,+ runId: string,+ recoveryPoint: RecoveryPointEvidence | null,+): BrainSwap {+ const outgoing = join(staged.root, "outgoing");+ const movedOut: string[] = [];+ const movedIn: string[] = [];++ const undo = (): string[] => {+ const failures: string[] = [];+ for (const name of movedIn.toReversed()) {+ try {+ renameWithRetry(join(liveBrain, name), join(staged.incoming, name));+ } catch (err) {+ failures.push(`could not move the restored ${name} back out: ${errorText(err)}`);+ }+ }+ for (const name of movedOut.toReversed()) {+ try {+ renameWithRetry(join(outgoing, name), join(liveBrain, name));+ } catch (err) {+ failures.push(`could not move the live ${name} back: ${errorText(err)}`);+ }+ }+ return failures;+ };++ const fail = (+ what: string,+ cause: unknown,+ opts: { readonly storeIntact?: boolean } = {},+ ): BrainSnapshotError => {+ const storeIntact = opts.storeIntact ?? true;+ const swapped =+ `${movedOut.length} of ${liveEntries.length} live Brain/ top-level entries had been moved ` ++ `aside and ${movedIn.length} of ${staged.entries.length} restored ones moved in`;+ const failures = undo();+ const recovery =+ recoveryPoint === null+ ? "no recovery point was taken for this restore"+ : `the recovery point ${recoveryPoint.runId} (${recoveryPoint.path}) holds the ` ++ `pre-restore state; \`o2b brain rollback ${recoveryPoint.runId}\` puts it back`;+ if (failures.length === 0) {+ const state = storeIntact+ ? "Brain/ and the derived store are as they were before the restore"+ : `Brain/ is as it was before the restore, but the derived store is not (see above); ${recovery}`;+ return new BrainSnapshotError(+ `restore failed during the swap: ${what}. ${swapped}; every Brain/ rename was undone, so ${state}`,+ runId,+ { cause },+ );+ }+ // The outgoing entries are now the only copy of what did not make it+ // back: the staging directory must outlive this restore.+ staged.keep();+ return new BrainSnapshotError(+ `restore failed during the swap: ${what}. ${swapped}; undoing it also failed ` ++ `(${failures.join("; ")}), so Brain/ is a mix of the restored and the pre-restore ` ++ `tree. The pre-restore entries that were moved aside are kept under ${outgoing} and ` ++ `the restored ones not yet moved in under ${staged.incoming}; ${recovery}`,+ runId,+ { cause },+ );+ };++ try {+ mkdirSync(outgoing);+ for (const name of liveEntries) {+ renameWithRetry(join(liveBrain, name), join(outgoing, name));+ movedOut.push(name);+ }+ for (const name of staged.entries) {+ renameWithRetry(join(staged.incoming, name), join(liveBrain, name));+ movedIn.push(name);+ }+ } catch (err) {+ throw fail(`a Brain/ rename failed (${errorText(err)})`, err);+ }++ return {+ finish: () => {+ try {+ rmSync(outgoing, { recursive: true, force: true });+ } catch (err) {+ // The restore itself is complete; what is left is the discarded+ // tree, which the staging cleanup retries.+ console.error(+ `warning: could not remove the pre-restore Brain entries under ${outgoing}: ` ++ errorText(err),+ );+ }+ },+ fail,+ };+}++/** An error's message, or its string form. */+function errorText(err: unknown): string {+ return err instanceof Error ? err.message : String(err);+}+/*** Refuse a restore that cannot be completed, before it starts.*@@ -2355,104 +2777,659 @@ function assertDerivedStoreRestorable(vault: string, runId: string): void {}}-function restoreDerivedStore(+/**+ * Where the derived store sits relative to `Brain/`, which decides how the+ * Brain swap must treat it.+ *+ * - `outside`: the swap and the store never meet.+ * - `top`: the store is a direct child of `Brain/`. The swap leaves its+ * file family (see {@link isStoreFamilyEntry}) where it is, in both+ * directions; the store is written only by the store swap.+ * - `nested`: the store is deeper inside a top-level `Brain/` entry the+ * swap replaces wholesale; `top` names that entry.+ * - `brain`: the store path is the `Brain/` directory itself.+ */+export type StorePlacement =+ | { readonly kind: "outside" }+ | { readonly kind: "top"; readonly base: string }+ | { readonly kind: "nested"; readonly top: string }+ | { readonly kind: "brain" };++/**+ * True where the filesystem compares names without case by default+ * (APFS, HFS+, NTFS), so `Index.sqlite` and `index.sqlite` are one file.+ */+function foldsCase(platform: NodeJS.Platform): boolean {+ return platform === "darwin" || platform === "win32";+}++/** `name` as the platform compares it. */+function compareKey(name: string, platform: NodeJS.Platform): string {+ return foldsCase(platform) ? name.toLowerCase() : name;+}++/**+ * A path with every existing ancestor resolved through symlinks. Where the+ * platform folds case, an absent component takes the spelling of the one+ * entry of its parent that matches it without case, so a store configured+ * as `Index.sqlite` is recognised as the `index.sqlite` on disk and in the+ * archive.+ */+function canonicalPath(path: string, platform: NodeJS.Platform): string {+ const abs = resolve(path);+ try {+ return realpathSync.native(abs);+ } catch {+ const parent = dirname(abs);+ if (parent === abs) return abs;+ const canonicalParent = canonicalPath(parent, platform);+ return join(canonicalParent, existingSpelling(canonicalParent, basename(abs), platform));+ }+}++/** The spelling of `name` among `dir`'s entries where case is folded. */+function existingSpelling(dir: string, name: string, platform: NodeJS.Platform): string {+ if (!foldsCase(platform)) return name;+ let entries: string[];+ try {+ entries = readdirSync(dir);+ } catch {+ return name;+ }+ if (entries.includes(name)) return name;+ const key = compareKey(name, platform);+ const matches = entries.filter((e) => compareKey(e, platform) === key);+ return matches.length === 1 ? matches[0]! : name;+}++/**+ * The directory entry a rename of `path` acts on: every ancestor resolved+ * through symlinks, the last component itself not followed. Where the+ * platform folds case, an existing entry takes its on-disk spelling.+ */+function entryPath(path: string, platform: NodeJS.Platform): string {+ const abs = resolve(path);+ const parent = dirname(abs);+ if (parent === abs) return canonicalPath(abs, platform);+ const canonicalParent = canonicalPath(parent, platform);+ return join(canonicalParent, existingSpelling(canonicalParent, basename(abs), platform));+}++/**+ * Classify the store target against the live `Brain/` directory. The+ * containment test compares case-folded paths where the platform folds+ * case; the names it returns keep the target's canonical spelling.+ *+ * What is classified is the directory entry at the target path, not the+ * file a symlink there points to: the store swap renames that entry (it+ * stages `<target>.restore-*` beside it, moves it aside to `<target>.bak`+ * and renames the staged file over it), so the Brain swap must judge the+ * same entry. A `Brain/index.sqlite` that links to a database outside+ * `Brain/` is a store directly under `Brain/`: the link and the staged+ * file beside it stay out of the tree swap, and the store swap replaces+ * the link with the restored database. Classifying the link's target+ * instead called it `outside`, and the tree swap then moved the staged+ * file away before the store swap could rename it.+ */+export function storePlacement(+ liveBrain: string,+ target: string,+ platform: NodeJS.Platform = process.platform,+): StorePlacement {+ const brain = canonicalPath(liveBrain, platform);+ // A path that resolves to `Brain/` itself is refused whatever its last+ // component is: a link to the directory is still the directory.+ if (compareKey(canonicalPath(target, platform), platform) === compareKey(brain, platform)) {+ return { kind: "brain" };+ }+ const store = entryPath(target, platform);+ const rel = relative(compareKey(brain, platform), compareKey(store, platform));+ if (rel === "") return { kind: "brain" };+ if (rel === ".." || rel.startsWith(`..${sep}`) || isAbsolute(rel)) return { kind: "outside" };+ // The relative path's components, in the target's own spelling.+ const parts = store.split(sep).slice(-rel.split(sep).length);+ // An entry the swap never touches (`.snapshots/`, `.artifacts/`) keeps+ // whatever it holds, so a store inside one does not meet the swap.+ if (isSnapshotExcludedEntry(parts[0]!)) return { kind: "outside" };+ return parts.length === 1 ? { kind: "top", base: parts[0]! } : { kind: "nested", top: parts[0]! };+}++/** SQLite's journal siblings of a database file: WAL mode and DELETE mode. */+const STORE_JOURNAL_SUFFIXES: ReadonlyArray<string> = Object.freeze(["-wal", "-shm", "-journal"]);++/**+ * The database files the store layer keeps beside the store, as suffixes+ * of its name: the store itself, the reindex staging file (`.new`), the+ * parked previous index (`.bak`, also the restore's outgoing store, and+ * `.old`).+ */+const STORE_DATABASE_SUFFIXES: ReadonlyArray<string> = Object.freeze(["", ".new", ".bak", ".old"]);++/** The writer lock `proper-lockfile` takes on the store: a directory. */+const STORE_LOCK_SUFFIX = ".lock";++/** The restore's staged store: `<store>.restore-<pid>-<8 hex>`. */+const STAGED_STORE_PATTERN = /^\.restore-\d+-[0-9a-f]{8}$/u;++/**+ * True for a top-level `Brain/` entry that belongs to the store named+ * `base` when the store sits directly under `Brain/`. A closed list, so a+ * note or directory that merely starts with the store's name is restored+ * like any other entry:+ *+ * - the store and each database file the store layer derives from it+ * (`.new`, `.bak`, `.old`), each with its journals (`-wal`, `-shm`,+ * `-journal`);+ * - the restore's staged store (`.restore-<pid>-<hex>`) with its+ * journals;+ * - the writer lock directory (`.lock`), the only directory in the+ * family.+ *+ * Names compare without case where the platform folds it. A suffix the+ * store layer adds later must be listed here; until it is, the Brain swap+ * treats that file as Brain content.+ */+export function isStoreFamilyEntry(+ name: string,+ base: string,+ isDirectory: boolean,+ platform: NodeJS.Platform = process.platform,+): boolean {+ const key = compareKey(name, platform);+ const stem = compareKey(base, platform);+ if (!key.startsWith(stem)) return false;+ const suffix = key.slice(stem.length);+ if (isDirectory) return suffix === STORE_LOCK_SUFFIX;+ // At most one journal suffix comes off: `-wal-shm` is not a journal.+ const journal = STORE_JOURNAL_SUFFIXES.find((j) => suffix.endsWith(j));+ const file = journal === undefined ? suffix : suffix.slice(0, -journal.length);+ return STORE_DATABASE_SUFFIXES.includes(file) || STAGED_STORE_PATTERN.test(file);+}++/**+ * Refuse a restore whose store target the swap cannot write around, before+ * anything is extracted or touched.+ *+ * A store inside a top-level `Brain/` entry: the Brain swap renames whole+ * top-level entries; the store swap renames one file. A store nested+ * inside a renamed directory would be moved out with the live directory,+ * its staged replacement with it, and the restored directory may not even+ * contain the store's parent. Carving the store out of that rename would+ * add a third set of renames, each with its own undo, to a swap whose+ * whole contract is that every step can be reversed; refusing up front+ * keeps that contract and loses nothing.+ *+ * A store target that is `Brain/` itself: the store swap would rename a+ * database file over the directory the tree swap is restoring into.+ */+function refuseStoreTheSwapCannotWriteAround(+ placement: StorePlacement,+ target: string,+ runId: string,+): void {+ if (placement.kind === "brain") {+ throw new BrainSnapshotError(+ `the derived store path ${target} is the Brain/ directory itself; refusing before ` ++ "anything is touched. Point search_db_path at a file, directly under Brain/ or " ++ "outside it, then restore again",+ runId,+ );+ }+ if (placement.kind !== "nested") return;+ throw new BrainSnapshotError(+ `the derived store ${target} is inside Brain/${placement.top}, which a restore replaces ` ++ "as a whole; refusing before anything is touched. Move the store (search_db_path) " ++ "directly under Brain/ or outside it, then restore again",+ runId,+ );+}++/** A decompressed and verified store, waiting beside its live target. */+interface StagedDerivedStore {+ /** The live store path the staged file is renamed over. */+ readonly target: string;+ /** The staged file, on the target's filesystem. */+ readonly incoming: string;+ /** Remove the staged file if it is still there. Best-effort. */+ discard(): void;+}++/**+ * Decide what the restore does with the derived store, and when the+ * manifest says the snapshot carried one, decompress and verify it next+ * to the live store - without touching the live store.+ *+ * Three answers, and the caller reports whichever it gets rather than+ * leaving any of them implicit:+ *+ * - the manifest records inclusion: decompress and verify here, swap+ * later in {@link commitDerivedStoreSwap};+ * - the manifest records an exclusion: nothing is touched, and the+ * named reason travels back;+ * - there is no record at all: coverage is UNKNOWN, and whether a+ * sibling archive is on disk travels back with it. A restore must not+ * guess in either direction here - replacing a live store on a hunch+ * destroys embeddings the archive never held, and reporting+ * "excluded" would claim a check that never ran. The archive probe is+ * what lets the caller say WHY the record is missing instead of+ * asserting the snapshot is older than the feature.+ *+ * Runs without the writer lock: the staged file has a name of its own+ * beside the store, and the lock is held only across the renames. The+ * recovery point the caller takes after this step snapshots the live+ * store under that same lock, so holding it here would deadlock.+ */+function stageDerivedStore(vault: string,- liveVault: string,+ target: string,runId: string,- opts: SnapshotStoreOptions,-): RestoreDerivedStoreResult {+): { readonly result: RestoreDerivedStoreResult; readonly staged: StagedDerivedStore | null } {const record = readManifestSidecar(vault, runId)?.derived_store ?? null;// Probed once, for every branch, and deliberately independent of the// record: it is the only evidence that separates a snapshot with no// record from a snapshot whose record was lost after the archive landed.const storeArchivePresent = existsSync(snapshotStorePath(vault, runId));if (record === null) {- return Object.freeze({- replaced: false,- coverage_known: false,- path: null,- exclusion_reason: null,- store_archive_present: storeArchivePresent,- });+ return {+ result: Object.freeze({+ replaced: false,+ coverage_known: false,+ path: null,+ exclusion_reason: null,+ store_archive_present: storeArchivePresent,+ }),+ staged: null,+ };}if (!record.included) {- return Object.freeze({- replaced: false,- coverage_known: true,- path: null,- exclusion_reason: record.exclusion_reason,- store_archive_present: storeArchivePresent,- });+ return {+ result: Object.freeze({+ replaced: false,+ coverage_known: true,+ path: null,+ exclusion_reason: record.exclusion_reason,+ store_archive_present: storeArchivePresent,+ }),+ staged: null,+ };}const archive = snapshotStorePath(vault, runId);if (!existsSync(archive)) {// Unreachable through `restoreSnapshot`, which refuses this in its- // pre-flight. Kept because this function is also the one a future- // caller would reach directly, and a store swap must never begin- // against an archive that is not there.+ // pre-flight. Kept because a store swap must never begin against an+ // archive that is not there.throw new BrainSnapshotError(`manifest records a derived-store archive but ${archive} is absent`,runId,);}- // The manifest's `source_path` is where the store WAS; the live target- // is where it is now. They differ when the operator moved the vault or- // set a `search_db_path` override after the snapshot, and the live- // answer is the one a restore must write to.- // The index the search layer opens: resolved from the caller's vault.- const target = opts.derivedStorePath ?? liveDerivedStorePath(liveVault);- swapDerivedStore(archive, target, runId);- return Object.freeze({- replaced: true,- coverage_known: true,- path: target,- exclusion_reason: null,- store_archive_present: true,- });+ if (record.archive_sha256 !== null) {+ const actual = sha256Hex(readFileSync(archive));+ if (actual !== record.archive_sha256) {+ throw new BrainSnapshotError(+ `the derived-store archive ${archive} does not match the digest its manifest recorded ` ++ `(${record.archive_sha256}, found ${actual}); refusing before the Brain tree is touched`,+ runId,+ );+ }+ }+ mkdirSync(dirname(target), { recursive: true });+ // A name of this process's own: two restores staging at once must not+ // write into one file, and neither is under the lock yet.+ const incoming = `${target}${RESTORE_STAGING_SUFFIX}-${process.pid}-${randomBytes(4).toString("hex")}`;+ const discard = (): void => {+ for (const path of [incoming, ...WAL_SIBLING_SUFFIXES.map((s) => `${incoming}${s}`)]) {+ try {+ rmSync(path, { force: true });+ } catch {+ // best-effort: a leftover staged file is never opened as the store+ }+ }+ };+ try {+ decompressArchiveTo(archive, incoming, runId);+ try {+ assertStoreIsSound(incoming, runId);+ } catch (err) {+ throw new BrainSnapshotError(+ `the archived derived store failed verification after decompression: ${errorText(err)}`,+ runId,+ { cause: err },+ );+ }+ // Opening it for the check must not leave journal siblings to travel+ // with the rename; the decompressed file is complete on its own.+ for (const sidecar of WAL_SIBLING_SUFFIXES) rmSync(`${incoming}${sidecar}`, { force: true });+ } catch (err) {+ discard();+ if (err instanceof BrainSnapshotError || err instanceof BrainSnapshotToolingMissingError) {+ throw err;+ }+ throw new BrainSnapshotError(+ `failed to stage the derived store beside ${target}: ${errorText(err)}`,+ runId,+ { cause: err },+ );+ }+ return {+ result: Object.freeze({+ replaced: true,+ coverage_known: true,+ path: target,+ exclusion_reason: null,+ store_archive_present: true,+ }),+ staged: Object.freeze({ target, incoming, discard }),+ };}/**- * Decompress `archive` beside the live store and rename it into place- * under the writer lock.- *- * The same swap discipline `reindexVault` already uses: build the- * replacement at a sibling path on the same filesystem, keep the- * outgoing file as `.bak`, then a single atomic rename. Holding the- * writer lock across it is what stops an index run from writing into the- * file being replaced.+ * The store's writer lock, taken before the Brain swap starts. Holding it+ * across the renames is what stops an index run from writing into the+ * file being replaced; taking it BEFORE the first rename is what turns+ * contention into a refusal rather than a half-finished restore.+ *+ * The acquisition is the only place a {@link StoreLockToken} is minted:+ * the recovery point skips its own acquire for that token, live and on+ * the same path, and for nothing that merely names the path. The token+ * stops counting before the lock is released, so a reference kept past+ * the restore never stands in for a lock nobody holds.+ */+function acquireStoreLockForRestore(+ target: string,+ runId: string,+): { readonly token: StoreLockToken; readonly release: () => void } {+ let unlock: () => void;+ try {+ unlock = acquireWriterLockSync(target);+ } catch (err) {+ throw new BrainSnapshotError(+ `cannot take the derived store's writer lock (${errorText(err)}); refusing before the ` ++ "Brain tree is touched",+ runId,+ { cause: err },+ );+ }+ const token = new StoreLockToken(STORE_LOCK_TOKEN_MINT, target);+ liveStoreLockTokens.set(token, target);+ return {+ token,+ release: (): void => {+ liveStoreLockTokens.delete(token);+ unlock();+ },+ };+}++/**+ * Refuse a swap whose recovery point archived a different store than the+ * one the swap replaces.+ *+ * The restore writes the snapshot's store to the live target it resolved+ * (an explicit `derivedStorePath`, or the search layer's answer). A+ * recovery point that resolved its own source - the configured store+ * rather than the override - would archive one file while the swap+ * discards another, and the discarded store would have no copy anywhere.+ * The manifest's `source_path` is what the recovery point actually+ * archived (or declined to), so that is what is compared. A recovery+ * point with no readable manifest is left to the recoverability verdict,+ * which already reads it as unproven.+ */+function assertRecoveryPointCoversTarget(+ vault: string,+ evidence: RecoveryPointEvidence,+ target: string,+ runId: string,+): void {+ const record = readManifestSidecar(vault, evidence.runId)?.derived_store ?? null;+ if (record === null) return;+ if (record.source_path !== target) {+ throw new BrainSnapshotError(+ `the recovery point ${evidence.runId} recorded the derived store at ${record.source_path}, ` ++ `but this restore replaces ${target}; refusing before the Brain tree is touched, so the ` ++ "store being discarded is not left without a copy",+ runId,+ );+ }+ if (!record.included && liveStoreWorthPreserving(target)) {+ throw new BrainSnapshotError(+ `the recovery point ${evidence.runId} did not include the derived store at ${target}, ` ++ "which this restore replaces; refusing before the Brain tree is touched, so the store " ++ "being discarded is not left without a copy",+ runId,+ );+ }+}++/**+ * Release the writer lock {@link acquireStoreLockForRestore} took.+ *+ * A failed release must not hide what happened before it. When the+ * restore already failed, that error stays the one thrown and the+ * release failure is appended to its message as a second line. When the+ * swap committed, the live tree and store are the snapshot's now, and the+ * error says so and names the recovery point, rather than reading like a+ * restore that changed nothing.+ */+function releaseStoreLockAfterRestore(+ release: () => void,+ runId: string,+ committed: boolean,+ primary: { readonly err: unknown } | null,+ recoveryPoint: RecoveryPointEvidence | null,+): void {+ try {+ release();+ return;+ } catch (releaseErr) {+ const what = `releasing the derived store's writer lock failed (${errorText(releaseErr)})`;+ if (primary !== null) {+ // The primary error is rethrown by the caller's catch once this+ // finally returns; it only gains the release failure as a line.+ if (primary.err instanceof Error) {+ primary.err.message += `\nalso: ${what}`;+ } else {+ // A non-Error throw has no message to extend; the release failure+ // still must not vanish.+ console.error(`warning: ${what}`);+ }+ return;+ }+ if (committed) {+ const recovery =+ recoveryPoint === null+ ? "no recovery point was taken for this restore"+ : `the recovery point ${recoveryPoint.runId} (${recoveryPoint.path}) holds the ` ++ `pre-restore state; \`o2b brain rollback ${recoveryPoint.runId}\` puts it back`;+ throw new BrainSnapshotError(+ `the restore committed: Brain/ and the derived store now hold snapshot ${runId}, but ` ++ `${what}; the lock is stale-released after ${WRITER_LOCK_STALE_MS} ms; ${recovery}`,+ runId,+ { cause: releaseErr },+ );+ }+ // Unreachable: nothing failed and nothing committed means the try+ // block did not finish, which is a throw, which is `primary`.+ throw new BrainSnapshotError(`restore: ${what}`, runId, { cause: releaseErr });+ }+}++/**+ * A store swap that failed between its two renames and could not put the+ * outgoing store back: the live path has no store, and the pre-restore+ * one is at the `.bak` path.+ */+class StoreSwapTornError extends Error {+ constructor(message: string, options?: { readonly cause?: unknown }) {+ super(message, options);+ this.name = "StoreSwapTornError";+ }+}++/**+ * Rename the staged store over the live one. The caller holds the writer+ * lock.+ *+ * The same swap discipline `reindexVault` uses: the outgoing file is kept+ * as `.bak`, then a single atomic rename puts the replacement in place.+ * If that second rename fails, the `.bak` is renamed back, so the live+ * store is untouched; a {@link StoreSwapTornError} means even that failed.+ *+ * A rollback journal (`-journal`, DELETE journal mode, which the store+ * uses on remote filesystems and SQLite leaves hot after a crash) belongs+ * to the database it was written for and travels with it: it is moved to+ * `.bak-journal` right after the outgoing store moves to `.bak`, and back+ * before the store on every undo. Left at the live path, SQLite would+ * replay the old database's journal over the replacement on its next+ * open, which is the database/journal mismatch it documents as a cause of+ * corruption. A journal with no live store beside it moves the same way.+ * A journal that cannot be moved undoes the swap like any other failed+ * step, so the replacement never lands beside it.** The WAL siblings of the OUTGOING file are removed after the swap. The* decompressed archive came from `VACUUM INTO` and is a complete* database on its own; an orphan `-wal` from the file it replaced would* make the next open fail with SQLITE_IOERR_SHORT_READ, which is the* exact failure `consolidateWal` exists to prevent on the reindex path.+ * A removal that still fails after the Windows retry budget (a reader+ * holding the file) is not a warning: the swap is undone, the new store+ * back at its staging name and the `.bak` back at the live path, and the+ * error is thrown so the caller undoes the Brain/ renames too.*/-function swapDerivedStore(archive: string, target: string, runId: string): void {- mkdirSync(dirname(target), { recursive: true });- const incoming = `${target}${RESTORE_STAGING_SUFFIX}`;+function commitDerivedStoreSwap(store: StagedDerivedStore): void {+ const { target, incoming } = store;const outgoing = `${target}${RESTORE_BACKUP_SUFFIX}`;- const release = acquireWriterLockSync(target);+ const journal = `${target}${ROLLBACK_JOURNAL_SUFFIX}`;+ const outgoingJournal = `${outgoing}${ROLLBACK_JOURNAL_SUFFIX}`;+ const moved: MovedOutgoing = { target, outgoing, store: false, journal: false };try {- rmSync(incoming, { force: true });- decompressArchiveTo(archive, incoming, runId);+ // A `.bak` and its journal from an earlier swap are replaced together:+ // the earlier journal must not end up beside the store moved aside now.rmSync(outgoing, { force: true });- if (existsSync(target)) renameWithRetry(target, outgoing);- renameWithRetry(incoming, target);- for (const sidecar of WAL_SIBLING_SUFFIXES) {- rmSync(`${target}${sidecar}`, { force: true });+ rmSync(outgoingJournal, { force: true });+ if (existsSync(target)) {+ try {+ renameWithRetry(target, outgoing);+ } catch (err) {+ throw new Error(+ `could not move the live store ${target} aside to ${outgoing}: ${errorText(err)}; ` ++ "the live store was not touched",+ { cause: err },+ );+ }+ moved.store = true;+ }+ if (existsSync(journal)) {+ try {+ renameWithRetry(journal, outgoingJournal);+ } catch (err) {+ throw new Error(+ `could not move the rollback journal ${journal} aside to ${outgoingJournal}: ` ++ `${errorText(err)}; the restored store must not open beside it`,+ { cause: err },+ );+ }+ moved.journal = true;}+ renameWithRetry(incoming, target);} catch (err) {- rmSync(incoming, { force: true });- if (err instanceof BrainSnapshotError || err instanceof BrainSnapshotToolingMissingError) {- throw err;+ putBackOutgoingStore(moved, errorText(err), err);+ throw err;+ }+ try {+ removeWalSiblings(target);+ } catch (err) {+ const what =+ `could not remove ${errorText(err)} beside the restored store, and an orphan ` ++ "write-ahead log would corrupt the next open";+ try {+ renameWithRetry(target, incoming);+ } catch (backErr) {+ throw new StoreSwapTornError(+ `${what}; the restored store could not be moved back off ${target} either ` ++ `(${errorText(backErr)})` ++ (moved.store ? `, and the pre-restore store is at ${outgoing}` : "") ++ (moved.journal ? `, its rollback journal at ${outgoingJournal}` : ""),+ { cause: err },+ );}- throw new BrainSnapshotError(- `failed to restore the derived store to ${target}: ${(err as Error).message ?? String(err)}`,- runId,+ putBackOutgoingStore(moved, what, err);+ throw new Error(+ `${what}; the store swap was undone` ++ (moved.store+ ? ", the pre-restore store is back in place"+ : ", and no store was live before it"),+ { cause: err },);- } finally {- release();+ }+}++/** What {@link commitDerivedStoreSwap} has moved aside so far. */+interface MovedOutgoing {+ readonly target: string;+ readonly outgoing: string;+ /** The live store is at `.bak`. */+ store: boolean;+ /** The live store's rollback journal is at `.bak-journal`. */+ journal: boolean;+}++/**+ * Put what a failed swap step moved aside back at the live path, or throw+ * a {@link StoreSwapTornError} naming where it was left.+ *+ * The journal goes back first. A crash between the two renames then+ * leaves the journal at the live path with no store and the store at+ * `.bak`, which the crash-recovery preamble of every `Store.open` puts+ * back beside it; the other order would leave the old store live without+ * the journal it needs to roll back.+ */+function putBackOutgoingStore(moved: MovedOutgoing, what: string, cause: unknown): void {+ const { target, outgoing } = moved;+ if (moved.journal) {+ const journal = `${target}${ROLLBACK_JOURNAL_SUFFIX}`;+ const outgoingJournal = `${outgoing}${ROLLBACK_JOURNAL_SUFFIX}`;+ try {+ renameWithRetry(outgoingJournal, journal);+ } catch (backErr) {+ throw new StoreSwapTornError(+ `${what}; the rollback journal of the pre-restore store could not be renamed back ` ++ `(${errorText(backErr)}) and is at ${outgoingJournal}` ++ (moved.store ? `; the pre-restore store is at ${outgoing}` : ""),+ { cause },+ );+ }+ }+ if (!moved.store) return;+ try {+ renameWithRetry(outgoing, target);+ } catch (backErr) {+ throw new StoreSwapTornError(+ `${what}; the pre-restore store could not be renamed back either ` ++ `(${errorText(backErr)}) and is at ${outgoing}, where the next store open restores it`,+ { cause },+ );+ }+}++/**+ * Remove the WAL siblings the outgoing store left at `target`, throwing+ * the path that could not be removed.+ *+ * `-shm` goes first: it is only an index of the `-wal` and SQLite+ * rebuilds it, so when the `-wal` removal then fails, the swap can still+ * be undone in full - the pre-restore store returns beside its own log.+ */+function removeWalSiblings(target: string): void {+ for (const sidecar of ["-shm", "-wal"] as const) {+ const path = `${target}${sidecar}`;+ try {+ unlinkWithRetry(path);+ } catch (err) {+ if ((err as NodeJS.ErrnoException | null)?.code === "ENOENT") continue;+ throw new Error(`${path} (${errorText(err)})`, { cause: err });+ }}}@@ -2464,6 +3441,8 @@ const RESTORE_STAGING_SUFFIX = ".restore";* it if a rollback dies between the two renames.*/const RESTORE_BACKUP_SUFFIX = ".bak";+/** SQLite's rollback journal of a database file (DELETE journal mode). */+const ROLLBACK_JOURNAL_SUFFIX = "-journal";/** SQLite's write-ahead-log siblings of a database file. */const WAL_SIBLING_SUFFIXES: ReadonlyArray<string> = Object.freeze(["-wal", "-shm"]);diff --git a/tests/core/brain/snapshot-restore-order.test.ts b/tests/core/brain/snapshot-restore-order.test.tsnew file mode 100644index 00000000..2a585fc1--- /dev/null+++ b/tests/core/brain/snapshot-restore-order.test.ts@@ -0,0 +1,1492 @@+/**+ * A restore changes nothing live until every step that can fail has run.+ *+ * `restoreSnapshot` used to replace the live `Brain/` tree and only then+ * decompress the archived derived store, so a `zstd` failure or timeout, a+ * corrupt payload or a full disk at that point left the rolled-back tree+ * beside the store it was meant to replace. The tree and the store are now+ * both staged and verified first, the recovery-point seam runs after that,+ * and the swap is a run of renames that is undone when one of them fails.+ *+ * Every failure case below asserts the same thing: `Brain/` (file by file),+ * `.snapshots/` (entry by entry) and the derived store (byte for byte) are+ * exactly what they were before the call, and no staging file is left.+ */++import { afterEach, beforeEach, describe, expect, setDefaultTimeout, spyOn, test } from "bun:test";+import { Database } from "bun:sqlite";+import * as fs from "node:fs";+import {+ chmodSync,+ existsSync,+ lstatSync,+ mkdirSync,+ mkdtempSync,+ readdirSync,+ readFileSync,+ readlinkSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { basename, dirname, join, relative } from "node:path";+import { gzipSync } from "node:zlib";+import lockfile from "proper-lockfile";++import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { manifestSidecarPath } from "../../../src/core/brain/manifest.ts";+import { BRAIN_ARTIFACTS_DIR } from "../../../src/core/brain/path-constants.ts";+import { brainDirs, snapshotPath, snapshotStorePath } from "../../../src/core/brain/paths.ts";+import {+ BrainSnapshotError,+ createSnapshot,+ isStoreFamilyEntry,+ restoreSnapshot,+ SNAPSHOT_TOOL_TIMEOUT_ENV,+ StoreLockToken,+ storePlacement,+} from "../../../src/core/brain/snapshot.ts";+import { restoreSnapshotWithRecoveryPoint } from "../../../src/core/brain/snapshot-gate.ts";+import { BRAIN_SNAPSHOT_REASON } from "../../../src/core/brain/types.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import { sha256Hex } from "../../../src/core/integrity/digest.ts";+import { indexVault } from "../../../src/core/search/indexer.ts";+import { acquireWriterLockSync } from "../../../src/core/search/store/writer-lock.ts";+import { resolveIndexPath } from "../../../src/core/search/paths.ts";+import { CLI_SPAWN_BUDGET_MS } from "../../helpers/cli-timeout.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { makeConfig } from "../../helpers/search-fixtures.ts";++// Tests here spawn processes; see tests/helpers/cli-timeout.ts.+setDefaultTimeout(CLI_SPAWN_BUDGET_MS);++const COVERED = { include: true, maxBytes: 1024 * 1024 * 1024 } as const;+const DREAM = BRAIN_SNAPSHOT_REASON.dream;+const RUN_ID = "dream-restore-order";++let root: string;+let vault: string;+let dbPath: string;+let originalPath: string | undefined;+let originalToolTimeout: string | undefined;++beforeEach(async () => {+ root = mkdtempSync(join(tmpdir(), "o2b-restore-order-"));+ vault = join(root, "vault");+ mkdirSync(vault);+ const configPath = join(root, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath });+ writeFileSync(+ join(brainDirs(vault).preferences, "pref-foo.md"),+ "---\nkind: brain-preference\n---\n\n## Principle\n\nseed\n",+ );++ dbPath = resolveIndexPath(vault, null);+ const notes = join(vault, "Notes");+ mkdirSync(notes, { recursive: true });+ writeFileSync(join(notes, "seed.md"), "# Seed\n\nA paragraph long enough to be indexed.\n");+ await indexVault(makeConfig({ vault, dbPath }));++ originalPath = process.env["PATH"];+ originalToolTimeout = process.env[SNAPSHOT_TOOL_TIMEOUT_ENV];+});++afterEach(() => {+ process.env["PATH"] = originalPath;+ if (originalToolTimeout === undefined) delete process.env[SNAPSHOT_TOOL_TIMEOUT_ENV];+ else process.env[SNAPSHOT_TOOL_TIMEOUT_ENV] = originalToolTimeout;+ // A test that made `Brain/` read-only must not leave the cleanup stuck.+ try {+ chmodSync(brainDirs(vault).brain, 0o755);+ } catch {+ // gone already+ }+ rmSync(root, { recursive: true, force: true });+});++/**+ * Take the covered snapshot, then move the live state on: a changed file,+ * a new file the archive never held, and a table only the live store has.+ */+function snapshotThenDiverge(): void {+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ diverge();+}++/** Move the live tree and store on from whatever the snapshot holds. */+function diverge(): void {+ const dirs = brainDirs(vault);+ writeFileSync(+ join(dirs.preferences, "pref-foo.md"),+ "---\nkind: brain-preference\n---\n\n## Principle\n\nchanged after the snapshot\n",+ );+ writeFileSync(join(dirs.brain, "post-snapshot-marker.md"), "# only in the live tree\n");+ const live = new Database(dbPath);+ live.exec("CREATE TABLE post_snapshot_marker (id INTEGER PRIMARY KEY)");+ live.close();+}++/** Everything under `dir`, as relative path -> content (or link target). */+function treeOf(dir: string): Record<string, string> {+ const out: Record<string, string> = {};+ const walk = (d: string): void => {+ for (const entry of readdirSync(d, { withFileTypes: true })) {+ const abs = join(d, entry.name);+ const rel = relative(dir, abs);+ if (entry.isSymbolicLink()) out[rel] = `-> ${readlinkSync(abs)}`;+ else if (entry.isDirectory()) {+ out[`${rel}/`] = "";+ walk(abs);+ } else out[rel] = readFileSync(abs, "utf8");+ }+ };+ walk(dir);+ return out;+}++/** The observable live state a failed restore must leave as it was. */+function liveState(): { brain: Record<string, string>; store: string; storeDir: string[] } {+ return {+ brain: treeOf(brainDirs(vault).brain),+ store: sha256Hex(readFileSync(dbPath)),+ storeDir: readdirSync(dirname(dbPath)).toSorted(),+ };+}++/** Rewrite the store archive and keep the manifest digest in step with it. */+function replaceStoreArchive(bytes: Buffer): void {+ writeFileSync(snapshotStorePath(vault, RUN_ID), bytes);+ const sidecar = manifestSidecarPath(vault, RUN_ID);+ const parsed = JSON.parse(readFileSync(sidecar, "utf8")) as {+ derived_store: Record<string, unknown>;+ };+ parsed.derived_store["archive_sha256"] = sha256Hex(bytes);+ writeFileSync(sidecar, JSON.stringify(parsed, null, 2) + "\n");+}++function markerTableExists(): boolean {+ const db = new Database(dbPath, { readonly: true });+ try {+ return (+ db+ .query<{ name: string }, [string]>(+ "SELECT name FROM sqlite_master WHERE type = 'table' AND name = ?",+ )+ .get("post_snapshot_marker") !== null+ );+ } finally {+ db.close();+ }+}++/** Run `fn` expecting a refusal, and prove nothing live changed. */+function expectUntouched(fn: () => unknown, message: RegExp): void {+ const before = liveState();+ let threw: unknown = null;+ try {+ fn();+ } catch (err) {+ threw = err;+ }+ expect(threw).toBeInstanceOf(BrainSnapshotError);+ expect((threw as Error).message).toMatch(message);+ expect(liveState()).toEqual(before);+ expect(markerTableExists()).toBe(true);+}++describe("a derived-store failure before the swap leaves the live state untouched", () => {+ test("a corrupt store payload (decompression fails)", () => {+ snapshotThenDiverge();+ // gzip magic so the restore inflates it in-process on every host,+ // followed by bytes no inflater accepts. The digest is updated, so this+ // is a payload that is wrong, not an archive that was swapped.+ replaceStoreArchive(Buffer.from([0x1f, 0x8b, 0x08, 0x00, 0xde, 0xad, 0xbe, 0xef, 0x00]));+ let recoveryPointTaken = false;+ expectUntouched(+ () =>+ restoreSnapshot(vault, RUN_ID, {+ beforeDiscard: () => {+ recoveryPointTaken = true;+ },+ }),+ /gzip decompress failed/u,+ );+ // The failure arrived before the recovery-point seam: nothing was+ // about to be discarded yet.+ expect(recoveryPointTaken).toBe(false);+ });++ test("a payload that decompresses but is not a sound database", () => {+ snapshotThenDiverge();+ replaceStoreArchive(gzipSync(Buffer.from("not a sqlite database at all\n")));+ expectUntouched(() => restoreSnapshot(vault, RUN_ID), /failed verification/u);+ });++ test("a store archive that does not match its recorded digest", () => {+ snapshotThenDiverge();+ writeFileSync(snapshotStorePath(vault, RUN_ID), gzipSync(Buffer.from("swapped")));+ expectUntouched(() => restoreSnapshot(vault, RUN_ID), /does not match the digest/u);+ });++ test("the store's writer lock held by another writer", () => {+ snapshotThenDiverge();+ const release = lockfile.lockSync(dbPath, { realpath: false });+ try {+ expectUntouched(() => restoreSnapshot(vault, RUN_ID), /writer lock/u);+ } finally {+ release();+ }+ });+});++const realZstd = Bun.which("zstd");+const realTar = Bun.which("tar");+const realSleep = Bun.which("sleep");++/**+ * A PATH holding the real `tar` and a stand-in `zstd` that runs the real+ * one for the Brain tar (staged as `brain.tar`) and runs `body` for any+ * other target, which in a restore is only the derived store.+ */+function storeOnlyZstd(body: string): void {+ const bin = join(root, "bin");+ mkdirSync(bin);+ symlinkSync(realTar!, join(bin, "tar"));+ const script = [+ "#!/bin/sh",+ 'case "$*" in',+ ` *brain.tar*) exec '${realZstd}' "$@" ;;`,+ "esac",+ body,+ "",+ ].join("\n");+ writeFileSync(join(bin, "zstd"), script);+ chmodSync(join(bin, "zstd"), 0o755);+ process.env["PATH"] = bin;+}++function expectZstdArchives(): void {+ // The stand-in is only reached for zstd archives; a gzip fallback here+ // would make the test pass without exercising the failure.+ for (const archive of [snapshotPath(vault, RUN_ID), snapshotStorePath(vault, RUN_ID)]) {+ const magic = readFileSync(archive).subarray(0, 4);+ expect([...magic]).toEqual([0x28, 0xb5, 0x2f, 0xfd]);+ }+}++describe.skipIf(+ process.platform === "win32" || realZstd === null || realTar === null || realSleep === null,+)("a failing zstd on the store archive", () => {+ test("exits non-zero", () => {+ snapshotThenDiverge();+ expectZstdArchives();+ storeOnlyZstd("exit 3");+ expectUntouched(() => restoreSnapshot(vault, RUN_ID), /zstd exited with status 3/u);+ });++ test("runs past the snapshot tool timeout", () => {+ snapshotThenDiverge();+ expectZstdArchives();+ // Absolute: the PATH here holds only `tar` and the stand-in.+ storeOnlyZstd(`exec '${realSleep}' 30`);+ process.env[SNAPSHOT_TOOL_TIMEOUT_ENV] = "1500";+ expectUntouched(() => restoreSnapshot(vault, RUN_ID), /zstd did not finish within 1500 ms/u);+ });+});++describe.skipIf(CHMOD_CANNOT_DENY)("a rename that fails during the swap", () => {+ test("is undone, and the error says so", () => {+ snapshotThenDiverge();+ // Live entries cannot be moved out of a read-only `Brain/`, while the+ // staging directory under `.snapshots/` stays writable: the restore+ // gets all the way to the swap and fails at its first rename.+ chmodSync(brainDirs(vault).brain, 0o555);+ expectUntouched(+ () => restoreSnapshot(vault, RUN_ID),+ /failed during the swap.*every Brain\/ rename was undone.*as they were/u,+ );+ });+});++/** An errno-shaped error, as a failing `rename(2)` or `unlink(2)` raises it. */+function fsError(code: string, path: string): Error {+ return Object.assign(new Error(`${code}: simulated by the test, ${path}`), { code });+}++/**+ * Run `fn` with `renameSync` failing wherever `failOn` returns true. The+ * real rename runs otherwise, so every other step of the restore is real.+ * `renameWithRetry` (`src/core/fs-atomic.ts`) calls the namespace's+ * `renameSync`, which is what the spy replaces.+ */+function withFailingRename<T>(failOn: (from: string, to: string) => boolean, fn: () => T): T {+ const realRename = fs.renameSync;+ const spy = spyOn(fs, "renameSync").mockImplementation((from, to) => {+ if (failOn(String(from), String(to))) throw fsError("EIO", String(to));+ realRename(from, to);+ });+ try {+ return fn();+ } finally {+ spy.mockRestore();+ }+}++/** True for a rename of a staged Brain/ entry into the live `Brain/`. */+function isMoveIn(from: string, to: string): boolean {+ return (+ from.includes(".restore-staging-") &&+ basename(dirname(from)) === "incoming" &&+ dirname(to) === brainDirs(vault).brain+ );+}++/** Run `fn` expecting a {@link BrainSnapshotError}; return its message. */+function refusal(fn: () => unknown): string {+ let threw: unknown = null;+ try {+ fn();+ } catch (err) {+ threw = err;+ }+ expect(threw).toBeInstanceOf(BrainSnapshotError);+ return (threw as Error).message;+}++describe("a failure in the middle of the swap", () => {+ test("a later move-in failing undoes the entries moved both ways, in reverse", () => {+ snapshotThenDiverge();+ let moveIns = 0;+ expectUntouched(+ () =>+ withFailingRename(+ (from, to) => isMoveIn(from, to) && ++moveIns === 2,+ () => restoreSnapshot(vault, RUN_ID),+ ),+ // Every live entry was out and one restored entry in when it failed.+ /(\d+) of \1 live Brain\/ top-level entries had been moved aside and 1 of \d+ restored ones moved in; every Brain\/ rename was undone, so Brain\/ and the derived store are as they were/u,+ );+ expect(moveIns).toBe(2);+ expect(readdirSync(brainDirs(vault).snapshots).some((n) => n.startsWith(".restore-"))).toBe(+ false,+ );+ });++ test("an undo that also fails keeps the staging directory and says Brain/ is mixed", () => {+ snapshotThenDiverge();+ const dirs = brainDirs(vault);+ let moveIns = 0;+ let firstIn = "";+ const message = refusal(() =>+ withFailingRename(+ (from, to) => {+ if (isMoveIn(from, to)) {+ moveIns += 1;+ if (moveIns === 1) firstIn = basename(to);+ return moveIns === 2;+ }+ // The undo of the one entry that made it in.+ return from === join(dirs.brain, firstIn) && basename(dirname(to)) === "incoming";+ },+ () => restoreSnapshot(vault, RUN_ID),+ ),+ );+ expect(message).toMatch(/undoing it also failed \(could not move the restored /u);+ expect(message).toMatch(/Brain\/ is a mix of the restored and the pre-restore tree/u);+ // The staging directory outlived the restore, with the pre-restore+ // entries put back and the stuck restored one still live.+ const staging = readdirSync(dirs.snapshots).filter((n) => n.startsWith(".restore-staging-"));+ expect(staging).toHaveLength(1);+ const kept = join(dirs.snapshots, staging[0]!);+ expect(message).toContain(join(kept, "outgoing"));+ expect(existsSync(join(kept, "outgoing"))).toBe(true);+ expect(existsSync(join(dirs.brain, firstIn))).toBe(true);+ expect(existsSync(join(dirs.brain, "post-snapshot-marker.md"))).toBe(true);+ });++ test("a failing store rename-back is reported as a torn store, with Brain/ undone", () => {+ snapshotThenDiverge();+ const before = liveState();+ const backup = `${dbPath}.bak`;+ const message = refusal(() =>+ withFailingRename(+ (from, to) => to === dbPath && (from.startsWith(`${dbPath}.restore-`) || from === backup),+ () => restoreSnapshot(vault, RUN_ID),+ ),+ );+ expect(message).toMatch(/the pre-restore store could not be renamed back either/u);+ expect(message).toContain(`is at ${backup}`);+ expect(message).toMatch(+ /Brain\/ is as it was before the restore, but the derived store is not/u,+ );+ expect(treeOf(brainDirs(vault).brain)).toEqual(before.brain);+ expect(existsSync(dbPath)).toBe(false);+ expect(sha256Hex(readFileSync(backup))).toBe(before.store);+ });++ test("a live store that cannot be moved aside fails with the store-swap context", () => {+ snapshotThenDiverge();+ expectUntouched(+ () =>+ withFailingRename(+ (from, to) => from === dbPath && to === `${dbPath}.bak`,+ () => restoreSnapshot(vault, RUN_ID),+ ),+ /derived store swap failed \(could not move the live store .* aside .*the live store was not touched\).*as they were/u,+ );+ });++ test("a -wal that cannot be removed after the store swap undoes the whole restore", () => {+ snapshotThenDiverge();+ // A log left beside the live store, held open the way a Windows+ // reader holds it: the removal fails on every attempt.+ const wal = `${dbPath}-wal`;+ writeFileSync(wal, "");+ const realUnlink = fs.unlinkSync;+ const spy = spyOn(fs, "unlinkSync").mockImplementation((path) => {+ if (String(path) === wal) throw fsError("EBUSY", wal);+ realUnlink(path);+ });+ try {+ expectUntouched(+ () => restoreSnapshot(vault, RUN_ID),+ /could not remove .*-wal .*the store swap was undone.*every Brain\/ rename was undone, so Brain\/ and the derived store are as they were/u,+ );+ } finally {+ spy.mockRestore();+ }+ expect(existsSync(wal)).toBe(true);+ });+});++describe("a restore staging directory left under Brain/", () => {+ test("is neither archived nor swapped by a later restore", () => {+ const dirs = brainDirs(vault);+ // What a restore killed mid-swap leaves when it staged under Brain/.+ const leftover = join(dirs.brain, ".restore-staging-killed-abc123");+ mkdirSync(join(leftover, "incoming"), { recursive: true });+ writeFileSync(join(leftover, "incoming", "half.md"), "half-written\n");++ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ expect(readFileSync(manifestSidecarPath(vault, RUN_ID), "utf8")).not.toContain(+ ".restore-staging-",+ );+ diverge();+ writeFileSync(join(leftover, "incoming", "half.md"), "changed after the snapshot\n");++ restoreSnapshot(vault, RUN_ID);++ // Left exactly where and as it was: not moved aside, not replaced by+ // an archived copy.+ expect(readFileSync(join(leftover, "incoming", "half.md"), "utf8")).toBe(+ "changed after the snapshot\n",+ );+ expect(readdirSync(leftover)).toEqual(["incoming"]);+ expect(markerTableExists()).toBe(false);+ });+});++describe("a successful restore", () => {+ test("restores the tree and the store, keeps the excluded entries, leaves no staging", () => {+ const dirs = brainDirs(vault);+ // Read before the snapshot: `createSnapshot` logs itself after the+ // archive is written, so the tree afterwards holds one entry more.+ const snapshotTree = treeOf(dirs.brain);+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ diverge();+ const artifacts = join(dirs.brain, BRAIN_ARTIFACTS_DIR);+ mkdirSync(artifacts, { recursive: true });+ writeFileSync(join(artifacts, "run.json"), "{}");+ const snapshotsBefore = readdirSync(dirs.snapshots).toSorted();++ let staged = null as { tree: boolean; store: boolean } | null;+ const result = restoreSnapshot(vault, RUN_ID, {+ // Both stagings exist by the time the recovery point is taken.+ beforeDiscard: () => {+ staged = {+ tree: readdirSync(dirs.snapshots).some((n) => n.startsWith(".restore-staging-")),+ store: readdirSync(dirname(dbPath)).some((n) =>+ n.startsWith(`${basename(dbPath)}.restore-`),+ ),+ };+ },+ });++ expect(staged).toEqual({ tree: true, store: true });+ expect(result.derived_store.replaced).toBe(true);+ expect(result.derived_store.path).toBe(dbPath);+ expect(markerTableExists()).toBe(false);+ expect(existsSync(`${dbPath}-wal`)).toBe(false);++ // Brain/ equals the snapshot, except the two excluded entries, which+ // are exactly what they were before the restore.+ const after = treeOf(dirs.brain);+ expect(withoutExcluded(after)).toEqual(withoutExcluded(snapshotTree));+ expect(readFileSync(join(artifacts, "run.json"), "utf8")).toBe("{}");+ expect(readdirSync(dirs.snapshots).toSorted()).toEqual(snapshotsBefore);+ expect(+ readdirSync(dirname(dbPath)).filter((n) => n.startsWith(`${basename(dbPath)}.restore`)),+ ).toEqual([]);+ expect(result.restored_files).toBe(+ Object.keys(withoutExcluded(snapshotTree)).filter((p) => !p.endsWith("/")).length,+ );+ });++ test.skipIf(process.platform === "win32")("keeps an in-tree relative symlink", () => {+ const dirs = brainDirs(vault);+ symlinkSync("pref-foo.md", join(dirs.preferences, "alias.md"));+ createSnapshot(vault, RUN_ID, { reason: DREAM });+ rmSync(join(dirs.preferences, "alias.md"));++ restoreSnapshot(vault, RUN_ID);++ const link = join(dirs.preferences, "alias.md");+ expect(lstatSync(link).isSymbolicLink()).toBe(true);+ expect(readlinkSync(link)).toBe("pref-foo.md");+ expect(readFileSync(link, "utf8")).toContain("seed");+ });++ test.skipIf(process.platform === "win32")(+ "keeps a top-level link to another top-level entry that is staged after it",+ () => {+ // `aa-link.md` sorts before `zz-target.md`, so it is staged first; the+ // staged link resolves only once its target is staged too.+ writeFileSync(join(vault, "Brain", "zz-target.md"), "target\n");+ symlinkSync("zz-target.md", join(vault, "Brain", "aa-link.md"));+ createSnapshot(vault, RUN_ID, { reason: DREAM });+ rmSync(join(vault, "Brain", "aa-link.md"));+ rmSync(join(vault, "Brain", "zz-target.md"));++ restoreSnapshot(vault, RUN_ID);++ const link = join(vault, "Brain", "aa-link.md");+ expect(lstatSync(link).isSymbolicLink()).toBe(true);+ expect(readFileSync(link, "utf8")).toBe("target\n");+ },+ );+});++/** A {@link treeOf} result without the entries a restore never touches. */+function withoutExcluded(tree: Record<string, string>): Record<string, string> {+ return Object.fromEntries(+ Object.entries(tree).filter(+ ([p]) => !p.startsWith(".snapshots") && !p.startsWith(BRAIN_ARTIFACTS_DIR),+ ),+ );+}++/** Turn on derived-store coverage in the vault's own policy, as an operator does. */+function coverStoreByPolicy(): void {+ atomicWriteFileSync(+ join(brainDirs(vault).brain, "_brain.yaml"),+ "schema_version: 1\nsnapshots:\n include_derived_store: true\n",+ );+}++/** Names of the tables in the store at `path`. */+function tablesOf(path: string): string[] {+ const db = new Database(path, { readonly: true });+ try {+ return db+ .query<{ name: string }, []>("SELECT name FROM sqlite_master WHERE type = 'table'")+ .all()+ .map((r) => r.name)+ .toSorted();+ } finally {+ db.close();+ }+}++type LockSync = typeof lockfile.lockSync;++/**+ * Run `fn` with every writer lock taken on the live store wrapped by+ * `wrapRelease`. Locks on other paths, and locks the test itself takes+ * while `bypass()` is true, are the real ones.+ */+function withStoreLockRelease<T>(+ wrapRelease: (real: () => void) => () => void,+ fn: () => T,+ bypass: () => boolean = () => false,+): T {+ const realLockSync: LockSync = lockfile.lockSync.bind(lockfile);+ const spy = spyOn(lockfile, "lockSync").mockImplementation(((path, opts) => {+ const release = realLockSync(path, opts);+ return String(path) === dbPath && !bypass() ? wrapRelease(release) : release;+ }) as LockSync);+ try {+ return fn();+ } finally {+ spy.mockRestore();+ }+}++describe("the gated restore holds the store lock from the recovery point through the swap", () => {+ test("a writer cannot commit between the recovery point and the swap", () => {+ coverStoreByPolicy();+ snapshotThenDiverge();++ // A writer waiting on the lock: the moment any holder releases it+ // before the store is swapped, it takes the lock and commits.+ let swapped = false;+ let inWriter = false;+ let lateCommits = 0;+ const writer = (): void => {+ inWriter = true;+ try {+ const release = acquireWriterLockSync(dbPath);+ try {+ const db = new Database(dbPath);+ db.exec(`CREATE TABLE late_commit_${lateCommits} (id INTEGER PRIMARY KEY)`);+ db.close();+ lateCommits++;+ } finally {+ release();+ }+ } finally {+ inWriter = false;+ }+ };++ // What the swap replaces, read at the swap itself, and whether a+ // writer could take the lock at that point.+ let replaced: string[] | null = null;+ let writerAtSwap: unknown = null;+ const realRename = fs.renameSync;+ const renameSpy = spyOn(fs, "renameSync").mockImplementation((from, to) => {+ if (String(from) === dbPath && String(to) === `${dbPath}.bak`) {+ replaced = tablesOf(dbPath);+ try {+ inWriter = true;+ acquireWriterLockSync(dbPath)();+ } catch (err) {+ writerAtSwap = err;+ } finally {+ inWriter = false;+ }+ swapped = true;+ }+ realRename(from, to);+ });+ let recoveryRunId: string;+ try {+ const result = withStoreLockRelease(+ (real) => () => {+ real();+ if (!swapped) writer();+ },+ () => restoreSnapshotWithRecoveryPoint(vault, RUN_ID),+ () => inWriter,+ );+ recoveryRunId = result.recoveryPoint.runId;+ } finally {+ renameSpy.mockRestore();+ }++ // The lock was never released before the swap, so no writer got in,+ // and one that tried at the swap was refused.+ expect(lateCommits).toBe(0);+ expect(String(writerAtSwap)).toMatch(/another writer holds the search index lock/u);+ expect(replaced).not.toBeNull();+ expect(replaced!).toContain("post_snapshot_marker");+ expect(markerTableExists()).toBe(false);++ // The recovery point holds every table the restore replaced.+ restoreSnapshot(vault, recoveryRunId);+ expect(tablesOf(dbPath)).toEqual(replaced!);+ });++ test("a lock release that fails after the swap committed says so and names the recovery point", () => {+ coverStoreByPolicy();+ snapshotThenDiverge();+ let threw: unknown = null;+ try {+ withStoreLockRelease(+ (real) => () => {+ real();+ throw new Error("simulated unlock failure");+ },+ () => restoreSnapshotWithRecoveryPoint(vault, RUN_ID),+ );+ } catch (err) {+ threw = err;+ }+ expect(threw).toBeInstanceOf(BrainSnapshotError);+ expect((threw as Error).message).toMatch(+ /the restore committed: Brain\/ and the derived store now hold snapshot dream-restore-order, but releasing the derived store's writer lock failed \(simulated unlock failure\).*the recovery point manual-\S+ \(.+\) holds the pre-restore state/u,+ );+ // It did commit: the live state is the snapshot's.+ expect(markerTableExists()).toBe(false);+ expect(existsSync(join(brainDirs(vault).brain, "post-snapshot-marker.md"))).toBe(false);+ });++ test("a lock release that fails after a failed swap does not mask the swap error", () => {+ snapshotThenDiverge();+ expectUntouched(+ () =>+ withStoreLockRelease(+ (real) => () => {+ real();+ throw new Error("simulated unlock failure");+ },+ () => withFailingRename(isMoveIn, () => restoreSnapshot(vault, RUN_ID)),+ ),+ /^snapshot\[dream-restore-order\]: restore failed during the swap: a Brain\/ rename failed.*as they were before the restore\nalso: releasing the derived store's writer lock failed \(simulated unlock failure\)$/su,+ );+ });+});++describe("a -wal removal failure with no live store before the restore", () => {+ test("does not claim a pre-restore store was put back", () => {+ snapshotThenDiverge();+ for (const suffix of ["", "-wal", "-shm"]) rmSync(`${dbPath}${suffix}`, { force: true });+ const wal = `${dbPath}-wal`;+ writeFileSync(wal, "");+ const realUnlink = fs.unlinkSync;+ const spy = spyOn(fs, "unlinkSync").mockImplementation((path) => {+ if (String(path) === wal) throw fsError("EBUSY", wal);+ realUnlink(path);+ });+ let threw: unknown = null;+ try {+ restoreSnapshot(vault, RUN_ID);+ } catch (err) {+ threw = err;+ } finally {+ spy.mockRestore();+ }+ expect(threw).toBeInstanceOf(BrainSnapshotError);+ const message = (threw as Error).message;+ expect(message).toMatch(/the store swap was undone, and no store was live before it/u);+ expect(message).not.toMatch(/back in place/u);+ expect(existsSync(dbPath)).toBe(false);+ });+});++/** A store at an override path, distinct from the configured one. */+function overrideStore(): { derivedStorePath: string; derivedStore: typeof COVERED } {+ const override = join(root, "override", "index.sqlite");+ mkdirSync(dirname(override), { recursive: true });+ const live = new Database(dbPath);+ live.exec(`VACUUM INTO '${override.replaceAll("'", "''")}'`);+ live.close();+ return { derivedStorePath: override, derivedStore: COVERED };+}++function addTable(path: string, name: string): void {+ const db = new Database(path);+ db.exec(`CREATE TABLE ${name} (id INTEGER PRIMARY KEY)`);+ db.close();+}++describe("the recovery point archives exactly the store the restore replaces", () => {+ test("with a derivedStorePath override, the recovery point holds the override store", () => {+ const store = overrideStore();+ createSnapshot(vault, RUN_ID, { reason: DREAM, ...store });+ addTable(store.derivedStorePath, "override_marker");++ const res = restoreSnapshotWithRecoveryPoint(vault, RUN_ID, store);++ expect(tablesOf(store.derivedStorePath)).not.toContain("override_marker");+ const recorded = JSON.parse(+ readFileSync(manifestSidecarPath(vault, res.recoveryPoint.runId), "utf8"),+ ) as { derived_store: { included: boolean; source_path: string } };+ expect(recorded.derived_store.included).toBe(true);+ expect(recorded.derived_store.source_path).toBe(store.derivedStorePath);++ // The discarded override store comes back from the recovery point.+ restoreSnapshot(vault, res.recoveryPoint.runId, store);+ expect(tablesOf(store.derivedStorePath)).toContain("override_marker");+ });++ test("a store the restore replaces is archived even when the policy leaves it out", () => {+ // The snapshot covered the store; the restore runs under the default+ // policy, which would leave the store out of an ordinary snapshot.+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ addTable(dbPath, "late_marker");++ const res = restoreSnapshotWithRecoveryPoint(vault, RUN_ID, {+ derivedStore: { include: false, maxBytes: COVERED.maxBytes },+ });++ expect(tablesOf(dbPath)).not.toContain("late_marker");+ const recorded = JSON.parse(+ readFileSync(manifestSidecarPath(vault, res.recoveryPoint.runId), "utf8"),+ ) as { derived_store: { included: boolean } };+ expect(recorded.derived_store.included).toBe(true);+ restoreSnapshot(vault, res.recoveryPoint.runId);+ expect(tablesOf(dbPath)).toContain("late_marker");+ });++ test("an absent live store does not block the restore that recreates it", () => {+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ for (const suffix of ["", "-wal", "-shm"]) rmSync(dbPath + suffix, { force: true });++ restoreSnapshotWithRecoveryPoint(vault, RUN_ID);++ expect(existsSync(dbPath)).toBe(true);+ expect(tablesOf(dbPath).length).toBeGreaterThan(0);+ });++ test("a condemned live store is healed by the restore, not preserved by refusing it", () => {+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ for (const suffix of ["-wal", "-shm"]) rmSync(dbPath + suffix, { force: true });+ writeFileSync(dbPath, "this is not a database");++ restoreSnapshotWithRecoveryPoint(vault, RUN_ID);++ expect(tablesOf(dbPath).length).toBeGreaterThan(0);+ });++ test("with coverage on by the vault's policy, an absent live store is recreated", () => {+ coverStoreByPolicy();+ createSnapshot(vault, RUN_ID, { reason: DREAM });+ const archived = tablesOf(dbPath);+ for (const suffix of ["", "-wal", "-shm"]) rmSync(dbPath + suffix, { force: true });++ const res = restoreSnapshotWithRecoveryPoint(vault, RUN_ID);++ expect(tablesOf(dbPath)).toEqual(archived);+ expect(recordedStore(res.recoveryPoint.runId)).toEqual({+ included: false,+ exclusion_reason: "not-requested",+ });+ });++ test("with coverage on by the vault's policy, a condemned live store is healed", () => {+ coverStoreByPolicy();+ createSnapshot(vault, RUN_ID, { reason: DREAM });+ const archived = tablesOf(dbPath);+ for (const suffix of ["-wal", "-shm"]) rmSync(dbPath + suffix, { force: true });+ writeFileSync(dbPath, "this is not a database");++ const res = restoreSnapshotWithRecoveryPoint(vault, RUN_ID);++ expect(tablesOf(dbPath)).toEqual(archived);+ expect(recordedStore(res.recoveryPoint.runId)).toEqual({+ included: false,+ exclusion_reason: "not-requested",+ });+ });++ test("a recovery point that left the replaced store out refuses the swap", () => {+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ addTable(dbPath, "late_marker");+ const storeBefore = sha256Hex(readFileSync(dbPath));++ expect(() =>+ restoreSnapshot(vault, RUN_ID, {+ beforeDiscard: ({ heldStoreLock }) => {+ const point = createSnapshot(vault, "manual-no-store", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ derivedStore: { include: false, maxBytes: COVERED.maxBytes },+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ });+ return { runId: "manual-no-store", path: point.path };+ },+ }),+ ).toThrow(/did not include the derived store/);+ expect(sha256Hex(readFileSync(dbPath))).toBe(storeBefore);+ expect(tablesOf(dbPath)).toContain("late_marker");+ });++ test("a recovery point of another store refuses the swap before anything is touched", () => {+ const store = overrideStore();+ createSnapshot(vault, RUN_ID, { reason: DREAM, ...store });+ addTable(store.derivedStorePath, "override_marker");+ // Read after the recovery point lands: its log lines are the only+ // change to the tree the refused restore may leave.+ let brainBefore: Record<string, string> = {};++ let threw: unknown = null;+ try {+ restoreSnapshot(vault, RUN_ID, {+ ...store,+ beforeDiscard: () => {+ // Resolves its own source: the configured store, not the override.+ const point = createSnapshot(vault, "manual-wrong-store", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ derivedStore: COVERED,+ });+ brainBefore = treeOf(brainDirs(vault).brain);+ return { runId: "manual-wrong-store", path: point.path };+ },+ });+ } catch (err) {+ threw = err;+ }+ expect(threw).toBeInstanceOf(BrainSnapshotError);+ expect((threw as Error).message).toMatch(+ /recorded the derived store at .+, but this restore replaces .+override.+refusing before the Brain tree is touched/u,+ );+ expect(tablesOf(store.derivedStorePath)).toContain("override_marker");+ expect(withoutExcluded(treeOf(brainDirs(vault).brain))).toEqual(withoutExcluded(brainBefore));+ });+});++describe("createSnapshot skips its own store lock only for the restore's live token on that store", () => {+ const LOCKED = /another writer holds the search index lock/u;++ test("the restore's token is used; a path string, a look-alike or a hand-made token is not", () => {+ coverStoreByPolicy();+ snapshotThenDiverge();+ let seen: { storeTarget: string | null; token: StoreLockToken | null } | null = null;++ const res = restoreSnapshot(vault, RUN_ID, {+ beforeDiscard: ({ storeTarget, heldStoreLock }) => {+ seen = { storeTarget, token: heldStoreLock };+ // The old hint: a string naming the locked path. The restore still+ // holds the lock, so the snapshot's own acquire refuses.+ expect(() =>+ createSnapshot(vault, "dream-string-hint", {+ reason: DREAM,+ heldStoreLock: dbPath as unknown as StoreLockToken,+ }),+ ).toThrow(LOCKED);+ // Objects that look like a token without being minted by the restore.+ expect(() =>+ createSnapshot(vault, "dream-look-alike", {+ reason: DREAM,+ heldStoreLock: { path: dbPath } as unknown as StoreLockToken,+ }),+ ).toThrow(LOCKED);+ expect(() =>+ createSnapshot(vault, "dream-prototype-forgery", {+ reason: DREAM,+ heldStoreLock: Object.create(StoreLockToken.prototype) as StoreLockToken,+ }),+ ).toThrow(LOCKED);+ // A token cannot be constructed without the module-private mint.+ expect(() => new StoreLockToken(Symbol("store-lock-token-mint"), dbPath)).toThrow(+ TypeError,+ );++ const point = createSnapshot(vault, "manual-under-token", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ });+ expect(point.derived_store.included).toBe(true);+ return { runId: "manual-under-token", path: point.path };+ },+ });++ expect(res.derived_store.replaced).toBe(true);+ expect(seen).not.toBeNull();+ expect(seen!.storeTarget).toBe(dbPath);+ expect(seen!.token).toBeInstanceOf(StoreLockToken);+ expect(seen!.token!.path).toBe(dbPath);+ expect(markerTableExists()).toBe(false);+ });++ test("a token kept past the restore no longer skips the lock", () => {+ coverStoreByPolicy();+ snapshotThenDiverge();+ let kept: StoreLockToken | null = null;+ restoreSnapshot(vault, RUN_ID, {+ beforeDiscard: ({ heldStoreLock }) => {+ kept = heldStoreLock;+ const point = createSnapshot(vault, "manual-before-keep", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ });+ return { runId: "manual-before-keep", path: point.path };+ },+ });+ expect(kept).toBeInstanceOf(StoreLockToken);++ // Another writer holds the lock now; the released token must not+ // stand in for it.+ const release = acquireWriterLockSync(dbPath);+ try {+ expect(() =>+ createSnapshot(vault, "dream-released-token", { reason: DREAM, heldStoreLock: kept! }),+ ).toThrow(LOCKED);+ } finally {+ release();+ }+ });++ test("a token whose path getter is shadowed does not skip another store's lock", () => {+ const store = overrideStore();+ createSnapshot(vault, RUN_ID, { reason: DREAM, ...store });++ restoreSnapshot(vault, RUN_ID, {+ ...store,+ beforeDiscard: ({ heldStoreLock }) => {+ expect(heldStoreLock).not.toBeNull();+ // Redirect the genuine token to the configured store, which another+ // writer holds: the module reads the path it minted, not the getter.+ Object.defineProperty(heldStoreLock, "path", { value: dbPath });+ const release = acquireWriterLockSync(dbPath);+ try {+ expect(() =>+ createSnapshot(vault, "dream-shadowed-token", {+ reason: DREAM,+ derivedStore: COVERED,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ }),+ ).toThrow(LOCKED);+ } finally {+ release();+ }+ const point = createSnapshot(vault, "manual-shadow-point", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ ...store,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ });+ return { runId: "manual-shadow-point", path: point.path };+ },+ });+ });++ test("a live token on another store does not skip the lock of this one", () => {+ const store = overrideStore();+ createSnapshot(vault, RUN_ID, { reason: DREAM, ...store });+ addTable(store.derivedStorePath, "override_marker");++ restoreSnapshot(vault, RUN_ID, {+ ...store,+ beforeDiscard: ({ heldStoreLock }) => {+ expect(heldStoreLock?.path).toBe(store.derivedStorePath);+ // The configured store is held by another writer; the restore's+ // live token names the override store, so it does not count here.+ const release = acquireWriterLockSync(dbPath);+ try {+ expect(() =>+ createSnapshot(vault, "dream-other-store-token", {+ reason: DREAM,+ derivedStore: COVERED,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ }),+ ).toThrow(LOCKED);+ } finally {+ release();+ }+ const point = createSnapshot(vault, "manual-override-point", {+ reason: BRAIN_SNAPSHOT_REASON.manual,+ ...store,+ ...(heldStoreLock !== null ? { heldStoreLock } : {}),+ });+ return { runId: "manual-override-point", path: point.path };+ },+ });+ expect(tablesOf(store.derivedStorePath)).not.toContain("override_marker");+ });+});++/** The derived-store record a snapshot's manifest carries. */+function recordedStore(runId: string): { included: boolean; exclusion_reason: string | null } {+ const { derived_store } = JSON.parse(readFileSync(manifestSidecarPath(vault, runId), "utf8")) as {+ derived_store: { included: boolean; exclusion_reason: string | null };+ };+ return { included: derived_store.included, exclusion_reason: derived_store.exclusion_reason };+}++/** A copy of the configured store at `path`, as a `derivedStorePath` override. */+function storeAt(path: string): string {+ mkdirSync(dirname(path), { recursive: true });+ const live = new Database(dbPath);+ live.exec(`VACUUM INTO '${path.replaceAll("'", "''")}'`);+ live.close();+ return path;+}++describe("a derived store that lives inside Brain/", () => {+ test("directly under Brain/, it is restored by the store swap and kept out of the tree swap", () => {+ const brain = brainDirs(vault).brain;+ const derivedStorePath = storeAt(join(brain, "index.sqlite"));+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStorePath, derivedStore: COVERED });+ const archived = tablesOf(derivedStorePath);+ writeFileSync(join(brain, "post-snapshot-marker.md"), "# only in the live tree\n");+ addTable(derivedStorePath, "late_marker");++ const res = restoreSnapshotWithRecoveryPoint(vault, RUN_ID, { derivedStorePath });++ expect(res.derived_store.replaced).toBe(true);+ expect(tablesOf(derivedStorePath)).toEqual(archived);+ expect(existsSync(join(brain, "post-snapshot-marker.md"))).toBe(false);+ expect(readdirSync(brain).filter((e) => e.startsWith("index.sqlite.restore"))).toEqual([]);+ // The store family is not counted as restored Brain content.+ const restored = Object.keys(treeOf(brain)).filter(+ (p) =>+ !p.endsWith("/") &&+ !p.startsWith(".snapshots") &&+ !p.startsWith(BRAIN_ARTIFACTS_DIR) &&+ !p.startsWith("index.sqlite"),+ );+ expect(res.restored_files).toBe(restored.length);++ // The recovery point holds the store the restore discarded.+ restoreSnapshot(vault, res.recoveryPoint.runId, { derivedStorePath });+ expect(tablesOf(derivedStorePath)).toContain("late_marker");+ });++ test("directly under Brain/ and not covered, the Brain archive's copy is not restored over it", () => {+ const brain = brainDirs(vault).brain;+ const derivedStorePath = storeAt(join(brain, "index.sqlite"));+ // The Brain archive still holds a raw copy of the store as a top-level+ // entry; the store record says the store was left out.+ createSnapshot(vault, RUN_ID, {+ reason: DREAM,+ derivedStorePath,+ derivedStore: { include: false, maxBytes: COVERED.maxBytes },+ });+ addTable(derivedStorePath, "late_marker");+ const storeBefore = sha256Hex(readFileSync(derivedStorePath));++ const res = restoreSnapshot(vault, RUN_ID, { derivedStorePath });++ expect(res.derived_store.replaced).toBe(false);+ expect(sha256Hex(readFileSync(derivedStorePath))).toBe(storeBefore);+ expect(tablesOf(derivedStorePath)).toContain("late_marker");+ });++ test("directly under Brain/, entries that only share the store's name prefix are restored", () => {+ const brain = brainDirs(vault).brain;+ const derivedStorePath = storeAt(join(brain, "index.sqlite"));+ writeFileSync(join(brain, "index.sqlite-notes.md"), "# snapshot notes\n");+ mkdirSync(join(brain, "index.sqlite-assets"));+ writeFileSync(join(brain, "index.sqlite-assets", "a.md"), "# snapshot asset\n");+ writeFileSync(join(brain, "index.sqlite.notes.md"), "# only in the snapshot\n");+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStorePath, derivedStore: COVERED });+ writeFileSync(join(brain, "index.sqlite-notes.md"), "# changed after the snapshot\n");+ writeFileSync(join(brain, "index.sqlite-assets", "b.md"), "# added after the snapshot\n");+ rmSync(join(brain, "index.sqlite.notes.md"));++ const res = restoreSnapshot(vault, RUN_ID, { derivedStorePath });++ expect(readFileSync(join(brain, "index.sqlite-notes.md"), "utf8")).toBe("# snapshot notes\n");+ expect(readdirSync(join(brain, "index.sqlite-assets"))).toEqual(["a.md"]);+ expect(readFileSync(join(brain, "index.sqlite.notes.md"), "utf8")).toBe(+ "# only in the snapshot\n",+ );+ // They count as restored Brain content; the store's own files (the+ // store, its journals, the outgoing store parked at `.bak`) do not.+ const restored = Object.keys(treeOf(brain)).filter(+ (p) =>+ !p.endsWith("/") &&+ !p.startsWith(".snapshots") &&+ !p.startsWith(BRAIN_ARTIFACTS_DIR) &&+ !/^index\.sqlite(\.bak)?(-wal|-shm|-journal)?$/u.test(p),+ );+ expect(res.restored_files).toBe(restored.length);+ });++ test("a store path that is the Brain/ directory itself is refused before anything is touched", () => {+ const brain = brainDirs(vault).brain;+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED });+ writeFileSync(join(brain, "post-snapshot-marker.md"), "# only in the live tree\n");+ const brainBefore = treeOf(brain);+ const snapshotsBefore = readdirSync(brainDirs(vault).snapshots).toSorted();++ let threw: unknown = null;+ try {+ restoreSnapshotWithRecoveryPoint(vault, RUN_ID, { derivedStorePath: brain });+ } catch (err) {+ threw = err;+ }++ expect(threw).toBeInstanceOf(BrainSnapshotError);+ expect((threw as Error).message).toMatch(+ /the derived store path .+ is the Brain\/ directory itself; refusing before anything is touched/u,+ );+ expect(treeOf(brain)).toEqual(brainBefore);+ expect(readdirSync(brainDirs(vault).snapshots).toSorted()).toEqual(snapshotsBefore);+ });++ test("nested inside a Brain/ directory, the restore refuses before anything is touched", () => {+ const brain = brainDirs(vault).brain;+ const derivedStorePath = storeAt(join(brain, "data", "index.sqlite"));+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStorePath, derivedStore: COVERED });+ writeFileSync(join(brain, "post-snapshot-marker.md"), "# only in the live tree\n");+ addTable(derivedStorePath, "late_marker");+ const brainBefore = treeOf(brain);+ const snapshotsBefore = readdirSync(brainDirs(vault).snapshots).toSorted();++ let threw: unknown = null;+ try {+ restoreSnapshotWithRecoveryPoint(vault, RUN_ID, { derivedStorePath });+ } catch (err) {+ threw = err;+ }++ expect(threw).toBeInstanceOf(BrainSnapshotError);+ expect((threw as Error).message).toMatch(+ /the derived store .+ is inside Brain\/data, which a restore replaces as a whole; refusing before anything is touched/u,+ );+ expect(treeOf(brain)).toEqual(brainBefore);+ expect(readdirSync(brainDirs(vault).snapshots).toSorted()).toEqual(snapshotsBefore);+ expect(tablesOf(derivedStorePath)).toContain("late_marker");+ });+});++describe("the store family kept out of the tree swap is a closed list", () => {+ test("the store's own artifacts match; other names sharing its prefix do not", () => {+ const family = [+ "index.sqlite",+ "index.sqlite-wal",+ "index.sqlite-shm",+ "index.sqlite-journal",+ "index.sqlite.new",+ "index.sqlite.new-wal",+ "index.sqlite.bak",+ "index.sqlite.bak-shm",+ "index.sqlite.old",+ "index.sqlite.restore-4242-0a1b2c3d",+ "index.sqlite.restore-4242-0a1b2c3d-wal",+ ];+ for (const name of family)+ expect(isStoreFamilyEntry(name, "index.sqlite", false, "linux")).toBe(true);+ expect(isStoreFamilyEntry("index.sqlite.lock", "index.sqlite", true, "linux")).toBe(true);++ const content = [+ "index.sqlite-notes.md",+ "index.sqlite.notes.md",+ "index.sqlite.old-notes",+ "index.sqlite-wal-shm",+ "index.sqlite.restore-notes",+ "index.sqlite.lock",+ "index.sqlite.takeover",+ "index.sqlite.1234.tmp",+ ];+ for (const name of content) {+ expect(isStoreFamilyEntry(name, "index.sqlite", false, "linux")).toBe(false);+ }+ // Directories are family only as the writer lock.+ for (const name of ["index.sqlite", "index.sqlite-assets", "index.sqlite.bak"]) {+ expect(isStoreFamilyEntry(name, "index.sqlite", true, "linux")).toBe(false);+ }+ });++ test("names compare without case only where the platform folds case", () => {+ for (const platform of ["darwin", "win32"] as const) {+ expect(isStoreFamilyEntry("index.sqlite-WAL", "Index.sqlite", false, platform)).toBe(true);+ expect(isStoreFamilyEntry("INDEX.SQLITE.LOCK", "index.sqlite", true, platform)).toBe(true);+ expect(isStoreFamilyEntry("index.SQLITE-notes.md", "Index.sqlite", false, platform)).toBe(+ false,+ );+ }+ expect(isStoreFamilyEntry("index.sqlite-wal", "Index.sqlite", false, "linux")).toBe(false);+ });++ test("an absent store takes the spelling of the entry beside it where case folds", () => {+ const brain = brainDirs(vault).brain;+ writeFileSync(join(brain, "index.sqlite"), "");+ const configured = join(brain, "Index.sqlite");++ // Both hold on any host: only the ancestors go through the native+ // realpath, and the last component is spelled by the platform the call+ // names, so a case-folding filesystem under the test does not rewrite+ // the `linux` answer to the on-disk `index.sqlite`.++ expect(storePlacement(brain, configured, "darwin")).toEqual({+ kind: "top",+ base: "index.sqlite",+ });+ expect(storePlacement(brain, configured, "linux")).toEqual({+ kind: "top",+ base: "Index.sqlite",+ });+ });++ test("Brain/ itself, an entry under it in another case, and paths outside are told apart", () => {+ const brain = brainDirs(vault).brain;+ const upper = join(dirname(brain), basename(brain).toUpperCase());++ expect(storePlacement(brain, brain, "linux")).toEqual({ kind: "brain" });+ expect(storePlacement(brain, upper, "darwin")).toEqual({ kind: "brain" });+ expect(storePlacement(brain, join(upper, "index.sqlite"), "darwin")).toEqual({+ kind: "top",+ base: "index.sqlite",+ });+ expect(storePlacement(brain, join(root, "index.sqlite"), "darwin")).toEqual({+ kind: "outside",+ });+ });++ // Skipped where the filesystem under the test folds case (NTFS, default+ // APFS): there `BRAIN` IS `Brain/`, and the native realpath answers with+ // the on-disk spelling whatever platform the call names, so the+ // case-sensitive comparison has two equal paths to compare. The+ // platform-independent cases above still run on those hosts.+ test.skipIf(filesystemFoldsCase(tmpdir()))(+ "on a case-sensitive filesystem, Brain/ in another case is another directory",+ () => {+ const brain = brainDirs(vault).brain;+ const upper = join(dirname(brain), basename(brain).toUpperCase());+ expect(storePlacement(brain, join(upper, "index.sqlite"), "linux")).toEqual({+ kind: "outside",+ });+ },+ );+});++/**+ * True when the filesystem under `dir` folds case, so `CASE-PROBE` names+ * the file created as `case-probe`.+ */+function filesystemFoldsCase(dir: string): boolean {+ const probeDir = mkdtempSync(join(dir, "o2b-case-probe-"));+ try {+ writeFileSync(join(probeDir, "case-probe"), "");+ return existsSync(join(probeDir, "CASE-PROBE"));+ } finally {+ rmSync(probeDir, { recursive: true, force: true });+ }+}++describe("the store target is one directory entry for the placement and the swap", () => {+ // Symlinks need elevated rights on Windows; the other suites building+ // symlink fixtures skip there for the same reason.+ test.skipIf(process.platform === "win32")(+ "a live Brain/index.sqlite linking to a database outside Brain/ is restored at the link",+ () => {+ const brain = brainDirs(vault).brain;+ const external = storeAt(join(root, "external", "index.sqlite"));+ // The snapshot predates the link: its store sat outside Brain/.+ createSnapshot(vault, RUN_ID, {+ reason: DREAM,+ derivedStorePath: external,+ derivedStore: COVERED,+ });+ const archived = tablesOf(external);+ addTable(external, "late_marker");+ const link = join(brain, "index.sqlite");+ symlinkSync(external, link);++ // The entry the swap renames is what is classified.+ expect(storePlacement(brain, link)).toEqual({ kind: "top", base: "index.sqlite" });+ // A link to Brain/ itself is still Brain/.+ symlinkSync(brain, join(root, "brain-link"));+ expect(storePlacement(brain, join(root, "brain-link"))).toEqual({ kind: "brain" });++ const res = restoreSnapshot(vault, RUN_ID, { derivedStorePath: link });++ expect(res.derived_store.replaced).toBe(true);+ // The restored database replaced the link; the link went aside with+ // the outgoing store's name, and the database it pointed to is as+ // it was.+ expect(lstatSync(link).isSymbolicLink()).toBe(false);+ expect(tablesOf(link)).toEqual(archived);+ expect(readlinkSync(`${link}.bak`)).toBe(external);+ expect(tablesOf(external)).toContain("late_marker");+ expect(readdirSync(brain).filter((e) => e.startsWith("index.sqlite.restore"))).toEqual([]);+ },+ );+});++/**+ * {@link expectUntouched} without opening the store: SQLite would take a+ * stand-in journal beside it for a hot one and try to roll it back.+ */+function expectUntouchedOnDisk(fn: () => unknown, message: RegExp): void {+ const before = liveState();+ expect(refusal(fn)).toMatch(message);+ expect(liveState()).toEqual(before);+}++describe("the outgoing store's rollback journal moves with it", () => {+ /**+ * Stand-in for a hot rollback journal: the swap moves the file by name+ * and never reads it, so its bytes only need to be recognisable.+ */+ const HOT_JOURNAL = "hot rollback journal of the pre-restore store\n";++ test("a store outside Brain/: the journal does not stay beside the replacement", () => {+ snapshotThenDiverge();+ const journal = `${dbPath}-journal`;+ writeFileSync(journal, HOT_JOURNAL);++ restoreSnapshot(vault, RUN_ID);++ expect(markerTableExists()).toBe(false);+ expect(existsSync(journal)).toBe(false);+ expect(readFileSync(`${dbPath}.bak-journal`, "utf8")).toBe(HOT_JOURNAL);+ });++ test("a store directly under Brain/: the journal does not stay beside the replacement", () => {+ const brain = brainDirs(vault).brain;+ const derivedStorePath = storeAt(join(brain, "index.sqlite"));+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStorePath, derivedStore: COVERED });+ const archived = tablesOf(derivedStorePath);+ addTable(derivedStorePath, "late_marker");+ writeFileSync(`${derivedStorePath}-journal`, HOT_JOURNAL);++ restoreSnapshot(vault, RUN_ID, { derivedStorePath });++ expect(tablesOf(derivedStorePath)).toEqual(archived);+ expect(existsSync(`${derivedStorePath}-journal`)).toBe(false);+ expect(readFileSync(`${derivedStorePath}.bak-journal`, "utf8")).toBe(HOT_JOURNAL);+ });++ test("a journal with no live store beside it does not meet the replacement either", () => {+ snapshotThenDiverge();+ rmSync(dbPath);+ const journal = `${dbPath}-journal`;+ writeFileSync(journal, HOT_JOURNAL);++ restoreSnapshot(vault, RUN_ID);++ expect(existsSync(dbPath)).toBe(true);+ expect(existsSync(journal)).toBe(false);+ expect(readFileSync(`${dbPath}.bak-journal`, "utf8")).toBe(HOT_JOURNAL);+ });++ test("an undo puts the journal back beside the pre-restore store", () => {+ snapshotThenDiverge();+ writeFileSync(`${dbPath}-journal`, HOT_JOURNAL);+ expectUntouchedOnDisk(+ () =>+ withFailingRename(+ (from, to) => to === dbPath && from.startsWith(`${dbPath}.restore-`),+ () => restoreSnapshot(vault, RUN_ID),+ ),+ /derived store swap failed.*as they were/u,+ );+ expect(readFileSync(`${dbPath}-journal`, "utf8")).toBe(HOT_JOURNAL);+ expect(existsSync(`${dbPath}.bak-journal`)).toBe(false);+ });++ test("a journal that cannot be moved aside undoes the whole restore", () => {+ snapshotThenDiverge();+ const journal = `${dbPath}-journal`;+ writeFileSync(journal, HOT_JOURNAL);+ expectUntouchedOnDisk(+ () =>+ withFailingRename(+ (from) => from === journal,+ () => restoreSnapshot(vault, RUN_ID),+ ),+ /could not move the rollback journal .*the restored store must not open beside it.*every Brain\/ rename was undone, so Brain\/ and the derived store are as they were/u,+ );+ expect(readFileSync(journal, "utf8")).toBe(HOT_JOURNAL);+ });++ test("a journal that cannot be put back is reported as a torn store, with Brain/ undone", () => {+ snapshotThenDiverge();+ const journal = `${dbPath}-journal`;+ const outgoingJournal = `${dbPath}.bak-journal`;+ writeFileSync(journal, HOT_JOURNAL);+ const before = liveState();+ const message = refusal(() =>+ withFailingRename(+ (from, to) =>+ (to === dbPath && from.startsWith(`${dbPath}.restore-`)) ||+ (from === outgoingJournal && to === journal),+ () => restoreSnapshot(vault, RUN_ID),+ ),+ );+ expect(message).toMatch(+ /the rollback journal of the pre-restore store could not be renamed back/u,+ );+ expect(message).toContain(`is at ${outgoingJournal}`);+ expect(message).toMatch(+ /Brain\/ is as it was before the restore, but the derived store is not/u,+ );+ expect(treeOf(brainDirs(vault).brain)).toEqual(before.brain);+ expect(readFileSync(outgoingJournal, "utf8")).toBe(HOT_JOURNAL);+ expect(sha256Hex(readFileSync(`${dbPath}.bak`))).toBe(before.store);+ });+});diff --git a/tests/core/store/fixtures/append-worker.ts b/tests/core/store/fixtures/append-worker.tsindex ee6e7adc..34ec86db 100644--- a/tests/core/store/fixtures/append-worker.ts+++ b/tests/core/store/fixtures/append-worker.ts@@ -8,7 +8,8 @@* for the go file the parent writes when all are ready, so they append at* the same time rather than one after another as each process happens to* start, however slowly the runner starts them. A go file that never- * comes ends the worker after 30 s instead of leaving it behind.+ * comes ends the worker after 30 s instead of leaving it behind. A store+ * lock refusal (ELOCKED) is retried within a bound; see the append loop.** argv: <root> <device> <stateHome> <worker> <count> <ready dir> <go file>*/@@ -48,12 +49,33 @@ while (!existsSync(goFile)) {}Bun.sleepSync(1);}+// Six workers each firing their appends back to back is far more contention+// than the store lock's wait budget is sized for (one append per agent+// action, not a tight loop), and on a slow Windows runner a worker could+// lose every round past it with `store lock busy` (ELOCKED). That refusal+// is documented as retryable ("retry once the other writer finishes"), and+// it arrives before the record's line is written: the clock tick it may+// have spent is never reused and a spilled blob is content-addressed, so a+// retried append writes the same blob name and one new line. The worker+// therefore retries the refused append with a short randomized backoff,+// within a bound; what the test proves is the shard under concurrency+// (every record present once, unique ids and clocks, one blob, interleaving).+const RETRY_UNTIL_MS = 30_000;for (let i = 0; i < Number(countText); i++) {- appendRecord(store, {- partition: "shared",- stream: "events",- kind: "feedback",- agent: `worker-${worker}`,- body: i % 3 === 0 ? { large } : { worker, i },- });+ const giveUpAt = Date.now() + RETRY_UNTIL_MS;+ for (;;) {+ try {+ appendRecord(store, {+ partition: "shared",+ stream: "events",+ kind: "feedback",+ agent: `worker-${worker}`,+ body: i % 3 === 0 ? { large } : { worker, i },+ });+ break;+ } catch (err) {+ if ((err as NodeJS.ErrnoException).code !== "ELOCKED" || Date.now() > giveUpAt) throw err;+ Bun.sleepSync(10 + Math.floor(Math.random() * 20));+ }+ }}