Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Pre-`dream` snapshot, rotation, and rollback support.
*
* The contract (design doc §7.4 "Pre-run snapshot" + §15 Step 9a):
*
* - Before any state-changing operation in a `dream` run we write
* `Brain/.snapshots/<run_id>.tar.zst` containing the entire
* `Brain/` tree **excluding** {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES}.
* Including the snapshots dir would either explode the archive or
* racy-clobber an in-progress write; including the artifact cache
* would hash TTL'd tool output into a drift gate that no rollback
* would ever act on.
*
* - Retention is enforced by `pruneSnapshots`: keep the
* `snapshots.retention_count` newest files, delete the rest.
*
* - `restoreSnapshot` extracts the archive over `Brain/`. Critical
* constraint: the restore must NOT touch `Brain/.snapshots/`,
* otherwise rolling back to an older state could destroy newer
* snapshots (and with them the user's only path forward again).
* We achieve this by extracting into a sibling temp directory,
* verifying the contents, then replacing every top-level entry
* under `Brain/` *except* `.snapshots/`.
*
* - Tooling: we shell out to system `tar` (GNU tar on Linux, bsdtar
* on macOS and in `System32\tar.exe` on Windows 10+) and use `zstd`
* when it is on PATH. Without `zstd` the archive is gzip-compressed
* in-process through `node:zlib`, so no `gzip` binary is needed -
* native Windows has none. A missing `tar` throws
* {@link BrainSnapshotToolingMissingError} with an actionable
* message.
*
* No external dependencies. Everything is `node:child_process` +
* `node:fs` + `node:zlib`, so the cost is one subprocess per archive
* operation.
*
* ## On a switched machine
*
* Once this machine is switched to the brain store (design D14 item 9),
* the store needs no snapshot: its shards are append-only with record
* sums and partition digests. What a snapshot covers then is this
* machine's device-local state, which keeps the same `Brain/` layout
* under its own root (`state/machine-state.ts`). Every public entry point
* below maps its `vault` argument to that root first, so the archive is
* of `<device state>/migration/local/Brain/`, it lives in that tree's
* `.snapshots/`, and a restore replaces that tree. Nothing under the vault
* is read or written. An unswitched machine maps to the vault itself.
*
* ## Derived-store coverage, and why it is off by default
*
* The derived SQLite store (`<vault>/.open-second-brain/brain.sqlite`)
* is a sibling of `Brain/`, and until now it was in no snapshot, no
* manifest and no rollback - with nothing saying so, which made a
* complete-looking pre-restore diff while the embeddings stayed at
* whatever the live store happened to hold.
*
* What coverage buys is SPEND, not information. Feedback, activation and
* tuning are replayable JSON folds inside `Brain/`; only the embeddings
* and a tier baseline are database-only, so a lost store costs an
* embedding bill and a reindex, never a fact. Against that: retention
* keeps ten archives, and `.snapshots/` sits inside a vault replicated
* peer-to-peer, so every retained copy is pushed to every device. Ten
* multiples of a store on every peer is a real cost to pay for a
* regenerable artifact.
*
* Hence the three-part answer. Coverage is opt-in. The ceiling REFUSES
* rather than truncating, because half a database is not a recovery
* point. And the live store size is recorded in every manifest whether
* or not it was included, so an operator who has never enabled coverage
* can still read what enabling it would cost.
*
* ## Every recovery point says why it exists
*
* `createSnapshot` requires a {@link BrainSnapshotReason}. It goes into
* the manifest sidecar as an additive key at the existing schema version,
* and into the Brain event log as one `snapshot` event carrying the run
* id, the reason and the archive size — the counterpart the log has been
* missing since `rollback` shipped, which recorded the restore while the
* point it restores to left no trace but a filename.
*
* The reason is also the run-id prefix at every call site, which is
* exactly why the READ path must not parse it back out: recovering the
* field from the filename would look right on almost every archive this
* project writes and would invent provenance for one it did not.
*/
import { Database } from "bun:sqlite";
import { spawnSync, type SpawnSyncOptions, type SpawnSyncReturns } from "node:child_process";
import { randomBytes } from "node:crypto";
import { gunzipSync, gzipSync } from "node:zlib";
import {
closeSync,
cpSync,
existsSync,
linkSync,
lstatSync,
mkdirSync,
mkdtempSync,
openSync,
readFileSync,
readSync,
readdirSync,
readlinkSync,
realpathSync,
rmSync,
statSync,
writeFileSync,
writeSync,
type Dirent,
} from "node:fs";
import { tmpdir } from "node:os";
import { basename, dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
import { FileAlreadyExistsError, renameWithRetry, unlinkWithRetry } from "../fs-atomic.ts";
import { classifyRecoverability, type RecoverabilityVerdict } from "./gates/recoverability.ts";
import { sha256Hex } from "../integrity/digest.ts";
import { resolveConfiguredIndexPath } from "../search/paths.ts";
import { runIntegrityCheck } from "../search/store/lifecycle.ts";
import {
acquireWriterLeaseSync,
acquireWriterLockSync,
keepWriterLeasesFresh,
resolveWriterLockStaleMs,
WRITER_LEASE_BLOCKING_BUDGET_MS,
type WriterLease,
} from "../search/store/writer-lock.ts";
import { appendLogEvent } from "./log.ts";
import {
buildManifest,
manifestSidecarPath,
readManifestSidecar,
SNAPSHOT_STORE_EXCLUSION,
writeManifestSidecar,
type BrainManifestDerivedStore,
type SnapshotStoreExclusionReason,
} from "./manifest.ts";
import {
BRAIN_RESTORE_STAGING_PREFIX,
BRAIN_ROOT_REL,
BRAIN_SNAPSHOT_EXCLUDED_ENTRIES,
BRAIN_SNAPSHOTS_REL,
brainDirs,
brainDirsForWrite,
SNAPSHOT_ARCHIVE_SUFFIX,
snapshotPath,
snapshotStorePath,
validateRunId,
} from "./paths.ts";
import { loadSnapshotDerivedStorePolicySafe, type BrainDerivedStorePolicy } from "./policy.ts";
import { isoSecond } from "./time.ts";
import { BRAIN_LOG_EVENT_KIND, type BrainSnapshotReason } from "./types.ts";
import { assertVaultIdentityForWrite } from "./vault-identity.ts";
import type { WriteLane } from "./freeze-marker.ts";
import { closeDatabase } from "../sqlite-close.ts";
import { machineStateRoot } from "../state/machine-state.ts";
// ----- Errors ---------------------------------------------------------------
/**
* Thrown when the host lacks both `tar` and any compressor we know how
* to use. Distinct from `BrainSnapshotError` so callers can pattern-
* match and offer install instructions.
*/
export class BrainSnapshotToolingMissingError extends Error {
constructor(tool: string, hint: string) {
super(`snapshot tooling missing: '${tool}' not found on PATH. ${hint}`);
this.name = "BrainSnapshotToolingMissingError";
}
}
/**
* Generic snapshot failure (archive write failed, restore failed,
* archive contents corrupted). Includes the runId in the message so
* the operator can locate it.
*/
export class BrainSnapshotError extends Error {
readonly runId: string;
/**
* `cause` is forwarded rather than flattened into the message because the
* caller that allocates a unique run id retries a lost race and has to
* recognise a collision by its errno AND the path it names.
* `createUniqueSnapshot` keys on
* `isFileAlreadyExistsAt`, which walks the cause chain; an error that only
* described its errno in prose would turn a retriable collision into a
* terminal failure, which is the shape this release removes.
*/
constructor(message: string, runId: string, options?: { readonly cause?: unknown }) {
super(`snapshot[${runId}]: ${message}`, options);
this.name = "BrainSnapshotError";
this.runId = runId;
}
}
/**
* Thrown when `.snapshots/` is present and cannot be enumerated.
*
* {@link listSnapshots} used to answer an unreadable directory with `[]`,
* which put "this vault has taken no recovery points" and "nobody could
* read the recovery points" on one wire - and every surface above it then
* printed the first sentence over the second. It is not a
* {@link BrainSnapshotError} because it belongs to no run id: the failure
* is the directory, and naming an arbitrary archive in the message would
* imply the read got as far as one.
*/
export class BrainSnapshotListingError extends Error {
/** Absolute path of the directory that could not be read. */
readonly path: string;
constructor(path: string, cause: unknown) {
super(
`snapshots directory ${path} could not be read: ${
cause instanceof Error ? cause.message : String(cause)
}`,
{ cause },
);
this.name = "BrainSnapshotListingError";
this.path = path;
}
}
/**
* Thrown when derived-store coverage was REQUESTED and could not be
* honoured. Carries the reason from the same closed vocabulary a
* manifest records, so the caller that reports the refusal and the
* manifest that records an omission speak one language.
*
* It is a `BrainSnapshotError` because it is a snapshot failure in every
* sense that matters to a caller: `createSnapshot` throws, nothing is
* left on disk, and the destructive-snapshot gate consequently never
* runs the operation it was protecting. That last property needs no
* plumbing - it follows from refusing here rather than downgrading to a
* partial snapshot that reports success.
*/
export class BrainSnapshotStoreError extends BrainSnapshotError {
readonly reason: SnapshotStoreExclusionReason;
constructor(message: string, runId: string, reason: SnapshotStoreExclusionReason) {
super(`derived store not archived (${reason}): ${message}`, runId);
this.name = "BrainSnapshotStoreError";
this.reason = reason;
}
}
// ----- Types ---------------------------------------------------------------
export interface CreateSnapshotResult {
/** Absolute path of the resulting archive. */
readonly path: string;
/** What this snapshot did about the derived store. */
readonly derived_store: BrainManifestDerivedStore;
}
/**
* The derived-store knobs, shared by the create and the restore path
* because both resolve the same live store: one to archive it, one to
* write it back.
*/
export interface SnapshotStoreOptions {
/**
* Derived-store coverage policy. Defaults to the vault's
* `snapshots:` block, so every existing call site picks the operator's
* setting up without threading it through.
*/
readonly derivedStore?: BrainDerivedStorePolicy;
/**
* Absolute path of the live derived store. Defaults to
* {@link liveDerivedStorePath}, which resolves it the way the search
* layer does - honouring both the `OPEN_SECOND_BRAIN_SEARCH_DB`
* environment variable and the `search_db_path` config key. A caller
* that already holds a resolved search config passes its own answer.
*/
readonly derivedStorePath?: string;
}
export interface CreateSnapshotOptions extends SnapshotStoreOptions {
/**
* Why this recovery point is being taken. REQUIRED, and required
* precisely because it used to be optional in effect: the reasons
* existed only as run-id prefixes, three of them inline literals, and
* nothing read them back. A default here would restore that state -
* every unnamed call site would silently claim whatever the default
* said - so there is none, and the compiler asks each caller instead.
*/
readonly reason: BrainSnapshotReason;
/**
* The write lane the guard checks. Content by default, so a frozen vault
* refuses a snapshot like any content write. The store migration takes
* its recovery point after it froze the legacy writers, on the audit
* lane, because an archive of a tree still being written is not one.
*/
readonly lane?: WriteLane;
/**
* Clock for the `snapshot` audit line's timestamp. Defaults to wall
* clock. A caller that already has an injected clock passes it, so a
* pass whose output is byte-reproducible given `now` - the dream pass
* is, and a test asserts it - does not become non-reproducible merely
* by recording that it took a recovery point.
*/
readonly now?: Date;
/**
* The token of a derived-store writer lock the CALLER already holds.
* When it is a live token on exactly the store this snapshot archives,
* the archive is written under that lock instead of acquiring it again
* (the lock is not re-entrant, so a second acquire would refuse).
* Anything else - a released token, a token on another path, a string
* naming the path, an object that merely looks like a token - takes
* the lock as usual. A token is minted only where the restore acquires
* the lock, so naming the store is not enough to skip it.
*
* One caller needs it: the gated restore takes the lock before its
* recovery point and keeps it through the swap, so no write lands
* between the two.
*/
readonly heldStoreLock?: StoreLockToken;
}
/** Mints {@link StoreLockToken}s; never leaves this module. */
const STORE_LOCK_TOKEN_MINT: unique symbol = Symbol("store-lock-token-mint");
/**
* The tokens whose lock is still held. A token leaves the set when its
* lock is released, so a kept reference stops counting at that moment.
*/
// Live token -> the store path its lock is held on. The path is read from
// here, never from the token, so a caller cannot redirect a genuine token.
const liveStoreLockTokens = new WeakMap<StoreLockToken, string>();
/**
* Proof that this module holds the derived store's writer lock on
* {@link StoreLockToken.path}. Constructed only with the module-private
* mint symbol, by the restore's lock acquisition, and honoured only
* while the lock is held (see {@link isLiveStoreLockToken}).
*/
export class StoreLockToken {
readonly #path: string;
constructor(mint: symbol, path: string) {
if (mint !== STORE_LOCK_TOKEN_MINT) {
throw new TypeError("a store lock token is minted only by the restore's lock acquisition");
}
this.#path = path;
}
/** The live store path the lock is held on. */
get path(): string {
return this.#path;
}
}
/** True only for a minted, unreleased token on exactly `path`. */
function isLiveStoreLockToken(token: unknown, path: string): boolean {
return token instanceof StoreLockToken && liveStoreLockTokens.get(token) === path;
}
export interface SnapshotInfo {
readonly run_id: string;
readonly path: string;
/**
* ISO-8601 UTC creation time: the `generated_at` the manifest sidecar
* recorded when the archive was taken, or the archive file's mtime when
* there is no readable sidecar. The manifest wins because an mtime is a
* property of the copy, not of the snapshot: `rsync -t`, `cp -p` or a
* stepped clock rewrite it, and retention orders by this field.
*/
readonly created_at: string;
/**
* Present only when the sidecar's `generated_at` was set aside for the
* mtime: why it was not believed (later than now, or earlier than the
* time the run id itself carries). A hand-edited sidecar or a broken
* clock would otherwise pin one archive as the newest forever, and every
* prune would then delete the genuinely newest ones.
*/
readonly created_at_untrusted?: string;
readonly size_bytes: number;
/**
* Absolute path of the sidecar manifest, or `null` when the
* sidecar write failed at snapshot time (read-only directory or
* similar). Rollback gracefully degrades on `null`.
*/
readonly manifest_path: string | null;
/**
* Absolute path of the sibling derived-store archive when one is on
* disk, `null` otherwise. Probed rather than believed: the manifest
* says what the snapshot INTENDED, this says what retention has to
* delete.
*/
readonly store_archive_path: string | null;
/**
* The sidecar's derived-store record, or `null` when there is no
* sidecar or the sidecar predates the feature. `null` is UNKNOWN
* coverage and must never be rendered as "excluded".
*/
readonly derived_store: BrainManifestDerivedStore | null;
/**
* Why the recovery point was taken, read back off the sidecar, or
* `null` when there is no sidecar, the sidecar is unreadable, or it
* predates the reason. `null` is UNKNOWN and is NOT recovered from the
* run id, even though every run id this project mints begins with the
* reason: parsing it back would report provenance for an archive that
* never recorded any.
*/
readonly reason: BrainSnapshotReason | null;
}
/**
* Why one `.snapshots/` entry could not become a {@link SnapshotInfo}.
*
* Both members describe an entry that IS an archive by name - the suffix
* matched - and could not be turned into a listing row. A file that is not
* an archive at all (a sidecar manifest, a store archive, an operator's
* stray note) is not in this vocabulary: it is not a recovery point that
* could not be read, it is not a recovery point.
*/
export const SNAPSHOT_ENTRY_SKIP_REASON = Object.freeze({
/** The name carries the archive suffix and the run id inside it does not validate. */
runIdUnparseable: "run_id_unparseable",
/** The archive is named in the directory and could not be stat'ed. */
entryUnreadable: "entry_unreadable",
} as const);
export type SnapshotEntrySkipReason =
(typeof SNAPSHOT_ENTRY_SKIP_REASON)[keyof typeof SNAPSHOT_ENTRY_SKIP_REASON];
export const SNAPSHOT_ENTRY_SKIP_REASONS: ReadonlyArray<SnapshotEntrySkipReason> = Object.freeze(
Object.values(SNAPSHOT_ENTRY_SKIP_REASON),
);
export function isSnapshotEntrySkipReason(value: unknown): value is SnapshotEntrySkipReason {
return (
typeof value === "string" &&
(SNAPSHOT_ENTRY_SKIP_REASONS as ReadonlyArray<string>).includes(value)
);
}
/** One archive-shaped entry that is missing from {@link SnapshotListing.snapshots}. */
export interface SnapshotEntrySkip {
/** Directory entry name, as `readdir` reported it. */
readonly name: string;
/** Absolute path of that entry. */
readonly path: string;
readonly reason: SnapshotEntrySkipReason;
/** The underlying failure, in the words of whatever refused. */
readonly detail: string;
}
/**
* One skipped entry as a phrase, spelled once for every surface that
* reports one: the doctor's uncertain stream and both CLI listings say the
* same thing about the same file, and a second wording would be a second
* claim about it.
*/
export function describeSnapshotEntrySkip(skip: SnapshotEntrySkip): string {
return `${skip.name} (${skip.reason}: ${skip.detail})`;
}
/**
* What {@link listSnapshots} found: the recovery points it could describe,
* and the archive-shaped entries it could not.
*
* Two fields rather than one array because "these are the recovery points"
* and "this listing is complete" are two different facts, and the second
* one used to have no wire at all: every per-entry failure was dropped, so
* a directory full of archives nobody could stat answered exactly like a
* vault that had never taken one. Callers that only count archives read
* {@link snapshots}; callers that tell an operator what their vault
* contains have to read {@link skipped} too, and the compiler now makes
* them look at it.
*/
export interface SnapshotListing {
/** Newest-first by archive mtime. */
readonly snapshots: SnapshotInfo[];
/** Empty when every archive-shaped entry became a row. */
readonly skipped: ReadonlyArray<SnapshotEntrySkip>;
}
/**
* Why a prune did not remove what it was asked to.
*
* One member, and it is the one that mattered: `snapshots.retention_count`
* is operator-supplied, and a zero silently removed every recovery point
* in the vault on the next dream. A refusal is a value here rather than a
* throw because the prune runs as a hygiene step behind every snapshot -
* throwing would fail the operation the snapshot was protecting, which
* trades a real guarantee for a tidy directory.
*/
export const SNAPSHOT_PRUNE_REFUSAL = Object.freeze({
/** The requested retention is under {@link SNAPSHOT_RETENTION_FLOOR}. */
belowRetentionFloor: "below_retention_floor",
} as const);
/** Closed union over {@link SNAPSHOT_PRUNE_REFUSAL}. */
export type SnapshotPruneRefusal =
(typeof SNAPSHOT_PRUNE_REFUSAL)[keyof typeof SNAPSHOT_PRUNE_REFUSAL];
/** Membership list, in declaration order. */
export const SNAPSHOT_PRUNE_REFUSALS: ReadonlyArray<SnapshotPruneRefusal> = Object.freeze(
Object.values(SNAPSHOT_PRUNE_REFUSAL),
);
/**
* `unknown` rather than `string`: the value rides out of TypeScript in
* the prune report a caller may persist or print, and the vocabulary
* census probes every guard with `null`, `42` and `{}`.
*/
export function isSnapshotPruneRefusal(value: unknown): value is SnapshotPruneRefusal {
return (
typeof value === "string" && (SNAPSHOT_PRUNE_REFUSALS as ReadonlyArray<string>).includes(value)
);
}
/**
* The smallest retention {@link pruneSnapshots} will act on.
*
* One, because one archive is the difference between a vault with a way
* back and a vault without one. A retention of zero is a configuration
* that asks the most destructive operation in this module to leave
* nothing behind, and obeying it silently is what made it dangerous.
*/
export const SNAPSHOT_RETENTION_FLOOR = 1;
export interface PruneSnapshotsOptions {
/**
* Run ids this pass must never remove, whatever the ordering says.
*
* The retention order was by MTIME when this was added (it is now the
* manifest's creation time, falling back to mtime for an archive with
* no readable sidecar), and the caller that most needs this
* is the one that just wrote an archive: "the newest file is the newest
* mtime" is an assumption about the filesystem, not arithmetic. `rsync
* -t`, `cp -p`, an NFS server with clock skew, or a system clock
* stepped backwards all leave an older archive claiming a newer mtime,
* and at retention 1 the prune then evicted the recovery point the gate
* had made one line earlier - which the gate reported as `covered`.
*
* The rollback is the second such caller: it takes a recovery point
* mid-restore, and the archive being restored is still being read from
* `.snapshots/` (its manifest sidecar, its derived-store archive) after
* that point lands. Both are named here rather than trusted to sort.
*/
readonly protectRunIds?: ReadonlyArray<string>;
/** The write lane the prune's guard checks; see {@link CreateSnapshotOptions.lane}. */
readonly lane?: WriteLane;
}
export interface PruneSnapshotsResult {
/** Vault-relative path of each deleted archive. */
readonly deleted: ReadonlyArray<string>;
/**
* Archives the prune tried and failed to remove. Previously swallowed
* whole: a permission error left the archive in place and reported a
* clean prune, so retention silently stopped working and nothing said
* so.
*/
readonly failed: ReadonlyArray<string>;
/**
* How many archives the listing could DESCRIBE when the prune returns.
*
* An entry `listSnapshots` had to skip - a run id that will not validate,
* a `stat` that failed - is still on disk and is not in this count, and
* is not a prune candidate either: this function removes what it can
* name. {@link SnapshotListing.skipped} is where those are reported.
*/
readonly retained: number;
/** Why nothing was pruned, or `null` when the prune ran. */
readonly refusal: SnapshotPruneRefusal | null;
}
/**
* What a restore did about the derived store. Separate fields rather than
* a single verdict string, because "not replaced" splits into answers an
* operator must be able to tell apart: the snapshot recorded that it did
* not include the store, or the snapshot carries no record at all.
*/
export interface RestoreDerivedStoreResult {
/** True when the archived store replaced the live one. */
readonly replaced: boolean;
/**
* False when the snapshot carries no derived-store record: coverage is
* unknown, and the live store was deliberately left untouched.
*/
readonly coverage_known: boolean;
/** Absolute path of the store that was replaced; null when none was. */
readonly path: string | null;
/** The manifest's named reason; null when included or unknown. */
readonly exclusion_reason: SnapshotStoreExclusionReason | null;
/**
* Whether a sibling store archive is on disk for this snapshot, probed
* rather than believed.
*
* Load-bearing exactly when {@link coverage_known} is false. The sidecar
* write is non-fatal in `createSnapshot`, so coverage can run to
* completion and leave no record of itself - and an archive sitting
* beside the tar proves that happened. Without this field the reporting
* surface had one sentence for two facts and chose the wrong one: it
* called a lost RECORD a snapshot older than the FEATURE, while the
* evidence to the contrary was in the same directory.
*/
readonly store_archive_present: boolean;
}
export interface RestoreSnapshotResult {
/** Number of regular files restored under `Brain/` (excluding the excluded entries). */
readonly restored_files: number;
readonly derived_store: RestoreDerivedStoreResult;
/**
* What the LIVE tree's recoverability was worth at the moment it was
* discarded - not the archive's. A restore deletes every top-level
* entry under `Brain/` and, before this field existed, said nothing
* about the state it destroyed doing so. Without a
* {@link RestoreSnapshotOptions.beforeDiscard} this reads `unproven`,
* which is the honest answer and the reason `restoreSnapshotWithRecoveryPoint`
* exists.
*/
readonly recoverability: RecoverabilityVerdict;
}
/**
* What a {@link RestoreSnapshotOptions.beforeDiscard} callback hands back
* to prove it archived the tree about to be discarded.
*
* A structural pair rather than the gate's own `DestructiveSnapshot`,
* because this module is the one `snapshot-gate.ts` composes: importing
* its type back would make the pair a cycle. The `path` is the field that
* matters - {@link restoreSnapshot} probes it on disk, so the verdict
* rests on an archive rather than on a promise that one was taken.
*/
export interface RecoveryPointEvidence {
/** Run id of the archive the callback wrote. */
readonly runId: string;
/** Absolute path of that archive. */
readonly path: string;
}
/**
* What {@link restoreSnapshot} tells its
* {@link RestoreSnapshotOptions.beforeDiscard} callback.
*/
export interface BeforeDiscardContext {
/**
* The live derived store the restore will replace, or null when it
* does not replace the store. A recovery point taken here archives
* exactly this path (`derivedStorePath`), and the restore refuses when
* the recovery point's manifest records another source.
*/
readonly storeTarget: string | null;
/**
* The token of the writer lock the restore holds on
* {@link storeTarget}, or null when it holds none. The lock is taken
* before the callback and kept through the swap, so a recovery point
* taken here sees every commit the swap replaces. A snapshot taken in
* the callback passes it as `heldStoreLock`, or its own acquire of the
* same lock refuses. The token stops counting once the restore
* releases the lock.
*/
readonly heldStoreLock: StoreLockToken | null;
}
export interface RestoreSnapshotOptions extends SnapshotStoreOptions {
/**
* The write lane the restore is guarded on (default: content), as the
* recovery point taken before it is: a restore that passes `audit` on a
* frozen vault is not refused after its recovery point was taken.
*/
readonly lane?: WriteLane;
/**
* Called once, AFTER the archive has been extracted into a temp
* directory and BEFORE the first live entry is removed. That ordering
* is the whole contract: extraction can fail on a corrupt or missing
* archive, and a rollback that never touched the tree must not leave a
* recovery point of it.
*
* Extraction being done is NOT the same as the restore being done with
* `.snapshots/`: the manifest sidecar and the derived-store archive are
* read after this callback returns, which is why a callback that prunes
* is checked rather than trusted - see the re-validation in
* {@link restoreSnapshot}.
*
* Return {@link RecoveryPointEvidence} to have the restore's
* recoverability verdict count the archive. Returning nothing is a
* legitimate answer and reads as `unproven`: the verdict is derived
* from an archive being on disk when the restore finishes, never from
* the fact that a callback was supplied.
*
* Injected rather than called directly because the recovery point is
* minted by `snapshot-gate.ts`, which composes THIS module - taking
* `takeSnapshot` from there would make the pair a cycle.
*/
readonly beforeDiscard?: (ctx: BeforeDiscardContext) => RecoveryPointEvidence | void;
}
// ----- Tooling detection ---------------------------------------------------
interface ToolAvailability {
readonly tar: boolean;
readonly zstd: boolean;
}
/**
* Detect tool availability by walking `process.env.PATH` and checking
* the candidate binary exists. We deliberately avoid `spawnSync(cmd,
* ["--version"])` because the underlying Node/Bun runtime resolves
* commands against an internal PATH snapshot taken at process start
* — so tests that mutate `process.env.PATH` between calls would have
* no effect on a `spawnSync` probe. Reading the filesystem at probe
* time keeps the detection honest and test-controllable.
*
* The cost (a handful of `existsSync` calls) is dwarfed by the actual
* archive operation that follows.
*/
function detectTooling(): ToolAvailability {
const pathEnv = process.env["PATH"] ?? "";
const dirs = pathEnv.split(process.platform === "win32" ? ";" : ":").filter((d) => d.length > 0);
const winExts =
process.platform === "win32"
? (process.env["PATHEXT"] ?? ".COM;.EXE;.BAT;.CMD").split(";")
: [];
const probe = (cmd: string): boolean => {
for (const d of dirs) {
if (existsSync(join(d, cmd))) return true;
for (const ext of winExts) {
if (existsSync(join(d, cmd + ext.toLowerCase()))) return true;
}
}
return false;
};
return {
tar: probe("tar"),
zstd: probe("zstd"),
};
}
/**
* A filesystem path spelled for a `tar` argument on this host.
*
* On Windows two tars answer to the name `tar`: bsdtar in `System32`, and
* the MSYS GNU tar that Git for Windows puts on PATH - ahead of `System32`
* inside Git Bash (where hooks run) and on GitHub's Windows runners. GNU
* tar under MSYS fails to extract into a backslashed `-C C:\...`
* ("Cannot open: No such file or directory"). Both tars accept the same
* path with forward slashes, which Windows itself treats as separators, so
* every path this module hands to tar goes through here. POSIX paths are
* returned unchanged.
*
* Archives are always passed through stdin/stdout (`-f -`), never by name:
* GNU tar reads the drive colon in `-f C:\...` as a remote host.
*/
export function tarPathArg(path: string, platform: NodeJS.Platform = process.platform): string {
return platform === "win32" ? path.replaceAll("\\", "/") : path;
}
// ----- createSnapshot ------------------------------------------------------
/**
* `rel` (vault-relative, `/`-separated) followed by everything beneath it,
* depth-first with siblings in byte order. Symlinked directories are listed
* but not entered - the same as tar's own default. Exported for tests.
*
* Names are raw bytes (`Buffer`), not strings: a POSIX file name need not be
* UTF-8, and a latin1 `caf\xe9.md` decoded to a string comes back as
* `caf\uFFFD.md` - a name that does not exist, which GNU tar then refuses
* to archive. Bytes round-trip exactly (on Windows they are the UTF-8 of
* the UTF-16 name, which is what tar reads the list as).
*
* The walk is exhaustive or it fails. tar is told not to recurse, so a
* directory this walk could not read would be archived EMPTY, and bsdtar
* (macOS, Windows `tar.exe`) reports success for that - a recovery point
* silently missing data. An entry that vanished mid-walk (ENOENT) is simply
* not in the tree any more and is left out; any other error is thrown as a
* {@link BrainSnapshotError} naming the path.
*/
export function sortedMembers(vault: string, rel: Buffer, runId: string): Buffer[] {
const abs = Buffer.concat([Buffer.from(vault), SLASH, rel]);
let entries: Buffer[];
try {
if (!lstatSync(abs).isDirectory()) return [rel];
entries = readdirSync(abs, { encoding: "buffer" });
} catch (err) {
if ((err as NodeJS.ErrnoException).code === "ENOENT") return [];
throw new BrainSnapshotError(
`cannot read ${rel.toString()} to archive it: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
const out = [rel];
for (const name of entries.toSorted(Buffer.compare)) {
out.push(...sortedMembers(vault, Buffer.concat([rel, SLASH, name]), runId));
}
return out;
}
const SLASH = Buffer.from("/");
const NUL = Buffer.from([0]);
/**
* Make `Brain/.snapshots/` exist, or say what is there instead.
*
* This `mkdir` is the first disk touch of every snapshot, which makes it the
* first thing that can fail and the worst place to fail opaquely. It used to
* be a bare `mkdirSync`, so its errno left the module naming a syscall and a
* path and nothing else - and, worse, reached the run-id allocator in
* `snapshot-gate.ts`, which read the EEXIST as a lost race over the run id
* and reported an id exhaustion that had never happened (GitHub #167).
*
* `recursive: true` already tolerates the normal case of the directory
* existing, so a throw from here means the path is NOT a usable directory.
* Which of the several ways that can be true is the only question the
* operator has, and it is answerable on the spot: `lstat` the path and say
* what is really there. A regular file, a symlink, a socket left by some
* other tool, a parent that is not a directory, a permission fault - each has
* a different remedy and the errno alone distinguishes none of them.
*
* The errno is carried as `cause`, not folded away: the diagnosis explains
* the failure, it does not replace the evidence. A {@link BrainSnapshotError}
* rather than a bare throw so this leaves the module in the one shape every
* caller of {@link createSnapshot} already handles, run id included.
*/
function ensureSnapshotsDirectory(path: string, runId: string): void {
try {
mkdirSync(path, { recursive: true });
return;
} catch (err) {
throw new BrainSnapshotError(
`cannot use the archive directory ${BRAIN_SNAPSHOTS_REL} at ${path}: ` +
`${describeExistingPath(path)} (${(err as Error).message ?? String(err)}); ` +
`no recovery point can be written until ${BRAIN_SNAPSHOTS_REL} is a writable directory`,
runId,
{ cause: err },
);
}
}
/**
* What is actually at `path`, in one clause an operator can act on.
*
* `lstat` rather than `stat` because a dangling or looping symlink is one of
* the states that produces this failure, and `stat` would report the target's
* absence instead of the link's presence. Its own failure is reported rather
* than swallowed: "the path cannot even be inspected" is a different remedy
* (permissions on the parent) from "the path holds the wrong kind of thing".
*/
function describeExistingPath(path: string): string {
let info: ReturnType<typeof lstatSync>;
try {
info = lstatSync(path);
} catch (err) {
const code = (err as NodeJS.ErrnoException).code;
if (code === "ENOENT") return "nothing is there, so the directory itself could not be created";
return `it could not be inspected either (${code ?? (err as Error).message})`;
}
if (info.isDirectory()) return "it is a directory, but it could not be created or opened as one";
if (info.isSymbolicLink()) {
return existsSync(path)
? "it is a symbolic link to something that is not a usable directory"
: "it is a symbolic link whose target does not exist";
}
if (info.isFile()) return "it is a regular file";
if (info.isSocket()) return "it is a socket";
if (info.isFIFO()) return "it is a FIFO";
return "it is not a directory";
}
/**
* Archive `Brain/` (minus {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES}) into
* `Brain/.snapshots/<run_id>.tar.zst`, optionally beside a compressed
* copy of the derived store.
*
* Implementation:
*
* 1. Detect available tools. If `tar` is absent we throw.
* 2. Cover the derived store FIRST, so a refusal leaves no tar behind.
* 3. List the top-level entries under `Brain/`, drop the excluded ones.
* 4. `tar -c -C <vault> Brain/<entry> ...` streamed into `zstd -19`
* (or `gzip -9` fallback) → output file.
*
* Using `--exclude=Brain/.snapshots` is tempting, but tar's exclude
* pattern matching is shell-dependent and prone to subtle surprises
* on filenames containing whitespace. Enumerating the kept entries
* explicitly is byte-stable and easy to reason about.
*/
export function createSnapshot(
vault: string,
runId: string,
opts: CreateSnapshotOptions,
): CreateSnapshotResult {
// The log event names the caller's vault; the archive is of the state root.
const logVault = vault;
vault = machineStateRoot(vault);
validateRunId(runId);
const dirs = brainDirsForWrite(vault, undefined, opts.lane);
ensureSnapshotsDirectory(dirs.snapshots, runId);
const outPath = snapshotPath(vault, runId);
const tools = detectTooling();
if (!tools.tar) {
throw new BrainSnapshotToolingMissingError(
"tar",
"install GNU tar or BSD tar; both are supported.",
);
}
// The derived store goes first. It is the step that can REFUSE, and a
// refusal must leave nothing behind - running it before the tar is
// written is what makes "no partial archive" a property of the order
// rather than of a cleanup path that could be forgotten.
const derivedStore = coverDerivedStore(vault, logVault, runId, tools, opts);
// List top-level entries of Brain/ that we want to capture. Sort the
// result so the resulting archive's contents are deterministic
// across filesystems (readdirSync's order is FS-dependent).
let topEntries: Buffer[];
try {
topEntries = readdirSync(dirs.brain, { encoding: "buffer" }).filter(
(e) => !isSnapshotExcludedEntry(bufferEntryName(e)),
);
} catch (err) {
discardStoreArchive(derivedStore, vault, runId);
throw new BrainSnapshotError(
`failed to list Brain/: ${(err as Error).message ?? String(err)}`,
runId,
);
}
topEntries.sort(Buffer.compare);
// Paths inside the archive start at `Brain/` — matching the rollback
// contract that the archive is "the Brain/ tree". Every member is
// listed explicitly, depth-first in sorted order, and tar is told not
// to recurse: left to itself tar walks each directory in `readdir`
// order, which differs between two copies of the same tree (ext4 hash
// order, NTFS, APFS). The member ORDER is therefore deterministic; the
// bytes are not (tar headers still carry mtime, owner and mode).
// The list travels NUL-separated through a file (`--null -T`), which
// GNU tar and bsdtar (macOS, Windows `tar.exe`) both accept and which
// survives any character a filename can hold.
const listDir = mkdtempSync(join(tmpdir(), `o2b-snapshot-list-${runId}-`));
const listPath = join(listDir, "members");
try {
const members = topEntries.flatMap((e) =>
sortedMembers(vault, Buffer.concat([Buffer.from(`${BRAIN_ROOT_REL}/`), e]), runId),
);
writeFileSync(listPath, Buffer.concat(members.flatMap((m) => [m, NUL])));
const tarArgs = [
"-c",
"-f",
"-",
"-C",
tarPathArg(vault),
"--no-recursion",
"--null",
"-T",
tarPathArg(listPath),
];
// tar writes into a staging file beside the member list rather than
// into a captured pipe, and the compressor reads that file: neither
// the tar stream nor the compressed archive ever sits in this
// process's memory, so the size of `Brain/` sets no ceiling here.
const tarPath = join(listDir, "brain.tar");
runArchiveProducer("tar", tarArgs, tarPath, runId);
compressInto({ kind: "file", path: tarPath }, outPath, tools, runId);
if (!existsSync(outPath)) {
throw new BrainSnapshotError(
`archive write reported success but ${outPath} is absent`,
runId,
);
}
} catch (err) {
// The store archive we just wrote belongs to a snapshot that does
// not exist. Leaving it would make the run id look taken to the
// collision resolver and would survive retention, which only prunes
// archives it can list.
discardStoreArchive(derivedStore, vault, runId);
throw err;
} finally {
rmSync(listDir, { recursive: true, force: true });
}
// Sidecar manifest. Failure is non-fatal: the archive is the
// load-bearing artifact, and a snapshot without a manifest just
// degrades rollback's drift detection to a silent-overwrite path
// (with a warning at rollback time). The alternative — failing
// the whole snapshot because the sidecar could not be written —
// would block dream from making any progress on a read-only
// `.snapshots/` directory.
try {
writeManifestSidecar(
vault,
runId,
buildManifest(dirs.brain, { derivedStore, snapshotReason: opts.reason }),
opts.lane,
);
} catch (err) {
process.stderr.write(
`warning: manifest sidecar write failed for snapshot ` +
`'${runId}': ${(err as Error).message ?? String(err)}; ` +
`rollback drift detection will be skipped for this snapshot.\n`,
);
}
logSnapshotEvent(logVault, runId, opts.reason, outPath, opts.now ?? new Date());
return { path: outPath, derived_store: derivedStore };
}
/**
* Record the recovery point in the Brain event log — the counterpart the
* log has been missing since `rollback` was added, which recorded the
* restore while the point it restores to left no trace but a filename.
*
* Best-effort, in the same shape the `rollback` verb already uses for its
* own event, and for a stronger reason here: this function runs AFTER the
* archive is on disk, and `createSnapshot` is called by the
* destructive-snapshot gate before the mutation it protects. A throw would
* therefore abort an operation whose recovery point already exists,
* turning a lost audit line into a refused mutation. The archive is the
* load-bearing artifact; the event is how it is found later.
*/
function logSnapshotEvent(
vault: string,
runId: string,
reason: BrainSnapshotReason,
archivePath: string,
now: Date,
): void {
try {
// Measured off the file rather than carried from the compressor, so
// the number describes the archive an operator can `ls`. An
// unmeasurable archive raises into the handler below and no event is
// written: a `0` here would read as an empty archive, which is a real
// and very different state.
const sizeBytes = statSync(archivePath).size;
appendLogEvent(vault, {
timestamp: isoSecond(now),
eventType: BRAIN_LOG_EVENT_KIND.snapshot,
body: { run_id: runId, reason, size_bytes: String(sizeBytes) },
});
} catch (err) {
process.stderr.write(
`warning: append snapshot log event failed for '${runId}': ` +
`${(err as Error).message ?? String(err)}; the archive itself is intact.\n`,
);
}
}
/**
* Decode one `readdirSync(..., { encoding: "buffer" })` entry to its UTF-8
* name.
*
* The exclusion below is a string comparison, so it holds only while the
* entry decodes to its name. Bun and Node document `Buffer` entries, whose
* `toString()` is the name, but a runtime that hands back a bare
* `Uint8Array` renders comma-joined byte values (`"46,100,..."`) instead:
* nothing matches, and `.snapshots` - the archive home - is packed into
* every archive, so each one carries its predecessors. `Buffer.from`
* decodes either shape.
*/
export function bufferEntryName(e: Uint8Array | Buffer): string {
return Buffer.from(e).toString("utf8");
}
/**
* True for a top-level `Brain/` entry the snapshot family never touches:
* the excluded entries, and a restore's staging directory left directly
* under `Brain/` (see {@link BRAIN_RESTORE_STAGING_PREFIX}), which must
* never be archived, nor moved aside and put back as Brain content.
*/
export function isSnapshotExcludedEntry(name: string): boolean {
return (
BRAIN_SNAPSHOT_EXCLUDED_ENTRIES.includes(name) || name.startsWith(BRAIN_RESTORE_STAGING_PREFIX)
);
}
// ----- Derived-store coverage ----------------------------------------------
/**
* Byte length of `path`, or `null` when it is not there to measure.
* Never zero-as-absent: a freshly created SQLite file is legitimately
* small, and an operator reading "0" needs it to mean zero bytes.
*/
function fileSizeOrNull(path: string): number | null {
try {
const st = statSync(path);
return st.isFile() ? st.size : null;
} catch {
return null;
}
}
/**
* The live derived store, resolved the way the search layer resolves it.
*
* The resolver honours the `search_db_path` override; the bare path
* helper answers only the default location. That distinction is the whole
* point: on a vault carrying the override, an archive built from the
* default would describe a file the search layer never reads, and if a
* stale database happened to sit there it would have been archived and
* reported as a success. It is imported from the resolver leaf rather
* than the search barrel because the barrel is reachable from this module
* and importing it back closes a cycle the architecture ratchet refuses.
*/
function liveDerivedStorePath(vault: string): string {
return resolveConfiguredIndexPath(vault);
}
/**
* Decide and, when asked, perform derived-store coverage for one
* snapshot. Returns the record the manifest will carry; THROWS
* {@link BrainSnapshotStoreError} when coverage was requested and any
* step could not be completed.
*
* The refusal is the point. A snapshot that quietly dropped the store it
* was asked to protect would report success, and the destructive
* operation it guards would then run against a recovery point that is
* not the one the operator configured.
*/
function coverDerivedStore(
vault: string,
liveVault: string,
runId: string,
tools: ToolAvailability,
opts: SnapshotStoreOptions & { readonly heldStoreLock?: StoreLockToken },
): BrainManifestDerivedStore {
// The policy and the live store are the caller's vault's (the search
// index is keyed by the real vault), never the mapped state root's; the
// archive lands beside the tree it goes with (`vault`).
const policy = opts.derivedStore ?? loadSnapshotDerivedStorePolicySafe(liveVault);
// Resolved through the search layer's resolver, never re-derived here.
const sourcePath = opts.derivedStorePath ?? liveDerivedStorePath(liveVault);
const liveSize = fileSizeOrNull(sourcePath);
const excluded = (reason: SnapshotStoreExclusionReason): BrainManifestDerivedStore =>
Object.freeze({
included: false,
source_path: sourcePath,
archive_name: null,
archive_sha256: null,
archive_size: null,
live_size: liveSize,
exclusion_reason: reason,
});
if (!policy.include) return excluded(SNAPSHOT_STORE_EXCLUSION.not_requested);
if (liveSize === null) {
throw new BrainSnapshotStoreError(
`no store file at ${sourcePath}`,
runId,
SNAPSHOT_STORE_EXCLUSION.absent,
);
}
const archivePath = snapshotStorePath(vault, runId);
if (existsSync(archivePath)) {
throw new BrainSnapshotError(
`refusing to overwrite an existing archive: ${archivePath}`,
runId,
);
}
// The writer lock on the LIVE store path, held across the integrity
// scan and the copy, so an index run cannot mutate the file underneath
// the snapshot. It is the same lock every other writer serialises on,
// taken synchronously because this whole module is synchronous. A
// caller that holds it on this exact path (the gated restore) hands
// down the live token the acquisition minted, and the archive is
// written under the caller's lock.
const release = isLiveStoreLockToken(opts.heldStoreLock, sourcePath)
? (): void => {}
: acquireWriterLockSync(sourcePath);
try {
// Everything under the lock blocks this thread - the integrity scan,
// `VACUUM INTO`, the compressor - for minutes on a large store, so the
// lock is kept fresh off the thread meanwhile, bounded by the scan and
// copy budget plus the compressor's own timeout.
let published = false;
try {
keepWriterLeasesFresh(
WRITER_LEASE_BLOCKING_BUDGET_MS + resolveSnapshotToolTimeoutMs(),
() => {
assertStoreIsSound(sourcePath, runId);
// Measured against the LIVE size, which is what the operator's
// ceiling is expressed in and what they can check with `ls`. The
// compressed archive is smaller, but refusing on a number nobody can
// predict before the work is done would be a worse contract.
if (liveSize > policy.maxBytes) {
throw new BrainSnapshotStoreError(
`store is ${liveSize} bytes, over the ${policy.maxBytes}-byte ceiling ` +
`(snapshots.derived_store_max_bytes)`,
runId,
SNAPSHOT_STORE_EXCLUSION.over_size_ceiling,
);
}
writeStoreArchive(sourcePath, archivePath, tools, runId);
published = true;
},
);
} catch (err) {
// An archive published before the lock turned out lost (the region's
// exit check) is this run's own: the name was free at the check above.
// It is removed, so the run leaves nothing a retry under the same id
// would be refused for.
if (published) rmSync(archivePath, { force: true });
if (err instanceof BrainSnapshotError) throw err;
throw new BrainSnapshotError(
`the derived store's writer lock was lost while its archive was written ` +
`(${(err as Error).message ?? String(err)}); no store archive was kept`,
runId,
{ cause: err },
);
}
} finally {
release();
}
const archiveSize = fileSizeOrNull(archivePath);
if (archiveSize === null) {
throw new BrainSnapshotError(
`store archive write reported success but ${archivePath} is absent`,
runId,
);
}
return Object.freeze({
included: true,
source_path: sourcePath,
archive_name: basename(archivePath),
archive_sha256: sha256Hex(readFileSync(archivePath)),
archive_size: archiveSize,
live_size: liveSize,
exclusion_reason: null,
});
}
/**
* Whether the live store at `path` holds data a recovery point must keep:
* the file exists and passes the same integrity scan a snapshot runs. An
* absent store has nothing to lose, and a condemned one is what a restore
* replaces to heal it, so neither forces store coverage on a recovery
* point.
*/
export function liveStoreWorthPreserving(path: string): boolean {
if (!existsSync(path)) return false;
try {
assertStoreIsSound(path, "recovery-point-probe");
return true;
} catch {
return false;
}
}
/**
* Refuse a condemned store rather than archiving it over a good
* snapshot.
*
* This runs the integrity scanner the previous release already shipped -
* `PRAGMA quick_check`, classified by {@link runIntegrityCheck} - rather
* than inventing a second gate with its own opinion. A file that is not
* a SQLite database at all fails here too: `new Database` is lazy, so
* the first statement raises and the scanner classifies the raise as a
* fault, which is the honest answer.
*/
function assertStoreIsSound(sourcePath: string, runId: string): void {
let db: Database;
try {
db = new Database(sourcePath, { readonly: true });
} catch (err) {
throw new BrainSnapshotStoreError(
`cannot open ${sourcePath}: ${(err as Error).message ?? String(err)}`,
runId,
SNAPSHOT_STORE_EXCLUSION.integrity_fault,
);
}
try {
const verdict = runIntegrityCheck(db);
if (!verdict.ok) {
throw new BrainSnapshotStoreError(
`${sourcePath} failed a structural integrity check: ${verdict.fault}`,
runId,
SNAPSHOT_STORE_EXCLUSION.integrity_fault,
);
}
} finally {
closeDatabase(db);
}
}
/**
* `VACUUM INTO` a private temp file, then compress that into
* `archivePath`.
*
* `VACUUM INTO` rather than a file copy because the store runs in WAL
* mode: the main file alone is not a consistent database, and the
* runtime exposes no online-backup API. `VACUUM INTO` is read-only with
* respect to the source, produces a single self-contained and compacted
* file, and is the native answer to exactly this question.
*
* The temp file lives in the OS temp directory rather than beside the
* store, so a crash between the vacuum and the compression leaves
* nothing in the vault for the next reader to find.
*/
function writeStoreArchive(
sourcePath: string,
archivePath: string,
tools: ToolAvailability,
runId: string,
): void {
const tmp = mkdtempSync(join(tmpdir(), `o2b-store-vacuum-${runId}-`));
const vacuumed = join(tmp, DERIVED_STORE_VACUUM_FILE);
try {
const db = new Database(sourcePath, { readonly: true });
try {
db.query("VACUUM INTO ?").run(vacuumed);
} finally {
closeDatabase(db);
}
compressInto({ kind: "file", path: vacuumed }, archivePath, tools, runId);
} catch (err) {
// Nothing to remove here: the compressor wrote a partial file that
// `compressInto` discards, and the archive name is only ever created by
// a successful publish. A failure on that name is a peer's archive,
// which must survive the refusal.
if (err instanceof BrainSnapshotError) throw err;
throw new BrainSnapshotError(
`failed to archive the derived store: ${(err as Error).message ?? String(err)}`,
runId,
);
} finally {
try {
rmSync(tmp, { recursive: true, force: true });
} catch {
// tmp cleanup is best-effort; the OS will reclaim it eventually.
}
}
}
/** Filename of the vacuumed copy inside its private temp directory. */
const DERIVED_STORE_VACUUM_FILE = "store.sqlite";
/** Remove a store archive whose snapshot is being abandoned. */
function discardStoreArchive(
record: BrainManifestDerivedStore,
vault: string,
runId: string,
): void {
if (!record.included) return;
try {
rmSync(snapshotStorePath(vault, runId), { force: true });
} catch (err) {
process.stderr.write(
`warning: could not remove the orphaned derived-store archive for ` +
`'${runId}': ${(err as Error).message ?? String(err)}\n`,
);
}
}
/**
* Compress the producer's stdout into `outPath`, choosing zstd when the
* host has it and gzip otherwise.
*
* ## Both compressors refuse an existing target, and that is load-bearing
*
* `zstd -o` has always refused to overwrite. The gzip fallback wrote
* through `writeFileSync`, which does not - so on a host without zstd
* two concurrent destructive operations that landed on the same run id
* would silently destroy each other's recovery point. That was not a
* cosmetic asymmetry: `createUniqueSnapshot` in `snapshot-gate.ts`
* resolves a run-id collision by RETRYING when the create fails and the
* archive now exists, so on a gzip-only host the collision path had no
* failure to retry on and the whole mechanism was inert.
*
* The refusal is therefore made explicit here, before either compressor
* runs, so both paths fail with the same typed error and the same
* message. Both compressors then write a partial file, and
* {@link publishArchive} gives it the final name with an exclusive hard
* link, which closes the window between the check and the publish that a
* probe alone would leave open and means a killed compression never
* leaves a truncated archive under a recovery point's name.
*
* The on-disk extension stays `.tar.zst` under gzip. The restore probes
* the magic bytes rather than the name, and one suffix keeps listing and
* retention a single comparison.
*/
function compressInto(
payload: CompressionSource,
outPath: string,
tools: ToolAvailability,
runId: string,
): void {
if (existsSync(outPath)) throw archiveCollision(outPath, runId);
// The compressor writes a partial name beside the archive, and the
// archive appears under its final name only once the compressor has
// exited cleanly. A process killed part-way through compression (an
// operator's kill, a host timeout, a power cut) used to leave a
// truncated `<run_id>.tar.zst` with no manifest, which the listing,
// retention and rollback all treat as a recovery point. A partial file
// never ends in the archive suffix, so nothing lists it.
const partial = `${outPath}${PARTIAL_ARCHIVE_INFIX}${process.pid}-${randomBytes(4).toString("hex")}`;
try {
if (tools.zstd) {
runCompressor("zstd", ["-19", "-q", "-o", partial, payload.path], runId);
} else {
// No zstd: gzip in-process. `node:zlib` ships with every runtime this
// tool supports, so a snapshot never depends on a `gzip` binary -
// which native Windows does not have. Level 9 matches the `gzip -9`
// this replaced, byte format included, so older readers restore it
// unchanged.
gzipFileExclusive(payload.path, partial, runId);
}
publishArchive(partial, outPath, runId);
} finally {
discardPartial(partial);
}
}
/**
* Remove a partial archive this process wrote. Best-effort and never
* throwing: it runs in a `finally`, where a throw would replace the real
* outcome - turning a published archive into a failed snapshot, or hiding
* the typed collision the run-id allocator ladders on. A partial left
* behind is never listed, and {@link pruneSnapshots} removes stale ones.
*/
function discardPartial(partial: string): void {
try {
unlinkWithRetry(partial);
} catch (err) {
if ((err as NodeJS.ErrnoException).code === "ENOENT") return;
process.stderr.write(
`warning: could not remove the partial snapshot archive ${partial}: ` +
`${(err as Error).message ?? String(err)}\n`,
);
}
}
/** Infix of a partial archive's name: `<archive>.partial-<pid>-<hex>`. */
const PARTIAL_ARCHIVE_INFIX = ".partial-";
/**
* A partial archive older than this is left over from a process that was
* killed during compression. Generous against the longest compression a
* live snapshot could still be running.
*/
const STALE_PARTIAL_MS = 24 * 60 * 60 * 1000;
/**
* Remove partial archives a killed snapshot left in `.snapshots/`. They are
* this tool's own unfinished output, never a recovery point (nothing lists
* them), and each can be as large as the whole Brain tar. Best-effort.
*/
function removeStalePartials(snapshotsDirPath: string, now: number): void {
let names: string[];
try {
names = readdirSync(snapshotsDirPath);
} catch {
return;
}
for (const name of names) {
if (!name.includes(PARTIAL_ARCHIVE_INFIX)) continue;
const full = join(snapshotsDirPath, name);
try {
const st = lstatSync(full);
if (!st.isFile() || now - st.mtimeMs < STALE_PARTIAL_MS) continue;
unlinkWithRetry(full);
} catch {
// gone already, or not ours to remove right now; the next prune retries
}
}
}
/**
* The typed "this name is taken" refusal. It carries an EEXIST-shaped
* cause because the id allocator in `snapshot-gate.ts` ladders past a
* taken name and needs to recognise this as the collision it is. Without
* it the allocator sees an opaque snapshot failure, declines to retry, and
* aborts the destructive operation the snapshot exists to protect.
*/
function archiveCollision(outPath: string, runId: string): BrainSnapshotError {
return new BrainSnapshotError(`refusing to overwrite an existing archive: ${outPath}`, runId, {
cause: new FileAlreadyExistsError(outPath, { kind: "snapshot archive" }),
});
}
/**
* errno values from the hard link that rule out the rename fallback too: the
* partial file or the directory is gone, the volume is full or read-only.
* Every other failure (a filesystem without hard links reports EPERM,
* ENOTSUP, or on Windows FAT and exFAT an `ERROR_INVALID_FUNCTION` that
* libuv maps to EISDIR) falls through to the checked rename.
*/
const LINK_FATAL_CODES: ReadonlySet<string> = new Set(["ENOENT", "ENOSPC", "EROFS"]);
/**
* Give the finished `partial` file its final archive name, refusing a name
* that is already taken.
*
* A hard link is an exclusive create of the name with the content already
* in place: it fails with EEXIST when a peer published first and never
* replaces anything, on POSIX and on NTFS alike. `rename` would replace an
* existing archive silently, so it is used only when the link fails for
* another reason (typically a filesystem without hard links), behind an
* existence check. That leaves a window between
* the check and the rename on such a volume; the run id already carries
* the second it was taken in, and the allocator probes the name first.
* The partial file itself is removed by the caller.
*/
function publishArchive(partial: string, outPath: string, runId: string): void {
try {
linkSync(partial, outPath);
return;
} catch (err) {
const code = (err as NodeJS.ErrnoException).code ?? "";
if (code === "EEXIST") throw archiveCollision(outPath, runId);
if (LINK_FATAL_CODES.has(code)) {
throw new BrainSnapshotError(
`failed to publish ${outPath}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
}
if (existsSync(outPath)) throw archiveCollision(outPath, runId);
try {
renameWithRetry(partial, outPath);
} catch (err) {
throw new BrainSnapshotError(
`failed to publish ${outPath}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
}
/**
* Input read per gzip member by {@link gzipFileExclusive}. Bounds the
* memory the gzip fallback holds at once, whatever the archive's size.
*/
const GZIP_MEMBER_INPUT_BYTES = 8 * 1024 * 1024;
/**
* gzip `source` into `outPath` one bounded chunk at a time.
*
* `node:zlib` has no synchronous streaming API, and the snapshot path is
* synchronous by contract (the destructive gate runs it inline before the
* mutation it protects). So each chunk is compressed on its own and the
* results are written back to back: a multi-member gzip file (RFC 1952,
* section 2.2), which `gunzip`, `node:zlib` and every `tar -z` read as the
* concatenation of the members. The restore path reads it unchanged.
*
* The destination is opened `wx`, so the exclusivity {@link compressInto}
* relies on holds here exactly as it did for the single-buffer write. A
* file this call created and could not finish is removed: it is ours, and
* left behind it would read as a recovery point.
*/
function gzipFileExclusive(source: string, outPath: string, runId: string): void {
let out: number;
try {
out = openSync(outPath, "wx");
} catch (err) {
throw new BrainSnapshotError(
`failed to write ${outPath}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
let finished = false;
try {
const input = openSync(source, "r");
try {
const chunk = Buffer.allocUnsafe(GZIP_MEMBER_INPUT_BYTES);
let wroteMember = false;
for (;;) {
const read = readSync(input, chunk, 0, chunk.length, null);
if (read === 0) break;
writeFully(out, gzipSync(chunk.subarray(0, read), { level: 9 }));
wroteMember = true;
}
// An empty source still produces a valid (one-member) gzip file.
if (!wroteMember) writeFully(out, gzipSync(Buffer.alloc(0), { level: 9 }));
} finally {
closeSync(input);
}
finished = true;
} catch (err) {
throw new BrainSnapshotError(
`failed to write ${outPath}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
} finally {
closeSync(out);
if (!finished) rmSync(outPath, { force: true });
}
}
/** `writeSync` until every byte of `bytes` is on `fd`. */
function writeFully(fd: number, bytes: Buffer): void {
let offset = 0;
while (offset < bytes.length) {
offset += writeSync(fd, bytes, offset, bytes.length - offset);
}
}
/**
* Exclusive create of a decompressed payload. `wx` refuses an existing
* destination the way `zstd -o` does, so neither decompressor can replace
* a file it did not create.
*/
function writeArchiveExclusive(outPath: string, bytes: Buffer, runId: string): void {
try {
writeFileSync(outPath, bytes, { flag: "wx" });
} catch (err) {
throw new BrainSnapshotError(
`failed to write ${outPath}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
}
/**
* What a compressor reads: a file, handed over by path - the staged tar
* stream or the vacuumed store copy - so a multi-hundred-megabyte payload
* never round-trips through memory.
*/
type CompressionSource = { readonly kind: "file"; readonly path: string };
/**
* Hard ceiling on any single CAPTURED subprocess stream. Only diagnostic
* output (stderr) is captured on the archive path; archive bytes go
* through files, so this never bounds the size of a snapshot.
*/
const SUBPROCESS_MAX_BUFFER_BYTES = 256 * 1024 * 1024;
/** Environment variable that overrides {@link SNAPSHOT_TOOL_TIMEOUT_MS}. */
export const SNAPSHOT_TOOL_TIMEOUT_ENV = "OPEN_SECOND_BRAIN_SNAPSHOT_TIMEOUT_MS";
/**
* How long one `tar` or `zstd` run of a snapshot or a restore may take
* before it is killed and the snapshot fails.
*
* The whole module is synchronous by contract, so a child that never
* exits used to block the calling process forever: nothing else in the
* process can run, not even a timer that would notice. Fifteen minutes is
* far above what `zstd -19` needs for a `Brain/` tree of a few hundred
* megabytes, and finite, so a wedged tool turns into a named failure
* instead of a frozen hook, CLI or MCP call.
*/
export const SNAPSHOT_TOOL_TIMEOUT_MS = 15 * 60 * 1000;
/**
* The tool timeout in force: {@link SNAPSHOT_TOOL_TIMEOUT_ENV} when it is a
* whole number of milliseconds above zero, {@link SNAPSHOT_TOOL_TIMEOUT_MS}
* when it is unset or empty. Any other value is an error rather than a
* silent fallback, so a typo cannot quietly change the bound.
*/
export function resolveSnapshotToolTimeoutMs(env: NodeJS.ProcessEnv = process.env): number {
const raw = env[SNAPSHOT_TOOL_TIMEOUT_ENV];
if (raw === undefined || raw.trim() === "") return SNAPSHOT_TOOL_TIMEOUT_MS;
const text = raw.trim();
const parsed = /^\d+$/.test(text) ? Number(text) : Number.NaN;
if (!Number.isSafeInteger(parsed) || parsed <= 0) {
throw new Error(
`${SNAPSHOT_TOOL_TIMEOUT_ENV}=${raw} is not a timeout; set it to a whole number of ` +
`milliseconds > 0 (the default is ${SNAPSHOT_TOOL_TIMEOUT_MS}), or unset it`,
);
}
return parsed;
}
/**
* `spawnSync` with the snapshot tool timeout. A run that hits it is killed
* (SIGKILL: a tool that ignored its input closing will not honour a polite
* signal either) and reported as a {@link BrainSnapshotError} that names
* the tool and the bound. Every other outcome is returned unchanged for
* the caller's own status handling.
*/
function spawnSnapshotTool(
cmd: string,
args: ReadonlyArray<string>,
options: SpawnSyncOptions,
runId: string,
): SpawnSyncReturns<Buffer> {
const timeoutMs = resolveSnapshotToolTimeoutMs();
// A writer lock held across this run (the restore's, a store archive's)
// is kept fresh while the thread is blocked in the child, bounded by the
// timeout the child is killed at plus one stale window for the kill.
const r = keepWriterLeasesFresh(
timeoutMs + resolveWriterLockStaleMs(),
() =>
spawnSync(cmd, [...args], {
...options,
timeout: timeoutMs,
killSignal: "SIGKILL",
}) as SpawnSyncReturns<Buffer>,
);
if ((r.error as NodeJS.ErrnoException | undefined)?.code === "ETIMEDOUT") {
throw new BrainSnapshotError(
`${cmd} did not finish within ${timeoutMs} ms and was stopped; ` +
`no archive was published (raise ${SNAPSHOT_TOOL_TIMEOUT_ENV} for a very large tree)`,
runId,
{ cause: r.error },
);
}
return r;
}
/**
* Run one compressor that writes the destination named after its `-o`
* itself, so only its stderr is captured. Every destination handed to it
* is a name this process owns: a partial file with a random suffix, or a
* file in a private temp directory. A collision on the published archive
* name is detected by {@link publishArchive}, not here.
*/
function runCompressor(cmd: string, args: ReadonlyArray<string>, runId: string): void {
const r = spawnSnapshotTool(
cmd,
args,
{ stdio: ["ignore", "ignore", "pipe"], maxBuffer: SUBPROCESS_MAX_BUFFER_BYTES },
runId,
);
if (r.error) {
throw new BrainSnapshotError(`${cmd} failed to start: ${r.error.message}`, runId);
}
if (r.status !== 0) {
const stderr = (r.stderr ?? Buffer.from("")).toString("utf8").trim();
throw new BrainSnapshotError(`${cmd} exited with status ${r.status}: ${stderr}`, runId);
}
}
/**
* Run the archive producer with its stdout on `outFile`.
*
* No shell pipe (the old `sh -c "tar ... | zstd ..."` broke on quoting of
* paths with whitespace, and native Windows has no `sh`), and no captured
* pipe either: tar's stdout is a file descriptor opened here, so the
* stream goes straight to disk. It used to be captured through
* `spawnSync` with a 256 MB `maxBuffer`, which a `Brain/` tree past that
* size overran with `ENOBUFS` after tar had already done all of its work.
* Only stderr is captured, for the error message.
*
* `outFile` is opened `wx`: it lives in a private temp directory, so an
* existing file there is a bug to surface rather than a file to reuse.
*
* Argument quoting: we deliberately do NOT shell-escape paths, because
* `tar -C <vault>` already roots everything relative to the vault, and
* the only user-supplied bytes in `tarArgs` are the run id (which
* `validateRunId` constrains to `[A-Za-z0-9._-]`) and the member list,
* which travels through a file.
*/
function runArchiveProducer(
cmd: string,
args: ReadonlyArray<string>,
outFile: string,
runId: string,
): void {
const fd = openSync(outFile, "wx");
let r: SpawnSyncReturns<Buffer>;
try {
r = spawnSnapshotTool(
cmd,
args,
{ maxBuffer: SUBPROCESS_MAX_BUFFER_BYTES, stdio: ["ignore", fd, "pipe"] },
runId,
);
} finally {
closeSync(fd);
}
if (r.error) {
throw new BrainSnapshotError(`${cmd} failed to start: ${r.error.message}`, runId);
}
if (r.status !== 0) {
const stderr = Buffer.from(r.stderr ?? "")
.toString("utf8")
.trim();
throw new BrainSnapshotError(`${cmd} exited with status ${r.status}: ${stderr}`, runId);
}
if (statSync(outFile).size === 0) {
throw new BrainSnapshotError(`${cmd} produced no stdout`, runId);
}
}
// ----- listSnapshots / pruneSnapshots --------------------------------------
/**
* Enumerate `.snapshots/*.tar.zst` in newest-first order (by the
* manifest's recorded creation time, see {@link SnapshotInfo.created_at}),
* alongside the archive-shaped entries that could not be described.
*
* Files outside the canonical naming pattern are skipped and NOT reported:
* a sidecar manifest, a store archive or a stray text file is not a
* recovery point, so naming it would be noise.
*
* The three answers this returns are three different facts:
*
* - a directory that is not there is an empty history: no snapshot has
* ever been taken, and both fields are empty.
* - a directory that IS there and cannot be enumerated throws
* {@link BrainSnapshotListingError}, because `[]` for that too is what
* let the listing surfaces print "no snapshots available" over a read
* that never happened.
* - an archive that is named in the directory and cannot be turned into
* a row - an unvalidatable run id, or a `stat` that failed - lands in
* {@link SnapshotListing.skipped}. It used to be dropped with a bare
* `continue`, which put a populated history nobody could stat and a
* vault that never took one on the same wire; the caller decides what
* to say about it, but it can no longer fail to know.
*/
export function listSnapshots(vault: string): SnapshotListing {
vault = machineStateRoot(vault);
const dirs = brainDirs(vault);
if (!existsSync(dirs.snapshots)) return { snapshots: [], skipped: [] };
let entries: string[];
try {
entries = readdirSync(dirs.snapshots);
} catch (err) {
throw new BrainSnapshotListingError(dirs.snapshots, err);
}
const infos: SnapshotInfo[] = [];
// Archive mtimes, kept beside the rows only to break a creation-time
// tie: the manifest records whole seconds, so two archives taken in
// the same second compare equal on `created_at`.
const mtimes = new Map<string, number>();
const skipped: SnapshotEntrySkip[] = [];
const nowMs = Date.now();
for (const name of entries) {
if (!name.endsWith(SNAPSHOT_ARCHIVE_SUFFIX)) continue;
const runId = name.slice(0, -SNAPSHOT_ARCHIVE_SUFFIX.length);
const full = join(dirs.snapshots, name);
// The `.snapshots/` dir is ours, so a malformed run_id here indicates
// manual tampering — the listing stays tolerant and keeps describing
// the archives it can, but the entry is carried out rather than
// dropped: an operator hunting a recovery point they know they took
// needs to be told the file is there under a name this build will not
// accept.
try {
validateRunId(runId);
} catch (err) {
skipped.push({
name,
path: full,
reason: SNAPSHOT_ENTRY_SKIP_REASON.runIdUnparseable,
detail: err instanceof Error ? err.message : String(err),
});
continue;
}
let st;
try {
st = statSync(full);
} catch (err) {
skipped.push({
name,
path: full,
reason: SNAPSHOT_ENTRY_SKIP_REASON.entryUnreadable,
detail: err instanceof Error ? err.message : String(err),
});
continue;
}
const sidecar = manifestSidecarPath(vault, runId);
const storeArchive = snapshotStorePath(vault, runId);
// One read for both sidecar-derived columns: two calls would parse
// the same file twice and could disagree if a peer rewrote it between
// them.
const manifest = readManifestSidecar(vault, runId);
mtimes.set(runId, st.mtimeMs);
const created = snapshotCreatedAt(manifest?.generated_at, st.mtimeMs, runId, nowMs);
infos.push({
run_id: runId,
path: full,
created_at: created.at,
...(created.untrusted !== null ? { created_at_untrusted: created.untrusted } : {}),
size_bytes: st.size,
manifest_path: existsSync(sidecar) ? sidecar : null,
store_archive_path: existsSync(storeArchive) ? storeArchive : null,
// `?? null` collapses "no sidecar", "unreadable sidecar" and
// "sidecar predating the feature" into one answer, which is
// correct: all three mean the coverage is UNKNOWN, and none of
// them is evidence that the store was excluded.
derived_store: manifest?.derived_store ?? null,
// Same three states, same single answer, and the same refusal to
// improve on it: the run id is not evidence of a reason.
reason: manifest?.snapshot_reason ?? null,
});
}
// Sort newest-first by recorded creation time, then by mtime within
// the same second, then by run id so the order is total. We avoid a
// lexicographic run_id sort as the primary key because manual rollback
// runs might use a non-timestamped id and we still want them to land
// where the operator's mental model expects (most recent first).
infos.sort(
(a, b) =>
Date.parse(b.created_at) - Date.parse(a.created_at) ||
(mtimes.get(b.run_id) ?? 0) - (mtimes.get(a.run_id) ?? 0) ||
(a.run_id < b.run_id ? 1 : a.run_id > b.run_id ? -1 : 0),
);
return { snapshots: infos, skipped };
}
/** How far ahead of now a sidecar stamp may be before it reads as a broken clock or an edit. */
const SNAPSHOT_STAMP_FUTURE_SKEW_MS = 5 * 60 * 1000;
/** Slack below the run id's time: both are whole seconds, and the sidecar is written after. */
const SNAPSHOT_STAMP_RUN_ID_SLACK_MS = 1000;
/**
* The UTC time a run id carries (`<reason>-YYYY-MM-DD-HHMMSS`, with an
* optional `-<n>` collision suffix), or `null` for an id without one.
*/
function runIdStampMs(runId: string): number | null {
const m = /-(\d{4})-(\d{2})-(\d{2})-(\d{2})(\d{2})(\d{2})(?:-\d+)?$/.exec(runId);
if (m === null) return null;
const [year, month, day, hour, minute, second] = m.slice(1).map(Number) as [
number,
number,
number,
number,
number,
number,
];
const ms = Date.UTC(year, month - 1, day, hour, minute, second);
return Number.isFinite(ms) ? ms : null;
}
/**
* The creation time retention orders by: the manifest's `generated_at`
* when it parses as a date and is believable, else the archive's mtime.
* An unparseable stamp falls back rather than sorting as `NaN`, which
* would make the comparator inconsistent and the prune order undefined.
* A stamp later than now (beyond a small skew) or earlier than the run
* id's own time falls back too, and the reason is returned so the
* listing can report it.
*/
function snapshotCreatedAt(
generatedAt: string | undefined,
mtimeMs: number,
runId: string,
nowMs: number,
): { readonly at: string; readonly untrusted: string | null } {
let untrusted: string | null = null;
if (generatedAt !== undefined) {
const ms = Date.parse(generatedAt);
if (Number.isFinite(ms)) {
const runMs = runIdStampMs(runId);
// Quoted as JSON: the stamp is printed, and a parseable one can still
// carry a newline or a terminal escape inside a `(...)` comment. JSON
// leaves DEL and the C1 controls (U+009B is a one-character CSI) as
// they are, so those are escaped too.
const shown = JSON.stringify(generatedAt).replace(
/[\u007f-\u009f]/g,
(c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, "0")}`,
);
if (ms > nowMs + SNAPSHOT_STAMP_FUTURE_SKEW_MS) {
untrusted = `the manifest's generated_at ${shown} is later than now`;
} else if (runMs !== null && ms < runMs - SNAPSHOT_STAMP_RUN_ID_SLACK_MS) {
untrusted =
`the manifest's generated_at ${shown} is earlier than the run id's own time ` +
new Date(runMs).toISOString();
} else {
return { at: new Date(ms).toISOString(), untrusted: null };
}
}
}
return { at: new Date(mtimeMs).toISOString(), untrusted };
}
/**
* Delete all but the `retention_count` newest archives. Returns what it
* removed, what resisted removal, how many archives survive, and - when
* it declined to run at all - why. Idempotent: a second run on the same
* directory removes nothing.
*
* This is the most destructive operation in the module, and it cannot be
* gated on taking a recovery point: gating the thing that destroys
* recovery points on making one is circular. It gets a floor and a named
* refusal instead. Below {@link SNAPSHOT_RETENTION_FLOOR} nothing is
* removed and the refusal travels back, because a configured retention of
* zero asks this function to leave a vault with no way back, and it used
* to comply on every snapshot and every dream without saying a word.
*
* A removal that fails is REPORTED rather than swallowed. It used to be
* silently skipped, so a permission problem stopped retention working and
* every prune still read as clean. A failure also STOPS the companion
* removal for that archive: the sidecar manifest and the store archive go
* only on the branch where the archive itself went, because an archive
* left on disk without its manifest is worse than either - rollback
* against it skips drift detection and silently never restores its
* derived store.
*
* {@link PruneSnapshotsOptions.protectRunIds} is how a caller keeps the
* archive it is standing on. See that field for why the ordering alone
* was never enough.
*/
export function pruneSnapshots(
vault: string,
retentionCount: number,
opts: PruneSnapshotsOptions = {},
): PruneSnapshotsResult {
vault = machineStateRoot(vault);
// Vault-identity write guard (context-integrity-gates, Unit J). This is
// the most destructive operation in the module - an `rmSync` over
// archives - and it was the only one of the three without the guard
// its `createSnapshot` and `restoreSnapshot` siblings carry.
assertVaultIdentityForWrite(vault, undefined, opts.lane);
if (!Number.isInteger(retentionCount)) {
throw new Error(`pruneSnapshots: retentionCount must be an integer; got ${retentionCount}`);
}
removeStalePartials(brainDirs(vault).snapshots, Date.now());
if (retentionCount < SNAPSHOT_RETENTION_FLOOR) {
// Refused, not obeyed, and not thrown: see the docblock.
return Object.freeze({
deleted: Object.freeze([]),
failed: Object.freeze([]),
retained: listSnapshots(vault).snapshots.length,
refusal: SNAPSHOT_PRUNE_REFUSAL.belowRetentionFloor,
});
}
const all = listSnapshots(vault).snapshots;
if (all.length <= retentionCount) {
return Object.freeze({
deleted: Object.freeze([]),
failed: Object.freeze([]),
retained: all.length,
refusal: null,
});
}
const protectedIds = new Set(opts.protectRunIds ?? []);
// Protected archives are moved to the FRONT of the retention order and
// then filtered out of the victim list a second time. Two mechanisms
// for one guarantee, because they answer different failures: the
// reordering keeps the retained count at exactly `retentionCount` in
// the normal case, and the filter is what holds when a caller protects
// more archives than the retention allows.
const ordered = [
...all.filter((s) => protectedIds.has(s.run_id)),
...all.filter((s) => !protectedIds.has(s.run_id)),
];
const victims = ordered.slice(retentionCount).filter((s) => !protectedIds.has(s.run_id));
const deleted: string[] = [];
const failed: string[] = [];
for (const v of victims) {
try {
rmSync(v.path, { force: true });
} catch {
// The archive stays put and the caller is told which one, so a
// recurring permission problem is visible instead of being
// rediscovered when retention has quietly stopped working. The
// companions below are deliberately NOT reached: an archive
// without its manifest is an archive a rollback misreads.
failed.push(v.path);
continue;
}
deleted.push(v.path);
// Remove the matching sidecar manifest and derived-store archive if
// present. Independent try/catch per companion so a missing one (a
// snapshot whose sidecar write failed at creation time, or one taken
// without store coverage) must not abort the prune of subsequent
// victims.
for (const companion of [v.manifest_path, v.store_archive_path]) {
if (companion === null) continue;
try {
rmSync(companion, { force: true });
} catch {
// Same rationale as above — best-effort.
}
}
}
return Object.freeze({
deleted: Object.freeze(deleted),
failed: Object.freeze(failed),
retained: all.length - deleted.length,
refusal: null,
});
}
// ----- restoreSnapshot -----------------------------------------------------
/**
* Result of {@link extractSnapshotToTemp}. `brainRoot` is the
* extracted `Brain/` directory (sibling to the live tree, inside a
* private tmp dir); `tmpRoot` is the parent directory the caller
* owns. {@link cleanup} removes the tmp dir best-effort.
*/
export interface ExtractSnapshotResult {
readonly tmpRoot: string;
readonly brainRoot: string;
readonly cleanup: () => void;
}
/**
* Extract a snapshot archive into a private tmp directory and return
* pointers to the materialised tree. The caller is responsible for
* invoking {@link ExtractSnapshotResult.cleanup} once the data is no
* longer needed.
*
* Used by:
* - {@link restoreSnapshot} — actually replaces the live tree.
* - `o2b brain rollback --dry-run` — previews the restore plan.
* - `o2b brain snapshot diff` — read-only inspector across two
* snapshots or a snapshot and the live tree.
*
* Shared so the tar / zstd / gzip decompression logic stays in one
* place. Throws {@link BrainSnapshotError} on archive corruption /
* missing root, {@link BrainSnapshotToolingMissingError} when the
* host lacks the required external tool.
*/
export function extractSnapshotToTemp(vault: string, runId: string): ExtractSnapshotResult {
vault = machineStateRoot(vault);
validateRunId(runId);
const archive = snapshotPath(vault, runId);
if (!existsSync(archive)) {
throw new BrainSnapshotError(`archive does not exist: ${archive}`, runId);
}
const tools = detectTooling();
if (!tools.tar) {
throw new BrainSnapshotToolingMissingError(
"tar",
"install GNU tar or BSD tar; both support the same -x command.",
);
}
const tmp = mkdtempSync(join(tmpdir(), `o2b-brain-extract-${runId}-`));
const cleanup = (): void => {
try {
rmSync(tmp, { recursive: true, force: true });
} catch {
// tmp cleanup is best-effort; the OS will reclaim it eventually.
}
};
try {
// Probe the magic bytes to decide how to decompress. zstd starts
// with `28 B5 2F FD`; gzip with `1F 8B`. Anything else is rejected
// — we don't blindly try every decompressor.
const decompressor = detectArchiveCompression(archive);
if (decompressor === "zstd" && !tools.zstd) {
throw new BrainSnapshotToolingMissingError(
"zstd",
"archive is zstd-compressed; install zstd to restore it.",
);
}
if (decompressor === "zstd") {
// Decompressed to a staging file OUTSIDE the extraction root, then
// handed to tar as its stdin descriptor: neither stream is captured
// in memory, so an archive whose tar stream is larger than any
// capture buffer restores as well as a small one. Captured through
// a 256 MB `maxBuffer` before, which such an archive overran.
const staging = mkdtempSync(join(tmpdir(), `o2b-brain-extract-tar-${runId}-`));
try {
const tarball = join(staging, "brain.tar");
runCompressor("zstd", ["-d", "-q", "-o", tarball, archive], runId);
const input = openSync(tarball, "r");
let tar: SpawnSyncReturns<Buffer>;
try {
// `-f -` is explicit stdin: GNU tar defaults to stdin without
// it, but BSD tar and busybox tar do not, so passing the flag
// keeps the extraction portable across hosts.
tar = spawnSnapshotTool(
"tar",
["-x", "-f", "-", "-C", tarPathArg(tmp)],
{ stdio: [input, "inherit", "pipe"] },
runId,
);
} finally {
closeSync(input);
}
if (tar.error || tar.status !== 0) {
const stderr = Buffer.from(tar.stderr ?? "")
.toString("utf8")
.trim();
throw new BrainSnapshotError(
`tar extract failed: ${tar.error?.message ?? stderr}`,
runId,
);
}
} finally {
rmSync(staging, { recursive: true, force: true });
}
} else {
// gzip is inflated in-process and piped to `tar -f -`, like the zstd
// branch: no `gzip` binary needed (native Windows has none), and no
// reliance on `tar -z` spawning one.
let tarball: Buffer;
try {
tarball = gunzipSync(readFileSync(archive));
} catch (err) {
throw new BrainSnapshotError(
`gzip decompress failed: ${(err as Error).message ?? String(err)}`,
runId,
);
}
const tar = spawnSnapshotTool(
"tar",
["-x", "-f", "-", "-C", tarPathArg(tmp)],
{ input: tarball, stdio: ["pipe", "inherit", "pipe"] },
runId,
);
if (tar.error || tar.status !== 0) {
const stderr = (tar.stderr ?? Buffer.from("")).toString("utf8").trim();
throw new BrainSnapshotError(`tar extract failed: ${tar.error?.message ?? stderr}`, runId);
}
}
const extractedBrain = join(tmp, BRAIN_ROOT_REL);
if (!existsSync(extractedBrain)) {
throw new BrainSnapshotError(`archive does not contain a ${BRAIN_ROOT_REL}/ root`, runId);
}
refuseEscapingSymlinks(extractedBrain, runId);
return Object.freeze({ tmpRoot: tmp, brainRoot: extractedBrain, cleanup });
} catch (err) {
cleanup();
throw err;
}
}
/**
* Refuse an extracted tree that carries a symlink pointing outside it.
*
* The archive is vault content - agent-writable, and in a peer-replicated
* vault writable by every peer - so its members are untrusted input, and
* the restore step after this sweep copies top-level entries into the
* live tree with `dereference` off, which would re-plant such a link
* verbatim into `Brain/`. Symlinks that stay INSIDE the extracted tree
* are kept: the member walk that builds an archive lists symlinked files
* and directories as members without following them, so a vault that
* legitimately uses in-tree links produces archives carrying them, and a
* restore must not break on its own output.
*
* Every link is checked twice. Its recorded target, read with
* `readlink` and resolved against the link's own directory, must stay
* inside the tree - this is the only check a DANGLING link can get, and
* it is the one that matters for it: a link to a file that does not
* exist yet outside the vault is restored verbatim into `Brain/` and
* becomes a write-through escape the moment anything creates that file.
* When the target does resolve, its real path must stay inside too, so
* a chain through another link cannot leave the tree either.
*
* Both comparisons accept the tree under its resolved AND unresolved
* root: the tmpdir this tree extracts into may itself sit behind a
* platform symlink (macOS `/var`), and a compare against only one
* spelling would refuse the tree's own honest links.
*/
function refuseEscapingSymlinks(root: string, runId: string): void {
const realRoot = realpathSync(root);
const lexicalRoot = resolve(root);
const inside = (p: string): boolean =>
p === realRoot ||
p.startsWith(realRoot + sep) ||
p === lexicalRoot ||
p.startsWith(lexicalRoot + sep);
const stack: string[] = [root];
while (stack.length > 0) {
const dir = stack.pop()!;
let entries: Dirent[];
try {
entries = readdirSync(dir, { withFileTypes: true });
} catch (err) {
throw new BrainSnapshotError(
`cannot inspect the extracted tree at ${dir}: ${(err as Error).message ?? String(err)}`,
runId,
{ cause: err },
);
}
for (const entry of entries) {
const abs = join(dir, entry.name);
if (entry.isSymbolicLink()) {
const recorded = resolve(dirname(abs), readlinkSync(abs));
let target: string | null;
try {
target = realpathSync(abs);
} catch {
// Dangling (or a loop): only the recorded target can be judged.
target = null;
}
for (const candidate of target === null ? [recorded] : [recorded, target]) {
if (!inside(candidate)) {
throw new BrainSnapshotError(
`snapshot contains a symlink that escapes the extracted tree: ` +
`${relative(root, abs)} -> ${candidate}`,
runId,
);
}
}
continue;
}
if (entry.isDirectory()) stack.push(abs);
}
}
}
/**
* Restore the archive identified by `runId` over `Brain/`, and the
* derived store alongside it when the snapshot recorded one. The current
* `.snapshots/` directory is preserved verbatim so older rollbacks still
* have a path back.
*
* Every step that can fail for a reason other than the swap itself runs
* BEFORE anything live changes, so a failed restore leaves `Brain/` and
* the derived store exactly as they were:
*
* 1. Pre-flight the derived-store records: refuse an unreadable record
* or a recorded archive that is not on disk, and refuse a live store
* nested inside a top-level `Brain/` directory the swap replaces.
* 2. Extract the archive into a private temp dir and verify it has a
* `Brain/` root with no symlink escaping it.
* 3. Stage the extracted top-level entries (minus
* {@link BRAIN_SNAPSHOT_EXCLUDED_ENTRIES}, and minus the store's file
* family when the store sits directly under `Brain/`) on the live
* tree's own filesystem, and verify the staged copy holds every file.
* 4. When the manifest says the snapshot included the derived store:
* check the archive against the recorded digest, decompress it
* beside the live store, and run the integrity check on the result.
* 5. Take the derived store's writer lock (when it is being replaced),
* so lock contention also refuses before anything is touched. It is
* held from here through the swap.
* 6. Run {@link RestoreSnapshotOptions.beforeDiscard}, the seam where a
* recovery point of the state about to be discarded is taken, then
* re-check that the derived-store records did not change under it.
* The lock taken in step 5 means no write lands between the recovery
* point and the swap, so the recovery point holds every commit the
* swap replaces.
* 7. Swap: move each live top-level entry aside and rename the staged
* one into place, then rename the staged store over the live one.
* Renames only, on one filesystem, so the window is short. If any
* rename fails, the renames already done are undone in reverse.
* 8. Remove the outgoing entries and every staging directory.
*
* Step 7 is a data-loss operation with the strongest confirmation ladder
* in the codebase in front of it and, until step 5 existed, nothing at
* all behind it: an operator who rolled back to the wrong run id had no
* way back. Callers that want one use `restoreSnapshotWithRecoveryPoint`
* in `snapshot-gate.ts`; callers that do not get a `recoverability`
* verdict saying so rather than silence. When the swap fails AND its undo
* fails too, the error names what was swapped, where the outgoing entries
* were left, and the recovery point when one was taken. When the lock
* cannot be released after the swap committed, the error says the restore
* committed and names the recovery point; a release failure never
* replaces an earlier error, it is added to it.
*/
export function restoreSnapshot(
vault: string,
runId: string,
opts: RestoreSnapshotOptions = {},
): RestoreSnapshotResult {
const liveVault = vault;
vault = machineStateRoot(vault);
const dirs = brainDirsForWrite(vault, undefined, opts.lane);
// Everything a restore needs is checked before anything is replaced.
//
// The store step used to run after the Markdown tree had already been
// deleted and re-copied, so a missing store archive threw over work
// that had in fact completed: the caller reported a failed rollback,
// logged nothing, and the vault was rolled back anyway. Reachable
// whenever the sidecar survives without its companion archive - partial
// replication, a sync rule excluding large binaries, or an operator
// reclaiming disk. The create path already refuses before it writes;
// this is the same ordering on the way back.
assertDerivedStoreRestorable(vault, runId);
// The records the pre-flight just validated, as one comparable value.
const storeStateBefore = derivedStoreRestoreState(vault, runId);
// The live store target, resolved once: the store step writes to it,
// and the Brain swap must keep out of its way when it sits inside
// `Brain/`. The manifest's `source_path` is where the store WAS; the
// live target is where it is now. They differ when the operator moved
// the vault or set a `search_db_path` override after the snapshot, and
// the live answer is the one a restore must write to.
const storeTarget = opts.derivedStorePath ?? liveDerivedStorePath(liveVault);
const placement = storePlacement(dirs.brain, storeTarget);
refuseStoreTheSwapCannotWriteAround(placement, storeTarget, runId);
// Names the Brain swap leaves alone in both directions, beyond the
// excluded entries: with the store directly under `Brain/`, its file
// family. The live store, its journals and lock, and the staged
// replacement stay where the store swap expects them, and the raw copy
// of the store the Brain archive carries (it archives every top-level
// entry) is not restored over it, so the store is written once, by the
// store swap, from the store archive - or not at all when the snapshot
// did not cover it.
const keptOutOfSwap = (name: string, isDirectory: boolean): boolean =>
placement.kind === "top" && isStoreFamilyEntry(name, placement.base, isDirectory);
const ext = extractSnapshotToTemp(vault, runId);
let staged: StagedBrainTree | null = null;
let store: StagedDerivedStore | null = null;
try {
// Replace every top-level entry under Brain/, except the entries the
// snapshot family never touches. The `.snapshots/` exclusion is the
// load-bearing safety guarantee: rolling back an older state must
// not erase newer snapshots, otherwise the operator is one click
// away from losing their forward path. `.artifacts/` is excluded for
// the opposite reason - it is not in the archive at all, so deleting
// the live copy would restore nothing over it.
//
// `.snapshots/` is created first, as `createSnapshot` does: the
// staging directory goes there when it shares `Brain/`'s device, and
// a missing one would otherwise read as "another device" and send the
// staging directory into `Brain/` itself.
ensureSnapshotsDirectory(dirs.snapshots, runId);
staged = stageBrainTree(ext.brainRoot, dirs.brain, dirs.snapshots, runId, keptOutOfSwap);
// The derived store is decompressed and verified here, before the
// recovery point and before the Brain tree is touched. It used to be
// decompressed only after the tree had been replaced, so a `zstd`
// timeout, a corrupt payload or a full disk left a rolled-back tree
// beside the store it was meant to replace.
const plan = stageDerivedStore(vault, storeTarget, runId);
store = plan.staged;
// The correct semantics are "live tree == snapshot tree minus the
// excluded entries". Every live top-level entry except those is moved
// out, and every staged one moved in. This makes restore
// deterministic.
const liveEntries = existsSync(dirs.brain)
? readdirSync(dirs.brain, { withFileTypes: true })
.filter(
(e) =>
!isSnapshotExcludedEntry(e.name) &&
e.name !== staged!.name &&
!keptOutOfSwap(e.name, e.isDirectory()),
)
.map((e) => e.name)
: [];
// The writer lock is taken before the recovery point, not inside the
// store swap, and held through the swap. Taken before the recovery
// point because a writer committing between the recovery point and
// the swap would land a change the swap discards and the recovery
// point never saw. Taken before the first rename because an index run
// holding it is a refusal like any other, and it must arrive while
// nothing has been replaced yet.
const lock = store === null ? null : acquireStoreLockForRestore(store.target, runId);
let evidence: RecoveryPointEvidence | null = null;
let committed = false;
let primary: { readonly err: unknown } | null = null;
try {
// The last moment at which the live state still exists. A throw
// here propagates and nothing has been replaced yet, which is the
// same abort-before-the-operation rule the destructive gate applies.
//
// The recovery point runs tar, zstd and a store archive synchronously
// under the lock. Each of those keeps the lease fresh for its own
// bound, and the rest of the callback (the manifest, the retention
// pass) for this outer one; the bounds add up rather than nest, so a
// wedged recovery point holds the lock at most for their sum and then
// lets it go stale, never forever.
try {
evidence =
keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>
opts.beforeDiscard?.({
storeTarget: store?.target ?? null,
heldStoreLock: lock?.token ?? null,
}),
) ?? null;
} catch (err) {
// A lease lost while the recovery point ran is reported as the
// restore's own refusal, with nothing touched.
if (lock !== null && lock.lease.lost) assertStoreLeaseHeld(lock.lease, runId);
throw err;
}
if (store !== null && evidence !== null) {
assertRecoveryPointCoversTarget(vault, evidence, store.target, runId);
}
// The callback is the one step that can change `.snapshots/` between
// the pre-flight and the store swap - it takes a recovery point, and
// a retention pass rides behind that. `assertDerivedStoreRestorable`
// above validated a state this callback may have invalidated, so the
// pre-flight is re-run against the state that actually reaches the
// rest of the restore, and the RECORD is compared rather than only
// re-checked: a manifest that has been removed outright would pass a
// fresh pre-flight while turning a covered store into "coverage
// unknown, nothing restored".
if (opts.beforeDiscard !== undefined) {
assertDerivedStoreRestorable(vault, runId);
const after = derivedStoreRestoreState(vault, runId);
if (after !== storeStateBefore) {
throw new BrainSnapshotError(
"the derived-store records of this snapshot changed while the recovery point was " +
`being taken (${storeStateBefore} -> ${after}); refusing before the Brain tree is ` +
"touched, so nothing is half-restored",
runId,
);
}
}
// The lease is proven once more right before the first rename. One
// that another writer took over (the recovery point stalled past the
// stale window) excludes nothing any more, and the swap would discard
// whatever that writer committed.
if (lock !== null) assertStoreLeaseHeld(lock.lease, runId);
// The Brain/ renames run under the lease too, kept fresh off this
// thread, and the lease is proven once more when they are done, right
// before the store replacement: a lease lost during the renames
// undoes them instead of replacing a store another writer now owns.
const held: { swap?: ReturnType<typeof swapBrainTree> } = {};
const tree = staged;
const recovery = evidence;
let swapStarted = false;
const runSwap = (): void => {
swapStarted = true;
held.swap = swapBrainTree(tree, dirs.brain, liveEntries, runId, recovery);
};
try {
if (lock === null) runSwap();
else keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, runSwap);
} catch (err) {
// A failed rename has undone its own renames and says so, undo
// result and recovery included; that error stands. Only when the
// swap never started (the lease was found lost on entry) is this
// the restore's own refusal, with nothing touched.
if (held.swap === undefined) {
if (!swapStarted && lock !== null && lock.lease.lost) {
assertStoreLeaseHeld(lock.lease, runId);
}
throw err;
}
throw held.swap.fail(
`the derived store's writer lock was lost during the Brain/ swap (${errorText(err)})`,
err,
{ storeIntact: true },
);
}
const swap = held.swap;
if (swap === undefined)
throw new BrainSnapshotError("restore: the Brain/ swap did not run", runId);
if (store !== null) {
try {
commitDerivedStoreSwap(store);
} catch (err) {
throw swap.fail(`the derived store swap failed (${errorText(err)})`, err, {
storeIntact: !(err instanceof StoreSwapTornError),
});
}
}
committed = true;
swap.finish();
} catch (err) {
primary = { err };
throw err;
} finally {
if (lock !== null)
releaseStoreLockAfterRestore(lock.release, runId, committed, primary, evidence);
}
const derivedStore = plan.result;
return {
restored_files: staged.files,
derived_store: derivedStore,
// What the DISCARDED tree was worth, not the archive's. The live
// derived store is replaced only when the archive carried one, so
// it joins the blast radius exactly when it was actually swapped.
recoverability: classifyRecoverability({
// An ARCHIVE, probed here at the end of the restore rather than
// the presence of a callback. `beforeDiscard !== undefined` said
// `covered` for any caller that supplied a function - including
// one that took no snapshot, and including the case where the
// recovery point was written and then evicted by the retention
// pass behind it.
recoveryPoint: evidence !== null && existsSync(evidence.path),
blastRadius: { brainTopLevel: true, derivedStore: derivedStore.replaced },
}),
};
} finally {
store?.discard();
staged?.cleanup();
ext.cleanup();
}
}
/**
* The extracted `Brain/` entries, copied onto the live tree's filesystem
* so the swap is a run of renames rather than a delete-and-copy.
*
* `root` holds `incoming/` (the staged entries) and, once the swap has
* started, `outgoing/` (the live entries moved aside). {@link cleanup}
* removes it best-effort, and keeps it when {@link keep} has been called
* because the outgoing entries are the only copy left.
*/
interface StagedBrainTree {
/** Top-level name of {@link root} inside its parent directory. */
readonly name: string;
readonly root: string;
readonly incoming: string;
readonly entries: ReadonlyArray<string>;
/** Regular files under {@link incoming}. */
readonly files: number;
keep(): void;
cleanup(): void;
}
/**
* Copy the extracted tree to a staging directory on the live tree's
* filesystem and verify the copy.
*
* The staging directory goes under `.snapshots/` - excluded from every
* archive, manifest walk and restore, and never listed as a snapshot
* since it carries no archive suffix - when that directory shares a
* device with `Brain/`. When it does not (a separately mounted snapshot
* directory), a rename from it into `Brain/` would fail mid-swap with
* EXDEV, so the staging directory goes directly under `Brain/` instead
* and the swap leaves it alone.
*
* Entries `keptOut` names (the store's file family, when the store sits
* directly under `Brain/`) are neither staged nor counted: the swap leaves
* the live ones in place, and the store is restored by the store swap.
*/
function stageBrainTree(
extractedBrain: string,
liveBrain: string,
snapshotsDirPath: string,
runId: string,
keptOut: (name: string, isDirectory: boolean) => boolean,
): StagedBrainTree {
mkdirSync(liveBrain, { recursive: true });
const parent = sameDevice(snapshotsDirPath, liveBrain) ? snapshotsDirPath : liveBrain;
let root: string;
try {
root = mkdtempSync(join(parent, `${BRAIN_RESTORE_STAGING_PREFIX}${runId}-`));
} catch (err) {
throw new BrainSnapshotError(
`cannot create a restore staging directory under ${parent}: ${errorText(err)}`,
runId,
{ cause: err },
);
}
let kept = false;
const cleanup = (): void => {
if (kept) return;
try {
rmSync(root, { recursive: true, force: true });
} catch (err) {
console.error(
`warning: could not remove the restore staging directory ${root}: ${errorText(err)}`,
);
}
};
try {
const incoming = join(root, "incoming");
mkdirSync(incoming);
const entries = readdirSync(extractedBrain, { withFileTypes: true })
.filter((e) => !isSnapshotExcludedEntry(e.name) && !keptOut(e.name, e.isDirectory()))
.map((e) => e.name);
let files = 0;
let expected = 0;
for (const name of entries) {
// `verbatimSymlinks` is load-bearing: without it `cpSync` rewrites
// every relative link to the ABSOLUTE path of its target inside the
// temp extraction, which the cleanup deletes - the in-tree links the
// containment sweep kept would all restore dangling, pointing outside
// the vault. A relative link survives the later rename unchanged,
// because the staged entry sits at the same depth as the live one.
const from = join(extractedBrain, name);
const to = join(incoming, name);
cpSync(from, to, { recursive: true, verbatimSymlinks: true });
}
// Counted only once every entry is staged: a top-level link to another
// top-level entry resolves in the staged copy only after its target is.
for (const name of entries) {
expected += countFiles(join(extractedBrain, name));
files += countFiles(join(incoming, name));
}
if (files !== expected) {
throw new BrainSnapshotError(
`the staged Brain tree holds ${files} file(s) but the archive extracted ${expected}; ` +
"refusing before the Brain tree is touched",
runId,
);
}
return {
name: basename(root),
root,
incoming,
entries,
files,
keep: () => {
kept = true;
},
cleanup,
};
} catch (err) {
cleanup();
if (err instanceof BrainSnapshotError) throw err;
throw new BrainSnapshotError(
`failed to stage the extracted Brain tree under ${root}: ${errorText(err)}`,
runId,
{ cause: err },
);
}
}
/** Whether two existing paths live on the same device; false when unsure. */
function sameDevice(a: string, b: string): boolean {
try {
return statSync(a).dev === statSync(b).dev;
} catch {
return false;
}
}
/** An in-progress Brain swap: {@link finish} it, or {@link fail} it. */
interface BrainSwap {
/** Remove the outgoing entries; the swap is complete. */
finish(): void;
/**
* Undo the swap and return the error to throw, naming what failed,
* whether the undo succeeded and how to recover when it did not.
*/
fail(what: string, cause: unknown, opts?: { readonly storeIntact?: boolean }): BrainSnapshotError;
}
/**
* Move every live top-level entry into the staging directory's
* `outgoing/`, then rename every staged entry into `Brain/`.
*
* Renames on one filesystem: each one either happens or does not, and
* the outgoing entries stay on disk until {@link BrainSwap.finish}, so a
* failure at any point - here, or in the store swap that follows - is
* undone by renaming everything back in reverse order.
*/
function swapBrainTree(
staged: StagedBrainTree,
liveBrain: string,
liveEntries: ReadonlyArray<string>,
runId: string,
recoveryPoint: RecoveryPointEvidence | null,
): BrainSwap {
const outgoing = join(staged.root, "outgoing");
const movedOut: string[] = [];
const movedIn: string[] = [];
const undo = (): string[] => {
const failures: string[] = [];
for (const name of movedIn.toReversed()) {
try {
renameWithRetry(join(liveBrain, name), join(staged.incoming, name));
} catch (err) {
failures.push(`could not move the restored ${name} back out: ${errorText(err)}`);
}
}
for (const name of movedOut.toReversed()) {
try {
renameWithRetry(join(outgoing, name), join(liveBrain, name));
} catch (err) {
failures.push(`could not move the live ${name} back: ${errorText(err)}`);
}
}
return failures;
};
const fail = (
what: string,
cause: unknown,
opts: { readonly storeIntact?: boolean } = {},
): BrainSnapshotError => {
const storeIntact = opts.storeIntact ?? true;
const swapped =
`${movedOut.length} of ${liveEntries.length} live Brain/ top-level entries had been moved ` +
`aside and ${movedIn.length} of ${staged.entries.length} restored ones moved in`;
const failures = undo();
const recovery =
recoveryPoint === null
? "no recovery point was taken for this restore"
: `the recovery point ${recoveryPoint.runId} (${recoveryPoint.path}) holds the ` +
`pre-restore state; \`o2b brain rollback ${recoveryPoint.runId}\` puts it back`;
if (failures.length === 0) {
const state = storeIntact
? "Brain/ and the derived store are as they were before the restore"
: `Brain/ is as it was before the restore, but the derived store is not (see above); ${recovery}`;
return new BrainSnapshotError(
`restore failed during the swap: ${what}. ${swapped}; every Brain/ rename was undone, so ${state}`,
runId,
{ cause },
);
}
// The outgoing entries are now the only copy of what did not make it
// back: the staging directory must outlive this restore.
staged.keep();
return new BrainSnapshotError(
`restore failed during the swap: ${what}. ${swapped}; undoing it also failed ` +
`(${failures.join("; ")}), so Brain/ is a mix of the restored and the pre-restore ` +
`tree. The pre-restore entries that were moved aside are kept under ${outgoing} and ` +
`the restored ones not yet moved in under ${staged.incoming}; ${recovery}`,
runId,
{ cause },
);
};
try {
mkdirSync(outgoing);
for (const name of liveEntries) {
renameWithRetry(join(liveBrain, name), join(outgoing, name));
movedOut.push(name);
}
for (const name of staged.entries) {
renameWithRetry(join(staged.incoming, name), join(liveBrain, name));
movedIn.push(name);
}
} catch (err) {
throw fail(`a Brain/ rename failed (${errorText(err)})`, err);
}
return {
finish: () => {
try {
rmSync(outgoing, { recursive: true, force: true });
} catch (err) {
// The restore itself is complete; what is left is the discarded
// tree, which the staging cleanup retries.
console.error(
`warning: could not remove the pre-restore Brain entries under ${outgoing}: ` +
errorText(err),
);
}
},
fail,
};
}
/** An error's message, or its string form. */
function errorText(err: unknown): string {
return err instanceof Error ? err.message : String(err);
}
/**
* Refuse a restore that cannot be completed, before it starts.
*
* Two conditions, both of which used to surface only after the Markdown
* tree had been replaced. A derived-store record this build cannot read
* means coverage is indeterminate, so neither replacing the live store
* nor leaving it can be justified; a record that names an archive which
* is not on disk means the restore is already incomplete. Either way the
* honest answer is to touch nothing and say why.
*/
/**
* The three facts a restore's store step depends on, as one comparable
* token: whether the sidecar is readable, what it says about coverage,
* and whether the store archive is on disk.
*
* Compared rather than re-checked because the failure this exists to
* catch REMOVES a record: a fresh pre-flight over a vanished manifest
* passes (no record, nothing to validate) and the restore then reports
* coverage as unknown over a store it was asked to put back.
*/
function derivedStoreRestoreState(vault: string, runId: string): string {
const manifest = readManifestSidecar(vault, runId);
const record = manifest?.derived_store ?? null;
return [
manifest === null ? 0 : 1,
manifest?.derived_store_unreadable === true ? 1 : 0,
record === null ? 0 : record.included ? 1 : 2,
record?.archive_sha256 ?? "",
existsSync(snapshotStorePath(vault, runId)) ? 1 : 0,
].join("/");
}
function assertDerivedStoreRestorable(vault: string, runId: string): void {
const manifest = readManifestSidecar(vault, runId);
if (manifest === null) return;
if (manifest.derived_store_unreadable === true) {
throw new BrainSnapshotError(
"the manifest carries a derived-store record this build cannot read, so whether the " +
"snapshot covered the store is indeterminate; refusing to restore rather than " +
"guessing at the live store",
runId,
);
}
const record = manifest.derived_store ?? null;
if (record === null || !record.included) return;
const archive = snapshotStorePath(vault, runId);
if (!existsSync(archive)) {
throw new BrainSnapshotError(
`manifest records a derived-store archive but ${archive} is absent; refusing before the ` +
"Brain tree is touched, so nothing is half-restored",
runId,
);
}
}
/**
* Where the derived store sits relative to `Brain/`, which decides how the
* Brain swap must treat it.
*
* - `outside`: the swap and the store never meet.
* - `top`: the store is a direct child of `Brain/`. The swap leaves its
* file family (see {@link isStoreFamilyEntry}) where it is, in both
* directions; the store is written only by the store swap.
* - `nested`: the store is deeper inside a top-level `Brain/` entry the
* swap replaces wholesale; `top` names that entry.
* - `brain`: the store path is the `Brain/` directory itself.
*/
export type StorePlacement =
| { readonly kind: "outside" }
| { readonly kind: "top"; readonly base: string }
| { readonly kind: "nested"; readonly top: string }
| { readonly kind: "brain" };
/**
* True where the filesystem compares names without case by default
* (APFS, HFS+, NTFS), so `Index.sqlite` and `index.sqlite` are one file.
*/
function foldsCase(platform: NodeJS.Platform): boolean {
return platform === "darwin" || platform === "win32";
}
/** `name` as the platform compares it. */
function compareKey(name: string, platform: NodeJS.Platform): string {
return foldsCase(platform) ? name.toLowerCase() : name;
}
/**
* A path with every existing ancestor resolved through symlinks. Where the
* platform folds case, an absent component takes the spelling of the one
* entry of its parent that matches it without case, so a store configured
* as `Index.sqlite` is recognised as the `index.sqlite` on disk and in the
* archive.
*/
function canonicalPath(path: string, platform: NodeJS.Platform): string {
const abs = resolve(path);
try {
return realpathSync.native(abs);
} catch {
const parent = dirname(abs);
if (parent === abs) return abs;
const canonicalParent = canonicalPath(parent, platform);
return join(canonicalParent, existingSpelling(canonicalParent, basename(abs), platform));
}
}
/** The spelling of `name` among `dir`'s entries where case is folded. */
function existingSpelling(dir: string, name: string, platform: NodeJS.Platform): string {
if (!foldsCase(platform)) return name;
let entries: string[];
try {
entries = readdirSync(dir);
} catch {
return name;
}
if (entries.includes(name)) return name;
const key = compareKey(name, platform);
const matches = entries.filter((e) => compareKey(e, platform) === key);
return matches.length === 1 ? matches[0]! : name;
}
/**
* The directory entry a rename of `path` acts on: every ancestor resolved
* through symlinks, the last component itself not followed. Where the
* platform folds case, an existing entry takes its on-disk spelling.
*/
function entryPath(path: string, platform: NodeJS.Platform): string {
const abs = resolve(path);
const parent = dirname(abs);
if (parent === abs) return canonicalPath(abs, platform);
const canonicalParent = canonicalPath(parent, platform);
return join(canonicalParent, existingSpelling(canonicalParent, basename(abs), platform));
}
/**
* Classify the store target against the live `Brain/` directory. The
* containment test compares case-folded paths where the platform folds
* case; the names it returns keep the target's canonical spelling.
*
* What is classified is the directory entry at the target path, not the
* file a symlink there points to: the store swap renames that entry (it
* stages `<target>.restore-*` beside it, moves it aside to `<target>.bak`
* and renames the staged file over it), so the Brain swap must judge the
* same entry. A `Brain/index.sqlite` that links to a database outside
* `Brain/` is a store directly under `Brain/`: the link and the staged
* file beside it stay out of the tree swap, and the store swap replaces
* the link with the restored database. Classifying the link's target
* instead called it `outside`, and the tree swap then moved the staged
* file away before the store swap could rename it.
*/
export function storePlacement(
liveBrain: string,
target: string,
platform: NodeJS.Platform = process.platform,
): StorePlacement {
const brain = canonicalPath(liveBrain, platform);
// A path that resolves to `Brain/` itself is refused whatever its last
// component is: a link to the directory is still the directory.
if (compareKey(canonicalPath(target, platform), platform) === compareKey(brain, platform)) {
return { kind: "brain" };
}
const store = entryPath(target, platform);
const rel = relative(compareKey(brain, platform), compareKey(store, platform));
if (rel === "") return { kind: "brain" };
if (rel === ".." || rel.startsWith(`..${sep}`) || isAbsolute(rel)) return { kind: "outside" };
// The relative path's components, in the target's own spelling.
const parts = store.split(sep).slice(-rel.split(sep).length);
// An entry the swap never touches (`.snapshots/`, `.artifacts/`) keeps
// whatever it holds, so a store inside one does not meet the swap.
if (isSnapshotExcludedEntry(parts[0]!)) return { kind: "outside" };
return parts.length === 1 ? { kind: "top", base: parts[0]! } : { kind: "nested", top: parts[0]! };
}
/** SQLite's journal siblings of a database file: WAL mode and DELETE mode. */
const STORE_JOURNAL_SUFFIXES: ReadonlyArray<string> = Object.freeze(["-wal", "-shm", "-journal"]);
/**
* The database files the store layer keeps beside the store, as suffixes
* of its name: the store itself, the reindex staging file (`.new`), the
* parked previous index (`.bak`, also the restore's outgoing store, and
* `.old`).
*/
const STORE_DATABASE_SUFFIXES: ReadonlyArray<string> = Object.freeze(["", ".new", ".bak", ".old"]);
/** The writer lock `proper-lockfile` takes on the store: a directory. */
const STORE_LOCK_SUFFIX = ".lock";
/** The restore's staged store: `<store>.restore-<pid>-<8 hex>`. */
const STAGED_STORE_PATTERN = /^\.restore-\d+-[0-9a-f]{8}$/u;
/**
* True for a top-level `Brain/` entry that belongs to the store named
* `base` when the store sits directly under `Brain/`. A closed list, so a
* note or directory that merely starts with the store's name is restored
* like any other entry:
*
* - the store and each database file the store layer derives from it
* (`.new`, `.bak`, `.old`), each with its journals (`-wal`, `-shm`,
* `-journal`);
* - the restore's staged store (`.restore-<pid>-<hex>`) with its
* journals;
* - the writer lock directory (`.lock`), the only directory in the
* family.
*
* Names compare without case where the platform folds it. A suffix the
* store layer adds later must be listed here; until it is, the Brain swap
* treats that file as Brain content.
*/
export function isStoreFamilyEntry(
name: string,
base: string,
isDirectory: boolean,
platform: NodeJS.Platform = process.platform,
): boolean {
const key = compareKey(name, platform);
const stem = compareKey(base, platform);
if (!key.startsWith(stem)) return false;
const suffix = key.slice(stem.length);
if (isDirectory) return suffix === STORE_LOCK_SUFFIX;
// At most one journal suffix comes off: `-wal-shm` is not a journal.
const journal = STORE_JOURNAL_SUFFIXES.find((j) => suffix.endsWith(j));
const file = journal === undefined ? suffix : suffix.slice(0, -journal.length);
return STORE_DATABASE_SUFFIXES.includes(file) || STAGED_STORE_PATTERN.test(file);
}
/**
* Refuse a restore whose store target the swap cannot write around, before
* anything is extracted or touched.
*
* A store inside a top-level `Brain/` entry: the Brain swap renames whole
* top-level entries; the store swap renames one file. A store nested
* inside a renamed directory would be moved out with the live directory,
* its staged replacement with it, and the restored directory may not even
* contain the store's parent. Carving the store out of that rename would
* add a third set of renames, each with its own undo, to a swap whose
* whole contract is that every step can be reversed; refusing up front
* keeps that contract and loses nothing.
*
* A store target that is `Brain/` itself: the store swap would rename a
* database file over the directory the tree swap is restoring into.
*/
function refuseStoreTheSwapCannotWriteAround(
placement: StorePlacement,
target: string,
runId: string,
): void {
if (placement.kind === "brain") {
throw new BrainSnapshotError(
`the derived store path ${target} is the Brain/ directory itself; refusing before ` +
"anything is touched. Point search_db_path at a file, directly under Brain/ or " +
"outside it, then restore again",
runId,
);
}
if (placement.kind !== "nested") return;
throw new BrainSnapshotError(
`the derived store ${target} is inside Brain/${placement.top}, which a restore replaces ` +
"as a whole; refusing before anything is touched. Move the store (search_db_path) " +
"directly under Brain/ or outside it, then restore again",
runId,
);
}
/** A decompressed and verified store, waiting beside its live target. */
interface StagedDerivedStore {
/** The live store path the staged file is renamed over. */
readonly target: string;
/** The staged file, on the target's filesystem. */
readonly incoming: string;
/** Remove the staged file if it is still there. Best-effort. */
discard(): void;
}
/**
* Decide what the restore does with the derived store, and when the
* manifest says the snapshot carried one, decompress and verify it next
* to the live store - without touching the live store.
*
* Three answers, and the caller reports whichever it gets rather than
* leaving any of them implicit:
*
* - the manifest records inclusion: decompress and verify here, swap
* later in {@link commitDerivedStoreSwap};
* - the manifest records an exclusion: nothing is touched, and the
* named reason travels back;
* - there is no record at all: coverage is UNKNOWN, and whether a
* sibling archive is on disk travels back with it. A restore must not
* guess in either direction here - replacing a live store on a hunch
* destroys embeddings the archive never held, and reporting
* "excluded" would claim a check that never ran. The archive probe is
* what lets the caller say WHY the record is missing instead of
* asserting the snapshot is older than the feature.
*
* Runs without the writer lock: the staged file has a name of its own
* beside the store, and the lock is held only across the renames. The
* recovery point the caller takes after this step snapshots the live
* store under that same lock, so holding it here would deadlock.
*/
function stageDerivedStore(
vault: string,
target: string,
runId: string,
): { readonly result: RestoreDerivedStoreResult; readonly staged: StagedDerivedStore | null } {
const record = readManifestSidecar(vault, runId)?.derived_store ?? null;
// Probed once, for every branch, and deliberately independent of the
// record: it is the only evidence that separates a snapshot with no
// record from a snapshot whose record was lost after the archive landed.
const storeArchivePresent = existsSync(snapshotStorePath(vault, runId));
if (record === null) {
return {
result: Object.freeze({
replaced: false,
coverage_known: false,
path: null,
exclusion_reason: null,
store_archive_present: storeArchivePresent,
}),
staged: null,
};
}
if (!record.included) {
return {
result: Object.freeze({
replaced: false,
coverage_known: true,
path: null,
exclusion_reason: record.exclusion_reason,
store_archive_present: storeArchivePresent,
}),
staged: null,
};
}
const archive = snapshotStorePath(vault, runId);
if (!existsSync(archive)) {
// Unreachable through `restoreSnapshot`, which refuses this in its
// pre-flight. Kept because a store swap must never begin against an
// archive that is not there.
throw new BrainSnapshotError(
`manifest records a derived-store archive but ${archive} is absent`,
runId,
);
}
if (record.archive_sha256 !== null) {
const actual = sha256Hex(readFileSync(archive));
if (actual !== record.archive_sha256) {
throw new BrainSnapshotError(
`the derived-store archive ${archive} does not match the digest its manifest recorded ` +
`(${record.archive_sha256}, found ${actual}); refusing before the Brain tree is touched`,
runId,
);
}
}
mkdirSync(dirname(target), { recursive: true });
// A name of this process's own: two restores staging at once must not
// write into one file, and neither is under the lock yet.
const incoming = `${target}${RESTORE_STAGING_SUFFIX}-${process.pid}-${randomBytes(4).toString("hex")}`;
const discard = (): void => {
for (const path of [incoming, ...WAL_SIBLING_SUFFIXES.map((s) => `${incoming}${s}`)]) {
try {
rmSync(path, { force: true });
} catch {
// best-effort: a leftover staged file is never opened as the store
}
}
};
try {
decompressArchiveTo(archive, incoming, runId);
try {
assertStoreIsSound(incoming, runId);
} catch (err) {
throw new BrainSnapshotError(
`the archived derived store failed verification after decompression: ${errorText(err)}`,
runId,
{ cause: err },
);
}
// Opening it for the check must not leave journal siblings to travel
// with the rename; the decompressed file is complete on its own.
for (const sidecar of WAL_SIBLING_SUFFIXES) rmSync(`${incoming}${sidecar}`, { force: true });
} catch (err) {
discard();
if (err instanceof BrainSnapshotError || err instanceof BrainSnapshotToolingMissingError) {
throw err;
}
throw new BrainSnapshotError(
`failed to stage the derived store beside ${target}: ${errorText(err)}`,
runId,
{ cause: err },
);
}
return {
result: Object.freeze({
replaced: true,
coverage_known: true,
path: target,
exclusion_reason: null,
store_archive_present: true,
}),
staged: Object.freeze({ target, incoming, discard }),
};
}
/**
* The store's writer lock, taken before the Brain swap starts. Holding it
* across the renames is what stops an index run from writing into the
* file being replaced; taking it BEFORE the first rename is what turns
* contention into a refusal rather than a half-finished restore.
*
* The acquisition is the only place a {@link StoreLockToken} is minted:
* the recovery point skips its own acquire for that token, live and on
* the same path, and for nothing that merely names the path. The token
* stops counting before the lock is released, so a reference kept past
* the restore never stands in for a lock nobody holds.
*/
function acquireStoreLockForRestore(
target: string,
runId: string,
): {
readonly token: StoreLockToken;
readonly lease: WriterLease;
readonly release: () => void;
} {
let lease: WriterLease;
try {
lease = acquireWriterLeaseSync(target);
} catch (err) {
throw new BrainSnapshotError(
`cannot take the derived store's writer lock (${errorText(err)}); refusing before the ` +
"Brain tree is touched",
runId,
{ cause: err },
);
}
const token = new StoreLockToken(STORE_LOCK_TOKEN_MINT, target);
liveStoreLockTokens.set(token, target);
return {
token,
lease,
release: (): void => {
liveStoreLockTokens.delete(token);
lease.release();
},
};
}
/** Refuse the swap, with nothing touched, when the restore's lease was lost. */
function assertStoreLeaseHeld(lease: WriterLease, runId: string): void {
try {
lease.assertHeld();
} catch (err) {
throw new BrainSnapshotError(
`the derived store's writer lock was lost before the swap (${errorText(err)}); ` +
"refusing before the Brain tree is touched",
runId,
{ cause: err },
);
}
}
/**
* Refuse a swap whose recovery point archived a different store than the
* one the swap replaces.
*
* The restore writes the snapshot's store to the live target it resolved
* (an explicit `derivedStorePath`, or the search layer's answer). A
* recovery point that resolved its own source - the configured store
* rather than the override - would archive one file while the swap
* discards another, and the discarded store would have no copy anywhere.
* The manifest's `source_path` is what the recovery point actually
* archived (or declined to), so that is what is compared. A recovery
* point with no readable manifest is left to the recoverability verdict,
* which already reads it as unproven.
*/
function assertRecoveryPointCoversTarget(
vault: string,
evidence: RecoveryPointEvidence,
target: string,
runId: string,
): void {
const record = readManifestSidecar(vault, evidence.runId)?.derived_store ?? null;
if (record === null) return;
if (record.source_path !== target) {
throw new BrainSnapshotError(
`the recovery point ${evidence.runId} recorded the derived store at ${record.source_path}, ` +
`but this restore replaces ${target}; refusing before the Brain tree is touched, so the ` +
"store being discarded is not left without a copy",
runId,
);
}
if (!record.included && liveStoreWorthPreserving(target)) {
throw new BrainSnapshotError(
`the recovery point ${evidence.runId} did not include the derived store at ${target}, ` +
"which this restore replaces; refusing before the Brain tree is touched, so the store " +
"being discarded is not left without a copy",
runId,
);
}
}
/**
* Release the writer lock {@link acquireStoreLockForRestore} took.
*
* A failed release must not hide what happened before it. When the
* restore already failed, that error stays the one thrown and the
* release failure is appended to its message as a second line. When the
* swap committed, the live tree and store are the snapshot's now, and the
* error says so and names the recovery point, rather than reading like a
* restore that changed nothing.
*/
function releaseStoreLockAfterRestore(
release: () => void,
runId: string,
committed: boolean,
primary: { readonly err: unknown } | null,
recoveryPoint: RecoveryPointEvidence | null,
): void {
try {
release();
return;
} catch (releaseErr) {
const what = `releasing the derived store's writer lock failed (${errorText(releaseErr)})`;
if (primary !== null) {
// The primary error is rethrown by the caller's catch once this
// finally returns; it only gains the release failure as a line.
if (primary.err instanceof Error) {
primary.err.message += `\nalso: ${what}`;
} else {
// A non-Error throw has no message to extend; the release failure
// still must not vanish.
console.error(`warning: ${what}`);
}
return;
}
if (committed) {
const recovery =
recoveryPoint === null
? "no recovery point was taken for this restore"
: `the recovery point ${recoveryPoint.runId} (${recoveryPoint.path}) holds the ` +
`pre-restore state; \`o2b brain rollback ${recoveryPoint.runId}\` puts it back`;
throw new BrainSnapshotError(
`the restore committed: Brain/ and the derived store now hold snapshot ${runId}, but ` +
`${what}; the lock is stale-released after ${resolveWriterLockStaleMs()} ms; ${recovery}`,
runId,
{ cause: releaseErr },
);
}
// Unreachable: nothing failed and nothing committed means the try
// block did not finish, which is a throw, which is `primary`.
throw new BrainSnapshotError(`restore: ${what}`, runId, { cause: releaseErr });
}
}
/**
* A store swap that failed between its two renames and could not put the
* outgoing store back: the live path has no store, and the pre-restore
* one is at the `.bak` path.
*/
class StoreSwapTornError extends Error {
constructor(message: string, options?: { readonly cause?: unknown }) {
super(message, options);
this.name = "StoreSwapTornError";
}
}
/**
* Rename the staged store over the live one. The caller holds the writer
* lock.
*
* The same swap discipline `reindexVault` uses: the outgoing file is kept
* as `.bak`, then a single atomic rename puts the replacement in place.
* If that second rename fails, the `.bak` is renamed back, so the live
* store is untouched; a {@link StoreSwapTornError} means even that failed.
*
* A rollback journal (`-journal`, DELETE journal mode, which the store
* uses on remote filesystems and SQLite leaves hot after a crash) belongs
* to the database it was written for and travels with it: it is moved to
* `.bak-journal` right after the outgoing store moves to `.bak`, and back
* before the store on every undo. Left at the live path, SQLite would
* replay the old database's journal over the replacement on its next
* open, which is the database/journal mismatch it documents as a cause of
* corruption. A journal with no live store beside it moves the same way.
* A journal that cannot be moved undoes the swap like any other failed
* step, so the replacement never lands beside it.
*
* The WAL siblings of the OUTGOING file are removed after the swap. The
* decompressed archive came from `VACUUM INTO` and is a complete
* database on its own; an orphan `-wal` from the file it replaced would
* make the next open fail with SQLITE_IOERR_SHORT_READ, which is the
* exact failure `consolidateWal` exists to prevent on the reindex path.
* A removal that still fails after the Windows retry budget (a reader
* holding the file) is not a warning: the swap is undone, the new store
* back at its staging name and the `.bak` back at the live path, and the
* error is thrown so the caller undoes the Brain/ renames too.
*/
function commitDerivedStoreSwap(store: StagedDerivedStore): void {
const { target, incoming } = store;
const outgoing = `${target}${RESTORE_BACKUP_SUFFIX}`;
const journal = `${target}${ROLLBACK_JOURNAL_SUFFIX}`;
const outgoingJournal = `${outgoing}${ROLLBACK_JOURNAL_SUFFIX}`;
const moved: MovedOutgoing = { target, outgoing, store: false, journal: false };
try {
// A `.bak` and its journal from an earlier swap are replaced together:
// the earlier journal must not end up beside the store moved aside now.
rmSync(outgoing, { force: true });
rmSync(outgoingJournal, { force: true });
if (existsSync(target)) {
try {
renameWithRetry(target, outgoing);
} catch (err) {
throw new Error(
`could not move the live store ${target} aside to ${outgoing}: ${errorText(err)}; ` +
"the live store was not touched",
{ cause: err },
);
}
moved.store = true;
}
if (existsSync(journal)) {
try {
renameWithRetry(journal, outgoingJournal);
} catch (err) {
throw new Error(
`could not move the rollback journal ${journal} aside to ${outgoingJournal}: ` +
`${errorText(err)}; the restored store must not open beside it`,
{ cause: err },
);
}
moved.journal = true;
}
renameWithRetry(incoming, target);
} catch (err) {
putBackOutgoingStore(moved, errorText(err), err);
throw err;
}
try {
removeWalSiblings(target);
} catch (err) {
const what =
`could not remove ${errorText(err)} beside the restored store, and an orphan ` +
"write-ahead log would corrupt the next open";
try {
renameWithRetry(target, incoming);
} catch (backErr) {
throw new StoreSwapTornError(
`${what}; the restored store could not be moved back off ${target} either ` +
`(${errorText(backErr)})` +
(moved.store ? `, and the pre-restore store is at ${outgoing}` : "") +
(moved.journal ? `, its rollback journal at ${outgoingJournal}` : ""),
{ cause: err },
);
}
putBackOutgoingStore(moved, what, err);
throw new Error(
`${what}; the store swap was undone` +
(moved.store
? ", the pre-restore store is back in place"
: ", and no store was live before it"),
{ cause: err },
);
}
}
/** What {@link commitDerivedStoreSwap} has moved aside so far. */
interface MovedOutgoing {
readonly target: string;
readonly outgoing: string;
/** The live store is at `.bak`. */
store: boolean;
/** The live store's rollback journal is at `.bak-journal`. */
journal: boolean;
}
/**
* Put what a failed swap step moved aside back at the live path, or throw
* a {@link StoreSwapTornError} naming where it was left.
*
* The journal goes back first. A crash between the two renames then
* leaves the journal at the live path with no store and the store at
* `.bak`, which the crash-recovery preamble of every `Store.open` puts
* back beside it; the other order would leave the old store live without
* the journal it needs to roll back.
*/
function putBackOutgoingStore(moved: MovedOutgoing, what: string, cause: unknown): void {
const { target, outgoing } = moved;
if (moved.journal) {
const journal = `${target}${ROLLBACK_JOURNAL_SUFFIX}`;
const outgoingJournal = `${outgoing}${ROLLBACK_JOURNAL_SUFFIX}`;
try {
renameWithRetry(outgoingJournal, journal);
} catch (backErr) {
throw new StoreSwapTornError(
`${what}; the rollback journal of the pre-restore store could not be renamed back ` +
`(${errorText(backErr)}) and is at ${outgoingJournal}` +
(moved.store ? `; the pre-restore store is at ${outgoing}` : ""),
{ cause },
);
}
}
if (!moved.store) return;
try {
renameWithRetry(outgoing, target);
} catch (backErr) {
throw new StoreSwapTornError(
`${what}; the pre-restore store could not be renamed back either ` +
`(${errorText(backErr)}) and is at ${outgoing}, where the next store open restores it`,
{ cause },
);
}
}
/**
* Remove the WAL siblings the outgoing store left at `target`, throwing
* the path that could not be removed.
*
* `-shm` goes first: it is only an index of the `-wal` and SQLite
* rebuilds it, so when the `-wal` removal then fails, the swap can still
* be undone in full - the pre-restore store returns beside its own log.
*/
function removeWalSiblings(target: string): void {
for (const sidecar of ["-shm", "-wal"] as const) {
const path = `${target}${sidecar}`;
try {
unlinkWithRetry(path);
} catch (err) {
if ((err as NodeJS.ErrnoException | null)?.code === "ENOENT") continue;
throw new Error(`${path} (${errorText(err)})`, { cause: err });
}
}
}
/** Staging name of the store being restored, beside its destination. */
const RESTORE_STAGING_SUFFIX = ".restore";
/**
* The outgoing store, kept under the SAME `.bak` name the reindex swap
* uses - so the crash-recovery preamble every `Store.open` runs restores
* it if a rollback dies between the two renames.
*/
const RESTORE_BACKUP_SUFFIX = ".bak";
/** SQLite's rollback journal of a database file (DELETE journal mode). */
const ROLLBACK_JOURNAL_SUFFIX = "-journal";
/** SQLite's write-ahead-log siblings of a database file. */
const WAL_SIBLING_SUFFIXES: ReadonlyArray<string> = Object.freeze(["-wal", "-shm"]);
/** Decompress a snapshot archive to `outPath`, probing the magic bytes. */
function decompressArchiveTo(archive: string, outPath: string, runId: string): void {
const tools = detectTooling();
const compressor = detectArchiveCompression(archive);
if (compressor === "zstd" && !tools.zstd) {
throw new BrainSnapshotToolingMissingError(
"zstd",
"archive is zstd-compressed; install zstd to restore it.",
);
}
if (compressor === "zstd") {
// `zstd -o` opens the destination itself and refuses an existing one.
runCompressor("zstd", ["-d", "-q", "-o", outPath, archive], runId);
return;
}
// gzip is inflated in-process (no `gzip` binary on native Windows);
// `wx` keeps the destination exclusive.
let bytes: Buffer;
try {
bytes = gunzipSync(readFileSync(archive));
} catch (err) {
throw new BrainSnapshotError(
`gzip decompress failed: ${(err as Error).message ?? String(err)}`,
runId,
);
}
writeArchiveExclusive(outPath, bytes, runId);
}
// ----- Helpers -------------------------------------------------------------
function detectArchiveCompression(archive: string): "zstd" | "gzip" {
const buf = Buffer.alloc(4);
const fd = openSync(archive, "r");
try {
readSync(fd, buf, 0, 4, 0);
} finally {
closeSync(fd);
}
// zstd magic: 0x28 0xB5 0x2F 0xFD (little-endian view of 0xFD2FB528).
if (buf[0] === 0x28 && buf[1] === 0xb5 && buf[2] === 0x2f && buf[3] === 0xfd) {
return "zstd";
}
// gzip magic: 0x1F 0x8B
if (buf[0] === 0x1f && buf[1] === 0x8b) {
return "gzip";
}
// Default to zstd since that's our preferred writer — restore will
// fail loudly through the zstd subprocess if the bytes don't match.
return "zstd";
}
function countFiles(path: string): number {
try {
const st = statSync(path);
if (st.isFile()) return 1;
if (!st.isDirectory()) return 0;
} catch {
return 0;
}
let count = 0;
for (const entry of readdirSync(path, { withFileTypes: true })) {
const sub = join(path, entry.name);
if (entry.isDirectory()) {
count += countFiles(sub);
} else if (entry.isFile()) {
count++;
}
}
return count;
}