Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
fix(store): keep the writer lock fresh through long synchronous work (#303)
Serg Eroshea4ab96254654
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3d3b77dd..c56ba621 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **The search index writer lock stays fresh through long synchronous work, and a writer that lost it no longer touches its successor's lock.** The lock's heartbeat ran on the event loop, so a holder blocked in synchronous work - a snapshot's `tar` or `zstd` run, the store archive's integrity scan and `VACUUM INTO`, the restore that holds the lock from its recovery point through the swap, an FTS rebuild, a migration, the index walk between embedding batches - left the lock untouched for longer than its 60-second stale window on a large vault, and a second writer took it over while the first was still writing. Each of those steps now keeps the lock fresh from a worker thread for a stated bound (the tool's kill timeout, or fifteen minutes for an in-process step without one), and the index walk refreshes it as it goes; past the bound the worker stops, so a holder wedged inside the step still goes stale and is taken over one window later, and a crashed or killed holder goes stale as before. Every refresh and the release first check that the lock is still the one this holder stamped (its mtime, which carries a random sub-millisecond fraction, and its inode): a lock another writer took over is never refreshed or removed, also at process exit, and neither is one so close to its stale window that a contender may be taking it over at that moment. Each protected step refuses with `INDEX_LOCKED` when the lock is not provably this holder's before it starts and again when it ends, and the holder's commits - the reindex swap, right before its first rename, and the restore swap, before the Brain renames and again before the store replacement, undoing the renames when the lock was lost in between - refuse with nothing touched instead of replacing what the other writer committed, and a write-mode store stops at the moment it sees the loss: its connection turns read-only and the loss is raised as `INDEX_LOCKED`, which from the heartbeat ends the process as the old lock library did. The brain store catch-up and the link and relation passes of an index run keep the lock fresh too. The on-disk lock is unchanged, so older releases on the same index exclude and are excluded by it; an older release still removes the lock without checking whose it is, which this release detects and refuses its commit on. `OPEN_SECOND_BRAIN_WRITER_LOCK_STALE_MS` sets the stale window (one minute by default, at most one day); the lock beats at least every 20 seconds whatever the window, so an older release never sees a live lock as stale, and a value below one minute is unsafe while an older release shares the index and is warned about.+- **Two Windows-flaky tests are fixed at their cause.** The hung and TERM-ignoring version-child tests read the child's pid from a shell `$$` that the 150 ms budget could cut off mid-write, and that is not the native pid on Windows; the child is now a Bun process that writes its native pid atomically, and a run whose child the budget cut off before it reported is repeated at a doubled budget instead of failing on a pid it never wrote. The concurrent `o2b store init` test started six full CLI processes at once and raced their startup against its 30-second budget on a loaded runner; six processes now load the init verb's handler, report ready and start together on one signal, so the test races the store creation itself and process start is no longer part of the race, and a racer never outlives the test.- **A failed snapshot restore no longer leaves `Brain/` replaced beside the old derived store.** `restoreSnapshot` replaced the live `Brain/` tree before it decompressed the archived derived store, so a `zstd` failure or timeout, a corrupt payload or a full disk at that point stopped the rollback with the tree already restored and the store not. Every step that can fail now runs before anything live changes: the archive is extracted and the `Brain/` entries staged and verified on the live tree's filesystem (under `.snapshots/` when it shares a device with `Brain/`), the store archive is checked against the digest its manifest recorded, decompressed beside the live store and integrity-checked, and only then is the store's writer lock acquired and the recovery point taken, so lock contention also refuses with nothing touched. The lock is held from before the recovery point through the swap, and the recovery point's store archive is written under it, so no write can land between the two and the recovery point holds every commit the swap replaces. The recovery point skips its own lock only for an opaque token minted where the restore acquires the lock, while that lock is held and on the same store; a string naming the store's path, an object that merely looks like a token or a token whose lock was already released takes the lock as usual. The recovery point archives exactly the store the swap replaces, under the restore's coverage policy, also with a `derivedStorePath` override, and the restore refuses before the swap when the recovery point's manifest records a different store. A lock release that fails after the swap committed is reported as a committed restore that names the recovery point, and one that fails after an earlier error is added to that error instead of replacing it. The swap itself is a run of renames: live entries are moved aside, staged ones moved in, then the staged store renamed over the live one, and a rename that fails is undone in reverse. If the undo fails too, the error says how many entries were swapped, where the pre-restore entries were kept and which recovery point puts the tree back. Excluded entries, the preserved `.snapshots/`, manifest-driven store inclusion and the reported restore states are unchanged. A rollback journal (`-journal`, the DELETE journal mode the store uses on remote filesystems) that the old store left beside it, hot after a crash, moves with the outgoing store to `.bak-journal` and back on every undo, for a store inside `Brain/` or outside it, so SQLite never replays it over the replacement; a journal that cannot be moved undoes the whole restore. The old store's `-wal`/`-shm` are removed after the store swap with the Windows retry, and a removal that still fails undoes the whole restore instead of reporting success beside an orphan write-ahead log; when no store was live before the restore, the error says so instead of claiming a pre-restore store was put back. A restore staging directory left directly under `Brain/` (a restore killed mid-swap, when `.snapshots/` is on another device) is skipped by the archiver, the manifest walker and the restore, so it is never archived or swapped back as Brain content. A sound store the restore replaces is always archived in its recovery point, whatever the coverage policy or size cap, and the restore refuses when the recovery point records it as left out; an absent store or one that fails its integrity scan does not block the restore that replaces it, also when the vault's policy turns store coverage on: the recovery point then leaves that store out explicitly instead of refusing it, so the restore recreates or heals it. A derived store that lives directly under `Brain/` (a `search_db_path` or `derivedStorePath` such as `Brain/index.sqlite`) is kept out of the `Brain/` tree swap together with its journals, writer lock, `.bak` and the restore's staged file, so the store swap finds its staged file and the restore completes; the placement is judged for the directory entry the store swap renames, not the file a symlink there points to, so a `Brain/index.sqlite` that links to a database outside `Brain/` is restored at the link instead of having its staged replacement moved away by the tree swap; the raw copy of the store that the `Brain/` archive carries as a top-level entry is not restored over it, so the store is written once, from the store archive, or left untouched when the snapshot did not cover it. A store nested deeper inside a `Brain/` directory the swap replaces as a whole is refused before anything is extracted or touched, with an error that names the directory.- **The concurrent appenders test no longer fails on a slow runner with `store lock busy`.** Six workers appending back to back can outlast the store lock's wait budget on a loaded Windows runner. That refusal is documented as retryable and arrives before the record is written, so each worker now retries a refused append with a short randomized backoff within a bound; the assertions on record count, unique ids and clocks, the single blob and interleaving are unchanged.- **A snapshot or restore can no longer hang forever on a stuck `tar` or `zstd`.** The snapshot module is synchronous, so a child that never exited froze the hook, CLI verb or MCP call that took the recovery point, with nothing able to interrupt it. Every `tar` and `zstd` run of a snapshot, a derived-store archive and a restore now has a timeout (fifteen minutes by default, `OPEN_SECOND_BRAIN_SNAPSHOT_TIMEOUT_MS` to change it); a run that hits it is killed and the snapshot fails with an error that names the tool and the bound, leaving no partial archive.diff --git a/bun.lock b/bun.lockindex b2ba989c..098f220c 100644--- a/bun.lock+++ b/bun.lock@@ -6,6 +6,7 @@"name": "open-second-brain","dependencies": {"proper-lockfile": "^4.1.2",+ "signal-exit": "^3.0.7",},"devDependencies": {"@types/bun": "^1.4.4",diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex bcb15a5e..3180b4a3 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -1809,15 +1809,16 @@ For the two verbs that do hold a handle, a second interrupt is not intercepted a## Lock and cache overrides (since v1.61.0)-Four environment variables tune how long a writer waits for a shared lock,-how long a snapshot tool may run, and where the machine-local dedup cache-lives. None of them has a+Five environment variables tune how long a writer waits for a shared lock,+when the search index writer lock counts as abandoned, how long a snapshot+tool may run, and where the machine-local dedup cache lives. None of them has a`_brain.yaml` key: they describe the host, not the vault.| Variable | Default | Effect || ----------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ || `OPEN_SECOND_BRAIN_LOCK_WAIT_MS` | `5000` | How long a write to shared ingest state (content manifest, plan checkpoint, session ledger, git record store) waits for its lock before it is refused with `ELOCKED`. A whole number of milliseconds; `0` means one attempt; anything else is an error. The one-second interactive wait is not affected. See [Source pipeline integrity](#source-pipeline-integrity-and-operator-tooling-since-v1340). || `OPEN_SECOND_BRAIN_SNAPSHOT_TIMEOUT_MS` | `900000` (fifteen minutes) | How long one `tar` or `zstd` run of a Brain snapshot, a derived-store archive or a restore may take before it is killed and the snapshot fails naming the tool. A whole number of milliseconds above zero; anything else is an error. |+| `OPEN_SECOND_BRAIN_WRITER_LOCK_STALE_MS` | `60000` (one minute) | The stale window of the search index writer lock: a lock whose holder has not refreshed it for this long counts as abandoned and the next writer takes it over. A live holder refreshes it every third of the window, from a worker thread while it is blocked in a bounded synchronous step (a `tar` or `zstd` run, an integrity scan, `VACUUM INTO`, an FTS rebuild, a migration), so a long snapshot, restore or rebuild keeps it; a holder stuck in such a step past its bound stops refreshing and is taken over one window later. Every process sharing an index should use the same value. Releases before this one always use one minute and ignore the variable, so a value below `60000` lets this process take over an older release's live lock: do not set one while an older release shares the index (a process that reads such a value warns about it once). A whole number of milliseconds from `1000` to `86400000` (one day); anything else is an error. || `OPEN_SECOND_BRAIN_DEDUP_CACHE_DIR` | the user cache directory, `open-second-brain/dedup-index/` | Where the signal dedup index cache is kept, one `<vault-digest>.json` per vault, outside the vault. || `OPEN_SECOND_BRAIN_DEDUP_CACHE` | on | `0` turns the dedup index cache off; every capture then walks the inbox, `processed/` and `archived/` in full. |diff --git a/package.json b/package.jsonindex ea4eb6ab..c285ad32 100644--- a/package.json+++ b/package.json@@ -61,7 +61,8 @@"prepare": "bun scripts/prepare.ts || exit 0"},"dependencies": {- "proper-lockfile": "^4.1.2"+ "proper-lockfile": "^4.1.2",+ "signal-exit": "^3.0.7"},"devDependencies": {"@types/bun": "^1.4.4",diff --git a/src/cli/search/verbs/check.ts b/src/cli/search/verbs/check.tsindex eb0ccc70..afde7a33 100644--- a/src/cli/search/verbs/check.ts+++ b/src/cli/search/verbs/check.ts@@ -93,6 +93,10 @@ import type {} from "../../../core/search/index.ts";import { PROVIDER_PROBE } from "../../../core/search/provider-probe.ts";import { acquireWriterLock } from "../../../core/search/store.ts";+import {+ keepWriterLeasesFresh,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+} from "../../../core/search/store/writer-lock.ts";import { runIntegrityCheck } from "../../../core/search/store/lifecycle.ts";import { nowIso } from "../../../core/search/store/sql.ts";import {@@ -342,7 +346,11 @@ async function scanIntegrity(config: ResolvedSearchConfig): Promise<IntegrityRepconst priorCheck = readPriorCheck(db);const startedAt = performance.now();- const verdict = runIntegrityCheck(db);+ // Held under the writer lock, which is kept fresh off this blocked+ // thread while the scan reads a large index.+ const verdict = keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>+ runIntegrityCheck(db),+ );const elapsedMs = Math.round(performance.now() - startedAt);// `quick_check` passes any well-formed sqlite file, including onediff --git a/src/core/brain/operator-snapshot.ts b/src/core/brain/operator-snapshot.tsindex 641358ed..fca1d1ee 100644--- a/src/core/brain/operator-snapshot.ts+++ b/src/core/brain/operator-snapshot.ts@@ -430,12 +430,11 @@ function lockAgeMs(dbPath: string, now: Date): number | null {}/**- * The lock directory `proper-lockfile` creates for an index path: a bare- * `${dbPath}.lock` directory whose only metadata is its mtime (`getLockFile`- * in `proper-lockfile/lib/lockfile.js`; `writer-lock.ts` takes it with- * `mkdir`). Spelled here because the age reads that directory's mtime, and- * the state probe (`isWriterLockHeld`) asks the same path behind the same- * library.+ * The writer lock directory for an index path: a bare `${dbPath}.lock`+ * directory whose only metadata is its mtime (the `proper-lockfile` on-disk+ * protocol, which `writer-lock.ts` keeps; it takes the lock with `mkdir`).+ * Spelled here because the age reads that directory's mtime, and the state+ * probe (`isWriterLockHeld`) asks the same path.*/const WRITER_LOCK_SUFFIX = ".lock";diff --git a/src/core/brain/snapshot.ts b/src/core/brain/snapshot.tsindex 094ae001..80c13e92 100644--- a/src/core/brain/snapshot.ts+++ b/src/core/brain/snapshot.ts@@ -116,7 +116,14 @@ import { classifyRecoverability, type RecoverabilityVerdict } from "./gates/recoimport { sha256Hex } from "../integrity/digest.ts";import { resolveConfiguredIndexPath } from "../search/paths.ts";import { runIntegrityCheck } from "../search/store/lifecycle.ts";-import { acquireWriterLockSync, WRITER_LOCK_STALE_MS } from "../search/store/writer-lock.ts";+import {+ acquireWriterLeaseSync,+ acquireWriterLockSync,+ keepWriterLeasesFresh,+ resolveWriterLockStaleMs,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+ type WriterLease,+} from "../search/store/writer-lock.ts";import { appendLogEvent } from "./log.ts";import {buildManifest,@@ -1158,22 +1165,48 @@ function coverDerivedStore(? (): void => {}: acquireWriterLockSync(sourcePath);try {- assertStoreIsSound(sourcePath, runId);+ // Everything under the lock blocks this thread - the integrity scan,+ // `VACUUM INTO`, the compressor - for minutes on a large store, so the+ // lock is kept fresh off the thread meanwhile, bounded by the scan and+ // copy budget plus the compressor's own timeout.+ let published = false;+ try {+ keepWriterLeasesFresh(+ WRITER_LEASE_BLOCKING_BUDGET_MS + resolveSnapshotToolTimeoutMs(),+ () => {+ assertStoreIsSound(sourcePath, runId);++ // Measured against the LIVE size, which is what the operator's+ // ceiling is expressed in and what they can check with `ls`. The+ // compressed archive is smaller, but refusing on a number nobody can+ // predict before the work is done would be a worse contract.+ if (liveSize > policy.maxBytes) {+ throw new BrainSnapshotStoreError(+ `store is ${liveSize} bytes, over the ${policy.maxBytes}-byte ceiling ` ++ `(snapshots.derived_store_max_bytes)`,+ runId,+ SNAPSHOT_STORE_EXCLUSION.over_size_ceiling,+ );+ }- // Measured against the LIVE size, which is what the operator's- // ceiling is expressed in and what they can check with `ls`. The- // compressed archive is smaller, but refusing on a number nobody can- // predict before the work is done would be a worse contract.- if (liveSize > policy.maxBytes) {- throw new BrainSnapshotStoreError(- `store is ${liveSize} bytes, over the ${policy.maxBytes}-byte ceiling ` +- `(snapshots.derived_store_max_bytes)`,+ writeStoreArchive(sourcePath, archivePath, tools, runId);+ published = true;+ },+ );+ } catch (err) {+ // An archive published before the lock turned out lost (the region's+ // exit check) is this run's own: the name was free at the check above.+ // It is removed, so the run leaves nothing a retry under the same id+ // would be refused for.+ if (published) rmSync(archivePath, { force: true });+ if (err instanceof BrainSnapshotError) throw err;+ throw new BrainSnapshotError(+ `the derived store's writer lock was lost while its archive was written ` ++ `(${(err as Error).message ?? String(err)}); no store archive was kept`,runId,- SNAPSHOT_STORE_EXCLUSION.over_size_ceiling,+ { cause: err },);}-- writeStoreArchive(sourcePath, archivePath, tools, runId);} finally {release();}@@ -1645,11 +1678,18 @@ function spawnSnapshotTool(runId: string,): SpawnSyncReturns<Buffer> {const timeoutMs = resolveSnapshotToolTimeoutMs();- const r = spawnSync(cmd, [...args], {- ...options,- timeout: timeoutMs,- killSignal: "SIGKILL",- }) as SpawnSyncReturns<Buffer>;+ // A writer lock held across this run (the restore's, a store archive's)+ // is kept fresh while the thread is blocked in the child, bounded by the+ // timeout the child is killed at plus one stale window for the kill.+ const r = keepWriterLeasesFresh(+ timeoutMs + resolveWriterLockStaleMs(),+ () =>+ spawnSync(cmd, [...args], {+ ...options,+ timeout: timeoutMs,+ killSignal: "SIGKILL",+ }) as SpawnSyncReturns<Buffer>,+ );if ((r.error as NodeJS.ErrnoException | undefined)?.code === "ETIMEDOUT") {throw new BrainSnapshotError(`${cmd} did not finish within ${timeoutMs} ms and was stopped; ` +@@ -2392,11 +2432,27 @@ export function restoreSnapshot(// The last moment at which the live state still exists. A throw// here propagates and nothing has been replaced yet, which is the// same abort-before-the-operation rule the destructive gate applies.- evidence =- opts.beforeDiscard?.({- storeTarget: store?.target ?? null,- heldStoreLock: lock?.token ?? null,- }) ?? null;+ //+ // The recovery point runs tar, zstd and a store archive synchronously+ // under the lock. Each of those keeps the lease fresh for its own+ // bound, and the rest of the callback (the manifest, the retention+ // pass) for this outer one; the bounds add up rather than nest, so a+ // wedged recovery point holds the lock at most for their sum and then+ // lets it go stale, never forever.+ try {+ evidence =+ keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>+ opts.beforeDiscard?.({+ storeTarget: store?.target ?? null,+ heldStoreLock: lock?.token ?? null,+ }),+ ) ?? null;+ } catch (err) {+ // A lease lost while the recovery point ran is reported as the+ // restore's own refusal, with nothing touched.+ if (lock !== null && lock.lease.lost) assertStoreLeaseHeld(lock.lease, runId);+ throw err;+ }if (store !== null && evidence !== null) {assertRecoveryPointCoversTarget(vault, evidence, store.target, runId);}@@ -2423,7 +2479,47 @@ export function restoreSnapshot(}}- const swap = swapBrainTree(staged, dirs.brain, liveEntries, runId, evidence);+ // The lease is proven once more right before the first rename. One+ // that another writer took over (the recovery point stalled past the+ // stale window) excludes nothing any more, and the swap would discard+ // whatever that writer committed.+ if (lock !== null) assertStoreLeaseHeld(lock.lease, runId);++ // The Brain/ renames run under the lease too, kept fresh off this+ // thread, and the lease is proven once more when they are done, right+ // before the store replacement: a lease lost during the renames+ // undoes them instead of replacing a store another writer now owns.+ const held: { swap?: ReturnType<typeof swapBrainTree> } = {};+ const tree = staged;+ const recovery = evidence;+ let swapStarted = false;+ const runSwap = (): void => {+ swapStarted = true;+ held.swap = swapBrainTree(tree, dirs.brain, liveEntries, runId, recovery);+ };+ try {+ if (lock === null) runSwap();+ else keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, runSwap);+ } catch (err) {+ // A failed rename has undone its own renames and says so, undo+ // result and recovery included; that error stands. Only when the+ // swap never started (the lease was found lost on entry) is this+ // the restore's own refusal, with nothing touched.+ if (held.swap === undefined) {+ if (!swapStarted && lock !== null && lock.lease.lost) {+ assertStoreLeaseHeld(lock.lease, runId);+ }+ throw err;+ }+ throw held.swap.fail(+ `the derived store's writer lock was lost during the Brain/ swap (${errorText(err)})`,+ err,+ { storeIntact: true },+ );+ }+ const swap = held.swap;+ if (swap === undefined)+ throw new BrainSnapshotError("restore: the Brain/ swap did not run", runId);if (store !== null) {try {commitDerivedStoreSwap(store);@@ -3139,10 +3235,14 @@ function stageDerivedStore(function acquireStoreLockForRestore(target: string,runId: string,-): { readonly token: StoreLockToken; readonly release: () => void } {- let unlock: () => void;+): {+ readonly token: StoreLockToken;+ readonly lease: WriterLease;+ readonly release: () => void;+} {+ let lease: WriterLease;try {- unlock = acquireWriterLockSync(target);+ lease = acquireWriterLeaseSync(target);} catch (err) {throw new BrainSnapshotError(`cannot take the derived store's writer lock (${errorText(err)}); refusing before the ` +@@ -3155,13 +3255,28 @@ function acquireStoreLockForRestore(liveStoreLockTokens.set(token, target);return {token,+ lease,release: (): void => {liveStoreLockTokens.delete(token);- unlock();+ lease.release();},};}+/** Refuse the swap, with nothing touched, when the restore's lease was lost. */+function assertStoreLeaseHeld(lease: WriterLease, runId: string): void {+ try {+ lease.assertHeld();+ } catch (err) {+ throw new BrainSnapshotError(+ `the derived store's writer lock was lost before the swap (${errorText(err)}); ` ++ "refusing before the Brain tree is touched",+ runId,+ { cause: err },+ );+ }+}+/*** Refuse a swap whose recovery point archived a different store than the* one the swap replaces.@@ -3244,7 +3359,7 @@ function releaseStoreLockAfterRestore(`pre-restore state; \`o2b brain rollback ${recoveryPoint.runId}\` puts it back`;throw new BrainSnapshotError(`the restore committed: Brain/ and the derived store now hold snapshot ${runId}, but ` +- `${what}; the lock is stale-released after ${WRITER_LOCK_STALE_MS} ms; ${recovery}`,+ `${what}; the lock is stale-released after ${resolveWriterLockStaleMs()} ms; ${recovery}`,runId,{ cause: releaseErr },);diff --git a/src/core/brain/sync-lockfile.ts b/src/core/brain/sync-lockfile.tsindex 582efdaf..3fbcb556 100644--- a/src/core/brain/sync-lockfile.ts+++ b/src/core/brain/sync-lockfile.ts@@ -48,9 +48,10 @@* SIGKILL never wedges the index for longer than that. This lock* deliberately has no such window, and the asymmetry is not an* oversight: the search lock can self-heal only because its holder- * HEARTBEATS - `acquireWriterLock` passes `update:- * WRITER_LOCK_HEARTBEAT_MS` so a live holder keeps refreshing the mtime,- * which is what makes "old mtime" mean "dead holder". A holder of THIS+ * HEARTBEATS - a live writer lease refreshes its mtime every+ * `WRITER_LOCK_HEARTBEAT_MS`, from a worker thread while it is blocked+ * in a bounded synchronous call, which is what makes "old mtime" mean+ * "dead or wedged holder". A holder of THIS* lock is inside a synchronous critical section: the single thread is* busy doing the very work the lock protects, so no timer can refresh* anything. Here an age threshold would only mean "held for longer thandiff --git a/src/core/search/brain-store/catch-up.ts b/src/core/search/brain-store/catch-up.tsindex 1b0f1613..40f7d1dd 100644--- a/src/core/search/brain-store/catch-up.ts+++ b/src/core/search/brain-store/catch-up.ts@@ -49,7 +49,12 @@ import {writeWatermark,type StoreWatermark,} from "../store/brain-records.ts";-import { isWriterLockHeld } from "../store/writer-lock.ts";+import {+ isWriterLockHeld,+ keepWriterLeasesFresh,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+ writerLeaseCheckIn,+} from "../store/writer-lock.ts";import { SearchError, type ResolvedSearchConfig } from "../types.ts";import { STORE_CATCH_UP_INTERVAL_ENV, STORE_CATCH_UP_INTERVAL_MS } from "./constants.ts";import { completePendingRecord, ingestRecord, removeRecord } from "./ingest.ts";@@ -259,6 +264,9 @@ function readOneShard(totals.shardsRead++;const seen = new Set<string>();for (const r of read.records) {+ // A shard read from its start can hold every record the store has; the+ // writer lock this pass runs under beats as it goes.+ writerLeaseCheckIn();const outcome = ingestRecord(index, config, r.record, shard, r.bodyState === "pending");// A copy refused for sitting in the other partition does not hold the// record: it must not keep an orphaned row alive.@@ -349,12 +357,29 @@ function dropVoidedRecords(return removed;}-/** Bring `index` level with the shards in `shards`; returns what changed. */+/**+ * Bring `index` level with the shards in `shards`; returns what changed.+ *+ * A shard read from its start is one synchronous read of the whole file,+ * which a check-in cannot interrupt, so the pass runs with the writer lock+ * it holds kept fresh off this thread; the check-ins below cover the rest.+ */function syncShards(index: Store,config: ResolvedSearchConfig,store: IndexedStore,shards: ReadonlyArray<ShardStat>,+): SyncTotals {+ return keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>+ syncShardsInPass(index, config, store, shards),+ );+}++function syncShardsInPass(+ index: Store,+ config: ResolvedSearchConfig,+ store: IndexedStore,+ shards: ReadonlyArray<ShardStat>,): SyncTotals {const db = index.brainRecordsConnection();const marks = readWatermarks(db);@@ -379,6 +404,7 @@ function syncShards(const eventsReadInFull = new Set<string>();const next: StoreWatermark[] = [];for (const shard of shards) {+ writerLeaseCheckIn();present.add(shard.key);const mark = marks.get(shard.key);if (mark !== undefined && sameStat(mark, shard)) continue;@@ -393,7 +419,9 @@ function syncShards(for (const key of vanished) {for (const id of recordIdsOfShard(db, key)) if (!orphans.has(id)) orphans.set(id, key);}+ writerLeaseCheckIn();const settled = settleOrphans(index, config, store, shards, orphans, readInFull);+ writerLeaseCheckIn();totals.removed += settled.removed;// A forgotten signal leaves the index with its tombstone (FORMAT.md,// "Removal"): after a pass that read one, that re-read an events sharddiff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex 18fbc25e..4d1d64b6 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -94,7 +94,12 @@ import { compareStamps, formatStampMismatch, type StampMismatch } from "../integimport { pathCovers } from "../vault-scope/defaults.ts";import {- acquireWriterLock,+ acquireWriterLease,+ keepWriterLeasesFresh,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+ writerLeaseCheckIn,+} from "./store/writer-lock.ts";+import {describeUnreadableIndex,embeddingAbiFixCommand,formatEmbedderRecordContradiction,@@ -517,6 +522,9 @@ async function indexIntoRun(// Per-file document upserts are transactional, so a tripped// guard leaves a consistent (partially refreshed) index.opts?.safeguard?.checkpoint();+ // The walk is synchronous between embed batches, so the lock's+ // event-loop heartbeat cannot fire during it; this beats instead.+ writerLeaseCheckIn();progress.advance(INDEX_STAGE.walk);// On-demand cancellation, same boundary as the deadline.throwIfAborted(opts?.signal, "index");@@ -541,6 +549,10 @@ async function indexIntoRun(}const content = readUtf8(file.absPath);+ // Reading (and below, chunking) one large note can take a while:+ // the lease is checked again before this file's first write, so a+ // write store that lost its lock meanwhile stops here.+ writerLeaseCheckIn();const contentHash = sha256Hex(content);// Fallback: fastpath missed (mtime or size changed). If@@ -583,6 +595,7 @@ async function indexIntoRun(// and the resolved event-time window below must come from the// same anchor answer.const eventAnchor = resolveEventAnchor(frontmatter, body);+ writerLeaseCheckIn();const docId = store.upsertDocument({path: file.relPath,title: chunkResult.title,@@ -717,6 +730,8 @@ async function indexIntoRun(opts?.onFile?.({ path: file.relPath, kind: "updated" });}} catch (e) {+ // A lost writer lock is the run's end, not this file's error.+ if (e instanceof SearchError && e.code === "INDEX_LOCKED") throw e;const msg = e instanceof Error ? e.message : String(e);stats.errors.push({ path: file.relPath, message: msg });opts?.onFile?.({ path: file.relPath, kind: "error", message: msg });@@ -724,6 +739,7 @@ async function indexIntoRun(}for (const [path] of existing) {+ writerLeaseCheckIn();if (!seen.has(path)) {markLinkResolutionPending();store.deleteDocument(path);@@ -760,10 +776,14 @@ async function indexIntoRun() {stats.linkResolutionSkipped = true;} else {- store.resolveLinkTargets();- // Alias post-pass (v7): exact path matches above always win; this- // only fills still-unresolved slash-free targets from doc_aliases.- stats.aliasResolved = store.resolveAliasTargets();+ // Whole-table statements that block for as long as the index is+ // large: the writer lock is kept fresh off this thread meanwhile.+ stats.aliasResolved = keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () => {+ store.resolveLinkTargets();+ // Alias post-pass (v7): exact path matches above always win; this+ // only fills still-unresolved slash-free targets from doc_aliases.+ return store.resolveAliasTargets();+ });store.setState(LINK_RESOLUTION_PENDING_STATE_KEY, "0");}@@ -781,7 +801,9 @@ async function indexIntoRun(stats.errors.push({ path: "Brain/_brain.yaml", message: `schema pack unreadable: ${msg}` });}stats.relationViolations = Object.freeze(- store.recomputeRelationConstraintFlags(pack?.link_constraints ?? {}),+ keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>+ store.recomputeRelationConstraintFlags(pack?.link_constraints ?? {}),+ ),);// Tier-guard post-pass: for each changed framework-kind doc,@@ -1509,7 +1531,7 @@ async function reindexInto(// in-progress staging DB and then having its empty seed swapped over the// live index (INDEX_UNREADABLE, silent data loss). The staging-DB opens// below lock a different path (`.new`), so there is no self-deadlock.- const release = await acquireWriterLock(config.dbPath);+ const lease = await acquireWriterLease(config.dbPath);progress.advance(INDEX_STAGE.lock);try {if (stillCurrent !== null && stillCurrent()) return null;@@ -1611,7 +1633,18 @@ async function reindexInto(if (typeof Bun !== "undefined") Bun.gc(true);else (globalThis as { gc?: () => void }).gc?.();progress.start(INDEX_STAGE.swap, 1);- const keepBak = countStoredEmbeddings(bakPath) > countStoredEmbeddings(config.dbPath);+ // Counting the stored embeddings reads both indexes in full, which on+ // a large one blocks for a while; the lock is kept fresh meanwhile.+ const keepBak = keepWriterLeasesFresh(+ WRITER_LEASE_BLOCKING_BUDGET_MS,+ () => countStoredEmbeddings(bakPath) > countStoredEmbeddings(config.dbPath),+ );+ // The last moment the rebuild can still be thrown away harmlessly,+ // right before the first destructive step. A lease another writer took+ // over (this run stalled past the stale window) no longer excludes that+ // writer, and swapping now would replace whatever it committed with+ // this run's staging build.+ lease.assertHeld();const parkPath = keepBak ? config.dbPath + ".old" : bakPath;tryUnlink(parkPath);tryRename(config.dbPath, parkPath); // no-op (ENOENT) on fresh reindex@@ -1625,7 +1658,7 @@ async function reindexInto(progress.advance(INDEX_STAGE.swap);return stats;} finally {- await release();+ lease.release();}}diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsindex a09f6544..060b9484 100644--- a/src/core/search/schema.ts+++ b/src/core/search/schema.ts@@ -10,6 +10,7 @@import { Database } from "bun:sqlite";import { SearchError } from "./types.ts";+import { keepWriterLeasesFresh, WRITER_LEASE_BLOCKING_BUDGET_MS } from "./store/writer-lock.ts";import { hasPrivacyMarkerKey, remoteDenyVisibilityTokens } from "../graph/privacy-markers.ts";import { parseFrontmatterText } from "../vault.ts";import { closeDatabase } from "../sqlite-close.ts";@@ -1456,18 +1457,23 @@ export function applyMigrations(db: Database, opts: ApplyMigrationsOptions = {})if (current === LATEST_SCHEMA_VERSION) return current;- db.exec("BEGIN");- try {- for (const migration of MIGRATIONS) {- if (migration.version <= current) continue;- migration.up(db, ctx);- setSchemaVersion(db, migration.version);+ // A migration can rewrite a large table in one synchronous run; the+ // write open that calls this holds the writer lock, kept fresh off the+ // blocked thread meanwhile.+ keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () => {+ db.exec("BEGIN");+ try {+ for (const migration of MIGRATIONS) {+ if (migration.version <= current) continue;+ migration.up(db, ctx);+ setSchemaVersion(db, migration.version);+ }+ db.exec("COMMIT");+ } catch (e) {+ db.exec("ROLLBACK");+ throw e;}- db.exec("COMMIT");- } catch (e) {- db.exec("ROLLBACK");- throw e;- }+ });return LATEST_SCHEMA_VERSION;}diff --git a/src/core/search/store/keyword.ts b/src/core/search/store/keyword.tsindex 32e79ce2..8b5b1e58 100644--- a/src/core/search/store/keyword.ts+++ b/src/core/search/store/keyword.ts@@ -11,7 +11,11 @@ import type { SearchKind } from "../property-pushdown.ts";import { pushdownSqlFragment } from "../property-pushdown.ts";import { SearchError } from "../types.ts";import { pathPrefixRange } from "./path-prefix.ts";-import { acquireWriterLockSync } from "./writer-lock.ts";+import {+ acquireWriterLockSync,+ keepWriterLeasesFresh,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+} from "./writer-lock.ts";export interface KeywordHit {readonly chunkId: number;@@ -82,13 +86,17 @@ export function rebuildFtsIndexWithWriterLock(dbPath: string,heldWriterLock: boolean,): void {+ // One statement that can run for minutes on a large index, so the lock+ // is kept fresh off the blocked thread for its duration.+ const rebuild = (): void =>+ keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () => rebuildFtsIndex(db));if (heldWriterLock) {- rebuildFtsIndex(db);+ rebuild();return;}const release = acquireWriterLockSync(dbPath);try {- rebuildFtsIndex(db);+ rebuild();} finally {release();}diff --git a/src/core/search/store/lifecycle.ts b/src/core/search/store/lifecycle.tsindex ae9ab3e9..77998755 100644--- a/src/core/search/store/lifecycle.ts+++ b/src/core/search/store/lifecycle.ts@@ -32,7 +32,12 @@ import {setState,} from "./state.ts";import { loadVecExtension } from "./vectors.ts";-import { acquireWriterLock, acquireWriterLockSync } from "./writer-lock.ts";+import {+ acquireWriterLease,+ acquireWriterLockSync,+ keepWriterLeasesFresh,+ WRITER_LEASE_BLOCKING_BUDGET_MS,+} from "./writer-lock.ts";import { closeDatabase } from "../../sqlite-close.ts";import { renameWithRetry } from "../../fs-atomic.ts";import {@@ -413,7 +418,11 @@ function runWriteOpenIntegrityGate(db: Database, dbPath: string): void {return;}- const verdict = runIntegrityCheck(db);+ // A full scan of a large index blocks for as long as it reads; the open+ // holds the writer lock, kept fresh off the blocked thread meanwhile.+ const verdict = keepWriterLeasesFresh(WRITER_LEASE_BLOCKING_BUDGET_MS, () =>+ runIntegrityCheck(db),+ );try {setState(db, INTEGRITY_CHECKED_AT_STATE_KEY, nowIso());if (verdict.ok) deleteState(db, INTEGRITY_FAULT_STATE_KEY);@@ -512,11 +521,29 @@ export async function openWriteDatabase(closeDatabase(seed);}- const release = await acquireWriterLock(config.dbPath);+ // A write session has no single commit point to check its lease at, so+ // a lease lost to a takeover stops the session where the loss is seen:+ // the connection turns read-only (a write that slips past a caught+ // throw is refused by SQLite) and the loss is thrown as INDEX_LOCKED -+ // into the index walk from its check-in, or out of the heartbeat timer,+ // which ends the process as `proper-lockfile` did for a lost lock.+ let session: Database | null = null;+ const lease = await acquireWriterLease(config.dbPath, {+ onLost: (err) => {+ try {+ session?.exec("PRAGMA query_only = ON");+ } catch {+ /* the throw below still stops the session */+ }+ throw err;+ },+ });+ const release = async (): Promise<void> => lease.release();let db: Database;try {db = new Database(config.dbPath);+ session = db;} catch (e) {await release();const msg = e instanceof Error ? e.message : String(e);diff --git a/src/core/search/store/signal-exit.d.ts b/src/core/search/store/signal-exit.d.tsnew file mode 100644index 00000000..e3f6f364--- /dev/null+++ b/src/core/search/store/signal-exit.d.ts@@ -0,0 +1,11 @@+/**+ * `signal-exit` 3.x ships no types. The one call this codebase makes:+ * register a handler that runs on any process exit, a fatal signal+ * included, and get back a function that removes it.+ */+declare module "signal-exit" {+ export default function onExit(+ callback: (code: number | null, signal: string | null) => void,+ options?: { readonly alwaysLast?: boolean },+ ): () => void;+}diff --git a/src/core/search/store/writer-lock.ts b/src/core/search/store/writer-lock.tsindex 007665b1..39d0f22f 100644--- a/src/core/search/store/writer-lock.ts+++ b/src/core/search/store/writer-lock.ts@@ -1,54 +1,855 @@/*** The exclusive writer lock on the index path. Every writer - a write* open, the reindex rebuild+swap, the crash-recovery restore, an- * out-of-band FTS rebuild - serialises here, on the LIVE `dbPath`, so- * there is exactly one lock and one stale window in the system.+ * out-of-band FTS rebuild, a snapshot's store archive - serialises here,+ * on the LIVE `dbPath`, so there is exactly one lock and one stale window+ * in the system.+ *+ * On disk the lock is the protocol `proper-lockfile` uses, unchanged: the+ * directory `<dbPath>.lock`, created with `mkdir` (atomic), whose mtime is+ * the holder's heartbeat. A lock whose mtime is older than the stale+ * window is abandoned and the next writer removes and re-creates it.+ * Processes running an older release, which take this lock through+ * `proper-lockfile` itself, are excluded by it and exclude it. The safety+ * is not symmetric, though: an older release's release and exit hook+ * remove the directory whoever owns it, so an older holder that went stale+ * and wakes up can remove this module's lock. This module then sees the+ * lock gone, marks its lease lost and refuses its next commit.+ *+ * What this module adds over `proper-lockfile` is the lease discipline:+ *+ * 1. Ownership. Every refresh and the release first check that the+ * lock is still the one this holder last stamped (its mtime, and its+ * device and inode where the filesystem reports one). A lock another+ * writer took over is never refreshed and never removed: the holder+ * marks its lease lost instead. Before a commit and at release, a+ * lock past the stale window minus a margin is not touched either,+ * even when it still looks like ours, because a contender may be+ * taking it over at that very moment; a routine beat after a long+ * stall still refreshes a lock nobody took over, up to one second+ * (or the margin, when that is shorter) before the stale window.+ * 2. Freshness through synchronous work. The heartbeat normally runs on+ * the event loop (a timer, plus {@link writerLeaseCheckIn} from long+ * synchronous loops). A call that blocks the thread for longer - a+ * `spawnSync` of tar or zstd, a `PRAGMA quick_check`, `VACUUM INTO`,+ * an FTS rebuild, a migration - is wrapped in+ * {@link keepWriterLeasesFresh}, which refreshes from a worker thread+ * for at most a stated budget. Past the budget the worker stops, so a+ * holder wedged inside the call still goes stale and is taken over.+ * 3. Loss is surfaced. A holder about to commit (the reindex swap, the+ * restore swap) calls {@link WriterLease.assertHeld}, which refuses+ * with `INDEX_LOCKED`. A holder with no single commit point (a+ * write-mode store) passes an `onLost` handler, which runs the moment+ * the main thread sees the loss.*/-import lockfile from "proper-lockfile";+import {+ closeSync,+ fstatSync,+ futimesSync,+ mkdirSync,+ openSync,+ rmdirSync,+ statSync,+ utimesSync,+} from "node:fs";+import { resolve } from "node:path";+import { Worker } from "node:worker_threads";++import onExit from "signal-exit";import { SearchError } from "../types.ts";/**- * Stale window for the writer lock (ms). A lock whose mtime is older- * than this is treated as abandoned (crashed holder) and taken over by- * the next writer, so a SIGKILL never wedges the index for longer than- * this window.+ * Default stale window for the writer lock (ms). A lock whose mtime is+ * older than this is treated as abandoned (crashed or wedged holder) and+ * taken over by the next writer, so a SIGKILL never wedges the index for+ * longer than this window. {@link WRITER_LOCK_STALE_ENV} overrides it.*/export const WRITER_LOCK_STALE_MS = 60_000;/**- * Heartbeat interval (ms): the async writer lock refreshes its mtime- * this often so a legitimate long-running index is never mistaken for- * a stale lock. Must stay below {@link WRITER_LOCK_STALE_MS}. NOTE: the- * per-file document walk is synchronous, so this timer only fires- * across the await points (embed batches); a multi-minute fully- * synchronous walk still relies on the stale window, which 60s amply- * covers for real vaults.+ * Default heartbeat (ms): a live lease refreshes its mtime once it is this+ * old, a third of the stale window, so two missed beats still leave the+ * lock fresh. Derived from the resolved stale window when that is+ * overridden.+ */+export const WRITER_LOCK_HEARTBEAT_MS = 20_000;++/** Environment variable overriding {@link WRITER_LOCK_STALE_MS}. */+export const WRITER_LOCK_STALE_ENV = "OPEN_SECOND_BRAIN_WRITER_LOCK_STALE_MS";++/** The smallest stale window the override accepts (ms). */+const MIN_WRITER_LOCK_STALE_MS = 1_000;++/** The largest stale window the override accepts (ms): one day. */+const MAX_WRITER_LOCK_STALE_MS = 24 * 60 * 60_000;++/** Longest delay a timer accepts; a longer one fires every millisecond. */+const MAX_TIMER_MS = 2 ** 31 - 1;++/**+ * Budget (ms) for an in-process blocking call that has no timeout of its+ * own (an integrity scan, `VACUUM INTO`, an FTS rebuild, a migration):+ * {@link keepWriterLeasesFresh} keeps the lease fresh this long, and a+ * call still running past it counts as wedged.+ */+export const WRITER_LEASE_BLOCKING_BUDGET_MS = 15 * 60_000;++let shortWindowWarned = false;++/**+ * The stale window this process uses, from {@link WRITER_LOCK_STALE_ENV}+ * or the default. Every process sharing an index should agree on it: a+ * process with a shorter window takes over a lock a longer-window holder+ * still considers fresh. Older releases always use the default and cannot+ * be told otherwise, so a window below it is unsafe while one of them+ * shares the index; that is warned about once per process.+ */+export function resolveWriterLockStaleMs(env: NodeJS.ProcessEnv = process.env): number {+ const raw = env[WRITER_LOCK_STALE_ENV];+ if (raw === undefined || raw.trim() === "") return WRITER_LOCK_STALE_MS;+ const text = raw.trim();+ const parsed = /^\d+$/.test(text) ? Number(text) : Number.NaN;+ if (+ !Number.isSafeInteger(parsed) ||+ parsed < MIN_WRITER_LOCK_STALE_MS ||+ parsed > MAX_WRITER_LOCK_STALE_MS+ ) {+ throw new Error(+ `${WRITER_LOCK_STALE_ENV}=${raw} is not a stale window; set it to a whole number of ` ++ `milliseconds from ${MIN_WRITER_LOCK_STALE_MS} to ${MAX_WRITER_LOCK_STALE_MS} ` ++ `(the default is ${WRITER_LOCK_STALE_MS}), or unset it`,+ );+ }+ if (parsed < WRITER_LOCK_STALE_MS && !shortWindowWarned) {+ shortWindowWarned = true;+ // eslint-disable-next-line no-console+ console.error(+ `warning: ${WRITER_LOCK_STALE_ENV}=${parsed} is below the ${WRITER_LOCK_STALE_MS} ms ` ++ "older releases always use; do not share the index with an older release, whose " ++ "live lock this process would take over",+ );+ }+ return parsed;+}++function heartbeatFor(staleMs: number): number {+ // Never longer than the default beat, whatever the window: an older+ // release judges this lock by its own fixed 60-second window, so a+ // longer window must not let the lock's mtime fall that far behind.+ return Math.min(WRITER_LOCK_HEARTBEAT_MS, Math.floor(staleMs / 3));+}++/**+ * How far below the stale window a holder stops trusting its own lock+ * (ms): half a heartbeat. A refresh or release that finds the lock this+ * old does not write or remove it, since a contender may already be+ * removing it as stale.*/-export const WRITER_LOCK_HEARTBEAT_MS = 30_000;+function marginFor(staleMs: number): number {+ return Math.floor(heartbeatFor(staleMs) / 2);+}-/** The one message both lock paths report contention with. */+/** The one message both acquire paths report contention with. */function contentionError(cause: unknown): SearchError {const msg = cause instanceof Error ? cause.message : String(cause);return new SearchError("INDEX_LOCKED", `another writer holds the search index lock: ${msg}`);}-export function acquireWriterLockSync(path: string): () => void {- const maxAttempts = 10;- const sleepMs = 50;- let lastError: unknown;- for (let attempt = 0; attempt < maxAttempts; attempt++) {+function lockDirOf(path: string): string {+ return `${resolve(path)}.lock`;+}++function errnoCode(err: unknown): string | undefined {+ return (err as NodeJS.ErrnoException | null)?.code;+}++// ---------------------------------------------------------------------------+// Shared lease state+//+// Each lease keeps its state in a SharedArrayBuffer so the main thread and+// a refresher worker see one copy: Int32 slot 0 is the refresh mutex+// (whoever refreshes or releases holds it), slot 1 the lease state, and+// from byte 8 four Float64s: the mtime this holder last stamped, the+// device and inode of the directory, and 1 when the filesystem kept the+// stamp to the sub-millisecond (0 when it rounded it). The Float64s are+// plain, non-atomic memory: they are read and written only by whoever+// holds the mutex, whose Atomics operations order those accesses.+//+// Each stamp carries a random sub-millisecond fraction, so on a filesystem+// with fine timestamps two holders never record the same mtime, even when+// the filesystem reuses an inode number for a successor's directory.++const SLOT_BUSY = 0;+const SLOT_STATE = 1;+const STATE_HELD = 0;+const STATE_LOST = 1;+const STATE_RELEASED = 2;+const LEASE_BYTES = 40;++/**+ * Refresh one lease whose mutex the caller holds. Shared verbatim by the+ * main thread and the refresher worker (its source is embedded there), so+ * it references only its parameters and globals.+ *+ * Nothing is written, and the lease is marked lost, when the lock's mtime+ * or directory identity is not the one recorded, when its directory is+ * gone, or when it is too old: for a forced check (`minAgeMs` 0, before a+ * commit) past the stale window minus `marginMs`, since a contender may be+ * taking it over; for a due beat past the stale window minus the smaller+ * of `marginMs` and one second. The identity is checked once more+ * after the write, so a successor that replaced the directory between the+ * check and the write is not adopted. A refresh that fails for another+ * reason is retried on the next beat; the age rule gives it up in time.+ *+ * Returns true only when it stamped the lock and proved it ours after the+ * write; a skipped or inconclusive refresh (an EIO or EACCES on the stat)+ * returns false, which a commit check treats as not holding the lock.+ */+function refreshHeldLease(+ state: Int32Array,+ own: Float64Array,+ lockDir: string,+ staleMs: number,+ marginMs: number,+ minAgeMs: number,+ fs: LockFs,+): boolean {+ if (Atomics.load(state, 1) !== 0) return false;+ const known = own[0] as number;+ const age = Date.now() - known;+ // Only a due-check may skip the ownership check; a forced refresh+ // (minAgeMs 0) always looks, also when a clock that went backwards+ // makes the lock's age read as negative.+ if (minAgeMs > 0 && age < minAgeMs) return false;+ // How old the lock may be and still be written. A forced check (a commit+ // about to happen) stops at the margin below the stale window: it must not+ // write a lock a contender may be taking over. A due beat after a long+ // stall of this thread (a sleeping laptop) goes further, so a holder+ // nobody took over keeps its lease, but not to the window itself: a lock+ // that crosses the window between this check and the write is exactly+ // the one a contender removes and re-creates in that gap. One second+ // covers a stat and a utimes on a slow filesystem many times over and+ // still recovers almost the whole stall; with a short window the margin+ // itself is the smaller guard.+ const limit = minAgeMs > 0 ? staleMs - Math.min(marginMs, 1_000) : staleMs - marginMs;+ if (age >= limit) {+ Atomics.store(state, 1, 1);+ return false;+ }+ // An inode of 0 is a filesystem that reports none; the mtime alone counts.+ const sameDir = (st: { dev: number; ino: number }): boolean =>+ own[2] === 0 || (st.ino === own[2] && st.dev === own[1]);+ const stamp = Date.now() + Math.random() * 0.9;+ // The directory is opened, and the check and the stamp go through that+ // handle: a handle stays bound to the directory it opened, so a holder+ // that is paused between the check and the write and resumes after a+ // takeover stamps its own removed directory, never the successor's.+ // On Windows the path is used: a directory handle opens there but+ // refuses `futimes` (EPERM), and the pause between the check and the+ // write is the residual window. Elsewhere a directory that cannot be+ // opened falls back to the path too.+ let fd = -1;+ if (process.platform !== "win32") {try {- return lockfile.lockSync(path, { stale: WRITER_LOCK_STALE_MS, realpath: false });+ fd = fs.openSync(lockDir, "r");} catch (err) {- const code = (err as NodeJS.ErrnoException).code;- if (code !== "ELOCKED") throw err;- lastError = err;- if (attempt < maxAttempts - 1) Bun.sleepSync(sleepMs);+ if ((err as { code?: string } | null)?.code === "ENOENT") {+ Atomics.store(state, 1, 1);+ return false;+ }}}- throw contentionError(lastError);+ try {+ let current: { mtimeMs: number; dev: number; ino: number };+ try {+ current = fd >= 0 ? fs.fstatSync(fd) : fs.statSync(lockDir);+ } catch (err) {+ if ((err as { code?: string } | null)?.code === "ENOENT") Atomics.store(state, 1, 1);+ return false;+ }+ // Ownership, and the age of the lock as it is on disk, read against the+ // clock after the stat returned: a lock that crept up to the limit while+ // the stat was pending (a slow network filesystem) is not written either.+ if (current.mtimeMs !== known || !sameDir(current) || Date.now() - current.mtimeMs >= limit) {+ Atomics.store(state, 1, 1);+ return false;+ }+ try {+ if (fd >= 0) fs.futimesSync(fd, stamp / 1000, stamp / 1000);+ else fs.utimesSync(lockDir, stamp / 1000, stamp / 1000);+ // The path must still name the directory that was stamped.+ const after = fs.statSync(lockDir);+ if (!sameDir(after) || (fd >= 0 && after.mtimeMs !== fs.fstatSync(fd).mtimeMs)) {+ Atomics.store(state, 1, 1);+ return false;+ }+ own[0] = after.mtimeMs;+ own[3] = Math.abs(after.mtimeMs - stamp) < 0.01 ? 1 : 0;+ return true;+ } catch (err) {+ if ((err as { code?: string } | null)?.code === "ENOENT") Atomics.store(state, 1, 1);+ return false;+ }+ } finally {+ if (fd >= 0) {+ try {+ fs.closeSync(fd);+ } catch {+ // A handle that cannot be closed is released with the process.+ }+ }+ }+}++interface LockFs {+ readonly statSync: typeof statSync;+ readonly utimesSync: typeof utimesSync;+ readonly openSync: typeof openSync;+ readonly fstatSync: typeof fstatSync;+ readonly futimesSync: typeof futimesSync;+ readonly closeSync: typeof closeSync;+}++const FS_OPS: LockFs = { statSync, utimesSync, openSync, fstatSync, futimesSync, closeSync };++/**+ * How long a blocking mutex take (assert, release, a worker's stop) waits+ * for a refresh in flight (ms). A refresh is a stat, a utimes and a stat,+ * so this is only reached on a filesystem that has stopped answering; the+ * caller then treats ownership as unproven.+ */+const MUTEX_WAIT_MS = 2_000;++// Control words of a refresher worker.+const CONTROL_STOP = 0;+const CONTROL_PHASE = 1;+const PHASE_PENDING = 0;+const PHASE_RUNNING = 1;+const PHASE_DONE = 2;+const PHASE_CANCELLED = 3;++/**+ * Refreshes the given leases while the main thread is blocked, until told+ * to stop, until the budget runs out, or until no lease is held any more.+ * It waits on the control word with `Atomics.wait`, so it needs no event+ * loop, and it only takes a lease's mutex without blocking, so a lease the+ * main thread is releasing is skipped. It claims the start with a+ * compare-exchange, so a worker the main thread cancelled before it began+ * never touches a lease.+ */+const REFRESHER_SOURCE = `+const { workerData } = require("node:worker_threads");+const fs = require("node:fs");+const refreshHeldLease = (${refreshHeldLease.toString()});+const control = new Int32Array(workerData.control);+if (Atomics.compareExchange(control, ${CONTROL_PHASE}, ${PHASE_PENDING}, ${PHASE_RUNNING}) === ${PHASE_PENDING}) {+ const started = performance.now();+ const leases = workerData.leases.map((l) => ({+ state: new Int32Array(l.sab, 0, 2),+ own: new Float64Array(l.sab, 8, 4),+ lockDir: l.lockDir,+ staleMs: l.staleMs,+ marginMs: l.marginMs,+ heartbeatMs: l.heartbeatMs,+ }));+ const tick = Math.max(25, Math.min(...leases.map((l) => l.heartbeatMs)) / 2);+ while (performance.now() - started < workerData.budgetMs) {+ let held = 0;+ for (const l of leases) {+ if (Atomics.load(control, ${CONTROL_STOP}) !== 0) break;+ if (Atomics.load(l.state, 1) !== 0) continue;+ held++;+ if (Atomics.compareExchange(l.state, ${SLOT_BUSY}, 0, 1) !== 0) continue;+ try {+ refreshHeldLease(l.state, l.own, l.lockDir, l.staleMs, l.marginMs, l.heartbeatMs, fs);+ } catch {+ // A refresh that throws is retried on the next tick.+ } finally {+ Atomics.store(l.state, ${SLOT_BUSY}, 0);+ Atomics.notify(l.state, ${SLOT_BUSY});+ }+ }+ if (held === 0 || Atomics.load(control, ${CONTROL_STOP}) !== 0) break;+ Atomics.wait(control, ${CONTROL_STOP}, 0, tick);+ }+ Atomics.store(control, ${CONTROL_PHASE}, ${PHASE_DONE});+ Atomics.notify(control, ${CONTROL_PHASE});+}+`;++/**+ * An exclusive writer lock held by this process. It stays fresh on its own+ * while the event loop runs; synchronous work keeps it fresh through+ * {@link writerLeaseCheckIn} and {@link keepWriterLeasesFresh}.+ */+export interface WriterLease {+ /** The index path the lock is held on (the lock is `<path>.lock`). */+ readonly path: string;+ /**+ * True once the lock stopped being this holder's: another writer took it+ * over, its directory was removed, or it went so long without a refresh+ * that it may be taken over. A lost lease excludes nothing.+ */+ readonly lost: boolean;+ /**+ * Refuse with `INDEX_LOCKED` unless the lock is still this holder's,+ * refreshing it when it is. Called right before a commit that must not+ * race another writer (a swap), so a lease lost to a takeover aborts the+ * commit instead of overwriting what the other writer did.+ */+ assertHeld(): void;+ /**+ * Remove the lock when it is provably still this holder's; leave it+ * otherwise (a successor's lock, or one the next writer will find stale+ * and take over anyway). Idempotent. Throws only when removing our own+ * lock fails.+ */+ release(): void;+}++export interface WriterLeaseOptions {+ /**+ * Runs once, on the main thread, when this process first sees that the+ * lease was lost: from the heartbeat timer, a check-in, the end of a+ * {@link keepWriterLeasesFresh} call, or {@link WriterLease.assertHeld}.+ * It receives the `INDEX_LOCKED` error and may throw it; from the timer,+ * that ends the process, as `proper-lockfile` did for a lock it lost.+ */+ readonly onLost?: (err: SearchError) => void;+}++const liveLeases = new Set<Lease>();+let roundingFilesystemWarned = false;+let exitHookInstalled = false;++function installExitHook(): void {+ if (exitHookInstalled) return;+ exitHookInstalled = true;+ // On any exit, a signal included, leases still held are released the+ // same ownership-checked way, so an interrupted writer neither keeps+ // the index locked for a stale window nor removes a successor's lock.+ onExit(() => {+ for (const lease of [...liveLeases]) {+ try {+ lease.release();+ } catch {+ // The process is exiting; the lock goes stale on its own.+ }+ }+ });+}++class Lease implements WriterLease {+ readonly path: string;+ readonly lockDir: string;+ readonly staleMs: number;+ readonly heartbeatMs: number;+ readonly marginMs: number;+ readonly sab: SharedArrayBuffer;+ readonly #state: Int32Array;+ readonly #own: Float64Array;+ readonly #timer: ReturnType<typeof setInterval>;+ readonly #onLost: ((err: SearchError) => void) | undefined;+ #lossReported = false;++ constructor(+ path: string,+ lockDir: string,+ staleMs: number,+ identity: { readonly mtimeMs: number; readonly dev: number; readonly ino: number },+ precise: boolean,+ options: WriterLeaseOptions,+ ) {+ this.path = path;+ this.lockDir = lockDir;+ this.staleMs = staleMs;+ this.heartbeatMs = heartbeatFor(staleMs);+ this.marginMs = marginFor(staleMs);+ this.#onLost = options.onLost;+ this.sab = new SharedArrayBuffer(LEASE_BYTES);+ this.#state = new Int32Array(this.sab, 0, 2);+ this.#own = new Float64Array(this.sab, 8, 4);+ this.#own[0] = identity.mtimeMs;+ this.#own[1] = identity.dev;+ this.#own[2] = identity.ino;+ this.#own[3] = precise ? 1 : 0;+ if (identity.ino === 0 && !precise && !roundingFilesystemWarned) {+ roundingFilesystemWarned = true;+ // eslint-disable-next-line no-console+ console.error(+ `warning: the filesystem holding ${lockDir} reports no inode and rounds timestamps, ` ++ "so a writer cannot prove the search index lock is still its own when it releases " ++ `it; each release leaves the lock to go stale, and the next writer waits up to ${staleMs} ms`,+ );+ }+ // The event-loop heartbeat. Unref'd: a held lease never keeps the+ // process alive by itself.+ this.#timer = setInterval(+ () => this.checkIn(),+ Math.min(MAX_TIMER_MS, Math.max(25, this.heartbeatMs / 2)),+ );+ this.#timer.unref?.();+ liveLeases.add(this);+ installExitHook();+ }++ get lost(): boolean {+ return Atomics.load(this.#state, SLOT_STATE) === STATE_LOST;+ }++ /** Whether its holder released it. */+ get released(): boolean {+ return Atomics.load(this.#state, SLOT_STATE) === STATE_RELEASED;+ }++ /** Whether this lease still needs refreshing (held, not lost or released). */+ get active(): boolean {+ return Atomics.load(this.#state, SLOT_STATE) === STATE_HELD;+ }++ /** Refresh when due, without waiting for a refresh already in flight. */+ refreshIfDue(minAgeMs: number = this.heartbeatMs): void {+ if (!this.#tryLock()) return;+ try {+ refreshHeldLease(+ this.#state,+ this.#own,+ this.lockDir,+ this.staleMs,+ this.marginMs,+ minAgeMs,+ FS_OPS,+ );+ } finally {+ this.#unlock();+ }+ }++ /** A due refresh, then the loss report if the lease was found lost. */+ checkIn(): void {+ this.refreshIfDue();+ if (this.lost) this.reportLost();+ }++ #lostError(): SearchError {+ return new SearchError(+ "INDEX_LOCKED",+ `the search index writer lock on ${this.path} is no longer held by this writer ` ++ "(it went stale and may have been taken over by another writer)",+ );+ }++ /**+ * Report the loss once: stop the heartbeat, warn, and run `onLost`,+ * whose throw propagates to the caller.+ */+ reportLost(): void {+ if (this.#lossReported) return;+ this.#lossReported = true;+ this.#settle();+ const err = this.#lostError();+ // eslint-disable-next-line no-console+ console.error(`warning: ${err.message}`);+ this.#onLost?.(err);+ }++ assertHeld(): void {+ const locked = this.#lockBlocking();+ let proven = false;+ try {+ if (locked) {+ proven = refreshHeldLease(+ this.#state,+ this.#own,+ this.lockDir,+ this.staleMs,+ this.marginMs,+ 0,+ FS_OPS,+ );+ }+ } finally {+ if (locked) this.#unlock();+ }+ if (Atomics.load(this.#state, SLOT_STATE) === STATE_HELD && proven) return;+ if (Atomics.load(this.#state, SLOT_STATE) === STATE_HELD) {+ // A refresh that never let go of the mutex, or one that could not+ // read the lock: ownership is not proven, which for a commit is the+ // same as not having it.+ Atomics.store(this.#state, SLOT_STATE, STATE_LOST);+ }+ this.reportLost();+ throw this.#lostError();+ }++ release(): void {+ const locked = this.#lockBlocking();+ try {+ if (!locked || Atomics.load(this.#state, SLOT_STATE) !== STATE_HELD) return;+ // No inode and a rounded stamp: the mtime alone cannot tell this+ // holder from a successor, so the lock is left to go stale.+ if (this.#own[2] === 0 && this.#own[3] === 0) return;+ // Everything else is decided by one stat taken right before the+ // removal, so only the stat and the rmdir themselves are a window+ // for a contender to swap the directory in.+ let current: { mtimeMs: number; dev: number; ino: number };+ try {+ current = statSync(this.lockDir);+ } catch (err) {+ if (errnoCode(err) === "ENOENT") return;+ throw err;+ }+ // Someone else's lock now: theirs to release.+ const sameDir =+ this.#own[2] === 0 || (current.ino === this.#own[2] && current.dev === this.#own[1]);+ if (current.mtimeMs !== this.#own[0] || !sameDir) return;+ // Past the stale window minus the margin, on disk and now, a+ // contender may be removing it right now; it is left for whoever+ // takes it next.+ if (Date.now() - current.mtimeMs >= this.staleMs - this.marginMs) return;+ try {+ rmdirSync(this.lockDir);+ } catch (err) {+ if (errnoCode(err) !== "ENOENT") throw err;+ }+ } finally {+ if (Atomics.load(this.#state, SLOT_STATE) === STATE_HELD) {+ Atomics.store(this.#state, SLOT_STATE, STATE_RELEASED);+ }+ if (locked) this.#unlock();+ this.#settle();+ }+ }++ #settle(): void {+ clearInterval(this.#timer);+ liveLeases.delete(this);+ }++ #tryLock(): boolean {+ return Atomics.compareExchange(this.#state, SLOT_BUSY, 0, 1) === 0;+ }++ #lockBlocking(): boolean {+ const deadline = Date.now() + MUTEX_WAIT_MS;+ while (!this.#tryLock()) {+ const left = deadline - Date.now();+ if (left <= 0) return false;+ Atomics.wait(this.#state, SLOT_BUSY, 1, left);+ }+ return true;+ }++ #unlock(): void {+ Atomics.store(this.#state, SLOT_BUSY, 0);+ Atomics.notify(this.#state, SLOT_BUSY);+ }+}++/**+ * One attempt at the lock: true when this call created it, false when a+ * live holder has it. A lock past the stale window is removed and created+ * again, as `proper-lockfile` does; any other filesystem error is thrown.+ */+function tryCreateLock(lockDir: string, staleMs: number): boolean {+ try {+ mkdirSync(lockDir);+ return true;+ } catch (err) {+ if (errnoCode(err) !== "EEXIST") throw err;+ }+ let mtimeMs: number;+ try {+ mtimeMs = statSync(lockDir).mtimeMs;+ } catch (err) {+ if (errnoCode(err) !== "ENOENT") throw err;+ mtimeMs = Number.NEGATIVE_INFINITY; // released meanwhile+ }+ if (mtimeMs >= Date.now() - staleMs) return false;+ try {+ rmdirSync(lockDir);+ } catch (err) {+ if (errnoCode(err) !== "ENOENT") throw err;+ }+ try {+ mkdirSync(lockDir);+ return true;+ } catch (err) {+ if (errnoCode(err) === "EEXIST") return false;+ throw err;+ }+}++/**+ * Stamp a directory this call just created (no earlier than `createdAt`,+ * monotonic)+ * with this attempt's own mtime and read back its identity.+ *+ * Nothing can tell from the directory alone that it is the one this call's+ * `mkdir` made, so it is proven by time instead: a contender removes it+ * only once it is a stale window old, so a directory opened less than the+ * window minus `marginMs` after the `mkdir` is necessarily ours. It is+ * opened first and stamped through that handle, so a pause after the check+ * stamps our own directory even if it was replaced meanwhile, and the+ * path is checked to still name it afterwards. Null - contention, nothing+ * more touched - when the directory was removed or replaced, or when too+ * long passed to prove it ours.+ */+function stampNewLock(+ lockDir: string,+ createdAt: number,+ staleMs: number,+ marginMs: number,+): { identity: { mtimeMs: number; dev: number; ino: number }; precise: boolean } | null {+ let fd = -1;+ let created: { dev: number; ino: number } | null = null;+ try {+ // On Windows the path is used: a directory handle opens there but+ // refuses `futimes`. Elsewhere one that cannot be opened falls back too.+ if (process.platform !== "win32") {+ try {+ fd = openSync(lockDir, "r");+ } catch (err) {+ if (errnoCode(err) === "ENOENT") return null;+ }+ }+ created = fd >= 0 ? fstatSync(fd) : statSync(lockDir);+ if (performance.now() - createdAt >= staleMs - marginMs) return null;+ const stamp = Date.now() + Math.random() * 0.9;+ if (fd >= 0) futimesSync(fd, stamp / 1000, stamp / 1000);+ else utimesSync(lockDir, stamp / 1000, stamp / 1000);+ const identity = statSync(lockDir);+ if (identity.ino !== created.ino || identity.dev !== created.dev) return null;+ if (fd >= 0 && identity.mtimeMs !== fstatSync(fd).mtimeMs) return null;+ return { identity, precise: Math.abs(identity.mtimeMs - stamp) < 0.01 };+ } catch (err) {+ if (errnoCode(err) === "ENOENT") return null;+ // The stamp failed (a filesystem that refuses to set a directory's+ // times): the directory this attempt made is removed, so a failed+ // acquire leaves no lock behind, but only under the same proof that it+ // is ours - still inside the window minus the margin since the+ // `mkdir`, and the path still naming the directory that was opened.+ removeOwnNewLock(lockDir, created, createdAt, staleMs, marginMs);+ // A lock that cannot be stamped is a filesystem fault, not another+ // writer: it is reported as itself, never as contention.+ throw Object.assign(+ new Error(+ `cannot stamp the search index writer lock ${lockDir}: ` ++ `${(err as Error).message ?? String(err)}`,+ { cause: err },+ ),+ { code: errnoCode(err), lockStampFailure: true },+ );+ } finally {+ if (fd >= 0) {+ try {+ closeSync(fd);+ } catch {+ /* released with the process */+ }+ }+ }+}++function removeOwnNewLock(+ lockDir: string,+ created: { readonly dev: number; readonly ino: number } | null,+ createdAt: number,+ staleMs: number,+ marginMs: number,+): void {+ if (created === null || performance.now() - createdAt >= staleMs - marginMs) return;+ try {+ const now = statSync(lockDir);+ if (now.ino !== created.ino || now.dev !== created.dev) return;+ rmdirSync(lockDir);+ } catch {+ /* left to go stale */+ }+}++function attempt(path: string, staleMs: number, options: WriterLeaseOptions): Lease | null {+ const lockDir = lockDirOf(path);+ // Taken before the `mkdir`, so the time it proves ownership by can only+ // be overstated, never understated.+ const startedAt = performance.now();+ if (!tryCreateLock(lockDir, staleMs)) return null;+ const stamped = stampNewLock(lockDir, startedAt, staleMs, marginFor(staleMs));+ if (stamped === null) return null;+ return new Lease(path, lockDir, staleMs, stamped.identity, stamped.precise, options);+}++function lockedError(path: string): Error {+ return Object.assign(new Error("Lock file is already being held"), {+ code: "ELOCKED",+ file: path,+ });+}++/**+ * Take the writer lock synchronously as a {@link WriterLease}. Retries a+ * few times over half a second, then refuses with `INDEX_LOCKED`. Other+ * filesystem errors are thrown as they are. `path` need not exist; its+ * parent directory must.+ */+export function acquireWriterLeaseSync(+ path: string,+ options: WriterLeaseOptions = {},+): WriterLease {+ const staleMs = resolveWriterLockStaleMs();+ const maxAttempts = 10;+ const sleepMs = 50;+ for (let i = 0; i < maxAttempts; i++) {+ const lease = attempt(path, staleMs, options);+ if (lease !== null) return lease;+ if (i < maxAttempts - 1) Bun.sleepSync(sleepMs);+ }+ throw contentionError(lockedError(path));+}++export function acquireWriterLockSync(path: string): () => void {+ const lease = acquireWriterLeaseSync(path);+ return () => lease.release();+}++/**+ * Take the writer lock on the LIVE index path as a {@link WriterLease}.+ * Shared by `Store.open({ mode: "write" })` and `reindexVault`'s+ * rebuild+swap so both serialise on the SAME lock (keyed on `dbPath`, not+ * on the `.new` staging path). Fast-fails to `INDEX_LOCKED` after a few+ * retries a second apart rather than blocking indefinitely, matching the+ * module's fail-fast contention contract; every failure, a filesystem one+ * included, is reported that way, except a lock directory that was created+ * but cannot be stamped, which is a fault of the filesystem and is thrown+ * as itself. `path` need not exist yet (a fresh+ * reindex has no live index); only its parent directory must.+ */+export async function acquireWriterLease(+ path: string,+ options: WriterLeaseOptions = {},+): Promise<WriterLease> {+ const retries = 3;+ const delayMs = 1_000;+ // A malformed override is a configuration error, not contention.+ const staleMs = resolveWriterLockStaleMs();+ try {+ for (let i = 0; i <= retries; i++) {+ const lease = attempt(path, staleMs, options);+ if (lease !== null) return lease;+ if (i < retries) await Bun.sleep(delayMs);+ }+ throw lockedError(path);+ } catch (e) {+ if ((e as { lockStampFailure?: boolean } | null)?.lockStampFailure === true) throw e;+ throw contentionError(e);+ }+}++export async function acquireWriterLock(dbPath: string): Promise<() => Promise<void>> {+ const lease = await acquireWriterLease(dbPath);+ return async () => lease.release();}/**@@ -61,7 +862,7 @@ export function acquireWriterLockSync(path: string): () => void {* cheaper contention, not exclusion: a caller about to start a rebuild it* can already see will lose can decline to start it at all.*- * Same `stale` window as both acquire paths, so a lock whose holder was+ * Same stale window as both acquire paths, so a lock whose holder was* SIGKILLed reads as free here exactly when it would be taken over there.* A stale lock is reported free without being removed - removing it is the* acquiring writer's business.@@ -72,30 +873,130 @@ export function acquireWriterLockSync(path: string): () => void {* the second as the first has been told something untrue.*/export function isWriterLockHeld(dbPath: string): boolean {- return lockfile.checkSync(dbPath, { stale: WRITER_LOCK_STALE_MS, realpath: false });+ const staleMs = resolveWriterLockStaleMs();+ try {+ return statSync(lockDirOf(dbPath)).mtimeMs >= Date.now() - staleMs;+ } catch (err) {+ if (errnoCode(err) === "ENOENT") return false;+ throw err;+ }}/**- * Acquire the exclusive writer lock on the LIVE index path. Shared by- * `Store.open({ mode: "write" })` and `reindexVault`'s rebuild+swap so both- * serialise on the SAME lock (keyed on `dbPath`, not on the `.new` staging- * path). Fast-fails to `INDEX_LOCKED` after a few retries rather than- * blocking indefinitely, matching the module's fail-fast contention- * contract. `realpath: false` lets the lock be taken before `dbPath` exists- * (a fresh reindex has no live index yet); only the parent directory must- * already exist.+ * A cheap check-in for long synchronous loops (the index walk, the brain+ * store catch-up): refreshes every lease this process holds that is due,+ * and reports a lease found lost (a write-mode store's `onLost` refuses+ * the loop from here). A no-op syscall-wise until a lease is a heartbeat+ * old.*/-export async function acquireWriterLock(dbPath: string): Promise<() => Promise<void>> {+export function writerLeaseCheckIn(): void {+ for (const lease of [...liveLeases]) lease.checkIn();+}++/**+ * Run `fn`, which blocks this thread, with every lease this process holds+ * kept fresh from a worker thread for at most `budgetMs`.+ *+ * The budget is the bound on the lease, not on `fn`: once it runs out the+ * worker stops refreshing, so a holder wedged inside `fn` goes stale one+ * stale window later and the next writer takes the lock over. Callers pass+ * the timeout the call is killed at (a `spawnSync` timeout) plus a margin,+ * or {@link WRITER_LEASE_BLOCKING_BUDGET_MS} for a call without one. The+ * worker refreshes under the same ownership check as the main thread, so a+ * lock another writer took over is marked lost, never refreshed, and the+ * loss is reported when `fn` returns. A worker that dies or never starts+ * only stops refreshing: the lease goes stale, which is the safe failure,+ * and {@link WriterLease.assertHeld} reports it.+ *+ * When `fn` returns, the worker is stopped and, once it has let go of every+ * lease, terminated; it never outlives the call by more than a refresh in+ * flight. Leases taken inside `fn` are not covered. Nested calls each run+ * their own worker with their own budget, so an outer budget does not cut+ * an inner one short; the shared mutex keeps their refreshes from+ * overlapping.+ */+export function keepWriterLeasesFresh<T>(budgetMs: number, fn: () => T): T {+ const leases = [...liveLeases].filter((l) => l.active);+ if (leases.length === 0) return fn();+ // Ownership proven, and a fresh beat, before the protected work starts:+ // a lease already lost refuses it here with INDEX_LOCKED, and a fresh+ // beat gives the worker a whole stale window to start in.+ for (const lease of leases) lease.assertHeld();+ const control = new Int32Array(new SharedArrayBuffer(8));+ let worker: Worker | null = null;try {- return await lockfile.lock(dbPath, {- retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },- stale: WRITER_LOCK_STALE_MS,- // Explicit heartbeat: refresh the lock mtime mid-run so a long index- // is never mistaken for a stale lock and taken over.- update: WRITER_LOCK_HEARTBEAT_MS,- realpath: false,+ worker = new Worker(REFRESHER_SOURCE, {+ eval: true,+ workerData: {+ control: control.buffer,+ budgetMs,+ leases: leases.map((l) => ({+ sab: l.sab,+ lockDir: l.lockDir,+ staleMs: l.staleMs,+ marginMs: l.marginMs,+ heartbeatMs: l.heartbeatMs,+ })),+ },});- } catch (e) {- throw contentionError(e);+ worker.unref();+ // A worker failure is the safe one (the lease goes stale); it must+ // not surface as an unhandled error event.+ worker.on("error", () => {});+ } catch {+ worker = null;+ }+ const finish = (fnFailed: boolean): void => {+ if (worker !== null) stopRefresher(worker, control);+ // Ownership proven again once the blocking work is over: a worker that+ // ran out of budget stops without marking anything, so only a fresh+ // check tells whether the lock outlived `fn`. A loss refuses with+ // INDEX_LOCKED when `fn` itself succeeded; a failure of `fn` stays the+ // error the caller sees.+ let loss: unknown = null;+ for (const lease of leases) {+ // A lease `fn` released itself is not a loss.+ if (lease.released) continue;+ try {+ lease.assertHeld();+ } catch (err) {+ loss ??= err;+ }+ }+ if (loss !== null && !fnFailed) throw loss;+ };+ let result: T;+ try {+ result = fn();+ } catch (err) {+ finish(true);+ throw err;+ }+ finish(false);+ return result;+}++/**+ * Stop a refresher. One that never started is cancelled and terminated at+ * once. A running one is woken, waited for until it has let go of every+ * lease (bounded by {@link MUTEX_WAIT_MS}), then terminated; one that does+ * not answer in time (a filesystem call that hangs) is left to finish on+ * its own, since terminating it mid-refresh could leave a lease's mutex+ * taken for good.+ */+function stopRefresher(worker: Worker, control: Int32Array): void {+ Atomics.store(control, CONTROL_STOP, 1);+ Atomics.notify(control, CONTROL_STOP);+ const cancelled =+ Atomics.compareExchange(control, CONTROL_PHASE, PHASE_PENDING, PHASE_CANCELLED) ===+ PHASE_PENDING;+ if (!cancelled) {+ const deadline = Date.now() + MUTEX_WAIT_MS;+ while (Atomics.load(control, CONTROL_PHASE) === PHASE_RUNNING) {+ const left = deadline - Date.now();+ if (left <= 0) return;+ Atomics.wait(control, CONTROL_PHASE, PHASE_RUNNING, left);+ }}+ void worker.terminate().catch(() => {});}diff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.tsindex 753c78fa..26a85d3c 100644--- a/tests/core/architecture/write-site-census.test.ts+++ b/tests/core/architecture/write-site-census.test.ts@@ -1036,6 +1036,17 @@ const DIRECT_WRITE_EXCLUSIONS: Readonly<Record<string, WriteExclusion>> = Object"takeover claim on that file won.",},+ "src/core/search/store/writer-lock.ts": {+ categories: [C.lockPrimitive],+ calls: ["rmdirSync", "utimesSync"],+ reason:+ "the search index writer lock itself, an empty `<dbPath>.lock` directory. Its " ++ "semantics come from the exclusive `mkdir`; `utimesSync` is the heartbeat that " ++ "stamps the directory's mtime, and `rmdirSync` is the release (only of a lock whose " ++ "mtime and inode are still this holder's) or the removal of a lock past its stale " ++ "window by the contender about to re-create it. No file bytes are written.",+ },+// --- Metadata only -----------------------------------------------------"src/core/brain/health/remediation.ts": {categories: [C.metadataOnly],@@ -1386,8 +1397,9 @@ const DIRECT_ROWS = ROWS.filter((row) => row.directCalls.length > 0);* 78 -> 79: the migration places accepted skills from Brain/skills in the vault's skills/ folder as files (task 2.10).* 79 -> 80: the skill accept journal writes its repair markers as machine-state files, not lane records.* 80 -> 81: machine-state reads a bare vault's own `.open-second-brain/` to seed the device-local state (device-local writes only).+ * 81 -> 82: the search index writer lock is taken, refreshed and released by its own module instead of `proper-lockfile`, to check ownership.*/-const DIRECT_WRITE_ROWS = 81;+const DIRECT_WRITE_ROWS = 82;/*** Measured modules reaching a write through a shared helper. An equality.@@ -1759,8 +1771,9 @@ const STAMPED_PATHS: ReadonlySet<string> = new Set(* 77 -> 78: the migration places accepted skills from Brain/skills in the vault's skills/ folder as files (task 2.10).* 78 -> 79: the skill accept journal's repair markers are machine-state files.* 79 -> 80: machine-state reads a bare vault's own `.open-second-brain/` to seed the device-local state (device-local writes only).+ * 80 -> 81: the search index writer lock's own module (lock directory metadata only, no stamped bytes).*/-const UNSTAMPED_DIRECT_ROWS = 80;+const UNSTAMPED_DIRECT_ROWS = 81;/*** Shared-helper write sites the stamp does not reach, measured the samediff --git a/tests/core/brain/snapshot-restore-order.test.ts b/tests/core/brain/snapshot-restore-order.test.tsindex 2a585fc1..30e1fd40 100644--- a/tests/core/brain/snapshot-restore-order.test.ts+++ b/tests/core/brain/snapshot-restore-order.test.ts@@ -52,7 +52,12 @@ import { BRAIN_SNAPSHOT_REASON } from "../../../src/core/brain/types.ts";import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";import { sha256Hex } from "../../../src/core/integrity/digest.ts";import { indexVault } from "../../../src/core/search/indexer.ts";-import { acquireWriterLockSync } from "../../../src/core/search/store/writer-lock.ts";+import {+ acquireWriterLeaseSync,+ acquireWriterLockSync,+ isWriterLockHeld,+ type WriterLease,+} from "../../../src/core/search/store/writer-lock.ts";import { resolveIndexPath } from "../../../src/core/search/paths.ts";import { CLI_SPAWN_BUDGET_MS } from "../../helpers/cli-timeout.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";@@ -599,23 +604,27 @@ function tablesOf(path: string): string[] {}}-type LockSync = typeof lockfile.lockSync;-/**- * Run `fn` with every writer lock taken on the live store wrapped by- * `wrapRelease`. Locks on other paths, and locks the test itself takes- * while `bypass()` is true, are the real ones.+ * Run `fn` with every release of the live store's writer lock wrapped by+ * `wrapRelease`. A release removes the lock directory `<dbPath>.lock`+ * while the lock is live, so that removal is what is wrapped. The lock+ * module's other removals - of a lock past its stale window, before a+ * takeover - happen on a lock that is not live and are left alone, as are+ * removals on other paths and those the test itself makes while+ * `bypass()` is true.*/function withStoreLockRelease<T>(wrapRelease: (real: () => void) => () => void,fn: () => T,bypass: () => boolean = () => false,): T {- const realLockSync: LockSync = lockfile.lockSync.bind(lockfile);- const spy = spyOn(lockfile, "lockSync").mockImplementation(((path, opts) => {- const release = realLockSync(path, opts);- return String(path) === dbPath && !bypass() ? wrapRelease(release) : release;- }) as LockSync);+ const realRmdir = fs.rmdirSync;+ const spy = spyOn(fs, "rmdirSync").mockImplementation(((path, opts) => {+ const real = (): void => realRmdir(path, opts);+ const release = String(path) === `${dbPath}.lock` && isWriterLockHeld(dbPath);+ if (release && !bypass()) wrapRelease(real)();+ else real();+ }) as typeof fs.rmdirSync);try {return fn();} finally {@@ -698,6 +707,112 @@ describe("the gated restore holds the store lock from the recovery point throughexpect(tablesOf(dbPath)).toEqual(replaced!);});+ test("a lock another writer took over before the swap refuses it with nothing touched", () => {+ snapshotThenDiverge();+ let successor: WriterLease | null = null;+ let successorLockKept = false;+ expectUntouched(() => {+ try {+ restoreSnapshot(vault, RUN_ID, {+ beforeDiscard: () => {+ // What a takeover does on disk once the restore's lock went+ // stale: the old lock directory is removed and the next+ // writer creates its own.+ rmSync(`${dbPath}.lock`, { recursive: true });+ successor = acquireWriterLeaseSync(dbPath);+ },+ });+ } finally {+ // The restore's release left the successor's lock alone.+ successorLockKept = existsSync(`${dbPath}.lock`);+ (successor as WriterLease | null)?.release();+ }+ }, /writer lock was lost before the swap.*refusing before the Brain tree is touched/u);+ expect(successor).not.toBeNull();+ expect(successorLockKept).toBe(true);+ });++ test("a lock taken over during the Brain/ swap undoes it before the store is replaced", () => {+ snapshotThenDiverge();+ let successor: WriterLease | null = null;+ const realRename = fs.renameSync;+ const spy = spyOn(fs, "renameSync").mockImplementation((from, to) => {+ if (successor === null && isMoveIn(String(from), String(to))) {+ // A takeover in the middle of the renames: the restore's lock+ // directory replaced by another writer's.+ rmSync(`${dbPath}.lock`, { recursive: true });+ successor = acquireWriterLeaseSync(dbPath);+ }+ realRename(from, to);+ });+ try {+ expectUntouched(() => {+ try {+ restoreSnapshot(vault, RUN_ID);+ } finally {+ (successor as WriterLease | null)?.release();+ }+ }, /writer lock was lost during the Brain\/ swap.*every Brain\/ rename was undone, so Brain\/ and the derived store are as they were before the restore/su);+ } finally {+ spy.mockRestore();+ }+ expect(successor).not.toBeNull();+ });++ test("a rename that fails after the lock was taken over reports the swap, not a refusal", () => {+ snapshotThenDiverge();+ let successor: WriterLease | null = null;+ const realRename = fs.renameSync;+ const spy = spyOn(fs, "renameSync").mockImplementation((from, to) => {+ if (successor === null && isMoveIn(String(from), String(to))) {+ // The swap is under way: the lock is taken over, and this rename+ // fails. What the restore reports is the swap and its undo.+ rmSync(`${dbPath}.lock`, { recursive: true });+ successor = acquireWriterLeaseSync(dbPath);+ throw fsError("EIO", String(to));+ }+ realRename(from, to);+ });+ try {+ expectUntouched(() => {+ try {+ restoreSnapshot(vault, RUN_ID);+ } finally {+ (successor as WriterLease | null)?.release();+ }+ }, /^snapshot\[dream-restore-order\]: restore failed during the swap: a Brain\/ rename failed.*every Brain\/ rename was undone/su);+ } finally {+ spy.mockRestore();+ }+ expect(successor).not.toBeNull();+ });++ test("a store archive written while its lock was taken over is not kept", () => {+ let successor: WriterLease | null = null;+ const archive = snapshotStorePath(vault, RUN_ID);+ const realLink = fs.linkSync;+ const spy = spyOn(fs, "linkSync").mockImplementation((from, to) => {+ if (successor === null && String(to) === archive) {+ // The store's lock taken over while the archive is published.+ rmSync(`${dbPath}.lock`, { recursive: true });+ successor = acquireWriterLeaseSync(dbPath);+ }+ realLink(from, to);+ });+ let message = "";+ try {+ message = refusal(() =>+ createSnapshot(vault, RUN_ID, { reason: DREAM, derivedStore: COVERED }),+ );+ } finally {+ spy.mockRestore();+ (successor as WriterLease | null)?.release();+ }+ expect(successor).not.toBeNull();+ expect(message).toMatch(/writer lock was lost while its archive was written/u);+ expect(existsSync(archive)).toBe(false);+ });+test("a lock release that fails after the swap committed says so and names the recovery point", () => {coverStoreByPolicy();snapshotThenDiverge();diff --git a/tests/core/doctor-readiness.test.ts b/tests/core/doctor-readiness.test.tsindex fa02f21c..fdd67fa7 100644--- a/tests/core/doctor-readiness.test.ts+++ b/tests/core/doctor-readiness.test.ts@@ -1,5 +1,13 @@import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync } from "node:fs";+import {+ existsSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";import { tmpdir } from "node:os";import { dirname, join } from "node:path";@@ -1148,29 +1156,62 @@ describe("probeClientVersions", () => {expect(probe.detail).toContain("timed out");});+ /**+ * Ask the production runner, at a test-sized budget, to run a Bun child+ * that reports its own pid and then hangs for `hangMs`; with `ignoreTerm`+ * it ignores SIGTERM from before it reports.+ * Bun's `process.pid` is the native pid on every platform, which a+ * shell's `$$` is not on Windows, so the liveness poll below asks about+ * the process the budget must kill.+ *+ * The pid is written to a temp name and renamed into place, so a child+ * the budget cuts off mid-write never leaves a partial pid that reads as+ * a report. A child cut off before it reported at all proves nothing+ * either way - on a slow runner the interpreter's start alone can outlast+ * the smallest budget - so that run is repeated at a doubled budget, a+ * bounded number of times. Every run's verdict is checked; only the+ * liveness proof needs a run whose child got as far as reporting.+ */+ async function askReportingChild(+ name: string,+ hangMs: number,+ ignoreTerm = false,+ ): Promise<{ readonly pid: number }> {+ for (let attempt = 0, budgetMs = 150; attempt < 6; attempt++, budgetMs *= 2) {+ const pidFile = join(tmp, `${name}-${attempt}.pid`);+ const source = [+ ignoreTerm ? 'process.on("SIGTERM", () => {});' : "",+ 'const fs = require("node:fs");',+ `fs.writeFileSync(${JSON.stringify(`${pidFile}.tmp`)}, String(process.pid));`,+ `fs.renameSync(${JSON.stringify(`${pidFile}.tmp`)}, ${JSON.stringify(pidFile)});`,+ `setTimeout(() => {}, ${hangMs});`,+ ].join("\n");+ const v = await probeClientVersions(+ versionsOpts(() => spawnClientVersion([process.execPath, "-e", source], budgetMs)),+ );+ expect(v.rows![0]!.status).toBe(READINESS_STATUS.unknown);+ expect(v.rows![0]!.detail).toContain("timed out");+ if (!existsSync(pidFile)) continue;+ const pid = Number.parseInt(readFileSync(pidFile, "utf8").trim(), 10);+ expect(Number.isInteger(pid)).toBe(true);+ return { pid };+ }+ throw new Error(`the ${name} child never reported its pid before its budget cut it off`);+ }+test("a hung version child is killed at its ask's budget, never orphaned", async () => {writeConfig("");codexManifest();- const pidFile = join(tmp, "version-child.pid");// The production runner itself, through the same seam every other- // row is graded by, at a test-sized budget, against a child that- // would sleep far past it: `exec` replaces the shell with the- // sleeper in place, so the pid the script wrote is the one process- // that must die when the budget cuts the ask off.- const v = await probeClientVersions(- versionsOpts(() =>- spawnClientVersion(["sh", "-c", `echo $$ > '${pidFile}'; exec sleep 30`], 150),- ),- );- expect(v.rows![0]!.status).toBe(READINESS_STATUS.unknown);- expect(v.rows![0]!.detail).toContain("timed out");- const pid = Number.parseInt(readFileSync(pidFile, "utf8").trim(), 10);- expect(Number.isInteger(pid)).toBe(true);+ // row is graded by, against a child that would sleep far past the+ // budget: the pid it reported is the one process that must die when+ // the budget cuts the ask off.+ const { pid } = await askReportingChild("hung", 30_000);// The kill is issued asynchronously: poll until the pid is gone,// bounded, rather than sleeping a fixed grace period.const gone = await pollUntilGone(pid, Date.now() + 5_000);expect(gone).toBe(true);- }, 20000);+ }, 30000);test("a client whose helper holds stdout still delivers the version it printed", async () => {writeConfig("");@@ -1194,26 +1235,19 @@ describe("probeClientVersions", () => {test("a TERM-ignoring version child is escalated to SIGKILL and reaped", async () => {writeConfig("");codexManifest();- const pidFile = join(tmp, "term-child.pid");- // The M10 shape: a client that ignores SIGTERM (`trap "" TERM` keeps- // the disposition across `exec`, so the sleeper itself never dies of- // the first kill) outlived doctor together with its children, because- // the budget's cut was one SIGTERM with no follow-up. The sleeper is- // bounded at 8s so a regression cannot leak an unbounded orphan.- const v = await probeClientVersions(- versionsOpts(() =>- spawnClientVersion(["sh", "-c", `echo $$ > '${pidFile}'; trap "" TERM; exec sleep 8`], 150),- ),- );- expect(v.rows![0]!.status).toBe(READINESS_STATUS.unknown);- expect(v.rows![0]!.detail).toContain("timed out");- const pid = Number.parseInt(readFileSync(pidFile, "utf8").trim(), 10);- expect(Number.isInteger(pid)).toBe(true);+ // The M10 shape: a client that ignores SIGTERM outlived doctor+ // together with its children, because the budget's cut was one+ // SIGTERM with no follow-up. The child ignores TERM before it reports,+ // so a reported pid is one the first kill cannot stop. It lives+ // 30 s, past the largest ask budget plus the poll window, so only the+ // kill can make it go in time, and a regression cannot leak an+ // unbounded orphan.+ const { pid } = await askReportingChild("term", 30_000, true);// The escalation lands a grace period after the SIGTERM: poll until// the pid is gone, bounded, rather than sleeping a fixed window.const gone = await pollUntilGone(pid, Date.now() + 5_000);expect(gone).toBe(true);- }, 20000);+ }, 30000);test("a manifest recording no installs is skipped", async () => {writeConfig("");diff --git a/tests/core/search/writer-lock-lease.test.ts b/tests/core/search/writer-lock-lease.test.tsnew file mode 100644index 00000000..92969354--- /dev/null+++ b/tests/core/search/writer-lock-lease.test.ts@@ -0,0 +1,517 @@+/**+ * The writer lock as a lease: fresh while its holder is blocked in+ * synchronous work, bounded so a wedged holder still goes stale, and+ * ownership-checked so a holder that lost it never touches its+ * successor's lock.+ *+ * The stale window is shortened through its documented override. A lock+ * is made stale by backdating its directory rather than by waiting, and+ * a takeover is what a contender does on disk: the old directory removed,+ * a new one created. The other writer is a real second process wherever+ * the holder is blocked, since a blocked thread cannot run it itself.+ */++import {+ afterEach,+ beforeEach,+ expect,+ setDefaultTimeout,+ setSystemTime,+ spyOn,+ test,+} from "bun:test";+import { spawn, spawnSync } from "node:child_process";+import * as fs from "node:fs";+import {+ chmodSync,+ existsSync,+ mkdirSync,+ readFileSync,+ rmSync,+ statSync,+ utimesSync,+ writeFileSync,+} from "node:fs";+import { join } from "node:path";+import lockfile from "proper-lockfile";++import { indexVault, reindexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import {+ acquireWriterLease,+ acquireWriterLeaseSync,+ isWriterLockHeld,+ keepWriterLeasesFresh,+ writerLeaseCheckIn,+ WRITER_LOCK_STALE_ENV,+ type WriterLease,+} from "../../../src/core/search/store/writer-lock.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { tempDirs } from "../../helpers/temp-dir.ts";++setDefaultTimeout(30_000);++const STALE_MS = 1_500;+const HEARTBEAT_MS = STALE_MS / 3;+const WRITER_LOCK_MODULE = join(import.meta.dir, "../../../src/core/search/store/writer-lock.ts");++const mkTemp = tempDirs();+let dir: string;+let dbPath: string;+let lockDir: string;+let savedStale: string | undefined;++beforeEach(() => {+ dir = mkTemp("o2b-writer-lease-");+ dbPath = join(dir, "index.sqlite");+ lockDir = `${dbPath}.lock`;+ savedStale = process.env[WRITER_LOCK_STALE_ENV];+ process.env[WRITER_LOCK_STALE_ENV] = String(STALE_MS);+});++afterEach(() => {+ if (savedStale === undefined) delete process.env[WRITER_LOCK_STALE_ENV];+ else process.env[WRITER_LOCK_STALE_ENV] = savedStale;+});++/** Make the lock look abandoned: its mtime two stale windows in the past. */+function backdateLock(): void {+ const past = (Date.now() - 2 * STALE_MS) / 1000;+ utimesSync(lockDir, past, past);+}++/** What a takeover leaves on disk: the old directory gone, the successor's in its place. */+function takeOver(): WriterLease {+ rmSync(lockDir, { recursive: true });+ return acquireWriterLeaseSync(dbPath);+}++function codeOf(fn: () => unknown): string | null {+ try {+ fn();+ return null;+ } catch (err) {+ return (err as SearchError).code ?? String(err);+ }+}++/** Run a Bun script as another process, synchronously, with the short window. */+function runChild(name: string, source: string): string {+ const script = join(dir, `${name}.ts`);+ writeFileSync(script, source);+ const r = spawnSync(process.execPath, [script], {+ encoding: "utf8",+ env: { ...process.env, [WRITER_LOCK_STALE_ENV]: String(STALE_MS) },+ timeout: 20_000,+ });+ expect(r.status).toBe(0);+ return r.stdout;+}++const importLease = `import { acquireWriterLeaseSync } from ${JSON.stringify(WRITER_LOCK_MODULE)};`;++test("a holder blocked in a child process keeps the lock past two stale windows", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ try {+ // Two windows into the block, long after an event-loop heartbeat alone+ // would have let the lock go stale, another writer is still refused.+ const got = keepWriterLeasesFresh(60_000, () =>+ runChild(+ "contender",+ [+ importLease,+ `Bun.sleepSync(${2 * STALE_MS});`,+ "try {",+ ` acquireWriterLeaseSync(${JSON.stringify(dbPath)}).release();`,+ ' process.stdout.write("acquired");',+ "} catch (err) {",+ " process.stdout.write(String(err.code ?? err));",+ "}",+ ].join("\n"),+ ),+ );+ expect(got).toBe("INDEX_LOCKED");+ expect(lease.lost).toBe(false);+ expect(() => lease.assertHeld()).not.toThrow();+ } finally {+ lease.release();+ }+ expect(existsSync(lockDir)).toBe(false);+});++test("a holder still blocked past its budget goes stale and is taken over", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ try {+ const budgetMs = 300;+ // The contender keeps trying from the earliest moment a takeover could+ // succeed, so a refresher that started late cannot make it give up:+ // what it proves is that the lock does go stale while the holder+ // stays blocked. When the block ends, the call refuses: the lock is+ // not this holder's any more.+ let got = "";+ const refused = codeOf(() =>+ keepWriterLeasesFresh(budgetMs, () => {+ got = runChild(+ "late-contender",+ [+ importLease,+ `Bun.sleepSync(${budgetMs + STALE_MS});`,+ "const deadline = Date.now() + 10_000;",+ "let got = 'INDEX_LOCKED';",+ "while (Date.now() < deadline) {",+ " try {",+ ` acquireWriterLeaseSync(${JSON.stringify(dbPath)}).release();`,+ " got = 'acquired';",+ " break;",+ " } catch (err) {",+ " if (err.code !== 'INDEX_LOCKED') throw err;",+ " }",+ "}",+ "process.stdout.write(got);",+ ].join("\n"),+ );+ }),+ );+ expect(got).toBe("acquired");+ expect(refused).toBe("INDEX_LOCKED");+ expect(codeOf(() => lease.assertHeld())).toBe("INDEX_LOCKED");+ } finally {+ lease.release();+ }+});++test("a takeover while the refresher runs is not refreshed, and the call refuses when it returns", () => {+ const losses: string[] = [];+ const lease = acquireWriterLeaseSync(dbPath, { onLost: (err) => void losses.push(err.code) });+ try {+ // The successor takes the lock over and then sleeps through several+ // of the refresher's beats; its lock's mtime must be exactly what it+ // stamped when it woke up.+ let got = "";+ const refused = codeOf(() =>+ keepWriterLeasesFresh(60_000, () => {+ got = runChild(+ "successor",+ [+ 'import { rmSync, statSync } from "node:fs";',+ importLease,+ `rmSync(${JSON.stringify(lockDir)}, { recursive: true });`,+ `const lease = acquireWriterLeaseSync(${JSON.stringify(dbPath)});`,+ `const before = statSync(${JSON.stringify(lockDir)}).mtimeMs;`,+ `Bun.sleepSync(${4 * HEARTBEAT_MS});`,+ `const after = statSync(${JSON.stringify(lockDir)}).mtimeMs;`,+ 'process.stdout.write(after === before ? "untouched" : "stamped");',+ "lease.release();",+ ].join("\n"),+ );+ }),+ );+ expect(got).toBe("untouched");+ expect(refused).toBe("INDEX_LOCKED");+ expect(lease.lost).toBe(true);+ expect(losses).toEqual(["INDEX_LOCKED"]);+ } finally {+ lease.release();+ }+});++test("a holder that lost the lock to a takeover leaves its successor's lock in place", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ backdateLock();+ const successor = acquireWriterLeaseSync(dbPath);+ try {+ lease.release();+ expect(existsSync(lockDir)).toBe(true);+ expect(isWriterLockHeld(dbPath)).toBe(true);+ expect(codeOf(() => lease.assertHeld())).toBe("INDEX_LOCKED");+ expect(() => successor.assertHeld()).not.toThrow();+ } finally {+ successor.release();+ }+ expect(existsSync(lockDir)).toBe(false);+});++test("a lock near the end of its stale window is neither refreshed nor removed, even when it is ours", () => {+ // The holder's own lock, read on a clock that makes it almost stale: a+ // contender may be removing it at this moment, so stamping or removing+ // it could hit the contender's new lock instead.+ const nearlyStale = (): void => {+ setSystemTime(new Date(Date.now() + STALE_MS - 100));+ };+ const kept = acquireWriterLeaseSync(dbPath);+ nearlyStale();+ try {+ kept.release();+ } finally {+ setSystemTime();+ }+ expect(existsSync(lockDir)).toBe(true);+ rmSync(lockDir, { recursive: true });++ const checked = acquireWriterLeaseSync(dbPath);+ const mtime = statSync(lockDir).mtimeMs;+ nearlyStale();+ try {+ expect(codeOf(() => checked.assertHeld())).toBe("INDEX_LOCKED");+ } finally {+ setSystemTime();+ }+ expect(statSync(lockDir).mtimeMs).toBe(mtime);+ checked.release();+ rmSync(lockDir, { recursive: true });+});++test("a holder whose thread stalled past the margin, with nobody taking over, keeps its lease", () => {+ // A window long enough that the commit margin (half a heartbeat, 1.5 s)+ // and the beat's one-second guard below the window are apart.+ const windowMs = 9_000;+ process.env[WRITER_LOCK_STALE_ENV] = String(windowMs);+ const lease = acquireWriterLeaseSync(dbPath);+ try {+ // The next beat after a long stall (a sleeping laptop): the lock is+ // past the commit margin but still this holder's, so the beat+ // refreshes it.+ setSystemTime(new Date(Date.now() + windowMs - 1_200));+ try {+ writerLeaseCheckIn();+ } finally {+ setSystemTime();+ }+ expect(lease.lost).toBe(false);+ expect(() => lease.assertHeld()).not.toThrow();+ } finally {+ lease.release();+ }+});++test("a clock that went backwards does not let a lost lock pass for held", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ const successor = takeOver();+ try {+ // The lock's age now reads as negative; the forced check still looks.+ setSystemTime(new Date(Date.now() - 60 * 60_000));+ expect(codeOf(() => lease.assertHeld())).toBe("INDEX_LOCKED");+ } finally {+ setSystemTime();+ lease.release();+ successor.release();+ }+});++test("with a long stale window the lock still beats often enough for an older release", () => {+ // Older releases judge the lock by a fixed 60-second window, whatever+ // this one is set to: a 25-second-old lock must already be refreshed.+ process.env[WRITER_LOCK_STALE_ENV] = String(5 * 60_000);+ const lease = acquireWriterLeaseSync(dbPath);+ try {+ const stamped = statSync(lockDir).mtimeMs;+ setSystemTime(new Date(Date.now() + 25_000));+ try {+ writerLeaseCheckIn();+ } finally {+ setSystemTime();+ }+ expect(statSync(lockDir).mtimeMs).not.toBe(stamped);+ } finally {+ lease.release();+ }+});++test.skipIf(CHMOD_CANNOT_DENY)("a commit check that cannot read the lock refuses", () => {+ // The lock behind a directory this process may not search: whose lock+ // it is cannot be found out, which for a commit is not holding it.+ const guarded = join(dir, "guarded");+ mkdirSync(guarded);+ const path = join(guarded, "index.sqlite");+ const lease = acquireWriterLeaseSync(path);+ chmodSync(guarded, 0o600);+ try {+ expect(codeOf(() => lease.assertHeld())).toBe("INDEX_LOCKED");+ } finally {+ chmodSync(guarded, 0o700);+ lease.release();+ rmSync(`${path}.lock`, { recursive: true, force: true });+ }+});++test("a lock the filesystem refuses to stamp is reported as that fault, not as another writer", async () => {+ // Whichever call stamps the directory on this platform refuses it.+ const refuse = (): never => {+ throw Object.assign(new Error("EPERM: operation not permitted, futime"), { code: "EPERM" });+ };+ const spies = [+ spyOn(fs, "futimesSync").mockImplementation(refuse),+ spyOn(fs, "utimesSync").mockImplementation(refuse),+ ];+ let refused: unknown = null;+ try {+ await acquireWriterLease(dbPath);+ } catch (err) {+ refused = err;+ } finally {+ for (const spy of spies) spy.mockRestore();+ }+ expect((refused as NodeJS.ErrnoException | null)?.code).toBe("EPERM");+ expect(String(refused)).toContain("cannot stamp the search index writer lock");+ expect(String(refused)).not.toContain("another writer");+ // The directory the failed attempt made is not left behind.+ expect(existsSync(lockDir)).toBe(false);+});++test("protected work does not start on a lease that is already lost", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ const successor = takeOver();+ try {+ let ran = false;+ expect(+ codeOf(() =>+ keepWriterLeasesFresh(60_000, () => {+ ran = true;+ }),+ ),+ ).toBe("INDEX_LOCKED");+ expect(ran).toBe(false);+ } finally {+ lease.release();+ successor.release();+ }+});++test.skipIf(process.platform === "win32")(+ "the exit hook releases the lock on a signal, and leaves a successor's lock in place",+ async () => {+ const script = join(dir, "holder.ts");+ writeFileSync(+ script,+ [+ importLease,+ `acquireWriterLeaseSync(${JSON.stringify(dbPath)});`,+ 'process.stdout.write("ready\\n");',+ "setInterval(() => {}, 1_000);",+ ].join("\n"),+ );+ const holder = async (): Promise<{ kill: () => Promise<void> }> => {+ const child = spawn(process.execPath, [script], {+ env: { ...process.env, [WRITER_LOCK_STALE_ENV]: String(STALE_MS) },+ stdio: ["ignore", "pipe", "ignore"],+ });+ const exited = new Promise<void>((done) => child.once("exit", () => done()));+ await new Promise<void>((ready, fail) => {+ child.stdout.on("data", (chunk: Buffer) => {+ if (chunk.toString().includes("ready")) ready();+ });+ child.once("exit", () => fail(new Error("the holder exited before it was ready")));+ });+ return {+ kill: async () => {+ child.kill("SIGTERM");+ await exited;+ },+ };+ };++ // Its own lock: removed on the way out.+ const first = await holder();+ expect(existsSync(lockDir)).toBe(true);+ await first.kill();+ expect(existsSync(lockDir)).toBe(false);++ // A lock taken over from it: left in place.+ const second = await holder();+ backdateLock();+ const successor = acquireWriterLeaseSync(dbPath);+ try {+ await second.kill();+ expect(existsSync(lockDir)).toBe(true);+ expect(() => successor.assertHeld()).not.toThrow();+ } finally {+ successor.release();+ }+ },+);++test("the lock excludes, and is excluded by, an older release's proper-lockfile", () => {+ const lease = acquireWriterLeaseSync(dbPath);+ try {+ let refused: unknown = null;+ try {+ lockfile.lockSync(dbPath, { stale: 60_000, realpath: false });+ } catch (err) {+ refused = err;+ }+ expect((refused as NodeJS.ErrnoException | null)?.code).toBe("ELOCKED");+ expect(lockfile.checkSync(dbPath, { stale: 60_000, realpath: false })).toBe(true);+ } finally {+ lease.release();+ }++ const release = lockfile.lockSync(dbPath, { stale: 60_000, realpath: false });+ try {+ expect(codeOf(() => acquireWriterLeaseSync(dbPath))).toBe("INDEX_LOCKED");+ } finally {+ release();+ }++ // An older release's abandoned lock is taken over like any other.+ lockfile.lockSync(dbPath, { stale: 60_000, realpath: false, onCompromised: () => {} });+ backdateLock();+ const taken = acquireWriterLeaseSync(dbPath);+ expect(() => taken.assertHeld()).not.toThrow();+ taken.release();+});++test("a write-mode store whose lock was taken over refuses to go on writing", async () => {+ const v = createTempVault("lease-store");+ try {+ const config = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ const store = await Store.open(config, { mode: "write" });+ const successor = (() => {+ rmSync(`${v.dbPath}.lock`, { recursive: true });+ return acquireWriterLeaseSync(v.dbPath);+ })();+ try {+ // The next due beat sees the loss.+ Bun.sleepSync(HEARTBEAT_MS + 100);+ expect(codeOf(() => writerLeaseCheckIn())).toBe("INDEX_LOCKED");+ expect(() => store.setState("after_loss", "1")).toThrow();+ } finally {+ await store.close();+ successor.release();+ }+ } finally {+ v.cleanup();+ }+});++test("a reindex whose lock was taken over refuses its swap and leaves the live index", async () => {+ const v = createTempVault("lease-reindex");+ try {+ writeMd(v.vault, "note.md", "# Note\n\nbody text\n");+ const config = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ await indexVault(config);+ const live = readFileSync(v.dbPath);+ const liveIno = statSync(v.dbPath).ino;+ let successor: WriterLease | null = null;+ let refused: unknown = null;+ try {+ await reindexVault(config, {+ onFile: () => {+ if (successor !== null) return;+ rmSync(`${v.dbPath}.lock`, { recursive: true });+ successor = acquireWriterLeaseSync(v.dbPath);+ },+ });+ } catch (err) {+ refused = err;+ } finally {+ (successor as WriterLease | null)?.release();+ }+ expect((refused as SearchError | null)?.code).toBe("INDEX_LOCKED");+ expect(statSync(v.dbPath).ino).toBe(liveIno);+ expect(readFileSync(v.dbPath).equals(live)).toBe(true);+ } finally {+ v.cleanup();+ }+});diff --git a/tests/core/store/store-init.test.ts b/tests/core/store/store-init.test.tsindex c6860ea8..83b55f15 100644--- a/tests/core/store/store-init.test.ts+++ b/tests/core/store/store-init.test.ts@@ -167,18 +167,93 @@ describe("o2b store init", () => {expect([...(inspectStore(root)?.partitions ?? [])]).toEqual(["shared"]);});- test("concurrent runs agree on one store and only one reports creating it", async () => {- // The sandboxed HOME also holds Bun's transpiler cache. One run first- // fills it, so the six racing children start warm instead of each- // transpiling the whole CLI at once (26 to 32 s on the Windows runner).- const warm = await runCli(["version"], { env: env(), subprocess: true });- expect(warm.returncode).toBe(0);- const runs = await Promise.all(- [1, 2, 3, 4, 5, 6].map(() =>- runCli(["store", "init", "--json"], { env: env(), subprocess: true }),- ),+ test("six processes running the init verb at once agree on one store and only one reports creating it", async () => {+ // Six processes run the `store init` verb (its handler, not the whole+ // CLI entry; the single-run tests above cover the CLI) against one root. Each loads the verb,+ // says it is ready and waits for one go signal, so the six really do+ // race on the store, whatever the runner's load: a process start+ // (a transpile of the CLI on a cold Windows runner takes seconds) is+ // not part of the race and cannot stagger it into six serial runs or+ // eat the test's budget while one run is still starting.+ const runs = 6;+ const go = join(parent, "go");+ const child = join(parent, "racer.ts");+ const storeVerb = join(import.meta.dir, "../../../src/cli/store.ts");+ writeFileSync(+ child,+ [+ 'import { existsSync, writeFileSync } from "node:fs";',+ `import { handleStoreSubcommand } from ${JSON.stringify(storeVerb)};`,+ "writeFileSync(process.argv[2], '');",+ // Bounded on its own, so a racer whose test already gave up exits+ // instead of spinning after the run.+ "const deadline = Date.now() + 30_000;",+ `while (!existsSync(${JSON.stringify(go)})) {`,+ " if (Date.now() > deadline) process.exit(2);",+ " Bun.sleepSync(5);",+ "}",+ 'process.exitCode = await handleStoreSubcommand(["init", "--json"]);',+ ].join("\n"),);- const answers = runs.map((r) => JSON.parse(r.stdout) as Record<string, unknown>);+ const spawnRacer = (i: number) => {+ const home = join(parent, "home");+ const proc = Bun.spawn([process.execPath, child, join(parent, `ready-${i}`)], {+ cwd: parent,+ env: {+ ...process.env,+ ...env(),+ ...(process.platform === "win32" ? { USERPROFILE: home } : {}),+ // A machine config of its own per run, as each CLI run gets.+ OPEN_SECOND_BRAIN_CONFIG: join(parent, `config-${i}.yaml`),+ },+ stdout: "pipe",+ stderr: "pipe",+ });+ const output = Promise.all([+ new Response(proc.stdout).text(),+ new Response(proc.stderr).text(),+ proc.exited,+ ]);+ return { proc, output };+ };+ const racers: Array<ReturnType<typeof spawnRacer>> = [];+ const readyUpTo = (count: number): boolean =>+ Array.from({ length: count }, (_, i) => join(parent, `ready-${i}`)).every((f) =>+ existsSync(f),+ );+ const waitReady = async (count: number): Promise<void> => {+ while (!readyUpTo(count)) {+ // A run that died before it got ready would leave the others waiting.+ for (const { proc } of racers) {+ if (proc.exitCode !== null) {+ throw new Error(`a racer exited ${proc.exitCode} before the start`);+ }+ }+ await Bun.sleep(10);+ }+ };+ let results: Array<[string, string, number]>;+ try {+ // The first racer starts alone and fills Bun's transpiler cache, so+ // the other five start warm instead of transpiling the verb's whole+ // module graph six times at once on a slow runner.+ racers.push(spawnRacer(0));+ await waitReady(1);+ for (let i = 1; i < runs; i++) racers.push(spawnRacer(i));+ await waitReady(runs);+ writeFileSync(go, "");+ results = await Promise.all(racers.map((r) => r.output));+ } finally {+ // Whatever happened above, no racer outlives the test.+ for (const { proc } of racers) if (proc.exitCode === null) proc.kill("SIGKILL");+ }+ const answers = results.map(([stdout, stderr]) => {+ try {+ return JSON.parse(stdout) as Record<string, unknown>;+ } catch {+ throw new Error(`a racer answered no JSON: ${stdout}${stderr}`);+ }+ });const done = answers.filter((a) => a["ok"] === true);// A run that lost is refused for the store the others made, never for another reason.const refused = answers.filter((a) => a["ok"] !== true).map((a) => a["reason"]);@@ -188,7 +263,6 @@ describe("o2b store init", () => {const written = done.flatMap((a) => a["created"] as string[]);expect(written.toSorted()).toEqual(["private", "shared"]);expect(new Set(done.map((a) => a["storeId"]))).toEqual(new Set([inspectStore(root)?.storeId]));- // Six cold CLI processes start at once: a loaded CI runner needs more than the 5 s default.}, 30_000);test("with --json a usage error is a JSON answer too", async () => {