Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
37.9 KB
Raw Blame
/**
 * Synchronous lockfile primitive for the brain write path.
 *
 * The brain write path is sync end-to-end (`writePreference`, `dream`,
 * `moveToRetired`, `writeFrontmatterAtomic`) and migrating it to async
 * would touch every caller signature for one async-only ingredient.
 * Instead, this module provides a tiny single-attempt sync lock built
 * on `fs.openSync(target
 * + '.lock', 'wx')`. EEXIST surfaces as `Error` with
 * `.code === 'ELOCKED'`; the brain txn layer maps that to a
 * `BrainCollisionError({ kind: 'SourceLock' })`.
 *
 * No retry/backoff. Contention in OSB is rare (single operator, single
 * MCP server, dream runs from cron). When it does happen, we prefer a
 * loud typed error over a silent retry-then-still-fail loop.
 *
 * ## Why this exists beside `reliability/lock.ts`
 *
 * `withFileLock` there is the general-purpose ladder and it is ASYNC -
 * it awaits `proper-lockfile`. Every consumer of the waiting form here
 * (`updateManifest`, `appendGitRecords`, `recordCompleted`,
 * `generateRun`) is a synchronous function, and `ingestSource` above two
 * of them is synchronous on the published SDK surface, so awaiting a
 * lock would colour that function, its callers and the SDK contract.
 * Two ladders, two reasons; a reader who finds this one and wonders why
 * the other was not used is reading the right question, and this is the
 * answer.
 *
 * Ownership: a handle remembers the identity (device and inode) of the file
 * its exclusive create made, and {@link LockHandle.release} and the exit hook
 * remove the lock name only while it still names that file. A lock name
 * that something else removed and another process re-created belongs to
 * that process, and deleting it would let a third process in beside it; an
 * abandoned-lock takeover ({@link removeStaleLockSync}) is one remover, an
 * operator clearing a lock by hand another.
 *
 * Stale-lock recovery: on normal process exit the cleanup hook unlinks
 * any still-held locks. On hard crash (SIGKILL, OOM, or the second
 * Ctrl-C that falls through to the default SIGINT disposition) the
 * `.lock` file stays on disk and the next acquire fails with ELOCKED;
 * the brain doctor surfaces these via {@link scanStaleLocks} so an
 * operator can remove them by hand.
 *
 * ## Why this lock has no stale window, when the search writer lock does
 *
 * `search/store/writer-lock.ts` treats a lock older than
 * `WRITER_LOCK_STALE_MS` (60 s) as abandoned and takes it over, so a
 * SIGKILL never wedges the index for longer than that. This lock
 * deliberately has no such window, and the asymmetry is not an
 * oversight: the search lock can self-heal only because its holder
 * HEARTBEATS - a live writer lease refreshes its mtime every
 * `WRITER_LOCK_HEARTBEAT_MS`, from a worker thread while it is blocked
 * in a bounded synchronous call, which is what makes "old mtime" mean
 * "dead or wedged holder". A holder of THIS
 * lock is inside a synchronous critical section: the single thread is
 * busy doing the very work the lock protects, so no timer can refresh
 * anything. Here an age threshold would only mean "held for longer than
 * N", which a live writer on a large store reaches honestly - and
 * breaking a live writer's lock corrupts what it protects. So the
 * condition is REPORTED and the judgement is left to the operator; see
 * `doctor/uncertainty-probes.ts`.
 */

import {
  type BigIntStats,
  closeSync,
  constants as fsConstants,
  fstatSync,
  lstatSync,
  readFileSync,
  mkdirSync,
  openSync,
  readdirSync,
  unlinkSync,
  writeSync,
} from "node:fs";
import { randomUUID } from "node:crypto";
import { appendFileSync } from "node:fs";
import { basename, dirname, join } from "node:path";

import { BRAIN_ROOT_REL, DERIVED_STORE_DIR } from "./path-constants.ts";

export interface LockHandle {
  /** Filesystem path to the underlying `.lock` file. */
  readonly path: string;
  /**
   * Release the lock: remove the `.lock` file while its name still holds the
   * file this handle created, and leave a lock someone else now holds alone.
   * Idempotent: a second call is a no-op.
   */
  release(): void;
}

const LOCK_SUFFIX = ".lock";

/**
 * The file a holder created: device and inode as exact bigints. `null` when
 * the platform reports no inode (some network and FAT volumes answer 0).
 */
type LockIdentity = { readonly dev: bigint; readonly ino: bigint } | null;

/**
 * The body this holder wrote into its lock file: the unique token that
 * decides ownership when a POSIX filesystem hands a freed inode straight
 * back to the next create. Device and inode alone cannot tell a reused
 * identity apart, so release reads the stamp back and compares prefixes.
 * Empty on Windows, where ownership stays the device-and-inode check.
 */
type LockHolder = { readonly identity: LockIdentity; readonly token: string };

// Held locks and the file each holder created, tracked at module scope so the
// exit hook can clean up.
const heldLocks = new Map<string, LockHolder>();
let exitHookInstalled = false;

function ensureExitHook(): void {
  if (exitHookInstalled) return;
  exitHookInstalled = true;
  process.on("exit", () => {
    for (const [lockPath, holder] of heldLocks) {
      if (ownsLockName(lockPath, holder.identity, holder.token)) {
        try {
          unlinkSync(lockPath);
        } catch {
          // best-effort; the next acquire will see ELOCKED and surface it
        }
      }
    }
    heldLocks.clear();
  });
}

/** Identity of the file behind an open descriptor; `null` when it has none. */
function identityOf(fd: number): LockIdentity {
  try {
    const st = fstatSync(fd, { bigint: true });
    return st.ino === 0n ? null : { dev: st.dev, ino: st.ino };
  } catch {
    return null;
  }
}

/**
 * Whether `lockPath` still names the file this holder created. A missing name
 * is not owned (nothing to remove). The device and inode must match, and the
 * body must still open with this holder's token: a filesystem that reuses a
 * freed inode for the next holder's create answers the same identity for a
 * file this holder no longer owns, and only the stamp tells them apart. On a
 * volume without inodes the token alone decides.
 */
function ownsLockName(lockPath: string, identity: LockIdentity, token: string): boolean {
  let body: Buffer | null = null;
  try {
    body = readFileSync(lockPath);
  } catch {
    // A body that cannot be read (Windows holds a just-written file
    // briefly against readers) cannot be told from another holder's:
    // fall back to the device-and-inode check alone, the behaviour
    // before ownership tokens.
    body = null;
  }
  if (body !== null && token !== "") {
    // The stamp answers for the whole file: a matching token is owned,
    // a foreign one (a reused identity, a takeover survivor) is not. A
    // waiter mark that raced the stamp's own write clobbers the token's
    // first byte, so a body that still carries this holder's unique
    // token line is owned all the same; a foreign body cannot carry it.
    if (body.subarray(0, Buffer.byteLength(token, "utf8")).equals(Buffer.from(token, "utf8"))) {
      return true;
    }
    const uuidLine = Buffer.from(`\n${token.split("\n")[1] ?? ""}\n`, "utf8");
    return uuidLine.byteLength > 2 && body.includes(uuidLine);
  }
  try {
    const st = lstatSync(lockPath, { bigint: true });
    if (identity === null) return true;
    return st.dev === identity.dev && st.ino === identity.ino;
  } catch {
    return false;
  }
}

/**
 * Whether a failed exclusive create is Windows reporting a lock that is
 * mid-release rather than a real permission problem.
 *
 * Windows does not remove a file the moment another process unlinks it: the
 * name stays in a "delete pending" state until the last handle closes, and
 * any attempt to create that name meanwhile fails with `EPERM` (or `EACCES`)
 * instead of `EEXIST`. Under a multi-process race - the exact case these
 * locks exist for - a writer that loses by a few microseconds to a releasing
 * holder then crashed with a raw `EPERM` instead of waiting its turn. On
 * win32 that code is contention and goes through the same retry path as
 * `EEXIST`; the original error rides along as `cause`, so a directory that
 * genuinely refuses writes still names its real reason once the wait budget
 * runs out. POSIX never reports a held lock this way, so there it stays an
 * error.
 *
 * A directory that refuses to create files at all (an ACL, Defender's
 * Controlled Folder Access) answers the same `EPERM`, and treating THAT as
 * contention froze the caller for the whole wait budget and then reported
 * "lock busy". The two differ in whether anything holds the name: a
 * delete-pending or held lock file is there to `lstat` (or refuses the
 * `lstat` too), a refused create leaves nothing (`ENOENT`). Only the first
 * is contention. `platform` and `occupied` are test seams.
 */
export function isWindowsDeletePending(
  e: NodeJS.ErrnoException,
  lockPath: string,
  platform: NodeJS.Platform = process.platform,
  occupied: (path: string) => boolean = nameIsOccupied,
): boolean {
  if (platform !== "win32" || (e.code !== "EPERM" && e.code !== "EACCES")) return false;
  return occupied(lockPath);
}

/** Whether something holds `path`: it can be `lstat`ed, or refuses it with anything but ENOENT. */
function nameIsOccupied(path: string): boolean {
  try {
    lstatSync(path);
    return true;
  } catch (err) {
    return (err as NodeJS.ErrnoException).code !== "ENOENT";
  }
}

/**
 * Acquire an exclusive lock for `target`. Returns a handle whose
 * {@link LockHandle.release} method unlinks the underlying `.lock`
 * file while it is still the one this call created. Throws
 * `Error & { code: 'ELOCKED' }` if the lock is already held.
 *
 * The target file itself does NOT need to exist - first-time writes
 * acquire the lock before creating the target. The parent directory
 * is created on demand.
 */
export function acquireLockSync(target: string): LockHandle {
  ensureExitHook();
  const lockPath = target + LOCK_SUFFIX;
  mkdirSync(dirname(lockPath), { recursive: true });

  let fd = -1;
  for (let attempt = 0; fd < 0; attempt += 1) {
    try {
      fd = openSync(lockPath, "wx", 0o644);
    } catch (err) {
      const e = err as NodeJS.ErrnoException;
      if (e.code === "EEXIST" || isWindowsDeletePending(e, lockPath)) {
        const collision: NodeJS.ErrnoException = new Error(`lock busy: ${lockPath}`, { cause: e });
        collision.code = "ELOCKED";
        collision.path = lockPath;
        throw collision;
      }
      // A Windows EPERM with nothing at the name is either a directory that
      // refuses creates or a holder whose file finished going away between
      // the open and the lstat. One more try tells them apart.
      if (
        attempt === 0 &&
        process.platform === "win32" &&
        (e.code === "EPERM" || e.code === "EACCES")
      ) {
        continue;
      }
      throw err;
    }
  }

  const identity = identityOf(fd);

  // Stamp pid, a unique holder token and the timestamp into the lock
  // body. On POSIX the token decides ownership at release: a filesystem
  // there can hand a freed inode straight back to the next holder's
  // create, and only the stamp tells the files apart (the doctor
  // surface reads the rest when reporting stale locks). Windows keeps
  // the device-and-inode check alone: its delete-pending windows make
  // a just-written body unreliable to read back, and its volumes
  // allocate fresh identities for recreated files, where the reuse
  // race has never been observed. A failed write leaves no token and
  // release falls back to the same check.
  let stampBytes = 0;
  let token: string | null =
    process.platform === "win32" ? null : `${process.pid}\n${randomUUID()}`;
  try {
    const stamp = Buffer.from(`${token}\n${new Date().toISOString()}\n`, "utf8");
    stampBytes = writeSync(fd, stamp, 0, stamp.byteLength);
  } catch {
    // ignore; the body cannot be trusted as an ownership marker
    token = null;
  } finally {
    try {
      closeSync(fd);
    } catch {
      // ignore
    }
  }

  const holder: LockHolder =
    token !== null && stampBytes >= Buffer.byteLength(token, "utf8")
      ? { identity, token }
      : { identity, token: "" };
  heldLocks.set(lockPath, holder);
  let released = false;
  return {
    path: lockPath,
    release(): void {
      if (released) return;
      released = true;
      // A newer handle in this process may have re-registered the path; its
      // registration is the one the exit sweep must still see.
      if (heldLocks.get(lockPath) === holder) heldLocks.delete(lockPath);
      // The name no longer holds this holder's file: whoever holds it now
      // keeps it. Removing it would admit a second holder beside them.
      if (!ownsLockName(lockPath, holder.identity, holder.token)) return;
      // Anything past our own stamp was appended by a waiter (see
      // `markWaiting`): someone is queued for this lock, so a re-acquire from
      // this process steps back first. Best-effort, like the stamp itself.
      let waitedOn = false;
      try {
        waitedOn = lstatSync(lockPath).size > stampBytes;
      } catch {
        // gone already; nobody to hand over to
      }
      try {
        unlinkSync(lockPath);
      } catch {
        // already gone (race with exit hook or manual cleanup); ignore
      }
      if (waitedOn) contendedReleases.set(lockPath, Date.now());
    },
  };
}

/** Whether two stats describe the same file in the same state. */
function sameLockFile(a: BigIntStats, b: BigIntStats): boolean {
  return a.dev === b.dev && a.ino === b.ino && a.mtimeNs === b.mtimeNs;
}

/**
 * Remove an abandoned lock file: the file at `lockPath` when `stale` judges
 * it abandoned, only while the name still holds that same file, and only for
 * the one caller that claims that file first. Returns `true` when this call
 * removed it; the caller then makes a fresh acquire attempt like any other
 * contender. Never throws.
 *
 * A plain "stat, judge, unlink" lets two contenders that both judged the same
 * abandoned lock both remove something: the first removes the abandoned
 * lock and re-acquires, the second then removes that fresh lock and
 * acquires too, and from there each holder's release can remove the next
 * one's lock. Here the judgement and the removal are tied to one file.
 *
 * The claim is an exclusive create of a name derived from the judged file's
 * identity (`<lock>.takeover-<inode>-<mtime ns>`): every contender that judged
 * the same file races for the same claim name, and exclusive create is a
 * compare-and-swap on Linux, macOS and Windows alike, so exactly one of them
 * goes on. A fresh claim is never removed - it is the tombstone that turns
 * away a late contender still acting on its judgement of that same file
 * after the winner has re-acquired under the same name. A claim older than
 * {@link TAKEOVER_CLAIM_STALE_MS} turns nobody away any more: it is the
 * residue of a winner that died between the claim and the unlink, or of an
 * earlier takeover whose inode and mtime a new abandoned lock happens to
 * reuse. Such a claim does not block the takeover - contenders move on to
 * the next claim generation (`<lock>.takeover-<inode>-<mtime ns>-<n>`), again
 * one exclusive create every contender races for, so no claim is ever
 * deleted on the contended path. After a successful removal the winner
 * sweeps this lock's claims older than the threshold, so stores without a
 * sweep of their own (the session-capture worker lock in `tmpdir()`, the
 * lineage gap lock) do not accumulate them.
 *
 * The winner then re-reads the name and removes it only while it still
 * holds the judged file, so a lock that was replaced by a fresh one in the
 * meantime is never removed. Between that read and the unlink the name can
 * only change if the judged lock's own holder releases it, which the
 * staleness judgement presumes dead, as long as every other remover of the
 * lock goes through this function.
 */
export function removeStaleLockSync(
  lockPath: string,
  stale: (st: BigIntStats) => boolean,
): boolean {
  let seen: BigIntStats;
  try {
    seen = lstatSync(lockPath, { bigint: true });
    if (!stale(seen)) return false;
  } catch {
    return false;
  }
  const claimed = claimTakeover(`${lockPath}.takeover-${seen.ino}-${seen.mtimeNs}`);
  if (claimed === "exhausted") {
    // Every generation is a dead claim: clear them so the next attempt can claim.
    sweepTakeoverClaims(lockPath);
    return false;
  }
  if (!claimed) return false;
  try {
    if (!sameLockFile(lstatSync(lockPath, { bigint: true }), seen)) return false;
    unlinkSync(lockPath);
  } catch {
    return false;
  }
  sweepTakeoverClaims(lockPath);
  return true;
}

/**
 * Age past which a takeover claim no longer turns a contender away. A winner
 * goes from its claim to the unlink in microseconds, and a contender acts on
 * its judgement within one acquire attempt, so a minute-old claim belongs to
 * nobody still running.
 */
export const TAKEOVER_CLAIM_STALE_MS = 60_000;

/** How many claim generations one judged lock file may need (see {@link removeStaleLockSync}). */
const TAKEOVER_CLAIM_GENERATIONS = 4;

function claimIsStale(path: string): boolean {
  try {
    return Date.now() - lstatSync(path).mtimeMs > TAKEOVER_CLAIM_STALE_MS;
  } catch {
    return false;
  }
}

/**
 * Exclusive-create the first free claim generation for one judged lock file.
 * An existing fresh claim means another contender won; an existing stale
 * claim sends every contender to the next generation. `"exhausted"` means
 * every generation holds a stale claim.
 */
function claimTakeover(base: string): boolean | "exhausted" {
  for (let gen = 0; gen < TAKEOVER_CLAIM_GENERATIONS; gen++) {
    const claim = gen === 0 ? base : `${base}-${gen}`;
    try {
      closeSync(openSync(claim, "wx", 0o600));
      return true;
    } catch (err) {
      if ((err as NodeJS.ErrnoException).code !== "EEXIST") return false;
      if (!claimIsStale(claim)) return false;
    }
  }
  return "exhausted";
}

/** Remove this lock's takeover claims older than {@link TAKEOVER_CLAIM_STALE_MS}. Never throws. */
function sweepTakeoverClaims(lockPath: string): void {
  const prefix = `${basename(lockPath)}.takeover-`;
  let names: string[];
  try {
    names = readdirSync(dirname(lockPath));
  } catch {
    return;
  }
  for (const name of names) {
    if (!name.startsWith(prefix)) continue;
    const claim = join(dirname(lockPath), name);
    if (!claimIsStale(claim)) continue;
    try {
      unlinkSync(claim);
    } catch {
      // already gone
    }
  }
}

/**
 * How long {@link acquireLockSyncWithRetry} keeps waiting, and the ceiling on
 * one sleep between attempts.
 *
 * The budget is a WALL-CLOCK deadline rather than an attempt count because
 * what a waiter queues behind is other writers' work, not other writers'
 * sleeps: with n processes each folding one source into a shared file, the
 * last arrival waits for up to n-1 whole critical sections. An attempt count
 * bounds the number of naps, which is the wrong quantity.
 *
 * The sleep is JITTERED across `[1, RETRY_SLEEP_CEILING_MS]` rather than
 * fixed. A fixed sleep makes contenders that started together retry in
 * lockstep - they wake on the same tick, one wins, the rest sleep another
 * whole tick - so the same budget buys far fewer real attempts.
 *
 * ## Five seconds, for a different reason than the one first given
 *
 * The budget IS the freeze (see {@link acquireLockSyncWithRetry}), so five
 * seconds is a claim worth checking rather than assuming. The claim made
 * here was that it is "generous against a hold measured in milliseconds".
 * Both halves of that were measured, and the first half is false:
 *
 * | rows in the store | `appendGitRecords` hold | `updateManifest` hold |
 * |---|---|---|
 * | 1 000 | 23 ms | 19 ms |
 * | 10 000 | 134 ms | 81 ms |
 * | 50 000 | 465 ms | 262 ms |
 *
 * The holds are linear in the store they guard - a 50 000-commit repo is
 * most of a second per critical section, not milliseconds. But the hold is
 * not what sizes the budget. Measured across a four-process race over a
 * 400-entry manifest (100 acquisitions), the WAIT distribution is
 * `p50 = 0 ms, p90 = 1 ms, p99 = 732 ms, max = 732 ms`: almost every
 * acquisition is uncontended, and the tail is set by queueing, not by one
 * hold. The tail was long because nothing here was fair - a writer in a loop
 * re-acquired with no backoff at all while every waiter was mid-sleep, so a
 * waiter could lose many draws in a row. On a host whose hold is slower (the
 * Windows runner, about 80 ms per checkpoint write) that same unfairness
 * spent the whole budget: the checkpoint race was refused there, and a local
 * replay with an 80 ms hold refused 2 of 300 writes at 5 000 ms.
 *
 * ## The hand-over, and what it measured
 *
 * A waiter now marks the held lock file ({@link markWaiting}); a holder that
 * releases a marked lock and comes straight back for it steps back for longer
 * than any waiter sleeps, and a waiter that has already waited a while polls
 * on a shorter ceiling. The four-process checkpoint race, 100 acquisitions
 * per run, before and after:
 *
 * | host | before p50 / p99 / max | after p50 / p99 / max |
 * |---|---|---|
 * | Linux, 8 ms hold | 0 / 395 / 604 ms | 62 / 155 / 185 ms |
 * | Linux, 80 ms hold | 0 / 4 582 / 5 012 ms, 2 refused | 247 / 508 / 753 ms |
 * | Linux, 210 ms hold | 1 / 5 016 / 5 017 ms, 4 refused | 427 / 1 905 / 2 124 ms |
 * | WSL on an NTFS drive, 26 ms hold | 9 / 654 / 881 ms | 97 / 313 / 328 ms |
 *
 * The median rises because a waiter now actually waits its turn instead of
 * the looping writer taking every draw; the tail, which is what the budget
 * has to cover, falls to a few holds. The cost is idle time at each
 * hand-over (until the next waiter wakes, at most one sleep ceiling), paid
 * only when someone is waiting. The default stays at five seconds: after the
 * hand-over it is more than six times the longest wait at the Windows pace.
 * A slower host moves it with {@link LOCK_WAIT_BUDGET_ENV}.
 *
 * That tail is why the budget cannot simply be made small: at 1 000 ms the
 * branch's own designed-contention test (`updateManifest - concurrent
 * writers`, four real processes) starts failing intermittently with
 * `ELOCKED` on a perfectly healthy race - a budget at the p99 of the thing
 * it is meant to absorb is a coin flip. Five seconds is about six times
 * that p99, which is the margin that makes reaching it mean "something is
 * wrong" rather than "busy".
 *
 * What a caller gives up is therefore stated plainly rather than argued
 * away: in the worst case this freezes the process for five seconds. See
 * {@link acquireLockSyncWithRetry} for what bounds that, and
 * {@link LOCK_WAIT_INTERACTIVE_MS} for the callers that refuse to pay it.
 */
export const LOCK_WAIT_BUDGET_MS = 5_000;

/**
 * The environment variable that moves {@link LOCK_WAIT_BUDGET_MS} for every
 * caller that does not pass its own budget - the ingest writers, the session
 * ledger, the payload store. A host where one critical section is slow (an
 * antivirus scanning every write, a network or synced folder) can give
 * parallel ingest a longer wait without a rebuild. It never touches
 * {@link LOCK_WAIT_INTERACTIVE_MS}: a caller that chose to refuse rather
 * than freeze keeps that choice.
 */
export const LOCK_WAIT_BUDGET_ENV = "OPEN_SECOND_BRAIN_LOCK_WAIT_MS";

/**
 * The default wait budget this process uses, in milliseconds.
 *
 * A value that is not a non-negative integer THROWS rather than falling
 * back to the default, in the same discipline as `O2B_MCP_DRAIN_MS`: an
 * operator who set the variable did so to change the wait, and quietly
 * waiting five seconds because they typed `20s` would hide that.
 */
export function resolveLockWaitBudgetMs(env: NodeJS.ProcessEnv = process.env): number {
  const raw = env[LOCK_WAIT_BUDGET_ENV];
  if (raw === undefined || raw.trim() === "") return LOCK_WAIT_BUDGET_MS;
  const text = raw.trim();
  const parsed = /^\d+$/.test(text) ? Number(text) : Number.NaN;
  if (!Number.isSafeInteger(parsed)) {
    throw new Error(
      `${LOCK_WAIT_BUDGET_ENV}=${raw} is not a lock wait; set it to a whole number of ` +
        `milliseconds >= 0 (the default is ${LOCK_WAIT_BUDGET_MS}), or unset it`,
    );
  }
  return parsed;
}

/**
 * The budget for a caller whose contention is RARE and whose freeze is
 * expensive - an interactive CLI pass with a progress stream and a Ctrl-C
 * the operator expects to land.
 *
 * The default above is sized for the ingest fan-out: several processes
 * folding results into one file, by design, where a long wait buys a
 * healthy ingest and a freeze costs a short-lived worker nothing. The
 * architect is the opposite case on both counts. Its contender is a second
 * architect run over the same repo, which is not a designed workload, and
 * its process is one an operator is watching: five seconds parked in
 * `Bun.sleepSync` is five seconds of stalled progress events and an
 * undelivered SIGINT. One second is still above the p90 of the measured
 * wait distribution, so a genuine brief overlap is absorbed; beyond that
 * the operator gets a named `ELOCKED` and a prompt back, which is a better
 * answer than a frozen terminal.
 */
export const LOCK_WAIT_INTERACTIVE_MS = 1_000;

const RETRY_SLEEP_CEILING_MS = 25;

/**
 * Lock paths this process released while another process was waiting on
 * them, with the release time. Read (and cleared) by the next
 * {@link acquireLockSyncWithRetry} of the same path.
 */
const contendedReleases = new Map<string, number>();

/**
 * How long after a contended release a re-acquire still counts as "coming
 * straight back". A waiter sleeps at most {@link RETRY_SLEEP_CEILING_MS}
 * between draws, so past a few ceilings every waiter has had its turn and
 * stepping back would only slow the returning writer down.
 */
const HANDOFF_WINDOW_MS = RETRY_SLEEP_CEILING_MS * 4;

/**
 * The pause a returning writer takes before it re-acquires a lock it just
 * released while someone waited: always longer than a waiter's longest
 * sleep, so at least one waiter wakes and draws inside it, and randomised so
 * two returning writers do not step back in lockstep.
 */
function handoffPauseMs(): number {
  return RETRY_SLEEP_CEILING_MS * 2 + Math.floor(Math.random() * RETRY_SLEEP_CEILING_MS);
}

/**
 * How long an aged waiter's ceiling is. After a hand-over the lock goes to
 * whichever waiter draws first, so a waiter that has already waited through
 * several hand-overs polls on a shorter ceiling and wins the next one more
 * often: an approximation of first come, first served without a queue file
 * a crash could strand.
 */
const AGED_WAITER_AFTER_MS = RETRY_SLEEP_CEILING_MS * 10;
const AGED_WAITER_CEILING_MS = 5;

function waiterSleepMs(waitedMs: number): number {
  const ceiling =
    waitedMs >= AGED_WAITER_AFTER_MS ? AGED_WAITER_CEILING_MS : RETRY_SLEEP_CEILING_MS;
  return 1 + Math.floor(Math.random() * ceiling);
}

/** Upper bound on the bytes waiters append to one lock file. */
const WAITER_MARK_LIMIT = 4_096;
/**
 * The smallest body a COMPLETED stamp has, per platform. A mark at or below
 * it would land inside a stamp still being written (the exclusive create
 * exists a few microseconds before the stamp's bytes do), so such a file is
 * left unmarked. POSIX stamps `pid\nuuid\n<iso>\n` - never under 48 bytes.
 * Windows writes no token line, so its completed stamp is the literal
 * `null` token, the 24-character ISO timestamp and two separators: exactly
 * 30 bytes. Sizing one floor for the POSIX stamp kept every completed
 * Windows body under it forever: no waiter could ever announce itself, the
 * holder learned nothing at release, and the hand-over - the whole reason
 * the mark exists - was dead on win32.
 */
const POSIX_MIN_STAMP_BYTES = 48;
const WIN32_MIN_STAMP_BYTES = 30;
const MIN_STAMP_BYTES =
  process.platform === "win32" ? WIN32_MIN_STAMP_BYTES : POSIX_MIN_STAMP_BYTES;
const WAITER_MARK = Buffer.from("w", "utf8");
/** Read-write, never create, and never follow a symlink where the platform can refuse one. */
const MARK_OPEN_FLAGS = fsConstants.O_RDWR | (fsConstants.O_NOFOLLOW ?? 0);

/**
 * Tell the holder of `lockPath` that someone is waiting, by appending one
 * byte to its lock file. The file is opened WITHOUT create: a lock
 * released a moment ago must never be re-created by a waiter, which would be
 * a lock nobody holds. Every failure is ignored - the mark only buys
 * fairness, and the exclusive create is still what grants the lock.
 */
function markWaiting(lockPath: string): void {
  let fd = -1;
  try {
    fd = openSync(lockPath, MARK_OPEN_FLAGS);
    const stat = fstatSync(fd);
    // Only a lock file this module could have created: a regular file with
    // one name. A symlink or hard link planted at the lock path (a synced
    // vault can carry one) must not turn the mark into a write elsewhere.
    if (!stat.isFile() || stat.nlink !== 1) return;
    // A body still inside its own stamp write (a waiter can open the
    // name in the microseconds before the holder's stamp lands) is left
    // alone: an early mark would write over the holder token position.
    if (stat.size < MIN_STAMP_BYTES) return;
    if (stat.size < WAITER_MARK_LIMIT) {
      writeSync(fd, WAITER_MARK, 0, WAITER_MARK.byteLength, stat.size);
    }
  } catch {
    // released, delete-pending or unreadable: nothing to mark
  } finally {
    if (fd >= 0) {
      try {
        closeSync(fd);
      } catch {
        // ignore
      }
    }
  }
}

/** What an expired wait tells the operator to do next, by who set the budget. */
const REMEDY_EXPLICIT_BUDGET = "retry once the other writer finishes";

function defaultBudgetRemedy(budgetMs: number): string {
  return (
    `retry once the other writer finishes, or allow a longer wait with ` +
    `${LOCK_WAIT_BUDGET_ENV}=<milliseconds> (now ${budgetMs})`
  );
}

/**
 * The next step a refused INGEST write names. Passed by the writers the
 * parallel ingest fan-out shares (the content manifest, the plan and session
 * checkpoints, the git record store, the session ledger), where running the
 * workers one at a time is the other honest answer.
 */
export function ingestLockRemedy(budgetMs: number = resolveLockWaitBudgetMs()): string {
  return (
    `another process is writing the same file: retry the ingest, run parallel ingests ` +
    `one at a time, or allow a longer wait with ${LOCK_WAIT_BUDGET_ENV}=<milliseconds> ` +
    `(now ${budgetMs})`
  );
}

export interface LockWaitOptions {
  /** What the refusal tells the operator to do next, replacing the generic advice. */
  readonly remedy?: (budgetMs: number) => string;
}

/**
 * {@link acquireLockSync} with a bounded wait, for the shared files that
 * PARALLEL processes write by design.
 *
 * The single-attempt policy above is justified by contention being rare -
 * one operator, one MCP server, dream on cron. The ingest path breaks that
 * premise on purpose: `ingest/batch-plan.ts` exists so a caller dispatches
 * each batch as its own subagent, so several processes fold their result
 * into the content manifest, the plan checkpoint and the git record store
 * concurrently. There, an immediate `ELOCKED` would turn a millisecond
 * overlap into a failed ingest.
 *
 * Waiting is bounded and still loud: only `ELOCKED` is retried, any other
 * error propagates at once, and an expired budget throws an `ELOCKED`
 * (`lock busy: <lock file>`, how long it waited, and what to do next; the
 * last collision rides along as `cause`) rather than proceeding unlocked.
 * Nothing is ever silently skipped. `budget` omitted means the default,
 * moved by {@link LOCK_WAIT_BUDGET_ENV}; `options.remedy` replaces the
 * generic advice with the caller's own (see {@link ingestLockRemedy}).
 *
 * ## The wait is a FREEZE, and that is the cost
 *
 * `Bun.sleepSync` blocks the single JavaScript thread. This does not "wait",
 * it stops the process: for the whole budget no timer fires, no progress
 * event is emitted, no signal handler runs (a Ctrl-C is queued, not
 * delivered), and inside the MCP server no other tool call proceeds.
 * `tests/core/brain/sync-lockfile.test.ts` pins both halves of that - the
 * freeze is total, and it ends when the budget does.
 *
 * The freeze is accepted rather than removed, for three reasons that were
 * checked rather than assumed:
 *
 *  - Removing it is not a local change. The Brain write path is synchronous
 *    end to end, and `ingestSource` - the caller that reaches two of these
 *    three sites - is a synchronous function on the published SDK surface
 *    (`brain/sdk.ts`). An awaited lock would have to colour it, its callers
 *    and that contract async for one ingredient. Worth doing on its own
 *    terms; not worth smuggling in beside a lock fix.
 *  - Shrinking it to a value nobody would notice does not work either: at
 *    1 000 ms the four-process manifest race fails on healthy contention.
 *    See {@link LOCK_WAIT_BUDGET_MS} for the measured distribution.
 *  - Within ONE process this lock can never be contended. The whole
 *    acquire-mutate-release runs inside a single synchronous turn, so a
 *    second `brain_ingest_source` call on the same MCP server cannot even
 *    begin while the first holds it. An `ELOCKED` inside a server therefore
 *    always means a DIFFERENT process - never the parallel fan-out this
 *    function exists for. The server can still be frozen by an external
 *    writer; what it cannot do is freeze itself.
 *
 * A caller that would rather refuse than freeze says so with
 * {@link LOCK_WAIT_INTERACTIVE_MS}.
 */
export function acquireLockSyncWithRetry(
  target: string,
  budget?: number,
  options: LockWaitOptions = {},
): LockHandle {
  const budgetMs = budget ?? resolveLockWaitBudgetMs();
  // A budget this loop cannot honour is named rather than quietly
  // reinterpreted, in the same discipline as `requireCap` and the lease's
  // TTL guard. `NaN` is the reason the guard exists: `Date.now() >= NaN` is
  // false forever, so a non-finite budget turned a bounded freeze into an
  // unbounded one that no signal could break. Zero is legal and means
  // exactly one attempt - the deadline is only consulted after a failure.
  if (!Number.isInteger(budgetMs) || budgetMs < 0) {
    throw new Error(`lock wait budgetMs must be an integer >= 0, got ${budgetMs}`);
  }
  const lockPath = target + LOCK_SUFFIX;
  const started = Date.now();
  const deadline = started + budgetMs;
  // Fairness: a writer in a loop re-creates the lock file microseconds after
  // it unlinks it, while every waiter is mid-sleep, so without this a waiter
  // loses draw after draw until its budget runs out on a healthy race. When
  // this process released the lock while someone waited, it steps back for
  // longer than any waiter sleeps, so one of them takes the lock first.
  const releasedAt = contendedReleases.get(lockPath);
  if (releasedAt !== undefined) {
    contendedReleases.delete(lockPath);
    if (started - releasedAt < HANDOFF_WINDOW_MS && budgetMs > 0) {
      Bun.sleepSync(Math.min(handoffPauseMs(), budgetMs));
    }
  }
  for (;;) {
    try {
      return acquireLockSync(target);
    } catch (err) {
      if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
      if (Date.now() >= deadline) {
        const remedy =
          options.remedy?.(budgetMs) ??
          (budget === undefined ? defaultBudgetRemedy(budgetMs) : REMEDY_EXPLICIT_BUDGET);
        const refused: NodeJS.ErrnoException = new Error(
          `lock busy: ${lockPath}: still held after waiting ${Date.now() - started} ms; ${remedy}`,
          { cause: err },
        );
        refused.code = "ELOCKED";
        refused.path = lockPath;
        throw refused;
      }
      markWaiting(lockPath);
      Bun.sleepSync(waiterSleepMs(Date.now() - started));
    }
  }
}

/**
 * Every directory in `vault` a Brain `.lock` can appear under.
 *
 * ENUMERATED here, in the module that owns the lock, rather than chosen by
 * whoever calls the scan. The doctor used to walk one hard-coded root,
 * `<vault>/Brain/` - and two of the three locks the parallel ingest path
 * takes (the content manifest and the plan checkpoint) live in the
 * derived-store directory beside it, so a crashed ingest left a lock the
 * doctor answered `[]` about while it sat plainly on disk.
 *
 * These two are the only directories any lock is taken in today - checked
 * against every path builder that feeds a lock target, not assumed. A new
 * lock site under a third root has to be added here, and
 * the census in `tests/core/brain/sync-lockfile.test.ts` reads the sources
 * to make sure it is: the failure this closes is not one wrong path, it is
 * a scan that silently stops being complete.
 */
export function lockScanRoots(vault: string): string[] {
  return [join(vault, BRAIN_ROOT_REL), join(vault, DERIVED_STORE_DIR)];
}

/**
 * Walk every {@link lockScanRoots} directory of `vault` recursively and
 * return each `.lock` file path. Used by `brain_doctor` to surface stale
 * locks left behind by a crashed process.
 *
 * Takes the VAULT, not a root, so no caller can narrow the scan to a
 * subset of the places a lock lives. A root that does not exist
 * contributes nothing - `walkLocks` already treats an unreadable
 * directory as empty.
 */
export function scanStaleLocks(vault: string): string[] {
  const out: string[] = [];
  for (const root of lockScanRoots(vault)) walkLocks(root, out);
  return out;
}

function walkLocks(dir: string, out: string[]): void {
  let entries;
  try {
    entries = readdirSync(dir, { withFileTypes: true });
  } catch {
    return;
  }
  for (const entry of entries) {
    const full = join(dir, entry.name);
    if (entry.isDirectory()) {
      walkLocks(full, out);
      continue;
    }
    if (!entry.isFile()) continue;
    if (entry.name.endsWith(LOCK_SUFFIX)) {
      out.push(full);
    }
  }
}

/**
 * Test-only: clear the held-locks tracking set without unlinking. Used
 * to keep cross-test state from leaking when a test deliberately
 * leaves a lock on disk for the next test to discover.
 *
 * The leading underscore is this repo's marker for a test-only export,
 * which is exactly what the rule below flags - same disable as
 * `search/benchmark.ts` uses for `_benchmarkQueryPeakForTests`.
 */
// oxlint-disable-next-line no-underscore-dangle
export function _resetHeldLocksForTests(): void {
  heldLocks.clear();
}