Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Session-lineage ledger (continuity-hygiene-freshness suite).
*
* Append-only JSONL under `Brain/.state/` (dot-directory: excluded
* from vault walking and search indexing, like `.snapshots`). One line
* per lifecycle observation: session id, timestamp, cwd, event name,
* whether the event evidences a compression boundary, and the resolved
* lineage when the caller already knows it.
*
* The ledger exists for one consumer: the interim crutch resolution in
* `crutch.ts` (CRUTCH(t_1459706f)). Once the host emits native lineage
* fields the ledger keeps working as a cache of resolved links but is
* no longer load-bearing.
*
* Contract for capture callers: resolve lineage BEFORE recording the
* session's own observation - the crutch treats "this session already
* has ledger history without a link" as proof it is a parallel
* session, not a continuation.
*
* Everything here is fail-soft: a missing ledger reads as empty,
* corrupt lines are skipped, and the file is compacted in place
* (atomic rewrite) once it grows past the line cap. Skipped lines are
* COUNTED and returned ({@link scanLineageLedger}) rather than merely
* dropped - fail-soft for the resolver, reportable for the verifier.
*
* ## Integrity (context-integrity-gates, Unit D)
*
* Every line carries a SEQUENCE NUMBER and a HASH CHAIN link, so a line
* that was edited, removed, or never written is detectable after the
* fact - by {@link import('./verify.ts').verifyLineageLedger}, which
* REPORTS and never refuses. The read path below deliberately does not
* consult the chain: `readLineageLedger`'s catch-to-empty-map is
* load-bearing for both the anticipatory cache and the lifecycle hook,
* and a ledger that refused to load would break resolution rather than
* degrade it. Integrity here is an audit property, not a gate.
*
* The whole read-modify-write runs under a SYNCHRONOUS writer lock
* (`acquireLockSync`, the same primitive the continuity store and the
* idempotency ledger use, re-reading under the lock exactly as they
* do). Before it, two hooks crossing the compaction cap could both read
* the file and the second rewrite could clobber the first's
* observation. `acquireLockSync` has no retry by design; a busy lock
* therefore DROPS the observation, and the drop is written to a gap
* sidecar and returned as a named notice rather than vanishing.
*
* Compaction retains the most recent lines VERBATIM. It used to fold
* the file to one summary line per session, which destroyed per-session
* event history, dropped `firstSeenMs` (the summary was stamped at
* `lastSeenMs`, so a reread saw first === last), deleted every session
* past the retention rank, and - through a re-serializer that hard-
* coded its field list - silently dropped any field a later writer
* added. Verbatim retention removes all four at once, including the
* last as a class of bug rather than an instance.
*
* ## Declared work identity (signals-that-survive, Unit 9)
*
* A line may also carry a DECLARED work id and lane id (`wid` / `lane`),
* sourced at the capture boundary by `identity.ts` and never derived
* here. They are the durable half of continuation: the crutch's freshness
* window expires, a work id does not. Both are optional and are omitted
* when blank, so a ledger written by a host that declares nothing is
* byte-identical to the one this module wrote before they existed - and
* because the hash covers the line body by REMOVAL of the chain fields,
* they ride the integrity chain without any change to it.
*/
import { appendFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
import { dirname, join, posix } from "node:path";
import { atomicWriteFileSync } from "../../fs-atomic.ts";
import {
DEGRADATION_CODE,
type DegradationNotice,
degradationNotice,
} from "../../integrity/degradation.ts";
import { canonicalizeGitRemote, type GitWorkspaceIdentity } from "../git/reader.ts";
import { computePayloadHash } from "../idempotency-ledger.ts";
import {
jsonlLedgerGrammar,
listShardedFiles,
literalBase,
mergeShardedRows,
resolveAppendShardId,
shardedFileName,
type LedgerShardGrammar,
type ShardedRow,
} from "../ledger-shards.ts";
import { BRAIN_ROOT_REL, ensureInsideVault } from "../paths.ts";
import { machineStatePath } from "../../state/machine-state.ts";
import { acquireLockSync, type LockHandle, removeStaleLockSync } from "../sync-lockfile.ts";
import { assertVaultIdentityForWrite, VaultIdentityMismatchError } from "../vault-identity.ts";
import type { SessionLineage } from "./types.ts";
import { dualWrite } from "../../store/dual-write.ts";
import { readLedgerRows } from "../../store/read/ledger-rows.ts";
import { isStoreOnly } from "../../store/store-only.ts";
import { storeWrite } from "../../store/store-write.ts";
import { DUAL_WRITE_LEDGERS, ledgerRowRecord } from "../../store/dual-write-records.ts";
/** Crutch link window: a continuation must start this close to the
* predecessor's compression evidence. CRUTCH(t_1459706f). */
export const CRUTCH_LINK_WINDOW_MS = 900_000;
/** Compact the ledger once it holds more lines than this. */
const MAX_LEDGER_LINES = 512;
/**
* LINES retained verbatim by a compaction rewrite - not sessions. The
* unit changed with the fold: a summary kept one line per session and
* lost that session's history, whereas keeping raw lines keeps history
* and loses only the oldest of it.
*/
const RETAIN_LEDGER_LINES = 256;
/** Gap records retained; the sidecar must not grow without bound either. */
const MAX_GAP_LINES = 256;
/**
* How long a recorded gap keeps being reported.
*
* Without an aging policy one burst of writer contention makes
* `verifyLineageLedger().ok` false forever and re-emits the same notices
* on every doctor run, which trains an operator to ignore the stream the
* whole unit exists to populate. A gap is a point-in-time observation,
* not a standing defect, so it expires.
*/
const GAP_RETENTION_MS = 7 * 24 * 60 * 60 * 1000;
/**
* Age at which the SIDECAR's OWN trim lock is treated as abandoned.
*
* This is a breaker for one lock and one operation, and it is scoped
* that narrowly on purpose. The ledger's writer lock never gets one: no
* timeout can distinguish a crashed writer from a slow live one, and
* killing a live writer's lock corrupts the file it protects. The trim
* is different in kind - it is idempotent (keep the newest N records),
* it protects diagnostics rather than the ledger, and the alternative is
* that one crash inside the trim window disables the bound permanently.
*/
const GAP_LOCK_STALE_MS = 60_000;
const STATE_DIR_REL = posix.join(BRAIN_ROOT_REL, ".state");
/**
* Shard bases. The two are distinct stems rather than one stem with a
* suffix, so `session-lineage-gaps.jsonl` can never parse as a shard of
* `session-lineage`: the grammar splits on a dot, and `-gaps` is part of
* the base.
*/
const LEDGER_BASE = "session-lineage";
const GAPS_BASE = "session-lineage-gaps";
const LINEAGE_EXT = "jsonl";
/**
* The file-name layouts of the Brain state directory, exported for the
* attestation sweep: normal growth here is exactly a new shard of one of
* these two stems (the ledger, its gap sidecar), so a verifier can tell
* an append from a planted name without restating the stems.
*/
export const LINEAGE_STATE_GRAMMARS: ReadonlyArray<LedgerShardGrammar> = Object.freeze([
jsonlLedgerGrammar(LEDGER_BASE),
jsonlLedgerGrammar(GAPS_BASE),
]);
/** Site recorded on every notice this module emits. */
const LEDGER_SITE = "brain.lineage.ledger";
/** Why an observation never reached the ledger. Structural, closed. */
export const LINEAGE_GAP_REASON = Object.freeze({
/** Another writer held the lock; `acquireLockSync` has no retry. */
lockBusy: "writer-lock-busy",
/** The lock could not be taken for any other reason (permissions, io). */
lockFailed: "writer-lock-failed",
/** The lock was held but the read-modify-write itself failed. */
writeFailed: "write-failed",
} as const);
export type LineageGapReason = (typeof LINEAGE_GAP_REASON)[keyof typeof LINEAGE_GAP_REASON];
export const LINEAGE_RECORD_STATUS = Object.freeze({
appended: "appended",
dropped: "dropped",
} as const);
export type LineageRecordStatus =
(typeof LINEAGE_RECORD_STATUS)[keyof typeof LINEAGE_RECORD_STATUS];
/**
* Outcome of one {@link recordLineageObservation}. The function used to
* return `void`, so a swallowed failure and a successful append were
* indistinguishable to its caller; they no longer are.
*/
export interface LineageRecordResult {
readonly status: LineageRecordStatus;
/** Sequence number allocated to the appended line. */
readonly seq?: number;
/** True when this write also compacted the file. */
readonly compacted?: boolean;
/** Present only on `dropped`: the named gap. */
readonly notice?: DegradationNotice;
}
/** One recorded gap: an observation that was never appended. */
export interface LineageGapRecord {
readonly sessionId: string;
/** The OBSERVATION's timestamp, as the caller reported it. */
readonly at: string;
/**
* When the gap itself was written, on the recording process's clock.
*
* Distinct from {@link at} on purpose: `at` is host-supplied and can
* name any instant (an imported transcript backfills months-old
* events), so aging on it would expire a gap recorded seconds ago.
* Empty only for a record written before this field existed, which
* then ages on `at` as the best available evidence.
*/
readonly recordedAt: string;
readonly event: string;
readonly reason: string;
}
export interface LineageObservation {
readonly sessionId: string;
/** ISO-8601 timestamp of the observation (host clock, injected). */
readonly at: string;
readonly cwd?: string;
readonly event: string;
/** True when this event evidences a compression boundary. */
readonly compressionEvidence?: boolean;
/** Resolved lineage to persist alongside the observation. */
readonly lineage?: SessionLineage;
/**
* Git working state the observation was made in
* (context-integrity-gates, Unit C). Absent when the session's `cwd`
* is not inside a repository or the bounded probe found nothing; the
* remote is already canonicalized, so no credential reaches this
* record.
*/
readonly workspace?: GitWorkspaceIdentity;
/**
* DECLARED work identity for this observation (signals-that-survive,
* Unit 9). Never derived - see `identity.ts` for the three rungs it is
* sourced from. Absent when nothing declared one, which is why every
* pre-existing line is byte-identical.
*/
readonly workId?: string;
/** DECLARED lane. A hard separator between work items, never a tiebreak. */
readonly laneId?: string;
}
export interface LineageLedgerEntry {
readonly sessionId: string;
readonly firstSeenMs: number;
readonly lastSeenMs: number;
readonly cwd?: string;
readonly lastEvent: string;
/** Evidence flag of the LATEST observation (not sticky). */
readonly compressionEvidence: boolean;
/** Last persisted lineage for the session, if any. */
readonly lineage?: SessionLineage;
/**
* Last attested git working state for the session, if any. Carried
* forward from the newest observation that reported one, exactly as
* `cwd` is - a probe that came back empty on one event does not erase
* what the session already attested.
*/
readonly workspace?: GitWorkspaceIdentity;
/**
* Last declared work identity for the session, carried forward from
* the newest observation that declared one - for the same reason
* `cwd` and `workspace` are: an event that arrived without the
* declaration does not retract it.
*/
readonly workId?: string;
/** Last declared lane for the session. See {@link LineageObservation.laneId}. */
readonly laneId?: string;
}
export type LineageLedgerState = ReadonlyMap<string, LineageLedgerEntry>;
/**
* The Brain state directory: the dot-directory this module's ledger, its
* gap sidecar and the work-identity markers all live under. Exported so
* a sibling state file resolves through one definition rather than
* re-deriving the relative path.
*/
export function brainStateDirPath(vault: string): string {
return ensureInsideVault(join(vault, STATE_DIR_REL), vault);
}
/**
* This device's ledger shard: `session-lineage[.<deviceId>].jsonl`.
*
* One chain and one sequence per FILE, so a per-device shard is a
* per-device chain (who-wrote-what, Task B) and two machines syncing one
* vault never rewrite each other's compaction. The empty shard id is the
* legacy un-sharded name, still merged by every read and never renamed.
*/
export function sessionLineageLedgerPath(
vault: string,
shardId: string = resolveAppendShardId(),
): string {
return ensureInsideVault(
join(vault, STATE_DIR_REL, shardedFileName(LEDGER_BASE, shardId, LINEAGE_EXT)),
vault,
);
}
/**
* Sidecar holding observations that were never appended. Lives beside
* the ledger under the same dot-directory, which is excluded from vault
* walking and search indexing - a diagnostic must not become a note.
*
* Shards with the ledger: each device bounds and ages its OWN sidecar,
* carrying its own discard accumulator, and the report sums them. The
* sidecar's trim lock therefore never crosses a device boundary either.
*/
export function sessionLineageGapsPath(
vault: string,
shardId: string = resolveAppendShardId(),
): string {
// Machine state: the vault on an unswitched machine, the device-local
// state once switched, so a store-only machine records its drops where
// its own reads find them and writes nothing under `Brain/`.
return machineStatePath(vault, STATE_DIR_REL, shardedFileName(GAPS_BASE, shardId, LINEAGE_EXT));
}
/** One shard of the ledger (or of its gap sidecar) as found on disk. */
export interface LineageShardFile {
/** Empty string for the legacy un-sharded file. */
readonly shardId: string;
readonly path: string;
}
/**
* Every ledger shard under `Brain/.state/`, ordered by shard id so the
* legacy file leads and two devices agree on the sequence whatever order
* Syncthing delivered their files in. Fail-soft: a state directory that
* cannot be listed yields no shards, matching every other read here.
*/
export function listSessionLineageShards(vault: string): LineageShardFile[] {
return listLineageShards(vault, LEDGER_BASE);
}
/** The gap sidecar's shards. See {@link listSessionLineageShards}. */
export function listSessionLineageGapShards(vault: string): LineageShardFile[] {
return listLineageShards(vault, GAPS_BASE);
}
function listLineageShards(vault: string, base: string): LineageShardFile[] {
let dirs: string[];
try {
// The gap sidecar is machine state (see `sessionLineageGapsPath`).
// Once switched it lives in the device-local state, and the shards
// recorded under the vault before the switch are read too until the
// legacy tree is removed: the migration does not move them, and a gap
// dropped from the read would pass a session's history as complete.
const legacy = brainStateDirPath(vault);
dirs =
base === GAPS_BASE
? [...new Set([machineStatePath(vault, STATE_DIR_REL), legacy])]
: [legacy];
} catch {
return [];
}
const shards: LineageShardFile[] = [];
for (const dir of dirs) {
try {
for (const file of listShardedFiles(dir, {
base: literalBase(base),
extensions: [LINEAGE_EXT],
})) {
shards.push({ shardId: file.shardId, path: file.path });
}
} catch {
// A state directory that cannot be listed yields no shards.
}
}
return shards.toSorted((a, b) =>
a.shardId < b.shardId ? -1 : a.shardId > b.shardId ? 1 : a.path < b.path ? -1 : 1,
);
}
export interface LedgerLine {
readonly sid: string;
readonly at: string;
readonly cwd?: string;
readonly event: string;
readonly ce?: boolean;
readonly parent?: string | null;
readonly root?: string;
readonly depth?: number;
readonly src?: SessionLineage["source"];
/** Canonical remote identity (Unit C). Never carries a credential. */
readonly repo?: string;
readonly branch?: string;
readonly commit?: string;
/** Declared work id (Unit 9). Absent unless a rung declared one. */
readonly wid?: string;
/** Declared lane id (Unit 9). Absent unless a rung declared one. */
readonly lane?: string;
/** Monotonic sequence number (Unit D). Absent on pre-chain lines. */
readonly seq?: number;
/** Chain hash of the preceding line; `null` at a chain anchor. */
readonly prev?: string | null;
/** This line's chain hash. Absent on pre-chain lines. */
readonly h?: string;
}
/** A parsed line together with the exact text it came from. */
export interface LineageLedgerLine {
/** Verbatim source text - what compaction retains, unchanged. */
readonly raw: string;
readonly line: LedgerLine;
}
/** The chain-carrying fields, excluded from the body that is hashed. */
const CHAIN_FIELDS = ["seq", "prev", "h"] as const;
/**
* The hashed body of a line: everything EXCEPT the chain fields. Built
* by removal rather than by an allow-list, so a field a future writer
* adds is covered by the hash automatically - the same reason
* compaction retains raw text instead of re-serializing.
*/
function lineBody(line: LedgerLine): Record<string, unknown> {
const body: Record<string, unknown> = { ...(line as unknown as Record<string, unknown>) };
for (const field of CHAIN_FIELDS) delete body[field];
return body;
}
function chainHash(prev: string | null, seq: number, body: Record<string, unknown>): string {
return computePayloadHash({ prev, seq, body });
}
/**
* Recompute the chain hash a stored line SHOULD carry. A line whose
* recorded `h` differs from this was edited after it was written.
*/
export function lineageChainHash(line: LedgerLine): string {
return chainHash(line.prev ?? null, typeof line.seq === "number" ? line.seq : 0, lineBody(line));
}
/** True when a line carries a usable chain link. */
export function isChainedLine(line: LedgerLine): boolean {
return (
typeof line.h === "string" &&
line.h.length > 0 &&
typeof line.seq === "number" &&
Number.isInteger(line.seq) &&
line.seq > 0
);
}
/**
* Project a workspace identity onto its line fields, omitting the ones
* git could not attest. A key that is absent means "not attested" - the
* distinction the fail-closed match predicate turns on.
*/
function workspaceFields(
workspace: GitWorkspaceIdentity | undefined,
): Pick<LedgerLine, "repo" | "branch" | "commit"> {
if (workspace === undefined) return {};
return {
...(workspace.repo !== null ? { repo: workspace.repo } : {}),
...(workspace.branch !== null ? { branch: workspace.branch } : {}),
...(workspace.commit !== null ? { commit: workspace.commit } : {}),
};
}
/**
* Project a declared work identity onto its line fields (Unit 9). A
* blank declaration is not a declaration: it is omitted, so a host that
* sends an empty string leaves the line byte-identical to one that sent
* nothing at all.
*/
function identityFields(
workId: string | undefined,
laneId: string | undefined,
): Pick<LedgerLine, "wid" | "lane"> {
const wid = declared(workId);
const lane = declared(laneId);
return {
...(wid !== undefined ? { wid } : {}),
...(lane !== undefined ? { lane } : {}),
};
}
/** A trimmed non-empty declaration, or `undefined`. */
function declared(value: string | undefined): string | undefined {
if (typeof value !== "string") return undefined;
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : undefined;
}
/**
* Inverse of {@link workspaceFields}; `undefined` when nothing was attested.
*
* The recorded repo is re-canonicalized on the way in. The canonical form
* is a remote in its own right and canonicalizes to itself, so a line
* written by this version reads back unchanged, while one written before
* a canonicalization rule was added (the drive letter's case) reads in
* today's form and still matches the same checkout. A value that no
* longer canonicalizes is kept verbatim: it attests something, and
* dropping it would turn a disagreement into an absence.
*/
function readWorkspace(line: LedgerLine): GitWorkspaceIdentity | undefined {
const repo =
typeof line.repo === "string" ? (canonicalizeGitRemote(line.repo) ?? line.repo) : null;
const branch = typeof line.branch === "string" ? line.branch : null;
const commit = typeof line.commit === "string" ? line.commit : null;
if (repo === null && branch === null && commit === null) return undefined;
return Object.freeze({ repo, branch, commit });
}
function toLine(obs: LineageObservation): LedgerLine {
return {
sid: obs.sessionId,
at: obs.at,
...(obs.cwd !== undefined ? { cwd: obs.cwd } : {}),
event: obs.event,
...(obs.compressionEvidence === true ? { ce: true } : {}),
...(obs.lineage !== undefined
? {
parent: obs.lineage.parentId,
root: obs.lineage.rootId,
depth: obs.lineage.depth,
src: obs.lineage.source,
}
: {}),
...workspaceFields(obs.workspace),
...identityFields(obs.workId, obs.laneId),
};
}
/** What one parse of the ledger text found, including what it could not use. */
interface ParsedLedger {
readonly lines: LineageLedgerLine[];
/**
* 1-based file positions of the non-blank lines that are not a usable
* ledger record. Recorded rather than discarded: dropping them
* silently is what let a wholly destroyed ledger verify as clean, and
* the position is what makes one addressable.
*/
readonly skippedLineNumbers: number[];
}
/**
* Parse the file into lines, KEEPING the source text of each. The raw
* text is what compaction writes back, which is how a field this
* version does not know about survives a rewrite.
*/
function readLedgerLines(raw: string): ParsedLedger {
const lines: LineageLedgerLine[] = [];
const skippedLineNumbers: number[] = [];
const source = raw.split("\n");
for (let index = 0; index < source.length; index++) {
const trimmed = source[index]!.trim();
if (trimmed.length === 0) continue;
const lineNumber = index + 1;
try {
const parsed = JSON.parse(trimmed) as unknown;
if (parsed === null || typeof parsed !== "object") {
skippedLineNumbers.push(lineNumber);
continue;
}
const candidate = parsed as LedgerLine;
if (typeof candidate.sid !== "string" || typeof candidate.at !== "string") {
skippedLineNumbers.push(lineNumber);
continue;
}
lines.push({ raw: trimmed, line: candidate });
} catch {
// Fail-soft for the READ path, recorded for the VERIFIER: a corrupt
// line never poisons resolution, and never passes unreported.
skippedLineNumbers.push(lineNumber);
}
}
return { lines, skippedLineNumbers };
}
/**
* One read of the ledger file, with the three states the old
* `LineageLedgerLine[]` return collapsed into an empty array: the file
* is absent, the file is present but could not be read, or the file was
* read and some of it was not usable.
*/
export interface LineageLedgerScan {
/** The ledger file exists on disk. */
readonly exists: boolean;
/** The file's bytes were obtained. False for an absent or unreadable file. */
readonly readable: boolean;
/** Usable records, in file order. */
readonly lines: ReadonlyArray<LineageLedgerLine>;
/** 1-based positions of the non-blank lines that could not be read. */
readonly skippedLineNumbers: ReadonlyArray<number>;
}
/**
* Read the ledger as parsed lines in file order, reporting what it could
* not use. Fail-soft like every other read here - it never throws, and
* an unreadable file yields no lines - but the caller can now tell an
* absent ledger from a destroyed one. Exposed for `verify.ts`, which
* needs line order and the chain fields that per-session state folds
* away.
*/
export function scanLineageLedger(
vault: string,
shardId: string = resolveAppendShardId(),
): LineageLedgerScan {
let path: string;
try {
path = sessionLineageLedgerPath(vault, shardId);
} catch {
return absentScan();
}
return scanLineageShard(path);
}
function absentScan(): LineageLedgerScan {
return Object.freeze({
exists: false,
readable: false,
lines: Object.freeze([]),
skippedLineNumbers: Object.freeze([]),
});
}
function scanLineageShard(path: string): LineageLedgerScan {
if (!existsSync(path)) return absentScan();
let raw: string;
try {
raw = readFileSync(path, "utf8");
} catch {
return Object.freeze({
exists: true,
readable: false,
lines: Object.freeze([]),
skippedLineNumbers: Object.freeze([]),
});
}
const parsed = readLedgerLines(raw);
return Object.freeze({
exists: true,
readable: true,
lines: Object.freeze(parsed.lines),
skippedLineNumbers: Object.freeze(parsed.skippedLineNumbers),
});
}
/** One shard's scan, together with which shard it was. */
export interface LineageLedgerShardScan extends LineageLedgerScan, LineageShardFile {}
/**
* Scan EVERY ledger shard, in shard-id order. The unit of the chain is
* one file, so the verifier walks these one at a time and reports each
* by its own path; a break in one device's shard says nothing about
* another's.
*/
export function scanLineageLedgerShards(vault: string): LineageLedgerShardScan[] {
return listSessionLineageShards(vault).map((shard) => ({
...shard,
...scanLineageShard(shard.path),
}));
}
/** The sidecar as a whole: what it still lists, and what it no longer can. */
export interface LineageGapReport {
/** The sidecar file exists on disk. */
readonly exists: boolean;
/** Gap records still within the retention window, oldest first. */
readonly records: ReadonlyArray<LineageGapRecord>;
/**
* Records the sidecar's bound discarded. They are COUNTED, never
* listed, so a capped listing is never mistaken for a complete one.
*/
readonly discarded: number;
/** True when `discarded > 0`: {@link records} is a suffix, not the whole. */
readonly truncated: boolean;
/** Dropped observations still on the record: listed plus discarded. */
readonly total: number;
}
export interface ReadLineageGapsOptions {
/** Injected clock (epoch ms) for the retention window. */
readonly nowMs?: number;
}
/** One line of the sidecar: either a gap record or the truncation marker. */
interface ParsedGapFile {
readonly records: LineageGapRecord[];
/** Records earlier trims discarded, from the accumulator line. */
readonly discarded: number;
/** When the accumulator was last written; `null` when there is none. */
readonly discardedAt: string | null;
/** Records this parse dropped because they fell outside retention. */
readonly aged: number;
}
/** Key that distinguishes the accumulator line from a gap record. */
const GAP_DISCARDED_KEY = "discarded";
/** Key carrying {@link LineageGapRecord.recordedAt} on the wire. */
const GAP_RECORDED_AT_KEY = "rat";
function parseGapFile(raw: string, nowMs: number): ParsedGapFile {
const records: LineageGapRecord[] = [];
let discarded = 0;
let discardedAt: string | null = null;
let aged = 0;
for (const line of raw.split("\n")) {
const trimmed = line.trim();
if (trimmed.length === 0) continue;
let parsed: Record<string, unknown>;
try {
parsed = JSON.parse(trimmed) as Record<string, unknown>;
} catch {
// A corrupt gap line is itself a gap we cannot name; skip it.
continue;
}
const count = parsed[GAP_DISCARDED_KEY];
if (typeof count === "number" && Number.isFinite(count)) {
const at = typeof parsed["at"] === "string" ? parsed["at"] : "";
if (!isExpired(at, nowMs)) {
discarded += Math.max(0, Math.trunc(count));
discardedAt = at;
}
continue;
}
if (typeof parsed["sid"] !== "string") continue;
const at = typeof parsed["at"] === "string" ? parsed["at"] : "";
const recordedAt = typeof parsed[GAP_RECORDED_AT_KEY] === "string" ? parsed["rat"] : "";
if (isExpired(recordedAt.length > 0 ? recordedAt : at, nowMs)) {
aged++;
continue;
}
records.push({
sessionId: parsed["sid"],
at,
recordedAt,
event: typeof parsed["event"] === "string" ? parsed["event"] : "unknown",
reason: typeof parsed["reason"] === "string" ? parsed["reason"] : "unknown",
});
}
return { records, discarded, discardedAt, aged };
}
/**
* The on-disk form of one gap record. Written by the append AND by the
* trim's rewrite, so a trimmed sidecar is byte-comparable with a
* freshly appended one and no field is lost in the round trip.
*/
function gapLine(record: LineageGapRecord): Record<string, unknown> {
return {
sid: record.sessionId,
at: record.at,
...(record.recordedAt.length > 0 ? { [GAP_RECORDED_AT_KEY]: record.recordedAt } : {}),
event: record.event,
reason: record.reason,
};
}
/**
* True when a recorded instant is older than the retention window. An
* undatable record never expires: aging out something whose age is
* unknown would be a guess, and this module reports rather than guesses.
*/
function isExpired(at: string, nowMs: number): boolean {
const ms = Date.parse(at);
if (!Number.isFinite(ms)) return false;
return nowMs - ms > GAP_RETENTION_MS;
}
/**
* Read the sidecar. Fail-soft; a missing sidecar reads as none.
* Records older than {@link GAP_RETENTION_MS} are omitted here as well as
* pruned on the next trim, so a vault that has stopped dropping
* observations stops reporting them even if nothing writes again.
*/
export function readLineageGapReport(
vault: string,
opts: ReadLineageGapsOptions = {},
): LineageGapReport {
const nowMs = opts.nowMs ?? Date.now();
const rows: Array<ShardedRow<LineageGapRecord>> = [];
let exists = false;
let discarded = 0;
// One sidecar per device, each with its own bound and its own discard
// accumulator; the report is their union and the sum of what each
// device's trim threw away.
for (const shard of listSessionLineageGapShards(vault)) {
let raw: string;
try {
raw = readFileSync(shard.path, "utf8");
} catch {
continue;
}
exists = true;
const parsed = parseGapFile(raw, nowMs);
discarded += parsed.discarded;
parsed.records.forEach((value, index) =>
rows.push({ value, shardId: shard.shardId, line: index }),
);
}
if (!exists) {
return Object.freeze({
exists: false,
records: Object.freeze([]),
discarded: 0,
truncated: false,
total: 0,
});
}
const records = mergeShardedRows(rows, (record) => record.at);
return Object.freeze({
exists: true,
records: Object.freeze(records),
discarded,
truncated: discarded > 0,
total: records.length + discarded,
});
}
/** The listed gap records. See {@link readLineageGapReport} for the rest. */
export function readLineageGaps(
vault: string,
opts: ReadLineageGapsOptions = {},
): ReadonlyArray<LineageGapRecord> {
return readLineageGapReport(vault, opts).records;
}
/**
* Session ids that spoke but whose observation never reached the ledger.
*
* This is the OTHER half of "does this session have history of its own".
* The crutch's Rule 1 reads the ledger for the session's own line and
* treats its absence as proof of a brand-new session; a writer-lock drop
* removes exactly that line, so without this set a session that DID
* speak looks new and can be stitched onto an unrelated parallel one.
* Fail-soft like every read here.
*/
export function lineageGapSessionIds(vault: string): ReadonlySet<string> {
const ids = new Set<string>();
for (const gap of readLineageGaps(vault)) ids.add(gap.sessionId);
return ids;
}
function buildState(lines: ReadonlyArray<LedgerLine>): Map<string, LineageLedgerEntry> {
const state = new Map<string, LineageLedgerEntry>();
for (const line of lines) {
const atMs = Date.parse(line.at);
if (!Number.isFinite(atMs)) continue;
const prev = state.get(line.sid);
const lineage: SessionLineage | undefined =
typeof line.root === "string" && line.src !== undefined
? Object.freeze({
rootId: line.root,
parentId: line.parent ?? null,
depth: typeof line.depth === "number" ? line.depth : 0,
source: line.src,
})
: prev?.lineage;
const workspace = readWorkspace(line) ?? prev?.workspace;
const workId = declared(line.wid) ?? prev?.workId;
const laneId = declared(line.lane) ?? prev?.laneId;
state.set(line.sid, {
sessionId: line.sid,
firstSeenMs: prev === undefined ? atMs : Math.min(prev.firstSeenMs, atMs),
lastSeenMs: prev === undefined ? atMs : Math.max(prev.lastSeenMs, atMs),
...(line.cwd !== undefined
? { cwd: line.cwd }
: prev?.cwd !== undefined
? { cwd: prev.cwd }
: {}),
lastEvent: line.event ?? prev?.lastEvent ?? "unknown",
compressionEvidence: line.ce === true,
...(lineage !== undefined ? { lineage } : {}),
...(workspace !== undefined ? { workspace } : {}),
...(workId !== undefined ? { workId } : {}),
...(laneId !== undefined ? { laneId } : {}),
});
}
return state;
}
/**
* Read the ledger into per-session state. Missing file reads empty.
*
* Deliberately does NOT verify the chain. The catch-to-empty-map below
* is load-bearing for `resolveRootId` in the anticipatory cache and for
* the lifecycle hook, both of which treat "no lineage" as a normal
* state and neither of which can handle a throw. Integrity findings
* belong to `verify.ts`, which reports them to the doctor.
*/
export function readLineageLedger(vault: string): LineageLedgerState {
// A store-only machine (design D14 item 1) folds the `lineage` ledger
// rows: by `at`, with fold order (the order every machine agrees on)
// breaking ties. Never throws, like the legacy read.
if (isStoreOnly({ vault })) {
let rows: unknown[];
try {
rows = readLedgerRows<unknown>(DUAL_WRITE_LEDGERS.lineage, () => [], { vault });
} catch {
rows = [];
}
const lines = rows
.filter(
(row): row is LedgerLine =>
row !== null &&
typeof row === "object" &&
typeof (row as LedgerLine).sid === "string" &&
typeof (row as LedgerLine).at === "string",
)
.toSorted((a, b) => (a.at < b.at ? -1 : a.at > b.at ? 1 : 0));
return buildState(lines);
}
const rows: Array<ShardedRow<LedgerLine>> = [];
for (const shard of scanLineageLedgerShards(vault)) {
shard.lines.forEach((entry, index) =>
rows.push({ value: entry.line, shardId: shard.shardId, line: index }),
);
}
// (at, shard id, line): every device folds the same observations in the
// same order, so per-session state does not depend on which machine is
// asking or on when Syncthing delivered the other machine's shard.
return buildState(mergeShardedRows(rows, (line) => line.at));
}
/**
* Append one observation under the writer lock, compacting the file
* (atomic rewrite keeping the most recent lines verbatim) once it
* crosses the line cap.
*
* Fail-soft by contract: nothing here throws into a lifecycle hook. But
* a failure is no longer SILENT - a dropped observation is written to
* the gap sidecar and returned as a named notice, so the absence is
* attributable instead of being indistinguishable from a session that
* simply never spoke.
*/
export function recordLineageObservation(
vault: string,
obs: LineageObservation,
): LineageRecordResult {
// A store-only machine (design D14 items 1 and 10) appends the
// observation as one `lineage` ledger row, with no per-line chain: the
// store's integrity is the record checksums and partition digests. The
// capture hook must never fail, so a store failure is returned as a
// named drop, recorded in the gap sidecar, which lives in the
// device-local state on this machine (nothing is written under `Brain/`).
if (isStoreOnly({ vault })) {
try {
// Lineage says which sessions ran where (working directories, repos,
// branches): it goes to `private` when this machine holds it.
storeWrite(
() => ledgerRowRecord(DUAL_WRITE_LEDGERS.lineage, toLine(obs), "private-when-held"),
{ vault },
);
} catch {
// Recorded in the gap sidecar (device-local once switched), which
// `lineageGapSessionIds` reads, so the lifecycle crutch knows this
// session's history is incomplete.
return recordGap(vault, obs, LINEAGE_GAP_REASON.writeFailed);
}
return { status: LINEAGE_RECORD_STATUS.appended };
}
// Vault-identity write guard (Unit J), AHEAD OF THE FIRST BYTE: the
// `mkdirSync` below materializes `Brain/.state/` under whatever root
// it is handed, so asserting after it would refuse a store this
// function had already created. A refusal is returned as a named drop,
// never thrown, and deliberately does NOT reach the gap sidecar -
// writing the diagnostic into the wrong vault is the very thing the
// guard just refused.
try {
assertVaultIdentityForWrite(vault);
} catch (err) {
return {
status: LINEAGE_RECORD_STATUS.dropped,
notice:
err instanceof VaultIdentityMismatchError
? err.notice
: degradationNotice({
code: DEGRADATION_CODE.vaultMarkerMismatch,
site: LEDGER_SITE,
detail:
`observation for session ${obs.sessionId} (event ${obs.event}) ` +
"was not appended: the vault write guard refused this root",
}),
};
}
let handle: LockHandle | undefined;
let appended: LedgerLine | undefined;
try {
const path = sessionLineageLedgerPath(vault);
mkdirSync(dirname(path), { recursive: true });
try {
handle = acquireLockSync(path);
} catch (err) {
// No retry, by the lock's design: a loud named gap beats a silent
// retry loop that still fails. The gap is exactly what the
// sequence numbers above would otherwise leave unexplained. The
// lock file is NAMED on the notice: a lock left behind by a
// SIGKILLed writer disables capture for the life of the vault, and
// an operator cannot remove a file the diagnostic never mentions.
const errno = err as NodeJS.ErrnoException;
return recordGap(
vault,
obs,
errno.code === "ELOCKED" ? LINEAGE_GAP_REASON.lockBusy : LINEAGE_GAP_REASON.lockFailed,
typeof errno.path === "string" ? errno.path : undefined,
);
}
// Re-read UNDER the lock, exactly as the idempotency ledger does:
// the chain head and the next sequence number must come from the
// file as it is now, not as it was before the lock was taken.
const existing = existsSync(path) ? readFileSync(path, "utf8") : "";
const entries = readLedgerLines(existing).lines;
const body = toLine(obs);
const { seq, prev } = nextChainPosition(entries);
const line: LedgerLine = {
...body,
seq,
prev,
h: chainHash(prev, seq, body as unknown as Record<string, unknown>),
};
const serialized = JSON.stringify(line);
if (entries.length + 1 > MAX_LEDGER_LINES) {
// Verbatim retention: raw source text, never a re-serialization.
// Retained lines keep their own sequence numbers and chain links,
// so the chain stays contiguous across the truncation.
const retained = entries.slice(-RETAIN_LEDGER_LINES).map((entry) => entry.raw);
retained.push(serialized);
atomicWriteFileSync(path, `${retained.join("\n")}\n`);
appended = line;
return { status: LINEAGE_RECORD_STATUS.appended, seq, compacted: true };
}
appendFileSync(path, `${serialized}\n`, "utf8");
appended = line;
return { status: LINEAGE_RECORD_STATUS.appended, seq };
} catch {
return recordGap(vault, obs, LINEAGE_GAP_REASON.writeFailed);
} finally {
handle?.release();
// Brain store dual write (task 1.11): the new line only (the store never
// compacts it away), off by default and never failing this append. It
// runs after the release: this lock is taken without a retry, so every
// millisecond the store's own lock and fsync held it would turn a
// concurrent observation into a lockBusy gap.
if (appended !== undefined) {
const row = appended;
// Lineage says which sessions ran where (working directories, repos,
// branches): it goes to `private` when this machine holds it.
dualWrite(() => ledgerRowRecord(DUAL_WRITE_LEDGERS.lineage, row, "private-when-held"), {
vault,
});
}
}
}
/**
* Sequence number and chain link for the next line. Sequence numbers
* continue from the last CHAINED line, so a compacted file keeps
* counting from where its retained head left off, and a pre-chain
* ledger starts a fresh chain at 1 with a null anchor.
*/
function nextChainPosition(entries: ReadonlyArray<LineageLedgerLine>): {
seq: number;
prev: string | null;
} {
for (let index = entries.length - 1; index >= 0; index--) {
const line = entries[index]!.line;
if (!isChainedLine(line)) continue;
return { seq: line.seq! + 1, prev: line.h! };
}
return { seq: 1, prev: null };
}
/**
* Record an observation that never reached the ledger. Best-effort and
* itself fail-soft: if even the sidecar cannot be written, the returned
* notice is the only trace, which is still strictly more than the
* nothing this path used to leave.
*/
function recordGap(
vault: string,
obs: LineageObservation,
reason: LineageGapReason,
lockPath?: string,
): LineageRecordResult {
const notice = degradationNotice({
code: DEGRADATION_CODE.lineageObservationDropped,
site: LEDGER_SITE,
...(lockPath !== undefined ? { path: lockPath } : {}),
detail:
`observation for session ${obs.sessionId} (event ${obs.event}) ` +
`was not appended: ${reason}` +
(lockPath !== undefined ? `; the lock is held at ${lockPath}` : ""),
});
try {
const path = sessionLineageGapsPath(vault);
mkdirSync(dirname(path), { recursive: true });
const nowMs = Date.now();
// O_APPEND: concurrent gap writers cannot interleave a short line.
appendFileSync(
path,
`${JSON.stringify(gapLine({ sessionId: obs.sessionId, at: obs.at, recordedAt: new Date(nowMs).toISOString(), event: obs.event, reason }))}\n`,
"utf8",
);
boundGapSidecar(path, nowMs);
} catch {
// The gap could not be persisted; the notice still names it.
}
return { status: LINEAGE_RECORD_STATUS.dropped, notice };
}
/**
* Keep the gap sidecar bounded, aged, and HONEST about the difference.
*
* Three properties, each of which was previously absent:
*
* - the bound always applies. The trim takes the sidecar's OWN lock
* (never the ledger's, so it cannot deadlock against the writer that
* called it) and reclaims that lock once it is older than
* {@link GAP_LOCK_STALE_MS} - otherwise one crash inside the trim
* window leaves a lock nothing ever removes and the bound stops
* existing for the life of the vault;
* - records past {@link GAP_RETENTION_MS} are pruned, so a burst of
* contention stops being reported instead of pinning the verifier's
* verdict to `false` permanently;
* - every record the bound removes is ADDED TO AN ACCUMULATOR, so 400
* drops are never presented as the 256 that happen to still be
* listed.
*/
function boundGapSidecar(path: string, nowMs: number): void {
let handle: LockHandle | undefined;
try {
const before = parseGapFile(readFileSync(path, "utf8"), nowMs);
if (before.records.length <= MAX_GAP_LINES && before.aged === 0) return;
handle = acquireGapLock(path, nowMs);
// Re-read under the lock: another writer may have appended, and the
// count that decides the trim must be the current one.
const current = parseGapFile(readFileSync(path, "utf8"), nowMs);
const kept = current.records.slice(-MAX_GAP_LINES);
const discarded = current.discarded + (current.records.length - kept.length);
const lines: string[] = [];
if (discarded > 0) {
lines.push(
JSON.stringify({ [GAP_DISCARDED_KEY]: discarded, at: new Date(nowMs).toISOString() }),
);
}
for (const record of kept) lines.push(JSON.stringify(gapLine(record)));
atomicWriteFileSync(path, lines.length === 0 ? "" : `${lines.join("\n")}\n`);
} catch {
// Best-effort: the sidecar is diagnostics, and a failed trim must
// never propagate into the capture path that recorded the gap.
} finally {
handle?.release();
}
}
/**
* Take the sidecar's trim lock, reclaiming it once when it is provably
* abandoned. Scoped to this one lock and this one idempotent operation -
* see {@link GAP_LOCK_STALE_MS} for why the ledger's writer lock gets no
* such treatment.
*/
function acquireGapLock(path: string, nowMs: number): LockHandle {
try {
return acquireLockSync(path);
} catch (err) {
const errno = err as NodeJS.ErrnoException;
if (errno.code !== "ELOCKED") throw err;
const lockPath = typeof errno.path === "string" ? errno.path : `${path}.lock`;
// Removed only by the one contender that claims this abandoned file, so
// two trims that both judged it stale cannot both break it.
const abandoned = (st: { readonly mtimeMs: bigint }): boolean =>
nowMs - Number(st.mtimeMs) > GAP_LOCK_STALE_MS;
if (!removeStaleLockSync(lockPath, abandoned)) throw err;
// Exactly one retry. A second ELOCKED means a live writer took the
// lock in between, which is contention rather than abandonment.
return acquireLockSync(path);
}
}