Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

31.0 KB
Raw Blame
/**
 * CoS run event ledger — machine-local, append-only, bounded.
 *
 * Storage class `ephemeral-file` (see `docs/STORAGE.md`): the durable record of
 * a run is still `data/runs/{id}/metadata.json`; this file is the ordered trace
 * of how that record got where it is. It is **intentionally machine-local and
 * never federated** — every line describes a process on this box, so a peer's
 * copy would describe nothing that exists there. No sync cursor, no tombstone,
 * no `PORTOS_SCHEMA_VERSIONS` entry.
 *
 * On disk:
 *   data/cos/run-events.jsonl    — active generation
 *   data/cos/run-events.1.jsonl  — the one retained archive generation
 *   data/cos/mind-events.jsonl   — persistent-mind active generation
 *   data/cos/mind-events.1.jsonl — persistent-mind archive generation
 *
 * Retention is TWO bounds, and both are load-bearing.
 *
 * The primary bound is a count, not a byte size, because the useful question is
 * "how many transitions back can I look", and a count survives a payload-shape
 * change that a byte cap would not. Once the active file reaches
 * MAX_ACTIVE_EVENTS it becomes the archive and the previous archive is dropped,
 * so the ledger holds between MAX_ACTIVE_EVENTS and 2×MAX_ACTIVE_EVENTS events.
 *
 * Ordinary diagnostics additionally carry an AGE bound (MAX_EVENT_AGE_DAYS).
 * Persistent-mind events live in their own larger count-bounded pool: mind
 * chatter can neither evict ordinary diagnostics nor age out while stopped.
 *
 * Two invariants the callers depend on:
 *
 * - **Appends never reject.** These are called from agent lifecycle paths,
 *   several of them inside child-process exit handlers where a throw takes the
 *   Node process with it. Telemetry must never be able to fail a run, so the
 *   promise chain absorbs and logs instead of propagating.
 * - **Appends are serialized.** Every append goes through one module-level
 *   promise queue, so the file order matches the call order and the seen-id set
 *   can't be read between another append's check and its write.
 */

import { join } from 'path';
import { rename, unlink } from 'fs/promises';
import { EventEmitter } from 'events';
import {
  PATHS,
  atomicWrite,
  appendJSONLine,
  readJSONFileStrict,
  readJSONLFile,
  writeJSONLines,
  pathExists
} from '../lib/fileUtils.js';
import {
  buildRunEvent,
  isStoredRunEvent,
  projectRunStates,
  RUN_EVENT_KINDS,
  RUN_EVENT_READ_LIMITS
} from '../lib/agentRunEvents.js';
import {
  PERSISTENT_MIND_ID,
  PERSISTENT_MIND_TRAJECTORY_LIMITS,
  isPersistentMindEventKind,
  parsePersistentMindCursor,
  persistentMindEventCursor,
  projectPersistentMind,
} from '../lib/persistentMindTrajectory.js';

const ACTIVE_PATH = join(PATHS.cos, 'run-events.jsonl');
const ARCHIVE_PATH = join(PATHS.cos, 'run-events.1.jsonl');
const MIND_ACTIVE_PATH = join(PATHS.cos, 'mind-events.jsonl');
const MIND_ARCHIVE_PATH = join(PATHS.cos, 'mind-events.1.jsonl');
const MIND_SEQUENCE_PATH = join(PATHS.cos, 'persistent-mind-sequences.json');
const MIND_SEQUENCE_SCHEMA_VERSION = 1;

// Live delivery is only a hint that a cursor-aware client should backfill from
// the durable ledger. The event itself is already redacted at construction.
export const runEventLogEvents = new EventEmitter();

/**
 * Events per generation. 5000 covers weeks of a busy install's lifecycle
 * boundaries at a few hundred bytes per line — a couple of MB per generation.
 */
export const MAX_ACTIVE_EVENTS = 5000;
export const MAX_ACTIVE_MIND_EVENTS = 10_000;

/**
 * Age bound, on TOP of the count bound.
 *
 * The two answer different failure modes and neither subsumes the other. The
 * count alone lets a quiet install keep a trace of a run from last spring —
 * stale diagnostics that describe a workspace, a provider, and a code path that
 * no longer exist. A busy install has the opposite problem the count already
 * solves. So: keep at most 2×MAX_ACTIVE_EVENTS events, AND nothing older than
 * this, whichever bites first.
 *
 * 30 days is well past the window in which an ordinary run failure is still
 * worth a post-mortem. Persistent-mind events instead use the count bound so
 * their recent unsummarized window survives a long user-initiated stop.
 */
export const MAX_EVENT_AGE_DAYS = 30;
const MAX_EVENT_AGE_MS = MAX_EVENT_AGE_DAYS * 24 * 60 * 60 * 1000;

/**
 * How often the on-disk prune actually runs. Expiry is enforced on every READ
 * by filtering (so a reader never sees an expired event even between prunes);
 * this interval only bounds how often the files are rewritten to reclaim the
 * space, because rewriting a generation on every append would turn an O(1)
 * append into an O(n) one.
 */
const PRUNE_INTERVAL_MS = 60 * 60 * 1000;

// Default page size for the read API, and the ceiling a caller can request.
// Defined in the pure module so the route's Zod schema can share them without a
// lib → services import (see RUN_EVENT_READ_LIMITS).
export const DEFAULT_READ_LIMIT = RUN_EVENT_READ_LIMITS.default;
export const MAX_READ_LIMIT = RUN_EVENT_READ_LIMITS.max;

// Lazily hydrated on first append/read: Maps of eventId → retention metadata.
// `null` (not an empty Map) is the "never loaded" sentinel, so a genuinely
// empty ledger caches as empty instead of re-reading both files on every single
// append. The timestamp is carried so the duplicate check can apply the SAME age
// cutoff the read path applies — otherwise, in the window between prunes, a
// redelivery could be rejected as a duplicate of an event no reader can see.
let activeIds = null;
let archiveIds = null;
let activeCount = 0;
let mindActiveIds = null;
let mindArchiveIds = null;
let mindActiveCount = 0;
// Highest sequence observed per mind. The timestamp-based floor keeps a new
// event above a fully-retained-away generation after restart; the retained max
// keeps strict monotonic order when the wall clock moves backwards.
let mindSequenceHighWater = null;
let mindSequenceCheckpointError = null;
// `null` = never pruned in this process, so the first append/read always
// prunes; a timestamp means "pruned then, skip until the interval elapses".
let lastPrunedAt = null;

// Serializes appends; also the handle callers await.
let appendQueue = Promise.resolve();
// A read can be the first caller after a restart, while an append arrives in
// the same event-loop turn. Keep all first-load callers on one snapshot so a
// slower read cannot overwrite the append path's in-memory sequence/index
// state with an older view of disk.
let hydrationPromise = null;

/** eventId → retention metadata, for the duplicate check. */
function indexById(events, { persistentMind = false } = {}) {
  return new Map(events.filter((e) => e?.eventId).map((e) => [e.eventId, {
    at: e.at ?? null,
    persistentMind: persistentMind || isPersistentMindEventKind(e.kind),
  }]));
}

const isMindShapedEvent = (event) => isStoredRunEvent(event)
  && typeof event.mindId === 'string'
  && Number.isSafeInteger(event.sequence);

function normalizeMindGenerations(events) {
  const seen = new Set();
  const previousByMind = new Map();
  const ordered = events
    .filter(isMindShapedEvent)
    .filter((event) => !seen.has(event.eventId) && seen.add(event.eventId))
    .sort((a, b) => a.sequence - b.sequence || String(a.eventId).localeCompare(String(b.eventId)))
    .map((event) => {
      const previousSequence = previousByMind.get(event.mindId) ?? null;
      previousByMind.set(event.mindId, event.sequence);
      return Object.hasOwn(event.data, 'previousSequence')
        ? event
        : { ...event, data: { ...event.data, previousSequence } };
    });
  const active = ordered.slice(-MAX_ACTIVE_MIND_EVENTS);
  const archive = ordered.slice(
    Math.max(0, ordered.length - (MAX_ACTIVE_MIND_EVENTS * 2)),
    Math.max(0, ordered.length - MAX_ACTIVE_MIND_EVENTS)
  );
  return { archive, active };
}

const relinkMindPredecessors = (events) => {
  const previousByMind = new Map();
  return [...events]
    .sort((a, b) => a.sequence - b.sequence || String(a.eventId).localeCompare(String(b.eventId)))
    .map((event) => {
      const previousSequence = previousByMind.get(event.mindId) ?? null;
      previousByMind.set(event.mindId, event.sequence);
      return { ...event, data: { ...event.data, previousSequence } };
    });
};

/**
 * Have we already stored this event, in a copy a reader can still see?
 *
 * An id whose only copy has aged past the cutoff does NOT count: the read path
 * filters that event out, so treating a redelivery as a duplicate would leave
 * the ledger with an event nothing can read and no way to re-observe it. Same
 * reasoning as rotation — a duplicate we can no longer see is no longer a
 * duplicate — applied continuously rather than only at the moment of a prune.
 */
function isStoredDuplicate(eventId, cutoff) {
  const metadata = activeIds.has(eventId) ? activeIds.get(eventId)
    : archiveIds.has(eventId) ? archiveIds.get(eventId)
      : mindActiveIds.has(eventId) ? mindActiveIds.get(eventId)
        : mindArchiveIds.has(eventId) ? mindArchiveIds.get(eventId)
      : undefined;
  return metadata !== undefined
    && (metadata.persistentMind || (typeof metadata.at === 'string' && metadata.at > cutoff));
}

async function hydrate() {
  if (activeIds) return;
  if (hydrationPromise) return hydrationPromise;
  const pending = (async () => {
    let [active, archive, mindActive, mindArchive, sequenceStore] = await Promise.all([
      readJSONLFile(ACTIVE_PATH),
      readJSONLFile(ARCHIVE_PATH),
      readJSONLFile(MIND_ACTIVE_PATH),
      readJSONLFile(MIND_ARCHIVE_PATH),
      readJSONFileStrict(MIND_SEQUENCE_PATH, { schemaVersion: MIND_SEQUENCE_SCHEMA_VERSION, minds: {} }),
    ]);
    const legacyArchive = archive.filter((event) => isPersistentMindEventKind(event?.kind));
    const legacyActive = active.filter((event) => isPersistentMindEventKind(event?.kind));
    if (legacyArchive.length > 0 || legacyActive.length > 0) {
      const recovered = normalizeMindGenerations([
        ...mindArchive,
        ...mindActive,
        ...legacyArchive,
        ...legacyActive,
      ]);
      // Destination-first makes recovery retry-safe if a later source rewrite
      // fails. This path handles upgrade -> rollback -> upgrade installs after
      // migration 301 is already marked applied.
      await writeJSONLines(MIND_ARCHIVE_PATH, recovered.archive);
      await writeJSONLines(MIND_ACTIVE_PATH, recovered.active);
      archive = archive.filter((event) => !isPersistentMindEventKind(event?.kind));
      active = active.filter((event) => !isPersistentMindEventKind(event?.kind));
      await writeJSONLines(ARCHIVE_PATH, archive);
      await writeJSONLines(ACTIVE_PATH, active);
      mindArchive = recovered.archive;
      mindActive = recovered.active;
      console.log(`🧠 Re-homed ${legacyArchive.length + legacyActive.length} rollback-era persistent mind event(s)`);
    }
    const invalidSequenceStore = !sequenceStore.ok || !sequenceStore.value
        || typeof sequenceStore.value !== 'object' || Array.isArray(sequenceStore.value)
        || sequenceStore.value.schemaVersion !== MIND_SEQUENCE_SCHEMA_VERSION
        || !sequenceStore.value.minds || typeof sequenceStore.value.minds !== 'object'
        || Array.isArray(sequenceStore.value.minds)
        || Object.entries(sequenceStore.value.minds).some(([mindId, sequence]) => (
          !mindId || mindId.length > 128 || !Number.isSafeInteger(sequence) || sequence < 0
        ));
    const validSequenceStore = !invalidSequenceStore;
    activeIds = indexById(active);
    archiveIds = indexById(archive);
    mindActiveIds = indexById(mindActive, { persistentMind: true });
    mindArchiveIds = indexById(mindArchive, { persistentMind: true });
    activeCount = active.length;
    mindActiveCount = mindActive.length;
    mindSequenceCheckpointError = validSequenceStore
      ? null
      : 'Persistent mind sequence checkpoint is unreadable or invalid';
    mindSequenceHighWater = new Map(validSequenceStore ? Object.entries(sequenceStore.value.minds) : []);
    for (const event of [...mindArchive, ...mindActive]) {
      if (!event?.mindId || !Number.isSafeInteger(event.sequence)) continue;
      mindSequenceHighWater.set(
        event.mindId,
        Math.max(mindSequenceHighWater.get(event.mindId) ?? -1, event.sequence)
      );
    }
  })();
  hydrationPromise = pending;
  try {
    await pending;
  } finally {
    if (hydrationPromise === pending) hydrationPromise = null;
  }
}

function nextMindSequence(mindId) {
  const previous = mindSequenceHighWater.get(mindId) ?? -1;
  return Math.max(previous + 1, Date.now() * 1000);
}

const saveMindSequenceHighWater = () => atomicWrite(MIND_SEQUENCE_PATH, {
  schemaVersion: MIND_SEQUENCE_SCHEMA_VERSION,
  minds: Object.fromEntries(mindSequenceHighWater),
});

/**
 * Rotate when the active generation is full: active becomes the archive, the
 * previous archive is dropped. The seen-id sets move with the files, so an
 * event whose only copy just aged out of the ledger is appendable again — which
 * is correct: a duplicate we can no longer see is no longer a duplicate.
 */
async function rotateIfFull(mindEvent) {
  const count = mindEvent ? mindActiveCount : activeCount;
  const limit = mindEvent ? MAX_ACTIVE_MIND_EVENTS : MAX_ACTIVE_EVENTS;
  if (count < limit) return;
  const activePath = mindEvent ? MIND_ACTIVE_PATH : ACTIVE_PATH;
  const archivePath = mindEvent ? MIND_ARCHIVE_PATH : ARCHIVE_PATH;
  if (await pathExists(archivePath)) await unlink(archivePath);
  await rename(activePath, archivePath);
  if (mindEvent) {
    mindArchiveIds = mindActiveIds;
    mindActiveIds = new Map();
    mindActiveCount = 0;
  } else {
    archiveIds = activeIds;
    activeIds = new Map();
    activeCount = 0;
  }
  console.log(`🔁 Rotated ${mindEvent ? 'persistent mind' : 'CoS run'} event ledger at ${limit} events`);
}

/**
 * The instant before which an event has aged out. Exported-ish only through the
 * behaviour it drives; `now` is a parameter so tests can age a ledger without
 * faking the clock.
 */
function expiryCutoff(now = Date.now()) {
  return new Date(now - MAX_EVENT_AGE_MS).toISOString();
}

/**
 * Would a reader see this line — structurally sound and either a mind event or
 * still inside the ordinary-run age window?
 *
 * One predicate, used by the read path, the stats, and the prune, so all three
 * agree by construction. A structurally invalid or undatable line counts as
 * NOT retained: the read path drops it either way, so counting it would make
 * the stats disagree with the endpoint they describe, and leaving it on disk
 * would let unreadable bytes accumulate forever under a retention policy that
 * can never date them. The prune is the only thing that ever cleans them up.
 */
const isRetained = (event, cutoff) => isStoredRunEvent(event)
  && (isPersistentMindEventKind(event.kind) || event.at > cutoff);

/**
 * Rewrite both generations without their expired events, and drop the expired
 * ids from the seen-id sets so a fresh redelivery of a long-gone event is
 * appendable again (same reasoning as rotation: a duplicate we can no longer
 * see is no longer a duplicate).
 *
 * Only runs inside the append queue or behind `flushRunEvents`, so it never
 * races an append. Throttled by PRUNE_INTERVAL_MS because it is O(ledger);
 * `force` is for the read path's first call and for tests.
 */
async function pruneExpired({ now = Date.now(), force = false } = {}) {
  if (!force && lastPrunedAt !== null && now - lastPrunedAt < PRUNE_INTERVAL_MS) return;
  lastPrunedAt = now;
  const cutoff = expiryCutoff(now);

  const [archive, active] = await Promise.all([
    readJSONLFile(ARCHIVE_PATH),
    readJSONLFile(ACTIVE_PATH)
  ]);

  const keptArchive = archive.filter((e) => isRetained(e, cutoff));
  const keptActive = active.filter((e) => isRetained(e, cutoff));
  const dropped = (archive.length - keptArchive.length) + (active.length - keptActive.length);
  if (dropped === 0) return;

  // An archive with nothing left is unlinked rather than rewritten empty, so a
  // long-idle install ends up with no archive file at all instead of a 0-byte one.
  if (keptArchive.length === 0) {
    if (await pathExists(ARCHIVE_PATH)) await unlink(ARCHIVE_PATH);
  } else if (keptArchive.length !== archive.length) {
    await writeJSONLines(ARCHIVE_PATH, keptArchive);
  }
  if (keptActive.length !== active.length) await writeJSONLines(ACTIVE_PATH, keptActive);

  archiveIds = indexById(keptArchive);
  activeIds = indexById(keptActive);
  activeCount = keptActive.length;
  console.log(`🧹 Pruned ${dropped} CoS run events older than ${MAX_EVENT_AGE_DAYS} days`);
}

async function appendNow(input) {
  await hydrate();
  const mindEvent = isPersistentMindEventKind(input?.kind);
  if (mindEvent && mindSequenceCheckpointError) throw new Error(mindSequenceCheckpointError);
  // A stable explicit id lets a retry short-circuit before consuming a new
  // sequence. Every supervisor boundary supplies one; derived ids remain
  // available for one-off callers and ordinary lifecycle events.
  if (mindEvent && input?.eventId && isStoredDuplicate(input.eventId, expiryCutoff())) {
    return { appended: false, duplicate: true, event: null };
  }
  const previousSequence = mindEvent ? (mindSequenceHighWater.get(input.mindId) ?? null) : null;
  const sequence = mindEvent ? nextMindSequence(input.mindId) : undefined;
  const event = buildRunEvent(mindEvent ? {
    ...input,
    sequence,
    data: { ...(input.data && typeof input.data === 'object' ? input.data : {}), previousSequence },
  } : input);
  // Age out BEFORE the duplicate check: an event whose only copy just expired
  // must be appendable again, exactly as after a rotation.
  await pruneExpired();
  if (isStoredDuplicate(event.eventId, expiryCutoff())) {
    return { appended: false, duplicate: true, event };
  }
  await rotateIfFull(mindEvent);
  const activePath = mindEvent ? MIND_ACTIVE_PATH : ACTIVE_PATH;
  await appendJSONLine(activePath, event);
  const ids = mindEvent ? mindActiveIds : activeIds;
  ids.set(event.eventId, { at: event.at, persistentMind: mindEvent });
  if (mindEvent) mindActiveCount += 1;
  else activeCount += 1;
  if (mindEvent) {
    mindSequenceHighWater.set(event.mindId, event.sequence);
    // Mind generations are bounded by rotation, while rollups can live
    // indefinitely. Persist the high-water separately so a clock rollback
    // after raw retention rotates cannot reuse an old sequence range.
    await saveMindSequenceHighWater();
  }
  return { appended: true, duplicate: false, event };
}

/**
 * Append one lifecycle event.
 *
 * Never rejects and never throws: a bad envelope or an unwritable disk is
 * logged and swallowed, because no run may fail on account of its own
 * telemetry. Returns `{ appended, duplicate, event }` on success and
 * `{ appended: false, error }` when the append was dropped, so a caller that
 * wants to know can look — none currently needs to.
 *
 * @param {object} input - see `buildRunEvent` in `lib/agentRunEvents.js`
 * @returns {Promise<{appended: boolean, duplicate?: boolean, event?: object, error?: string}>}
 */
export function appendRunEvent(input) {
  appendQueue = appendQueue.then(() => appendNow(input)).catch((err) => {
    console.error(`❌ Failed to append CoS run event (${input?.kind}): ${err.message}`);
    return { appended: false, error: err.message };
  });
  return appendQueue;
}

/** Append one persistent-mind trajectory event through the shared queue. */
export async function appendMindEvent(input) {
  const result = await appendRunEvent({ ...input, mindId: input?.mindId || PERSISTENT_MIND_ID });
  if (result.appended && result.event) runEventLogEvents.emit('mind:event', result.event);
  return result;
}

/** Resolve once every queued append has landed. Used by tests; reads take their own queue turn via `withLedgerTurn`. */
export function flushRunEvents() {
  return appendQueue.then(() => undefined, () => undefined);
}

/**
 * Run `fn` as one exclusive turn on the ledger queue, shared with appends,
 * rotation, clearing and pruning.
 *
 * Every read of a two-generation snapshot goes through here: rotation renames
 * active over archive between two file reads, so a reader outside the queue can
 * pair the old archive with the new active file and silently omit the
 * generation that just rotated. Awaiting an empty queue first would not help —
 * a writer can start before the reads finish.
 *
 * `fn` runs ON the queue, so it must never call another queued operation
 * (appendRunEvent, clearPersistentMindHistory, withLedgerTurn): that would wait
 * on a tail that is itself waiting for `fn`.
 */
function withLedgerTurn(fn) {
  const turn = appendQueue.then(fn);
  appendQueue = turn.then(() => undefined, () => undefined);
  return turn;
}

/**
 * Reclaim expired generations from inside a queued turn. Absorbs its own
 * errors: a read must never fail on account of housekeeping.
 */
async function pruneWithinTurn() {
  await pruneExpired().catch((err) => {
    console.error(`❌ Failed to prune CoS run event ledger: ${err.message}`);
  });
}

/**
 * Read the ledger, oldest generation first so the result is in append order —
 * which is what `projectRunStates` folds and what a replay needs.
 *
 * Filters are applied before the limit, and the limit keeps the NEWEST events
 * (the tail), because a diagnostic asking for 50 events wants the last 50, not
 * the first 50 of an aged-out generation.
 *
 * @param {object} [options]
 * @param {string} [options.runId] - exact run id
 * @param {string} [options.agentId] - exact agent id
 * @param {string} [options.taskId] - exact task id
 * @param {string} [options.kind] - one of AGENT_RUN_EVENT_KINDS
 * @param {string} [options.since] - ISO timestamp; events strictly after it
 * @param {number} [options.limit] - newest-N cap (default DEFAULT_READ_LIMIT)
 * @returns {Promise<object[]>} validated events in append order
 */
function readRetainedEvents() {
  return withLedgerTurn(async () => {
    await hydrate();
    await pruneWithinTurn();
    const [archive, active] = await Promise.all([
      readJSONLFile(ARCHIVE_PATH),
      readJSONLFile(ACTIVE_PATH),
    ]);
    // The age bound is enforced on read as well as on disk, so it holds in the
    // window between prunes — a reader must never see an event the retention
    // policy says is gone.
    const cutoff = expiryCutoff();
    // A line that fails the STRUCTURAL check is a corrupt/truncated write, not
    // data — dropping it keeps one bad line from poisoning the whole projection.
    // The check deliberately admits kinds this build does not know (see
    // `isStoredRunEvent`): a newer install's ledger must still read here. The
    // same predicate bounds the age, so a reader never sees an expired event.
    return [...archive, ...active].filter((event) => isRetained(event, cutoff) && !isMindShapedEvent(event));
  });
}

export async function readRunEvents({ runId, agentId, taskId, kind, since, limit } = {}) {
  let events = await readRetainedEvents();

  if (runId) events = events.filter((e) => e.runId === runId);
  if (agentId) events = events.filter((e) => e.agentId === agentId);
  if (taskId) events = events.filter((e) => e.taskId === taskId);
  if (kind) events = events.filter((e) => e.kind === kind);
  if (since) events = events.filter((e) => e.at > since);

  const cap = Math.min(Number.isFinite(limit) && limit > 0 ? Math.floor(limit) : DEFAULT_READ_LIMIT, MAX_READ_LIMIT);
  return events.length > cap ? events.slice(events.length - cap) : events;
}

/** All retained events for one mind, ordered by durable sequence. */
export function readPersistentMindHistory(mindId = PERSISTENT_MIND_ID) {
  return withLedgerTurn(async () => {
    await hydrate();
    const [archive, active] = await Promise.all([
      readJSONLFile(MIND_ARCHIVE_PATH),
      readJSONLFile(MIND_ACTIVE_PATH),
    ]);
    return [...archive, ...active]
      // Preserve structurally valid future kinds in the physical mind stream.
      // Current projections can ignore an unknown kind without deleting history.
      .filter((event) => isMindShapedEvent(event) && event.mindId === mindId)
      .sort((a, b) => a.sequence - b.sequence || String(a.eventId).localeCompare(String(b.eventId)));
  });
}

/**
 * Remove one mind's retained trajectory behind the same queue as appends.
 * A cleanup requested by the mind itself may preserve its current turn and
 * originating message so the final reply still has visible provenance.
 */
export function clearPersistentMindHistory({
  mindId = PERSISTENT_MIND_ID,
  preserveTurnId = null,
  preserveMessageId = null,
} = {}) {
  const operation = appendQueue.then(async () => {
    await hydrate();
    const [archive, active] = await Promise.all([
      readJSONLFile(MIND_ARCHIVE_PATH),
      readJSONLFile(MIND_ACTIVE_PATH),
    ]);
    const stored = [...archive, ...active].filter(isMindShapedEvent);
    const shouldPreserve = (event) => event.mindId !== mindId
      || (preserveTurnId && event.turnId === preserveTurnId)
      || (preserveMessageId && event.data?.messageId === preserveMessageId);
    const kept = relinkMindPredecessors(stored.filter(shouldPreserve));
    const removed = stored.filter((event) => event.mindId === mindId && !shouldPreserve(event));
    const generations = normalizeMindGenerations(kept);

    if (generations.archive.length === 0) {
      if (await pathExists(MIND_ARCHIVE_PATH)) await unlink(MIND_ARCHIVE_PATH);
    } else {
      await writeJSONLines(MIND_ARCHIVE_PATH, generations.archive);
    }
    await writeJSONLines(MIND_ACTIVE_PATH, generations.active);

    mindArchiveIds = indexById(generations.archive, { persistentMind: true });
    mindActiveIds = indexById(generations.active, { persistentMind: true });
    mindActiveCount = generations.active.length;
    console.log(`🧹 Cleared ${removed.length} persistent mind history event(s)`);
    return {
      cleared: removed.length,
      preserved: generations.archive.filter((event) => event.mindId === mindId).length
        + generations.active.filter((event) => event.mindId === mindId).length,
    };
  });
  appendQueue = operation.then(
    () => undefined,
    (error) => console.error(`❌ Failed to clear persistent mind history: ${error.message}`),
  );
  return operation;
}

/**
 * Cursor-aware mind tail read. A missing predecessor is explicit `gap: true`;
 * the same response carries a fresh projection and retained tail so a client
 * can recover without treating the gap as an empty conversation.
 */
export async function readPersistentMindEvents({
  mindId = PERSISTENT_MIND_ID,
  cursor,
  limit = PERSISTENT_MIND_TRAJECTORY_LIMITS.defaultPageSize,
} = {}) {
  const retained = await readPersistentMindHistory(mindId);
  const cap = Math.min(
    Number.isFinite(limit) && limit > 0 ? Math.floor(limit) : PERSISTENT_MIND_TRAJECTORY_LIMITS.defaultPageSize,
    PERSISTENT_MIND_TRAJECTORY_LIMITS.maxPageSize
  );
  const parsed = cursor ? parsePersistentMindCursor(cursor) : null;
  const cursorIndex = parsed
    ? retained.findIndex((event) => event.sequence === parsed.sequence && event.eventId === parsed.eventId)
    : -1;
  const gap = Boolean(cursor) && cursorIndex < 0;
  const available = cursorIndex >= 0 ? retained.slice(cursorIndex + 1) : retained;
  const events = cursorIndex >= 0 ? available.slice(0, cap) : available.slice(-cap);
  const last = events.at(-1) || retained.at(-1) || null;
  return {
    mindId,
    events,
    cursor: persistentMindEventCursor(last),
    gap,
    hasMore: cursorIndex >= 0 && available.length > events.length,
    truncated: cursorIndex < 0 && available.length > events.length,
    snapshot: projectPersistentMind(retained, mindId),
  };
}

/**
 * Replay the ledger into per-run current state.
 *
 * This is the "how did it get here" answer the mutable run record can't give.
 * `limit` bounds the PROJECTIONS returned (newest activity first), never the
 * events folded — the fold always runs over the newest `MAX_READ_LIMIT` events,
 * so a run's spawn and finalize are read together rather than a paged read
 * lopping the head off a lifecycle. Pass `runId` to fold one run's full history
 * regardless of how much unrelated traffic followed it.
 *
 * @param {object} [options]
 * @param {string} [options.runId] - restrict the replay to one run
 * @param {string} [options.agentId] - restrict the replay to one agent
 * @param {number} [options.limit] - max projections returned
 * @returns {Promise<object[]>}
 */
export async function getRunProjections({ runId, agentId, limit } = {}) {
  const events = await readRunEvents({ runId, agentId, limit: MAX_READ_LIMIT });
  const states = projectRunStates(events);
  const cap = Math.min(Number.isFinite(limit) && limit > 0 ? Math.floor(limit) : DEFAULT_READ_LIMIT, MAX_READ_LIMIT);
  return states.slice(0, cap);
}

/**
 * One run's projection plus the events it was folded from.
 *
 * @param {string} id - a run id, or `agent:<agentId>` for a run that never got one
 * @returns {Promise<{projection: object|null, events: object[]}>}
 */
export async function getRunDiagnostic(id) {
  const isAgentKey = typeof id === 'string' && id.startsWith('agent:');
  const events = isAgentKey
    ? (await readRunEvents({ agentId: id.slice('agent:'.length), limit: MAX_READ_LIMIT })).filter((e) => !e.runId)
    : await readRunEvents({ runId: id, limit: MAX_READ_LIMIT });
  const projection = projectRunStates(events).find((state) => state.id === id) ?? null;
  return { projection, events };
}

/**
 * Ledger health for the diagnostics route: generation sizes and the bound they
 * are held to, so "why is this run missing" has an answer that isn't a guess.
 */
export async function getRunEventLedgerStats() {
  const [archive, active, mindArchive, mindActive] = await withLedgerTurn(async () => {
    await hydrate();
    await pruneWithinTurn();
    return Promise.all([
      readJSONLFile(ARCHIVE_PATH),
      readJSONLFile(ACTIVE_PATH),
      readJSONLFile(MIND_ARCHIVE_PATH),
      readJSONLFile(MIND_ACTIVE_PATH),
    ]);
  });
  // Counted through the SAME predicate the read path uses, so "stats say 40
  // events" and "the events endpoint returns 40" can never disagree.
  const cutoff = expiryCutoff();
  const freshArchive = archive.filter((e) => isRetained(e, cutoff) && !isMindShapedEvent(e));
  const freshActive = active.filter((e) => isRetained(e, cutoff) && !isMindShapedEvent(e));
  const oldest = freshArchive[0]?.at ?? freshActive[0]?.at ?? null;
  return {
    activeEvents: freshActive.length,
    archivedEvents: freshArchive.length,
    maxActiveEvents: MAX_ACTIVE_EVENTS,
    maxRetainedEvents: MAX_ACTIVE_EVENTS * 2,
    maxEventAgeDays: MAX_EVENT_AGE_DAYS,
    mindActiveEvents: mindActive.filter(isMindShapedEvent).length,
    mindArchivedEvents: mindArchive.filter(isMindShapedEvent).length,
    maxActiveMindEvents: MAX_ACTIVE_MIND_EVENTS,
    maxRetainedMindEvents: MAX_ACTIVE_MIND_EVENTS * 2,
    persistentMindAgeBounded: false,
    oldestEventAt: oldest,
    kinds: RUN_EVENT_KINDS
  };
}

/**
 * Drop every cached seen-id map so the next call re-reads from disk.
 *
 * Exported for tests, which write ledger files directly to simulate a restart —
 * the in-process caches are exactly what a real restart discards.
 */
export function __resetRunEventLogCache() {
  activeIds = null;
  archiveIds = null;
  activeCount = 0;
  mindActiveIds = null;
  mindArchiveIds = null;
  mindActiveCount = 0;
  mindSequenceHighWater = null;
  mindSequenceCheckpointError = null;
  lastPrunedAt = null;
  hydrationPromise = null;
  appendQueue = Promise.resolve();
}