Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
14.7 KB
Raw Blame
/**
 * JSONL conversation records built from recorded session transcripts.
 *
 * Two capabilities already existed and were not connected: five adapters
 * that read a runtime's session log, and an export surface that writes
 * vault-derived bytes through the egress guard. Nothing carried the
 * transcript corpus to anything a training or evaluation harness can read.
 * This module is that carriage, and nothing else - it is reached only
 * through `o2b brain export --format transcripts-jsonl`, never as a
 * pipeline of its own, because the corpus serves a secondary operator
 * persona and this product's thesis is preference distillation.
 *
 * ## The record shape, and why it is this one
 *
 * One JSON object per line, one object per CONVERSATION: a session
 * identity plus its ordered messages. The alternative - one
 * instruction/response PAIR per line, the shape a supervised fine-tuning
 * harness eats directly - was considered and rejected, and the reason is
 * worth stating because picking either silently is how a guess becomes an
 * authoritative fact downstream.
 *
 * A pair set is a LOSSY PROJECTION of a conversation. Producing it means
 * deciding which turn is the instruction, what to do with system and tool
 * turns, and where a multi-turn exchange breaks into pairs - three
 * judgements the runtime never recorded and this exporter cannot recover.
 * A harness that wants pairs can derive them from a conversation; nothing
 * can derive the conversation back from the pairs. So the superset is what
 * leaves, and the pairing decision stays with the reader who knows which
 * one they want.
 *
 * The cost, stated rather than discovered: one conversation is held in
 * memory while its record is built. The line reader underneath never
 * materialises the file (`sessions/read-lines.ts`), so the bound is the size of
 * one session's text, not of the corpus or of the largest file.
 *
 * ## What a message carries
 *
 * Role, timestamp, turn id, text, and the NAMES of the tools the turn
 * called. Not the tool inputs: those are host paths, command lines and
 * pasted payloads, and this is a conversation dataset rather than a
 * tool-trace one. A turn with neither text nor a tool call - a runtime
 * queue event, a payload-less meta line - is not a message and is counted
 * rather than emitted.
 *
 * ## What it refuses
 *
 * A `.jsonl` under the named source that no adapter recognises stops the
 * export, naming the file. This is the same answer `collectExportRows`
 * gives an unparseable preference, for the same reason: a corpus that
 * quietly omits a transcript reads exactly like a machine that never
 * recorded it, and the file that would say otherwise is the one the
 * recipient does not have.
 *
 * A ZERO-BYTE file is the one thing that does not refuse, and is counted
 * instead ({@link TranscriptExportSummary.empty}). It carries nothing to
 * omit, Claude Code leaves them behind whenever a session is killed
 * before its first turn flushes, and refusing on them made the export
 * unusable against a real store.
 *
 * Redaction is NOT this module's job - it composes records, and the export
 * verb hands each one to `redactForEgress` before any byte is written. The
 * split is deliberate: the guard call belongs at the boundary that names
 * the destination, which is where the egress census can see it.
 */

import { statSync } from "node:fs";
import { basename } from "node:path";

import { readFirstLine } from "./sessions/read-lines.ts";
import { detectAdapter, getAdapter, SESSION_ADAPTER_REGISTRY } from "./sessions/registry.ts";
import type { SessionAdapterRegistry } from "./sessions/registry.ts";
import { sessionFilesUnder } from "./sessions/session-files.ts";
import {
  SESSION_TIMESTAMP_UNKNOWN,
  SessionImportError,
  type SessionTurn,
} from "./sessions/types.ts";

/**
 * Versions the RECORD, not the corpus. A field added later is additive and
 * needs no bump - a reader written against version 1 keeps working, the
 * same contract `BRAIN_EXPORT_SCHEMA_VERSION` states for the
 * preference envelope. A field that changes meaning is what would force it.
 */
export const TRANSCRIPT_EXPORT_SCHEMA_VERSION = 1 as const;

/** One turn of a conversation, as a dataset row sees it. */
export interface TranscriptMessage {
  /** The adapter's own turn id - the join back to the source transcript. */
  readonly turn_id: string;
  readonly role: SessionTurn["role"];
  readonly timestamp: string;
  /** Flat text of the turn. Empty when the turn was only a tool call. */
  readonly text: string;
  /** Tool names, in call order. Absent when the turn called none. */
  readonly tools?: ReadonlyArray<string>;
}

/** One session transcript, as one JSONL line. */
export interface TranscriptConversation {
  readonly schema: typeof TRANSCRIPT_EXPORT_SCHEMA_VERSION;
  /** Adapter id the file was detected as (`claude`, `codex`, …). */
  readonly runtime: string;
  /**
   * The transcript's basename, never its absolute path: the same portable
   * identity `sessionRefIdentity` uses for imported provenance, so a
   * dataset row and a vault signal join on one key and neither bakes in a
   * host directory.
   */
  readonly session_id: string;
  readonly started_at: string;
  readonly ended_at: string;
  readonly message_count: number;
  readonly messages: ReadonlyArray<TranscriptMessage>;
}

export interface TranscriptExportOptions {
  /** A transcript file, or a directory walked for `*.jsonl`. */
  readonly source: string;
  /** Keep only conversations recorded by this adapter id. */
  readonly runtime?: string;
  /** Keep conversations that STARTED at or after this instant. */
  readonly since?: Date;
  /** Keep conversations that STARTED strictly before this instant. */
  readonly until?: Date;
  /** Adapter set to detect against. Defaults to the shipped registry. */
  readonly registry?: SessionAdapterRegistry;
}

/**
 * What the walk did, returned when the stream ends.
 *
 * Every scanned file is accounted for by exactly one counter, so an empty
 * export can say which filter emptied it. "No records" and "no transcripts
 * to begin with" are different facts, and a caller that cannot tell them
 * apart reports a filter typo as an idle machine.
 */
export interface TranscriptExportSummary {
  readonly scanned: number;
  readonly exported: number;
  /** Recognised, but written by a runtime the filter excluded. */
  readonly other_runtime: number;
  /** Recognised, but started outside the requested window. */
  readonly outside_window: number;
  /** Recognised, and carrying no turn a dataset can learn from. */
  readonly no_messages: number;
  /**
   * Holding no bytes at all, so there was nothing to detect a format
   * from.
   *
   * Counted rather than refused, which is the one exception to the
   * refuse-what-you-cannot-read rule below, and it is not a softening of
   * it. A zero-byte transcript is not a file this walk failed to
   * understand - it is a file with no content to understand, which is
   * what Claude Code leaves behind routinely when a session opens and the
   * process is killed before the first turn flushes. Refusing on those
   * made the whole export unusable against a real store and offered a
   * remedy ("move this file out of the source directory") that does not
   * scale to a thousand of them. A file that HAS content no adapter
   * recognises still refuses: there, something was recorded and this
   * build cannot say what.
   */
  readonly empty: number;
}

/** Tool names of a turn, in call order. */
function toolNames(turn: SessionTurn): ReadonlyArray<string> {
  return (turn.toolCalls ?? []).map((call) => call.name);
}

/** A turn as a message, or `null` when it carries nothing to learn from. */
function messageOf(turn: SessionTurn): TranscriptMessage | null {
  const text = typeof turn.text === "string" ? turn.text : "";
  const tools = toolNames(turn);
  if (text.trim() === "" && tools.length === 0) return null;
  return Object.freeze({
    turn_id: turn.turnId,
    role: turn.role,
    timestamp: turn.timestamp,
    text,
    ...(tools.length > 0 ? { tools: Object.freeze([...tools]) } : {}),
  });
}

/**
 * The conversation's start as an instant, for the window comparison only.
 *
 * A timestamp this module cannot READ is refused rather than treated as
 * either inside or outside the window: both readings are a guess, and one
 * of them silently drops a transcript from the corpus. Only reached when a
 * window is actually set - with no `--since` / `--until` nothing depends
 * on the value, and refusing an export over a field nobody read would be a
 * refusal for its own sake.
 *
 * "Cannot read" has TWO cases, and for a long time this function handled
 * only the one that never happens. A string `Date.parse` rejects is the
 * obvious case; it does not occur, because no adapter ever emits one. The
 * case that does occur is {@link SESSION_TIMESTAMP_UNKNOWN}: every adapter
 * synthesises the epoch when the line it read carried no clock, that
 * parses perfectly well, and it sorts before any window an operator would
 * type - so the transcript took the silently-dropped path this docblock
 * forbids, on the exact input the guarantee was written for.
 */
function startInstant(record: TranscriptMessage, path: string): number {
  if (record.timestamp === SESSION_TIMESTAMP_UNKNOWN) {
    throw new SessionImportError(
      "PARSE",
      `the first turn of ${path} carries no timestamp - the runtime did not record one, and ` +
        "the adapter reports that as the epoch - so --since / --until cannot say whether this " +
        "conversation is in the window; drop the window flags to export it, or narrow " +
        "--transcripts to the files that are timestamped",
    );
  }
  const ms = Date.parse(record.timestamp);
  if (!Number.isFinite(ms)) {
    throw new SessionImportError(
      "PARSE",
      `cannot read the first timestamp of ${path} ('${record.timestamp}'), so --since / ` +
        "--until cannot say whether this conversation is in the window; repair the " +
        "transcript or drop the window flags",
    );
  }
  return ms;
}

function inWindow(startedAtMs: number, options: TranscriptExportOptions): boolean {
  if (options.since !== undefined && startedAtMs < options.since.getTime()) return false;
  if (options.until !== undefined && startedAtMs >= options.until.getTime()) return false;
  return true;
}

/**
 * True when `path` holds zero bytes.
 *
 * A stat that fails answers `false` rather than `true`: the file was
 * listed a moment ago, so a failure here is a race or a permission
 * change, and reading that as "empty" would drop a transcript on the
 * strength of an error. It falls through to the detect refusal, which
 * names the file.
 */
function holdsNoBytes(path: string): boolean {
  try {
    return statSync(path).size === 0;
  } catch {
    return false;
  }
}

/** Why a scanned file produced no record, when it produced none. */
type SkipKind = "outside_window" | "no_messages";

async function conversationOf(
  path: string,
  runtime: string,
  iterate: AsyncIterable<SessionTurn>,
  options: TranscriptExportOptions,
): Promise<TranscriptConversation | SkipKind> {
  const windowed = options.since !== undefined || options.until !== undefined;
  const messages: TranscriptMessage[] = [];
  for await (const turn of iterate) {
    const message = messageOf(turn);
    if (message === null) continue;
    if (windowed && messages.length === 0 && !inWindow(startInstant(message, path), options)) {
      // The window selects whole conversations by their start, so the rest
      // of this file cannot change the answer and is never read.
      return "outside_window";
    }
    messages.push(message);
  }
  if (messages.length === 0) return "no_messages";
  return Object.freeze({
    schema: TRANSCRIPT_EXPORT_SCHEMA_VERSION,
    runtime,
    session_id: basename(path),
    started_at: messages[0]!.timestamp,
    ended_at: messages[messages.length - 1]!.timestamp,
    message_count: messages.length,
    messages: Object.freeze(messages),
  });
}

/**
 * Stream one conversation record per recognised transcript under
 * `options.source`, then return {@link TranscriptExportSummary}.
 *
 * A generator rather than an array: the caller writes each record as it
 * arrives, so the corpus is never held whole. The summary rides out as the
 * generator's RETURN value, which is why a caller that needs it drives
 * `next()` itself instead of using `for await` - `for await` discards it,
 * and a counter nobody can read is not a report.
 *
 * @throws SessionImportError when the source is unreadable, a file matches
 * no adapter, or a window comparison meets an unparseable timestamp.
 */
export async function* streamTranscriptConversations(
  options: TranscriptExportOptions,
): AsyncGenerator<TranscriptConversation, TranscriptExportSummary> {
  const registry = options.registry ?? SESSION_ADAPTER_REGISTRY;
  // Resolved for its refusal, which names the registered set: an unknown
  // `--runtime` would otherwise match nothing and report an empty corpus.
  if (options.runtime !== undefined) getAdapter(options.runtime, registry);

  const files = sessionFilesUnder(options.source);
  let exported = 0;
  let otherRuntime = 0;
  let outsideWindow = 0;
  let noMessages = 0;
  let empty = 0;

  for (const path of files) {
    // oxlint-disable-next-line no-await-in-loop
    const firstLine = await readFirstLine(path);
    if (firstLine === "" && holdsNoBytes(path)) {
      // Accounted for and stepped over - see `empty` on the summary for
      // why this one case is not the refusal below. The size check is what
      // keeps it to that case: `readFirstLine` also returns `""` for a
      // file whose first line is blank and whose second line is a turn,
      // and skipping THAT would be the silent omission the refusal exists
      // to prevent.
      empty += 1;
      continue;
    }
    const adapter = detectAdapter(firstLine, registry);
    if (adapter === null) {
      throw new SessionImportError(
        "DETECT_FAIL",
        `could not autodetect a session format for ${path}; every *.jsonl under the ` +
          "exported source must be a session transcript, so move this file out of the " +
          "source directory or point --transcripts at one that holds only transcripts",
      );
    }
    if (options.runtime !== undefined && adapter.id !== options.runtime) {
      otherRuntime += 1;
      continue;
    }
    // oxlint-disable-next-line no-await-in-loop
    const record = await conversationOf(path, adapter.id, adapter.iterate(path), options);
    if (record === "outside_window") {
      outsideWindow += 1;
      continue;
    }
    if (record === "no_messages") {
      noMessages += 1;
      continue;
    }
    exported += 1;
    yield record;
  }

  return Object.freeze({
    scanned: files.length,
    exported,
    other_runtime: otherRuntime,
    outside_window: outsideWindow,
    no_messages: noMessages,
    empty,
  });
}