Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
41.6 KB
Raw Blame
/**
 * Model-mined session signals (salience-lifecycle-enrichment, unit 2,
 * t_1dace26d).
 *
 * `fact-extract.ts` mines a session turn with regexes, which is why it can
 * run inside the live capture hook: it costs nothing and knows no language.
 * The price is that it only ever recognises structure - a URL, an address,
 * a unit-bound number - so every stated preference in a transcript is
 * invisible to it. This module is the batch counterpart: it hands the
 * calling agent the user turns of an ALREADY-IMPORTED session and asks for
 * the taste signals in them. OSB still calls no model; it owns the turns it
 * offers, the limits the answer must satisfy, and the write.
 *
 * Two phases, diarization-shaped:
 *
 *   1. {@link planExtractSignals} - read-only. Resolves the session's
 *      imported turns, keeps the USER ones (the import path's rule at
 *      `sessions/import.ts`: bare assistant output is never auto-extracted),
 *      and returns a report carrying exactly one needs-llm-step envelope.
 *   2. {@link commitExtractedSignals} - validates the returned payload
 *      structurally ({@link EXTRACTED_SIGNALS_SHAPE}) and semantically (the
 *      per-session cap and the per-item confidence floor, registered on the
 *      semantic-check registry because neither is expressible in a
 *      descriptor), then routes the accepted items through the same helper
 *      chain the deterministic extractor uses.
 *
 * Everything written here is SPECULATIVE. The items land in `Brain/inbox/`
 * as `source_type: auto_extract` signals - a member of its own so a
 * model-authored signal is distinguishable from a regex-authored one
 * forever - and the inbox trial lane plus the dream pass remains the only
 * route to a confirmed preference. Nothing here writes a preference.
 *
 * The commit is also the mining ledger's write side (session-mining-ledger.ts):
 * every successful commit records its outcome breakdown over the session's
 * current mined turns. A zero-yield record (nothing written, nothing
 * staged) is the tombstone the plan gate suppresses on, so it is written
 * only over material a plan put in front of a model: the hash the caller
 * relays, or else the one survey the plan surfaces recorded
 * (session-mining-surveys.ts). A yield record is the distilled state the
 * mining census reports. A partial write never records - a run that FAILED
 * did not end.
 *
 * Precision beats recall, as it does in `fact-extract.ts`: the cap and the
 * floor are refusals, not filters. A payload over either limit is rejected
 * whole and named, because an over-eager mine that silently kept its best
 * twelve would teach the caller nothing about what it asked for.
 */

import { posix } from "node:path";

import { redactRawOutput } from "../redactor.ts";
import { buildReconciliationReport, type ReconciliationReport } from "../reconciliation-report.ts";
import {
  buildCaptureBoundary,
  type CaptureBoundary,
  type SessionCaptureDecision,
} from "./capture-boundary.ts";
import {
  tryRecordDeadLetter,
  type DeadLetterLane,
  type DeadLetterOutcome,
  type DeadLetterPayload,
} from "./dead-letter.ts";
import { buildDedupIndex, computeDedupHash, type DedupIndexEntry } from "./dedup-hash.ts";
import { classifyDurability, resolveDurabilityDenylist } from "./gates/durability.ts";
import { buildNeedsLlmStep, type NeedsLlmStep } from "./llm-step.ts";
import { brainDirsForWrite } from "./paths.ts";
import { resolveWriteApprovalEnabled } from "./pending.ts";
import {
  assertResponseCheck,
  registerResponseCheck,
  SEMANTIC_VIOLATION_CODES,
  semanticViolation,
  type SemanticViolation,
} from "./response-checks.ts";
import {
  assertResponseShape,
  EXTRACTED_SIGNALS_SHAPE,
  EXTRACTED_SIGNALS_SURFACE,
  SHAPE_ROOT_PATH,
} from "./response-shape.ts";
import { listSessionRawTurns, type ExpandedRawTurn } from "./session-recall.ts";
import { minedTurnsHash, recordSessionMiningOutcome } from "./session-mining-ledger.ts";
import {
  clearMiningSurvey,
  closedSurveyFailure,
  readMiningSurvey,
} from "./session-mining-surveys.ts";
import { writeSignal } from "./signal.ts";
import { isoDate, isoSecond } from "./time.ts";
import { BRAIN_SIGNAL_SOURCE_TYPE, type BrainSignalSign } from "./types.ts";

/** Vault-relative directory an accepted auto-extracted signal lands in. */
export const AUTO_EXTRACT_TARGET_DIR_REL = "Brain/inbox";

/** The needs-llm-step step name for the deferred signal mining. */
export const EXTRACT_SIGNALS_STEP = "extract-signals";

/**
 * This lane's dead-letter name. It is the ONE multi-artifact commit lane
 * in the tree - one file per mined item from one validated payload - and
 * so the only one that can leave a partial write behind.
 */
const EXTRACT_SIGNALS_LANE: DeadLetterLane = "extract-signals";

/**
 * Most signals one session may yield. A transcript that genuinely states
 * more than this many durable preferences is a transcript worth reading by
 * hand; past the cap the mine is guessing, and guesses at inbox scale cost
 * more to retire than they ever pay back.
 */
export const AUTO_EXTRACT_PER_SESSION_CAP = 12;

/**
 * Lowest confidence an item may carry. The floor sits above a coin flip on
 * purpose: an item the caller is not clearly sure of is exactly the item
 * whose retirement will cost an operator a review.
 */
export const AUTO_EXTRACT_CONFIDENCE_FLOOR = 0.6;

/** The role whose turns are mining material - the import path's rule. */
const MINED_TURN_ROLE = "user";

/** Per-turn text budget in the prompt, so one huge turn cannot crowd out the rest. */
export const PROMPT_TURN_TEXT_MAX = 2000;

/** A signal extraction could not proceed, and the reason is in the message. */
export class ExtractSignalsError extends Error {
  constructor(message: string) {
    super(message);
    this.name = "ExtractSignalsError";
  }
}

/**
 * A write failed PART WAY through the accepted items.
 *
 * The commit writes one file per item, so a failure on the fourth leaves
 * three signals on disk. The house precedent for that state is
 * `ArchWriteError` (`architect/generate.ts`): a partial write reports how
 * far it got rather than throwing a bare cause and leaving the caller to
 * guess. This is that error for this lane - the ids already written are
 * ON it, so an operator can find them, and re-running the same payload is
 * safe because the dedup index recognises every one of them.
 *
 * Since nothing-writes-silently it carries two further things. The first
 * is {@link reconciliation}: the same `attempted` / `found` / `missing`
 * report shape the import read-back census and the embedder audit use, so
 * this lane's accounting is not a fourth dialect of the same three
 * numbers. The second is {@link deadLetter}: this error's own accounting
 * lives exactly as long as the response carrying it, and a caller that
 * drops the response must not be the only record that a partial write
 * happened - so the accounting is ALSO on disk, and this field says
 * where, or why it could not be.
 *
 * Not an {@link ExtractSignalsError}: that class is the caller's fault
 * and the surfaces report it as such. A filesystem that refused a write
 * is not.
 */
export class ExtractSignalsWriteError extends Error {
  /** The item whose write failed. */
  readonly topic: string;
  /** Signals whose bytes are already on disk, in write order. */
  readonly written: ReadonlyArray<ExtractedSignalWrite>;
  /** Payload items this call never reached, the failing one included. */
  readonly remaining: number;
  /** Attempted / found / missing for this commit, missing keys named. */
  readonly reconciliation: ReconciliationReport;
  /** The error that stopped the commit, kept as thrown. */
  readonly firstError: unknown;
  /** Where the durable record of this partial write landed, or why not. */
  readonly deadLetter: DeadLetterOutcome;

  constructor(params: {
    readonly topic: string;
    readonly written: ReadonlyArray<ExtractedSignalWrite>;
    readonly remaining: number;
    readonly reconciliation: ReconciliationReport;
    readonly cause: unknown;
    readonly deadLetter: DeadLetterOutcome;
  }) {
    const { topic, written, remaining, reconciliation, cause, deadLetter } = params;
    super(
      `failed to write extracted signal ${JSON.stringify(topic)}: ` +
        `${cause instanceof Error ? cause.message : String(cause)} - ` +
        `attempted ${reconciliation.attempted}, written ${reconciliation.found}, ` +
        `failed ${reconciliation.missing.length} (${reconciliation.missing.join(", ")}); ` +
        `${written.length} signal(s) are already on disk ` +
        `(${written.length === 0 ? "none" : written.map((w) => w.id).join(", ")}) and ` +
        `${remaining} payload item(s) were not reached; ${describeDeadLetter(deadLetter)}; ` +
        "fix the cause and re-run the same payload, which dedups what is already there",
      { cause },
    );
    this.name = "ExtractSignalsWriteError";
    this.topic = topic;
    this.written = Object.freeze([...written]);
    this.remaining = remaining;
    this.reconciliation = reconciliation;
    this.firstError = cause;
    this.deadLetter = deadLetter;
  }
}

/** The dead-letter half of the message above, either way it went. */
function describeDeadLetter(outcome: DeadLetterOutcome): string {
  return outcome.recorded
    ? `a dead letter recording this is at ${outcome.path} (${outcome.id}); ` +
        "'o2b brain dead-letters' classifies it and names the recovery exit " +
        "(diagnostic code extract-signals-dead-letter)"
    : `NO dead letter could be recorded for it (${outcome.reason})`;
}

/** One user turn offered to the caller as mining material. */
export interface MinedTurn {
  readonly turnId: string;
  readonly text: string;
  /**
   * The turn's stored timestamp, verbatim; "" when the source has none.
   * Never a formatted `Date` and never the plan's clock: the caller grounds
   * a relative time bound against it, so an invented value would ground it
   * against the wrong day.
   */
  readonly timestamp: string;
}

/** Phase-one report: what will be mined, and the one envelope that mines it. */
export interface SignalExtractionPlan {
  readonly sessionId: string;
  readonly generatedAt: string;
  /** Capture-boundary verdict for the session; always `capture` here. */
  readonly boundaryDecision: SessionCaptureDecision;
  /** Imported turns of every role, before the user filter. */
  readonly turnsScanned: number;
  /** The user turns the prompt embeds, in record order. */
  readonly turnsMined: ReadonlyArray<MinedTurn>;
  /**
   * {@link minedTurnsHash} of {@link turnsMined}: the material this plan
   * put in front of the caller. The commit that answers this plan may pass
   * it back as {@link CommitExtractedSignalsOptions.surveyedTurnsHash} to
   * check the session did not change in between. Without the relay the
   * commit reads the survey the plan surfaces recorded for the session
   * instead.
   */
  readonly surveyedTurnsHash: string;
  /** The limits the payload will be held to, echoed so the caller can read them. */
  readonly cap: number;
  readonly confidenceFloor: number;
  readonly llmStep: NeedsLlmStep;
}

/** One mined taste signal, as the calling agent returns it. */
export interface ExtractedSignalItem {
  readonly topic: string;
  readonly signal: BrainSignalSign;
  readonly principle: string;
  readonly confidence: number;
  readonly scope?: string;
  /**
   * Optional plan turn id the item was mined from. Used only to measure
   * the decision-model turn pre-filter; an item without it behaves as
   * before.
   */
  readonly source_turn?: string;
}

/** One item the durability gate refused, named rather than dropped. */
export interface ExtractedSignalRejection {
  readonly topic: string;
  readonly reason: string;
}

/** One item that reached disk, with the coordinates it landed at. */
export interface ExtractedSignalWrite {
  readonly id: string;
  readonly path: string;
  readonly topic: string;
  /** The item's `source_turn`, when it carried a non-empty one. */
  readonly sourceTurn?: string;
}

export interface CommitExtractedSignalsResult {
  readonly sessionId: string;
  readonly written: ReadonlyArray<ExtractedSignalWrite>;
  /**
   * How many of {@link written} were STAGED into `Brain/pending/` rather
   * than written straight to the inbox. Zero when write approval is off.
   */
  readonly staged: number;
  readonly deduped: number;
  /** Items the durability gate refused; each is named in {@link rejected}. */
  readonly durabilityRejected: number;
  readonly rejected: ReadonlyArray<ExtractedSignalRejection>;
  /**
   * Write accounting in the wave's shared vocabulary: `attempted` is the
   * items that reached the write call, `found` the ones whose bytes
   * landed, `missing` the keys of the rest. On this path `missing` is
   * always empty - a failure throws {@link ExtractSignalsWriteError},
   * which carries the same shape with the gap named.
   *
   * A deduped or durability-rejected item is NOT attempted: both are
   * deliberate refusals this result already names in {@link deduped} and
   * {@link rejected}, and counting a refusal the lane made as a write it
   * lost would misreport the one number an operator reads first.
   */
  readonly reconciliation: ReconciliationReport;
  /**
   * Present only when the commit itself SUCCEEDED but recording the mining
   * outcome failed (session-mining-ledger.ts). The signals this result
   * reports are already on disk, so throwing would misreport a successful
   * write as a failed one - and swallowing the failure would hide a ledger
   * that is no longer trustworthy, so the error message rides here by
   * name, and on both surface outputs.
   */
  readonly miningLedgerFailure?: string;
}

export interface PlanExtractSignalsOptions {
  readonly now: Date;
  /**
   * Add the optional `source_turn` item field to the envelope's schema
   * hints. Set only by the decision-model pre-filter path (use not
   * `off`), which measures regret from it; absent, the envelope is
   * exactly as it has always been.
   */
  readonly sourceTurnHint?: boolean;
}

export interface CommitExtractedSignalsOptions {
  readonly agent: string;
  readonly now: Date;
  /**
   * Pre-compiled durability denylist. When omitted it is resolved from the
   * plugin config, mirroring `routeExtractedFacts`; tests inject it so the
   * gate is exercised without the default config path.
   */
  readonly durabilityDenylist?: ReadonlyArray<RegExp>;
  /** Write-approval toggle; resolved from config when omitted. */
  readonly writeApprovalEnabled?: boolean;
  /** Shared dedup index; built from the vault when omitted. */
  readonly dedup?: Map<string, DedupIndexEntry>;
  /**
   * The answered plan's {@link SignalExtractionPlan.surveyedTurnsHash}: the
   * hash of the turns the caller was actually shown. Optional: the commit
   * records over the hash of the session's CURRENT mined turns, and a
   * zero-yield record (the tombstone) additionally needs evidence that a
   * model saw exactly that material - this relay, or else the single
   * pending survey the plan surfaces recorded, so the documented flow
   * (plan, answer the envelope, commit the items) writes its tombstone
   * without relaying anything. When the hash IS relayed it is a check: it
   * must be a SHA-256 hex digest (anything else refuses the commit before
   * a write), and when it differs from the current material - an import
   * landed between plan and commit, or the hash came from another plan -
   * nothing is recorded and the commit names that as its mining ledger
   * failure. Without a relay, a survey that differs from the current
   * material, several pending surveys, or none withholds the tombstone and
   * is named the same way; a yield record is recorded regardless.
   */
  readonly surveyedTurnsHash?: string;
  /**
   * Name a pending-survey store failure in `miningLedgerFailure` by its
   * store and errno code only, never by the raised message, which carries
   * the absolute host path. Set by surfaces that answer a caller below
   * local reach.
   */
  readonly closeHostErrors?: boolean;
}

// ----- Semantic rules the descriptor language cannot express ---------------

/**
 * Cap, floor, topic uniqueness and interval sanity, registered once at
 * module scope. Each reads across the payload or inside a value rather
 * than one value at a time - the cap counts the whole list, the floor
 * compares a number to a limit the descriptor cannot name, uniqueness
 * compares items with each other, and an interval compares the two dates
 * inside one principle - so they belong here rather than in
 * {@link EXTRACTED_SIGNALS_SHAPE}.
 *
 * Registration is a module side effect on purpose: the registry is
 * fail-closed, so a lane that forgot to register refuses its own writes
 * instead of performing them unchecked.
 */
registerResponseCheck(EXTRACTED_SIGNALS_SURFACE, (payload) => {
  const violations: SemanticViolation[] = [];
  const items = readItems(payload);
  if (items === null) return violations;
  if (items.length > AUTO_EXTRACT_PER_SESSION_CAP) {
    violations.push(
      semanticViolation(
        SEMANTIC_VIOLATION_CODES.threshold,
        `${SHAPE_ROOT_PATH}.items`,
        `at most ${AUTO_EXTRACT_PER_SESSION_CAP} signals may be mined from one session; got ${items.length}`,
      ),
    );
  }
  items.forEach((item, index) => {
    const confidence = (item as { confidence?: unknown }).confidence;
    if (typeof confidence !== "number") return;
    if (confidence < AUTO_EXTRACT_CONFIDENCE_FLOOR) {
      violations.push(
        semanticViolation(
          SEMANTIC_VIOLATION_CODES.threshold,
          `${SHAPE_ROOT_PATH}.items[${index}].confidence`,
          `confidence must be at least ${AUTO_EXTRACT_CONFIDENCE_FLOOR}; got ${confidence}`,
        ),
      );
    }
  });
  violations.push(...duplicateTopicViolations(items));
  violations.push(...intervalViolations(items));
  return violations;
});

/**
 * An ISO interval inside a principle, the form the dream pass turns into
 * a `[valid_from, valid_until)` window. Global so every interval in one
 * principle is checked, not only the first.
 */
const PRINCIPLE_INTERVAL_RE = /\b(\d{4}-\d{2}-\d{2})\/(\d{4}-\d{2}-\d{2})\b/g;

/**
 * One violation per interval in a principle that names an impossible
 * calendar date or ends on or before it starts. Either one yields a window
 * that is never active, and nothing downstream would say why, so the
 * payload is refused here where the caller can still correct it.
 */
function intervalViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
  const violations: SemanticViolation[] = [];
  items.forEach((item, index) => {
    const principle = (item as { principle?: unknown }).principle;
    if (typeof principle !== "string") return;
    for (const [interval, start, end] of principle.matchAll(PRINCIPLE_INTERVAL_RE)) {
      const impossible = [start!, end!].find((date) => !isCalendarDate(date));
      const message =
        impossible !== undefined
          ? `interval ${interval} names ${impossible}, which is not a calendar date`
          : end! <= start!
            ? `interval ${interval} must end after it starts; its end date is exclusive, the day after the last day the rule holds`
            : null;
      if (message === null) continue;
      violations.push(
        semanticViolation(
          SEMANTIC_VIOLATION_CODES.threshold,
          `${SHAPE_ROOT_PATH}.items[${index}].principle`,
          message,
        ),
      );
    }
  });
  return violations;
}

/** Whether a `YYYY-MM-DD` string names a real day, so `2026-02-30` does not. */
function isCalendarDate(date: string): boolean {
  const parsed = new Date(`${date}T00:00:00Z`);
  return !Number.isNaN(parsed.getTime()) && parsed.toISOString().slice(0, 10) === date;
}

/**
 * One violation per item that repeats an earlier item's topic, naming both
 * indices. The comparison is exact: two items under one topic are one rule
 * stated twice or two rules filed under one name, and the lane cannot tell
 * which - so the payload is refused whole rather than one of them kept by a
 * guess. A fuzzy comparison of principles is deliberately absent: two short
 * rules of opposite meaning read alike, and one false refusal here would
 * drop every valid signal in the run.
 */
function duplicateTopicViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
  const violations: SemanticViolation[] = [];
  const firstIndex = new Map<string, number>();
  items.forEach((item, index) => {
    const topic = (item as { topic?: unknown }).topic;
    if (typeof topic !== "string") return;
    const first = firstIndex.get(topic);
    if (first === undefined) {
      firstIndex.set(topic, index);
      return;
    }
    violations.push(
      semanticViolation(
        SEMANTIC_VIOLATION_CODES.crossItem,
        `${SHAPE_ROOT_PATH}.items[${index}].topic`,
        `items[${first}] and items[${index}] share topic ${JSON.stringify(topic)}`,
      ),
    );
  });
  return violations;
}

/**
 * The item list, or null when the payload is not shaped like one. The
 * shape layer has already refused that case by the time the check runs;
 * returning null keeps the check pure and total rather than throwing from
 * inside a validator whose contract says it never throws.
 */
function readItems(payload: unknown): ReadonlyArray<unknown> | null {
  if (typeof payload !== "object" || payload === null) return null;
  const items = (payload as { items?: unknown }).items;
  return Array.isArray(items) ? items : null;
}

// ----- Phase one: what to mine ---------------------------------------------

/**
 * The user turns the prompt embeds, in record order: the import path's
 * role rule, the message-level capture boundary, and blank text dropped.
 * Shared by the plan (which offers them) and the mining census (whose
 * `minedTurnMaterial` must hash exactly what the plan would mine), so the
 * two can never disagree on what a run covered.
 */
export function selectMinedTurns(
  boundary: CaptureBoundary,
  turns: ReadonlyArray<ExpandedRawTurn>,
): MinedTurn[] {
  const mined: MinedTurn[] = [];
  for (const turn of turns) {
    if (turn.role !== MINED_TURN_ROLE) continue;
    // Message-level boundary, mirroring the import path: suppressed text
    // never reaches extraction, so it never reaches the prompt either.
    if (boundary.suppressMessage(turn.text)) continue;
    const text = turn.text.trim();
    if (text.length === 0) continue;
    mined.push(Object.freeze({ turnId: turn.turn_id, text, timestamp: turn.timestamp }));
  }
  return mined;
}

/**
 * Assemble the mining plan for one imported session. Read-only; writes
 * nothing and calls no model.
 *
 * Three conditions are refusals rather than empty reports, because each
 * one means the caller asked for something this vault cannot answer:
 * a session the capture boundary ignores, a session with no imported turns
 * at all, and a session whose turns are all non-user. Reporting any of them
 * as "mined nothing" would read as a clean run over a barren transcript.
 */
export function planExtractSignals(
  vault: string,
  sessionId: string,
  opts: PlanExtractSignalsOptions,
): SignalExtractionPlan {
  const trimmed = sessionId.trim();
  if (trimmed.length === 0) {
    throw new ExtractSignalsError("extract-signals needs a session reference");
  }
  const boundary = buildCaptureBoundary(vault);
  const boundaryDecision = boundary.sessionDecision(trimmed);
  if (boundaryDecision !== "capture") {
    throw new ExtractSignalsError(
      `session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,
    );
  }
  const turns = listSessionRawTurns(vault, trimmed);
  if (turns.length === 0) {
    throw new ExtractSignalsError(
      `session ${trimmed} has no imported turns; import it first with 'o2b brain import-session --recall'`,
    );
  }
  const mined = selectMinedTurns(boundary, turns);
  if (mined.length === 0) {
    throw new ExtractSignalsError(
      `session ${trimmed} has ${turns.length} imported turn(s) but none authored by the user; there is nothing to mine`,
    );
  }
  return Object.freeze({
    sessionId: trimmed,
    generatedAt: opts.now.toISOString(),
    boundaryDecision,
    turnsScanned: turns.length,
    turnsMined: Object.freeze(mined),
    surveyedTurnsHash: minedTurnsHash(mined),
    cap: AUTO_EXTRACT_PER_SESSION_CAP,
    confidenceFloor: AUTO_EXTRACT_CONFIDENCE_FLOOR,
    llmStep: buildMiningStep(trimmed, mined, { sourceTurnHint: opts.sourceTurnHint === true }),
  });
}

/**
 * Hygiene rules the mined items are held to, in the instruction.
 *
 * Each names a CATEGORY and never an example word: the operator may write
 * in any language, and a kernel that listed one language's greetings or
 * date words would mine that language better than every other. Time is
 * grounded the same way - the caller resolves a relative bound against the
 * turn's timestamp and writes it as ISO, which the dream pass already turns
 * into a validity window, so no date parser lives here.
 */
const EXTRACT_HYGIENE_RULES: ReadonlyArray<string> = Object.freeze([
  "Write any time bound a rule carries as an ISO 8601 date or interval (YYYY-MM-DD, or " +
    "YYYY-MM-DD/YYYY-MM-DD), resolving a relative bound against the timestamp of the turn " +
    "that stated it; an interval's end date is exclusive, so write the day after the last " +
    "day the rule holds, and an end that is not after the start refuses the whole payload; " +
    "write a bound that only ends as an interval from that turn's date, " +
    "because a lone date reads as the day the rule starts; a rule with no time bound stays " +
    "undated.",
  "Skip conversational mechanics - greetings, thanks, acknowledgements, requests to go on, " +
    "and directions about the flow of this session - because they state no rule.",
  "Return one rule per item: a turn that states two rules yields two items with distinct topics.",
  "Keep every condition a rule was stated with - when, where, for which project or kind of " +
    "work - inside its principle; never widen a conditional rule into an unconditional one.",
  "Drop restatements: a rule the operator states more than once is returned once, under one " +
    "topic, because two items sharing a topic refuse the whole payload.",
]);

/**
 * The one envelope. The turns ride inside the prompt rather than beside it
 * because the receiving agent is answering a question about exactly these
 * turns and nothing else; a prompt that named a session id would invite the
 * caller to go and read more of it.
 */
export function buildMiningStep(
  sessionId: string,
  mined: ReadonlyArray<MinedTurn>,
  opts: { readonly sourceTurnHint?: boolean } = {},
): NeedsLlmStep {
  const transcript = mined
    .map((turn) => `${transcriptLabel(turn)} ${turn.text.slice(0, PROMPT_TURN_TEXT_MAX)}`)
    .join("\n");
  return buildNeedsLlmStep({
    step: EXTRACT_SIGNALS_STEP,
    prompt:
      `Mine durable taste signals from the ${mined.length} user turn(s) of session ${sessionId} below. ` +
      "A taste signal is a rule the operator stated about how work should be done - a preference, a " +
      "correction, a prohibition - not a fact, a task, or a one-off instruction about this session. " +
      `Return at most ${AUTO_EXTRACT_PER_SESSION_CAP} items, each with a confidence of at least ` +
      `${AUTO_EXTRACT_CONFIDENCE_FLOOR}; omit anything you are less sure of rather than lowering the ` +
      "number, because an item below the floor refuses the whole payload. " +
      EXTRACT_HYGIENE_RULES.join(" ") +
      "\n\n" +
      transcript,
    schema_hints: [
      'payload: { "items": [ { "topic", "signal", "principle", "confidence", "scope"? } ] }',
      "topic: stable kebab-slug naming the rule, unique within the payload",
      "signal: 'positive' when the principle is the rule to follow, 'negative' when it is what to avoid",
      "principle: one imperative line, in the language the operator used, with any time bound " +
        "written as an ISO 8601 date or interval",
      `confidence: number in [${AUTO_EXTRACT_CONFIDENCE_FLOOR}, 1]`,
      `items: at most ${AUTO_EXTRACT_PER_SESSION_CAP} entries`,
      ...(opts.sourceTurnHint === true
        ? ["source_turn: optional, the bracketed id of the turn the rule was stated in"]
        : []),
    ],
    target_path: posix.normalize(AUTO_EXTRACT_TARGET_DIR_REL),
  });
}

/** `[turnId @ timestamp]`, or `[turnId]` when the turn has no timestamp. */
function transcriptLabel(turn: MinedTurn): string {
  return turn.timestamp === "" ? `[${turn.turnId}]` : `[${turn.turnId} @ ${turn.timestamp}]`;
}

// ----- Phase two: what the caller mined ------------------------------------

/**
 * The payload as it is FROZEN into a dead letter. The letter sits in
 * `.open-second-brain/dead-letters/` forever (the retention rule deletes
 * nothing) and the dead-letters surfaces print it, so each item's
 * free-text fields - `principle` and `scope` - are redacted here exactly
 * as `writeSignal` redacts them before disk; the letter is never the one
 * place the raw text survives.
 *
 * What redaction must not cost is the delivery identity: the dedup hash
 * a re-run would have been recognised by is computed over the RAW text,
 * so the raw text's hash is frozen beside each item (`dedup_hash`, the
 * same field name the delivered pages carry). Recovery prefers it, and a
 * later delivery of the real content still reads as delivered.
 *
 * The fields are enumerated rather than spread so a field added to
 * {@link ExtractedSignalItem} later cannot ride into the letter
 * unredacted: freezing it becomes a decision, not an accident.
 */
function frozenDeadLetterPayload(items: ReadonlyArray<ExtractedSignalItem>): DeadLetterPayload {
  return {
    items: items.map((item) => {
      const scope = item.scope?.trim();
      return {
        topic: item.topic,
        signal: item.signal,
        principle: redactRawOutput(item.principle),
        confidence: item.confidence,
        ...(scope !== undefined && scope !== "" ? { scope: redactRawOutput(scope) } : {}),
        ...(typeof item.source_turn === "string" && item.source_turn.trim() !== ""
          ? { source_turn: item.source_turn }
          : {}),
        dedup_hash: computeDedupHash({
          topic: item.topic,
          signal: item.signal,
          principle: item.principle,
          ...(scope !== undefined && scope !== "" ? { scope } : {}),
        }),
      };
    }),
  };
}

/** A plan's surveyed-turns hash: a SHA-256 digest in lowercase hex. */
const SURVEYED_TURNS_HASH_RE = /^[0-9a-f]{64}$/;

/**
 * Validate the returned payload and write the accepted items.
 *
 * Order is the deterministic extractor's, kept deliberately identical so
 * the two lanes cannot diverge on what reaches the inbox: capture boundary,
 * then per item dedup, then the durability gate, then staging, then the
 * write. A deduped item never reaches the durability gate, and a
 * durability-rejected item is counted and NAMED in
 * {@link CommitExtractedSignalsResult.rejected} - never a silent drop.
 */
export function commitExtractedSignals(
  vault: string,
  sessionId: string,
  payload: unknown,
  opts: CommitExtractedSignalsOptions,
): CommitExtractedSignalsResult {
  const trimmed = sessionId.trim();
  if (trimmed.length === 0) {
    throw new ExtractSignalsError("extract-signals needs a session reference");
  }
  const boundary = buildCaptureBoundary(vault);
  const boundaryDecision = boundary.sessionDecision(trimmed);
  if (boundaryDecision !== "capture") {
    throw new ExtractSignalsError(
      `session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,
    );
  }
  if (
    opts.surveyedTurnsHash !== undefined &&
    !SURVEYED_TURNS_HASH_RE.test(opts.surveyedTurnsHash)
  ) {
    throw new ExtractSignalsError(
      `surveyed_turns_hash must be the 64-character lowercase hex digest a plan printed, got '${opts.surveyedTurnsHash}'`,
    );
  }
  // Structure first, then the cross-item rules. Nothing is written until
  // both gates pass, so a payload with one bad item lands nothing.
  assertResponseShape(EXTRACTED_SIGNALS_SURFACE, EXTRACTED_SIGNALS_SHAPE, payload);
  assertResponseCheck(EXTRACTED_SIGNALS_SURFACE, payload);

  const items = (payload as { items: ReadonlyArray<ExtractedSignalItem> }).items;
  const dedup = opts.dedup ?? buildDedupIndex(vault);
  const denylist = opts.durabilityDenylist ?? resolveDurabilityDenylist();
  const writeApprovalEnabled = opts.writeApprovalEnabled ?? resolveWriteApprovalEnabled();
  const targetDir = writeApprovalEnabled ? brainDirsForWrite(vault).pending : undefined;

  const written: ExtractedSignalWrite[] = [];
  const rejected: ExtractedSignalRejection[] = [];
  let deduped = 0;

  for (const [index, item] of items.entries()) {
    const scope = item.scope?.trim();
    const hash = computeDedupHash({
      topic: item.topic,
      signal: item.signal,
      principle: item.principle,
      ...(scope ? { scope } : {}),
    });
    if (dedup.has(hash)) {
      deduped++;
      continue;
    }
    const verdict = classifyDurability(item.principle, { denylist });
    if (!verdict.durable) {
      rejected.push(Object.freeze({ topic: item.topic, reason: verdict.reason! }));
      continue;
    }
    // Per-item writes, so a failure here is a PARTIAL write: everything
    // before this item is on disk and stays there. It is reported as such
    // rather than propagating a bare cause with no accounting - the same
    // choice `ArchWriteError` makes for the architect tree.
    let res;
    try {
      res = writeSignal(
        vault,
        {
          topic: item.topic,
          signal: item.signal,
          agent: opts.agent,
          principle: item.principle,
          created_at: isoSecond(opts.now),
          date: isoDate(opts.now),
          slug: item.topic,
          ...(scope ? { scope } : {}),
          source: [`[[${trimmed}]]`],
          source_type: BRAIN_SIGNAL_SOURCE_TYPE.autoExtract,
          dedup_hash: hash,
          session_ref: trimmed,
          // The confidence the caller claimed rides onto the file: it is the
          // one thing separating two auto-extracted signals whose text reads
          // equally plausible, and the dream pass has no other way to see it.
          raw: `confidence: ${item.confidence}`,
        },
        targetDir !== undefined ? { targetDir } : {},
      );
    } catch (err) {
      // Everything from the failing item onward is unwritten. Some of the
      // items this call never reached might have deduped away had it got
      // to them - unknowable from here, and "not written" is true of all
      // of them either way, so they are NAMED rather than guessed at.
      const remaining = items.slice(index);
      const reconciliation = buildReconciliationReport({
        attempted: written.length + remaining.length,
        found: written.length,
        missing: remaining.map((unwritten, offset) =>
          unwrittenKey(index + offset, unwritten.topic),
        ),
      });
      throw new ExtractSignalsWriteError({
        topic: item.topic,
        written,
        remaining: remaining.length,
        reconciliation,
        cause: err,
        deadLetter: tryRecordDeadLetter(vault, {
          envelope: {
            lane: EXTRACT_SIGNALS_LANE,
            step: EXTRACT_SIGNALS_STEP,
            reference: trimmed,
            target: AUTO_EXTRACT_TARGET_DIR_REL,
          },
          report: reconciliation,
          firstError: err,
          // Schema 2 freezes the validated payload - the SAME payload the
          // error message tells the caller to re-run - so recovery can
          // re-commit it without a second model call. The items already
          // written ride along on purpose: a re-run lets the dedup index
          // recognize them, exactly as the message says. The freeze
          // redacts the free-text fields and keeps each item's raw dedup
          // hash, so the letter never becomes the one unredacted copy of
          // the payload and recovery still matches the original identity.
          payload: frozenDeadLetterPayload(items),
          now: opts.now,
        }),
      });
    }
    dedup.set(hash, { id: res.id, path: res.path });
    const sourceTurn = typeof item.source_turn === "string" ? item.source_turn.trim() : "";
    written.push(
      Object.freeze({
        id: res.id,
        path: res.path,
        topic: item.topic,
        ...(sourceTurn !== "" ? { sourceTurn } : {}),
      }),
    );
  }

  // Every commit is the ledger's write trigger: the outcome breakdown is
  // recorded over the hash of the session's current mined turns - the
  // material a plan issued now would offer, and the one the plan gate and
  // the census compare against. A yield record is the distilled state the
  // census reports, and it is NOT a tombstone - the session stays minable.
  // The zero-yield record IS the tombstone the plan gate suppresses on, so
  // it is written only over material a model was shown: the relayed plan
  // hash when there is one, else the one survey a plan recorded since the
  // last commit (session-mining-surveys.ts) - and only while the session
  // still holds that material. A thrown ExtractSignalsWriteError never
  // reaches here: a run that FAILED did not end, so it records nothing. A
  // session with no mined material records nothing either, and every
  // withheld record says why: failing open to a re-mine is the safe
  // direction.
  const staged = writeApprovalEnabled ? written.length : 0;
  const zeroYield = written.length === 0;
  let miningLedgerFailure: string | undefined;
  let currentHash: string | null = null;
  const raw = (err: unknown): string => (err instanceof Error ? err.message : String(err));
  const surveyText = (err: unknown): string =>
    opts.closeHostErrors === true ? closedSurveyFailure(err) : raw(err);
  // Read, not consumed: the survey is cleared below only once this
  // commit's outcome is final, so a commit that fails transiently (the
  // turns unreadable, the ledger write refused) leaves it for the retry.
  let surveyed: ReadonlyArray<string> = [];
  let surveyFailure: string | undefined;
  try {
    surveyed = readMiningSurvey(vault, trimmed);
  } catch (err) {
    surveyFailure = surveyText(err);
  }
  let outcomeFinal = true;
  try {
    const mined = selectMinedTurns(boundary, listSessionRawTurns(vault, trimmed));
    currentHash = mined.length === 0 ? null : minedTurnsHash(mined);
  } catch (err) {
    miningLedgerFailure = raw(err);
    outcomeFinal = false;
  }
  if (currentHash !== null && opts.surveyedTurnsHash !== undefined) {
    if (opts.surveyedTurnsHash !== currentHash) {
      miningLedgerFailure =
        `the session's turns no longer match the surveyed_turns_hash this commit answers ` +
        `(${opts.surveyedTurnsHash}); no mining record was written, so the session stays minable`;
      currentHash = null;
    }
  } else if (currentHash !== null && zeroYield) {
    const withheld = withheldTombstoneReason(surveyed, surveyFailure, currentHash);
    if (withheld !== null) {
      miningLedgerFailure = withheld;
      currentHash = null;
    }
  }
  if (currentHash !== null) {
    try {
      recordSessionMiningOutcome(
        vault,
        {
          sessionId: trimmed,
          minedContentHash: currentHash,
          outcome: {
            written: written.length,
            staged,
            deduped,
            durabilityRejected: rejected.length,
          },
        },
        opts.now,
      );
    } catch (err) {
      // The signals already on disk are safe; throwing would misreport the
      // commit as failed, and dropping the error would leave a ledger that
      // silently stopped being written. Name it instead - the surfaces
      // carry it verbatim.
      miningLedgerFailure = raw(err);
      outcomeFinal = false;
    }
  }
  if (outcomeFinal && surveyFailure === undefined) {
    try {
      clearMiningSurvey(vault, trimmed, surveyed);
    } catch (err) {
      // The outcome stands; a survey left behind can only attest material
      // a plan really offered, so it is named rather than thrown.
      miningLedgerFailure ??= `the answered plan survey could not be cleared (${surveyText(err)})`;
    }
  }

  return Object.freeze({
    sessionId: trimmed,
    written: Object.freeze(written),
    staged,
    deduped,
    durabilityRejected: rejected.length,
    rejected: Object.freeze(rejected),
    reconciliation: buildReconciliationReport({
      attempted: written.length,
      found: written.length,
      missing: [],
    }),
    ...(miningLedgerFailure !== undefined ? { miningLedgerFailure } : {}),
  });
}

/**
 * Why a zero-yield commit that relayed no plan hash may not write its
 * tombstone, or `null` when it may: exactly one distinct material was
 * surveyed since the last commit and the session still holds it. Anything
 * else means the commit cannot attest that a model saw the turns it would
 * suppress.
 */
function withheldTombstoneReason(
  surveyed: ReadonlyArray<string>,
  surveyFailure: string | undefined,
  currentHash: string,
): string | null {
  const tail =
    "no zero-yield record was written, so the session stays minable; re-plan it, or relay the " +
    "plan's surveyed_turns_hash with the commit";
  if (surveyFailure !== undefined) {
    return `the plan survey of this session could not be read (${surveyFailure}); ${tail}`;
  }
  if (surveyed.length === 0) {
    return `no plan of this session was recorded since its last commit, so this commit cannot attest which turns a model saw; ${tail}`;
  }
  if (surveyed.length > 1) {
    return `${surveyed.length} plans over different turns of this session are pending, so this commit cannot tell which one it answers; ${tail}`;
  }
  if (surveyed[0] !== currentHash) {
    return `the session's turns changed since the plan this commit answers (${surveyed[0]}); ${tail}`;
  }
  return null;
}

/**
 * The key a missing item is NAMED by: its payload index and its topic.
 * The index carries the position and the topic carries the meaning,
 * which is what an operator matches against the payload they are about to
 * re-run.
 */
function unwrittenKey(index: number, topic: string): string {
  return `items[${index}]:${topic}`;
}