Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Model-mined session signals (salience-lifecycle-enrichment, unit 2,
* t_1dace26d).
*
* `fact-extract.ts` mines a session turn with regexes, which is why it can
* run inside the live capture hook: it costs nothing and knows no language.
* The price is that it only ever recognises structure - a URL, an address,
* a unit-bound number - so every stated preference in a transcript is
* invisible to it. This module is the batch counterpart: it hands the
* calling agent the user turns of an ALREADY-IMPORTED session and asks for
* the taste signals in them. OSB still calls no model; it owns the turns it
* offers, the limits the answer must satisfy, and the write.
*
* Two phases, diarization-shaped:
*
* 1. {@link planExtractSignals} - read-only. Resolves the session's
* imported turns, keeps the USER ones (the import path's rule at
* `sessions/import.ts`: bare assistant output is never auto-extracted),
* and returns a report carrying exactly one needs-llm-step envelope.
* 2. {@link commitExtractedSignals} - validates the returned payload
* structurally ({@link EXTRACTED_SIGNALS_SHAPE}) and semantically (the
* per-session cap and the per-item confidence floor, registered on the
* semantic-check registry because neither is expressible in a
* descriptor), then routes the accepted items through the same helper
* chain the deterministic extractor uses.
*
* Everything written here is SPECULATIVE. The items land in `Brain/inbox/`
* as `source_type: auto_extract` signals - a member of its own so a
* model-authored signal is distinguishable from a regex-authored one
* forever - and the inbox trial lane plus the dream pass remains the only
* route to a confirmed preference. Nothing here writes a preference.
*
* The commit is also the mining ledger's write side (session-mining-ledger.ts):
* every successful commit records its outcome breakdown over the session's
* current mined turns. A zero-yield record (nothing written, nothing
* staged) is the tombstone the plan gate suppresses on, so it is written
* only over material a plan put in front of a model: the hash the caller
* relays, or else the one survey the plan surfaces recorded
* (session-mining-surveys.ts). A yield record is the distilled state the
* mining census reports. A partial write never records - a run that FAILED
* did not end.
*
* Precision beats recall, as it does in `fact-extract.ts`: the cap and the
* floor are refusals, not filters. A payload over either limit is rejected
* whole and named, because an over-eager mine that silently kept its best
* twelve would teach the caller nothing about what it asked for.
*/
import { posix } from "node:path";
import { redactRawOutput } from "../redactor.ts";
import { buildReconciliationReport, type ReconciliationReport } from "../reconciliation-report.ts";
import {
buildCaptureBoundary,
type CaptureBoundary,
type SessionCaptureDecision,
} from "./capture-boundary.ts";
import {
tryRecordDeadLetter,
type DeadLetterLane,
type DeadLetterOutcome,
type DeadLetterPayload,
} from "./dead-letter.ts";
import { buildDedupIndex, computeDedupHash, type DedupIndexEntry } from "./dedup-hash.ts";
import { classifyDurability, resolveDurabilityDenylist } from "./gates/durability.ts";
import { buildNeedsLlmStep, type NeedsLlmStep } from "./llm-step.ts";
import { brainDirsForWrite } from "./paths.ts";
import { resolveWriteApprovalEnabled } from "./pending.ts";
import {
assertResponseCheck,
registerResponseCheck,
SEMANTIC_VIOLATION_CODES,
semanticViolation,
type SemanticViolation,
} from "./response-checks.ts";
import {
assertResponseShape,
EXTRACTED_SIGNALS_SHAPE,
EXTRACTED_SIGNALS_SURFACE,
SHAPE_ROOT_PATH,
} from "./response-shape.ts";
import { listSessionRawTurns, type ExpandedRawTurn } from "./session-recall.ts";
import { minedTurnsHash, recordSessionMiningOutcome } from "./session-mining-ledger.ts";
import {
clearMiningSurvey,
closedSurveyFailure,
readMiningSurvey,
} from "./session-mining-surveys.ts";
import { writeSignal } from "./signal.ts";
import { isoDate, isoSecond } from "./time.ts";
import { BRAIN_SIGNAL_SOURCE_TYPE, type BrainSignalSign } from "./types.ts";
/** Vault-relative directory an accepted auto-extracted signal lands in. */
export const AUTO_EXTRACT_TARGET_DIR_REL = "Brain/inbox";
/** The needs-llm-step step name for the deferred signal mining. */
export const EXTRACT_SIGNALS_STEP = "extract-signals";
/**
* This lane's dead-letter name. It is the ONE multi-artifact commit lane
* in the tree - one file per mined item from one validated payload - and
* so the only one that can leave a partial write behind.
*/
const EXTRACT_SIGNALS_LANE: DeadLetterLane = "extract-signals";
/**
* Most signals one session may yield. A transcript that genuinely states
* more than this many durable preferences is a transcript worth reading by
* hand; past the cap the mine is guessing, and guesses at inbox scale cost
* more to retire than they ever pay back.
*/
export const AUTO_EXTRACT_PER_SESSION_CAP = 12;
/**
* Lowest confidence an item may carry. The floor sits above a coin flip on
* purpose: an item the caller is not clearly sure of is exactly the item
* whose retirement will cost an operator a review.
*/
export const AUTO_EXTRACT_CONFIDENCE_FLOOR = 0.6;
/** The role whose turns are mining material - the import path's rule. */
const MINED_TURN_ROLE = "user";
/** Per-turn text budget in the prompt, so one huge turn cannot crowd out the rest. */
export const PROMPT_TURN_TEXT_MAX = 2000;
/** A signal extraction could not proceed, and the reason is in the message. */
export class ExtractSignalsError extends Error {
constructor(message: string) {
super(message);
this.name = "ExtractSignalsError";
}
}
/**
* A write failed PART WAY through the accepted items.
*
* The commit writes one file per item, so a failure on the fourth leaves
* three signals on disk. The house precedent for that state is
* `ArchWriteError` (`architect/generate.ts`): a partial write reports how
* far it got rather than throwing a bare cause and leaving the caller to
* guess. This is that error for this lane - the ids already written are
* ON it, so an operator can find them, and re-running the same payload is
* safe because the dedup index recognises every one of them.
*
* Since nothing-writes-silently it carries two further things. The first
* is {@link reconciliation}: the same `attempted` / `found` / `missing`
* report shape the import read-back census and the embedder audit use, so
* this lane's accounting is not a fourth dialect of the same three
* numbers. The second is {@link deadLetter}: this error's own accounting
* lives exactly as long as the response carrying it, and a caller that
* drops the response must not be the only record that a partial write
* happened - so the accounting is ALSO on disk, and this field says
* where, or why it could not be.
*
* Not an {@link ExtractSignalsError}: that class is the caller's fault
* and the surfaces report it as such. A filesystem that refused a write
* is not.
*/
export class ExtractSignalsWriteError extends Error {
/** The item whose write failed. */
readonly topic: string;
/** Signals whose bytes are already on disk, in write order. */
readonly written: ReadonlyArray<ExtractedSignalWrite>;
/** Payload items this call never reached, the failing one included. */
readonly remaining: number;
/** Attempted / found / missing for this commit, missing keys named. */
readonly reconciliation: ReconciliationReport;
/** The error that stopped the commit, kept as thrown. */
readonly firstError: unknown;
/** Where the durable record of this partial write landed, or why not. */
readonly deadLetter: DeadLetterOutcome;
constructor(params: {
readonly topic: string;
readonly written: ReadonlyArray<ExtractedSignalWrite>;
readonly remaining: number;
readonly reconciliation: ReconciliationReport;
readonly cause: unknown;
readonly deadLetter: DeadLetterOutcome;
}) {
const { topic, written, remaining, reconciliation, cause, deadLetter } = params;
super(
`failed to write extracted signal ${JSON.stringify(topic)}: ` +
`${cause instanceof Error ? cause.message : String(cause)} - ` +
`attempted ${reconciliation.attempted}, written ${reconciliation.found}, ` +
`failed ${reconciliation.missing.length} (${reconciliation.missing.join(", ")}); ` +
`${written.length} signal(s) are already on disk ` +
`(${written.length === 0 ? "none" : written.map((w) => w.id).join(", ")}) and ` +
`${remaining} payload item(s) were not reached; ${describeDeadLetter(deadLetter)}; ` +
"fix the cause and re-run the same payload, which dedups what is already there",
{ cause },
);
this.name = "ExtractSignalsWriteError";
this.topic = topic;
this.written = Object.freeze([...written]);
this.remaining = remaining;
this.reconciliation = reconciliation;
this.firstError = cause;
this.deadLetter = deadLetter;
}
}
/** The dead-letter half of the message above, either way it went. */
function describeDeadLetter(outcome: DeadLetterOutcome): string {
return outcome.recorded
? `a dead letter recording this is at ${outcome.path} (${outcome.id}); ` +
"'o2b brain dead-letters' classifies it and names the recovery exit " +
"(diagnostic code extract-signals-dead-letter)"
: `NO dead letter could be recorded for it (${outcome.reason})`;
}
/** One user turn offered to the caller as mining material. */
export interface MinedTurn {
readonly turnId: string;
readonly text: string;
/**
* The turn's stored timestamp, verbatim; "" when the source has none.
* Never a formatted `Date` and never the plan's clock: the caller grounds
* a relative time bound against it, so an invented value would ground it
* against the wrong day.
*/
readonly timestamp: string;
}
/** Phase-one report: what will be mined, and the one envelope that mines it. */
export interface SignalExtractionPlan {
readonly sessionId: string;
readonly generatedAt: string;
/** Capture-boundary verdict for the session; always `capture` here. */
readonly boundaryDecision: SessionCaptureDecision;
/** Imported turns of every role, before the user filter. */
readonly turnsScanned: number;
/** The user turns the prompt embeds, in record order. */
readonly turnsMined: ReadonlyArray<MinedTurn>;
/**
* {@link minedTurnsHash} of {@link turnsMined}: the material this plan
* put in front of the caller. The commit that answers this plan may pass
* it back as {@link CommitExtractedSignalsOptions.surveyedTurnsHash} to
* check the session did not change in between. Without the relay the
* commit reads the survey the plan surfaces recorded for the session
* instead.
*/
readonly surveyedTurnsHash: string;
/** The limits the payload will be held to, echoed so the caller can read them. */
readonly cap: number;
readonly confidenceFloor: number;
readonly llmStep: NeedsLlmStep;
}
/** One mined taste signal, as the calling agent returns it. */
export interface ExtractedSignalItem {
readonly topic: string;
readonly signal: BrainSignalSign;
readonly principle: string;
readonly confidence: number;
readonly scope?: string;
/**
* Optional plan turn id the item was mined from. Used only to measure
* the decision-model turn pre-filter; an item without it behaves as
* before.
*/
readonly source_turn?: string;
}
/** One item the durability gate refused, named rather than dropped. */
export interface ExtractedSignalRejection {
readonly topic: string;
readonly reason: string;
}
/** One item that reached disk, with the coordinates it landed at. */
export interface ExtractedSignalWrite {
readonly id: string;
readonly path: string;
readonly topic: string;
/** The item's `source_turn`, when it carried a non-empty one. */
readonly sourceTurn?: string;
}
export interface CommitExtractedSignalsResult {
readonly sessionId: string;
readonly written: ReadonlyArray<ExtractedSignalWrite>;
/**
* How many of {@link written} were STAGED into `Brain/pending/` rather
* than written straight to the inbox. Zero when write approval is off.
*/
readonly staged: number;
readonly deduped: number;
/** Items the durability gate refused; each is named in {@link rejected}. */
readonly durabilityRejected: number;
readonly rejected: ReadonlyArray<ExtractedSignalRejection>;
/**
* Write accounting in the wave's shared vocabulary: `attempted` is the
* items that reached the write call, `found` the ones whose bytes
* landed, `missing` the keys of the rest. On this path `missing` is
* always empty - a failure throws {@link ExtractSignalsWriteError},
* which carries the same shape with the gap named.
*
* A deduped or durability-rejected item is NOT attempted: both are
* deliberate refusals this result already names in {@link deduped} and
* {@link rejected}, and counting a refusal the lane made as a write it
* lost would misreport the one number an operator reads first.
*/
readonly reconciliation: ReconciliationReport;
/**
* Present only when the commit itself SUCCEEDED but recording the mining
* outcome failed (session-mining-ledger.ts). The signals this result
* reports are already on disk, so throwing would misreport a successful
* write as a failed one - and swallowing the failure would hide a ledger
* that is no longer trustworthy, so the error message rides here by
* name, and on both surface outputs.
*/
readonly miningLedgerFailure?: string;
}
export interface PlanExtractSignalsOptions {
readonly now: Date;
/**
* Add the optional `source_turn` item field to the envelope's schema
* hints. Set only by the decision-model pre-filter path (use not
* `off`), which measures regret from it; absent, the envelope is
* exactly as it has always been.
*/
readonly sourceTurnHint?: boolean;
}
export interface CommitExtractedSignalsOptions {
readonly agent: string;
readonly now: Date;
/**
* Pre-compiled durability denylist. When omitted it is resolved from the
* plugin config, mirroring `routeExtractedFacts`; tests inject it so the
* gate is exercised without the default config path.
*/
readonly durabilityDenylist?: ReadonlyArray<RegExp>;
/** Write-approval toggle; resolved from config when omitted. */
readonly writeApprovalEnabled?: boolean;
/** Shared dedup index; built from the vault when omitted. */
readonly dedup?: Map<string, DedupIndexEntry>;
/**
* The answered plan's {@link SignalExtractionPlan.surveyedTurnsHash}: the
* hash of the turns the caller was actually shown. Optional: the commit
* records over the hash of the session's CURRENT mined turns, and a
* zero-yield record (the tombstone) additionally needs evidence that a
* model saw exactly that material - this relay, or else the single
* pending survey the plan surfaces recorded, so the documented flow
* (plan, answer the envelope, commit the items) writes its tombstone
* without relaying anything. When the hash IS relayed it is a check: it
* must be a SHA-256 hex digest (anything else refuses the commit before
* a write), and when it differs from the current material - an import
* landed between plan and commit, or the hash came from another plan -
* nothing is recorded and the commit names that as its mining ledger
* failure. Without a relay, a survey that differs from the current
* material, several pending surveys, or none withholds the tombstone and
* is named the same way; a yield record is recorded regardless.
*/
readonly surveyedTurnsHash?: string;
/**
* Name a pending-survey store failure in `miningLedgerFailure` by its
* store and errno code only, never by the raised message, which carries
* the absolute host path. Set by surfaces that answer a caller below
* local reach.
*/
readonly closeHostErrors?: boolean;
}
// ----- Semantic rules the descriptor language cannot express ---------------
/**
* Cap, floor, topic uniqueness and interval sanity, registered once at
* module scope. Each reads across the payload or inside a value rather
* than one value at a time - the cap counts the whole list, the floor
* compares a number to a limit the descriptor cannot name, uniqueness
* compares items with each other, and an interval compares the two dates
* inside one principle - so they belong here rather than in
* {@link EXTRACTED_SIGNALS_SHAPE}.
*
* Registration is a module side effect on purpose: the registry is
* fail-closed, so a lane that forgot to register refuses its own writes
* instead of performing them unchecked.
*/
registerResponseCheck(EXTRACTED_SIGNALS_SURFACE, (payload) => {
const violations: SemanticViolation[] = [];
const items = readItems(payload);
if (items === null) return violations;
if (items.length > AUTO_EXTRACT_PER_SESSION_CAP) {
violations.push(
semanticViolation(
SEMANTIC_VIOLATION_CODES.threshold,
`${SHAPE_ROOT_PATH}.items`,
`at most ${AUTO_EXTRACT_PER_SESSION_CAP} signals may be mined from one session; got ${items.length}`,
),
);
}
items.forEach((item, index) => {
const confidence = (item as { confidence?: unknown }).confidence;
if (typeof confidence !== "number") return;
if (confidence < AUTO_EXTRACT_CONFIDENCE_FLOOR) {
violations.push(
semanticViolation(
SEMANTIC_VIOLATION_CODES.threshold,
`${SHAPE_ROOT_PATH}.items[${index}].confidence`,
`confidence must be at least ${AUTO_EXTRACT_CONFIDENCE_FLOOR}; got ${confidence}`,
),
);
}
});
violations.push(...duplicateTopicViolations(items));
violations.push(...intervalViolations(items));
return violations;
});
/**
* An ISO interval inside a principle, the form the dream pass turns into
* a `[valid_from, valid_until)` window. Global so every interval in one
* principle is checked, not only the first.
*/
const PRINCIPLE_INTERVAL_RE = /\b(\d{4}-\d{2}-\d{2})\/(\d{4}-\d{2}-\d{2})\b/g;
/**
* One violation per interval in a principle that names an impossible
* calendar date or ends on or before it starts. Either one yields a window
* that is never active, and nothing downstream would say why, so the
* payload is refused here where the caller can still correct it.
*/
function intervalViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
const violations: SemanticViolation[] = [];
items.forEach((item, index) => {
const principle = (item as { principle?: unknown }).principle;
if (typeof principle !== "string") return;
for (const [interval, start, end] of principle.matchAll(PRINCIPLE_INTERVAL_RE)) {
const impossible = [start!, end!].find((date) => !isCalendarDate(date));
const message =
impossible !== undefined
? `interval ${interval} names ${impossible}, which is not a calendar date`
: end! <= start!
? `interval ${interval} must end after it starts; its end date is exclusive, the day after the last day the rule holds`
: null;
if (message === null) continue;
violations.push(
semanticViolation(
SEMANTIC_VIOLATION_CODES.threshold,
`${SHAPE_ROOT_PATH}.items[${index}].principle`,
message,
),
);
}
});
return violations;
}
/** Whether a `YYYY-MM-DD` string names a real day, so `2026-02-30` does not. */
function isCalendarDate(date: string): boolean {
const parsed = new Date(`${date}T00:00:00Z`);
return !Number.isNaN(parsed.getTime()) && parsed.toISOString().slice(0, 10) === date;
}
/**
* One violation per item that repeats an earlier item's topic, naming both
* indices. The comparison is exact: two items under one topic are one rule
* stated twice or two rules filed under one name, and the lane cannot tell
* which - so the payload is refused whole rather than one of them kept by a
* guess. A fuzzy comparison of principles is deliberately absent: two short
* rules of opposite meaning read alike, and one false refusal here would
* drop every valid signal in the run.
*/
function duplicateTopicViolations(items: ReadonlyArray<unknown>): SemanticViolation[] {
const violations: SemanticViolation[] = [];
const firstIndex = new Map<string, number>();
items.forEach((item, index) => {
const topic = (item as { topic?: unknown }).topic;
if (typeof topic !== "string") return;
const first = firstIndex.get(topic);
if (first === undefined) {
firstIndex.set(topic, index);
return;
}
violations.push(
semanticViolation(
SEMANTIC_VIOLATION_CODES.crossItem,
`${SHAPE_ROOT_PATH}.items[${index}].topic`,
`items[${first}] and items[${index}] share topic ${JSON.stringify(topic)}`,
),
);
});
return violations;
}
/**
* The item list, or null when the payload is not shaped like one. The
* shape layer has already refused that case by the time the check runs;
* returning null keeps the check pure and total rather than throwing from
* inside a validator whose contract says it never throws.
*/
function readItems(payload: unknown): ReadonlyArray<unknown> | null {
if (typeof payload !== "object" || payload === null) return null;
const items = (payload as { items?: unknown }).items;
return Array.isArray(items) ? items : null;
}
// ----- Phase one: what to mine ---------------------------------------------
/**
* The user turns the prompt embeds, in record order: the import path's
* role rule, the message-level capture boundary, and blank text dropped.
* Shared by the plan (which offers them) and the mining census (whose
* `minedTurnMaterial` must hash exactly what the plan would mine), so the
* two can never disagree on what a run covered.
*/
export function selectMinedTurns(
boundary: CaptureBoundary,
turns: ReadonlyArray<ExpandedRawTurn>,
): MinedTurn[] {
const mined: MinedTurn[] = [];
for (const turn of turns) {
if (turn.role !== MINED_TURN_ROLE) continue;
// Message-level boundary, mirroring the import path: suppressed text
// never reaches extraction, so it never reaches the prompt either.
if (boundary.suppressMessage(turn.text)) continue;
const text = turn.text.trim();
if (text.length === 0) continue;
mined.push(Object.freeze({ turnId: turn.turn_id, text, timestamp: turn.timestamp }));
}
return mined;
}
/**
* Assemble the mining plan for one imported session. Read-only; writes
* nothing and calls no model.
*
* Three conditions are refusals rather than empty reports, because each
* one means the caller asked for something this vault cannot answer:
* a session the capture boundary ignores, a session with no imported turns
* at all, and a session whose turns are all non-user. Reporting any of them
* as "mined nothing" would read as a clean run over a barren transcript.
*/
export function planExtractSignals(
vault: string,
sessionId: string,
opts: PlanExtractSignalsOptions,
): SignalExtractionPlan {
const trimmed = sessionId.trim();
if (trimmed.length === 0) {
throw new ExtractSignalsError("extract-signals needs a session reference");
}
const boundary = buildCaptureBoundary(vault);
const boundaryDecision = boundary.sessionDecision(trimmed);
if (boundaryDecision !== "capture") {
throw new ExtractSignalsError(
`session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,
);
}
const turns = listSessionRawTurns(vault, trimmed);
if (turns.length === 0) {
throw new ExtractSignalsError(
`session ${trimmed} has no imported turns; import it first with 'o2b brain import-session --recall'`,
);
}
const mined = selectMinedTurns(boundary, turns);
if (mined.length === 0) {
throw new ExtractSignalsError(
`session ${trimmed} has ${turns.length} imported turn(s) but none authored by the user; there is nothing to mine`,
);
}
return Object.freeze({
sessionId: trimmed,
generatedAt: opts.now.toISOString(),
boundaryDecision,
turnsScanned: turns.length,
turnsMined: Object.freeze(mined),
surveyedTurnsHash: minedTurnsHash(mined),
cap: AUTO_EXTRACT_PER_SESSION_CAP,
confidenceFloor: AUTO_EXTRACT_CONFIDENCE_FLOOR,
llmStep: buildMiningStep(trimmed, mined, { sourceTurnHint: opts.sourceTurnHint === true }),
});
}
/**
* Hygiene rules the mined items are held to, in the instruction.
*
* Each names a CATEGORY and never an example word: the operator may write
* in any language, and a kernel that listed one language's greetings or
* date words would mine that language better than every other. Time is
* grounded the same way - the caller resolves a relative bound against the
* turn's timestamp and writes it as ISO, which the dream pass already turns
* into a validity window, so no date parser lives here.
*/
const EXTRACT_HYGIENE_RULES: ReadonlyArray<string> = Object.freeze([
"Write any time bound a rule carries as an ISO 8601 date or interval (YYYY-MM-DD, or " +
"YYYY-MM-DD/YYYY-MM-DD), resolving a relative bound against the timestamp of the turn " +
"that stated it; an interval's end date is exclusive, so write the day after the last " +
"day the rule holds, and an end that is not after the start refuses the whole payload; " +
"write a bound that only ends as an interval from that turn's date, " +
"because a lone date reads as the day the rule starts; a rule with no time bound stays " +
"undated.",
"Skip conversational mechanics - greetings, thanks, acknowledgements, requests to go on, " +
"and directions about the flow of this session - because they state no rule.",
"Return one rule per item: a turn that states two rules yields two items with distinct topics.",
"Keep every condition a rule was stated with - when, where, for which project or kind of " +
"work - inside its principle; never widen a conditional rule into an unconditional one.",
"Drop restatements: a rule the operator states more than once is returned once, under one " +
"topic, because two items sharing a topic refuse the whole payload.",
]);
/**
* The one envelope. The turns ride inside the prompt rather than beside it
* because the receiving agent is answering a question about exactly these
* turns and nothing else; a prompt that named a session id would invite the
* caller to go and read more of it.
*/
export function buildMiningStep(
sessionId: string,
mined: ReadonlyArray<MinedTurn>,
opts: { readonly sourceTurnHint?: boolean } = {},
): NeedsLlmStep {
const transcript = mined
.map((turn) => `${transcriptLabel(turn)} ${turn.text.slice(0, PROMPT_TURN_TEXT_MAX)}`)
.join("\n");
return buildNeedsLlmStep({
step: EXTRACT_SIGNALS_STEP,
prompt:
`Mine durable taste signals from the ${mined.length} user turn(s) of session ${sessionId} below. ` +
"A taste signal is a rule the operator stated about how work should be done - a preference, a " +
"correction, a prohibition - not a fact, a task, or a one-off instruction about this session. " +
`Return at most ${AUTO_EXTRACT_PER_SESSION_CAP} items, each with a confidence of at least ` +
`${AUTO_EXTRACT_CONFIDENCE_FLOOR}; omit anything you are less sure of rather than lowering the ` +
"number, because an item below the floor refuses the whole payload. " +
EXTRACT_HYGIENE_RULES.join(" ") +
"\n\n" +
transcript,
schema_hints: [
'payload: { "items": [ { "topic", "signal", "principle", "confidence", "scope"? } ] }',
"topic: stable kebab-slug naming the rule, unique within the payload",
"signal: 'positive' when the principle is the rule to follow, 'negative' when it is what to avoid",
"principle: one imperative line, in the language the operator used, with any time bound " +
"written as an ISO 8601 date or interval",
`confidence: number in [${AUTO_EXTRACT_CONFIDENCE_FLOOR}, 1]`,
`items: at most ${AUTO_EXTRACT_PER_SESSION_CAP} entries`,
...(opts.sourceTurnHint === true
? ["source_turn: optional, the bracketed id of the turn the rule was stated in"]
: []),
],
target_path: posix.normalize(AUTO_EXTRACT_TARGET_DIR_REL),
});
}
/** `[turnId @ timestamp]`, or `[turnId]` when the turn has no timestamp. */
function transcriptLabel(turn: MinedTurn): string {
return turn.timestamp === "" ? `[${turn.turnId}]` : `[${turn.turnId} @ ${turn.timestamp}]`;
}
// ----- Phase two: what the caller mined ------------------------------------
/**
* The payload as it is FROZEN into a dead letter. The letter sits in
* `.open-second-brain/dead-letters/` forever (the retention rule deletes
* nothing) and the dead-letters surfaces print it, so each item's
* free-text fields - `principle` and `scope` - are redacted here exactly
* as `writeSignal` redacts them before disk; the letter is never the one
* place the raw text survives.
*
* What redaction must not cost is the delivery identity: the dedup hash
* a re-run would have been recognised by is computed over the RAW text,
* so the raw text's hash is frozen beside each item (`dedup_hash`, the
* same field name the delivered pages carry). Recovery prefers it, and a
* later delivery of the real content still reads as delivered.
*
* The fields are enumerated rather than spread so a field added to
* {@link ExtractedSignalItem} later cannot ride into the letter
* unredacted: freezing it becomes a decision, not an accident.
*/
function frozenDeadLetterPayload(items: ReadonlyArray<ExtractedSignalItem>): DeadLetterPayload {
return {
items: items.map((item) => {
const scope = item.scope?.trim();
return {
topic: item.topic,
signal: item.signal,
principle: redactRawOutput(item.principle),
confidence: item.confidence,
...(scope !== undefined && scope !== "" ? { scope: redactRawOutput(scope) } : {}),
...(typeof item.source_turn === "string" && item.source_turn.trim() !== ""
? { source_turn: item.source_turn }
: {}),
dedup_hash: computeDedupHash({
topic: item.topic,
signal: item.signal,
principle: item.principle,
...(scope !== undefined && scope !== "" ? { scope } : {}),
}),
};
}),
};
}
/** A plan's surveyed-turns hash: a SHA-256 digest in lowercase hex. */
const SURVEYED_TURNS_HASH_RE = /^[0-9a-f]{64}$/;
/**
* Validate the returned payload and write the accepted items.
*
* Order is the deterministic extractor's, kept deliberately identical so
* the two lanes cannot diverge on what reaches the inbox: capture boundary,
* then per item dedup, then the durability gate, then staging, then the
* write. A deduped item never reaches the durability gate, and a
* durability-rejected item is counted and NAMED in
* {@link CommitExtractedSignalsResult.rejected} - never a silent drop.
*/
export function commitExtractedSignals(
vault: string,
sessionId: string,
payload: unknown,
opts: CommitExtractedSignalsOptions,
): CommitExtractedSignalsResult {
const trimmed = sessionId.trim();
if (trimmed.length === 0) {
throw new ExtractSignalsError("extract-signals needs a session reference");
}
const boundary = buildCaptureBoundary(vault);
const boundaryDecision = boundary.sessionDecision(trimmed);
if (boundaryDecision !== "capture") {
throw new ExtractSignalsError(
`session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,
);
}
if (
opts.surveyedTurnsHash !== undefined &&
!SURVEYED_TURNS_HASH_RE.test(opts.surveyedTurnsHash)
) {
throw new ExtractSignalsError(
`surveyed_turns_hash must be the 64-character lowercase hex digest a plan printed, got '${opts.surveyedTurnsHash}'`,
);
}
// Structure first, then the cross-item rules. Nothing is written until
// both gates pass, so a payload with one bad item lands nothing.
assertResponseShape(EXTRACTED_SIGNALS_SURFACE, EXTRACTED_SIGNALS_SHAPE, payload);
assertResponseCheck(EXTRACTED_SIGNALS_SURFACE, payload);
const items = (payload as { items: ReadonlyArray<ExtractedSignalItem> }).items;
const dedup = opts.dedup ?? buildDedupIndex(vault);
const denylist = opts.durabilityDenylist ?? resolveDurabilityDenylist();
const writeApprovalEnabled = opts.writeApprovalEnabled ?? resolveWriteApprovalEnabled();
const targetDir = writeApprovalEnabled ? brainDirsForWrite(vault).pending : undefined;
const written: ExtractedSignalWrite[] = [];
const rejected: ExtractedSignalRejection[] = [];
let deduped = 0;
for (const [index, item] of items.entries()) {
const scope = item.scope?.trim();
const hash = computeDedupHash({
topic: item.topic,
signal: item.signal,
principle: item.principle,
...(scope ? { scope } : {}),
});
if (dedup.has(hash)) {
deduped++;
continue;
}
const verdict = classifyDurability(item.principle, { denylist });
if (!verdict.durable) {
rejected.push(Object.freeze({ topic: item.topic, reason: verdict.reason! }));
continue;
}
// Per-item writes, so a failure here is a PARTIAL write: everything
// before this item is on disk and stays there. It is reported as such
// rather than propagating a bare cause with no accounting - the same
// choice `ArchWriteError` makes for the architect tree.
let res;
try {
res = writeSignal(
vault,
{
topic: item.topic,
signal: item.signal,
agent: opts.agent,
principle: item.principle,
created_at: isoSecond(opts.now),
date: isoDate(opts.now),
slug: item.topic,
...(scope ? { scope } : {}),
source: [`[[${trimmed}]]`],
source_type: BRAIN_SIGNAL_SOURCE_TYPE.autoExtract,
dedup_hash: hash,
session_ref: trimmed,
// The confidence the caller claimed rides onto the file: it is the
// one thing separating two auto-extracted signals whose text reads
// equally plausible, and the dream pass has no other way to see it.
raw: `confidence: ${item.confidence}`,
},
targetDir !== undefined ? { targetDir } : {},
);
} catch (err) {
// Everything from the failing item onward is unwritten. Some of the
// items this call never reached might have deduped away had it got
// to them - unknowable from here, and "not written" is true of all
// of them either way, so they are NAMED rather than guessed at.
const remaining = items.slice(index);
const reconciliation = buildReconciliationReport({
attempted: written.length + remaining.length,
found: written.length,
missing: remaining.map((unwritten, offset) =>
unwrittenKey(index + offset, unwritten.topic),
),
});
throw new ExtractSignalsWriteError({
topic: item.topic,
written,
remaining: remaining.length,
reconciliation,
cause: err,
deadLetter: tryRecordDeadLetter(vault, {
envelope: {
lane: EXTRACT_SIGNALS_LANE,
step: EXTRACT_SIGNALS_STEP,
reference: trimmed,
target: AUTO_EXTRACT_TARGET_DIR_REL,
},
report: reconciliation,
firstError: err,
// Schema 2 freezes the validated payload - the SAME payload the
// error message tells the caller to re-run - so recovery can
// re-commit it without a second model call. The items already
// written ride along on purpose: a re-run lets the dedup index
// recognize them, exactly as the message says. The freeze
// redacts the free-text fields and keeps each item's raw dedup
// hash, so the letter never becomes the one unredacted copy of
// the payload and recovery still matches the original identity.
payload: frozenDeadLetterPayload(items),
now: opts.now,
}),
});
}
dedup.set(hash, { id: res.id, path: res.path });
const sourceTurn = typeof item.source_turn === "string" ? item.source_turn.trim() : "";
written.push(
Object.freeze({
id: res.id,
path: res.path,
topic: item.topic,
...(sourceTurn !== "" ? { sourceTurn } : {}),
}),
);
}
// Every commit is the ledger's write trigger: the outcome breakdown is
// recorded over the hash of the session's current mined turns - the
// material a plan issued now would offer, and the one the plan gate and
// the census compare against. A yield record is the distilled state the
// census reports, and it is NOT a tombstone - the session stays minable.
// The zero-yield record IS the tombstone the plan gate suppresses on, so
// it is written only over material a model was shown: the relayed plan
// hash when there is one, else the one survey a plan recorded since the
// last commit (session-mining-surveys.ts) - and only while the session
// still holds that material. A thrown ExtractSignalsWriteError never
// reaches here: a run that FAILED did not end, so it records nothing. A
// session with no mined material records nothing either, and every
// withheld record says why: failing open to a re-mine is the safe
// direction.
const staged = writeApprovalEnabled ? written.length : 0;
const zeroYield = written.length === 0;
let miningLedgerFailure: string | undefined;
let currentHash: string | null = null;
const raw = (err: unknown): string => (err instanceof Error ? err.message : String(err));
const surveyText = (err: unknown): string =>
opts.closeHostErrors === true ? closedSurveyFailure(err) : raw(err);
// Read, not consumed: the survey is cleared below only once this
// commit's outcome is final, so a commit that fails transiently (the
// turns unreadable, the ledger write refused) leaves it for the retry.
let surveyed: ReadonlyArray<string> = [];
let surveyFailure: string | undefined;
try {
surveyed = readMiningSurvey(vault, trimmed);
} catch (err) {
surveyFailure = surveyText(err);
}
let outcomeFinal = true;
try {
const mined = selectMinedTurns(boundary, listSessionRawTurns(vault, trimmed));
currentHash = mined.length === 0 ? null : minedTurnsHash(mined);
} catch (err) {
miningLedgerFailure = raw(err);
outcomeFinal = false;
}
if (currentHash !== null && opts.surveyedTurnsHash !== undefined) {
if (opts.surveyedTurnsHash !== currentHash) {
miningLedgerFailure =
`the session's turns no longer match the surveyed_turns_hash this commit answers ` +
`(${opts.surveyedTurnsHash}); no mining record was written, so the session stays minable`;
currentHash = null;
}
} else if (currentHash !== null && zeroYield) {
const withheld = withheldTombstoneReason(surveyed, surveyFailure, currentHash);
if (withheld !== null) {
miningLedgerFailure = withheld;
currentHash = null;
}
}
if (currentHash !== null) {
try {
recordSessionMiningOutcome(
vault,
{
sessionId: trimmed,
minedContentHash: currentHash,
outcome: {
written: written.length,
staged,
deduped,
durabilityRejected: rejected.length,
},
},
opts.now,
);
} catch (err) {
// The signals already on disk are safe; throwing would misreport the
// commit as failed, and dropping the error would leave a ledger that
// silently stopped being written. Name it instead - the surfaces
// carry it verbatim.
miningLedgerFailure = raw(err);
outcomeFinal = false;
}
}
if (outcomeFinal && surveyFailure === undefined) {
try {
clearMiningSurvey(vault, trimmed, surveyed);
} catch (err) {
// The outcome stands; a survey left behind can only attest material
// a plan really offered, so it is named rather than thrown.
miningLedgerFailure ??= `the answered plan survey could not be cleared (${surveyText(err)})`;
}
}
return Object.freeze({
sessionId: trimmed,
written: Object.freeze(written),
staged,
deduped,
durabilityRejected: rejected.length,
rejected: Object.freeze(rejected),
reconciliation: buildReconciliationReport({
attempted: written.length,
found: written.length,
missing: [],
}),
...(miningLedgerFailure !== undefined ? { miningLedgerFailure } : {}),
});
}
/**
* Why a zero-yield commit that relayed no plan hash may not write its
* tombstone, or `null` when it may: exactly one distinct material was
* surveyed since the last commit and the session still holds it. Anything
* else means the commit cannot attest that a model saw the turns it would
* suppress.
*/
function withheldTombstoneReason(
surveyed: ReadonlyArray<string>,
surveyFailure: string | undefined,
currentHash: string,
): string | null {
const tail =
"no zero-yield record was written, so the session stays minable; re-plan it, or relay the " +
"plan's surveyed_turns_hash with the commit";
if (surveyFailure !== undefined) {
return `the plan survey of this session could not be read (${surveyFailure}); ${tail}`;
}
if (surveyed.length === 0) {
return `no plan of this session was recorded since its last commit, so this commit cannot attest which turns a model saw; ${tail}`;
}
if (surveyed.length > 1) {
return `${surveyed.length} plans over different turns of this session are pending, so this commit cannot tell which one it answers; ${tail}`;
}
if (surveyed[0] !== currentHash) {
return `the session's turns changed since the plan this commit answers (${surveyed[0]}); ${tail}`;
}
return null;
}
/**
* The key a missing item is NAMED by: its payload index and its topic.
* The index carries the position and the topic carries the meaning,
* which is what an operator matches against the payload they are about to
* re-run.
*/
function unwrittenKey(index: number, topic: string): string {
return `items[${index}]:${topic}`;
}