Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* Source distillation (Ingestion & Import Robustness suite, t_2e2e959f).
*
* Condenses one source into discrete atomic claims, each carrying provenance
* back to the exact block it was drawn from. Composes the primitives OSB
* already has - block-id wikilinks (`[[Note#^abc]]`), source sha256 provenance,
* and the idempotent per-source page write - rather than building extraction
* from scratch.
*
* Provider-agnostic: the calling agent supplies the atomic claims and their
* block references; this core runs NO model. It validates the claims
* structurally (non-empty text, well-formed block ids), checks every quoted
* span in a claim against the source ({@link checkClaimQuotes}), stamps a
* content sha256 the verifier can reproduce from the source file, and writes
* one distillation page per source identity, rewritten in place on
* re-distill (never duplicated). A byte-identical re-run is inert.
*
* QUOTES. A span in quotation marks is a claim that the source says exactly
* that. It is compared with the block the claim cites (or, without a block,
* the whole source) in the SAME bytes the content digest is computed over -
* one read feeds both. A span that does not verify loses its two marks on
* the page (its words stay) and is named in the result's `quotes` report, so
* the page never shows an unverified quote. `strictQuotes` refuses the whole
* write instead, with {@link QuoteCheckError}, before anything is written.
*
* CAPTURE SCOPE. Every result says how much of the source the vault holds
* (`captureScope`): a local file is `full-local` and adds nothing to the page;
* a source with no local bytes is `url-only` and the page says so. For such a
* source the caller may pass the verbatim `excerpt` it read: the page stores
* it in a fenced `## Excerpt` section with its digest, becomes
* `bounded-local`, and its quotes are checked against the excerpt. An excerpt
* beside a local source, an empty one or one past the byte ceiling is refused
* with {@link CaptureExcerptError} before anything is written.
*
* Language-agnostic: block-id validation is structural (the Obsidian `^id`
* grammar), never over natural-language vocabulary.
*
* TRUST. Both arguments are caller-supplied, and the caller is the same agent
* that read the material - so the source identity is a CLAIM, not a fact. This
* module used to take it at face value: it canonicalised the string, stat-ed
* and hashed whatever `join(vault, ...)` landed on, and wrote every page under
* `provenance: stated`, the top authority tier. It now asks the same question
* of the same classifier the other two claim-write paths ask
* ({@link readSourceOrigin}, the byte-keeping form of `classifySourceOrigin`),
* so a source this vault does not own commits in the quarantine lane the
* retrieval trust gate can actually see.
*
* MARKDOWN NEUTRALIZATION (t_c2ac4fc8, review M4). The claim text is caller
* text composed into the page verbatim - on the untrusted lane it is
* third-party material the same way an untrusted summary's prose is. Under
* the markdown-neutralization gate's enforce mode, an UNTRUSTED source's
* claim text passes through the pure neutralizer at body assembly, after
* the quote check (which must verify the spans the caller actually sent);
* the writer's own block citation after the text is composed live. The
* page stamps `neutralized_markers` with the families it escaped, on
* change only. Off, shadow and the trusted lane write the claims
* verbatim.
*/
import { relative } from "node:path";
import type { FrontmatterMap } from "../../types.ts";
import { canonicalNotePath } from "../../path-safety.ts";
import { formatFrontmatter, slugify } from "../../vault.ts";
import {
UNTRUSTED_ORIGIN,
isSourceHidden,
normalizeSourceIdentity,
readSourceOrigin,
} from "../intake/source-trust.ts";
import { distillationPagePath } from "../paths.ts";
import {
DISTILL_CLAIMS_SHAPE,
DISTILL_CLAIMS_SURFACE,
assertResponseShape,
} from "../response-shape.ts";
import {
INTAKE_TRUST,
sourceContentHashFrontmatter,
untrustedSourceFrontmatter,
type IntakeTrust,
} from "../trust/untrusted-provenance.ts";
import {
renderProvenanceSection,
sourceIdentityHash,
type Provenance,
} from "../provenance/provenance.ts";
import { isoSecond } from "../time.ts";
import {
MARKDOWN_NEUTRALIZE_MODE,
neutralizeMarkdown,
neutralizedMarkersFrontmatter,
resolveMarkdownNeutralizeModeForProcess,
type MarkdownStructureKind,
} from "../text/neutralize-markdown.ts";
import {
CAPTURE_SCOPE,
EXCERPT_HASH_KEY,
assertExcerptAdmissible,
captureScopeForTrust,
captureScopeFrontmatter,
excerptDigest,
renderExcerptSection,
type CaptureScope,
} from "../provenance/capture-scope.ts";
import { BLOCK_ID_RE } from "./block-resolve.ts";
import { BRAIN_DISTILLATION_KIND } from "./claim.ts";
import type { DistillClaim } from "./claim.ts";
import { checkClaimQuotes, type QuoteEvidence } from "./quote-check.ts";
import {
QUOTES_UNQUOTED_KEY,
QUOTES_VERIFIED_KEY,
QUOTE_CHECK_OUTCOME,
QuoteCheckError,
type QuoteCheckReport,
} from "./quote-verdict.ts";
import {
assertLaneWrite,
lanePageExists,
parseLanePage,
readLaneText,
writeLaneText,
} from "../../store/lane-pages.ts";
/** Frontmatter `kind:` marker of a distillation page (defined in the claim leaf). */
export { BRAIN_DISTILLATION_KIND };
export type { DistillClaim } from "./claim.ts";
/**
* The most claims one call may carry. Every claim is checked against the
* source, so the cap bounds the work one request can ask for.
*/
export const DISTILL_CLAIMS_MAX = 1000;
/** Line breaks a claim may not hold: a claim is one bullet line on the page. */
const CLAIM_LINE_BREAK_RE = /[\r\n]/;
/**
* Normalize one agent-supplied claim record into a {@link DistillClaim}. Shared
* by the CLI and MCP surfaces so both accept the same shape: a `text` string and
* an optional `block` id (a leading `^` sigil is stripped; an empty block is
* dropped). Structural validation of the block id happens later, in `validate`.
*/
export function normalizeClaim(rec: Record<string, unknown>): DistillClaim {
const text = typeof rec["text"] === "string" ? rec["text"] : "";
const block = typeof rec["block"] === "string" ? rec["block"].replace(/^\^/, "") : undefined;
return { text, ...(block !== undefined && block.length > 0 ? { block } : {}) };
}
/**
* Ingress of the agent-authored claims payload, shared by the CLI and MCP
* surfaces. The payload is checked against the frozen distill shape BEFORE
* {@link normalizeClaim} runs, so a non-string `text` is refused by name
* instead of being normalized into an empty claim and rejected later for the
* wrong reason. A violation throws `ResponseShapeError` and nothing is
* written - the whole batch, not just the offending item.
*/
export function parseDistillClaims(payload: unknown): DistillClaim[] {
assertResponseShape(DISTILL_CLAIMS_SURFACE, DISTILL_CLAIMS_SHAPE, payload);
return (payload as ReadonlyArray<Record<string, unknown>>).map(normalizeClaim);
}
export interface DistillSourceInput {
/** Source identity: a vault-relative path or a URL. Canonicalized on write. */
readonly sourcePath: string;
/** The atomic claims the agent distilled from the source (non-empty). */
readonly claims: readonly DistillClaim[];
/**
* The verbatim text the caller read from a source the vault holds no bytes
* of (`url-only`). Stored on the page and used as the quote evidence;
* refused for any other source.
*/
readonly excerpt?: string;
}
export interface DistillSourceOptions {
readonly agent: string;
readonly now: Date;
/**
* Refuse the whole write with {@link QuoteCheckError} when any quoted span
* would be unquoted, instead of unquoting it. Unpaired marks never refuse.
*/
readonly strictQuotes?: boolean;
/**
* May the caller read the vault file at this vault-relative path? Asked of
* the canonical source identity when the vault holds its bytes. A refusal
* makes the source answer exactly as an absent one: untrusted lane,
* `url-only` (an excerpt is admitted), every span `url-only`, and no digest
* in the result or on the page.
* Absent: the caller reads everything (the local CLI).
*/
readonly readable?: (rel: string) => boolean;
}
export interface DistillSourceResult {
/** Vault-relative path of the distillation page. */
readonly distillationPath: string;
/** `false` when the page already existed and was rewritten. */
readonly created: boolean;
/** Number of atomic claims written. */
readonly claimCount: number;
/**
* sha256 over the source bytes, ABSENT when there were none to hash.
*
* It used to be the literal string `missing` in that case, which recorded an
* answer where there was none: a caller comparing digests had to know that
* one particular value is not a digest at all. Absence is now the whole of
* the report, and the reason for it is {@link trust}.
*/
readonly sourceHash?: string;
/**
* The lane this distillation committed in, as the page was ACTUALLY written.
* A caller told only where its page landed is told nothing about whether an
* ordinary read can ever reach it again - the trust gate excludes an
* untrusted page from every scope.
*/
readonly trust: IntakeTrust;
/** How much of the source the page's evidence holds, as written. */
readonly captureScope: CaptureScope;
/**
* The quote check's account, present only when at least one claim contains
* a quoted span. Every finding is a span that was unquoted on the page.
*/
readonly quotes?: QuoteCheckReport;
}
/** Strict UTF-8: a source that is not text verifies no quote. */
const UTF8_STRICT = new TextDecoder("utf-8", { fatal: true });
/** The evidence a quote is checked against: the bytes the digest covers. */
function quoteEvidence(bytes: Uint8Array | undefined): QuoteEvidence {
if (bytes === undefined) return { kind: QUOTE_CHECK_OUTCOME.urlOnly };
try {
return { kind: "text", text: UTF8_STRICT.decode(bytes) };
} catch (cause) {
// A decode failure IS the verdict: these bytes are not text, so no span
// can be said to occur in them. It is reported per span, by name. The
// fatal decoder signals that with a TypeError; anything else is a fault.
if (!(cause instanceof TypeError)) throw cause;
return { kind: "not-text" };
}
}
/**
* The capture scope of this write and the evidence its quotes are checked
* against. An excerpt is admitted only for a `url-only` source, and then IS
* the evidence; otherwise the evidence is the bytes the digest covers.
*/
function captureOf(
trust: IntakeTrust,
bytes: Uint8Array | undefined,
excerpt: string | undefined,
): { readonly scope: CaptureScope; readonly evidence: QuoteEvidence } {
const scope = captureScopeForTrust(trust);
if (excerpt === undefined) return { scope, evidence: quoteEvidence(bytes) };
assertExcerptAdmissible(scope, excerpt);
return { scope: CAPTURE_SCOPE.boundedLocal, evidence: { kind: "text", text: excerpt } };
}
/** The page's quote counters; nothing when no claim held a span. */
function quotesFrontmatter(report: QuoteCheckReport | null): FrontmatterMap {
return report === null
? {}
: {
[QUOTES_VERIFIED_KEY]: report.verified_in_block + report.verified_in_source,
[QUOTES_UNQUOTED_KEY]: report.unquoted,
};
}
/** A distillation failed structural validation; nothing was written. */
export class DistillValidationError extends Error {
constructor(message: string) {
super(message);
this.name = "DistillValidationError";
}
}
function validate(input: DistillSourceInput): void {
if (input.claims.length === 0) {
throw new DistillValidationError("distillation requires at least one claim");
}
if (input.claims.length > DISTILL_CLAIMS_MAX) {
throw new DistillValidationError(
`distillation accepts at most ${DISTILL_CLAIMS_MAX} claims per call, got ${input.claims.length}`,
);
}
input.claims.forEach((claim, i) => {
if (claim.text.trim().length === 0) {
throw new DistillValidationError(`claim ${i} has empty text`);
}
if (CLAIM_LINE_BREAK_RE.test(claim.text)) {
throw new DistillValidationError(`claim ${i} spans more than one line; a claim is one line`);
}
if (claim.block !== undefined && !BLOCK_ID_RE.test(claim.block)) {
throw new DistillValidationError(
`claim ${i} has a malformed block id ${JSON.stringify(claim.block)} - expected an Obsidian block id (alphanumerics and hyphens)`,
);
}
});
}
/** Render one claim bullet, citing its source block when the claim carries one. */
function renderClaim(claim: DistillClaim, canonicalSource: string): string {
const text = claim.text.trim();
return claim.block !== undefined
? `- ${text} ([[${canonicalSource}#^${claim.block}]])`
: `- ${text}`;
}
/**
* Distill a source into atomic claims. Validates the claims (throwing
* {@link DistillValidationError} with no write on failure), classifies the
* source, then writes a distillation page listing each claim with its
* block-level citation and a `## Sources` provenance section. Idempotent on
* the source identity.
*
* The classification lives HERE rather than at either surface because there
* are two ways in - the MCP tool and `o2b brain distill` - and a guard in one
* of them is a guard in neither.
*/
export function distillSource(
vault: string,
input: DistillSourceInput,
opts: DistillSourceOptions,
): DistillSourceResult {
// Vault-identity write guard (context-integrity-gates, Unit J).
assertLaneWrite(vault);
// The write-time markdown neutralization gate (t_c2ac4fc8, review M4),
// resolved BEFORE ANY WRITE like every seam: a misconfigured mode is a
// misconfiguration of the process, not a verdict about this source, and
// refusing it after the page exists would half-apply the distillation.
// Whether the gate has work to do is decided below (untrusted origin,
// enforce only); resolving it here only moves the refusal earlier.
const gate = resolveMarkdownNeutralizeModeForProcess();
validate(input);
// The SAME normaliser the trust classifier uses, so the identity this page
// records and the identity that was classified cannot be two different
// strings. A bare `canonicalNotePath` left a caller's `[[Articles/x.md]]`
// wrapped, which cited `[[[[Articles/x.md]]]]` in the body and keyed a
// second page off a second identity hash for one source - the same defect
// `ingestSource` was moved off in v1.46.0.
const canonicalSource = normalizeSourceIdentity(input.sourcePath);
const sourceLink = `[[${canonicalSource}]]`;
const provenance: Provenance = { level: "stated", sources: [sourceLink], premises: [] };
// Before any write, so a source the filesystem refuses to answer for leaves
// no half-written page behind. The classifier owns both halves of the
// question this module used to answer with a bare `existsSync`: whether the
// identity is SHAPED like a location this vault owns (so `../../etc/passwd`
// is never stat-ed, let alone hashed), and whether bytes are actually there.
// `provenance.level` stays `stated` - that vocabulary bands how a conclusion
// was DERIVED, which is orthogonal to who was entitled to supply the
// material; the lane is carried by the marker below.
//
// A source the caller may not read answers exactly as an absent one: the
// untrusted lane, no bytes, no digest. No span is checked against it, an
// excerpt is admitted for it, and nothing in the result or on the page
// tells the caller that the page exists or what it says. Asked first, so
// the source is never stat-ed or read for such a caller.
const origin = isSourceHidden(vault, input.sourcePath, opts.readable)
? UNTRUSTED_ORIGIN
: readSourceOrigin(vault, input.sourcePath);
const sourceHash = origin.contentHash;
// The quote check runs on the bytes the digest above was computed over (or
// on the admitted excerpt), before any write, so a refusal of either kind
// leaves nothing behind.
const { excerpt } = input;
const capture = captureOf(origin.trust, origin.bytes, excerpt);
const checked = checkClaimQuotes({ claims: input.claims, evidence: capture.evidence });
const quotes = checked.report;
if (opts.strictQuotes === true && quotes !== null && quotes.unquoted > 0) {
throw new QuoteCheckError(quotes.findings);
}
const idHash = sourceIdentityHash([canonicalSource]);
const absPath = distillationPagePath(vault, `${slugify(canonicalSource)}-${idHash.slice(0, 12)}`);
const existed = lanePageExists(vault, absPath);
const stamp = isoSecond(opts.now);
const createdAt = existed ? readCreatedAt(vault, absPath, stamp) : stamp;
const priorUpdatedAt = existed ? readUpdatedAt(vault, absPath, stamp) : stamp;
// Claim-text neutralization (t_c2ac4fc8, review M4): the claims are the
// caller's text ABOUT the source, and an untrusted source makes them
// third-party material - under enforce mode each claim's text passes
// through the pure neutralizer before the bullets compose, so a planted
// `[[wikilink]]` (or any other structure row) lands escaped while the
// writer's own block citation after it composes live. The quote check
// above already ran on the text the caller actually sent: verification
// answers "does the source say this", which escaping would corrupt.
// Off, shadow and the trusted lane leave every claim verbatim.
let pageClaims = checked.claims;
let neutralized: FrontmatterMap = {};
if (origin.trust === INTAKE_TRUST.untrusted && gate === MARKDOWN_NEUTRALIZE_MODE.enforce) {
const kinds = new Set<MarkdownStructureKind>();
pageClaims = checked.claims.map((claim) => {
const neutral = neutralizeMarkdown(claim.text);
if (!neutral.changed) return claim;
for (const kind of neutral.kinds) kinds.add(kind);
return { ...claim, text: neutral.text };
});
neutralized = neutralizedMarkersFrontmatter([...kinds].toSorted());
}
const claimsSection = [
"## Claims",
"",
...pageClaims.map((c) => renderClaim(c, canonicalSource)),
].join("\n");
// The rendered section ends in a newline; the join supplies the separation.
const excerptSection = excerpt !== undefined ? renderExcerptSection(excerpt).trimEnd() : "";
const body = [claimsSection, excerptSection, renderProvenanceSection(provenance)]
.filter((section) => section.length > 0)
.join("\n\n");
const buildContents = (updatedAt: string): string => {
const meta: FrontmatterMap = {
kind: BRAIN_DISTILLATION_KIND,
source_path: canonicalSource,
// `source_hash` is this page kind's own historical spelling of the
// content digest, kept because distillation pages already on disk carry
// it and this release rewrites none of them. It cannot be the ONLY
// record: on a `Brain/sources` page the same key holds the source
// IDENTITY hash instead (`ingest/ingest.ts`), and `sources-registry.ts`
// reads it generically - so a consumer holding an unknown Brain page
// cannot tell which of the two it has. `source_content_hash` has one
// meaning everywhere it appears, and it is written by the same helper
// the entity registry writes it with.
...(sourceHash !== undefined ? { source_hash: sourceHash } : {}),
...sourceContentHashFrontmatter(sourceHash),
// Marks the page when the claims were distilled from material this vault
// does not own, so the retrieval trust gate excludes it with a reason
// rather than ranking it beside the operator's own notes. A trusted
// source adds nothing, keeping its page byte-identical to before.
...untrustedSourceFrontmatter(origin.trust),
// Names the structure families enforce-mode neutralization removed from
// the claim text, ON CHANGE ONLY - the same conditional-spread shape as
// the marker above, so an unchanged page gains zero bytes.
...neutralized,
// How much of the source the page holds; nothing for a local file, so
// its page stays byte-identical to before.
...captureScopeFrontmatter(capture.scope),
...(excerpt !== undefined ? { [EXCERPT_HASH_KEY]: excerptDigest(excerpt) } : {}),
provenance: provenance.level,
agent: opts.agent,
claim_count: input.claims.length,
...quotesFrontmatter(quotes),
created_at: createdAt,
updated_at: updatedAt,
tags: ["brain", "brain/distillation"],
};
return formatFrontmatter(meta, body);
};
// Idempotent no-op: if the page would be byte-identical with its EXISTING
// updated_at preserved, nothing meaningful changed - skip the write and leave
// updated_at (and the mtime) alone. A real content change bumps updated_at.
const onDisk = existed ? (readLaneText(vault, absPath) ?? "") : null;
if (onDisk === null || onDisk !== buildContents(priorUpdatedAt)) {
writeLaneText(vault, absPath, buildContents(stamp));
}
return {
distillationPath: canonicalNotePath(relative(vault, absPath)),
created: !existed,
claimCount: input.claims.length,
...(sourceHash !== undefined ? { sourceHash } : {}),
trust: origin.trust,
captureScope: capture.scope,
...(quotes !== null ? { quotes } : {}),
};
}
/** Read a stable `created_at` from an existing distillation page, else fall back. */
function readCreatedAt(vault: string, absPath: string, fallback: string): string {
return readStampField(vault, absPath, "created_at", fallback);
}
/** Read the existing `updated_at` so an unchanged re-run preserves it. */
function readUpdatedAt(vault: string, absPath: string, fallback: string): string {
return readStampField(vault, absPath, "updated_at", fallback);
}
/** A stamp of the page, read under the distillation's own vault (never one parsed from the path). */
function readStampField(vault: string, absPath: string, field: string, fallback: string): string {
const [meta] = parseLanePage(vault, absPath);
const value = meta[field];
return typeof value === "string" && value.length > 0 ? value : fallback;
}