Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
Non-Markdown source ingest - HTML parts, CSV and TSV table notes, named unsupported formats, and readers that answer at the caller's reach
Status: draft (phase 1 spec review applied)
Author: release orchestrator (via feature-release-playbook)
Audience: implementation
Base: origin/main f5c46615 (v1.68.0), branch feat/non-markdown-ingest, target version 1.69.0 (minor)
Problem statement
Ingest in Open Second Brain is text-only. DEFAULT_INGESTIBLE_EXTENSIONS (src/core/brain/ingest/batch-plan.ts:78-85) holds .md .markdown .txt .text .rst .org; every other file only adds to a per-extension unclassifiable count (batch-plan.ts:600-604). The search walker indexes .md files only (src/core/search/walker.ts:118), so an exported web page or a spreadsheet export in the vault never reaches search or recall. Nothing parses HTML into structure (the only reducer, htmlToText in src/core/brain/research/extract.ts:27-37, is a 4000-character web-finding helper with no headings and no entity decoding), nothing parses tables, and PDF, Office and image files are dropped by an anonymous count instead of being named. Six smaller defects ride along: the bounded source reader allocates its full cap for every file, the recall verdict's root coverage counts a root as reached through pages the caller cannot read, the daily and weekly brief views name preference ids from log events whatever the caller's reach, the architect says "no edges" when every declared edge touches a module a link cannot carry, the Hermes plugin's turn_id is refused by brain_recall_gate, and the surfacing gate has a dead newline branch.
Premise verification (live source, f5c46615)
Full evidence: the reconnaissance reports of this wave (HTML extraction and format registry, CSV and TSV table notes, v1.68.0 left-overs, the surfacing-gate facts). The orchestrator re-opened the anchors this design builds on: batch-plan.ts:75-100, 130-195, 255-330, 578-625, extractable-gate.ts:30-90, ingest.ts:168-240, 286-345, ingest-tools.ts:95-126, 255-275, registry-guard.ts:352, search-tools.ts:255-270, 566-610, surfacing-gate.ts:40-58, indexer.ts:1725-1752, brief-tools.ts:170-195, architect/generate.ts:335-345, 419-432, chunker.ts:28-36, 157, the OpenClaw bundle's module list, and ran the current CLI on a fixture tree (outputs in cli-output/expected-output.md).
Card A (source artifact ingest) - partly false, narrowed and built as narrowed
- False: "no general artifact indexer". Per-source summary pages with provenance (
ingestSource,ingest.ts:152-285), the sources registry, the content-hash manifest (it hashes raw bytes, so a changed HTML or CSV file is re-planned with no change), the batch planner and a closed skip-reason union already exist. - False in its framing: the pipeline never reads a source's text for the agent; the calling agent writes the summary. A "searchable part" can therefore only exist as text on the indexed Markdown summary page.
- Narrowed (validator ruling, kept): PDF, Office and images are named as unsupported, not extracted; OCR stays caller-side; watches and URL fetch are out.
- Built: a format registry, HTML text with heading-derived parts (headings and line spans only, never body text, on the page), a named
format-not-extractableskip, and a read-onlyo2b brain extractpreview verb.
Card B (plain-text notes for tabular data) - confirmed, narrowed
- Confirmed: nothing in
src/parses or renders tables;.csvis the canonicalunclassifiableexample in two test files. - Narrowed: CSV and TSV only, no formulas or workbook metadata (those live in XLSX and ODS, named unsupported under card A), no new dependency.
- New contract recorded: this is the first time the kernel writes derived source content onto a
kind: brain-sourcepage. It does so only for an in-vault file in the trusted lane, after the reach check, with two redaction passes, under named caps.
Fold-ins - confirmed, folded as narrowed
readSourceBoundedallocatesPRE_EXTRACT_MAX_SOURCE_BYTES + 1for every source (ingest.ts:329): confirmed; folded (thearchitect/manifests.ts:224-249pattern).- Coverage-divergent root flip (
indexRootCoverage,indexer.ts:1727-1750, no reach filter): confirmed; folded. - Whole-layer status and brief counts: confirmed and wider than recorded (daily and weekly envelopes name preference ids from log events). Only the id slice is folded; the counts are deferred (see Out of scope).
- Architect "no edges" sentence when only unlinkable edges exist (
generate.ts:419-432,NO_DEPENDS_ON:342): confirmed, plus the same false sentence on the module note; folded. - Hermes
turn_idrefused by the closedRECALL_GATE_INPUT_SCHEMA(search-tools.ts:566-609,additionalProperties: false): confirmed; folded. The sibling schema atsearch-tools.ts:263already declares the sameturn_id(512), which is the pattern copied. - Dead newline branch in
isShellOnlyPrompt(surfacing-gate.ts:53testsnormalized, already whitespace-collapsed at:45): confirmed. Premise corrected: testing the raw prompt for a line break admits multi-line prompts that start with a command name; a single-line prose prompt that starts with a command name ("git push keeps failing after the rebase") is still skipped asshell_command. The CHANGELOG and docs state exactly that, not "prose prompts are admitted".
Scope
- One format registry,
src/core/brain/ingest/source-formats.ts(Node-safe leaf, noBun.*): theSOURCE_FORMATvocabulary, one spec per format with its extractor, the extension map,sourceFormatOf, andDEFAULT_INGESTIBLE_EXTENSIONSderived from it (re-exported frombatch-plan.ts). - Planner:
.csv .tsv .html .htmplanned by default; PDF, Office, EPUB, RTF and image files reported per file asformat-not-extractablewith the format asdetail; a planned file whose format is nottextcarriesformat. - HTML: a linear scanner (
extract-html.ts) giving title, text and parts;## Partson the summary page. - CSV and TSV:
table-note.tsgiving a fenced plain-text## Tablesection and fivetable_*frontmatter keys. brain_ingest_source: automatic by format,source_formatplussource_content_hashon derived pages, additivepartsandtableresult fields (counts and tokens only).- CLI
o2b brain extract <file> [--json]for every registered format. - Fold-ins 1, 2, 3 (id slice), 4, 5 and 6 above.
Out of scope (deferred, by name)
- PDF, Office (DOCX, XLSX, PPTX, ODT, ODS, ODP), EPUB, RTF and images: named
format-not-extractable, not extracted. PDF text needs content-stream and font decoding that is unreliable without a library; Office is feasible later throughnode:zlibbut is its own change. - OCR: caller-side (validator ruling); the per-file skip with
detail: imageis the hook a caller routes on. - External-extractor command gate: a new subprocess capability reachable from MCP needs the
secrets/exec.tsaudit discipline, an apply gate and an allowlisted command form; theformat-not-extractableentries are the hook a later gate claims (a config key on thebench_judge_cmdprecedent,config.ts:1230). - Watches: overlap the maintenance lane; the content-hash manifest already re-plans changed bytes.
- URL fetch: ingest stays network-free.
- A read-only
brain_extract_sourceMCP tool: a new capability (tool count, parity, probe catalogue, docs) for a model that assumes the agent holds the source. - Whole-layer count aggregates of the status and brief views (
computeBrainStatus,buildOperatorSnapshot, monthly, operator and today views,osb://status): 12-16 files, a refusal-code decision and many tests that need a local reach; deferred as its own change. Only the daily and weekly slice ships now: their ids, and theirevents_by_kindandvault_deltacounts recomputed from the events the caller may see (review round). htmlToTextinresearch/extract.tsstays unchanged (a different contract: a bounded web finding).
Chosen approach
The consultant recommended its Variant 1 (a second lane of the opt-in pre_extract pass, nothing written to the page). The orchestrator overrides it for Variant 2's spine, adjusted (see variants.md): card B's deliverable is rows that search can find, and search indexes only the summary page, so a preview the agent may or may not copy leaves the gap open. The consultant's two objections to Variant 2 are answered in the design: the header-loss problem by token-bounded row groups that each repeat the header (no chunker rule change), and the redaction worry by two named passes with a pinned over-redaction test. HTML keeps the consultant's "no body copied" line: only headings and line spans reach the page.
Design decisions
Registry and planner (lane A)
- R1 one registry.
SOURCE_FORMAT(four-piece, census row) =text, csv, tsv, html, pdf, docx, xlsx, pptx, odt, ods, odp, epub, rtf, image. Markdown and the lightweight markups map totext(they are read as they are). Each spec is{format, extractor, extensions}withextractorfromSOURCE_EXTRACTOR(verbatim | html | table) ornull. Evidence: the pre-extractLANGUAGE_BY_EXTENSIONsingle-home pattern; a Map, not prose, decides. - R2 derived defaults.
DEFAULT_INGESTIBLE_EXTENSIONS= the extensions of every spec with a non-null extractor, in spec order: the existing six first, then.csv .tsv .html .htm. Same name and export frombatch-plan.ts, so nothing that iterates it reorders. - R3 skip routing. In
collectIngestible, after the ignore check and before the reach predicate: an extension in the effective set is discovered; an extension the registry names withextractor: nullbecomes{reason: "format-not-extractable", detail: <format>}; anything else staysunclassifiable.format-not-extractableis FIRST inSKIPPED_PAGE_REASONS("in the order a page meets the gates"). Format skips sort by path and precede schema skips inskippedNonExtractable. Evidence: visibility is a Markdown rule (docs/mcp.md:2062-2064) and calling the predicate on a binary reads it whole (vault.ts:200); the unclassifiable count already runs before the predicate for the same reason. A caller'sextensionsoverride still wins: a named format the caller lists is planned. - R4 planned format.
PlannedFile.format?: SourceFormat, set only when the format is nottext; serialized asformatbesidestatus. A tree of Markdown files plans byte-identically. The plan id of a tree holding HTML, CSV or TSV changes (those files join the discovered set), disclosed indocs/updating.md. - R5 bounded reader.
readSourceBounded(absolute, maxBytes)allocatesstat.size + 1and grows once tomaxBytes + 1when the buffer fills under the cap (themanifests.tspattern), exported for the CLI verb; the pre-extract pass passesPRE_EXTRACT_MAX_SOURCE_BYTESunchanged. - R6 one fence helper.
fenceFormoves fromprovenance/capture-scope.tstosrc/core/markdown-fence.ts; the excerpt section and both new sections use it.
Dispatch and the ingest write (lane A)
- I1 dispatch.
extract-source.tsexportsextractSource(path, bytes)with an exhaustiveswitch (spec.extractor)overnull,verbatim,htmlandtable. The orchestrator steer had lane A ship theverbatimandnullarms and lanes B and C each add one arm. Overturned: an exhaustive switch over a union that already holdshtmlandtabledoes not type-check without those arms, and a placeholder arm would be a stub. Lane A writes all four arms in its last task (A4), which waits on the two pure modules (B2, C2) whose exported signatures this plan pins;extract-source.tsthen has one owner. - I2 trigger. Automatic by format; no new MCP argument, no new tool, the
brain_ingest_sourcedescription (299 of 300 characters) untouched. Evidence: the derived section is the page content the card asks for; the pre-extract pass is opt-in because its output never reaches the page. - I3 reach order. Identity is a vault-relative file whose format has an extractor of
htmlortable; then the trusted lane; thenisSourceHidden(vault, source, readable); then onereadSourceOriginread (8 MiB,O_NOFOLLOW, digest of exactly the bytes returned). A URL, an absent file and a hidden file all answersource-not-localwith no digest and no section. A source above 8 MiB is already refused by the intake before this step. - I4 page. The derived section comes after
## Connections to existing notes. Frontmatter gainssource_formatandsource_content_hash(viasourceContentHashFrontmatter) whenever the bytes were read, plus the format's own keys. Atextsource gains nothing: byte-identical. - I5 visibility kept. An operator-set
visibilityon an existing summary page is preserved on rewrite, the waycreated_atis (ingest.ts:201-202). Applied to every summary page: dropping it on re-ingest is the same defect for a text source, and the key is the only control a rendered table has. Disclosed indocs/updating.md. - I6 wire.
parts(HTML) andtable(CSV, TSV) on the result, each absent for other formats, counts and tokens only.registry-guard.ts:352reason becomes "write; returns the summary path, bounded id lists and derived-section counts".
HTML (lane B)
- H1 scanner, not HTMLRewriter. A pure TypeScript linear scanner (index walk, no regex over unbounded input). Evidence: HTMLRewriter leaves entities undecoded, misses document-level text in element handlers, throws on
onEndTagfor void elements, and is Bun-only (recon probes on Bun 1.4.0 and 1.4.2). - H2 grammar. Comments, doctype and processing instructions dropped;
script style noscript template svg mathskipped whole;<private>content becomesPRIVATE_REGION_PLACEHOLDER; numeric entities (invalid code points become U+FFFD) andamp lt gt quot apos nbspdecoded, any other named entity kept verbatim; block elements emit line breaks;prekeeps whitespace, elsewhere whitespace collapses;<title>is the title; no attribute value is ever emitted; CRLF and CR normalised before lines are counted; strict UTF-8, refused by name. - H3 parts.
{index, level, heading, trail, lineStart, lineEnd, sourceOffset}; level 0 is a non-empty preamble;trailjoins ancestor headings with>; headings folded withoneLineand capped at 200 characters; at mostHTML_PARTS_MAX = 256parts, the rest counted inpartsOmitted. - H4 section.
## Partsholds one fenced block (info stringparts, fence byfenceFor) with one line per part,h<level> <trail> | lines <a>-<b>(preamble | lines <a>-<b>for level 0). Inside a fence,[[...]]and#wordfrom untrusted headings stay out of the link graph (stripCode,tags.ts:38-52) while FTS still indexes the text. - H5 CLI.
o2b brain extract <file> [--json], read-only, throughreadSourceBounded(file, SOURCE_HASH_MAX_BYTES), prints the dispatch result for any registered format; text and named-unsupported formats print{extracted: false, format, reason}.
Tables (lane C)
- T1 section. Fenced plain text, not a GFM table: one block per row group under
### Rows <a>-<b>, info stringtable, the header record first in every group, cells joined by|, cell escapes\->\\,|->\|, LF ->\n, CR ->\r, TAB ->\t, and (review round) a backtick ->\`so no row line can start a fence and every table body takes a three-backtick fence, and every other C0 control, DEL and C1 control ->\u{XXXX}. Evidence: a GFM table turns[[x]]and#wordcells into graph facts and loses its header in later chunks (splitBlock,chunker.ts:511). - T2 group size. At most
TABLE_NOTE_ROWS_PER_GROUP = 50rows AND at mostTABLE_NOTE_GROUP_MAX_TOKENS = 600tokens counted by the chunker's own counter (exported ascountChunkTokens), so a group block fits one 800-token chunk and every chunk carries a header line and aTable > Rows a-bheading path. Adjusted from the steer's fixed 50 rows: a 50-row group of a 15-column table exceeds one chunk and its later chunks lose the header. A single row over the budget forms its own group (residue: such a group can still split).CHUNKER_VERSIONuntouched. - T3 parsing. Delimiter by extension:
.tsvTAB without quote processing;.csvcomma with RFC 4180 lenient quoting, switching to semicolon when the first record has two or more fields with semicolon and more fields with semicolon than with comma (amended in the review round from "one field with comma": a comma-decimal export quotes only fields holding;, so its headername;"price, eur";qtyhas two comma fields;"a;b",cstays comma). BOM stripped, empty lines skipped, CRLF, LF and lone CR end records, bare quotes kept, an unterminated quoted field refused asmalformed-quotingwith the 1-based record number. The first record is always the header. Ragged rows keep their own cell count. - T4 caps.
TABLE_NOTE_MAX_ROWS = 1_000,TABLE_NOTE_MAX_COLUMNS = 64,TABLE_NOTE_MAX_CELL_CHARS = 256(code points, U+2026 marks a cut),TABLE_NOTE_MAX_BYTES = 262_144for the section. Every cut is named intable_truncatedand a closing line states rendered and total rows. - T5 redaction. A column whose header satisfies
isSecretKeyNamehas every value replaced byREDACTION_PLACEHOLDER(header kept); every other cell goes throughredactRawOutput(cell, {redactUrlCredentials: true, maxInput: Number.POSITIVE_INFINITY}); noredactTokens(it would erase order ids and hashes).keywordheaders over-redact, pinned by a test. Count onredacted_cells. - T6 refusals.
TABLE_NOTE_SKIP_REASON:source-not-local,not-utf8,contains-nul,malformed-quoting,empty. Data, not errors: the page and entities are still written.
Reach left-overs (lane D)
- L1 root coverage.
indexRootCoverage(config, roots, admit?)counts a root reached only through an admitted path; below local reachadmitisisPathReadableAtReachoverindexedVisibilityByPathsread in the same store session;probeRetrievalCorpus(resolveConfig, reach?)threads it; local reach unchanged. Thebrain_searchandbrain_recall_gateresidual sentences leave the registry. - L2 brief id slice. Below local reach the daily and weekly views drop
status_transitions,retired,contradictionsandsource_pointersrows whose ids failreaderRefView, and skipcaptureReportDelta(the digest precedent,brief-tools.ts:190). Counts stay as they are and the registry row says so. - Public wording: "answers at the caller's reach".
Small fixes (lane E; turn_id rides in lane D, which owns search-tools.ts)
- E1 architect. When every declared edge touches an unlinkable module the region prints a distinct fixed sentence and lists the dropped edges as code spans (
codeSpanName(from) -> codeSpanName(to)); with some edges drawn, one extra line under the diagram names the dropped ones; the module note saysNot linked:with the code spans instead ofDepends on: no other module.depends_onfrontmatter unchanged. - E2
turn_id. Optionalturn_id(string, max 512) onRECALL_GATE_INPUT_SCHEMA, recorded on thegate_telemetrypayload underCONTINUITY_TURN_ID_KEY;plugins/hermes/_schemas.pyre-vendored verbatim; aprefetch(..., turn_id=...)drive intests/python/test_static_schemas.py. - E3 newline branch.
isShellOnlyPromptreceives the raw prompt's line-break test before normalisation; the four-script contract test untouched.
File changes
About 57-59 files: 19 new (6 source modules, 13 tests), about 32 changed source and test files, 6 docs; plus the version mirrors and the bundle outside the lanes. Exact ownership in plan.md.
Risks and open questions
- The plan id of a tree holding HTML, CSV or TSV changes; an in-flight resume checkpoint for such a tree no longer matches (disclosed).
.pngand the other named formats move fromunclassifiabletoskipped_non_extractable; a reader that summedunclassifiablesees fewer files there (disclosed).- A re-ingest of an already-ingested CSV, TSV or HTML file rewrites its page once with the new section and keys (disclosed).
- Redaction of table cells is fail-safe by over-redaction on key-like headers; a credential that matches no pass stays in an indexed page, the same exposure the raw file already has in the vault, now also in embeddings (stated in docs/mcp.md).
- A summary page that carries a derived section is at most as visible as its source (review round):
deriveSourceSectionreturns the source's ownvisibilitytokens andingestSourcejoins them with the page's keptvisibility; a leading frontmatter block of the source is not handed to the extractor. - Redaction residue of table cells (review round, kept as decided in T5): bare vendor tokens, base64 or high-entropy blobs, PEM blocks in quoted multi-line cells and credential-like values under headers that name no credential are not redacted; stated in docs/how-it-works.md.
- No lane edited a module of the OpenClaw bundle; the review round did (
redactor.tsfor linear private-region walking,vault.tsfor a linear frontmatter opener), soopenclaw/index.jsis rebuilt and committed after the lastsrc/change.
Follow-ups (review round)
- Chunker fence rule.
readCodeBlock(src/core/search/chunker.ts) closes a fence of four or more backticks on any line starting with three. A CommonMark-length-aware close (a run of the same character at least as long as the opening one, the rulesrc/core/tags.tsalready implements) is deferred: it changes chunk boundaries and needs aCHUNKER_VERSIONbump that re-chunks every vault. Table notes no longer depend on it, since cells escape every backtick. @osbmarker fence rule.discoverMarkersDetailed(src/core/brain/inline.ts) recognises only three-backtick fences. The same length-aware rule is deferred because it changes marker parsing; latent today, since both walkers that act on markers skipBrain/.- Format skips before the reach predicate (accepted residual).
brain_ingest_batch_plannames a format-skipped file inskipped_non_extractablebefore asking the reach predicate. Visibility is a frontmatter rule, so a real PDF or image is always readable; only a non-text file that starts with a frontmatter block, or an unreadable one, would be named where a reserved page is not. - Single read in
deriveSourceSection. The section is derived from a second read of the source; a source that grows past the ceiling between the intake's read and this one now answerssource-not-local. Threading the intake's bytes through is the follow-up that removes the second read. oneLineleaf module.src/core/brain/ingest/extract-html.tsandsrc/cli/brain/verbs/extract.tsimportoneLinefromsrc/core/architect/manifests.ts, a Bun-dependent module (safe to load in the Node build, since the Bun call sits inside a function). MovingoneLineto a leaf module, re-exported frommanifests.ts, removes the coupling of ingest to the architect; deferred.- Frontmatter pattern, one copy. The leading-block strip of
extract-source.tscopied the vault's frontmatter pattern; it now importsFRONTMATTER_REfromsrc/core/vault.ts, as do the two link-graph readers that held their own copies. The pattern's opener takes horizontal whitespace only, so a run of blank lines after an unclosed---is read in linear time. - Other
<private>detectors (resolved in the review rounds).src/core/brain/session-recall.tsandsrc/core/brain/continuity/redaction.tsmatched the open tag with their own regular expressions; both now ask the redactor's span walker (privateRegionTexts), so one linear rule decides what opens a region. - Repeated remote re-ingest of a reserved summary page (accepted residual). A second remote brain_ingest_source of a source whose deterministic summary page is reserved still reveals that page's existence. Both calls answer created: true where the control answers true then false, and brain_search_by_source totals differ (2 against 3). A remote caller's summary write is discarded. Closing it means a caller-lane summary path below local reach (the R2S11 follow-up).
- Provenance as the summary id (R2S11, the full form of R3-6). An entity page from an intake still names the reserved source path in its
## Sourcessection (no content, no digest). The full fix reads the source's visibility once fromorigin.bytes, hands it to both the intake and the derivation, writes the provenance as the summary id rather than the source path, and teachestraceReferencesto follow summary-id provenance so a localbrain_delete_by_sourcestill cascades to the entity pages. Deferred because it changes the cascade and the provenance shape together. removed-tool-referencecap before the reach filter (count residual).REMOVED_TOOL_MAX_WARNINGScounts beforebrain_doctorfilters findings by the caller's reach, so a reserved page can take one of the capped slots. The check runs without the caller's view; moving the cap after the filter needs the view threaded into it.- Stale-dependency note over the whole Brain layer (count residual, released code). The
uncertainstale-dependency note ofbrain_doctorcounts "N state(s) stopped being current" over every record and moves by one when a reserved record is retired; it names no record. Answering it at the caller's reach needs the stale-dependency collector to take the reach view. - Shared ranking statistics (residual, stated on the
brain_searchregistry row). The bm25 corpus statistics and the diversity rerank run over the shared index, so scores can differ by reach while the returned paths do not.