Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
Source pipeline integrity and operator tooling - one wave, eleven units
Status: accepted Author: Claude orchestrator (via feature-release-playbook) Audience: implementation
Problem statement
Sources enter the vault through paths that cannot be scoped or gated, batches can silently lose documents, prose citations never reach the temporal layer, and vault health signals are scattered across six read-only surfaces with no repair action. This wave makes intake trustworthy end to end: scoped and gated discovery, reconciled dispatch, deterministic pre-extraction, citation provenance, configurable indexing, degree-aware querying, and an operator toolchain that can both summarize and repair.
Scope
Eleven kanban tasks ship as one PR and one release (v1.34.0):
t_2ed754d1early-closed stdout pipe exits clean ino2b/vault-logt_9654de80nested.gitignorecomposition in the hygiene file scant_e82101a5--src-subpath/--excludescoping on source ingestt_ed856388extractableflag honored during page discoveryt_ef786747local no-LLM code-structure extractor as a pre-ingest passt_d067a153dispatched-vs-ingested reconciliation for batch ingest planst_a3d1adb0inline[Source: <name>, YYYY-MM-DD]citations become dated provenance events on the temporal timelinet_618f7211configurable FTS tokenizer language and diacritic rulest_9bee8f0bgraph-degree cardinality predicates in the search/filter DSLt_bd6cc4cbo2b brain doctor --repairfor the issue classes doctor already detectst_9f9c5466unified operator status snapshot with per-problem next-command hints
Out of scope
- Rebuilding the reindex command (exists as
o2b search reindex). - Replacing or extending the codegraph partner integration; the pre-ingest extractor is a fallback pre-pass, not a codegraph substitute.
- Tree-sitter or any non-stdlib parsing backend for the extractor.
- An interactive profile explorer or onboarding scaffolder (different theme).
- Repair actions for issue classes doctor does not already detect.
t_71f96533(created_at bounds on raw-session grep) is excluded as a duplicate: v1.33.0 already shippedsince/beforeon session grep.
Chosen approach
Consultant Variant 1: two shared kernels, cluster-local units. Exactly two new shared abstractions, each riding inside the first feature commit that needs it:
- Path-scope engine
src/core/fs/ignore.ts- nested ignore-file composition with git semantics (deeper file scopes its subtree, nearer!re-include wins,.git/info/excludelayering). Introduced by the hygiene-scan unit (P1), consumed by ingest scoping (P2). It sits below bothhygiene/andbrain/ingest/, keeping layering one-directional. - Diagnostics-signal model - issue class + detector + optional fixer +
next-command hint, introduced by
doctor --repair(O2) and consumed by the unified status snapshot (O3), so hints travel with issue definitions instead of being duplicated in the snapshot formatter.
Every other unit is a local change to its existing module. Four clusters:
scope/gate (P1 -> P2 -> P3), ingest integrity (P4, P5), provenance/query
(Q1, Q2, Q3), operator surface (O1, O2 -> O3). Hard dependency edges are
only P1 before P2 and O2 before O3; every prefix of the ship order leaves
main releasable.
Design decisions
- Ignore engine location
src/core/fs/ignore.ts, not underhygiene/oringest/: both consume it; one home, no reach-across imports. - Byte-identical opt-out everywhere: unset config, absent flags, and unconfigured features must leave existing outputs unchanged; regression-tested per unit.
- Explicit errors, no silent fallbacks: malformed ignore patterns, invalid tokenizer configs, invalid degree predicates, malformed repair targets, and unknown citation dates surface as typed errors or explicit warnings; nothing is silently skipped without a report.
- Reconciliation is a report, not a retry:
reconcilePlan(planId)diffs the dispatched set against completed entries and returns/warns the gap; it does not re-dispatch (that stays operator-driven). - Extractor stays deterministic and stdlib-only: structural parsing per language family via regex/line grammar, emitting JSON entity/edge seeds that ingest passes to the agent as pre-extracted facts; no natural-language word lists, no LLM in the kernel.
- Citation syntax is structural:
[Source: <name>, YYYY-MM-DD]is a fixed marker grammar with an ISO-shaped date; no language-specific parsing. Dedup key is (normalized name, date) against already-logged source events. - Tokenizer config composes with CJK: config selects FTS5 tokenizer
options (stemming/diacritics); the existing trigram prefilter path is
untouched. Changing the config requires an explicit
o2b search reindex; the CLI says so rather than reindexing implicitly. - Degree predicates read the existing graph index: the DSL gains count
predicates over backlinks/outlinks backed by
graph-index.tsdegree data; no new graph computation. - Repair is opt-in and previewed:
doctor --repairdefaults to dry-run preview;--applyperforms fixes;--strictread-only behavior is unchanged. Every applied fix logs a typed event. - EPIPE handling is scoped to stdout write failures: an early-closed stdout pipe exits 0; all other I/O errors keep failing loudly.
- MCP parity: agent-relevant new surfaces (status snapshot, repair preview, degree predicates) get MCP counterparts; frozen parity lists and tool-count tests update accordingly. The citation scan ships CLI-only, following the existing precedent for operator-driven scan verbs; an MCP counterpart is deferred until an agent workflow needs it.
File changes
New: src/core/fs/ignore.ts, src/core/brain/ingest/pre-extract.ts (code
structure extractor), src/core/brain/ingest/reconcile.ts,
src/core/temporal/citations.ts, diagnostics-signal module next to
doctor.ts, snapshot verb module, tests per unit.
Modified: src/core/hygiene/scan-repo.ts, ingest CLI/MCP surface and
ingest.ts / batch-plan.ts / checkpoint.ts, discovery path consuming
extractable, src/core/search/schema.ts + config for tokenizer,
src/core/search/property-filter.ts + src/cli/search.ts for degree
predicates, src/cli/main.ts / scripts/* for EPIPE, doctor CLI/MCP,
docs and manifests (version 1.34.0).
Risks and open questions
- Gitignore composition has sharp edge cases (anchoring, directory-only
patterns,
!precedence); mitigate with a property/table test suite mirroring git's documented rules. Fullgit check-ignoreparity is not claimed; the supported subset is documented. - The extractor's language coverage is bounded; unknown languages must fall through to today's behavior explicitly (reported as unextracted, never a fake empty result).
- Repair fixers touch vault files; every fixer needs a preview diff and an idempotency test.
- The diagnostics-signal model shape is set by O2 and may need a small extension when O3 lands; acceptable churn inside one branch.
- MCP tool-count and description-budget guards will need deliberate updates; keep additions within the 300-char description cap.