Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
Memory write-path integrity and store safety - an 11-unit hardening wave
Status: draft
Author: claude-dev-agent (via feature-release-playbook)
Audience: implementation
Branch: feat/memory-write-path-integrity
Consultant: Claude Code (variants in variants.md, raw output in cli-output/claude.md)
Problem statement
Open Second Brain funnels every durable memory mutation through a small set of choke points (writeSignal, vecUpsert, deleteBySource, the embeddings client), but several of those choke points accept input they should reject and several destructive paths lack the safety machinery that already exists elsewhere in the codebase. A malformed entity label fragments the entity graph, a NaN vector poisons cosine distances silently, a bulk deleteBySource is unrecoverable despite a full snapshot engine sitting one import away, and an exhausted embeddings quota surfaces as a generic HTTP failure with blind retries. This wave closes those gaps as one coherent release: every unit strengthens the write path or the safety envelope around it, with typed, explicit errors and no silent fallbacks.
Scope
Eleven kanban units, grouped into four subsystem clusters:
Cluster A - brain write path:
- A1
t_657b365e(p4) - strip Markdown/punctuation decoration from entity labels, reject structurally-junk labels before graph persistence, snapshot-gated prune of historical malformed entity nodes. - A2
t_375e98fd(p3) - deterministic durability gate rejecting transient operational content before an extracted signal is persisted. - A3
t_e540b093(p3) - opt-in write-approval queue: stage extracted signal writes toBrain/pending/, apply or reject via a new CLI surface. - A4
t_f79b4fe0(p3) - synchronous write-time conflict advisory when an incoming signal contradicts a confirmed preference. - A5
t_66c12a67(p3) - retire lifecycle for extracted fact signals (mark superseded and move toBrain/retired/, keep queryable), modeled on the preference retire lifecycle.
Cluster B - vector store:
- B1
t_e2b182b6(p4) - finite/non-zero vector validation gate atvecUpsertplus a typed error fromunitNormaliseInPlaceon zero-norm or non-finite input. - B2
t_3beb374c(p3) - instruction prefixes (query:/passage:) applied at query and index time, preset-aware defaults for e5-family models, configurable.
Cluster C - embeddings client resilience:
- C1
t_8880a68d(p2) - classify quota/billing outcomes (HTTP 402, quota-shaped 429) distinctly from generic transient failures, with a dedicated error code and actionable message. - C2
t_144b680a(p2) - honorRetry-After, split rate-limit backoff from quota fail-fast, degrade implicit semantic search gracefully with an actionable warning.
Cluster D - store safety:
- D1
t_7965b04b(p4) - shared snapshot-before-destructive-write gate over the existingsnapshot.tsengine, wired intodeleteBySource(and reused by the A1 entity prune). - D2
t_29a63073(p4) - store hardening: bounded idempotent permission migration for existingBrain/files, a doctor lint for symlinks escaping the vault, opaque store references instead of absolute host paths in MCP responses.
Out of scope
- No LLM-backed classification anywhere in this wave; every gate is deterministic.
- No mem9-style server quota contract, OpenAPI schema, or HTTP envelope (the project has no hosted API surface).
- No rebuild of the existing conflict-resolution supersede machinery (
hygiene/resolve-conflicts.ts); A4 reuses detection, it does not replace resolution. - No full CRUD for facts; A5 ships the retire/supersede half only.
- No git-based snapshot mechanism; D1 extends the existing tar+zst engine.
- No temporal half-open interval model (
valid_from/valid_toat instant precision) - the atomic temporal fact-replacement task (t_3ba9c404) stays on the board. - No changes to existing public CLI flag or MCP tool contracts beyond additive fields, new verbs, and new configuration keys.
Chosen approach
Variant 3 from the consultant run: cluster-scoped kernels with per-unit atomic commits. A shared abstraction is introduced only where two or more units provably share a choke point:
- Cluster A gets a light pre-persist gate seam inside the extracted-signal path (
routeExtractedFacts-> durability gate -> approval staging ->writeSignal), one logged-skip convention, and reuses the existing retire/audit patterns for A5. - Cluster B gets one vector-validity kernel used by both the store write path and the provider normalisation path, and one
kind-aware embed choke point for prefixes. - Cluster C gets a single richer classification result (
category,retriable,retryAfterMs) consumed by both C1 and C2, avoiding a double rework ofclassifyError. - Cluster D gets one
withDestructiveSnapshotwrapper reused bydeleteBySourceand the A1 prune, and reuses the existing remediation plan/apply machinery for the permission migration.
Clusters stay mutually independent so any unit remains an independently revertable atomic commit; error vocabularies stay subsystem-native (SearchError codes for search/embeddings, typed brain-side errors and logged skips for vault writes).
Design decisions
- A1 keeps
normalizeEntityNamebyte-stable and adds a separate decoration-stripping quality pass. The identity-key shape<category>:<normalized name>(built byentityIdentityKey,src/core/brain/entities/canonical.ts) must not change for existing entities, so Markdown/punctuation stripping (**Foo**->Foo, trailing:/.etc.) runs as a new exportedsanitizeEntityLabel(raw)applied BEFOREnormalizeEntityNameat label-intake boundaries (entity creation, fact-extract anchoring, atomic-facts anchoring). Junk rejection is structural only: labels that are empty after stripping, contain no letter or digit in any script (Unicode property classes, not ASCII), collapse to pure punctuation/digits, or exceed a length bound are rejected with a typed error at creation and skipped-with-log at anchoring time. There is NO built-in natural-language denylist; an optionalentities.label_denylistconfig key accepts operator-supplied exact labels (compared post-normalization). The prune walksBrain/entities/, re-runs the same validator, and removes failing nodes plus their edges behind the D1 snapshot gate, surfaced as a dry-run-default CLI action. - A2 durability classification uses structural signals only.
classifyDurability(text)(newsrc/core/brain/gates/durability.ts) flags transient operational content via language-agnostic shapes: filesystem temp paths (/tmp/,*.tmp, OS temp dirs), progress counters (N/M,NN%), run-id/timestamp-suffixed identifiers, byte/duration measurements as the dominant token class, and process-exit/status codes. Word lists are not compiled in; an optionaldurability.denylistconfig key accepts operator-supplied regexes. The gate runs insiderouteExtractedFacts(the noisy automated path) beforewriteSignal; a rejected fact is not silently dropped - it is recorded viaappendLogEventwith a dedicated event kind sodoctorcan surface rejection counts. Operator-initiatedbrain_feedbackwrites are NOT gated (the operator is not noise). - A3 approval queue defaults off and reuses the signal file format verbatim. With
write_approval.enabled=true,routeExtractedFactswrites the identical frontmatter document toBrain/pending/instead ofBrain/inbox/. Apply moves the file intoBrain/inbox/unchanged (entity anchors were resolved at extraction time and live in the document); reject moves it toBrain/retired/with a rejection reason. New CLI verbo2b brain pending list|apply|reject. No new document schema. - A4 advisory reuses
detectContradictionsas a pure function at write ingress.src/core/brain/health/contradiction.ts:250already takes in-memory preferences and a Jaccard option; A4 loads confirmed preferences for the incoming signal's scope bucket only, runs the same pairwise similarity against the incoming principle, and returns an advisory (never blocks the write). The advisory surfaces in thebrain_feedbackMCP response and CLI output and is logged. Detection thresholds reuse the existing health-pass defaults. - A5 fact retire mirrors
moveToRetiredfor signals.retireSignal(vault, id, { reason, superseded_by? })movesBrain/inbox/sig-*.mdtoBrain/retired/, rewrites frontmatter (_status: "retired",retired_at,retired_reason, optionalsuperseded_by, tag swap, alias for the old id) and appends an audit line, followingpreference.ts:1005conventions. Because the dream pass consumesBrain/inbox/only, the directory move IS the exclusion mechanism; retired signals stay readable through the existing retired-directory query fallback. CLI:o2b brain signal retire <id> --reason <text> [--superseded-by <id>]. - B1 validates at the store choke point and fails loud in normalisation. A new
assertValidVector(vector, context)(insrc/core/search/vector-guard.ts) rejects non-finite values and all-zero vectors with a newSearchErrorcodeEMBEDDING_INVALID_VECTOR;vecUpsert(store.ts:821) calls it after the existing dimension gate, andsemanticTopKapplies it to query vectors.unitNormaliseInPlace(http-util.ts:48) stops returning unnormalised zeros: zero-norm or non-finite input now throws the same typed error, because a silent all-zero embedding is exactly the misleading no-op fallback this project forbids. - B2 threads an embed kind through the provider contract additively.
EmbeddingProvider.embed(texts, kind?: "query" | "passage")is an optional second parameter (backward compatible for any provider implementation). The openai-compat provider prepends the configured prefix per kind;indexer.ts:601passes"passage",semantic-phase.ts:94passes"query". New config keysembedding_prefix_query/embedding_prefix_passage(env twins included) default from the preset: a new optionalqueryPrefix/passagePrefixfield pair onEmbeddingModelPreset, populated for e5-family entries. The active prefix pair is recorded in the index meta and a mismatch between stored and configured prefixes surfaces the existing reindex-required warning path, since stored vectors embedded without prefixes are not comparable to prefixed queries. - C1 and C2 share one classification rework.
classifyErrorreturns{ category: "quota" | "rate_limit" | "auth" | "transient" | "fatal"; retriable: boolean; retryAfterMs: number | null; error: SearchError }. The HTTP status andRetry-Afterheader are carried as structured fields on the thrown error fromembedBatchOnce:SearchErrorgains an additive optional constructor argument (opts?: { status?: number; retryAfterMs?: number }) exposed as readonly fields, so no message-string parsing remains anywhere in the classifier. 402 always classifies as quota; 429 classifies as quota only on protocol-token evidence in the response body (provider error codes such asinsufficient_quota, which are wire-protocol identifiers, not natural language). Quota addsSearchErrorcodeEMBEDDING_QUOTA_EXHAUSTEDwith an actionable message (billing exhausted, semantic search degraded, check provider billing). C2 makes the retry loop honorretryAfterMs(capped) for rate-limit/transient retries, fail fast on quota, and enrich the implicit-semantic degrade warning (semantic-phase.ts) plussearchErrorToMcp(search-tools.ts:400) with the quota-specific actionable message. Explicit--semanticcontinues to throw the typed error - no silent lexical fallback on an explicit request. - D1 is a thin wrapper, not a new engine.
withDestructiveSnapshot(vault, label, op)(in a new siblingsrc/core/brain/snapshot-gate.ts, keeping the 656-line engine module untouched) mintsrunId = <label>-<isoSecondCompact>via the existingvalidateRunId, callscreateSnapshot, runsop, thenpruneSnapshotswith the configured retention. Snapshot failure aborts the operation with the existing typed snapshot errors - a destructive op never proceeds unsnapshotted.deleteBySourcecalls it around its confirm-path deletions (source-cleanup.ts); the A1 prune reuses it. - D2 reuses remediation machinery and secure-by-default redaction. Permission migration ships as a new
auto-saferemediation step class (harden-permissions) inhealth/remediation.ts: an idempotent bounded walk (existingstepCapbounds work per run, so re-running resumes naturally) that chmodsBrain/files to0o600and directories to0o700, dry-run by default through the existing--remediate --dry-runflow. The symlink lint adds a doctor issue code (symlink-escape) that reports vault-internal symlinks resolving outside the vault, reusingensureInsideVault(src/core/path-safety.ts) semantics. MCP responses stop exposingctx.vaultabsolute paths at the foursrc/mcp/tools.tssites: the value is replaced by an opaque store reference derived from the existing device-id primitive plus a stable vault hash, with anexpose_host_paths=trueconfig escape hatch (resolved insrc/core/config.ts, the vault-level config surface) for operators who depend on the raw path; the redaction default is on because MCP responses land in model context, which is exactly the leak surface the upstream fix targets. The prune surface for A1 ships under the existing entity CLI verb as a subcommand (dry-run default,--confirmto apply); the implementer wires it into whichever entity verb file already exists rather than creating a parallel verb.
File changes
New files:
src/core/brain/gates/durability.ts- structural transient-content classifier (A2).src/core/brain/pending.ts- approval queue staging/apply/reject (A3).src/core/brain/signal-retire.ts- signal retire lifecycle (A5).src/core/search/vector-guard.ts- vector validity kernel (B1).src/cli/brain/verbs/pending.ts- pending queue CLI verb (A3).tests/*mirrors for every new module.
Modified files:
src/core/brain/entities/canonical.ts(+sanitizeEntityLabel, structural label validation) and its intake call sitessrc/core/brain/fact-extract.ts,src/core/brain/atomic-facts.ts, entity creation path (A1).src/core/brain/fact-extract.ts- durability gate + approval staging seam inrouteExtractedFacts(A2, A3).src/core/brain/signal.ts- write-ingress advisory hook surface (A4), shared by MCP/CLI feedback paths.src/core/brain/health/contradiction.ts- export an incoming-vs-confirmed advisory helper reusing the existing pairwise kernel (A4).src/core/search/store.ts,src/core/search/embeddings/http-util.ts- vector gate wiring (B1).src/core/search/embeddings/contract.ts,openai-compat.ts,presets.ts,src/core/search/indexer.ts,src/core/search/semantic-phase.ts,src/core/search/index.ts(config keys), index meta - prefixes (B2).src/core/search/embeddings/openai-compat.ts,http-util.ts,src/core/search/types.ts,src/core/search/semantic-phase.ts,src/mcp/search-tools.ts- quota classification and Retry-After handling (C1, C2).src/core/brain/snapshot.ts(or sibling),src/core/brain/source-cleanup.ts- snapshot gate (D1).src/core/brain/health/remediation.ts,src/core/brain/doctor.ts,src/mcp/tools.ts,src/core/config.tsorsrc/core/search/index.tsconfig surface - hardening (D2).src/cli/brain.ts,src/cli/brain/help-text.ts- new verbs/flags registration (A1 prune surface, A3, A5, D2).README.md,CHANGELOG.md,package.json+ manifest mirrors viascripts/sync-version.ts- docs and version (final task).
Risks and open questions
- B2 reindex semantics: adding passage prefixes invalidates stored vectors; the mismatch must surface through the existing reindex-required path, not silently mix prefixed queries with unprefixed passages. The implementer must verify how index meta records embedding parameters today (
embeddingHash, dimension lock) and extend that mechanism rather than invent a parallel one. - C1 quota-shaped 429 detection varies by provider; the classifier stays conservative (default: 429 without protocol-token evidence remains a rate-limit) so false quota classifications cannot disable retries incorrectly.
- D2 redaction default-on changes the VALUE of existing MCP response fields (shape unchanged). This is deliberate (security default) and gets a CHANGELOG callout plus the
expose_host_pathsescape hatch. - A1 prune blast radius: prune only removes nodes failing the structural validator and their edges; it never touches nodes that merely LOOK unusual. Dry-run default plus D1 snapshot gate bound the damage of a bad run.
- Gate chain ordering in
routeExtractedFactsis fixed as: dedup -> durability (A2) -> staging decision (A3) -> write; A4 advisory attaches to the operator-facing feedback path, not the extracted path, so the two never double-fire on one write.