Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
Brain Integrity Suite - design
Status: draft Author: orchestrator (via feature-release-playbook) Audience: implementation
Problem statement
Open Second Brain's preference write path has three structural gaps. First, confirmed preferences can be silently mutated outside the dream-pass promotion/retirement flow (hand edits, write races, errant agents) with no observable trace. Second, the existing writePreference atomic write prevents torn writes but does not prevent stale-update, unsafe-shrink, source-lock, or duplicate-write collisions - the last writer wins silently. Third, the dream pass runs as a single synchronous pipeline with no on-disk progress trace, and there is no read-only way for an operator (or another agent) to inspect what brain_dream would do before invoking it.
This release bundles five features that close all three gaps inside one cohesive PR.
Scope
- F1: Content-hash drift detection on confirmed preferences. On promotion, write
_content_hash: sha256(principle + scope)into frontmatter. Reads recompute and compare; mismatch logs adrift_detectedevent and surfaces inbrain_doctor. - F2: Structured write-collision detection. Four typed collision modes (
StaleUpdate,UnsafeShrink,SourceLock,DuplicateWrite) wrapping the write path via a newwritePreferenceTxnchokepoint that builds on theproper-lockfilepattern already proven insrc/core/pay-memory/approval.ts:361. - F3: Destructive-proof gate on confirmed preferences. Two sub-gates inside the dream pass: shrink-gate (new principle < threshold% of existing length quarantines the candidate); retire-from-confirmed gate (single-signal retirement of a multi-evidence confirmed preference is rejected without operator action or evidence above a higher threshold).
- F4: Durable workrun checkpoints for dream pass. Per-invocation JSONL workrun at
Brain/log/dream-runs/<YYYY-MM-DD>-<run-id>.jsonlwith one line per phase transition. Recovery scan on startup;brain_doctorcheck for dangling files. - F5:
brain_review_candidatesMCP tool. Read-only projection overdream(vault, { dryRun: true }), exposed as a new MCP tool returning{ would_promote, would_retire, would_supersede, clusters_below_threshold }.
Out of scope
- Per-preference edit-history audit trail (
t_9f452292) - separate sibling task, not in this PR. - Cross-vault locking or multi-vault coordination - Pay Memory's lockfile is single-vault.
- Schema migration for existing v0.11.0 preferences - additive fields with absent-as-default reader semantics. No
o2b brain migrate-*verb. - Changes to
dedup-hash.ts(signal/import dedup) - different concern, untouched.
Chosen approach
Variant 2: Single transactional write-path chokepoint.
Refactor writePreference into a thin façade over a new writePreferenceTxn(vault, input, expectations, options) that wraps proper-lockfile: acquire lock on the preference file path, re-read current frontmatter inside the lock, run an ordered chain of expectations checks (revision, content-hash drift observer, shrink, duplicate-window, source-lock), mutate via writeFrontmatterAtomic, release. Dream's promotion and retirement paths call into the same txn with feature-specific expectations (shrink-gate and retire-from-confirmed gate are two predicates plugged into the same expectations chain).
brain_review_candidates is a thin MCP wrapper over the already-existing dream(vault, { dryRun: true }) path. Workrun checkpoints live in a separate small context object threaded through dream phases - they observe transitions but do not own the write semantics.
Variant 1 (bottom-up primitives) was rejected because integrity logic would drift across writePreference and dream's promote/retire call sites - two parallel implementations of the same idea. Variant 3 (validator-chain pipeline) was rejected because it invents a framework the codebase explicitly avoids (v0.11.0 removed pre-1.0 shims; the project is the opposite of framework-heavy).
Design decisions
- One chokepoint, multiple expectations.
writePreferenceTxnis the single function that writes to a preference file. Direct writes from CLI/MCP and indirect writes from dream both go through it; the expectations parameter carries feature-specific predicates (shrink-gate, retire-gate, revision check). This is the SOLID open/closed principle: new gates plug in as expectations without modifying the txn body. - Lockfile recipe verbatim from Pay Memory. Same
retries: { retries: 30, factor: 1.2, minTimeout: 30, maxTimeout: 500 }, stale: 10_000, realpath: falseparameters astransitionRequest:361. Single source of truth for the lock semantics; if Pay Memory ever tunes them, the brain mirrors. - Hash is opt-in via status.
_content_hashis written only when promoting toconfirmed. Unconfirmed and quarantine preferences do not get the field - they are mutable by design. - Drift detection is observation, not enforcement. A mismatch logs a
drift_detectedevent intoBrain/log/<today>.md(and the JSONL sidecar) and surfaces inbrain_doctoras a warning. The read still returns the live content; hand-edits remain legal. The operator chooses how to react. - Revision counter is monotonic.
_revision: <integer>starts at 0 on creation, increments on every txn write.StaleUpdatefires when the writer'sexpected_revisiondoes not match the current. Absent field reads as 0 for backward compatibility. - Shrink threshold is config-driven, not hardcoded. New field under
Brain/_brain.yaml's existingconfidenceblock:confidence.unsafe_shrink_min_ratio(default 0.5). Anything below half of the existing principle length triggers UnsafeShrink. - Workrun phases are deterministic. Five phase markers in fixed order:
started,cluster_complete,promote_complete,retire_complete,finalized. A sixth markerinterruptedmay replacefinalizedon caught crash. Recovery on dream startup is non-resuming: if a workrun is incomplete, logrecovered(informational), do not auto-resume - the next dream invocation processes the inbox fresh.brain_doctorsurfaces dangling files as a warning. brain_review_candidatesis pure projection. The MCP handler callsdream(vault, { dryRun: true })and returns its plan. No new persistent state, no log events, no inbox mutations. Same signature as other read-only tools:(ctx, args) => Promise<Record<string, unknown>>.- Error class hierarchy. A new
BrainCollisionError extends Errorwith akind: 'StaleUpdate' | 'UnsafeShrink' | 'SourceLock' | 'DuplicateWrite'discriminant. Error names are machine-friendly identifiers; no human-language strings hardcoded. - Sync lockfile primitive instead of
proper-lockfile. The consultant's Variant 2 recommendation cited the Pay Memoryproper-lockfilerecipe, but Pay Memory is async while the entire brain write path is sync (writePreference,dream,moveToRetired,writeFrontmatterAtomic, all callers).proper-lockfilev4 has nolockSync. Migrating the brain to async would touch 30+ caller files (CLI verbs, MCP handlers, internal call sites) for a feature whose only async-only ingredient is one lock acquire. The right cost is the opposite: write a tiny sync primitive insrc/core/brain/sync-lockfile.tsthat usesfs.openSync(target + '.lock', 'wx')for atomic exclusive create. Single-attempt semantics:EEXISTraisesBrainCollisionErrorwithkind: 'SourceLock', no retry loop. Real-world contention in OSB is rare (single operator, single MCP server, dream runs from cron) so the retry/backoff machinery ofproper-lockfilewould be dead code. Pay Memory keeps its async recipe untouched - the two subsystems have different concurrency profiles.
File changes
New files:
src/core/brain/preference-txn.ts- thewritePreferenceTxnchokepoint,BrainCollisionErrorclass, expectations types.src/core/brain/content-hash.ts- hash computation helper forprinciple + scope.src/core/brain/dream-workrun.ts- JSONL workrun writer + reader + recovery scan.src/core/brain/review-candidates.ts- core builder behindbrain_review_candidates(callsdreamin dry-run mode and projects the summary).tests/core/brain/preference-txn.test.ts- txn behaviour, lock semantics.tests/core/brain/content-hash.test.ts- hash compute + drift detection.tests/core/brain/preference-collision.test.ts- StaleUpdate, UnsafeShrink, SourceLock, DuplicateWrite.tests/core/brain/dream-destructive-gate.test.ts- shrink-gate + retire-from-confirmed gate.tests/core/brain/dream-workrun.test.ts- workrun emission, recovery scan, dangling check.tests/core/brain/review-candidates.test.ts- projection shape, dry-run isolation.tests/mcp/brain-review-candidates.test.ts- MCP tool wiring.
Modified files:
src/core/brain/preference.ts-writePreferencedelegates towritePreferenceTxnwith empty expectations.src/core/brain/types.ts- new optional fields onBrainPreference(_revision,_content_hash).src/core/brain/dream.ts- workrun emission at phase boundaries; promotion/retirement calls go through txn with gate expectations.src/core/brain/policy.ts- new config fieldsconfidence.unsafe_shrink_min_ratioandretire.confirmed_evidence_multiplier.src/core/brain/doctor.ts- new checks for drift, dangling workrun.src/core/brain/query.ts,src/core/brain/search.ts(if separate) - drift verification on read.src/mcp/brain-tools.ts- newtoolBrainReviewCandidateshandler.src/mcp/tools.ts- tool registration.src/mcp/instructions.ts- tool description.CHANGELOG.md- v0.12.0 entry.README.md- updated MCP tool table.package.json- version bump to 0.12.0.scripts/sync-version.tsand friends pick up the new version automatically.
Estimated diff: ~55-70 files including tests.
Risks and open questions
- Lock contention on hot preferences. Pay Memory's lockfile config (max 30 retries, ~500ms total) is sized for human-operator pace. The dream pass may write many preferences in rapid succession in one run. Verify in implementation that lock acquire latency stays bounded; consider a single in-process mutex sufficient for the same-run case rather than serialising through the filesystem lockfile for each preference.
- Lock cleanup on crash. A sync lockfile leaves a
.lockfile on disk if the process aborts between acquire and release. Mitigation: aprocess.on('exit', ...)cleanup hook unlinks every still-held lock during normal exit; for hard crashes (SIGKILL, OOM), the next acquire on the same target sees EEXIST and fails withSourceLock. The doctor check surfaces stale.lockfiles viascanStaleLocks(vault). Operators can rm them by hand. Pay Memory'sproper-lockfilehas the same class of issue (itsstale: 10_000timer is for that case); we accept the same hazard with a simpler recovery path. - Sync lockfile primitive scope. New file
src/core/brain/sync-lockfile.ts(~50 lines). Tests intests/core/brain/sync-lockfile.test.ts. Doctor integration insrc/core/brain/doctor.ts. Total deltas on the brain write surface:preference.ts,dream.ts, the newpreference-txn.tswrapping both. No caller signatures change. - Hash stability across encoding variations.
sha256(principle + scope)is sensitive to leading/trailing whitespace and Unicode normalization. Decision: hash the trimmedprinciple.trim() + "\n" + (scope ?? "").trim(), no NFD/NFC normalization (the codebase already does not normalize elsewhere). Document incontent-hash.tsso a future refactor does not silently invalidate every existing hash. - Dream dry-run + workrun interaction. Dry-run must not write a workrun file - the workrun is part of the durable side-effect surface. Implementation: skip workrun emission when
opts.dryRun === true. - Backward compat for existing v0.11.0 vaults. All new frontmatter fields are optional; readers fall back to defaults (
_revisionabsent reads as 0; missing_content_hashon confirmed pref reads as "never hashed yet, hash on next write"). No migration verb. - Test data isolation. Tests need disposable vaults under
os.tmpdir()per the existing tests/helpers/ pattern; verifyproper-lockfilecleans up on process exit during abun testinterrupt.