Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
/**
* The four ways agent memory fails, scored against shipped code
* (t_72d6eb23).
*
* The existing bench measures whether a query finds the right note. That
* is one failure mode out of several, and not the one operators complain
* about. This module adds the scorers for the rest, and it is deliberately
* narrow: it decides what a decision MEANS, and the phase pipeline decides
* when to make one. Everything here is pure.
*
* ## 1. Proactive know-to-ask, with a false-fire term
*
* Measured by driving `decideRecallInject`
* (`src/core/brain/recall-inject.ts`) - an explicitly I/O-free
* function with an injectable retriever - over a controlled vault. Two
* things must be said plainly about what this measures.
*
* First, WHAT SHIPS IS NOT WHAT IS MEASURED. `decideRecallInject` sits
* behind `recall_inject_enabled`, which defaults OFF (`src/core/config.ts`).
* Calling the function directly is what makes the metric possible at all,
* but the number it produces describes a CAPABILITY the code has, not
* behaviour a default install delivers. The always-on injection path
* (`hooks/active-inject.ts`) makes no relevance decision whatsoever - it
* injects `Brain/active.md` wholesale every session - so there is nothing
* there to score.
*
* Second, THE GATE IS SCOREABLE, AND THE BOUND IS THE GRADED KEY. The
* structural predicate (`evaluateSurfacingGate`,
* `src/core/search/surfacing-gate.ts`) still fails open by design - every
* non-empty, non-duplicate, non-slash, non-shell prompt retrieves - so a
* rate computed over THAT predicate alone would read ~1 by construction.
* Admission does not ride that predicate: it rides the shared graded
* surfacing key (`src/core/search/surfacing-key.ts`), channel-calibrated
* absolute match quality (`idfWeightedCoverage`) compared against the
* floor, which is a measured bound rather than a structural tautology.
* The surfacing-replay suite (`src/core/bench/replay.ts`, `--suite
* surfacing-replay`) is the instrument that scores it: it replays a
* recorded prompt corpus through the shipped decision core and posts the
* gate's false-fire and know-to-ask rates under the same
* {@link scoreProactiveRecall} denominations, so the calibration is a
* measured number that a re-run of the suite re-measures. The anti-gaming
* term still bites: {@link scoreProactiveRecall} counts a false fire
* exactly as heavily as a silent miss, which is what makes a strategy
* that always injects score strictly worse than one that abstains
* correctly.
*
* A measured caveat that used to belong with the number, kept because it
* is what the floor's current shape is a response to: through the shipped
* keyword lane the top-ranked result's SCORE sits at or above the
* configured keyword weight (`DEFAULT_KEYWORD_WEIGHT`,
* `src/core/search/index.ts`) whenever that lane is non-empty, because
* `rankResults` min-max normalizes it and then adds nine non-negative
* layers on top - measured at `FRESH_KEYWORD_ONLY_TOP_SCORE` (0.65) on a
* freshly written keyword-only vault, and higher still where a link or
* entity boost fires. While the floor was compared against that score,
* `below_floor` was unreachable via the default retriever and the abstain
* that actually fired on an off-topic prompt was `no_matches` - the term
* under pressure was not the term that fired. The floor now reads
* `idfWeightedCoverage`, which grades the match rather than the pool's
* shape, so a weak match reaches `below_floor` and this bench grades the
* bound it says it grades.
*
* Both figures are named rather than written out here, and
* `tests/core/search/keyword-lane-top-score.test.ts` drives the shipped
* pipeline and asserts what it measures against those two bindings - that
* test, not this sentence, is what keeps the stated number from drifting
* from the shipped one. It drifted once: this paragraph read 0.65 before
* the branch that rewrote it, the rewrite substituted 0.6, and 0.6 is the
* floor rather than the measurement.
*
* ## 2. Write-back fidelity
*
* Graded on the shipped write path with no model call, because there is no
* other kind: the server never calls a model on any write path by stated
* architectural contract (`src/mcp/brain/context-tools.ts`, "The calling
* agent generates; the Brain never does"). So there is no extractor seam
* to inject - the question a fidelity fixture actually answers is what
* structured intake stands in for a real agent's distillation, and whether
* the PROVENANCE survives segmentation, insertion and dedup intact.
*
* ## 3. Cross-source isolation
*
* Asserted on delivered pack CONTENTS, never on `hiddenByOwnerScope`: that
* counter is surfaced only under `warn`, where it counts items that WERE
* delivered, and under `fail` it is deliberately unreported. And it is run
* under a fixture that sets `integrity.owner_scope_delivery: fail`,
* because the shipped default is `off` - with no scope evaluated, nothing
* can be violated and a zero-gate would be vacuous.
*
* ## 4. Injected tokens
*
* Not a failure mode but a cost, so it stays in the report's cost family
* rather than here - see `context_cost.avg_injected_tokens`, which the
* pipeline computes with {@link estimateTokens} and nothing else.
*/
import { estimateTokens } from "../brain/text/tokenizer.ts";
import type { RecallInjectDecision, RecallResultSet } from "../brain/recall-inject.ts";
import { resolveSearchConfig, search } from "../search/index.ts";
import { TRANSPORT_REACH } from "../graph/transport-reach.ts";
/**
* How one proactive-recall decision failed, or nothing when it was right.
*
* Three members because three different repairs apply. A silent miss means
* the retrieval or the floor is too strict; a false fire means the
* opposite; a fault means the retriever itself broke, which is neither and
* must never be folded into either - a harness counting its own breakage
* as a memory failure reports a number about itself.
*/
export const RECALL_FAILURE = Object.freeze({
/** Memory held the answer and stayed silent. */
knowToAsk: "know_to_ask",
/** Memory spoke up when it had nothing worth saying. */
falseFire: "false_fire",
/** The decision core reported an error rather than a decision. */
faulted: "faulted",
} as const);
/** Closed union over {@link RECALL_FAILURE}. */
export type RecallFailure = (typeof RECALL_FAILURE)[keyof typeof RECALL_FAILURE];
/** Membership list; every surface renders its vocabulary from this array. */
export const RECALL_FAILURES: ReadonlyArray<RecallFailure> = Object.freeze([
RECALL_FAILURE.knowToAsk,
RECALL_FAILURE.falseFire,
RECALL_FAILURE.faulted,
]);
/** Narrow a value read back off disk or across a tool boundary. */
export function isRecallFailure(value: unknown): value is RecallFailure {
return typeof value === "string" && (RECALL_FAILURES as ReadonlyArray<string>).includes(value);
}
/**
* The retriever the bench hands to `decideRecallInject`: the shipped
* search pipeline over the disposable fixture vault.
*
* Mirrors `defaultRecallRetriever`, which adapts `searchAcrossVaults` the
* same way; the difference is only which search entry point is wrapped,
* because a bench vault has no registered recall sources to cross.
*/
export function benchRecallRetriever(
config: ReturnType<typeof resolveSearchConfig>,
limit: number,
): (query: string) => Promise<RecallResultSet> {
return async (query) => {
// Internal measurement lane: nothing here reaches a remote caller, so
// it measures the whole corpus. A lane that stopped seeing reserved
// pages would report a smaller vault than the one it is measuring.
const outcome = await search(config, {
query,
limit,
transportReach: TRANSPORT_REACH.local,
});
return Object.freeze({
candidates: Object.freeze(
outcome.results.map((result) =>
Object.freeze({
path: result.path,
title: result.title,
score: result.score,
searchType: result.searchType,
startLine: result.startLine,
endLine: result.endLine,
}),
),
),
total: outcome.total,
idfWeightedCoverage: outcome.idfWeightedCoverage,
});
};
}
/**
* What one decision costs the score, given what the fixture expected.
*
* `null` is a pass. An `error` decision is never charged to either rate:
* it is the harness failing, and attributing it to memory would make a
* broken retriever look like a cautious one.
*/
export function classifyRecallDecision(
expectInject: boolean,
decision: RecallInjectDecision,
): RecallFailure | null {
if (decision.kind === "error") return RECALL_FAILURE.faulted;
if (decision.kind === "inject") return expectInject ? null : RECALL_FAILURE.falseFire;
return expectInject ? RECALL_FAILURE.knowToAsk : null;
}
/** The two proactive-recall rates over a set of classified decisions. */
export interface ProactiveRecallScore {
readonly decisions: number;
readonly know_to_ask_failure_rate: number;
readonly false_fire_rate: number;
readonly faults: number;
}
/**
* Score a run of classified decisions.
*
* Both rates are denominated in DECISIONS, not in their own expectation
* class, and that is the anti-gaming term. Per-class denominators would
* let a strategy that always injects post a know-to-ask rate of 0 and hide
* its false fires in a rate nobody reads; sharing one denominator means
* every false fire displaces a pass, so the always-inject strategy is
* strictly worse than one that abstains correctly whenever the fixture
* contains a single prompt memory should stay quiet about.
*/
export function scoreProactiveRecall(
failures: ReadonlyArray<RecallFailure | null>,
): ProactiveRecallScore {
const decisions = failures.length;
const count = (kind: RecallFailure): number =>
failures.filter((failure) => failure === kind).length;
return Object.freeze({
decisions,
know_to_ask_failure_rate: rate(count(RECALL_FAILURE.knowToAsk), decisions),
false_fire_rate: rate(count(RECALL_FAILURE.falseFire), decisions),
faults: count(RECALL_FAILURE.faulted),
});
}
/**
* The forbidden ids a pack actually delivered.
*
* Deliberately reads the delivered CONTENTS rather than a counter: the one
* counter that exists (`hiddenByOwnerScope`) is reported only under `warn`,
* where its meaning is inverted, and is withheld under `fail` on purpose.
* A zero-gate has to look at what came back.
*/
export function isolationViolations(
deliveredIds: ReadonlyArray<string>,
forbiddenIds: ReadonlyArray<string>,
): ReadonlyArray<string> {
return Object.freeze(forbiddenIds.filter((id) => deliveredIds.includes(id)));
}
/**
* The single token estimator this suite is allowed to use.
*
* Re-exported by name so the bench has exactly one import site to point at
* and `tests/core/bench/token-estimator.test.ts` has exactly one thing to
* assert. Five mutually disagreeing estimators exist in this repo
* (`ceil(utf8_bytes/4)`, `ceil(utf16_len/4)`, a word count, an inline
* `ceil(chars/4)` the bench used to carry, and a ledger over integers a
* caller supplied); the bench converges on {@link estimateTokens} because
* it is the one the Brain-side accounting already uses everywhere and the
* only one that documents its own heuristic.
*/
export function injectedTokens(injected: string): number {
return estimateTokens(injected);
}
function rate(count: number, total: number): number {
return total > 0 ? Math.round((count / total) * 1000) / 1000 : 0;
}