Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
evals/ — evaluation harnesses
One Rust binary (ai-memory-eval) with four subcommands:
ab— runs the EXACT consolidation prompt ai-memory uses in production against two LLM providers side by side, saves both outputs to disk for human comparison, and scores literal fidelity (see Literal fidelity).projection— the zero-LLM half of that scoring: which fixture facts the observation projection lets into the prompt at all. No provider, no network.retrieval— LongMemEval and bundled coding-session retrieval checks, driven through a realai-memory servesubprocess. Published baselines live indocs/benchmarks/.abstention— offline evidence-sufficiency signals and threshold sweeps over versioned answerable, hard-negative, conflicting, partial, and missing-channel fixtures, with optional structured LLM answerability.
This is not part of the shipped binary. It's a workspace member
purely so it shares deps + builds with the rest. cargo build from
the repo root will compile it, but it's never bundled into the
docker image. CI builds it and runs its unit tests
(cargo nextest run --workspace --all-targets); the ab, projection,
retrieval, and abstention commands themselves never run there.
abstention — evidence sufficiency
The default run is deterministic, offline, and keyless:
cargo run -p ai-memory-eval -- abstention
It reads evals/fixtures/abstention/v1.json, validates the version, unique
ids, evidence paths, finite score/belief ranges, and required support for
answerable cases, then writes report.json and report.md under a timestamped
evals/runs/*-abstention/ directory. Each case records raw normalized lexical
coverage, top-score margin/distribution, matching-term density, snippet size,
and belief aggregation only where the fixture actually supplies belief.
The report sweeps lexical-coverage thresholds from 0.0 through 1.0 and records
coverage, false-supported rate (FSR), and false-abstention rate (FAR), including
undefined denominators. It does not designate a calibrated threshold: the
clean-room corpus is a regression probe, not a representative sample.
answerable maps to supported and unanswerable to unsupported;
conflicting and partial remain separate slices and are excluded from both
binary error-rate denominators.
The fixture complements rather than relabels LongMemEval. To add the pinned
LongMemEval-S *_abs questions as unanswerable cases, first fetch that dataset
through the retrieval command, then opt in without any network access:
cargo run -p ai-memory-eval -- retrieval --fetch --sample 1
cargo run -p ai-memory-eval -- abstention --longmemeval-abs
Those cases use the evaluator's existing whole-session grep baseline, not the production server: each question receives at most the top 10 term-overlap sessions, with normalized overlap scores and each rendered session clipped to 4,000 characters. This keeps optional QA inputs bounded and makes the signal's provenance explicit; it does not claim to measure production retrieval.
Optional --qa uses the exact --qa-* provider/auth construction shared with
the retrieval evaluator. Synthesis returns a structured answerability enum
(answerable, insufficient, conflicting, or partial); the evaluator never
detects abstention from answer wording. Missing credentials produce a reported
skip and leave the offline signal report intact. The retrieval QA path also
parses production's answer: { text, citations } object; empty validated
citations are an abstention, while nonempty citations are answerable for this
evaluation metadata. This is still identity evidence, not a claim-level
support proof.
When adding cases, keep them clean-room and synthetic. target_evidence must
name supplied paths, answerable cases require a gold_answer, and channel-loss
cases declare channels.fts / channels.vector explicitly. Do not tune a
threshold and the fixture against each other, and do not fold conflicting or
partial evidence into binary ground truth without changing the documented
mapping and report schema.
When to reach for this
After switching providers, models, or major prompt edits. Concretely:
- Replacing OpenRouter/Kimi with local Ollama (the case that motivated this harness — see commit log).
- Trying a new Ollama model (
qwen3:32b→qwen3-coder:30b). - Tuning the
BATCH_SYSTEM_PROMPTitself — does the rewrite preserve quality across providers?
The fixtures are deliberately small synthetic session logs that exercise the prompt's hard cases (durable rule extraction, multi-topic separation, "say nothing" sessions, decision/gotcha distinction).
What it does
For each *.json under evals/fixtures/:
- Builds the request via
[
ai_memory_consolidate::build_batch_request] — same code path the live consolidator uses. - Sends it to a baseline and a candidate provider, in parallel.
- Runs the result through
[
ai_memory_llm::complete_structured] — same JSON-schema validation the live system applies. Schema-parse failure is recorded (not fatal). - Persists to
evals/runs/<timestamp>/{baseline,candidate}/<fixture>.{json,md,meta.json}. - Scores each output against the fixture's
must_mentionandmust_not_mentionlists and writesreport.json.
The runner prints latency, parse status and the fidelity numbers per
fixture and run, and a tail summary. Structure and judgement are
still for you to read: open the markdown files in
runs/<timestamp>/baseline/ and …/candidate/ side by side.
Running it
Ollama qwen3:32b vs OpenRouter Kimi (the canonical comparison)
# OPENROUTER_API_KEY in env; LLM_API_KEY can be any non-empty
# string for the candidate (Ollama doesn't validate).
export OPENROUTER_API_KEY="sk-or-v1-..."
cargo run -p ai-memory-eval -- ab \
--baseline-provider openai-compat \
--baseline-base-url https://openrouter.ai/api/v1 \
--baseline-model moonshotai/kimi-k2.6 \
--baseline-api-key-env OPENROUTER_API_KEY \
--candidate-provider openai-compat \
--candidate-base-url http://192.168.0.90:11434/v1 \
--candidate-model qwen3:32b \
--candidate-api-key ollama-local
Two Ollama models against each other
cargo run -p ai-memory-eval -- ab \
--baseline-provider openai-compat \
--baseline-base-url http://192.168.0.90:11434/v1 \
--baseline-model qwen3:32b \
--baseline-api-key ollama-local \
--candidate-provider openai-compat \
--candidate-base-url http://192.168.0.90:11434/v1 \
--candidate-model qwen3-coder:30b \
--candidate-api-key ollama-local
ChatGPT/Codex OAuth as one side
Run ai-memory auth login openai-oauth first, then point the eval harness at
the same token file:
cargo run -p ai-memory-eval -- ab \
--baseline-provider openai-oauth \
--baseline-token-file ~/.local/share/ai-memory/auth.json \
--baseline-model gpt-5.5 \
--candidate-provider openai-compat \
--candidate-base-url http://192.168.0.90:11434/v1 \
--candidate-model qwen3:32b \
--candidate-api-key ollama-local
GitHub Copilot as one side
Run ai-memory auth login copilot first, then point the eval harness at the
same auth file:
cargo run -p ai-memory-eval -- ab \
--baseline-provider copilot \
--baseline-token-file ~/.local/share/ai-memory/auth.json \
--baseline-model gpt-5.5 \
--candidate-provider openai-compat \
--candidate-base-url http://192.168.0.90:11434/v1 \
--candidate-model qwen3:32b \
--candidate-api-key ollama-local
Reading the output
evals/runs/2026-05-22T18-30-00Z/
├── baseline/
│ ├── 01-rust-bug-fix.json ← raw structured output
│ ├── 01-rust-bug-fix.md ← flat markdown rendering, easy to read
│ └── 01-rust-bug-fix.meta.json ← {elapsed_ms, parsed_ok, update_count, fidelity}
├── candidate/
│ └── …
├── requests/01-rust-bug-fix.txt ← the exact system prompt + user message sent
├── generated-session.json ← with --generated-seed
└── report.json ← provenance + every run of both sides
With --runs N the per-case files carry a .rN suffix. Eyeball the
.md files in pairs. The runner also prints a
diff -ru baseline candidate command you can run.
Literal fidelity
Consolidation can lose a fact at two stages, and the fix differs:
- projection: the observation projection omits whole observations and cuts bodies at 3,000 characters to fit the budget. A fact it drops never reaches the model, so no prompt wording can bring it back.
- model: the fact was in the prompt and the page does not carry it.
Each fixture may list must_mention (literal strings the pages should
carry) and must_not_mention (strings no page may carry: injected
instructions, credential-shaped values). Matching is exact,
case-sensitive substring, so a paraphrase counts as a loss; that is
what "verbatim" means in the prompts. The scores, per fixture and run:
| Metric | Meaning |
|---|---|
projection_loss |
share of must_mention absent from the prompt (zero-LLM) |
llm_loss |
share of the prompt-reaching must_mention absent from the pages |
error_line_observation_coverage |
share of observations carrying a mined error line whose error line reaches the request |
file_error_line_share |
share of request-reaching error lines that came from file-shaped tool output |
recall_prompt / recall_page |
keep-token recall (mine_keep_tokens) of the prompt and the pages; a broad secondary view |
hallucinated_tokens |
keep-tokens on a page found in neither the request nor the batch's own paths |
invented_wikilinks |
[[links]] to pages outside the batch; the prompt lists no existing pages, so each one is invented (#1207) |
must_not_mention_hits |
forbidden strings found on a page |
The must_mention lists are the primary measure because they do not
depend on any token miner. A change that adds miner tokens to the
prompt raises recall_prompt by construction, so it cannot be judged
by that number.
--runs N (up to 20) repeats every side. The spread between a side's
own runs, printed as mean (min-max), is the noise band, and a delta
inside it is noise. Run the same model on both sides to measure
the band on its own.
--generated-seed 1215 adds the seeded long session from
src/generate.rs: 300 observations in production capture shapes with
fifteen planted facts. Seven sit where the default budget keeps them;
eight sit where the projection drops them (early in the session, in
output with no high-signal term, or past the excerpt cap of a pasted
prompt). Where a fact sits is a choice, so this session's
projection_loss describes the fixture, not real sessions; what it
supports is a before/after comparison. report.json records the seed
and the sha256 of the generated JSON next to the numbers; a test pins the
sha256 of seed 1215, so a generator change that alters it has to update the
pin and say so beside any republished number.
Two differences from a live server: fixtures and the generated session reach the projection without passing through the capture sanitizer (they are synthetic, and the injection fixture keeps its planted strings on purpose), and they are projected at the harness's 300,000-character default budget, not the server's 240,000-character one (100,000 tokens × 3 characters × the 0.8 safety margin). Compare runs against each other, not against a server's prompts.
Zero-LLM, no provider:
cargo run -p ai-memory-eval -- projection --generated-seed 1215
# compare the default projection with the opt-in residue, without a provider:
cargo run -p ai-memory-eval -- projection --generated-seed 1215 --residue
# also write every request and the generated session to a directory:
cargo run -p ai-memory-eval -- projection --generated-seed 1215 --dump evals/runs/projection
Point --dump and the fixtures at synthetic sessions only. A request
holds every observation verbatim, and real sessions do not belong in
files you may share.
For the model half, ab --residue keeps the baseline request at the
byte-compatible default and enables the residue only for the candidate. Use
the same provider and model on both sides and --runs 3; the within-side
spread is the noise band. The JSON report records the candidate projection
separately and sets verbatim_residue_ab so published results cannot be
mistaken for a provider comparison.
Adding fixtures
Each fixture is a JSON file:
{
"name": "human-readable",
"description": "what this case is meant to surface",
"observations": [
{"kind": "session-start", "title": "...", "body": "..."},
{"kind": "user-prompt", "title": "user prompt", "body": "..."},
{"kind": "pre-tool-use", "title": "Edit", "body": "..."},
...
],
"must_mention": ["error[E0382]: borrow of moved value"],
"must_not_mention": ["IGNORE ALL PREVIOUS RULES"]
}
Both lists are optional.
kind values: session-start, user-prompt, pre-tool-use,
post-tool-use, pre-compact, notification, stop,
session-end, other (see ObservationKind in ai-memory-core).
The description field isn't read by the runner — it's a comment
for the next human who opens the file.
What this harness does NOT do
- Judge quality. The literal checks above say whether a string survived, not whether the page is any good. Structure, scoping and judgement are still read by a human; an LLM-as-judge pass is deliberately out of scope.
- Test the embedding pipeline. This only exercises the consolidation LLM. Embedding A/B would be a parallel harness (probe queries + expected target pages, measure recall@5/MRR).
- Persist via the real wiki layer. No SQLite, no markdown writes, no git. Pure prompt → response.
Cleanup
evals/runs/ is in .gitignore. Drop it whenever it gets large.
retrieval — LongMemEval benchmark
Measures the real retrieval stack against LongMemEval (v1, S variant, MIT license, 500 questions over ~50-session chat haystacks). Nothing is mocked:
- a real
ai-memory servesubprocess starts on a fresh temp data dir (zero-LLM: no consolidation LLM, no embedder, no reranker — fully deterministic and offline); - each question's haystack replays through
POST /hook/batchat the production hook cadence (session-start,user-prompt-submitper user turn,stopwith the opt-in assistant excerpt per assistant turn,session-end), one project per question; - the question runs through MCP
tools/call memory_querywith explicit workspace/project scoping; - results are scored session-level:
hit@k(any evidence session in the top k),recall@k(fraction of evidence sessions found), andrecall-all@k(all evidence sessions found). Duplicate session hits count once. The 30 abstention questions are excluded and reported separately.
The dataset (278 MB) is NOT committed; it downloads to
evals/datasets/ (gitignored) with a pinned sha256 that fails loudly
on upstream drift.
cargo build --release -p ai-memory-cli # the server under test
cargo run --release -p ai-memory-eval -- retrieval --fetch # full 500
cargo run -p ai-memory-eval -- retrieval --sample 10 # smoke
cargo run -p ai-memory-eval -- retrieval --question-id 07741c44,07741c45 # chosen questions
Output: a per-slice table on stdout plus
evals/runs/<timestamp>-retrieval/report.{json,md} with full
provenance (commit, dataset sha, hardware, per-question scores).
When publishing a baseline, copy the markdown into docs/benchmarks/.
Bundled coding-session retrieval corpus
--dataset-variant coding-v1 selects four synthetic questions in
evals/fixtures/retrieval/coding-v1.json. The file is committed and pinned by
SHA-256, so a zero-LLM run needs neither a dataset download nor a provider.
Use the existing server or grep engine and A/B reporting:
cargo build -p ai-memory-cli -p ai-memory-eval
cargo run -p ai-memory-eval -- retrieval --dataset-variant coding-v1 \
--server-bin target/debug/ai-memory --candidate --concurrency 1
cargo run -p ai-memory-eval -- retrieval --dataset-variant coding-v1 --engine grep
| Case | Expected evidence | Incorrect alternative |
|---|---|---|
coding-architecture |
The write-pool failure and the later actor/read-pool decision | Unrelated implementation sessions |
coding-resolved-error |
The observed database error and the retry-path fix | Unrelated operational sessions |
coding-correction |
The later 45-second timeout correction | The earlier 15-second proposal |
coding-long-session |
The exact checksum header at the end of a 41-turn investigation | The unsuccessful initial header |
Each question has twelve sessions, including distractors, and one or two
expected evidence sessions. answer_session_ids drives the existing scorer;
has_answer marks the relevant turns. incorrect_session_ids is corpus
metadata checked by the fixture test, not an extra scoring rule. It partitions
the remaining sessions and makes stale alternatives explicit for review.
The gold answer is available to the optional QA mode, which still needs a
separately configured provider.
These checks measure evidence retrieval: hit@k, recall@k, recall-all@k,
context-token estimates, and query latency. Retrieving the correction does not
prove that an agent follows it, and a stale hit is not a measured answer error.
The corpus does not exercise as_of, typed supersession, handoff claiming,
actual harness switching, private/shared authorization, or concurrent writes.
The existing integration suites cover those contracts. Every replay still uses
the evaluator's Claude Code hook shape and one project per question.
Run identical baseline/candidate settings to check accuracy and context-token
parity; wall-clock latency can vary. Treat the four cases as regression probes,
not representative quality estimates or numbers comparable to LongMemEval.
If a fixture changes, update its pin and record the new corpus version beside
any reported results. --fetch cannot download a missing bundled fixture;
restore it from the repository instead.
coding-v1 baseline receipt (2026-10-25 UTC)
Measured with debug evaluator and server binaries from
e5ac18d8a819a76990d079ee70329973dd1a2e55, based on release/3.3 at
e7e17555efb6a95596480249598ad517ff587310. The change adds evaluation inputs;
production code is unchanged. Both A/B sides used that same source and
configuration. The published source commit
7e9ea65b15215d50cb1593956fb57e45954d5347 has the same Git tree
(5b896e1bbe3550341f25f64529656b919ee16970) as that locally built commit.
This is a repeatability baseline, not a measured improvement.
- Corpus SHA-256:
af3382663a3e4e969c8bcfa744b3a794de86cb5932393a88bea16f014193d467. - Server binary SHA-256:
2f501060445ff2bae166117b8143622c9f3bee606a8aa4ce34911ba7c827f84a. - Evaluator binary SHA-256:
498ee8917655b71438111afc30936a97bb213f8c722fb8dd5635efd16dfed29f. - Rust
1.95.0 (59807616e 2026-04-14); Linux6.18.55, x86_64; AMD EPYC 9V74, eight available threads. - Three A/B trials, four questions and 48 sessions per side, concurrency one. Default hook ingestion and per-turn dates; embeddings and reranker off; no LLM provider or QA calls.
| Metric | Baseline | Identical candidate | Delta |
|---|---|---|---|
| hit@1 / hit@5 / hit@10 | 1.000 | 1.000 | 0.000 |
| recall@1 | 0.750 | 0.750 | 0.000 |
| recall-all@1 | 0.500 | 0.500 | 0.000 |
| recall@5 / recall@10 | 1.000 | 1.000 | 0.000 |
| recall-all@5 / recall-all@10 | 1.000 | 1.000 | 0.000 |
| Estimated context tokens, mean | 225.0 | 225.0 | 0.0 |
Every question's retrieval scores and context estimate matched across all six runs. Context tokens use the existing Unicode-character-count divided by four estimator over returned titles and snippets; no provider token count was taken.
| Trial | Baseline p50 / p95 (ms) | Candidate p50 / p95 (ms) |
|---|---|---|
| 1 | 8 / 8 | 7 / 8 |
| 2 | 10 / 11 | 8 / 25 |
| 3 | 7 / 9 | 8 / 12 |
Latency covers the MCP round trip. Each percentile uses only four questions, and unrelated compilation ran on the same host, so these timings are descriptive and do not establish a performance difference. A separate offline grep run on the same fixture found the same retrieval scores. Its zero latency/context fields are evaluator placeholders and are not server measurements.
Reproduce with the coding-v1 --candidate --concurrency 1 command above, three
times. The ignored coding_corpus_replays_offline_with_identical_ab_results
smoke test also asserts full evidence recall@10 and identical A/B scores/context;
it needs a server binary at target/release/ai-memory.
The triple: accuracy + latency + context-tokens
Every run — single-config or A/B — reports three things per slice, not just accuracy:
- accuracy —
hit@k/recall@k/recall-all@k; - latency —
memory_queryMCP round-trip p50 / p95 in ms; - context tokens — the context an agent would ingest from the result,
estimated as chars / 4 over each returned hit's
title+snippet(a documented heuristic, not a real tokenizer), reported mean / median.
retrieval A/B mode (R2)
Pass a candidate config and the same question set runs through two
fresh servers (baseline + candidate) and gains a baseline→candidate delta
table (markdown + JSON). A/B mode turns on with --candidate or any
--candidate-* knob; without one, the single-config path is unchanged.
Baseline knobs: --embeddings none|local (also = vectors off/on),
--reranker (sets AI_MEMORY_RERANKER=llm; needs a provider via
--server-env), --server-env KEY=VALUE (any server env, repeatable),
--query-arg KEY=JSON (extra memory_query args, repeatable — so future
knobs like pin_first / include_superseded can be A/B'd;
query/workspace/project/limit are reserved). Each has a
--candidate-* twin (--candidate-embeddings, --candidate-reranker,
--candidate-server-env, --candidate-query-arg); a candidate knob left
unset mirrors the baseline.
# Determinism check — baseline == candidate ⇒ accuracy/context delta = 0:
cargo run --release -p ai-memory-eval -- retrieval --candidate --sample 2
# Vectors off vs on, full run:
cargo run --release -p ai-memory-eval -- retrieval --fetch \
--candidate-embeddings local
# A future memory_query knob, baseline off vs candidate on:
cargo run --release -p ai-memory-eval -- retrieval --fetch \
--candidate-query-arg include_expired=true
Output: evals/runs/<timestamp>-retrieval/report.{json,md} — the JSON is
{ baseline, candidate, delta }; the markdown carries both per-slice
triples and an overall delta table. Publish full-dataset A/B numbers in
docs/benchmarks/retrieval-ab-r2.md.
Strict recall-all@k scores each question as 1 only when every evidence
session appears in the top k distinct retrieved sessions, otherwise 0.
Slice scores average those per-question values. Finding one of two required
sessions gives hit@k = 1, recall@k = 0.5, and recall-all@k = 0.
JSON reports use recall_all_at for per-question, slice and A/B delta values.
retrieval QA-accuracy mode (R2b, opt-in, live LLM)
R2a scores retrieval. --qa adds an opt-in end-to-end answer-quality
metric so the answer/reasoning MCP features can be proven later. It is
OFF by default and makes REAL LLM API calls, gated behind a key. Per
scored question, with QA on, the harness:
- gets a candidate answer — the server's own
answerfield when a futureanswer=truefeature populates it, otherwise a synthesis LLM call over the retrieved snippets (the exact context an agent would receive); - grades that candidate against the dataset's gold answer with an
LLM-as-judge (
{correct, reason}), LongMemEval-style; - reports QA accuracy (fraction correct) per slice — plus the answer-step latency (p50/p95) and answer-token estimate — alongside the R2a triple, in both the single-config and A/B reports. Provider+model names are recorded; keys never are.
Flags: --qa, --qa-provider <name>, --qa-model <model> (required to
enable), --qa-base-url, --qa-api-key/--qa-api-key-env (defaults to the
provider's canonical env var, e.g. GEMINI_API_KEY), --qa-token-file
(oauth/codex/copilot). The grader defaults to the answerer and is overridable
per field: --qa-grader-provider/-model/-base-url/-api-key/
-api-key-env/-token-file.
Gating: if --qa is set but the provider/key cannot be resolved, the run
prints a SKIP line and continues with QA off (retrieval metrics unaffected),
exiting success — so a keyless CI stays green. The default non-QA path stays
zero-LLM and deterministic.
# Keep keys in the process env only for the run; never echo a key.
set -a; source ~/.config/zsh/secrets; set +a # or export GEMINI_API_KEY=...
cargo run -p ai-memory-eval -- retrieval --sample 3 \
--qa --qa-provider gemini --qa-model gemini-2.5-flash
retrieval decomposition instruments
These flags split a hit@k number into evidence the index never saw and
ranking on evidence it did. All are opt-in; the defaults reproduce the
published baselines. The results are in
docs/benchmarks/longmemeval-s-decomposition-2026-10-12.md.
On a 3.0 server the observation stream is on by default, so reproducing a
2.6-era baseline also needs
--server-env AI_MEMORY_RETRIEVAL__OBSERVATION_STREAM=false.
--reachabilityreads every evidence session back after the query and reports, per slice, how many answer-bearing turns (the dataset'shas_answerflag) survive into the session page, the observation bodies, and the first 512 tokens the local embedder reads. A turn counts as reachable when at least half of its distinctive tokens (content words that occur in at most 3 sessions of the haystack) are on that surface. The module docs insrc/retrieval/reachability.rsgive the exact definition. It needs the local model's tokenizer, which it fetches intoevals/models/if missing.--engine grepranks each question's raw sessions by query-term overlap in-process, with no server. It follows agentmemory's grep adapter exactly and matches its archived per-question results.--ingest whole-sessionwrites each raw session as one page throughPOST /admin/write-pageand then queries as usual: ranking on full evidence.--ingest captured-sessionwrites one page per session holding only the text capture keeps (user turns up to 16 KiB, assistant excerpts up to 2 KB).--date-prefix turn|session-start|nonemoves the session date off every turn (turnis the default).--stratify Nscores the first N questions of each type, so a cheap run covers every slice (--sampletakes a prefix, which is all one type).--dataset-variant s-cleanedruns on the cleaned LongMemEval-S re-release (pinned by sha256) that other systems report on.
--candidate-ingest and --candidate-date-prefix put either knob into an
A/B run. The A/B delta table also counts per-question flips (+gained/-lost),
which shows whether a net delta is many changes cancelling out or a few in
one direction.
cargo run --release -p ai-memory-eval -- retrieval --engine grep
cargo run --release -p ai-memory-eval -- retrieval --reachability \
--ingest hooks --candidate-ingest whole-session
cargo run --release -p ai-memory-eval -- retrieval --stratify 5 \
--date-prefix turn --candidate-date-prefix session-start
Honest-numbers notes:
- capture is production-shaped: excerpts are bounded at the 2 KB
privacy boundary. The decomposition measured what that costs: pages
holding exactly the captured text score 0.968 FTS
hit@5against 0.972 for whole raw sessions. The larger cost was the zero-LLM session page, which indexed only 80-character turn titles; the observation stream (on by default in 3.0) puts the observation bodies back into ranking; - each session's original date is prepended to its turn text, since a
replayed history must carry its own timestamps (the official
harness exposes the same information to retrievers). The prefix is 39
characters, about half of a turn title, and production capture adds
none, so the default
--date-prefix turnunderstates the pipeline on this content (+0.133 FTShit@5on a 120-question sample with the date moved to session start). It stays the default so earlier baselines remain reproducible; - other systems' published numbers on this dataset (e.g. agentmemory's
95.2%
recall_any@5, mcp-memory-service 0.804 R@5) use the same statistic as ourhit@5. agentmemory's indexes whole raw sessions directly, a component measurement rather than a pipeline one, so it is not like-for-like with the default--ingest hooksnumbers. Its 0.967 is a different benchmark (the grep baseline on its 15-query coding-agent-life-v1 corpus), not LongMemEval-S.