Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(evals): R2 retrieval A/B harness with accuracy+latency+context-tokens triple
AkitaOnRails926c23e43c2f
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.mdindex 78e565b1..95c17e81 100644--- a/docs/benchmarks/README.md+++ b/docs/benchmarks/README.md@@ -28,6 +28,13 @@ harness before the next change landed. For context, publishedembedding-based numbers on this dataset: agentmemory 0.967 R@5(hybrid + reranking), doobidoo/mcp-memory-service 0.804 R@5.+## A/B comparisons (R2)++The harness can also A/B two configs over the same question set and report+an **accuracy + latency + context-tokens** triple with a baseline→candidate+delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). Full-dataset A/B+baselines are published there once run.+## Reading the numbers- **mode: local embeddings** is the 2.0 default: the in-processdiff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.mdnew file mode 100644index 00000000..4b93220f--- /dev/null+++ b/docs/benchmarks/retrieval-ab-r2.md@@ -0,0 +1,56 @@+# Retrieval A/B (R2) — accuracy + latency + context-tokens triple++The `retrieval` harness can compare two server/query configurations over+the **same** LongMemEval-S question set in one run, reporting a triple per+config and a baseline→candidate delta:++- **accuracy** — `hit@k` / `recall@k` (as the single-config baselines do);+- **latency** — `memory_query` MCP round-trip p50 / p95, milliseconds;+- **context tokens** — the context an agent would ingest from a result,+ estimated as **chars / 4** over every returned hit's `title` + `snippet`+ (a documented, provider-agnostic heuristic — not a real tokenizer).++Provenance (commit, dataset sha256, hardware, per-config knob summary) is+carried exactly as the single-config reports carry it. See+`evals/README.md` for the full flag reference.++## How to run++```bash+cargo build --release -p ai-memory-cli+# baseline (FTS-only) vs candidate (local embeddings), full 500 questions:+cargo run --release -p ai-memory-eval -- retrieval --fetch \+ --candidate-embeddings local+```++A run without a candidate config is unchanged and still writes the+single-config report. A baseline-vs-baseline run (`--candidate`, no+differing knob) is the determinism check: accuracy and context-token+deltas are exactly zero (latency wobbles at wall-clock noise).++## Published A/B numbers++> Full-dataset (500-question) A/B baselines go here, one section per+> comparison, pasted verbatim from `evals/runs/<stamp>-retrieval/report.md`+> with its provenance header. **Not yet populated** — running the full+> matrix is a deliberate, separately-scheduled pass (the dataset is 278 MB+> and a full local-embeddings leg is minutes, not seconds). Do not paste+> `--sample` numbers here as if they were baselines.++### Illustrative smoke (NOT a baseline)++Recorded only to show the report shape. `--sample 4`, four+`single-session-user` questions, so the accuracy figures are noise;+latency and context-tokens show the expected direction (local embeddings+add query latency):++| metric | baseline (none) | candidate (local) | delta |+|---|---|---|---|+| hit@5 | 0.750 | 0.500 | -0.250 |+| latency p50 ms | 2 | 31 | +29 |+| latency p95 ms | 2 | 35 | +33 |+| ctx tok mean | 408.5 | 393.2 | -15.2 |+| ctx tok median | 494.5 | 389.0 | -105.5 |++Hardware: AMD Ryzen 9 7950X3D (32 threads). Replace this section with a+full-dataset run before citing any A/B number.diff --git a/evals/README.md b/evals/README.mdindex e0199b5a..3f946de6 100644--- a/evals/README.md+++ b/evals/README.md@@ -201,11 +201,57 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch # full 500cargo run -p ai-memory-eval -- retrieval --sample 10 # smoke```-Output: a per-category table on stdout plus+Output: a per-slice table on stdout plus`evals/runs/<timestamp>-retrieval/report.{json,md}` with fullprovenance (commit, dataset sha, hardware, per-question scores).When publishing a baseline, copy the markdown into `docs/benchmarks/`.+### The triple: accuracy + latency + context-tokens++Every run — single-config or A/B — reports three things per slice, not+just accuracy:++- **accuracy** — `hit@k` / `recall@k` (unchanged);+- **latency** — `memory_query` MCP round-trip **p50 / p95** in ms;+- **context tokens** — the context an agent would ingest from the result,+ estimated as **chars / 4** over each returned hit's `title` + `snippet`+ (a documented heuristic, not a real tokenizer), reported **mean / median**.++### `retrieval` A/B mode (R2)++Pass a **candidate** config and the same question set runs through two+fresh servers (baseline + candidate) and gains a baseline→candidate delta+table (markdown + JSON). A/B mode turns on with `--candidate` or any+`--candidate-*` knob; without one, the single-config path is unchanged.++Baseline knobs: `--embeddings none|local` (also = vectors off/on),+`--reranker` (sets `AI_MEMORY_RERANKER=llm`; needs a provider via+`--server-env`), `--server-env KEY=VALUE` (any server env, repeatable),+`--query-arg KEY=JSON` (extra `memory_query` args, repeatable — so future+knobs like `pin_first` / `include_superseded` can be A/B'd;+`query`/`workspace`/`project`/`limit` are reserved). Each has a+`--candidate-*` twin (`--candidate-embeddings`, `--candidate-reranker`,+`--candidate-server-env`, `--candidate-query-arg`); a candidate knob left+unset mirrors the baseline.++```bash+# Determinism check — baseline == candidate ⇒ accuracy/context delta = 0:+cargo run --release -p ai-memory-eval -- retrieval --candidate --sample 2++# Vectors off vs on, full run:+cargo run --release -p ai-memory-eval -- retrieval --fetch \+ --candidate-embeddings local++# A future memory_query knob, baseline off vs candidate on:+cargo run --release -p ai-memory-eval -- retrieval --fetch \+ --candidate-query-arg include_expired=true+```++Output: `evals/runs/<timestamp>-retrieval/report.{json,md}` — the JSON is+`{ baseline, candidate, delta }`; the markdown carries both per-slice+triples and an `overall` delta table. Publish full-dataset A/B numbers in+`docs/benchmarks/retrieval-ab-r2.md`.+Honest-numbers notes:- capture is production-shaped: excerpts are bounded at the 2 KBdiff --git a/evals/src/main.rs b/evals/src/main.rsindex ca38b7cb..dc4270d5 100644--- a/evals/src/main.rs+++ b/evals/src/main.rs@@ -31,7 +31,7 @@ enum Command {/// consolidation prompt.Ab(Box<ab::AbArgs>),/// LongMemEval retrieval benchmark through the real server stack.- Retrieval(retrieval::RetrievalArgs),+ Retrieval(Box<retrieval::RetrievalArgs>),}#[tokio::main]@@ -45,6 +45,6 @@ async fn main() -> Result<()> {match Cli::parse().command {Command::Ab(args) => ab::run(*args).await,- Command::Retrieval(args) => retrieval::run(args).await,+ Command::Retrieval(args) => retrieval::run(*args).await,}}diff --git a/evals/src/retrieval/mod.rs b/evals/src/retrieval/mod.rsindex 88f6774c..088715b2 100644--- a/evals/src/retrieval/mod.rs+++ b/evals/src/retrieval/mod.rs@@ -5,6 +5,16 @@//! score session-level hit@k / recall@k against the dataset's evidence//! labels. See `docs/benchmarks/` for published baselines and//! `evals/README.md` for how to run.+//!+//! ## R2: A/B + the accuracy/latency/context triple+//!+//! With a candidate config (`--candidate` plus any `--candidate-*` knob),+//! the same question set runs through TWO server configs — a baseline and+//! a candidate — and the report gains a baseline→candidate delta. Every+//! run, single or A/B, now reports a triple per slice: retrieval accuracy+//! (hit@k / recall@k), `memory_query` latency (p50/p95), and the context+//! tokens a result would cost the agent (chars/4 over returned snippets).+//! Without a candidate config the single-config path is unchanged.mod dataset;mod ingest;@@ -21,7 +31,7 @@ use jiff::Timestamp;use dataset::Question;use ingest::{project_for, stored_session_uuid};-use server::EvalServer;+use server::{EvalEmbeddings, EvalServer, LaunchConfig};mod server;@@ -61,11 +71,177 @@ pub struct RetrievalArgs {#[arg(long)]keep_data_dir: bool,- /// Embedding mode: `none` (zero-LLM, the default) or `local`- /// (in-process all-MiniLM-L6-v2; the model is fetched once into- /// `evals/models/`, checksum-pinned).+ // ---- baseline config knobs ----+ /// Baseline embedding mode: `none` (zero-LLM, vectors off, the+ /// default) or `local` (in-process all-MiniLM-L6-v2, vectors on; the+ /// model is fetched once into `evals/models/`, checksum-pinned).#[arg(long, default_value = "none")]embeddings: String,++ /// Turn the post-RRF reranker on for the baseline+ /// (`AI_MEMORY_RERANKER=llm`). Requires an LLM provider, which you+ /// must supply via `--server-env` — otherwise the server ignores it.+ #[arg(long)]+ reranker: bool,++ /// Extra server env for the baseline, `KEY=VALUE`, repeatable. The+ /// general escape hatch for any server knob (reranker provider,+ /// retrieval flags, future toggles).+ #[arg(long = "server-env", value_name = "KEY=VALUE")]+ server_env: Vec<String>,++ /// Extra `memory_query` argument for the baseline, `KEY=JSON`,+ /// repeatable (e.g. `--query-arg include_expired=true`). The value is+ /// parsed as JSON, falling back to a string. Lets future query knobs+ /// (`pin_first`, `include_superseded`, …) be A/B'd without new flags.+ /// `query`/`workspace`/`project`/`limit` are reserved by the harness.+ #[arg(long = "query-arg", value_name = "KEY=JSON")]+ query_arg: Vec<String>,++ // ---- candidate config knobs (A/B mode) ----+ /// Enable A/B mode with a candidate config. Implied by any+ /// `--candidate-*` knob. With no other candidate knob the candidate+ /// mirrors the baseline — the determinism check (delta ≈ 0).+ #[arg(long)]+ candidate: bool,++ /// Candidate embedding mode (`none`/`local`); defaults to the+ /// baseline's `--embeddings`.+ #[arg(long)]+ candidate_embeddings: Option<String>,++ /// Turn the reranker on for the candidate.+ #[arg(long)]+ candidate_reranker: bool,++ /// Extra server env for the candidate, `KEY=VALUE`, repeatable.+ #[arg(long = "candidate-server-env", value_name = "KEY=VALUE")]+ candidate_server_env: Vec<String>,++ /// Extra `memory_query` argument for the candidate, `KEY=JSON`,+ /// repeatable.+ #[arg(long = "candidate-query-arg", value_name = "KEY=JSON")]+ candidate_query_arg: Vec<String>,+}++impl RetrievalArgs {+ /// A/B mode is on when `--candidate` is set or any candidate knob is.+ fn ab_mode(&self) -> bool {+ self.candidate+ || self.candidate_embeddings.is_some()+ || self.candidate_reranker+ || !self.candidate_server_env.is_empty()+ || !self.candidate_query_arg.is_empty()+ }+}++/// One resolved server + query configuration.+struct Knobs {+ /// Short label for reports (`baseline` / `candidate`).+ side: &'static str,+ embeddings: EvalEmbeddings,+ /// `zero-llm` or `local-embeddings`, for report provenance.+ mode: &'static str,+ models_root: Option<PathBuf>,+ reranker: bool,+ env: Vec<(String, String)>,+ query_args: serde_json::Map<String, serde_json::Value>,+ /// Human-readable knob summary for the report.+ summary: String,+}++impl Knobs {+ fn parse(+ side: &'static str,+ embeddings: &str,+ reranker: bool,+ server_env: &[String],+ query_arg: &[String],+ ) -> Result<Self> {+ let (embeddings_kind, mode, models_root) = match embeddings {+ "none" => (EvalEmbeddings::None, "zero-llm", None),+ "local" => (+ EvalEmbeddings::Local,+ "local-embeddings",+ Some(PathBuf::from("evals/models")),+ ),+ other => bail!("{side}: --embeddings must be `none` or `local`, got {other}"),+ };+ let env = parse_env(side, server_env)?;+ let query_args = parse_query_args(side, query_arg)?;++ let mut summary = format!(+ "embeddings={embeddings} reranker={}",+ if reranker { "on" } else { "off" }+ );+ for (k, v) in &env {+ summary.push_str(&format!(" env:{k}={v}"));+ }+ for (k, v) in &query_args {+ summary.push_str(&format!(" arg:{k}={v}"));+ }++ Ok(Self {+ side,+ embeddings: embeddings_kind,+ mode,+ models_root,+ reranker,+ env,+ query_args,+ summary,+ })+ }++ fn launch_config(&self) -> LaunchConfig<'_> {+ LaunchConfig {+ embeddings: self.embeddings,+ models_root: self.models_root.as_deref(),+ reranker: self.reranker,+ env: &self.env,+ }+ }+}++fn parse_env(side: &str, items: &[String]) -> Result<Vec<(String, String)>> {+ items+ .iter()+ .map(|item| {+ let (k, v) = item.split_once('=').ok_or_else(|| {+ anyhow::anyhow!("{side}: --server-env `{item}` must be KEY=VALUE")+ })?;+ if k.is_empty() {+ bail!("{side}: --server-env `{item}` has an empty key");+ }+ Ok((k.to_string(), v.to_string()))+ })+ .collect()+}++/// Reserved args the harness always sets; a config may not override them.+const RESERVED_QUERY_ARGS: [&str; 4] = ["query", "workspace", "project", "limit"];++fn parse_query_args(+ side: &str,+ items: &[String],+) -> Result<serde_json::Map<String, serde_json::Value>> {+ let mut map = serde_json::Map::new();+ for item in items {+ let (k, v) = item+ .split_once('=')+ .ok_or_else(|| anyhow::anyhow!("{side}: --query-arg `{item}` must be KEY=JSON"))?;+ if k.is_empty() {+ bail!("{side}: --query-arg `{item}` has an empty key");+ }+ if RESERVED_QUERY_ARGS.contains(&k) {+ bail!("{side}: --query-arg key `{k}` is reserved by the harness");+ }+ // Parse as JSON, else treat the raw text as a string value.+ let value = serde_json::from_str::<serde_json::Value>(v)+ .unwrap_or_else(|_| serde_json::Value::String(v.to_string()));+ map.insert(k.to_string(), value);+ }+ Ok(map)}pub async fn run(args: RetrievalArgs) -> Result<()> {@@ -99,30 +275,100 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {args.server_bin.display());}- let (embeddings, mode, models_root) = match args.embeddings.as_str() {- "none" => (server::EvalEmbeddings::None, "zero-llm", None),- "local" => {- let root = PathBuf::from("evals/models");- if !ai_memory_llm::model_present(&root) {- tracing::info!("fetching the local embedding model into evals/models (~87 MB)");- ai_memory_llm::fetch_model(&root).await?;- }- (- server::EvalEmbeddings::Local,- "local-embeddings",- Some(root),- )- }- other => bail!("--embeddings must be `none` or `local`, got {other}"),++ let baseline = Knobs::parse(+ "baseline",+ &args.embeddings,+ args.reranker,+ &args.server_env,+ &args.query_arg,+ )?;+ let candidate = if args.ab_mode() {+ let embeddings = args+ .candidate_embeddings+ .clone()+ .unwrap_or_else(|| args.embeddings.clone());+ Some(Knobs::parse(+ "candidate",+ &embeddings,+ args.candidate_reranker,+ &args.candidate_server_env,+ &args.candidate_query_arg,+ )?)+ } else {+ None};- let server = EvalServer::launch(- &args.server_bin,- args.keep_data_dir,- embeddings,- models_root.as_deref(),- )- .await?;- tracing::info!(url = server.base_url, data_dir = %server.data_dir_path.display(), "eval server up");++ // Fetch the local embedding model once if any config needs it.+ let needs_local = baseline.embeddings == EvalEmbeddings::Local+ || candidate+ .as_ref()+ .is_some_and(|c| c.embeddings == EvalEmbeddings::Local);+ if needs_local {+ let root = PathBuf::from("evals/models");+ if !ai_memory_llm::model_present(&root) {+ tracing::info!("fetching the local embedding model into evals/models (~87 MB)");+ ai_memory_llm::fetch_model(&root).await?;+ }+ }++ let abstention_n = abstention.len();+ let baseline_report = run_config(&args, &baseline, &questions, abstention_n, &commit).await?;++ // Timestamped run dir shared by single-config and A/B outputs.+ let stamp: String = Timestamp::now()+ .to_string()+ .chars()+ .map(|c| if c == ':' || c == '.' { '-' } else { c })+ .collect();+ let run_dir = args.out.join(format!("{stamp}-retrieval"));+ std::fs::create_dir_all(&run_dir)?;++ match candidate {+ None => {+ std::fs::write(+ run_dir.join("report.json"),+ serde_json::to_vec_pretty(&baseline_report)?,+ )?;+ let md = report::to_markdown(&baseline_report);+ std::fs::write(run_dir.join("report.md"), &md)?;+ println!("{md}");+ }+ Some(candidate) => {+ let candidate_report =+ run_config(&args, &candidate, &questions, abstention_n, &commit).await?;+ let ab = report::AbReport::new(baseline_report, candidate_report);+ std::fs::write(run_dir.join("report.json"), serde_json::to_vec_pretty(&ab)?)?;+ let md = report::ab_to_markdown(&ab);+ std::fs::write(run_dir.join("report.md"), &md)?;+ println!("{md}");+ }+ }+ println!("run dir: {}", run_dir.display());+ Ok(())+}++/// Launch one server config, replay + query + score the whole question+/// set through it, and build its [`report::Report`]. Each config gets a+/// fresh server (and fresh data dir) so ingestion never bleeds between+/// baseline and candidate.+async fn run_config(+ args: &RetrievalArgs,+ knobs: &Knobs,+ questions: &[Question],+ abstention_n: usize,+ commit: &str,+) -> Result<report::Report> {+ let server = EvalServer::launch(&args.server_bin, args.keep_data_dir, &knobs.launch_config())+ .await+ .with_context(|| format!("launching {} server", knobs.side))?;+ tracing::info!(+ side = knobs.side,+ url = server.base_url,+ data_dir = %server.data_dir_path.display(),+ config = knobs.summary,+ "eval server up"+ );let client = reqwest::Client::builder().timeout(std::time::Duration::from_secs(120))@@ -131,24 +377,31 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {let ks = Arc::new(args.ks.clone());let base_url = Arc::new(server.base_url.clone());let client = Arc::new(client);+ let query_args = Arc::new(knobs.query_args.clone());let semaphore = Arc::new(tokio::sync::Semaphore::new(args.concurrency.max(1)));let mut tasks = tokio::task::JoinSet::new();- for q in questions.clone() {+ for q in questions.iter().cloned() {let permit = semaphore.clone().acquire_owned().await?;- let (client, base_url, ks) = (client.clone(), base_url.clone(), ks.clone());+ let (client, base_url, ks, query_args) = (+ client.clone(),+ base_url.clone(),+ ks.clone(),+ query_args.clone(),+ );tasks.spawn(async move {let _permit = permit;let events = ingest::ingest_question(&client, &base_url, &q).await.with_context(|| format!("ingesting {}", q.question_id))?;- let retrieved = query::memory_query(+ let outcome = query::memory_query(&client,&base_url,ingest::EVAL_WORKSPACE,&project_for(&q),&q.question,max_k,+ &query_args,).await.with_context(|| format!("querying {}", q.question_id))?;@@ -157,12 +410,21 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {.iter().map(|s| stored_session_uuid(s)).collect();- let score =- score::score_question(&q.question_id, &q.question_type, &evidence, &retrieved, &ks);+ let score = score::score_question(+ &q.question_id,+ &q.question_type,+ &evidence,+ &outcome.retrieved,+ &ks,+ outcome.latency.as_millis(),+ outcome.context_tokens,+ );tracing::info!(question = q.question_id,events,- retrieved = retrieved.len(),+ retrieved = outcome.retrieved.len(),+ latency_ms = outcome.latency.as_millis() as u64,+ context_tokens = outcome.context_tokens,hit_at_max = score.hit_at.values().next_back().copied().unwrap_or(0.0),"scored");@@ -186,7 +448,8 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {tracing::error!("{f}");}bail!(- "{} of {} questions failed; no report written",+ "{} config: {} of {} questions failed; no report written",+ knobs.side,failures.len(),failures.len() + scores.len());@@ -194,34 +457,78 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {scores.sort_by(|a, b| a.question_id.cmp(&b.question_id));let slices = score::aggregate(&scores, &args.ks);- let stamp: String = Timestamp::now()- .to_string()- .chars()- .map(|c| if c == ':' || c == '.' { '-' } else { c })- .collect();- let run_dir = args.out.join(format!("{stamp}-retrieval"));- std::fs::create_dir_all(&run_dir)?;-- let rep = report::Report {+ Ok(report::Report {generated_at: Timestamp::now().to_string(),- commit,+ commit: commit.to_string(),hardware: report::hardware(),dataset: "longmemeval_s",dataset_sha256: dataset::LONGMEMEVAL_S_SHA256,- mode,+ mode: knobs.mode,+ config: knobs.summary.clone(),questions_scored: scores.len(),- abstention_excluded: abstention.len(),+ abstention_excluded: abstention_n,ks: args.ks.clone(),slices,per_question: scores,- };- std::fs::write(- run_dir.join("report.json"),- serde_json::to_vec_pretty(&rep)?,- )?;- let md = report::to_markdown(&rep);- std::fs::write(run_dir.join("report.md"), &md)?;- println!("{md}");- println!("run dir: {}", run_dir.display());- Ok(())+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn query_args_parse_json_and_string_and_reject_reserved() {+ let m = parse_query_args(+ "baseline",+ &[+ "include_expired=true".into(),+ "note=hello world".into(),+ "top=5".into(),+ ],+ )+ .unwrap();+ assert_eq!(m["include_expired"], serde_json::json!(true));+ assert_eq!(m["note"], serde_json::json!("hello world"));+ assert_eq!(m["top"], serde_json::json!(5));++ let err = parse_query_args("baseline", &["query=x".into()])+ .unwrap_err()+ .to_string();+ assert!(err.contains("reserved"), "{err}");+ let err = parse_query_args("baseline", &["bad".into()])+ .unwrap_err()+ .to_string();+ assert!(err.contains("KEY=JSON"), "{err}");+ }++ #[test]+ fn server_env_parses_key_value_pairs() {+ let e = parse_env(+ "baseline",+ &["AI_MEMORY_RERANKER=llm".into(), "A=b=c".into()],+ )+ .unwrap();+ assert_eq!(e[0], ("AI_MEMORY_RERANKER".into(), "llm".into()));+ // split_once keeps everything after the first `=` in the value.+ assert_eq!(e[1], ("A".into(), "b=c".into()));+ assert!(parse_env("baseline", &["nope".into()]).is_err());+ }++ #[test]+ fn knobs_summary_reflects_the_matrix() {+ let k = Knobs::parse(+ "candidate",+ "local",+ true,+ &["AI_MEMORY_LLM_PROVIDER=openai".into()],+ &["include_expired=true".into()],+ )+ .unwrap();+ assert!(k.summary.contains("embeddings=local"));+ assert!(k.summary.contains("reranker=on"));+ assert!(k.summary.contains("env:AI_MEMORY_LLM_PROVIDER=openai"));+ assert!(k.summary.contains("arg:include_expired=true"));+ assert_eq!(k.mode, "local-embeddings");+ }}diff --git a/evals/src/retrieval/query.rs b/evals/src/retrieval/query.rsindex 7588f29a..fd73e457 100644--- a/evals/src/retrieval/query.rs+++ b/evals/src/retrieval/query.rs@@ -3,6 +3,8 @@//! `workspace` + `project` args scope each call to one question's//! haystack (the documented pattern for static MCP clients).+use std::time::{Duration, Instant};+use anyhow::{Context, Result, bail};use serde::Deserialize;use serde_json::json;@@ -18,6 +20,25 @@ pub struct Retrieved {pub session_uuid: Option<uuid::Uuid>,}+/// Outcome of one `memory_query` call: the ranked hits plus the two+/// R2 observability signals — how long the round trip took and how much+/// context the agent would have to read to consume the result.+#[derive(Debug, Clone)]+pub struct QueryOutcome {+ /// Flattened ranked session attributions (index 0 = best).+ pub retrieved: Vec<Retrieved>,+ /// Wall-clock of the `memory_query` MCP round trip (request send →+ /// response parsed).+ pub latency: Duration,+ /// Estimated tokens an agent would ingest from this result: the total+ /// character count of every returned hit's `title` + `snippet`+ /// (pages and raw observations), divided by 4 (the documented+ /// chars/4 heuristic). The snippet is exactly what `memory_query`+ /// puts in front of the agent, so this is the context cost of the+ /// retrieval, not of the underlying full pages.+ pub context_tokens: usize,+}+#[derive(Debug, Deserialize)]struct QueryResponse {#[serde(default)]@@ -29,11 +50,27 @@ struct QueryResponse {#[derive(Debug, Deserialize)]struct PageHitLite {path: String,+ #[serde(default)]+ title: String,+ #[serde(default)]+ snippet: String,}#[derive(Debug, Deserialize)]struct RawHitLite {session_id: uuid::Uuid,+ #[serde(default)]+ title: String,+ #[serde(default)]+ snippet: String,+}++/// Divisor for the chars/4 token estimator (see [`QueryOutcome::context_tokens`]).+const CHARS_PER_TOKEN: usize = 4;++/// chars/4 token estimate for one string (Unicode scalar count).+fn estimate_tokens(chars: usize) -> usize {+ chars.div_ceil(CHARS_PER_TOKEN)}/// Call `memory_query` and flatten the response into ranked session@@ -46,21 +83,27 @@ pub async fn memory_query(project: &str,query: &str,limit: usize,-) -> Result<Vec<Retrieved>> {+ extra_args: &serde_json::Map<String, serde_json::Value>,+) -> Result<QueryOutcome> {+ // Extra config-supplied args first, then the scoping/limit the harness+ // controls, so a config can A/B a `memory_query` knob (e.g. a future+ // `pin_first`/`include_superseded`) without ever redirecting the query+ // off its question's private haystack.+ let mut arguments = extra_args.clone();+ arguments.insert("query".into(), json!(query));+ arguments.insert("workspace".into(), json!(workspace));+ arguments.insert("project".into(), json!(project));+ arguments.insert("limit".into(), json!(limit));let request = json!({"jsonrpc": "2.0","id": 1,"method": "tools/call","params": {"name": "memory_query",- "arguments": {- "query": query,- "workspace": workspace,- "project": project,- "limit": limit,- }+ "arguments": arguments,}});+ let started = Instant::now();let resp = client.post(format!("{base_url}/mcp")).bearer_auth(EVAL_AUTH_TOKEN)@@ -120,22 +163,32 @@ pub async fn memory_query(.ok_or_else(|| anyhow::anyhow!("memory_query returned no text content: {result}"))?;let parsed: QueryResponse =serde_json::from_str(text).with_context(|| format!("parsing memory_query JSON: {text}"))?;- Ok(flatten(parsed))+ let (retrieved, context_tokens) = flatten(parsed);+ Ok(QueryOutcome {+ retrieved,+ latency: started.elapsed(),+ context_tokens,+ })}-fn flatten(resp: QueryResponse) -> Vec<Retrieved> {+/// Flatten pages + raw hits into ranked session attributions, and sum the+/// context-token estimate over every hit's `title` + `snippet`.+fn flatten(resp: QueryResponse) -> (Vec<Retrieved>, usize) {let mut out = Vec::new();+ let mut chars = 0usize;for hit in resp.hits {+ chars += hit.title.chars().count() + hit.snippet.chars().count();out.push(Retrieved {session_uuid: session_uuid_from_path(&hit.path),});}for hit in resp.raw_hits {+ chars += hit.title.chars().count() + hit.snippet.chars().count();out.push(Retrieved {session_uuid: Some(hit.session_id),});}- out+ (out, estimate_tokens(chars))}/// `sessions/<uuid>.md` → the owning session; anything else → None.@@ -166,12 +219,48 @@ mod tests {let resp = QueryResponse {hits: vec![PageHitLite {path: format!("sessions/{a}.md"),+ title: String::new(),+ snippet: String::new(),+ }],+ raw_hits: vec![RawHitLite {+ session_id: b,+ title: String::new(),+ snippet: String::new(),}],- raw_hits: vec![RawHitLite { session_id: b }],};- let flat = flatten(resp);+ let (flat, _) = flatten(resp);assert_eq!(flat.len(), 2);assert_eq!(flat[0].session_uuid, Some(a));assert_eq!(flat[1].session_uuid, Some(b));}++ #[test]+ fn context_tokens_sum_title_and_snippet_over_all_hits() {+ let a = uuid::Uuid::now_v7();+ let b = uuid::Uuid::now_v7();+ let resp = QueryResponse {+ // 4 + 4 = 8 chars → 2 tokens+ hits: vec![PageHitLite {+ path: format!("sessions/{a}.md"),+ title: "abcd".into(),+ snippet: "efgh".into(),+ }],+ // 4 + 4 = 8 chars → +2 tokens+ raw_hits: vec![RawHitLite {+ session_id: b,+ title: "ijkl".into(),+ snippet: "mnop".into(),+ }],+ };+ let (_, tokens) = flatten(resp);+ assert_eq!(tokens, 4);+ }++ #[test]+ fn token_estimate_rounds_up() {+ // 5 chars / 4 = 1.25 → ceil 2+ assert_eq!(estimate_tokens(5), 2);+ assert_eq!(estimate_tokens(0), 0);+ assert_eq!(estimate_tokens(8), 2);+ }}diff --git a/evals/src/retrieval/report.rs b/evals/src/retrieval/report.rsindex 0308bb4b..390de7fd 100644--- a/evals/src/retrieval/report.rs+++ b/evals/src/retrieval/report.rs@@ -1,6 +1,13 @@//! Render benchmark results as JSON (machine) + markdown (human), with//! enough provenance (dataset sha, commit, hardware, config) that a//! published number can be audited later.+//!+//! Two shapes:+//!+//! - [`Report`] — one config's run, reporting the R2 accuracy + latency ++//! context-tokens triple per slice.+//! - [`AbReport`] — a baseline vs candidate pair over the same question+//! set, adding a `overall`-slice delta table.use std::collections::BTreeMap;@@ -16,6 +23,9 @@ pub struct Report {pub dataset: &'static str,pub dataset_sha256: &'static str,pub mode: &'static str,+ /// Human-readable knob summary for this config (e.g.+ /// `embeddings=local reranker=off`).+ pub config: String,pub questions_scored: usize,pub abstention_excluded: usize,pub ks: Vec<usize>,@@ -48,21 +58,23 @@ pub fn hardware() -> String {format!("{model} ({threads} threads)")}-pub fn to_markdown(r: &Report) -> String {- let mut md = String::new();- md.push_str(&format!(- "# LongMemEval-S retrieval — {}\n\n\- - commit: `{}`\n- dataset: `{}` (sha256 `{}…`)\n- mode: {}\n- hardware: {}\n\- - questions scored: {} ({} abstention questions excluded)\n\n",- r.generated_at,+/// Header block shared by single-config and A/B reports.+fn provenance_md(r: &Report) -> String {+ format!(+ "- commit: `{}`\n- dataset: `{}` (sha256 `{}…`)\n- hardware: {}\n\+ - questions scored: {} ({} abstention questions excluded)\n",r.commit,r.dataset,&r.dataset_sha256[..16],- r.mode,r.hardware,r.questions_scored,r.abstention_excluded,- ));+ )+}++/// The per-slice triple table (accuracy + latency + context tokens).+fn slices_table_md(r: &Report) -> String {+ let mut md = String::new();md.push_str("| slice | n |");for k in &r.ks {md.push_str(&format!(" hit@{k} |"));@@ -70,11 +82,12 @@ pub fn to_markdown(r: &Report) -> String {for k in &r.ks {md.push_str(&format!(" recall@{k} |"));}- md.push_str("\n|---|---|");+ md.push_str(" p50 ms | p95 ms | ctx tok mean | ctx tok med |\n");+ md.push_str("|---|---|");for _ in r.ks.iter().chain(r.ks.iter()) {md.push_str("---|");}- md.push('\n');+ md.push_str("---|---|---|---|\n");for (name, m) in &r.slices {md.push_str(&format!("| {name} | {} |", m.questions));for k in &r.ks {@@ -83,15 +96,145 @@ pub fn to_markdown(r: &Report) -> String {for k in &r.ks {md.push_str(&format!(" {:.3} |", m.recall_at[k]));}- md.push('\n');+ md.push_str(&format!(+ " {} | {} | {:.1} | {:.1} |\n",+ m.latency_p50_ms, m.latency_p95_ms, m.context_tokens_mean, m.context_tokens_median,+ ));}- md.push_str(- "\nNotes: hit@k = any evidence session in top k (the \"Recall@k\" most \- systems publish); recall@k = fraction of evidence sessions found. \- Session attribution: `sessions/<id>.md` pages and raw observation \- hits; unattributable pages never score. Capture is production-shaped: \- excerpts bounded at the 2 KB privacy boundary.\n",- );+ md+}++const METRIC_NOTES: &str = "\nNotes: hit@k = any evidence session in top k (the \"Recall@k\" most \+ systems publish); recall@k = fraction of evidence sessions found. \+ Latency is the `memory_query` MCP round trip (send→parse); ctx tok is \+ the estimated context an agent ingests from the result = chars/4 over \+ returned hit titles + snippets. Session attribution: `sessions/<id>.md` \+ pages and raw observation hits; unattributable pages never score. \+ Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.\n";++pub fn to_markdown(r: &Report) -> String {+ let mut md = String::new();+ md.push_str(&format!(+ "# LongMemEval-S retrieval — {}\n\n",+ r.generated_at+ ));+ md.push_str(&provenance_md(r));+ md.push_str(&format!("- mode: {}\n- config: {}\n\n", r.mode, r.config));+ md.push_str(&slices_table_md(r));+ md.push_str(METRIC_NOTES);+ md+}++/// A baseline vs candidate run over the same question set.+#[derive(Debug, Serialize)]+pub struct AbReport {+ pub baseline: Report,+ pub candidate: Report,+ /// `overall`-slice baseline→candidate deltas, precomputed for machine+ /// consumers so a reader need not diff the two nested reports.+ pub delta: AbDelta,+}++/// candidate − baseline over the `overall` slice.+#[derive(Debug, Serialize)]+pub struct AbDelta {+ pub hit_at: BTreeMap<usize, f64>,+ pub recall_at: BTreeMap<usize, f64>,+ pub latency_p50_ms: i128,+ pub latency_p95_ms: i128,+ pub context_tokens_mean: f64,+ pub context_tokens_median: f64,+}++impl AbReport {+ pub fn new(baseline: Report, candidate: Report) -> Self {+ let delta = compute_delta(&baseline, &candidate);+ Self {+ baseline,+ candidate,+ delta,+ }+ }+}++/// The `overall` slice, present in every non-empty run.+fn overall(r: &Report) -> &SliceMetrics {+ r.slices+ .get("overall")+ .expect("every scored run has an `overall` slice")+}++fn compute_delta(baseline: &Report, candidate: &Report) -> AbDelta {+ let b = overall(baseline);+ let c = overall(candidate);+ let mut hit_at = BTreeMap::new();+ let mut recall_at = BTreeMap::new();+ for k in &candidate.ks {+ hit_at.insert(*k, c.hit_at[k] - b.hit_at[k]);+ recall_at.insert(*k, c.recall_at[k] - b.recall_at[k]);+ }+ AbDelta {+ hit_at,+ recall_at,+ latency_p50_ms: c.latency_p50_ms as i128 - b.latency_p50_ms as i128,+ latency_p95_ms: c.latency_p95_ms as i128 - b.latency_p95_ms as i128,+ context_tokens_mean: c.context_tokens_mean - b.context_tokens_mean,+ context_tokens_median: c.context_tokens_median - b.context_tokens_median,+ }+}++pub fn ab_to_markdown(r: &AbReport) -> String {+ let mut md = String::new();+ md.push_str(&format!(+ "# LongMemEval-S retrieval A/B — {}\n\n",+ r.candidate.generated_at+ ));+ // Provenance is shared (same dataset/commit/hardware/question set).+ md.push_str(&provenance_md(&r.candidate));+ md.push_str(&format!(+ "- baseline: {} ({})\n- candidate: {} ({})\n\n",+ r.baseline.config, r.baseline.mode, r.candidate.config, r.candidate.mode,+ ));++ md.push_str("## Baseline\n\n");+ md.push_str(&slices_table_md(&r.baseline));+ md.push_str("\n## Candidate\n\n");+ md.push_str(&slices_table_md(&r.candidate));++ // Delta table over the `overall` slice.+ let b = overall(&r.baseline);+ let c = overall(&r.candidate);+ md.push_str("\n## Delta (overall, candidate − baseline)\n\n");+ md.push_str("| metric | baseline | candidate | delta |\n|---|---|---|---|\n");+ for k in &r.candidate.ks {+ md.push_str(&format!(+ "| hit@{k} | {:.3} | {:.3} | {:+.3} |\n",+ b.hit_at[k], c.hit_at[k], r.delta.hit_at[k]+ ));+ }+ for k in &r.candidate.ks {+ md.push_str(&format!(+ "| recall@{k} | {:.3} | {:.3} | {:+.3} |\n",+ b.recall_at[k], c.recall_at[k], r.delta.recall_at[k]+ ));+ }+ md.push_str(&format!(+ "| latency p50 ms | {} | {} | {:+} |\n",+ b.latency_p50_ms, c.latency_p50_ms, r.delta.latency_p50_ms+ ));+ md.push_str(&format!(+ "| latency p95 ms | {} | {} | {:+} |\n",+ b.latency_p95_ms, c.latency_p95_ms, r.delta.latency_p95_ms+ ));+ md.push_str(&format!(+ "| ctx tok mean | {:.1} | {:.1} | {:+.1} |\n",+ b.context_tokens_mean, c.context_tokens_mean, r.delta.context_tokens_mean+ ));+ md.push_str(&format!(+ "| ctx tok median | {:.1} | {:.1} | {:+.1} |\n",+ b.context_tokens_median, c.context_tokens_median, r.delta.context_tokens_median+ ));+ md.push_str(METRIC_NOTES);md}@@ -99,33 +242,74 @@ pub fn to_markdown(r: &Report) -> String {mod tests {use super::*;- #[test]- fn markdown_carries_provenance_and_all_slices() {+ fn slice(questions: usize, hit5: f64, recall5: f64) -> SliceMetrics {+ SliceMetrics {+ questions,+ hit_at: BTreeMap::from([(5, hit5)]),+ recall_at: BTreeMap::from([(5, recall5)]),+ latency_p50_ms: 12,+ latency_p95_ms: 34,+ context_tokens_mean: 100.0,+ context_tokens_median: 90.0,+ }+ }++ fn report(config: &str, hit5: f64, recall5: f64) -> Report {let mut slices = BTreeMap::new();- slices.insert(- "overall".to_string(),- SliceMetrics {- questions: 2,- hit_at: BTreeMap::from([(5, 0.5)]),- recall_at: BTreeMap::from([(5, 0.25)]),- },- );- let r = Report {+ slices.insert("overall".to_string(), slice(2, hit5, recall5));+ Report {generated_at: "2026-09-01".into(),commit: "abc".into(),hardware: "cpu".into(),dataset: "longmemeval_s",dataset_sha256: crate::retrieval::dataset::LONGMEMEVAL_S_SHA256,mode: "zero-llm",+ config: config.into(),questions_scored: 2,abstention_excluded: 1,ks: vec![5],slices,per_question: vec![],- };+ }+ }++ #[test]+ fn markdown_carries_provenance_and_the_triple() {+ let r = report("embeddings=none reranker=off", 0.5, 0.25);let md = to_markdown(&r);assert!(md.contains("commit: `abc`"));- assert!(md.contains("| overall | 2 | 0.500 | 0.250 |"));+ assert!(md.contains("config: embeddings=none reranker=off"));+ // triple row: hit@5 recall@5 p50 p95 ctx-mean ctx-median+ assert!(+ md.contains("| overall | 2 | 0.500 | 0.250 | 12 | 34 | 100.0 | 90.0 |"),+ "{md}"+ );assert!(md.contains("abstention questions excluded"));+ assert!(md.contains("ctx tok is"));+ }++ #[test]+ fn ab_delta_is_candidate_minus_baseline_and_zero_for_identical_runs() {+ let baseline = report("embeddings=none", 0.5, 0.25);+ let candidate = report("embeddings=none", 0.5, 0.25);+ let ab = AbReport::new(baseline, candidate);+ assert_eq!(ab.delta.hit_at[&5], 0.0);+ assert_eq!(ab.delta.recall_at[&5], 0.0);+ assert_eq!(ab.delta.latency_p50_ms, 0);+ assert_eq!(ab.delta.context_tokens_mean, 0.0);+ let md = ab_to_markdown(&ab);+ assert!(md.contains("A/B"));+ assert!(md.contains("| hit@5 | 0.500 | 0.500 | +0.000 |"), "{md}");+ }++ #[test]+ fn ab_delta_reports_a_real_difference() {+ let baseline = report("embeddings=none", 0.4, 0.20);+ let candidate = report("embeddings=local", 0.6, 0.30);+ let ab = AbReport::new(baseline, candidate);+ assert!((ab.delta.hit_at[&5] - 0.2).abs() < 1e-9);+ let md = ab_to_markdown(&ab);+ assert!(md.contains("| hit@5 | 0.400 | 0.600 | +0.200 |"), "{md}");+ assert!(md.contains("candidate: embeddings=local"));}}diff --git a/evals/src/retrieval/score.rs b/evals/src/retrieval/score.rsindex beef382f..f305c9b0 100644--- a/evals/src/retrieval/score.rs+++ b/evals/src/retrieval/score.rs@@ -28,6 +28,11 @@ pub struct QuestionScore {pub recall_at: BTreeMap<usize, f64>,/// Deduped session attributions actually retrieved (for forensics).pub retrieved_sessions: usize,+ /// `memory_query` round-trip latency for this question, milliseconds.+ pub latency_ms: u128,+ /// Estimated context tokens the result would cost the agent (chars/4+ /// over returned hit titles + snippets).+ pub context_tokens: usize,}/// Score one question's retrieval run.@@ -42,6 +47,8 @@ pub fn score_question(evidence: &HashSet<uuid::Uuid>,retrieved: &[Retrieved],ks: &[usize],+ latency_ms: u128,+ context_tokens: usize,) -> QuestionScore {let mut seen = HashSet::new();let mut ranked_sessions = Vec::new();@@ -73,15 +80,55 @@ pub fn score_question(hit_at,recall_at,retrieved_sessions: ranked_sessions.len(),+ latency_ms,+ context_tokens,}}-/// Aggregated metrics for one slice (a category, or overall).+/// Aggregated metrics for one slice (a category, or overall): the R2+/// accuracy + latency + context-tokens triple.#[derive(Debug, Clone, Serialize)]pub struct SliceMetrics {pub questions: usize,pub hit_at: BTreeMap<usize, f64>,pub recall_at: BTreeMap<usize, f64>,+ /// Latency percentiles over the slice's per-question `memory_query`+ /// round trips, milliseconds.+ pub latency_p50_ms: u128,+ pub latency_p95_ms: u128,+ /// Context-token estimate central tendency over the slice.+ pub context_tokens_mean: f64,+ pub context_tokens_median: f64,+}++/// Nearest-rank percentile (`p` in 0.0..=1.0) over a copy-sorted slice.+/// Empty input yields 0.+fn percentile_u128(values: &[u128], p: f64) -> u128 {+ if values.is_empty() {+ return 0;+ }+ let mut sorted = values.to_vec();+ sorted.sort_unstable();+ // Nearest-rank: rank = ceil(p * n), clamped to [1, n], 1-based.+ let n = sorted.len();+ let rank = (p * n as f64).ceil().max(1.0) as usize;+ sorted[rank.min(n) - 1]+}++/// Median of an already-collected list of counts (mean of the two middle+/// elements for an even count). Empty input yields 0.+fn median_f64(values: &[usize]) -> f64 {+ if values.is_empty() {+ return 0.0;+ }+ let mut sorted = values.to_vec();+ sorted.sort_unstable();+ let n = sorted.len();+ if n % 2 == 1 {+ sorted[n / 2] as f64+ } else {+ (sorted[n / 2 - 1] + sorted[n / 2]) as f64 / 2.0+ }}/// Macro-average scores per category plus an `overall` slice.@@ -101,12 +148,19 @@ pub fn aggregate(scores: &[QuestionScore], ks: &[usize]) -> BTreeMap<String, Slihit_at.insert(k, group.iter().map(|s| s.hit_at[&k]).sum::<f64>() / n);recall_at.insert(k, group.iter().map(|s| s.recall_at[&k]).sum::<f64>() / n);}+ let latencies: Vec<u128> = group.iter().map(|s| s.latency_ms).collect();+ let tokens: Vec<usize> = group.iter().map(|s| s.context_tokens).collect();+ let tokens_sum: usize = tokens.iter().sum();(name,SliceMetrics {questions: group.len(),hit_at,recall_at,+ latency_p50_ms: percentile_u128(&latencies, 0.50),+ latency_p95_ms: percentile_u128(&latencies, 0.95),+ context_tokens_mean: tokens_sum as f64 / n,+ context_tokens_median: median_f64(&tokens),},)})@@ -134,6 +188,8 @@ mod tests {&HashSet::from([e]),&retrieved(&[Some(e)]),&[1, 5],+ 0,+ 0,);assert_eq!(s.hit_at[&1], 1.0);assert_eq!(s.hit_at[&5], 1.0);@@ -152,6 +208,8 @@ mod tests {&HashSet::from([e]),&retrieved(&all),&[1, 3, 5],+ 0,+ 0,);assert_eq!(s.hit_at[&1], 0.0);assert_eq!(s.hit_at[&3], 0.0);@@ -169,6 +227,8 @@ mod tests {&HashSet::from([e]),&retrieved(&[Some(noise), Some(noise), Some(noise), Some(e)]),&[2],+ 0,+ 0,);// dedup: noise occupies rank 0, evidence rank 1 → hit@2assert_eq!(s.hit_at[&2], 1.0);@@ -184,6 +244,8 @@ mod tests {&HashSet::from([e1, e2]),&retrieved(&[Some(e1)]),&[5],+ 0,+ 0,);assert_eq!(s.recall_at[&5], 0.5);assert_eq!(s.hit_at[&5], 1.0);@@ -198,6 +260,8 @@ mod tests {&HashSet::from([e]),&retrieved(&[None, None]),&[5],+ 0,+ 0,);assert_eq!(s.hit_at[&5], 0.0);assert_eq!(s.retrieved_sessions, 0);@@ -215,6 +279,8 @@ mod tests {&HashSet::from([e]),&retrieved(&random),&[1, 3, 5, 10],+ 0,+ 0,);for k in [1, 3, 5, 10] {assert_eq!(s.hit_at[&k], 0.0, "hit@{k}");@@ -231,6 +297,8 @@ mod tests {&HashSet::from([e]),&retrieved(&[Some(e)]),&[1],+ 10,+ 100,);let miss = score_question("q2",@@ -238,9 +306,35 @@ mod tests {&HashSet::from([uuid::Uuid::now_v7()]),&retrieved(&[]),&[1],+ 30,+ 200,);let agg = aggregate(&[hit, miss], &[1]);assert_eq!(agg["multi-session"].hit_at[&1], 0.5);assert_eq!(agg["overall"].questions, 2);+ // Latency + context-tokens triple is aggregated alongside accuracy.+ assert_eq!(agg["overall"].context_tokens_mean, 150.0);+ assert_eq!(agg["overall"].context_tokens_median, 150.0);+ // Nearest-rank p50 over [10, 30] → rank ceil(0.5*2)=1 → 10.+ assert_eq!(agg["overall"].latency_p50_ms, 10);+ assert_eq!(agg["overall"].latency_p95_ms, 30);+ }++ #[test]+ fn percentile_nearest_rank_and_empty() {+ assert_eq!(percentile_u128(&[], 0.5), 0);+ assert_eq!(percentile_u128(&[42], 0.95), 42);+ let v: Vec<u128> = (1..=100).collect();+ assert_eq!(percentile_u128(&v, 0.50), 50);+ assert_eq!(percentile_u128(&v, 0.95), 95);+ assert_eq!(percentile_u128(&v, 1.0), 100);+ }++ #[test]+ fn median_even_and_odd() {+ assert_eq!(median_f64(&[]), 0.0);+ assert_eq!(median_f64(&[7]), 7.0);+ assert_eq!(median_f64(&[1, 2, 3]), 2.0);+ assert_eq!(median_f64(&[1, 2, 3, 10]), 2.5);}}diff --git a/evals/src/retrieval/server.rs b/evals/src/retrieval/server.rsindex 34af935c..13a0e928 100644--- a/evals/src/retrieval/server.rs+++ b/evals/src/retrieval/server.rs@@ -4,9 +4,11 @@//! the same binary a user runs, configured for deterministic zero-LLM//! operation (no consolidation LLM, no embedder, no reranker).+use std::io::Read as _;use std::net::TcpListener;use std::path::PathBuf;use std::process::{Child, Command, Stdio};+use std::sync::{Arc, Mutex};use std::time::{Duration, Instant};use anyhow::{Context, Result, bail};@@ -20,27 +22,53 @@ pub struct EvalServer {/// Kept for the lifetime of the server unless --keep-data-dir moved it._data_dir: Option<tempfile::TempDir>,pub data_dir_path: PathBuf,+ /// Continuously drained child stderr. A piped stderr that nobody reads+ /// deadlocks the server the moment its startup logging fills the OS+ /// pipe buffer (~64 KiB) — and an A/B run launches two servers, so it+ /// hits reliably. A background reader keeps the pipe empty and the+ /// captured text is surfaced on a startup failure.+ stderr: Arc<Mutex<String>>,}/// Which embedding configuration the eval server runs with.#[derive(Clone, Copy, PartialEq, Eq)]pub enum EvalEmbeddings {- /// Deterministic zero-LLM mode: no embedder at all.+ /// Deterministic zero-LLM mode: no embedder at all (vectors OFF).None,- /// In-process local embeddings (all-MiniLM-L6-v2). The model files- /// must already sit in the given models root; the harness seeds the- /// server's data dir from there so the server never fetches.+ /// In-process local embeddings (all-MiniLM-L6-v2), i.e. vectors ON.+ /// The model files must already sit in the given models root; the+ /// harness seeds the server's data dir from there so the server never+ /// fetches.Local,}+/// One server launch configuration — the knob matrix an A/B run varies+/// between its baseline and candidate.+pub struct LaunchConfig<'a> {+ /// Embedding provider (`none` = zero-LLM/vectors off, `local` = vectors on).+ pub embeddings: EvalEmbeddings,+ /// When `embeddings == Local`, the models root to seed the server from.+ pub models_root: Option<&'a std::path::Path>,+ /// Turn the post-RRF reranker on (`AI_MEMORY_RERANKER=llm`). Requires+ /// an LLM provider to be supplied through `env` — the harness does not+ /// configure one, so a bare `reranker=true` with no provider is a no-op+ /// the server logs and ignores.+ pub reranker: bool,+ /// Extra server env overrides, applied last so a config can set any+ /// server knob (reranker provider, retrieval flags, future toggles)+ /// without a new field here. These win over the harness defaults.+ pub env: &'a [(String, String)],+}+impl EvalServer {/// Launch `server_bin` on a free loopback port with a fresh data dir.pub async fn launch(server_bin: &PathBuf,keep_data_dir: bool,- embeddings: EvalEmbeddings,- models_root: Option<&std::path::Path>,+ config: &LaunchConfig<'_>,) -> Result<Self> {+ let embeddings = config.embeddings;+ let models_root = config.models_root;let tmp = tempfile::Builder::new().prefix("ai-memory-eval-").tempdir()@@ -78,8 +106,10 @@ impl EvalServer {.arg(&data_dir_path).env("AI_MEMORY_AUTH_TOKEN", EVAL_AUTH_TOKEN).env("AI_MEMORY_CAPTURE_ASSISTANT", "true")- // No chat LLM and no reranker in either mode; the embedding env- // selects the one nondefault stream under test.+ // Start from a hermetic baseline: no chat LLM, no reranker, no+ // inherited provider keys. The knob matrix below re-enables only+ // what the config asks for, so a run never picks up the+ // developer's ambient environment..env_remove("AI_MEMORY_LLM_PROVIDER").env_remove("AI_MEMORY_RERANKER").env_remove("OPENAI_API_KEY")@@ -96,12 +126,41 @@ impl EvalServer {cmd.env("AI_MEMORY_EMBEDDING_PROVIDER", "local");}}- let child = cmd+ if config.reranker {+ // Only `llm` is a supported reranker; it needs a provider,+ // which the config must supply via `env`.+ cmd.env("AI_MEMORY_RERANKER", "llm");+ }+ // Config env overrides win over every default above.+ for (key, value) in config.env {+ cmd.env(key, value);+ }+ let mut child = cmd.stdout(Stdio::null()).stderr(Stdio::piped()).spawn().with_context(|| format!("spawning {} serve", server_bin.display()))?;+ // Drain stderr continuously so the server never blocks on a full+ // pipe, keeping the captured text for failure diagnostics.+ let stderr = Arc::new(Mutex::new(String::new()));+ if let Some(mut pipe) = child.stderr.take() {+ let sink = stderr.clone();+ std::thread::spawn(move || {+ let mut buf = [0u8; 8192];+ loop {+ match pipe.read(&mut buf) {+ Ok(0) | Err(_) => break,+ Ok(n) => {+ if let Ok(mut s) = sink.lock() {+ s.push_str(&String::from_utf8_lossy(&buf[..n]));+ }+ }+ }+ }+ });+ }+let base_url = format!("http://{bind}");let (guard, data_dir_path) = if keep_data_dir {// Persist the tempdir so it survives the run for inspection.@@ -114,23 +173,30 @@ impl EvalServer {base_url,_data_dir: guard,data_dir_path,+ stderr,};server.wait_ready().await?;Ok(server)}+ /// Snapshot of the child's captured stderr so far.+ fn stderr_snapshot(&self) -> String {+ self.stderr+ .lock()+ .map(|s| s.clone())+ .unwrap_or_else(|_| "(stderr capture poisoned)".into())+ }+/// Poll until the HTTP surface answers (any status counts as up).async fn wait_ready(&mut self) -> Result<()> {let client = reqwest::Client::new();let deadline = Instant::now() + Duration::from_secs(120);loop {if let Some(status) = self.child.try_wait()? {- let mut err = String::new();- if let Some(mut stderr) = self.child.stderr.take() {- use std::io::Read as _;- let _ = stderr.read_to_string(&mut err);- }- bail!("eval server exited during startup ({status}): {err}");+ bail!(+ "eval server exited during startup ({status}): {}",+ self.stderr_snapshot()+ );}if client.get(format!("{}/", self.base_url))@@ -143,8 +209,9 @@ impl EvalServer {}if Instant::now() > deadline {bail!(- "eval server did not answer on {} within 120s",- self.base_url+ "eval server did not answer on {} within 120s: {}",+ self.base_url,+ self.stderr_snapshot());}tokio::time::sleep(Duration::from_millis(200)).await;diff --git a/evals/tests/smoke.rs b/evals/tests/smoke.rsindex c9183b3e..e1068b6c 100644--- a/evals/tests/smoke.rs+++ b/evals/tests/smoke.rs@@ -50,6 +50,9 @@ fn the_harness_scores_a_tiny_sample_end_to_end() {assert_eq!(report["questions_scored"], 2);assert_eq!(report["mode"], "zero-llm");assert!(report["slices"]["overall"]["hit_at"]["5"].is_number());+ // The R2 triple lands next to accuracy.+ assert!(report["slices"]["overall"]["latency_p50_ms"].is_number());+ assert!(report["slices"]["overall"]["context_tokens_mean"].is_number());// Provenance that makes a published number auditable.assert!(report["commit"].as_str().unwrap().len() >= 7);assert!(@@ -60,3 +63,62 @@ fn the_harness_scores_a_tiny_sample_end_to_end() {);assert!(run_dir.join("report.md").exists());}++/// R2 A/B path: a baseline-vs-baseline run (`--candidate`, no differing+/// knob) must produce a delta report whose accuracy and context-token+/// deltas are exactly zero — the determinism guard.+#[test]+#[ignore = "needs the release server binary and the downloaded dataset"]+fn the_ab_path_reports_a_zero_delta_for_identical_configs() {+ let repo_root: PathBuf = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..");+ let server_bin = repo_root.join("target/release/ai-memory");+ assert!(+ server_bin.exists(),+ "build the server first: cargo build --release -p ai-memory-cli"+ );+ let dataset = repo_root.join("evals/datasets/longmemeval_s.json");+ assert!(+ dataset.exists(),+ "fetch the dataset first: cargo run -p ai-memory-eval -- retrieval --fetch --sample 1"+ );++ let out = tempfile::tempdir().unwrap();+ let status = Command::new(env!("CARGO_BIN_EXE_ai-memory-eval"))+ .current_dir(&repo_root)+ .args([+ "retrieval",+ "--candidate",+ "--sample",+ "2",+ "--concurrency",+ "2",+ ])+ .arg("--server-bin")+ .arg(&server_bin)+ .arg("--out")+ .arg(out.path())+ .status()+ .unwrap();+ assert!(status.success(), "A/B harness exited non-zero");++ let run_dir = std::fs::read_dir(out.path())+ .unwrap()+ .map(|e| e.unwrap().path())+ .find(|p| p.is_dir())+ .expect("run dir created");+ let report: serde_json::Value =+ serde_json::from_slice(&std::fs::read(run_dir.join("report.json")).unwrap()).unwrap();+ // A/B report shape: baseline + candidate + precomputed delta.+ assert_eq!(report["baseline"]["questions_scored"], 2);+ assert_eq!(report["candidate"]["questions_scored"], 2);+ // Identical configs ⇒ deterministic accuracy and context; only latency+ // (wall-clock) may wobble, so it is not asserted zero here.+ let delta = &report["delta"];+ for k in ["1", "3", "5", "10"] {+ assert_eq!(delta["hit_at"][k], 0.0, "hit@{k} delta");+ assert_eq!(delta["recall_at"][k], 0.0, "recall@{k} delta");+ }+ assert_eq!(delta["context_tokens_mean"], 0.0);+ assert_eq!(delta["context_tokens_median"], 0.0);+ assert!(run_dir.join("report.md").exists());+}