Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(evals): R2 retrieval A/B harness with accuracy+latency+context-tokens triple

AkitaOnRails926c23e43c2f
diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md
index 78e565b1..95c17e81 100644
--- a/docs/benchmarks/README.md
+++ b/docs/benchmarks/README.md
@@ -28,6 +28,13 @@ harness before the next change landed. For context, published
embedding-based numbers on this dataset: agentmemory 0.967 R@5
(hybrid + reranking), doobidoo/mcp-memory-service 0.804 R@5.
+## A/B comparisons (R2)
+
+The harness can also A/B two configs over the same question set and report
+an **accuracy + latency + context-tokens** triple with a baseline→candidate
+delta — see [retrieval-ab-r2.md](retrieval-ab-r2.md). Full-dataset A/B
+baselines are published there once run.
+
## Reading the numbers
- **mode: local embeddings** is the 2.0 default: the in-process
diff --git a/docs/benchmarks/retrieval-ab-r2.md b/docs/benchmarks/retrieval-ab-r2.md
new file mode 100644
index 00000000..4b93220f
--- /dev/null
+++ b/docs/benchmarks/retrieval-ab-r2.md
@@ -0,0 +1,56 @@
+# Retrieval A/B (R2) — accuracy + latency + context-tokens triple
+
+The `retrieval` harness can compare two server/query configurations over
+the **same** LongMemEval-S question set in one run, reporting a triple per
+config and a baseline→candidate delta:
+
+- **accuracy** — `hit@k` / `recall@k` (as the single-config baselines do);
+- **latency** — `memory_query` MCP round-trip p50 / p95, milliseconds;
+- **context tokens** — the context an agent would ingest from a result,
+ estimated as **chars / 4** over every returned hit's `title` + `snippet`
+ (a documented, provider-agnostic heuristic — not a real tokenizer).
+
+Provenance (commit, dataset sha256, hardware, per-config knob summary) is
+carried exactly as the single-config reports carry it. See
+`evals/README.md` for the full flag reference.
+
+## How to run
+
+```bash
+cargo build --release -p ai-memory-cli
+# baseline (FTS-only) vs candidate (local embeddings), full 500 questions:
+cargo run --release -p ai-memory-eval -- retrieval --fetch \
+ --candidate-embeddings local
+```
+
+A run without a candidate config is unchanged and still writes the
+single-config report. A baseline-vs-baseline run (`--candidate`, no
+differing knob) is the determinism check: accuracy and context-token
+deltas are exactly zero (latency wobbles at wall-clock noise).
+
+## Published A/B numbers
+
+> Full-dataset (500-question) A/B baselines go here, one section per
+> comparison, pasted verbatim from `evals/runs/<stamp>-retrieval/report.md`
+> with its provenance header. **Not yet populated** — running the full
+> matrix is a deliberate, separately-scheduled pass (the dataset is 278 MB
+> and a full local-embeddings leg is minutes, not seconds). Do not paste
+> `--sample` numbers here as if they were baselines.
+
+### Illustrative smoke (NOT a baseline)
+
+Recorded only to show the report shape. `--sample 4`, four
+`single-session-user` questions, so the accuracy figures are noise;
+latency and context-tokens show the expected direction (local embeddings
+add query latency):
+
+| metric | baseline (none) | candidate (local) | delta |
+|---|---|---|---|
+| hit@5 | 0.750 | 0.500 | -0.250 |
+| latency p50 ms | 2 | 31 | +29 |
+| latency p95 ms | 2 | 35 | +33 |
+| ctx tok mean | 408.5 | 393.2 | -15.2 |
+| ctx tok median | 494.5 | 389.0 | -105.5 |
+
+Hardware: AMD Ryzen 9 7950X3D (32 threads). Replace this section with a
+full-dataset run before citing any A/B number.
diff --git a/evals/README.md b/evals/README.md
index e0199b5a..3f946de6 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -201,11 +201,57 @@ cargo run --release -p ai-memory-eval -- retrieval --fetch # full 500
cargo run -p ai-memory-eval -- retrieval --sample 10 # smoke
```
-Output: a per-category table on stdout plus
+Output: a per-slice table on stdout plus
`evals/runs/<timestamp>-retrieval/report.{json,md}` with full
provenance (commit, dataset sha, hardware, per-question scores).
When publishing a baseline, copy the markdown into `docs/benchmarks/`.
+### The triple: accuracy + latency + context-tokens
+
+Every run — single-config or A/B — reports three things per slice, not
+just accuracy:
+
+- **accuracy** — `hit@k` / `recall@k` (unchanged);
+- **latency** — `memory_query` MCP round-trip **p50 / p95** in ms;
+- **context tokens** — the context an agent would ingest from the result,
+ estimated as **chars / 4** over each returned hit's `title` + `snippet`
+ (a documented heuristic, not a real tokenizer), reported **mean / median**.
+
+### `retrieval` A/B mode (R2)
+
+Pass a **candidate** config and the same question set runs through two
+fresh servers (baseline + candidate) and gains a baseline→candidate delta
+table (markdown + JSON). A/B mode turns on with `--candidate` or any
+`--candidate-*` knob; without one, the single-config path is unchanged.
+
+Baseline knobs: `--embeddings none|local` (also = vectors off/on),
+`--reranker` (sets `AI_MEMORY_RERANKER=llm`; needs a provider via
+`--server-env`), `--server-env KEY=VALUE` (any server env, repeatable),
+`--query-arg KEY=JSON` (extra `memory_query` args, repeatable — so future
+knobs like `pin_first` / `include_superseded` can be A/B'd;
+`query`/`workspace`/`project`/`limit` are reserved). Each has a
+`--candidate-*` twin (`--candidate-embeddings`, `--candidate-reranker`,
+`--candidate-server-env`, `--candidate-query-arg`); a candidate knob left
+unset mirrors the baseline.
+
+```bash
+# Determinism check — baseline == candidate ⇒ accuracy/context delta = 0:
+cargo run --release -p ai-memory-eval -- retrieval --candidate --sample 2
+
+# Vectors off vs on, full run:
+cargo run --release -p ai-memory-eval -- retrieval --fetch \
+ --candidate-embeddings local
+
+# A future memory_query knob, baseline off vs candidate on:
+cargo run --release -p ai-memory-eval -- retrieval --fetch \
+ --candidate-query-arg include_expired=true
+```
+
+Output: `evals/runs/<timestamp>-retrieval/report.{json,md}` — the JSON is
+`{ baseline, candidate, delta }`; the markdown carries both per-slice
+triples and an `overall` delta table. Publish full-dataset A/B numbers in
+`docs/benchmarks/retrieval-ab-r2.md`.
+
Honest-numbers notes:
- capture is production-shaped: excerpts are bounded at the 2 KB
diff --git a/evals/src/main.rs b/evals/src/main.rs
index ca38b7cb..dc4270d5 100644
--- a/evals/src/main.rs
+++ b/evals/src/main.rs
@@ -31,7 +31,7 @@ enum Command {
/// consolidation prompt.
Ab(Box<ab::AbArgs>),
/// LongMemEval retrieval benchmark through the real server stack.
- Retrieval(retrieval::RetrievalArgs),
+ Retrieval(Box<retrieval::RetrievalArgs>),
}
#[tokio::main]
@@ -45,6 +45,6 @@ async fn main() -> Result<()> {
match Cli::parse().command {
Command::Ab(args) => ab::run(*args).await,
- Command::Retrieval(args) => retrieval::run(args).await,
+ Command::Retrieval(args) => retrieval::run(*args).await,
}
}
diff --git a/evals/src/retrieval/mod.rs b/evals/src/retrieval/mod.rs
index 88f6774c..088715b2 100644
--- a/evals/src/retrieval/mod.rs
+++ b/evals/src/retrieval/mod.rs
@@ -5,6 +5,16 @@
//! score session-level hit@k / recall@k against the dataset's evidence
//! labels. See `docs/benchmarks/` for published baselines and
//! `evals/README.md` for how to run.
+//!
+//! ## R2: A/B + the accuracy/latency/context triple
+//!
+//! With a candidate config (`--candidate` plus any `--candidate-*` knob),
+//! the same question set runs through TWO server configs — a baseline and
+//! a candidate — and the report gains a baseline→candidate delta. Every
+//! run, single or A/B, now reports a triple per slice: retrieval accuracy
+//! (hit@k / recall@k), `memory_query` latency (p50/p95), and the context
+//! tokens a result would cost the agent (chars/4 over returned snippets).
+//! Without a candidate config the single-config path is unchanged.
mod dataset;
mod ingest;
@@ -21,7 +31,7 @@ use jiff::Timestamp;
use dataset::Question;
use ingest::{project_for, stored_session_uuid};
-use server::EvalServer;
+use server::{EvalEmbeddings, EvalServer, LaunchConfig};
mod server;
@@ -61,11 +71,177 @@ pub struct RetrievalArgs {
#[arg(long)]
keep_data_dir: bool,
- /// Embedding mode: `none` (zero-LLM, the default) or `local`
- /// (in-process all-MiniLM-L6-v2; the model is fetched once into
- /// `evals/models/`, checksum-pinned).
+ // ---- baseline config knobs ----
+ /// Baseline embedding mode: `none` (zero-LLM, vectors off, the
+ /// default) or `local` (in-process all-MiniLM-L6-v2, vectors on; the
+ /// model is fetched once into `evals/models/`, checksum-pinned).
#[arg(long, default_value = "none")]
embeddings: String,
+
+ /// Turn the post-RRF reranker on for the baseline
+ /// (`AI_MEMORY_RERANKER=llm`). Requires an LLM provider, which you
+ /// must supply via `--server-env` — otherwise the server ignores it.
+ #[arg(long)]
+ reranker: bool,
+
+ /// Extra server env for the baseline, `KEY=VALUE`, repeatable. The
+ /// general escape hatch for any server knob (reranker provider,
+ /// retrieval flags, future toggles).
+ #[arg(long = "server-env", value_name = "KEY=VALUE")]
+ server_env: Vec<String>,
+
+ /// Extra `memory_query` argument for the baseline, `KEY=JSON`,
+ /// repeatable (e.g. `--query-arg include_expired=true`). The value is
+ /// parsed as JSON, falling back to a string. Lets future query knobs
+ /// (`pin_first`, `include_superseded`, …) be A/B'd without new flags.
+ /// `query`/`workspace`/`project`/`limit` are reserved by the harness.
+ #[arg(long = "query-arg", value_name = "KEY=JSON")]
+ query_arg: Vec<String>,
+
+ // ---- candidate config knobs (A/B mode) ----
+ /// Enable A/B mode with a candidate config. Implied by any
+ /// `--candidate-*` knob. With no other candidate knob the candidate
+ /// mirrors the baseline — the determinism check (delta ≈ 0).
+ #[arg(long)]
+ candidate: bool,
+
+ /// Candidate embedding mode (`none`/`local`); defaults to the
+ /// baseline's `--embeddings`.
+ #[arg(long)]
+ candidate_embeddings: Option<String>,
+
+ /// Turn the reranker on for the candidate.
+ #[arg(long)]
+ candidate_reranker: bool,
+
+ /// Extra server env for the candidate, `KEY=VALUE`, repeatable.
+ #[arg(long = "candidate-server-env", value_name = "KEY=VALUE")]
+ candidate_server_env: Vec<String>,
+
+ /// Extra `memory_query` argument for the candidate, `KEY=JSON`,
+ /// repeatable.
+ #[arg(long = "candidate-query-arg", value_name = "KEY=JSON")]
+ candidate_query_arg: Vec<String>,
+}
+
+impl RetrievalArgs {
+ /// A/B mode is on when `--candidate` is set or any candidate knob is.
+ fn ab_mode(&self) -> bool {
+ self.candidate
+ || self.candidate_embeddings.is_some()
+ || self.candidate_reranker
+ || !self.candidate_server_env.is_empty()
+ || !self.candidate_query_arg.is_empty()
+ }
+}
+
+/// One resolved server + query configuration.
+struct Knobs {
+ /// Short label for reports (`baseline` / `candidate`).
+ side: &'static str,
+ embeddings: EvalEmbeddings,
+ /// `zero-llm` or `local-embeddings`, for report provenance.
+ mode: &'static str,
+ models_root: Option<PathBuf>,
+ reranker: bool,
+ env: Vec<(String, String)>,
+ query_args: serde_json::Map<String, serde_json::Value>,
+ /// Human-readable knob summary for the report.
+ summary: String,
+}
+
+impl Knobs {
+ fn parse(
+ side: &'static str,
+ embeddings: &str,
+ reranker: bool,
+ server_env: &[String],
+ query_arg: &[String],
+ ) -> Result<Self> {
+ let (embeddings_kind, mode, models_root) = match embeddings {
+ "none" => (EvalEmbeddings::None, "zero-llm", None),
+ "local" => (
+ EvalEmbeddings::Local,
+ "local-embeddings",
+ Some(PathBuf::from("evals/models")),
+ ),
+ other => bail!("{side}: --embeddings must be `none` or `local`, got {other}"),
+ };
+ let env = parse_env(side, server_env)?;
+ let query_args = parse_query_args(side, query_arg)?;
+
+ let mut summary = format!(
+ "embeddings={embeddings} reranker={}",
+ if reranker { "on" } else { "off" }
+ );
+ for (k, v) in &env {
+ summary.push_str(&format!(" env:{k}={v}"));
+ }
+ for (k, v) in &query_args {
+ summary.push_str(&format!(" arg:{k}={v}"));
+ }
+
+ Ok(Self {
+ side,
+ embeddings: embeddings_kind,
+ mode,
+ models_root,
+ reranker,
+ env,
+ query_args,
+ summary,
+ })
+ }
+
+ fn launch_config(&self) -> LaunchConfig<'_> {
+ LaunchConfig {
+ embeddings: self.embeddings,
+ models_root: self.models_root.as_deref(),
+ reranker: self.reranker,
+ env: &self.env,
+ }
+ }
+}
+
+fn parse_env(side: &str, items: &[String]) -> Result<Vec<(String, String)>> {
+ items
+ .iter()
+ .map(|item| {
+ let (k, v) = item.split_once('=').ok_or_else(|| {
+ anyhow::anyhow!("{side}: --server-env `{item}` must be KEY=VALUE")
+ })?;
+ if k.is_empty() {
+ bail!("{side}: --server-env `{item}` has an empty key");
+ }
+ Ok((k.to_string(), v.to_string()))
+ })
+ .collect()
+}
+
+/// Reserved args the harness always sets; a config may not override them.
+const RESERVED_QUERY_ARGS: [&str; 4] = ["query", "workspace", "project", "limit"];
+
+fn parse_query_args(
+ side: &str,
+ items: &[String],
+) -> Result<serde_json::Map<String, serde_json::Value>> {
+ let mut map = serde_json::Map::new();
+ for item in items {
+ let (k, v) = item
+ .split_once('=')
+ .ok_or_else(|| anyhow::anyhow!("{side}: --query-arg `{item}` must be KEY=JSON"))?;
+ if k.is_empty() {
+ bail!("{side}: --query-arg `{item}` has an empty key");
+ }
+ if RESERVED_QUERY_ARGS.contains(&k) {
+ bail!("{side}: --query-arg key `{k}` is reserved by the harness");
+ }
+ // Parse as JSON, else treat the raw text as a string value.
+ let value = serde_json::from_str::<serde_json::Value>(v)
+ .unwrap_or_else(|_| serde_json::Value::String(v.to_string()));
+ map.insert(k.to_string(), value);
+ }
+ Ok(map)
}
pub async fn run(args: RetrievalArgs) -> Result<()> {
@@ -99,30 +275,100 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
args.server_bin.display()
);
}
- let (embeddings, mode, models_root) = match args.embeddings.as_str() {
- "none" => (server::EvalEmbeddings::None, "zero-llm", None),
- "local" => {
- let root = PathBuf::from("evals/models");
- if !ai_memory_llm::model_present(&root) {
- tracing::info!("fetching the local embedding model into evals/models (~87 MB)");
- ai_memory_llm::fetch_model(&root).await?;
- }
- (
- server::EvalEmbeddings::Local,
- "local-embeddings",
- Some(root),
- )
- }
- other => bail!("--embeddings must be `none` or `local`, got {other}"),
+
+ let baseline = Knobs::parse(
+ "baseline",
+ &args.embeddings,
+ args.reranker,
+ &args.server_env,
+ &args.query_arg,
+ )?;
+ let candidate = if args.ab_mode() {
+ let embeddings = args
+ .candidate_embeddings
+ .clone()
+ .unwrap_or_else(|| args.embeddings.clone());
+ Some(Knobs::parse(
+ "candidate",
+ &embeddings,
+ args.candidate_reranker,
+ &args.candidate_server_env,
+ &args.candidate_query_arg,
+ )?)
+ } else {
+ None
};
- let server = EvalServer::launch(
- &args.server_bin,
- args.keep_data_dir,
- embeddings,
- models_root.as_deref(),
- )
- .await?;
- tracing::info!(url = server.base_url, data_dir = %server.data_dir_path.display(), "eval server up");
+
+ // Fetch the local embedding model once if any config needs it.
+ let needs_local = baseline.embeddings == EvalEmbeddings::Local
+ || candidate
+ .as_ref()
+ .is_some_and(|c| c.embeddings == EvalEmbeddings::Local);
+ if needs_local {
+ let root = PathBuf::from("evals/models");
+ if !ai_memory_llm::model_present(&root) {
+ tracing::info!("fetching the local embedding model into evals/models (~87 MB)");
+ ai_memory_llm::fetch_model(&root).await?;
+ }
+ }
+
+ let abstention_n = abstention.len();
+ let baseline_report = run_config(&args, &baseline, &questions, abstention_n, &commit).await?;
+
+ // Timestamped run dir shared by single-config and A/B outputs.
+ let stamp: String = Timestamp::now()
+ .to_string()
+ .chars()
+ .map(|c| if c == ':' || c == '.' { '-' } else { c })
+ .collect();
+ let run_dir = args.out.join(format!("{stamp}-retrieval"));
+ std::fs::create_dir_all(&run_dir)?;
+
+ match candidate {
+ None => {
+ std::fs::write(
+ run_dir.join("report.json"),
+ serde_json::to_vec_pretty(&baseline_report)?,
+ )?;
+ let md = report::to_markdown(&baseline_report);
+ std::fs::write(run_dir.join("report.md"), &md)?;
+ println!("{md}");
+ }
+ Some(candidate) => {
+ let candidate_report =
+ run_config(&args, &candidate, &questions, abstention_n, &commit).await?;
+ let ab = report::AbReport::new(baseline_report, candidate_report);
+ std::fs::write(run_dir.join("report.json"), serde_json::to_vec_pretty(&ab)?)?;
+ let md = report::ab_to_markdown(&ab);
+ std::fs::write(run_dir.join("report.md"), &md)?;
+ println!("{md}");
+ }
+ }
+ println!("run dir: {}", run_dir.display());
+ Ok(())
+}
+
+/// Launch one server config, replay + query + score the whole question
+/// set through it, and build its [`report::Report`]. Each config gets a
+/// fresh server (and fresh data dir) so ingestion never bleeds between
+/// baseline and candidate.
+async fn run_config(
+ args: &RetrievalArgs,
+ knobs: &Knobs,
+ questions: &[Question],
+ abstention_n: usize,
+ commit: &str,
+) -> Result<report::Report> {
+ let server = EvalServer::launch(&args.server_bin, args.keep_data_dir, &knobs.launch_config())
+ .await
+ .with_context(|| format!("launching {} server", knobs.side))?;
+ tracing::info!(
+ side = knobs.side,
+ url = server.base_url,
+ data_dir = %server.data_dir_path.display(),
+ config = knobs.summary,
+ "eval server up"
+ );
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(120))
@@ -131,24 +377,31 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
let ks = Arc::new(args.ks.clone());
let base_url = Arc::new(server.base_url.clone());
let client = Arc::new(client);
+ let query_args = Arc::new(knobs.query_args.clone());
let semaphore = Arc::new(tokio::sync::Semaphore::new(args.concurrency.max(1)));
let mut tasks = tokio::task::JoinSet::new();
- for q in questions.clone() {
+ for q in questions.iter().cloned() {
let permit = semaphore.clone().acquire_owned().await?;
- let (client, base_url, ks) = (client.clone(), base_url.clone(), ks.clone());
+ let (client, base_url, ks, query_args) = (
+ client.clone(),
+ base_url.clone(),
+ ks.clone(),
+ query_args.clone(),
+ );
tasks.spawn(async move {
let _permit = permit;
let events = ingest::ingest_question(&client, &base_url, &q)
.await
.with_context(|| format!("ingesting {}", q.question_id))?;
- let retrieved = query::memory_query(
+ let outcome = query::memory_query(
&client,
&base_url,
ingest::EVAL_WORKSPACE,
&project_for(&q),
&q.question,
max_k,
+ &query_args,
)
.await
.with_context(|| format!("querying {}", q.question_id))?;
@@ -157,12 +410,21 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
.iter()
.map(|s| stored_session_uuid(s))
.collect();
- let score =
- score::score_question(&q.question_id, &q.question_type, &evidence, &retrieved, &ks);
+ let score = score::score_question(
+ &q.question_id,
+ &q.question_type,
+ &evidence,
+ &outcome.retrieved,
+ &ks,
+ outcome.latency.as_millis(),
+ outcome.context_tokens,
+ );
tracing::info!(
question = q.question_id,
events,
- retrieved = retrieved.len(),
+ retrieved = outcome.retrieved.len(),
+ latency_ms = outcome.latency.as_millis() as u64,
+ context_tokens = outcome.context_tokens,
hit_at_max = score.hit_at.values().next_back().copied().unwrap_or(0.0),
"scored"
);
@@ -186,7 +448,8 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
tracing::error!("{f}");
}
bail!(
- "{} of {} questions failed; no report written",
+ "{} config: {} of {} questions failed; no report written",
+ knobs.side,
failures.len(),
failures.len() + scores.len()
);
@@ -194,34 +457,78 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
scores.sort_by(|a, b| a.question_id.cmp(&b.question_id));
let slices = score::aggregate(&scores, &args.ks);
- let stamp: String = Timestamp::now()
- .to_string()
- .chars()
- .map(|c| if c == ':' || c == '.' { '-' } else { c })
- .collect();
- let run_dir = args.out.join(format!("{stamp}-retrieval"));
- std::fs::create_dir_all(&run_dir)?;
-
- let rep = report::Report {
+ Ok(report::Report {
generated_at: Timestamp::now().to_string(),
- commit,
+ commit: commit.to_string(),
hardware: report::hardware(),
dataset: "longmemeval_s",
dataset_sha256: dataset::LONGMEMEVAL_S_SHA256,
- mode,
+ mode: knobs.mode,
+ config: knobs.summary.clone(),
questions_scored: scores.len(),
- abstention_excluded: abstention.len(),
+ abstention_excluded: abstention_n,
ks: args.ks.clone(),
slices,
per_question: scores,
- };
- std::fs::write(
- run_dir.join("report.json"),
- serde_json::to_vec_pretty(&rep)?,
- )?;
- let md = report::to_markdown(&rep);
- std::fs::write(run_dir.join("report.md"), &md)?;
- println!("{md}");
- println!("run dir: {}", run_dir.display());
- Ok(())
+ })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn query_args_parse_json_and_string_and_reject_reserved() {
+ let m = parse_query_args(
+ "baseline",
+ &[
+ "include_expired=true".into(),
+ "note=hello world".into(),
+ "top=5".into(),
+ ],
+ )
+ .unwrap();
+ assert_eq!(m["include_expired"], serde_json::json!(true));
+ assert_eq!(m["note"], serde_json::json!("hello world"));
+ assert_eq!(m["top"], serde_json::json!(5));
+
+ let err = parse_query_args("baseline", &["query=x".into()])
+ .unwrap_err()
+ .to_string();
+ assert!(err.contains("reserved"), "{err}");
+ let err = parse_query_args("baseline", &["bad".into()])
+ .unwrap_err()
+ .to_string();
+ assert!(err.contains("KEY=JSON"), "{err}");
+ }
+
+ #[test]
+ fn server_env_parses_key_value_pairs() {
+ let e = parse_env(
+ "baseline",
+ &["AI_MEMORY_RERANKER=llm".into(), "A=b=c".into()],
+ )
+ .unwrap();
+ assert_eq!(e[0], ("AI_MEMORY_RERANKER".into(), "llm".into()));
+ // split_once keeps everything after the first `=` in the value.
+ assert_eq!(e[1], ("A".into(), "b=c".into()));
+ assert!(parse_env("baseline", &["nope".into()]).is_err());
+ }
+
+ #[test]
+ fn knobs_summary_reflects_the_matrix() {
+ let k = Knobs::parse(
+ "candidate",
+ "local",
+ true,
+ &["AI_MEMORY_LLM_PROVIDER=openai".into()],
+ &["include_expired=true".into()],
+ )
+ .unwrap();
+ assert!(k.summary.contains("embeddings=local"));
+ assert!(k.summary.contains("reranker=on"));
+ assert!(k.summary.contains("env:AI_MEMORY_LLM_PROVIDER=openai"));
+ assert!(k.summary.contains("arg:include_expired=true"));
+ assert_eq!(k.mode, "local-embeddings");
+ }
}
diff --git a/evals/src/retrieval/query.rs b/evals/src/retrieval/query.rs
index 7588f29a..fd73e457 100644
--- a/evals/src/retrieval/query.rs
+++ b/evals/src/retrieval/query.rs
@@ -3,6 +3,8 @@
//! `workspace` + `project` args scope each call to one question's
//! haystack (the documented pattern for static MCP clients).
+use std::time::{Duration, Instant};
+
use anyhow::{Context, Result, bail};
use serde::Deserialize;
use serde_json::json;
@@ -18,6 +20,25 @@ pub struct Retrieved {
pub session_uuid: Option<uuid::Uuid>,
}
+/// Outcome of one `memory_query` call: the ranked hits plus the two
+/// R2 observability signals — how long the round trip took and how much
+/// context the agent would have to read to consume the result.
+#[derive(Debug, Clone)]
+pub struct QueryOutcome {
+ /// Flattened ranked session attributions (index 0 = best).
+ pub retrieved: Vec<Retrieved>,
+ /// Wall-clock of the `memory_query` MCP round trip (request send →
+ /// response parsed).
+ pub latency: Duration,
+ /// Estimated tokens an agent would ingest from this result: the total
+ /// character count of every returned hit's `title` + `snippet`
+ /// (pages and raw observations), divided by 4 (the documented
+ /// chars/4 heuristic). The snippet is exactly what `memory_query`
+ /// puts in front of the agent, so this is the context cost of the
+ /// retrieval, not of the underlying full pages.
+ pub context_tokens: usize,
+}
+
#[derive(Debug, Deserialize)]
struct QueryResponse {
#[serde(default)]
@@ -29,11 +50,27 @@ struct QueryResponse {
#[derive(Debug, Deserialize)]
struct PageHitLite {
path: String,
+ #[serde(default)]
+ title: String,
+ #[serde(default)]
+ snippet: String,
}
#[derive(Debug, Deserialize)]
struct RawHitLite {
session_id: uuid::Uuid,
+ #[serde(default)]
+ title: String,
+ #[serde(default)]
+ snippet: String,
+}
+
+/// Divisor for the chars/4 token estimator (see [`QueryOutcome::context_tokens`]).
+const CHARS_PER_TOKEN: usize = 4;
+
+/// chars/4 token estimate for one string (Unicode scalar count).
+fn estimate_tokens(chars: usize) -> usize {
+ chars.div_ceil(CHARS_PER_TOKEN)
}
/// Call `memory_query` and flatten the response into ranked session
@@ -46,21 +83,27 @@ pub async fn memory_query(
project: &str,
query: &str,
limit: usize,
-) -> Result<Vec<Retrieved>> {
+ extra_args: &serde_json::Map<String, serde_json::Value>,
+) -> Result<QueryOutcome> {
+ // Extra config-supplied args first, then the scoping/limit the harness
+ // controls, so a config can A/B a `memory_query` knob (e.g. a future
+ // `pin_first`/`include_superseded`) without ever redirecting the query
+ // off its question's private haystack.
+ let mut arguments = extra_args.clone();
+ arguments.insert("query".into(), json!(query));
+ arguments.insert("workspace".into(), json!(workspace));
+ arguments.insert("project".into(), json!(project));
+ arguments.insert("limit".into(), json!(limit));
let request = json!({
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "memory_query",
- "arguments": {
- "query": query,
- "workspace": workspace,
- "project": project,
- "limit": limit,
- }
+ "arguments": arguments,
}
});
+ let started = Instant::now();
let resp = client
.post(format!("{base_url}/mcp"))
.bearer_auth(EVAL_AUTH_TOKEN)
@@ -120,22 +163,32 @@ pub async fn memory_query(
.ok_or_else(|| anyhow::anyhow!("memory_query returned no text content: {result}"))?;
let parsed: QueryResponse =
serde_json::from_str(text).with_context(|| format!("parsing memory_query JSON: {text}"))?;
- Ok(flatten(parsed))
+ let (retrieved, context_tokens) = flatten(parsed);
+ Ok(QueryOutcome {
+ retrieved,
+ latency: started.elapsed(),
+ context_tokens,
+ })
}
-fn flatten(resp: QueryResponse) -> Vec<Retrieved> {
+/// Flatten pages + raw hits into ranked session attributions, and sum the
+/// context-token estimate over every hit's `title` + `snippet`.
+fn flatten(resp: QueryResponse) -> (Vec<Retrieved>, usize) {
let mut out = Vec::new();
+ let mut chars = 0usize;
for hit in resp.hits {
+ chars += hit.title.chars().count() + hit.snippet.chars().count();
out.push(Retrieved {
session_uuid: session_uuid_from_path(&hit.path),
});
}
for hit in resp.raw_hits {
+ chars += hit.title.chars().count() + hit.snippet.chars().count();
out.push(Retrieved {
session_uuid: Some(hit.session_id),
});
}
- out
+ (out, estimate_tokens(chars))
}
/// `sessions/<uuid>.md` → the owning session; anything else → None.
@@ -166,12 +219,48 @@ mod tests {
let resp = QueryResponse {
hits: vec![PageHitLite {
path: format!("sessions/{a}.md"),
+ title: String::new(),
+ snippet: String::new(),
+ }],
+ raw_hits: vec![RawHitLite {
+ session_id: b,
+ title: String::new(),
+ snippet: String::new(),
}],
- raw_hits: vec![RawHitLite { session_id: b }],
};
- let flat = flatten(resp);
+ let (flat, _) = flatten(resp);
assert_eq!(flat.len(), 2);
assert_eq!(flat[0].session_uuid, Some(a));
assert_eq!(flat[1].session_uuid, Some(b));
}
+
+ #[test]
+ fn context_tokens_sum_title_and_snippet_over_all_hits() {
+ let a = uuid::Uuid::now_v7();
+ let b = uuid::Uuid::now_v7();
+ let resp = QueryResponse {
+ // 4 + 4 = 8 chars → 2 tokens
+ hits: vec![PageHitLite {
+ path: format!("sessions/{a}.md"),
+ title: "abcd".into(),
+ snippet: "efgh".into(),
+ }],
+ // 4 + 4 = 8 chars → +2 tokens
+ raw_hits: vec![RawHitLite {
+ session_id: b,
+ title: "ijkl".into(),
+ snippet: "mnop".into(),
+ }],
+ };
+ let (_, tokens) = flatten(resp);
+ assert_eq!(tokens, 4);
+ }
+
+ #[test]
+ fn token_estimate_rounds_up() {
+ // 5 chars / 4 = 1.25 → ceil 2
+ assert_eq!(estimate_tokens(5), 2);
+ assert_eq!(estimate_tokens(0), 0);
+ assert_eq!(estimate_tokens(8), 2);
+ }
}
diff --git a/evals/src/retrieval/report.rs b/evals/src/retrieval/report.rs
index 0308bb4b..390de7fd 100644
--- a/evals/src/retrieval/report.rs
+++ b/evals/src/retrieval/report.rs
@@ -1,6 +1,13 @@
//! Render benchmark results as JSON (machine) + markdown (human), with
//! enough provenance (dataset sha, commit, hardware, config) that a
//! published number can be audited later.
+//!
+//! Two shapes:
+//!
+//! - [`Report`] — one config's run, reporting the R2 accuracy + latency +
+//! context-tokens triple per slice.
+//! - [`AbReport`] — a baseline vs candidate pair over the same question
+//! set, adding a `overall`-slice delta table.
use std::collections::BTreeMap;
@@ -16,6 +23,9 @@ pub struct Report {
pub dataset: &'static str,
pub dataset_sha256: &'static str,
pub mode: &'static str,
+ /// Human-readable knob summary for this config (e.g.
+ /// `embeddings=local reranker=off`).
+ pub config: String,
pub questions_scored: usize,
pub abstention_excluded: usize,
pub ks: Vec<usize>,
@@ -48,21 +58,23 @@ pub fn hardware() -> String {
format!("{model} ({threads} threads)")
}
-pub fn to_markdown(r: &Report) -> String {
- let mut md = String::new();
- md.push_str(&format!(
- "# LongMemEval-S retrieval — {}\n\n\
- - commit: `{}`\n- dataset: `{}` (sha256 `{}…`)\n- mode: {}\n- hardware: {}\n\
- - questions scored: {} ({} abstention questions excluded)\n\n",
- r.generated_at,
+/// Header block shared by single-config and A/B reports.
+fn provenance_md(r: &Report) -> String {
+ format!(
+ "- commit: `{}`\n- dataset: `{}` (sha256 `{}…`)\n- hardware: {}\n\
+ - questions scored: {} ({} abstention questions excluded)\n",
r.commit,
r.dataset,
&r.dataset_sha256[..16],
- r.mode,
r.hardware,
r.questions_scored,
r.abstention_excluded,
- ));
+ )
+}
+
+/// The per-slice triple table (accuracy + latency + context tokens).
+fn slices_table_md(r: &Report) -> String {
+ let mut md = String::new();
md.push_str("| slice | n |");
for k in &r.ks {
md.push_str(&format!(" hit@{k} |"));
@@ -70,11 +82,12 @@ pub fn to_markdown(r: &Report) -> String {
for k in &r.ks {
md.push_str(&format!(" recall@{k} |"));
}
- md.push_str("\n|---|---|");
+ md.push_str(" p50 ms | p95 ms | ctx tok mean | ctx tok med |\n");
+ md.push_str("|---|---|");
for _ in r.ks.iter().chain(r.ks.iter()) {
md.push_str("---|");
}
- md.push('\n');
+ md.push_str("---|---|---|---|\n");
for (name, m) in &r.slices {
md.push_str(&format!("| {name} | {} |", m.questions));
for k in &r.ks {
@@ -83,15 +96,145 @@ pub fn to_markdown(r: &Report) -> String {
for k in &r.ks {
md.push_str(&format!(" {:.3} |", m.recall_at[k]));
}
- md.push('\n');
+ md.push_str(&format!(
+ " {} | {} | {:.1} | {:.1} |\n",
+ m.latency_p50_ms, m.latency_p95_ms, m.context_tokens_mean, m.context_tokens_median,
+ ));
}
- md.push_str(
- "\nNotes: hit@k = any evidence session in top k (the \"Recall@k\" most \
- systems publish); recall@k = fraction of evidence sessions found. \
- Session attribution: `sessions/<id>.md` pages and raw observation \
- hits; unattributable pages never score. Capture is production-shaped: \
- excerpts bounded at the 2 KB privacy boundary.\n",
- );
+ md
+}
+
+const METRIC_NOTES: &str = "\nNotes: hit@k = any evidence session in top k (the \"Recall@k\" most \
+ systems publish); recall@k = fraction of evidence sessions found. \
+ Latency is the `memory_query` MCP round trip (send→parse); ctx tok is \
+ the estimated context an agent ingests from the result = chars/4 over \
+ returned hit titles + snippets. Session attribution: `sessions/<id>.md` \
+ pages and raw observation hits; unattributable pages never score. \
+ Capture is production-shaped: excerpts bounded at the 2 KB privacy boundary.\n";
+
+pub fn to_markdown(r: &Report) -> String {
+ let mut md = String::new();
+ md.push_str(&format!(
+ "# LongMemEval-S retrieval — {}\n\n",
+ r.generated_at
+ ));
+ md.push_str(&provenance_md(r));
+ md.push_str(&format!("- mode: {}\n- config: {}\n\n", r.mode, r.config));
+ md.push_str(&slices_table_md(r));
+ md.push_str(METRIC_NOTES);
+ md
+}
+
+/// A baseline vs candidate run over the same question set.
+#[derive(Debug, Serialize)]
+pub struct AbReport {
+ pub baseline: Report,
+ pub candidate: Report,
+ /// `overall`-slice baseline→candidate deltas, precomputed for machine
+ /// consumers so a reader need not diff the two nested reports.
+ pub delta: AbDelta,
+}
+
+/// candidate − baseline over the `overall` slice.
+#[derive(Debug, Serialize)]
+pub struct AbDelta {
+ pub hit_at: BTreeMap<usize, f64>,
+ pub recall_at: BTreeMap<usize, f64>,
+ pub latency_p50_ms: i128,
+ pub latency_p95_ms: i128,
+ pub context_tokens_mean: f64,
+ pub context_tokens_median: f64,
+}
+
+impl AbReport {
+ pub fn new(baseline: Report, candidate: Report) -> Self {
+ let delta = compute_delta(&baseline, &candidate);
+ Self {
+ baseline,
+ candidate,
+ delta,
+ }
+ }
+}
+
+/// The `overall` slice, present in every non-empty run.
+fn overall(r: &Report) -> &SliceMetrics {
+ r.slices
+ .get("overall")
+ .expect("every scored run has an `overall` slice")
+}
+
+fn compute_delta(baseline: &Report, candidate: &Report) -> AbDelta {
+ let b = overall(baseline);
+ let c = overall(candidate);
+ let mut hit_at = BTreeMap::new();
+ let mut recall_at = BTreeMap::new();
+ for k in &candidate.ks {
+ hit_at.insert(*k, c.hit_at[k] - b.hit_at[k]);
+ recall_at.insert(*k, c.recall_at[k] - b.recall_at[k]);
+ }
+ AbDelta {
+ hit_at,
+ recall_at,
+ latency_p50_ms: c.latency_p50_ms as i128 - b.latency_p50_ms as i128,
+ latency_p95_ms: c.latency_p95_ms as i128 - b.latency_p95_ms as i128,
+ context_tokens_mean: c.context_tokens_mean - b.context_tokens_mean,
+ context_tokens_median: c.context_tokens_median - b.context_tokens_median,
+ }
+}
+
+pub fn ab_to_markdown(r: &AbReport) -> String {
+ let mut md = String::new();
+ md.push_str(&format!(
+ "# LongMemEval-S retrieval A/B — {}\n\n",
+ r.candidate.generated_at
+ ));
+ // Provenance is shared (same dataset/commit/hardware/question set).
+ md.push_str(&provenance_md(&r.candidate));
+ md.push_str(&format!(
+ "- baseline: {} ({})\n- candidate: {} ({})\n\n",
+ r.baseline.config, r.baseline.mode, r.candidate.config, r.candidate.mode,
+ ));
+
+ md.push_str("## Baseline\n\n");
+ md.push_str(&slices_table_md(&r.baseline));
+ md.push_str("\n## Candidate\n\n");
+ md.push_str(&slices_table_md(&r.candidate));
+
+ // Delta table over the `overall` slice.
+ let b = overall(&r.baseline);
+ let c = overall(&r.candidate);
+ md.push_str("\n## Delta (overall, candidate − baseline)\n\n");
+ md.push_str("| metric | baseline | candidate | delta |\n|---|---|---|---|\n");
+ for k in &r.candidate.ks {
+ md.push_str(&format!(
+ "| hit@{k} | {:.3} | {:.3} | {:+.3} |\n",
+ b.hit_at[k], c.hit_at[k], r.delta.hit_at[k]
+ ));
+ }
+ for k in &r.candidate.ks {
+ md.push_str(&format!(
+ "| recall@{k} | {:.3} | {:.3} | {:+.3} |\n",
+ b.recall_at[k], c.recall_at[k], r.delta.recall_at[k]
+ ));
+ }
+ md.push_str(&format!(
+ "| latency p50 ms | {} | {} | {:+} |\n",
+ b.latency_p50_ms, c.latency_p50_ms, r.delta.latency_p50_ms
+ ));
+ md.push_str(&format!(
+ "| latency p95 ms | {} | {} | {:+} |\n",
+ b.latency_p95_ms, c.latency_p95_ms, r.delta.latency_p95_ms
+ ));
+ md.push_str(&format!(
+ "| ctx tok mean | {:.1} | {:.1} | {:+.1} |\n",
+ b.context_tokens_mean, c.context_tokens_mean, r.delta.context_tokens_mean
+ ));
+ md.push_str(&format!(
+ "| ctx tok median | {:.1} | {:.1} | {:+.1} |\n",
+ b.context_tokens_median, c.context_tokens_median, r.delta.context_tokens_median
+ ));
+ md.push_str(METRIC_NOTES);
md
}
@@ -99,33 +242,74 @@ pub fn to_markdown(r: &Report) -> String {
mod tests {
use super::*;
- #[test]
- fn markdown_carries_provenance_and_all_slices() {
+ fn slice(questions: usize, hit5: f64, recall5: f64) -> SliceMetrics {
+ SliceMetrics {
+ questions,
+ hit_at: BTreeMap::from([(5, hit5)]),
+ recall_at: BTreeMap::from([(5, recall5)]),
+ latency_p50_ms: 12,
+ latency_p95_ms: 34,
+ context_tokens_mean: 100.0,
+ context_tokens_median: 90.0,
+ }
+ }
+
+ fn report(config: &str, hit5: f64, recall5: f64) -> Report {
let mut slices = BTreeMap::new();
- slices.insert(
- "overall".to_string(),
- SliceMetrics {
- questions: 2,
- hit_at: BTreeMap::from([(5, 0.5)]),
- recall_at: BTreeMap::from([(5, 0.25)]),
- },
- );
- let r = Report {
+ slices.insert("overall".to_string(), slice(2, hit5, recall5));
+ Report {
generated_at: "2026-09-01".into(),
commit: "abc".into(),
hardware: "cpu".into(),
dataset: "longmemeval_s",
dataset_sha256: crate::retrieval::dataset::LONGMEMEVAL_S_SHA256,
mode: "zero-llm",
+ config: config.into(),
questions_scored: 2,
abstention_excluded: 1,
ks: vec![5],
slices,
per_question: vec![],
- };
+ }
+ }
+
+ #[test]
+ fn markdown_carries_provenance_and_the_triple() {
+ let r = report("embeddings=none reranker=off", 0.5, 0.25);
let md = to_markdown(&r);
assert!(md.contains("commit: `abc`"));
- assert!(md.contains("| overall | 2 | 0.500 | 0.250 |"));
+ assert!(md.contains("config: embeddings=none reranker=off"));
+ // triple row: hit@5 recall@5 p50 p95 ctx-mean ctx-median
+ assert!(
+ md.contains("| overall | 2 | 0.500 | 0.250 | 12 | 34 | 100.0 | 90.0 |"),
+ "{md}"
+ );
assert!(md.contains("abstention questions excluded"));
+ assert!(md.contains("ctx tok is"));
+ }
+
+ #[test]
+ fn ab_delta_is_candidate_minus_baseline_and_zero_for_identical_runs() {
+ let baseline = report("embeddings=none", 0.5, 0.25);
+ let candidate = report("embeddings=none", 0.5, 0.25);
+ let ab = AbReport::new(baseline, candidate);
+ assert_eq!(ab.delta.hit_at[&5], 0.0);
+ assert_eq!(ab.delta.recall_at[&5], 0.0);
+ assert_eq!(ab.delta.latency_p50_ms, 0);
+ assert_eq!(ab.delta.context_tokens_mean, 0.0);
+ let md = ab_to_markdown(&ab);
+ assert!(md.contains("A/B"));
+ assert!(md.contains("| hit@5 | 0.500 | 0.500 | +0.000 |"), "{md}");
+ }
+
+ #[test]
+ fn ab_delta_reports_a_real_difference() {
+ let baseline = report("embeddings=none", 0.4, 0.20);
+ let candidate = report("embeddings=local", 0.6, 0.30);
+ let ab = AbReport::new(baseline, candidate);
+ assert!((ab.delta.hit_at[&5] - 0.2).abs() < 1e-9);
+ let md = ab_to_markdown(&ab);
+ assert!(md.contains("| hit@5 | 0.400 | 0.600 | +0.200 |"), "{md}");
+ assert!(md.contains("candidate: embeddings=local"));
}
}
diff --git a/evals/src/retrieval/score.rs b/evals/src/retrieval/score.rs
index beef382f..f305c9b0 100644
--- a/evals/src/retrieval/score.rs
+++ b/evals/src/retrieval/score.rs
@@ -28,6 +28,11 @@ pub struct QuestionScore {
pub recall_at: BTreeMap<usize, f64>,
/// Deduped session attributions actually retrieved (for forensics).
pub retrieved_sessions: usize,
+ /// `memory_query` round-trip latency for this question, milliseconds.
+ pub latency_ms: u128,
+ /// Estimated context tokens the result would cost the agent (chars/4
+ /// over returned hit titles + snippets).
+ pub context_tokens: usize,
}
/// Score one question's retrieval run.
@@ -42,6 +47,8 @@ pub fn score_question(
evidence: &HashSet<uuid::Uuid>,
retrieved: &[Retrieved],
ks: &[usize],
+ latency_ms: u128,
+ context_tokens: usize,
) -> QuestionScore {
let mut seen = HashSet::new();
let mut ranked_sessions = Vec::new();
@@ -73,15 +80,55 @@ pub fn score_question(
hit_at,
recall_at,
retrieved_sessions: ranked_sessions.len(),
+ latency_ms,
+ context_tokens,
}
}
-/// Aggregated metrics for one slice (a category, or overall).
+/// Aggregated metrics for one slice (a category, or overall): the R2
+/// accuracy + latency + context-tokens triple.
#[derive(Debug, Clone, Serialize)]
pub struct SliceMetrics {
pub questions: usize,
pub hit_at: BTreeMap<usize, f64>,
pub recall_at: BTreeMap<usize, f64>,
+ /// Latency percentiles over the slice's per-question `memory_query`
+ /// round trips, milliseconds.
+ pub latency_p50_ms: u128,
+ pub latency_p95_ms: u128,
+ /// Context-token estimate central tendency over the slice.
+ pub context_tokens_mean: f64,
+ pub context_tokens_median: f64,
+}
+
+/// Nearest-rank percentile (`p` in 0.0..=1.0) over a copy-sorted slice.
+/// Empty input yields 0.
+fn percentile_u128(values: &[u128], p: f64) -> u128 {
+ if values.is_empty() {
+ return 0;
+ }
+ let mut sorted = values.to_vec();
+ sorted.sort_unstable();
+ // Nearest-rank: rank = ceil(p * n), clamped to [1, n], 1-based.
+ let n = sorted.len();
+ let rank = (p * n as f64).ceil().max(1.0) as usize;
+ sorted[rank.min(n) - 1]
+}
+
+/// Median of an already-collected list of counts (mean of the two middle
+/// elements for an even count). Empty input yields 0.
+fn median_f64(values: &[usize]) -> f64 {
+ if values.is_empty() {
+ return 0.0;
+ }
+ let mut sorted = values.to_vec();
+ sorted.sort_unstable();
+ let n = sorted.len();
+ if n % 2 == 1 {
+ sorted[n / 2] as f64
+ } else {
+ (sorted[n / 2 - 1] + sorted[n / 2]) as f64 / 2.0
+ }
}
/// Macro-average scores per category plus an `overall` slice.
@@ -101,12 +148,19 @@ pub fn aggregate(scores: &[QuestionScore], ks: &[usize]) -> BTreeMap<String, Sli
hit_at.insert(k, group.iter().map(|s| s.hit_at[&k]).sum::<f64>() / n);
recall_at.insert(k, group.iter().map(|s| s.recall_at[&k]).sum::<f64>() / n);
}
+ let latencies: Vec<u128> = group.iter().map(|s| s.latency_ms).collect();
+ let tokens: Vec<usize> = group.iter().map(|s| s.context_tokens).collect();
+ let tokens_sum: usize = tokens.iter().sum();
(
name,
SliceMetrics {
questions: group.len(),
hit_at,
recall_at,
+ latency_p50_ms: percentile_u128(&latencies, 0.50),
+ latency_p95_ms: percentile_u128(&latencies, 0.95),
+ context_tokens_mean: tokens_sum as f64 / n,
+ context_tokens_median: median_f64(&tokens),
},
)
})
@@ -134,6 +188,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&[Some(e)]),
&[1, 5],
+ 0,
+ 0,
);
assert_eq!(s.hit_at[&1], 1.0);
assert_eq!(s.hit_at[&5], 1.0);
@@ -152,6 +208,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&all),
&[1, 3, 5],
+ 0,
+ 0,
);
assert_eq!(s.hit_at[&1], 0.0);
assert_eq!(s.hit_at[&3], 0.0);
@@ -169,6 +227,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&[Some(noise), Some(noise), Some(noise), Some(e)]),
&[2],
+ 0,
+ 0,
);
// dedup: noise occupies rank 0, evidence rank 1 → hit@2
assert_eq!(s.hit_at[&2], 1.0);
@@ -184,6 +244,8 @@ mod tests {
&HashSet::from([e1, e2]),
&retrieved(&[Some(e1)]),
&[5],
+ 0,
+ 0,
);
assert_eq!(s.recall_at[&5], 0.5);
assert_eq!(s.hit_at[&5], 1.0);
@@ -198,6 +260,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&[None, None]),
&[5],
+ 0,
+ 0,
);
assert_eq!(s.hit_at[&5], 0.0);
assert_eq!(s.retrieved_sessions, 0);
@@ -215,6 +279,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&random),
&[1, 3, 5, 10],
+ 0,
+ 0,
);
for k in [1, 3, 5, 10] {
assert_eq!(s.hit_at[&k], 0.0, "hit@{k}");
@@ -231,6 +297,8 @@ mod tests {
&HashSet::from([e]),
&retrieved(&[Some(e)]),
&[1],
+ 10,
+ 100,
);
let miss = score_question(
"q2",
@@ -238,9 +306,35 @@ mod tests {
&HashSet::from([uuid::Uuid::now_v7()]),
&retrieved(&[]),
&[1],
+ 30,
+ 200,
);
let agg = aggregate(&[hit, miss], &[1]);
assert_eq!(agg["multi-session"].hit_at[&1], 0.5);
assert_eq!(agg["overall"].questions, 2);
+ // Latency + context-tokens triple is aggregated alongside accuracy.
+ assert_eq!(agg["overall"].context_tokens_mean, 150.0);
+ assert_eq!(agg["overall"].context_tokens_median, 150.0);
+ // Nearest-rank p50 over [10, 30] → rank ceil(0.5*2)=1 → 10.
+ assert_eq!(agg["overall"].latency_p50_ms, 10);
+ assert_eq!(agg["overall"].latency_p95_ms, 30);
+ }
+
+ #[test]
+ fn percentile_nearest_rank_and_empty() {
+ assert_eq!(percentile_u128(&[], 0.5), 0);
+ assert_eq!(percentile_u128(&[42], 0.95), 42);
+ let v: Vec<u128> = (1..=100).collect();
+ assert_eq!(percentile_u128(&v, 0.50), 50);
+ assert_eq!(percentile_u128(&v, 0.95), 95);
+ assert_eq!(percentile_u128(&v, 1.0), 100);
+ }
+
+ #[test]
+ fn median_even_and_odd() {
+ assert_eq!(median_f64(&[]), 0.0);
+ assert_eq!(median_f64(&[7]), 7.0);
+ assert_eq!(median_f64(&[1, 2, 3]), 2.0);
+ assert_eq!(median_f64(&[1, 2, 3, 10]), 2.5);
}
}
diff --git a/evals/src/retrieval/server.rs b/evals/src/retrieval/server.rs
index 34af935c..13a0e928 100644
--- a/evals/src/retrieval/server.rs
+++ b/evals/src/retrieval/server.rs
@@ -4,9 +4,11 @@
//! the same binary a user runs, configured for deterministic zero-LLM
//! operation (no consolidation LLM, no embedder, no reranker).
+use std::io::Read as _;
use std::net::TcpListener;
use std::path::PathBuf;
use std::process::{Child, Command, Stdio};
+use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use anyhow::{Context, Result, bail};
@@ -20,27 +22,53 @@ pub struct EvalServer {
/// Kept for the lifetime of the server unless --keep-data-dir moved it.
_data_dir: Option<tempfile::TempDir>,
pub data_dir_path: PathBuf,
+ /// Continuously drained child stderr. A piped stderr that nobody reads
+ /// deadlocks the server the moment its startup logging fills the OS
+ /// pipe buffer (~64 KiB) — and an A/B run launches two servers, so it
+ /// hits reliably. A background reader keeps the pipe empty and the
+ /// captured text is surfaced on a startup failure.
+ stderr: Arc<Mutex<String>>,
}
/// Which embedding configuration the eval server runs with.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum EvalEmbeddings {
- /// Deterministic zero-LLM mode: no embedder at all.
+ /// Deterministic zero-LLM mode: no embedder at all (vectors OFF).
None,
- /// In-process local embeddings (all-MiniLM-L6-v2). The model files
- /// must already sit in the given models root; the harness seeds the
- /// server's data dir from there so the server never fetches.
+ /// In-process local embeddings (all-MiniLM-L6-v2), i.e. vectors ON.
+ /// The model files must already sit in the given models root; the
+ /// harness seeds the server's data dir from there so the server never
+ /// fetches.
Local,
}
+/// One server launch configuration — the knob matrix an A/B run varies
+/// between its baseline and candidate.
+pub struct LaunchConfig<'a> {
+ /// Embedding provider (`none` = zero-LLM/vectors off, `local` = vectors on).
+ pub embeddings: EvalEmbeddings,
+ /// When `embeddings == Local`, the models root to seed the server from.
+ pub models_root: Option<&'a std::path::Path>,
+ /// Turn the post-RRF reranker on (`AI_MEMORY_RERANKER=llm`). Requires
+ /// an LLM provider to be supplied through `env` — the harness does not
+ /// configure one, so a bare `reranker=true` with no provider is a no-op
+ /// the server logs and ignores.
+ pub reranker: bool,
+ /// Extra server env overrides, applied last so a config can set any
+ /// server knob (reranker provider, retrieval flags, future toggles)
+ /// without a new field here. These win over the harness defaults.
+ pub env: &'a [(String, String)],
+}
+
impl EvalServer {
/// Launch `server_bin` on a free loopback port with a fresh data dir.
pub async fn launch(
server_bin: &PathBuf,
keep_data_dir: bool,
- embeddings: EvalEmbeddings,
- models_root: Option<&std::path::Path>,
+ config: &LaunchConfig<'_>,
) -> Result<Self> {
+ let embeddings = config.embeddings;
+ let models_root = config.models_root;
let tmp = tempfile::Builder::new()
.prefix("ai-memory-eval-")
.tempdir()
@@ -78,8 +106,10 @@ impl EvalServer {
.arg(&data_dir_path)
.env("AI_MEMORY_AUTH_TOKEN", EVAL_AUTH_TOKEN)
.env("AI_MEMORY_CAPTURE_ASSISTANT", "true")
- // No chat LLM and no reranker in either mode; the embedding env
- // selects the one nondefault stream under test.
+ // Start from a hermetic baseline: no chat LLM, no reranker, no
+ // inherited provider keys. The knob matrix below re-enables only
+ // what the config asks for, so a run never picks up the
+ // developer's ambient environment.
.env_remove("AI_MEMORY_LLM_PROVIDER")
.env_remove("AI_MEMORY_RERANKER")
.env_remove("OPENAI_API_KEY")
@@ -96,12 +126,41 @@ impl EvalServer {
cmd.env("AI_MEMORY_EMBEDDING_PROVIDER", "local");
}
}
- let child = cmd
+ if config.reranker {
+ // Only `llm` is a supported reranker; it needs a provider,
+ // which the config must supply via `env`.
+ cmd.env("AI_MEMORY_RERANKER", "llm");
+ }
+ // Config env overrides win over every default above.
+ for (key, value) in config.env {
+ cmd.env(key, value);
+ }
+ let mut child = cmd
.stdout(Stdio::null())
.stderr(Stdio::piped())
.spawn()
.with_context(|| format!("spawning {} serve", server_bin.display()))?;
+ // Drain stderr continuously so the server never blocks on a full
+ // pipe, keeping the captured text for failure diagnostics.
+ let stderr = Arc::new(Mutex::new(String::new()));
+ if let Some(mut pipe) = child.stderr.take() {
+ let sink = stderr.clone();
+ std::thread::spawn(move || {
+ let mut buf = [0u8; 8192];
+ loop {
+ match pipe.read(&mut buf) {
+ Ok(0) | Err(_) => break,
+ Ok(n) => {
+ if let Ok(mut s) = sink.lock() {
+ s.push_str(&String::from_utf8_lossy(&buf[..n]));
+ }
+ }
+ }
+ }
+ });
+ }
+
let base_url = format!("http://{bind}");
let (guard, data_dir_path) = if keep_data_dir {
// Persist the tempdir so it survives the run for inspection.
@@ -114,23 +173,30 @@ impl EvalServer {
base_url,
_data_dir: guard,
data_dir_path,
+ stderr,
};
server.wait_ready().await?;
Ok(server)
}
+ /// Snapshot of the child's captured stderr so far.
+ fn stderr_snapshot(&self) -> String {
+ self.stderr
+ .lock()
+ .map(|s| s.clone())
+ .unwrap_or_else(|_| "(stderr capture poisoned)".into())
+ }
+
/// Poll until the HTTP surface answers (any status counts as up).
async fn wait_ready(&mut self) -> Result<()> {
let client = reqwest::Client::new();
let deadline = Instant::now() + Duration::from_secs(120);
loop {
if let Some(status) = self.child.try_wait()? {
- let mut err = String::new();
- if let Some(mut stderr) = self.child.stderr.take() {
- use std::io::Read as _;
- let _ = stderr.read_to_string(&mut err);
- }
- bail!("eval server exited during startup ({status}): {err}");
+ bail!(
+ "eval server exited during startup ({status}): {}",
+ self.stderr_snapshot()
+ );
}
if client
.get(format!("{}/", self.base_url))
@@ -143,8 +209,9 @@ impl EvalServer {
}
if Instant::now() > deadline {
bail!(
- "eval server did not answer on {} within 120s",
- self.base_url
+ "eval server did not answer on {} within 120s: {}",
+ self.base_url,
+ self.stderr_snapshot()
);
}
tokio::time::sleep(Duration::from_millis(200)).await;
diff --git a/evals/tests/smoke.rs b/evals/tests/smoke.rs
index c9183b3e..e1068b6c 100644
--- a/evals/tests/smoke.rs
+++ b/evals/tests/smoke.rs
@@ -50,6 +50,9 @@ fn the_harness_scores_a_tiny_sample_end_to_end() {
assert_eq!(report["questions_scored"], 2);
assert_eq!(report["mode"], "zero-llm");
assert!(report["slices"]["overall"]["hit_at"]["5"].is_number());
+ // The R2 triple lands next to accuracy.
+ assert!(report["slices"]["overall"]["latency_p50_ms"].is_number());
+ assert!(report["slices"]["overall"]["context_tokens_mean"].is_number());
// Provenance that makes a published number auditable.
assert!(report["commit"].as_str().unwrap().len() >= 7);
assert!(
@@ -60,3 +63,62 @@ fn the_harness_scores_a_tiny_sample_end_to_end() {
);
assert!(run_dir.join("report.md").exists());
}
+
+/// R2 A/B path: a baseline-vs-baseline run (`--candidate`, no differing
+/// knob) must produce a delta report whose accuracy and context-token
+/// deltas are exactly zero — the determinism guard.
+#[test]
+#[ignore = "needs the release server binary and the downloaded dataset"]
+fn the_ab_path_reports_a_zero_delta_for_identical_configs() {
+ let repo_root: PathBuf = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("..");
+ let server_bin = repo_root.join("target/release/ai-memory");
+ assert!(
+ server_bin.exists(),
+ "build the server first: cargo build --release -p ai-memory-cli"
+ );
+ let dataset = repo_root.join("evals/datasets/longmemeval_s.json");
+ assert!(
+ dataset.exists(),
+ "fetch the dataset first: cargo run -p ai-memory-eval -- retrieval --fetch --sample 1"
+ );
+
+ let out = tempfile::tempdir().unwrap();
+ let status = Command::new(env!("CARGO_BIN_EXE_ai-memory-eval"))
+ .current_dir(&repo_root)
+ .args([
+ "retrieval",
+ "--candidate",
+ "--sample",
+ "2",
+ "--concurrency",
+ "2",
+ ])
+ .arg("--server-bin")
+ .arg(&server_bin)
+ .arg("--out")
+ .arg(out.path())
+ .status()
+ .unwrap();
+ assert!(status.success(), "A/B harness exited non-zero");
+
+ let run_dir = std::fs::read_dir(out.path())
+ .unwrap()
+ .map(|e| e.unwrap().path())
+ .find(|p| p.is_dir())
+ .expect("run dir created");
+ let report: serde_json::Value =
+ serde_json::from_slice(&std::fs::read(run_dir.join("report.json")).unwrap()).unwrap();
+ // A/B report shape: baseline + candidate + precomputed delta.
+ assert_eq!(report["baseline"]["questions_scored"], 2);
+ assert_eq!(report["candidate"]["questions_scored"], 2);
+ // Identical configs ⇒ deterministic accuracy and context; only latency
+ // (wall-clock) may wobble, so it is not asserted zero here.
+ let delta = &report["delta"];
+ for k in ["1", "3", "5", "10"] {
+ assert_eq!(delta["hit_at"][k], 0.0, "hit@{k} delta");
+ assert_eq!(delta["recall_at"][k], 0.0, "recall@{k} delta");
+ }
+ assert_eq!(delta["context_tokens_mean"], 0.0);
+ assert_eq!(delta["context_tokens_median"], 0.0);
+ assert!(run_dir.join("report.md").exists());
+}