Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(evals): measure answerability and abstention
AkitaOnRails6ae64ffb6417
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 85ed1d80..07809c90 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,17 @@ version in [`docs/changelog/`](docs/changelog/):### Added+- Added an offline `ai-memory-eval abstention` benchmark with versioned+ clean-room answerable, hard-negative, conflicting, partial and+ missing-channel fixtures; raw lexical/score/density/belief signals;+ FSR/FAR/coverage threshold sweeps with undefined denominators preserved;+ optional reuse of pinned LongMemEval `*_abs` cases; and optional structured+ LLM answerability through the existing QA provider path. Conflicting and+ partial cases remain separate from binary ground truth, no threshold is+ presented as calibrated, and production runtime behavior is unchanged.+ The retrieval evaluator now also parses production's structured+ `{text,citations}` answer shape instead of a stale string (#1717).+- Added `[briefing] inject_on_subagent_start` to `.ai-memory.toml`. A projectthat sets it hands each `claude-code` subagent the project brief when thesubagent starts: the same pinned, `_rules/` and `_slots/` brief thatdiff --git a/evals/README.md b/evals/README.mdindex 72f2e560..dccc4e1b 100644--- a/evals/README.md+++ b/evals/README.md@@ -1,6 +1,6 @@# `evals/` — evaluation harnesses-One Rust binary (`ai-memory-eval`) with three subcommands:+One Rust binary (`ai-memory-eval`) with four subcommands:- **`ab`** — runs the EXACT consolidation prompt ai-memory uses inproduction against two LLM providers side by side, saves both@@ -12,13 +12,65 @@ One Rust binary (`ai-memory-eval`) with three subcommands:- **`retrieval`** — LongMemEval and bundled coding-session retrieval checks,driven through a real `ai-memory serve` subprocess. Published baselineslive in `docs/benchmarks/`.+- **`abstention`** — offline evidence-sufficiency signals and threshold sweeps+ over versioned answerable, hard-negative, conflicting, partial, and+ missing-channel fixtures, with optional structured LLM answerability.**This is not part of the shipped binary.** It's a workspace memberpurely so it shares deps + builds with the rest. `cargo build` fromthe repo root will compile it, but it's never bundled into thedocker image. CI builds it and runs its unit tests-(`cargo nextest run --workspace --all-targets`); the `ab`,-`projection` and `retrieval` commands themselves never run there.+(`cargo nextest run --workspace --all-targets`); the `ab`, `projection`,+`retrieval`, and `abstention` commands themselves never run there.++## `abstention` — evidence sufficiency++The default run is deterministic, offline, and keyless:++```bash+cargo run -p ai-memory-eval -- abstention+```++It reads `evals/fixtures/abstention/v1.json`, validates the version, unique+ids, evidence paths, finite score/belief ranges, and required support for+answerable cases, then writes `report.json` and `report.md` under a timestamped+`evals/runs/*-abstention/` directory. Each case records raw normalized lexical+coverage, top-score margin/distribution, matching-term density, snippet size,+and belief aggregation only where the fixture actually supplies belief.++The report sweeps lexical-coverage thresholds from 0.0 through 1.0 and records+coverage, false-supported rate (FSR), and false-abstention rate (FAR), including+undefined denominators. It does not designate a calibrated threshold: the+clean-room corpus is a regression probe, not a representative sample.+`answerable` maps to supported and `unanswerable` to unsupported;+`conflicting` and `partial` remain separate slices and are excluded from both+binary error-rate denominators.++The fixture complements rather than relabels LongMemEval. To add the pinned+LongMemEval-S `*_abs` questions as unanswerable cases, first fetch that dataset+through the retrieval command, then opt in without any network access:++```bash+cargo run -p ai-memory-eval -- retrieval --fetch --sample 1+cargo run -p ai-memory-eval -- abstention --longmemeval-abs+```++Optional `--qa` uses the exact `--qa-*` provider/auth construction shared with+the retrieval evaluator. Synthesis returns a structured `answerability` enum+(`answerable`, `insufficient`, `conflicting`, or `partial`); the evaluator never+detects abstention from answer wording. Missing credentials produce a reported+skip and leave the offline signal report intact. The retrieval QA path also+parses production's `answer: { text, citations }` object; empty validated+citations are an abstention, while nonempty citations are answerable for this+evaluation metadata. This is still identity evidence, not a claim-level+support proof.++When adding cases, keep them clean-room and synthetic. `target_evidence` must+name supplied paths, answerable cases require a `gold_answer`, and channel-loss+cases declare `channels.fts` / `channels.vector` explicitly. Do not tune a+threshold and the fixture against each other, and do not fold conflicting or+partial evidence into binary ground truth without changing the documented+mapping and report schema.## When to reach for thisdiff --git a/evals/fixtures/abstention/v1.json b/evals/fixtures/abstention/v1.jsonnew file mode 100644index 00000000..78a04b8d--- /dev/null+++ b/evals/fixtures/abstention/v1.json@@ -0,0 +1,108 @@+{+ "version": 1,+ "cases": [+ {+ "id": "answerable-server-port",+ "question": "Which port does the ai-memory server use?",+ "expected_verdict": "answerable",+ "slice": "baseline",+ "evidence": [+ {+ "path": "rules/server.md",+ "snippet": "The ai-memory server listens on port 49374.",+ "score": 0.92,+ "belief_strength": 0.88+ }+ ],+ "target_evidence": ["rules/server.md"],+ "gold_answer": "49374"+ },+ {+ "id": "unrelated-coffee-question",+ "question": "Which coffee grinder does the user own?",+ "expected_verdict": "unanswerable",+ "slice": "unrelated",+ "evidence": [+ {+ "path": "rules/server.md",+ "snippet": "The ai-memory server listens on port 49374.",+ "score": 0.04+ }+ ]+ },+ {+ "id": "hard-negative-backup-schedule",+ "question": "At what exact time does the nightly backup start?",+ "expected_verdict": "unanswerable",+ "slice": "hard_negative",+ "evidence": [+ {+ "path": "operations/backups.md",+ "snippet": "Nightly backups upload encrypted archives and retain seven daily copies. The schedule is managed by the home server.",+ "score": 0.89,+ "belief_strength": 0.91+ },+ {+ "path": "operations/restore.md",+ "snippet": "Use the newest nightly backup when testing restore procedures.",+ "score": 0.73+ }+ ]+ },+ {+ "id": "conflicting-timeout-values",+ "question": "What request timeout is active?",+ "expected_verdict": "conflicting",+ "slice": "conflicting",+ "evidence": [+ {+ "path": "decisions/timeout-old.md",+ "snippet": "The active request timeout is 15 seconds.",+ "score": 0.86,+ "belief_strength": 0.65+ },+ {+ "path": "decisions/timeout-new.md",+ "snippet": "The active request timeout is 45 seconds.",+ "score": 0.84,+ "belief_strength": 0.65+ }+ ],+ "target_evidence": [+ "decisions/timeout-old.md",+ "decisions/timeout-new.md"+ ]+ },+ {+ "id": "partial-host-and-port",+ "question": "Which host and port does the remote server use?",+ "expected_verdict": "partial",+ "slice": "partial",+ "evidence": [+ {+ "path": "rules/server.md",+ "snippet": "The remote server uses port 49374. Its hostname is not recorded here.",+ "score": 0.82+ }+ ],+ "target_evidence": ["rules/server.md"]+ },+ {+ "id": "answerable-with-vector-channel-missing",+ "question": "Which database stores the derived memory index?",+ "expected_verdict": "answerable",+ "slice": "missing_channel",+ "channels": {"fts": true, "vector": false},+ "evidence": [+ {+ "path": "architecture/storage.md",+ "snippet": "SQLite stores the derived memory index; Markdown in git remains the source of truth.",+ "score": 0.77,+ "belief_strength": 0.94+ }+ ],+ "target_evidence": ["architecture/storage.md"],+ "gold_answer": "SQLite"+ }+ ]+}diff --git a/evals/src/abstention.rs b/evals/src/abstention.rsnew file mode 100644index 00000000..2d2461a3--- /dev/null+++ b/evals/src/abstention.rs@@ -0,0 +1,722 @@+//! Offline evidence-sufficiency and abstention evaluation.++use std::collections::{BTreeMap, HashSet};+use std::path::{Path, PathBuf};++use anyhow::{Context, Result, bail};+use jiff::Timestamp;+use serde::{Deserialize, Serialize};++use crate::retrieval::dataset::{self, DatasetVariant};+use crate::retrieval::qa::{Answerability, QaArgs, QaEngine};+use crate::retrieval::query::HitContext;+use crate::retrieval::report::QaReportMeta;++const FIXTURE_VERSION: u32 = 1;+const DEFAULT_THRESHOLDS: [f64; 11] = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0];++#[derive(clap::Args, Debug)]+pub struct AbstentionArgs {+ /// Versioned clean-room answerability fixture.+ #[arg(long, default_value = "evals/fixtures/abstention/v1.json")]+ fixtures: PathBuf,+ /// Output root; a timestamped run directory is created below it.+ #[arg(long, default_value = "evals/runs")]+ out: PathBuf,+ /// Also evaluate the pinned LongMemEval-S `*_abs` questions. The pinned+ /// dataset must already exist; this command never downloads it.+ #[arg(long)]+ longmemeval_abs: bool,+ /// Existing QA provider flags, shared with `retrieval --qa`.+ #[command(flatten)]+ qa: QaArgs,+}++#[derive(Debug, Deserialize)]+#[serde(deny_unknown_fields)]+struct FixtureSet {+ version: u32,+ cases: Vec<Case>,+}++#[derive(Debug, Clone, Deserialize)]+#[serde(deny_unknown_fields)]+struct Case {+ id: String,+ question: String,+ expected_verdict: Verdict,+ slice: Slice,+ evidence: Vec<Evidence>,+ #[serde(default)]+ target_evidence: Vec<String>,+ #[serde(default)]+ gold_answer: Option<String>,+ #[serde(default)]+ channels: Channels,+}++#[derive(Debug, Clone, Deserialize)]+#[serde(deny_unknown_fields)]+struct Evidence {+ path: String,+ snippet: String,+ score: f64,+ #[serde(default)]+ belief_strength: Option<f64>,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+enum Verdict {+ Answerable,+ Unanswerable,+ Conflicting,+ Partial,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+enum Slice {+ Baseline,+ Unrelated,+ HardNegative,+ Conflicting,+ Partial,+ MissingChannel,+ LongmemevalAbstention,+}++impl Slice {+ fn name(self) -> &'static str {+ match self {+ Self::Baseline => "baseline",+ Self::Unrelated => "unrelated",+ Self::HardNegative => "hard_negative",+ Self::Conflicting => "conflicting",+ Self::Partial => "partial",+ Self::MissingChannel => "missing_channel",+ Self::LongmemevalAbstention => "longmemeval_abstention",+ }+ }+}++#[derive(Debug, Clone, Copy, Deserialize, Serialize)]+#[serde(default, deny_unknown_fields)]+struct Channels {+ fts: bool,+ vector: bool,+}++impl Default for Channels {+ fn default() -> Self {+ Self {+ fts: true,+ vector: true,+ }+ }+}++#[derive(Debug, Clone, Serialize)]+struct Signals {+ lexical_coverage: f64,+ top_score: f64,+ top_score_margin: Option<f64>,+ score_mean: f64,+ score_stddev: f64,+ snippet_matching_term_density: f64,+ snippet_chars: usize,+ belief_mean: Option<f64>,+ belief_observations: usize,+}++#[derive(Debug, Clone, Serialize)]+struct CaseResult {+ id: String,+ expected_verdict: Verdict,+ slice: Slice,+ channels: Channels,+ signals: Signals,+ #[serde(skip_serializing_if = "Option::is_none")]+ qa: Option<QaResult>,+}++#[derive(Debug, Clone, Serialize)]+struct QaResult {+ answerability: Answerability,+ matches_expected: bool,+ answer: String,+ latency_ms: u128,+ estimated_tokens: usize,+}++#[derive(Debug, Clone, Serialize)]+struct ThresholdMetrics {+ threshold: f64,+ supported: usize,+ unsupported: usize,+ ambiguous_excluded: usize,+ predicted_answerable: usize,+ false_supported_rate: Option<f64>,+ false_abstention_rate: Option<f64>,+ coverage: f64,+}++#[derive(Debug, Serialize)]+struct SliceReport {+ cases: usize,+ thresholds: Vec<ThresholdMetrics>,+}++#[derive(Debug, Serialize)]+struct Report {+ generated_at: String,+ commit: String,+ hardware: String,+ fixture_version: u32,+ fixture_path: String,+ decision_signal: &'static str,+ binary_mapping: &'static str,+ qa_status: String,+ #[serde(skip_serializing_if = "Option::is_none")]+ qa: Option<QaReportMeta>,+ qa_cases: usize,+ qa_verdict_matches: usize,+ qa_latency_p50_ms: Option<u128>,+ qa_latency_p95_ms: Option<u128>,+ seed: Option<u64>,+ cases: Vec<CaseResult>,+ thresholds: Vec<ThresholdMetrics>,+ slices: BTreeMap<String, SliceReport>,+}++pub async fn run(args: AbstentionArgs) -> Result<()> {+ let mut fixture = load_fixture(&args.fixtures)?;+ if args.longmemeval_abs {+ fixture.cases.extend(load_longmemeval_abstentions()?);+ }++ let (qa_engine, qa_status) = if args.qa.enabled {+ match QaEngine::resolve(&args.qa) {+ Ok(engine) => {+ let status = format!("enabled: {}", engine.summary());+ (Some(engine), status)+ }+ Err(error) => (None, format!("skipped: {error:#}")),+ }+ } else {+ (None, "disabled (offline zero-LLM run)".to_string())+ };++ let mut results = Vec::with_capacity(fixture.cases.len());+ for case in &fixture.cases {+ let qa = if let Some(engine) = &qa_engine {+ let contexts = case+ .evidence+ .iter()+ .map(|e| HitContext {+ title: e.path.clone(),+ snippet: e.snippet.clone(),+ })+ .collect::<Vec<_>>();+ let synthesized = engine.synthesize(&case.question, &contexts).await?;+ Some(QaResult {+ answerability: synthesized.answerability,+ matches_expected: answerability_matches(+ case.expected_verdict,+ synthesized.answerability,+ ),+ answer: synthesized.answer,+ latency_ms: synthesized.latency_ms,+ estimated_tokens: synthesized.tokens,+ })+ } else {+ None+ };+ results.push(CaseResult {+ id: case.id.clone(),+ expected_verdict: case.expected_verdict,+ slice: case.slice,+ channels: case.channels,+ signals: signals(case),+ qa,+ });+ }++ let thresholds = threshold_sweep(&results);+ let mut slices = BTreeMap::new();+ for slice in [+ Slice::Baseline,+ Slice::Unrelated,+ Slice::HardNegative,+ Slice::Conflicting,+ Slice::Partial,+ Slice::MissingChannel,+ Slice::LongmemevalAbstention,+ ] {+ let group = results+ .iter()+ .filter(|result| result.slice == slice)+ .cloned()+ .collect::<Vec<_>>();+ if !group.is_empty() {+ slices.insert(+ slice.name().to_string(),+ SliceReport {+ cases: group.len(),+ thresholds: threshold_sweep(&group),+ },+ );+ }+ }++ let qa_latencies = results+ .iter()+ .filter_map(|result| result.qa.as_ref().map(|qa| qa.latency_ms))+ .collect::<Vec<_>>();+ let qa_cases = qa_latencies.len();+ let qa_verdict_matches = results+ .iter()+ .filter_map(|result| result.qa.as_ref())+ .filter(|qa| qa.matches_expected)+ .count();+ let report = Report {+ generated_at: Timestamp::now().to_string(),+ commit: crate::retrieval::report::commit_sha(),+ hardware: crate::retrieval::report::hardware(),+ fixture_version: fixture.version,+ fixture_path: args.fixtures.display().to_string(),+ decision_signal: "normalized lexical query-term coverage",+ binary_mapping: "answerable=supported; unanswerable=unsupported; conflicting/partial excluded from FSR/FAR denominators",+ qa_status,+ qa: qa_engine.as_ref().map(QaEngine::report_meta),+ qa_cases,+ qa_verdict_matches,+ qa_latency_p50_ms: percentile(&qa_latencies, 0.50),+ qa_latency_p95_ms: percentile(&qa_latencies, 0.95),+ seed: None,+ cases: results,+ thresholds,+ slices,+ };++ let stamp: String = Timestamp::now()+ .to_string()+ .chars()+ .map(|c| if c == ':' || c == '.' { '-' } else { c })+ .collect();+ let run_dir = args.out.join(format!("{stamp}-abstention"));+ std::fs::create_dir_all(&run_dir)?;+ std::fs::write(+ run_dir.join("report.json"),+ serde_json::to_vec_pretty(&report)?,+ )?;+ let markdown = report_markdown(&report);+ std::fs::write(run_dir.join("report.md"), &markdown)?;+ println!("{markdown}");+ println!("run dir: {}", run_dir.display());+ Ok(())+}++fn load_fixture(path: &Path) -> Result<FixtureSet> {+ let bytes = std::fs::read(path).with_context(|| format!("reading {}", path.display()))?;+ let fixture: FixtureSet =+ serde_json::from_slice(&bytes).with_context(|| format!("parsing {}", path.display()))?;+ validate_fixture(&fixture)?;+ Ok(fixture)+}++fn validate_fixture(fixture: &FixtureSet) -> Result<()> {+ if fixture.version != FIXTURE_VERSION {+ bail!(+ "fixture version {} is unsupported; expected {FIXTURE_VERSION}",+ fixture.version+ );+ }+ if fixture.cases.is_empty() {+ bail!("fixture contains no cases");+ }+ let mut ids = HashSet::new();+ for case in &fixture.cases {+ if case.id.trim().is_empty() || !ids.insert(case.id.as_str()) {+ bail!("fixture ids must be nonempty and unique: `{}`", case.id);+ }+ if case.question.trim().is_empty() || case.evidence.is_empty() {+ bail!("{}: question and evidence must be nonempty", case.id);+ }+ let paths = case+ .evidence+ .iter()+ .map(|e| e.path.as_str())+ .collect::<HashSet<_>>();+ for evidence in &case.evidence {+ if evidence.path.trim().is_empty() || evidence.snippet.trim().is_empty() {+ bail!("{}: evidence path and snippet must be nonempty", case.id);+ }+ if !evidence.score.is_finite() || !(0.0..=1.0).contains(&evidence.score) {+ bail!("{}: evidence scores must be finite and in [0,1]", case.id);+ }+ if evidence+ .belief_strength+ .is_some_and(|v| !v.is_finite() || !(0.0..=1.0).contains(&v))+ {+ bail!("{}: belief strength must be finite and in [0,1]", case.id);+ }+ }+ if case+ .target_evidence+ .iter()+ .any(|target| !paths.contains(target.as_str()))+ {+ bail!(+ "{}: target_evidence must name a supplied evidence path",+ case.id+ );+ }+ if case.expected_verdict == Verdict::Answerable+ && (case.target_evidence.is_empty()+ || case.gold_answer.as_deref().is_none_or(str::is_empty))+ {+ bail!(+ "{}: answerable cases need target evidence and a gold answer",+ case.id+ );+ }+ }+ Ok(())+}++fn load_longmemeval_abstentions() -> Result<Vec<Case>> {+ let variant: &'static DatasetVariant = &dataset::VARIANT_S;+ let path = Path::new(variant.default_path);+ let questions = dataset::load(path, variant).with_context(|| {+ "loading pinned LongMemEval abstentions; run `ai-memory-eval retrieval --fetch` first"+ .to_string()+ })?;+ let cases = questions+ .into_iter()+ .filter(|question| question.is_abstention())+ .map(|question| {+ let evidence = question+ .haystack_sessions+ .into_iter()+ .zip(question.haystack_session_ids)+ .map(|(turns, session_id)| Evidence {+ path: format!("sessions/{session_id}.md"),+ snippet: turns+ .into_iter()+ .map(|turn| turn.content)+ .collect::<Vec<_>>()+ .join("\n"),+ score: 0.0,+ belief_strength: None,+ })+ .collect();+ Case {+ id: format!("longmemeval-{}", question.question_id),+ question: question.question,+ expected_verdict: Verdict::Unanswerable,+ slice: Slice::LongmemevalAbstention,+ evidence,+ target_evidence: Vec::new(),+ gold_answer: None,+ channels: Channels::default(),+ }+ })+ .collect();+ Ok(cases)+}++fn content_terms(text: &str) -> Vec<String> {+ const STOP: &[&str] = &[+ "a", "an", "and", "are", "do", "does", "for", "how", "in", "is", "of", "on", "the", "to",+ "was", "what", "which", "with",+ ];+ text.split(|c: char| !c.is_alphanumeric() && c != '_')+ .map(str::to_lowercase)+ .filter(|term| term.len() > 1 && !STOP.contains(&term.as_str()))+ .collect()+}++fn signals(case: &Case) -> Signals {+ let query_terms = content_terms(&case.question)+ .into_iter()+ .collect::<HashSet<_>>();+ let snippets = case+ .evidence+ .iter()+ .flat_map(|e| content_terms(&e.snippet))+ .collect::<Vec<_>>();+ let snippet_terms = snippets.iter().cloned().collect::<HashSet<_>>();+ let matched = query_terms+ .iter()+ .filter(|term| snippet_terms.contains(*term))+ .count();+ let lexical_coverage = ratio(matched, query_terms.len()).unwrap_or(0.0);+ let matching_occurrences = snippets+ .iter()+ .filter(|term| query_terms.contains(*term))+ .count();+ let snippet_matching_term_density = ratio(matching_occurrences, snippets.len()).unwrap_or(0.0);++ let mut scores = case.evidence.iter().map(|e| e.score).collect::<Vec<_>>();+ scores.sort_by(f64::total_cmp);+ scores.reverse();+ let top_score = scores.first().copied().unwrap_or(0.0);+ let top_score_margin = scores.get(1).map(|second| top_score - second);+ let score_mean = scores.iter().sum::<f64>() / scores.len() as f64;+ let score_stddev = (scores+ .iter()+ .map(|score| (score - score_mean).powi(2))+ .sum::<f64>()+ / scores.len() as f64)+ .sqrt();+ let beliefs = case+ .evidence+ .iter()+ .filter_map(|e| e.belief_strength)+ .collect::<Vec<_>>();+ let belief_mean =+ (!beliefs.is_empty()).then(|| beliefs.iter().sum::<f64>() / beliefs.len() as f64);+ Signals {+ lexical_coverage,+ top_score,+ top_score_margin,+ score_mean,+ score_stddev,+ snippet_matching_term_density,+ snippet_chars: case+ .evidence+ .iter()+ .map(|e| e.snippet.chars().count())+ .sum(),+ belief_mean,+ belief_observations: beliefs.len(),+ }+}++fn ratio(numerator: usize, denominator: usize) -> Option<f64> {+ (denominator != 0).then(|| numerator as f64 / denominator as f64)+}++fn answerability_matches(expected: Verdict, actual: Answerability) -> bool {+ matches!(+ (expected, actual),+ (Verdict::Answerable, Answerability::Answerable)+ | (Verdict::Unanswerable, Answerability::Insufficient)+ | (Verdict::Conflicting, Answerability::Conflicting)+ | (Verdict::Partial, Answerability::Partial)+ )+}++fn percentile(values: &[u128], quantile: f64) -> Option<u128> {+ if values.is_empty() {+ return None;+ }+ let mut sorted = values.to_vec();+ sorted.sort_unstable();+ let rank = (quantile * sorted.len() as f64).ceil().max(1.0) as usize;+ sorted.get(rank - 1).copied()+}++fn threshold_sweep(cases: &[CaseResult]) -> Vec<ThresholdMetrics> {+ DEFAULT_THRESHOLDS+ .into_iter()+ .map(|threshold| {+ let supported = cases+ .iter()+ .filter(|case| case.expected_verdict == Verdict::Answerable)+ .count();+ let unsupported = cases+ .iter()+ .filter(|case| case.expected_verdict == Verdict::Unanswerable)+ .count();+ let ambiguous_excluded = cases.len() - supported - unsupported;+ let predicted_answerable = cases+ .iter()+ .filter(|case| case.signals.lexical_coverage >= threshold)+ .count();+ let false_supported = cases+ .iter()+ .filter(|case| {+ case.expected_verdict == Verdict::Unanswerable+ && case.signals.lexical_coverage >= threshold+ })+ .count();+ let false_abstained = cases+ .iter()+ .filter(|case| {+ case.expected_verdict == Verdict::Answerable+ && case.signals.lexical_coverage < threshold+ })+ .count();+ ThresholdMetrics {+ threshold,+ supported,+ unsupported,+ ambiguous_excluded,+ predicted_answerable,+ false_supported_rate: ratio(false_supported, unsupported),+ false_abstention_rate: ratio(false_abstained, supported),+ coverage: ratio(predicted_answerable, cases.len()).unwrap_or(0.0),+ }+ })+ .collect()+}++fn metric(value: Option<f64>) -> String {+ value.map_or_else(|| "undefined".into(), |v| format!("{v:.3}"))+}++fn report_markdown(report: &Report) -> String {+ let mut markdown = format!(+ "# Abstention evaluation\n\n- commit: `{}`\n- fixture: `{}` (v{})\n- cases: {}\n- QA: {}\n- binary mapping: {}\n\n",+ report.commit,+ report.fixture_path,+ report.fixture_version,+ report.cases.len(),+ report.qa_status,+ report.binary_mapping,+ );+ if report.qa_cases > 0 {+ markdown.push_str(&format!(+ "- QA verdict matches: {}/{}; latency p50/p95: {}/{} ms\n\n",+ report.qa_verdict_matches,+ report.qa_cases,+ report.qa_latency_p50_ms.unwrap_or_default(),+ report.qa_latency_p95_ms.unwrap_or_default(),+ ));+ }+ markdown.push_str("| lexical threshold | coverage | FSR | FAR | ambiguous excluded |\n");+ markdown.push_str("|---:|---:|---:|---:|---:|\n");+ for row in &report.thresholds {+ markdown.push_str(&format!(+ "| {:.1} | {:.3} | {} | {} | {} |\n",+ row.threshold,+ row.coverage,+ metric(row.false_supported_rate),+ metric(row.false_abstention_rate),+ row.ambiguous_excluded,+ ));+ }+ markdown.push_str("\nNo threshold is designated as calibrated; the table is a trade-off sweep over a small synthetic corpus.\n");+ markdown+}++#[cfg(test)]+mod tests {+ use super::*;++ fn case(id: &str, verdict: Verdict, coverage_text: &str) -> Case {+ Case {+ id: id.into(),+ question: "Which deployment port is configured?".into(),+ expected_verdict: verdict,+ slice: Slice::Baseline,+ evidence: vec![Evidence {+ path: "rules/server.md".into(),+ snippet: coverage_text.into(),+ score: 0.8,+ belief_strength: Some(0.7),+ }],+ target_evidence: if verdict == Verdict::Answerable {+ vec!["rules/server.md".into()]+ } else {+ Vec::new()+ },+ gold_answer: (verdict == Verdict::Answerable).then(|| "49374".into()),+ channels: Channels::default(),+ }+ }++ #[test]+ fn fixture_validation_rejects_duplicate_ids_and_missing_support() {+ let duplicate = FixtureSet {+ version: 1,+ cases: vec![+ case("same", Verdict::Answerable, "deployment port 49374"),+ case("same", Verdict::Unanswerable, "deployment docs"),+ ],+ };+ assert!(+ validate_fixture(&duplicate)+ .unwrap_err()+ .to_string()+ .contains("unique")+ );++ let mut missing = case("missing", Verdict::Answerable, "deployment port 49374");+ missing.target_evidence.clear();+ assert!(+ validate_fixture(&FixtureSet {+ version: 1,+ cases: vec![missing]+ })+ .unwrap_err()+ .to_string()+ .contains("target evidence")+ );+ }++ #[test]+ fn raw_signals_are_deterministic_and_belief_is_optional() {+ let supported = case(+ "supported",+ Verdict::Answerable,+ "The deployment uses configured port 49374.",+ );+ let first = signals(&supported);+ let second = signals(&supported);+ assert_eq!(first.lexical_coverage, second.lexical_coverage);+ assert!(first.lexical_coverage > 0.5);+ assert_eq!(first.belief_mean, Some(0.7));++ let mut absent = supported;+ absent.evidence[0].belief_strength = None;+ assert_eq!(signals(&absent).belief_mean, None);+ }++ #[test]+ fn threshold_metrics_preserve_undefined_denominators_and_ambiguous_cases() {+ let cases = [+ case("yes", Verdict::Answerable, "configured deployment port"),+ case("no", Verdict::Unanswerable, "deployment guide only"),+ case("partial", Verdict::Partial, "deployment port"),+ ]+ .into_iter()+ .map(|case| CaseResult {+ id: case.id.clone(),+ expected_verdict: case.expected_verdict,+ slice: case.slice,+ channels: case.channels,+ signals: signals(&case),+ qa: None,+ })+ .collect::<Vec<_>>();+ let sweep = threshold_sweep(&cases);+ assert!(sweep.iter().all(|row| row.ambiguous_excluded == 1));+ assert!(sweep.iter().all(|row| row.false_supported_rate.is_some()));+ assert!(sweep.iter().all(|row| row.false_abstention_rate.is_some()));++ let only_partial = vec![cases[2].clone()];+ assert!(+ threshold_sweep(&only_partial).iter().all(+ |row| row.false_supported_rate.is_none() && row.false_abstention_rate.is_none()+ )+ );+ }++ #[test]+ fn verdict_mapping_and_latency_percentiles_are_explicit() {+ assert!(answerability_matches(+ Verdict::Unanswerable,+ Answerability::Insufficient+ ));+ assert!(!answerability_matches(+ Verdict::Conflicting,+ Answerability::Insufficient+ ));+ assert_eq!(percentile(&[], 0.95), None);+ assert_eq!(percentile(&[30, 10, 20], 0.50), Some(20));+ assert_eq!(percentile(&[30, 10, 20], 0.95), Some(30));+ }+}diff --git a/evals/src/main.rs b/evals/src/main.rsindex dae05cc1..0e44af4f 100644--- a/evals/src/main.rs+++ b/evals/src/main.rs@@ -1,6 +1,6 @@//! `ai-memory-eval` — offline evaluation harnesses for ai-memory.//!-//! Three subcommands:+//! Four subcommands://!//! - `ab` — live A/B comparison of two LLM providers on the production//! consolidation prompt, with literal-fidelity scoring (see [`ab`],@@ -11,8 +11,11 @@//! through a real `ai-memory serve` subprocess: hook-shaped ingestion,//! real search stack, R@k / hit@k per question category (see//! [`retrieval`] and `docs/benchmarks/`).+//! - `abstention` — offline evidence-sufficiency signals, threshold sweeps,+//! and optional structured LLM answerability evaluation.mod ab;+mod abstention;mod fidelity;mod generate;mod projection;@@ -41,6 +44,8 @@ enum Command {Projection(projection::ProjectionArgs),/// Pinned retrieval benchmarks through the real server stack.Retrieval(Box<retrieval::RetrievalArgs>),+ /// Offline evidence-sufficiency and abstention benchmark.+ Abstention(Box<abstention::AbstentionArgs>),}#[tokio::main]@@ -56,5 +61,6 @@ async fn main() -> Result<()> {Command::Ab(args) => ab::run(*args).await,Command::Projection(args) => projection::run(args),Command::Retrieval(args) => retrieval::run(*args).await,+ Command::Abstention(args) => abstention::run(*args).await,}}diff --git a/evals/src/retrieval/mod.rs b/evals/src/retrieval/mod.rsindex 94804e35..fd9080c4 100644--- a/evals/src/retrieval/mod.rs+++ b/evals/src/retrieval/mod.rs@@ -30,11 +30,11 @@//! through the real page write path instead of replaying hooks;//! - `--date-prefix turn|session-start|none` moves the per-turn date.-mod dataset;+pub(crate) mod dataset;mod grep;mod ingest;-mod qa;-mod query;+pub(crate) mod qa;+pub(crate) mod query;mod reachability;pub(crate) mod report;mod score;@@ -195,66 +195,8 @@ pub struct RetrievalArgs {candidate_date_prefix: Option<DatePrefix>,// ---- QA-accuracy mode (R2b, opt-in, live LLM) ----- /// Enable end-to-end QA-accuracy grading (real LLM API calls). OFF by- /// default. Per scored question the harness synthesizes an answer from- /// the retrieved snippets (or uses a server-provided `answer` field when- /// present) and grades it against the gold answer with an LLM judge.- /// Requires `--qa-provider` + `--qa-model` and a resolvable key; if the- /// key/provider cannot be resolved the run SKIPS QA (retrieval metrics- /// are unaffected) instead of failing.- #[arg(long)]- qa: bool,-- /// QA answerer/judge provider- /// (`anthropic|openai|openai-compat|openai-oauth|codex|copilot|gemini`).- #[arg(long)]- qa_provider: Option<String>,-- /// QA answerer model id.- #[arg(long)]- qa_model: Option<String>,-- /// QA answerer base URL (omit for native vendors).- #[arg(long)]- qa_base_url: Option<String>,-- /// QA answerer API key (raw value). Prefer `--qa-api-key-env` or the- /// provider's default env var.- #[arg(long)]- qa_api_key: Option<String>,-- /// Env var to read the QA answerer API key from (defaults to the- /// provider's canonical env var, e.g. `GEMINI_API_KEY`).- #[arg(long)]- qa_api_key_env: Option<String>,-- /// Auth file for a QA answerer using `openai-oauth`/`codex`/`copilot`.- #[arg(long)]- qa_token_file: Option<PathBuf>,-- /// QA grader provider; defaults to `--qa-provider`.- #[arg(long)]- qa_grader_provider: Option<String>,-- /// QA grader model; defaults to `--qa-model`.- #[arg(long)]- qa_grader_model: Option<String>,-- /// QA grader base URL; defaults to `--qa-base-url`.- #[arg(long)]- qa_grader_base_url: Option<String>,-- /// QA grader API key (raw); defaults to `--qa-api-key`.- #[arg(long)]- qa_grader_api_key: Option<String>,-- /// Env var for the QA grader API key; defaults to `--qa-api-key-env`.- #[arg(long)]- qa_grader_api_key_env: Option<String>,-- /// Auth file for the QA grader; defaults to `--qa-token-file`.- #[arg(long)]- qa_grader_token_file: Option<PathBuf>,+ #[command(flatten)]+ qa: qa::QaArgs,}impl RetrievalArgs {@@ -471,7 +413,7 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {let run_dir = args.out.join(format!("{stamp}-retrieval"));if args.engine == Engine::Grep {- if args.ab_mode() || args.reachability || args.qa {+ if args.ab_mode() || args.reachability || args.qa.enabled {bail!("--engine grep runs alone: no candidate, --reachability, or --qa");}let report = run_grep(&args, variant, &questions, abstention.len(), &commit);@@ -538,8 +480,8 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {// grader once and share them across both configs. If QA is requested but// a provider/key can't be resolved, SKIP cleanly (retrieval metrics are// unaffected) rather than failing the run — mirroring live-LLM tests.- let qa_engine = if args.qa {- match qa::QaEngine::resolve(&args) {+ let qa_engine = if args.qa.enabled {+ match qa::QaEngine::resolve(&args.qa) {Ok(engine) => {println!("QA-accuracy mode ON (live LLM): {}", engine.summary());tracing::info!(qa = engine.summary(), "QA-accuracy mode enabled");diff --git a/evals/src/retrieval/qa.rs b/evals/src/retrieval/qa.rsindex e70eeb53..26980e9c 100644--- a/evals/src/retrieval/qa.rs+++ b/evals/src/retrieval/qa.rs@@ -35,10 +35,55 @@ use ai_memory_llm::{ProviderConfig, Role, complete_structured,};-use super::RetrievalArgs;use super::query::{HitContext, QueryOutcome};use super::report::QaReportMeta;+/// Shared opt-in QA provider arguments. Both retrieval QA and the abstention+/// benchmark use this exact construction path so auth/provider behavior cannot+/// drift between evaluators.+#[derive(clap::Args, Debug, Default)]+pub struct QaArgs {+ /// Enable live-LLM QA. OFF by default; missing credentials produce a skip.+ #[arg(long = "qa")]+ pub enabled: bool,+ /// QA answerer/judge provider.+ #[arg(long)]+ qa_provider: Option<String>,+ /// QA answerer model id.+ #[arg(long)]+ qa_model: Option<String>,+ /// QA answerer base URL (omit for native vendors).+ #[arg(long)]+ qa_base_url: Option<String>,+ /// QA answerer API key. Prefer `--qa-api-key-env`.+ #[arg(long)]+ qa_api_key: Option<String>,+ /// Env var containing the QA answerer API key.+ #[arg(long)]+ qa_api_key_env: Option<String>,+ /// Auth file for OAuth/Codex/Copilot answerers.+ #[arg(long)]+ qa_token_file: Option<std::path::PathBuf>,+ /// QA grader provider; defaults to the answerer.+ #[arg(long)]+ qa_grader_provider: Option<String>,+ /// QA grader model; defaults to the answerer.+ #[arg(long)]+ qa_grader_model: Option<String>,+ /// QA grader base URL; defaults to the answerer.+ #[arg(long)]+ qa_grader_base_url: Option<String>,+ /// QA grader API key; defaults to the answerer.+ #[arg(long)]+ qa_grader_api_key: Option<String>,+ /// Env var for the QA grader API key; defaults to the answerer.+ #[arg(long)]+ qa_grader_api_key_env: Option<String>,+ /// QA grader auth file; defaults to the answerer.+ #[arg(long)]+ qa_grader_token_file: Option<std::path::PathBuf>,+}+/// chars/4 heuristic divisor, matching `query.rs`'s context-token estimate.const CHARS_PER_TOKEN: usize = 4;/// Cap on synthesized-answer output tokens (LongMemEval answers are short).@@ -52,6 +97,9 @@ const GRADE_MAX_TOKENS: u32 = 256;pub struct QaOutcome {/// LLM-judge verdict: did the candidate correctly answer the question?pub correct: bool,+ /// Structured evidence-sufficiency verdict from synthesis (or from the+ /// server answer's validated citation presence).+ pub answerability: Answerability,/// Where the candidate came from: `synthesized` (harness LLM over the/// retrieved snippets) or `memory_query` (a server-provided `answer`).pub answer_source: &'static str,@@ -70,12 +118,27 @@ pub struct QaOutcome {/// Answer synthesized from retrieved snippets (structured output).#[derive(Debug, Clone, Deserialize, JsonSchema)]-struct SynthesizedAnswer {- /// The answer to the question, grounded strictly in the snippets, or- /// "I don't know." when the snippets do not contain it.+pub(crate) struct SynthesizedAnswer {+ /// Typed verdict; evaluators never infer abstention from answer prose.+ answerability: Answerability,+ /// The answer to the question, grounded strictly in the snippets.answer: String,}+/// Structured answerability classification used by both QA evaluators.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]+#[serde(rename_all = "snake_case")]+pub enum Answerability {+ /// The snippets contain enough evidence for the requested answer.+ Answerable,+ /// The snippets do not contain enough evidence.+ Insufficient,+ /// The snippets contain incompatible live claims.+ Conflicting,+ /// The snippets support only part of the requested answer.+ Partial,+}+/// LLM-as-judge verdict (structured output), LongMemEval-style.#[derive(Debug, Clone, Deserialize, JsonSchema)]struct Grade {@@ -106,7 +169,7 @@ impl QaEngine {/// provider's key is used when the operator does not name one, so/// `--qa --qa-provider gemini --qa-model <m>` works with `GEMINI_API_KEY`/// already in the environment.- pub fn resolve(args: &RetrievalArgs) -> Result<Self> {+ pub fn resolve(args: &QaArgs) -> Result<Self> {let Some(answer_provider_str) = args.qa_provider.as_deref() else {bail!("QA mode needs --qa-provider (e.g. anthropic|gemini|openai|openai-compat)");};@@ -178,18 +241,36 @@ impl QaEngine {gold_answer: &str,outcome: &QueryOutcome,) -> Result<QaOutcome> {- let (answer, answer_source, answer_latency_ms, answer_tokens) = match &outcome.answer {- // Future `answer=true` feature: the server already produced the- // answer, so grade it directly (no synthesis call).- Some(server_answer) => (server_answer.clone(), "memory_query", 0u128, 0usize),- None => {- let synth = self.synthesize(question, &outcome.contexts).await?;- (synth.answer, "synthesized", synth.latency_ms, synth.tokens)- }- };+ let (answer, answerability, answer_source, answer_latency_ms, answer_tokens) =+ match &outcome.answer {+ // Future `answer=true` feature: the server already produced the+ // answer, so grade it directly (no synthesis call).+ Some(server_answer) => (+ server_answer.text.clone(),+ if server_answer.citations.is_empty() {+ Answerability::Insufficient+ } else {+ Answerability::Answerable+ },+ "memory_query",+ 0u128,+ 0usize,+ ),+ None => {+ let synth = self.synthesize(question, &outcome.contexts).await?;+ (+ synth.answer,+ synth.answerability,+ "synthesized",+ synth.latency_ms,+ synth.tokens,+ )+ }+ };let grade = self.grade(question, gold_answer, &answer).await?;Ok(QaOutcome {correct: grade.correct,+ answerability,answer_source,answer,grade_reason: grade.reason,@@ -200,7 +281,11 @@ impl QaEngine {/// Synthesize an answer from the retrieved snippets (the context an agent/// would actually get), measuring latency and a chars/4 token estimate.- async fn synthesize(&self, question: &str, contexts: &[HitContext]) -> Result<SynthResult> {+ pub(crate) async fn synthesize(+ &self,+ question: &str,+ contexts: &[HitContext],+ ) -> Result<SynthResult> {let user = synthesis_user_prompt(question, contexts);let prompt_chars = SYNTHESIS_SYSTEM.chars().count() + user.chars().count();let request = ChatRequest {@@ -218,6 +303,7 @@ impl QaEngine {let tokens = (prompt_chars + out.answer.chars().count()).div_ceil(CHARS_PER_TOKEN);Ok(SynthResult {answer: out.answer,+ answerability: out.answerability,latency_ms,tokens,})@@ -240,10 +326,11 @@ impl QaEngine {}/// Internal synthesis result carrier.-struct SynthResult {- answer: String,- latency_ms: u128,- tokens: usize,+pub(crate) struct SynthResult {+ pub answer: String,+ pub answerability: Answerability,+ pub latency_ms: u128,+ pub tokens: usize,}/// Convenience: borrow an inner `Option<String>` as `Option<&str>`, falling@@ -259,8 +346,9 @@ impl OrRef for Option<String> {const SYNTHESIS_SYSTEM: &str = "You answer a user's question using ONLY the memory snippets \retrieved for them. Base your answer strictly on those snippets — do not use outside \- knowledge or guess. Be concise and factual. If the snippets do not contain the answer, \- reply exactly: I don't know.";+ knowledge or guess. Classify `answerability` as `answerable`, `insufficient`, `conflicting`, \+ or `partial`, then give a concise factual `answer`. For anything except `answerable`, explain \+ the evidence limitation without filling the gap from outside knowledge.";const GRADE_SYSTEM: &str = "You are a strict grader for a long-term-memory QA benchmark. Given a \question, the gold (reference) answer, and a candidate answer, decide whether the candidate \@@ -429,6 +517,22 @@ mod tests {assert!(p.contains("Candidate answer:\ncand-y"));}+ #[test]+ fn synthesis_requires_structured_answerability() {+ let parsed: SynthesizedAnswer = serde_json::from_value(serde_json::json!({+ "answerability": "insufficient",+ "answer": "The snippets do not contain the port."+ }))+ .unwrap();+ assert_eq!(parsed.answerability, Answerability::Insufficient);+ assert!(+ serde_json::from_value::<SynthesizedAnswer>(serde_json::json!({+ "answer": "I don't know."+ }))+ .is_err()+ );+ }+#[test]fn unknown_provider_is_rejected() {let err = build_llm_provider("answerer", "nope", "m", None, None, None, None)@@ -495,6 +599,7 @@ mod tests {.await.expect("QA evaluate succeeds with a live provider");assert_eq!(qa.answer_source, "synthesized");+ assert_eq!(qa.answerability, Answerability::Answerable);assert!(qa.correct, "expected the judge to accept: {qa:?}");}}diff --git a/evals/src/retrieval/query.rs b/evals/src/retrieval/query.rsindex 89c7358a..fe85820e 100644--- a/evals/src/retrieval/query.rs+++ b/evals/src/retrieval/query.rs@@ -47,7 +47,7 @@ pub struct QueryOutcome {/// `None` today; the QA path falls back to harness synthesis when it is/// absent or blank. Detecting it now future-proofs R2b for the B4/// `answer`/`reasoning` MCP features without a harness change.- pub answer: Option<String>,+ pub answer: Option<ServerAnswer>,/// Wall-clock of the `memory_query` MCP round trip (request send →/// response parsed).pub latency: Duration,@@ -74,7 +74,17 @@ struct QueryResponse {/// Unknown fields are ignored by serde, so this is forward-compatible:/// today's server simply never sets it.#[serde(default)]- answer: Option<String>,+ answer: Option<ServerAnswer>,+}++/// Production `memory_query(answer=true)` response shape.+#[derive(Debug, Clone, Deserialize)]+pub struct ServerAnswer {+ /// Synthesized answer text.+ pub text: String,+ /// Server-validated evidence paths. Empty means the server abstained.+ #[serde(default)]+ pub citations: Vec<String>,}#[derive(Debug, Deserialize)]@@ -133,17 +143,15 @@ pub async fn memory_query().await?;let latency = started.elapsed();- let parsed: QueryResponse = serde_json::from_str(&text)+ let mut parsed: QueryResponse = serde_json::from_str(&text).with_context(|| format!("parsing memory_query JSON: {text}"))?;let raw_hits = parsed.raw_hits.len();// A blank/whitespace `answer` is treated as absent so the QA path falls// back to harness synthesis rather than grading an empty string.let answer = parsed.answer- .as_deref()- .map(str::trim)- .filter(|s| !s.is_empty())- .map(str::to_owned);+ .take()+ .filter(|answer| !answer.text.trim().is_empty());let (retrieved, contexts, context_tokens) = flatten(parsed);Ok(QueryOutcome {retrieved,@@ -342,4 +350,16 @@ mod tests {assert_eq!(estimate_tokens(0), 0);assert_eq!(estimate_tokens(8), 2);}++ #[test]+ fn production_answer_object_is_parsed_with_citations() {+ let parsed: QueryResponse = serde_json::from_value(serde_json::json!({+ "hits": [],+ "answer": {"text": "Use port 49374.", "citations": ["rules/server.md"]}+ }))+ .unwrap();+ let answer = parsed.answer.unwrap();+ assert_eq!(answer.text, "Use port 49374.");+ assert_eq!(answer.citations, ["rules/server.md"]);+ }}diff --git a/evals/src/retrieval/score.rs b/evals/src/retrieval/score.rsindex 1d9273a6..91ab5450 100644--- a/evals/src/retrieval/score.rs+++ b/evals/src/retrieval/score.rs@@ -440,6 +440,7 @@ mod tests {fn qa(correct: bool, latency_ms: u128, tokens: usize) -> QaOutcome {QaOutcome {correct,+ answerability: crate::retrieval::qa::Answerability::Answerable,answer_source: "synthesized",answer: "a".into(),grade_reason: "r".into(),