Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(evals): measure answerability and abstention

AkitaOnRails6ae64ffb6417
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 85ed1d80..07809c90 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,17 @@ version in [`docs/changelog/`](docs/changelog/):
### Added
+- Added an offline `ai-memory-eval abstention` benchmark with versioned
+ clean-room answerable, hard-negative, conflicting, partial and
+ missing-channel fixtures; raw lexical/score/density/belief signals;
+ FSR/FAR/coverage threshold sweeps with undefined denominators preserved;
+ optional reuse of pinned LongMemEval `*_abs` cases; and optional structured
+ LLM answerability through the existing QA provider path. Conflicting and
+ partial cases remain separate from binary ground truth, no threshold is
+ presented as calibrated, and production runtime behavior is unchanged.
+ The retrieval evaluator now also parses production's structured
+ `{text,citations}` answer shape instead of a stale string (#1717).
+
- Added `[briefing] inject_on_subagent_start` to `.ai-memory.toml`. A project
that sets it hands each `claude-code` subagent the project brief when the
subagent starts: the same pinned, `_rules/` and `_slots/` brief that
diff --git a/evals/README.md b/evals/README.md
index 72f2e560..dccc4e1b 100644
--- a/evals/README.md
+++ b/evals/README.md
@@ -1,6 +1,6 @@
# `evals/` — evaluation harnesses
-One Rust binary (`ai-memory-eval`) with three subcommands:
+One Rust binary (`ai-memory-eval`) with four subcommands:
- **`ab`** — runs the EXACT consolidation prompt ai-memory uses in
production against two LLM providers side by side, saves both
@@ -12,13 +12,65 @@ One Rust binary (`ai-memory-eval`) with three subcommands:
- **`retrieval`** — LongMemEval and bundled coding-session retrieval checks,
driven through a real `ai-memory serve` subprocess. Published baselines
live in `docs/benchmarks/`.
+- **`abstention`** — offline evidence-sufficiency signals and threshold sweeps
+ over versioned answerable, hard-negative, conflicting, partial, and
+ missing-channel fixtures, with optional structured LLM answerability.
**This is not part of the shipped binary.** It's a workspace member
purely so it shares deps + builds with the rest. `cargo build` from
the repo root will compile it, but it's never bundled into the
docker image. CI builds it and runs its unit tests
-(`cargo nextest run --workspace --all-targets`); the `ab`,
-`projection` and `retrieval` commands themselves never run there.
+(`cargo nextest run --workspace --all-targets`); the `ab`, `projection`,
+`retrieval`, and `abstention` commands themselves never run there.
+
+## `abstention` — evidence sufficiency
+
+The default run is deterministic, offline, and keyless:
+
+```bash
+cargo run -p ai-memory-eval -- abstention
+```
+
+It reads `evals/fixtures/abstention/v1.json`, validates the version, unique
+ids, evidence paths, finite score/belief ranges, and required support for
+answerable cases, then writes `report.json` and `report.md` under a timestamped
+`evals/runs/*-abstention/` directory. Each case records raw normalized lexical
+coverage, top-score margin/distribution, matching-term density, snippet size,
+and belief aggregation only where the fixture actually supplies belief.
+
+The report sweeps lexical-coverage thresholds from 0.0 through 1.0 and records
+coverage, false-supported rate (FSR), and false-abstention rate (FAR), including
+undefined denominators. It does not designate a calibrated threshold: the
+clean-room corpus is a regression probe, not a representative sample.
+`answerable` maps to supported and `unanswerable` to unsupported;
+`conflicting` and `partial` remain separate slices and are excluded from both
+binary error-rate denominators.
+
+The fixture complements rather than relabels LongMemEval. To add the pinned
+LongMemEval-S `*_abs` questions as unanswerable cases, first fetch that dataset
+through the retrieval command, then opt in without any network access:
+
+```bash
+cargo run -p ai-memory-eval -- retrieval --fetch --sample 1
+cargo run -p ai-memory-eval -- abstention --longmemeval-abs
+```
+
+Optional `--qa` uses the exact `--qa-*` provider/auth construction shared with
+the retrieval evaluator. Synthesis returns a structured `answerability` enum
+(`answerable`, `insufficient`, `conflicting`, or `partial`); the evaluator never
+detects abstention from answer wording. Missing credentials produce a reported
+skip and leave the offline signal report intact. The retrieval QA path also
+parses production's `answer: { text, citations }` object; empty validated
+citations are an abstention, while nonempty citations are answerable for this
+evaluation metadata. This is still identity evidence, not a claim-level
+support proof.
+
+When adding cases, keep them clean-room and synthetic. `target_evidence` must
+name supplied paths, answerable cases require a `gold_answer`, and channel-loss
+cases declare `channels.fts` / `channels.vector` explicitly. Do not tune a
+threshold and the fixture against each other, and do not fold conflicting or
+partial evidence into binary ground truth without changing the documented
+mapping and report schema.
## When to reach for this
diff --git a/evals/fixtures/abstention/v1.json b/evals/fixtures/abstention/v1.json
new file mode 100644
index 00000000..78a04b8d
--- /dev/null
+++ b/evals/fixtures/abstention/v1.json
@@ -0,0 +1,108 @@
+{
+ "version": 1,
+ "cases": [
+ {
+ "id": "answerable-server-port",
+ "question": "Which port does the ai-memory server use?",
+ "expected_verdict": "answerable",
+ "slice": "baseline",
+ "evidence": [
+ {
+ "path": "rules/server.md",
+ "snippet": "The ai-memory server listens on port 49374.",
+ "score": 0.92,
+ "belief_strength": 0.88
+ }
+ ],
+ "target_evidence": ["rules/server.md"],
+ "gold_answer": "49374"
+ },
+ {
+ "id": "unrelated-coffee-question",
+ "question": "Which coffee grinder does the user own?",
+ "expected_verdict": "unanswerable",
+ "slice": "unrelated",
+ "evidence": [
+ {
+ "path": "rules/server.md",
+ "snippet": "The ai-memory server listens on port 49374.",
+ "score": 0.04
+ }
+ ]
+ },
+ {
+ "id": "hard-negative-backup-schedule",
+ "question": "At what exact time does the nightly backup start?",
+ "expected_verdict": "unanswerable",
+ "slice": "hard_negative",
+ "evidence": [
+ {
+ "path": "operations/backups.md",
+ "snippet": "Nightly backups upload encrypted archives and retain seven daily copies. The schedule is managed by the home server.",
+ "score": 0.89,
+ "belief_strength": 0.91
+ },
+ {
+ "path": "operations/restore.md",
+ "snippet": "Use the newest nightly backup when testing restore procedures.",
+ "score": 0.73
+ }
+ ]
+ },
+ {
+ "id": "conflicting-timeout-values",
+ "question": "What request timeout is active?",
+ "expected_verdict": "conflicting",
+ "slice": "conflicting",
+ "evidence": [
+ {
+ "path": "decisions/timeout-old.md",
+ "snippet": "The active request timeout is 15 seconds.",
+ "score": 0.86,
+ "belief_strength": 0.65
+ },
+ {
+ "path": "decisions/timeout-new.md",
+ "snippet": "The active request timeout is 45 seconds.",
+ "score": 0.84,
+ "belief_strength": 0.65
+ }
+ ],
+ "target_evidence": [
+ "decisions/timeout-old.md",
+ "decisions/timeout-new.md"
+ ]
+ },
+ {
+ "id": "partial-host-and-port",
+ "question": "Which host and port does the remote server use?",
+ "expected_verdict": "partial",
+ "slice": "partial",
+ "evidence": [
+ {
+ "path": "rules/server.md",
+ "snippet": "The remote server uses port 49374. Its hostname is not recorded here.",
+ "score": 0.82
+ }
+ ],
+ "target_evidence": ["rules/server.md"]
+ },
+ {
+ "id": "answerable-with-vector-channel-missing",
+ "question": "Which database stores the derived memory index?",
+ "expected_verdict": "answerable",
+ "slice": "missing_channel",
+ "channels": {"fts": true, "vector": false},
+ "evidence": [
+ {
+ "path": "architecture/storage.md",
+ "snippet": "SQLite stores the derived memory index; Markdown in git remains the source of truth.",
+ "score": 0.77,
+ "belief_strength": 0.94
+ }
+ ],
+ "target_evidence": ["architecture/storage.md"],
+ "gold_answer": "SQLite"
+ }
+ ]
+}
diff --git a/evals/src/abstention.rs b/evals/src/abstention.rs
new file mode 100644
index 00000000..2d2461a3
--- /dev/null
+++ b/evals/src/abstention.rs
@@ -0,0 +1,722 @@
+//! Offline evidence-sufficiency and abstention evaluation.
+
+use std::collections::{BTreeMap, HashSet};
+use std::path::{Path, PathBuf};
+
+use anyhow::{Context, Result, bail};
+use jiff::Timestamp;
+use serde::{Deserialize, Serialize};
+
+use crate::retrieval::dataset::{self, DatasetVariant};
+use crate::retrieval::qa::{Answerability, QaArgs, QaEngine};
+use crate::retrieval::query::HitContext;
+use crate::retrieval::report::QaReportMeta;
+
+const FIXTURE_VERSION: u32 = 1;
+const DEFAULT_THRESHOLDS: [f64; 11] = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0];
+
+#[derive(clap::Args, Debug)]
+pub struct AbstentionArgs {
+ /// Versioned clean-room answerability fixture.
+ #[arg(long, default_value = "evals/fixtures/abstention/v1.json")]
+ fixtures: PathBuf,
+ /// Output root; a timestamped run directory is created below it.
+ #[arg(long, default_value = "evals/runs")]
+ out: PathBuf,
+ /// Also evaluate the pinned LongMemEval-S `*_abs` questions. The pinned
+ /// dataset must already exist; this command never downloads it.
+ #[arg(long)]
+ longmemeval_abs: bool,
+ /// Existing QA provider flags, shared with `retrieval --qa`.
+ #[command(flatten)]
+ qa: QaArgs,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct FixtureSet {
+ version: u32,
+ cases: Vec<Case>,
+}
+
+#[derive(Debug, Clone, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct Case {
+ id: String,
+ question: String,
+ expected_verdict: Verdict,
+ slice: Slice,
+ evidence: Vec<Evidence>,
+ #[serde(default)]
+ target_evidence: Vec<String>,
+ #[serde(default)]
+ gold_answer: Option<String>,
+ #[serde(default)]
+ channels: Channels,
+}
+
+#[derive(Debug, Clone, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct Evidence {
+ path: String,
+ snippet: String,
+ score: f64,
+ #[serde(default)]
+ belief_strength: Option<f64>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+enum Verdict {
+ Answerable,
+ Unanswerable,
+ Conflicting,
+ Partial,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+enum Slice {
+ Baseline,
+ Unrelated,
+ HardNegative,
+ Conflicting,
+ Partial,
+ MissingChannel,
+ LongmemevalAbstention,
+}
+
+impl Slice {
+ fn name(self) -> &'static str {
+ match self {
+ Self::Baseline => "baseline",
+ Self::Unrelated => "unrelated",
+ Self::HardNegative => "hard_negative",
+ Self::Conflicting => "conflicting",
+ Self::Partial => "partial",
+ Self::MissingChannel => "missing_channel",
+ Self::LongmemevalAbstention => "longmemeval_abstention",
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, Deserialize, Serialize)]
+#[serde(default, deny_unknown_fields)]
+struct Channels {
+ fts: bool,
+ vector: bool,
+}
+
+impl Default for Channels {
+ fn default() -> Self {
+ Self {
+ fts: true,
+ vector: true,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Serialize)]
+struct Signals {
+ lexical_coverage: f64,
+ top_score: f64,
+ top_score_margin: Option<f64>,
+ score_mean: f64,
+ score_stddev: f64,
+ snippet_matching_term_density: f64,
+ snippet_chars: usize,
+ belief_mean: Option<f64>,
+ belief_observations: usize,
+}
+
+#[derive(Debug, Clone, Serialize)]
+struct CaseResult {
+ id: String,
+ expected_verdict: Verdict,
+ slice: Slice,
+ channels: Channels,
+ signals: Signals,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ qa: Option<QaResult>,
+}
+
+#[derive(Debug, Clone, Serialize)]
+struct QaResult {
+ answerability: Answerability,
+ matches_expected: bool,
+ answer: String,
+ latency_ms: u128,
+ estimated_tokens: usize,
+}
+
+#[derive(Debug, Clone, Serialize)]
+struct ThresholdMetrics {
+ threshold: f64,
+ supported: usize,
+ unsupported: usize,
+ ambiguous_excluded: usize,
+ predicted_answerable: usize,
+ false_supported_rate: Option<f64>,
+ false_abstention_rate: Option<f64>,
+ coverage: f64,
+}
+
+#[derive(Debug, Serialize)]
+struct SliceReport {
+ cases: usize,
+ thresholds: Vec<ThresholdMetrics>,
+}
+
+#[derive(Debug, Serialize)]
+struct Report {
+ generated_at: String,
+ commit: String,
+ hardware: String,
+ fixture_version: u32,
+ fixture_path: String,
+ decision_signal: &'static str,
+ binary_mapping: &'static str,
+ qa_status: String,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ qa: Option<QaReportMeta>,
+ qa_cases: usize,
+ qa_verdict_matches: usize,
+ qa_latency_p50_ms: Option<u128>,
+ qa_latency_p95_ms: Option<u128>,
+ seed: Option<u64>,
+ cases: Vec<CaseResult>,
+ thresholds: Vec<ThresholdMetrics>,
+ slices: BTreeMap<String, SliceReport>,
+}
+
+pub async fn run(args: AbstentionArgs) -> Result<()> {
+ let mut fixture = load_fixture(&args.fixtures)?;
+ if args.longmemeval_abs {
+ fixture.cases.extend(load_longmemeval_abstentions()?);
+ }
+
+ let (qa_engine, qa_status) = if args.qa.enabled {
+ match QaEngine::resolve(&args.qa) {
+ Ok(engine) => {
+ let status = format!("enabled: {}", engine.summary());
+ (Some(engine), status)
+ }
+ Err(error) => (None, format!("skipped: {error:#}")),
+ }
+ } else {
+ (None, "disabled (offline zero-LLM run)".to_string())
+ };
+
+ let mut results = Vec::with_capacity(fixture.cases.len());
+ for case in &fixture.cases {
+ let qa = if let Some(engine) = &qa_engine {
+ let contexts = case
+ .evidence
+ .iter()
+ .map(|e| HitContext {
+ title: e.path.clone(),
+ snippet: e.snippet.clone(),
+ })
+ .collect::<Vec<_>>();
+ let synthesized = engine.synthesize(&case.question, &contexts).await?;
+ Some(QaResult {
+ answerability: synthesized.answerability,
+ matches_expected: answerability_matches(
+ case.expected_verdict,
+ synthesized.answerability,
+ ),
+ answer: synthesized.answer,
+ latency_ms: synthesized.latency_ms,
+ estimated_tokens: synthesized.tokens,
+ })
+ } else {
+ None
+ };
+ results.push(CaseResult {
+ id: case.id.clone(),
+ expected_verdict: case.expected_verdict,
+ slice: case.slice,
+ channels: case.channels,
+ signals: signals(case),
+ qa,
+ });
+ }
+
+ let thresholds = threshold_sweep(&results);
+ let mut slices = BTreeMap::new();
+ for slice in [
+ Slice::Baseline,
+ Slice::Unrelated,
+ Slice::HardNegative,
+ Slice::Conflicting,
+ Slice::Partial,
+ Slice::MissingChannel,
+ Slice::LongmemevalAbstention,
+ ] {
+ let group = results
+ .iter()
+ .filter(|result| result.slice == slice)
+ .cloned()
+ .collect::<Vec<_>>();
+ if !group.is_empty() {
+ slices.insert(
+ slice.name().to_string(),
+ SliceReport {
+ cases: group.len(),
+ thresholds: threshold_sweep(&group),
+ },
+ );
+ }
+ }
+
+ let qa_latencies = results
+ .iter()
+ .filter_map(|result| result.qa.as_ref().map(|qa| qa.latency_ms))
+ .collect::<Vec<_>>();
+ let qa_cases = qa_latencies.len();
+ let qa_verdict_matches = results
+ .iter()
+ .filter_map(|result| result.qa.as_ref())
+ .filter(|qa| qa.matches_expected)
+ .count();
+ let report = Report {
+ generated_at: Timestamp::now().to_string(),
+ commit: crate::retrieval::report::commit_sha(),
+ hardware: crate::retrieval::report::hardware(),
+ fixture_version: fixture.version,
+ fixture_path: args.fixtures.display().to_string(),
+ decision_signal: "normalized lexical query-term coverage",
+ binary_mapping: "answerable=supported; unanswerable=unsupported; conflicting/partial excluded from FSR/FAR denominators",
+ qa_status,
+ qa: qa_engine.as_ref().map(QaEngine::report_meta),
+ qa_cases,
+ qa_verdict_matches,
+ qa_latency_p50_ms: percentile(&qa_latencies, 0.50),
+ qa_latency_p95_ms: percentile(&qa_latencies, 0.95),
+ seed: None,
+ cases: results,
+ thresholds,
+ slices,
+ };
+
+ let stamp: String = Timestamp::now()
+ .to_string()
+ .chars()
+ .map(|c| if c == ':' || c == '.' { '-' } else { c })
+ .collect();
+ let run_dir = args.out.join(format!("{stamp}-abstention"));
+ std::fs::create_dir_all(&run_dir)?;
+ std::fs::write(
+ run_dir.join("report.json"),
+ serde_json::to_vec_pretty(&report)?,
+ )?;
+ let markdown = report_markdown(&report);
+ std::fs::write(run_dir.join("report.md"), &markdown)?;
+ println!("{markdown}");
+ println!("run dir: {}", run_dir.display());
+ Ok(())
+}
+
+fn load_fixture(path: &Path) -> Result<FixtureSet> {
+ let bytes = std::fs::read(path).with_context(|| format!("reading {}", path.display()))?;
+ let fixture: FixtureSet =
+ serde_json::from_slice(&bytes).with_context(|| format!("parsing {}", path.display()))?;
+ validate_fixture(&fixture)?;
+ Ok(fixture)
+}
+
+fn validate_fixture(fixture: &FixtureSet) -> Result<()> {
+ if fixture.version != FIXTURE_VERSION {
+ bail!(
+ "fixture version {} is unsupported; expected {FIXTURE_VERSION}",
+ fixture.version
+ );
+ }
+ if fixture.cases.is_empty() {
+ bail!("fixture contains no cases");
+ }
+ let mut ids = HashSet::new();
+ for case in &fixture.cases {
+ if case.id.trim().is_empty() || !ids.insert(case.id.as_str()) {
+ bail!("fixture ids must be nonempty and unique: `{}`", case.id);
+ }
+ if case.question.trim().is_empty() || case.evidence.is_empty() {
+ bail!("{}: question and evidence must be nonempty", case.id);
+ }
+ let paths = case
+ .evidence
+ .iter()
+ .map(|e| e.path.as_str())
+ .collect::<HashSet<_>>();
+ for evidence in &case.evidence {
+ if evidence.path.trim().is_empty() || evidence.snippet.trim().is_empty() {
+ bail!("{}: evidence path and snippet must be nonempty", case.id);
+ }
+ if !evidence.score.is_finite() || !(0.0..=1.0).contains(&evidence.score) {
+ bail!("{}: evidence scores must be finite and in [0,1]", case.id);
+ }
+ if evidence
+ .belief_strength
+ .is_some_and(|v| !v.is_finite() || !(0.0..=1.0).contains(&v))
+ {
+ bail!("{}: belief strength must be finite and in [0,1]", case.id);
+ }
+ }
+ if case
+ .target_evidence
+ .iter()
+ .any(|target| !paths.contains(target.as_str()))
+ {
+ bail!(
+ "{}: target_evidence must name a supplied evidence path",
+ case.id
+ );
+ }
+ if case.expected_verdict == Verdict::Answerable
+ && (case.target_evidence.is_empty()
+ || case.gold_answer.as_deref().is_none_or(str::is_empty))
+ {
+ bail!(
+ "{}: answerable cases need target evidence and a gold answer",
+ case.id
+ );
+ }
+ }
+ Ok(())
+}
+
+fn load_longmemeval_abstentions() -> Result<Vec<Case>> {
+ let variant: &'static DatasetVariant = &dataset::VARIANT_S;
+ let path = Path::new(variant.default_path);
+ let questions = dataset::load(path, variant).with_context(|| {
+ "loading pinned LongMemEval abstentions; run `ai-memory-eval retrieval --fetch` first"
+ .to_string()
+ })?;
+ let cases = questions
+ .into_iter()
+ .filter(|question| question.is_abstention())
+ .map(|question| {
+ let evidence = question
+ .haystack_sessions
+ .into_iter()
+ .zip(question.haystack_session_ids)
+ .map(|(turns, session_id)| Evidence {
+ path: format!("sessions/{session_id}.md"),
+ snippet: turns
+ .into_iter()
+ .map(|turn| turn.content)
+ .collect::<Vec<_>>()
+ .join("\n"),
+ score: 0.0,
+ belief_strength: None,
+ })
+ .collect();
+ Case {
+ id: format!("longmemeval-{}", question.question_id),
+ question: question.question,
+ expected_verdict: Verdict::Unanswerable,
+ slice: Slice::LongmemevalAbstention,
+ evidence,
+ target_evidence: Vec::new(),
+ gold_answer: None,
+ channels: Channels::default(),
+ }
+ })
+ .collect();
+ Ok(cases)
+}
+
+fn content_terms(text: &str) -> Vec<String> {
+ const STOP: &[&str] = &[
+ "a", "an", "and", "are", "do", "does", "for", "how", "in", "is", "of", "on", "the", "to",
+ "was", "what", "which", "with",
+ ];
+ text.split(|c: char| !c.is_alphanumeric() && c != '_')
+ .map(str::to_lowercase)
+ .filter(|term| term.len() > 1 && !STOP.contains(&term.as_str()))
+ .collect()
+}
+
+fn signals(case: &Case) -> Signals {
+ let query_terms = content_terms(&case.question)
+ .into_iter()
+ .collect::<HashSet<_>>();
+ let snippets = case
+ .evidence
+ .iter()
+ .flat_map(|e| content_terms(&e.snippet))
+ .collect::<Vec<_>>();
+ let snippet_terms = snippets.iter().cloned().collect::<HashSet<_>>();
+ let matched = query_terms
+ .iter()
+ .filter(|term| snippet_terms.contains(*term))
+ .count();
+ let lexical_coverage = ratio(matched, query_terms.len()).unwrap_or(0.0);
+ let matching_occurrences = snippets
+ .iter()
+ .filter(|term| query_terms.contains(*term))
+ .count();
+ let snippet_matching_term_density = ratio(matching_occurrences, snippets.len()).unwrap_or(0.0);
+
+ let mut scores = case.evidence.iter().map(|e| e.score).collect::<Vec<_>>();
+ scores.sort_by(f64::total_cmp);
+ scores.reverse();
+ let top_score = scores.first().copied().unwrap_or(0.0);
+ let top_score_margin = scores.get(1).map(|second| top_score - second);
+ let score_mean = scores.iter().sum::<f64>() / scores.len() as f64;
+ let score_stddev = (scores
+ .iter()
+ .map(|score| (score - score_mean).powi(2))
+ .sum::<f64>()
+ / scores.len() as f64)
+ .sqrt();
+ let beliefs = case
+ .evidence
+ .iter()
+ .filter_map(|e| e.belief_strength)
+ .collect::<Vec<_>>();
+ let belief_mean =
+ (!beliefs.is_empty()).then(|| beliefs.iter().sum::<f64>() / beliefs.len() as f64);
+ Signals {
+ lexical_coverage,
+ top_score,
+ top_score_margin,
+ score_mean,
+ score_stddev,
+ snippet_matching_term_density,
+ snippet_chars: case
+ .evidence
+ .iter()
+ .map(|e| e.snippet.chars().count())
+ .sum(),
+ belief_mean,
+ belief_observations: beliefs.len(),
+ }
+}
+
+fn ratio(numerator: usize, denominator: usize) -> Option<f64> {
+ (denominator != 0).then(|| numerator as f64 / denominator as f64)
+}
+
+fn answerability_matches(expected: Verdict, actual: Answerability) -> bool {
+ matches!(
+ (expected, actual),
+ (Verdict::Answerable, Answerability::Answerable)
+ | (Verdict::Unanswerable, Answerability::Insufficient)
+ | (Verdict::Conflicting, Answerability::Conflicting)
+ | (Verdict::Partial, Answerability::Partial)
+ )
+}
+
+fn percentile(values: &[u128], quantile: f64) -> Option<u128> {
+ if values.is_empty() {
+ return None;
+ }
+ let mut sorted = values.to_vec();
+ sorted.sort_unstable();
+ let rank = (quantile * sorted.len() as f64).ceil().max(1.0) as usize;
+ sorted.get(rank - 1).copied()
+}
+
+fn threshold_sweep(cases: &[CaseResult]) -> Vec<ThresholdMetrics> {
+ DEFAULT_THRESHOLDS
+ .into_iter()
+ .map(|threshold| {
+ let supported = cases
+ .iter()
+ .filter(|case| case.expected_verdict == Verdict::Answerable)
+ .count();
+ let unsupported = cases
+ .iter()
+ .filter(|case| case.expected_verdict == Verdict::Unanswerable)
+ .count();
+ let ambiguous_excluded = cases.len() - supported - unsupported;
+ let predicted_answerable = cases
+ .iter()
+ .filter(|case| case.signals.lexical_coverage >= threshold)
+ .count();
+ let false_supported = cases
+ .iter()
+ .filter(|case| {
+ case.expected_verdict == Verdict::Unanswerable
+ && case.signals.lexical_coverage >= threshold
+ })
+ .count();
+ let false_abstained = cases
+ .iter()
+ .filter(|case| {
+ case.expected_verdict == Verdict::Answerable
+ && case.signals.lexical_coverage < threshold
+ })
+ .count();
+ ThresholdMetrics {
+ threshold,
+ supported,
+ unsupported,
+ ambiguous_excluded,
+ predicted_answerable,
+ false_supported_rate: ratio(false_supported, unsupported),
+ false_abstention_rate: ratio(false_abstained, supported),
+ coverage: ratio(predicted_answerable, cases.len()).unwrap_or(0.0),
+ }
+ })
+ .collect()
+}
+
+fn metric(value: Option<f64>) -> String {
+ value.map_or_else(|| "undefined".into(), |v| format!("{v:.3}"))
+}
+
+fn report_markdown(report: &Report) -> String {
+ let mut markdown = format!(
+ "# Abstention evaluation\n\n- commit: `{}`\n- fixture: `{}` (v{})\n- cases: {}\n- QA: {}\n- binary mapping: {}\n\n",
+ report.commit,
+ report.fixture_path,
+ report.fixture_version,
+ report.cases.len(),
+ report.qa_status,
+ report.binary_mapping,
+ );
+ if report.qa_cases > 0 {
+ markdown.push_str(&format!(
+ "- QA verdict matches: {}/{}; latency p50/p95: {}/{} ms\n\n",
+ report.qa_verdict_matches,
+ report.qa_cases,
+ report.qa_latency_p50_ms.unwrap_or_default(),
+ report.qa_latency_p95_ms.unwrap_or_default(),
+ ));
+ }
+ markdown.push_str("| lexical threshold | coverage | FSR | FAR | ambiguous excluded |\n");
+ markdown.push_str("|---:|---:|---:|---:|---:|\n");
+ for row in &report.thresholds {
+ markdown.push_str(&format!(
+ "| {:.1} | {:.3} | {} | {} | {} |\n",
+ row.threshold,
+ row.coverage,
+ metric(row.false_supported_rate),
+ metric(row.false_abstention_rate),
+ row.ambiguous_excluded,
+ ));
+ }
+ markdown.push_str("\nNo threshold is designated as calibrated; the table is a trade-off sweep over a small synthetic corpus.\n");
+ markdown
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn case(id: &str, verdict: Verdict, coverage_text: &str) -> Case {
+ Case {
+ id: id.into(),
+ question: "Which deployment port is configured?".into(),
+ expected_verdict: verdict,
+ slice: Slice::Baseline,
+ evidence: vec![Evidence {
+ path: "rules/server.md".into(),
+ snippet: coverage_text.into(),
+ score: 0.8,
+ belief_strength: Some(0.7),
+ }],
+ target_evidence: if verdict == Verdict::Answerable {
+ vec!["rules/server.md".into()]
+ } else {
+ Vec::new()
+ },
+ gold_answer: (verdict == Verdict::Answerable).then(|| "49374".into()),
+ channels: Channels::default(),
+ }
+ }
+
+ #[test]
+ fn fixture_validation_rejects_duplicate_ids_and_missing_support() {
+ let duplicate = FixtureSet {
+ version: 1,
+ cases: vec![
+ case("same", Verdict::Answerable, "deployment port 49374"),
+ case("same", Verdict::Unanswerable, "deployment docs"),
+ ],
+ };
+ assert!(
+ validate_fixture(&duplicate)
+ .unwrap_err()
+ .to_string()
+ .contains("unique")
+ );
+
+ let mut missing = case("missing", Verdict::Answerable, "deployment port 49374");
+ missing.target_evidence.clear();
+ assert!(
+ validate_fixture(&FixtureSet {
+ version: 1,
+ cases: vec![missing]
+ })
+ .unwrap_err()
+ .to_string()
+ .contains("target evidence")
+ );
+ }
+
+ #[test]
+ fn raw_signals_are_deterministic_and_belief_is_optional() {
+ let supported = case(
+ "supported",
+ Verdict::Answerable,
+ "The deployment uses configured port 49374.",
+ );
+ let first = signals(&supported);
+ let second = signals(&supported);
+ assert_eq!(first.lexical_coverage, second.lexical_coverage);
+ assert!(first.lexical_coverage > 0.5);
+ assert_eq!(first.belief_mean, Some(0.7));
+
+ let mut absent = supported;
+ absent.evidence[0].belief_strength = None;
+ assert_eq!(signals(&absent).belief_mean, None);
+ }
+
+ #[test]
+ fn threshold_metrics_preserve_undefined_denominators_and_ambiguous_cases() {
+ let cases = [
+ case("yes", Verdict::Answerable, "configured deployment port"),
+ case("no", Verdict::Unanswerable, "deployment guide only"),
+ case("partial", Verdict::Partial, "deployment port"),
+ ]
+ .into_iter()
+ .map(|case| CaseResult {
+ id: case.id.clone(),
+ expected_verdict: case.expected_verdict,
+ slice: case.slice,
+ channels: case.channels,
+ signals: signals(&case),
+ qa: None,
+ })
+ .collect::<Vec<_>>();
+ let sweep = threshold_sweep(&cases);
+ assert!(sweep.iter().all(|row| row.ambiguous_excluded == 1));
+ assert!(sweep.iter().all(|row| row.false_supported_rate.is_some()));
+ assert!(sweep.iter().all(|row| row.false_abstention_rate.is_some()));
+
+ let only_partial = vec![cases[2].clone()];
+ assert!(
+ threshold_sweep(&only_partial).iter().all(
+ |row| row.false_supported_rate.is_none() && row.false_abstention_rate.is_none()
+ )
+ );
+ }
+
+ #[test]
+ fn verdict_mapping_and_latency_percentiles_are_explicit() {
+ assert!(answerability_matches(
+ Verdict::Unanswerable,
+ Answerability::Insufficient
+ ));
+ assert!(!answerability_matches(
+ Verdict::Conflicting,
+ Answerability::Insufficient
+ ));
+ assert_eq!(percentile(&[], 0.95), None);
+ assert_eq!(percentile(&[30, 10, 20], 0.50), Some(20));
+ assert_eq!(percentile(&[30, 10, 20], 0.95), Some(30));
+ }
+}
diff --git a/evals/src/main.rs b/evals/src/main.rs
index dae05cc1..0e44af4f 100644
--- a/evals/src/main.rs
+++ b/evals/src/main.rs
@@ -1,6 +1,6 @@
//! `ai-memory-eval` — offline evaluation harnesses for ai-memory.
//!
-//! Three subcommands:
+//! Four subcommands:
//!
//! - `ab` — live A/B comparison of two LLM providers on the production
//! consolidation prompt, with literal-fidelity scoring (see [`ab`],
@@ -11,8 +11,11 @@
//! through a real `ai-memory serve` subprocess: hook-shaped ingestion,
//! real search stack, R@k / hit@k per question category (see
//! [`retrieval`] and `docs/benchmarks/`).
+//! - `abstention` — offline evidence-sufficiency signals, threshold sweeps,
+//! and optional structured LLM answerability evaluation.
mod ab;
+mod abstention;
mod fidelity;
mod generate;
mod projection;
@@ -41,6 +44,8 @@ enum Command {
Projection(projection::ProjectionArgs),
/// Pinned retrieval benchmarks through the real server stack.
Retrieval(Box<retrieval::RetrievalArgs>),
+ /// Offline evidence-sufficiency and abstention benchmark.
+ Abstention(Box<abstention::AbstentionArgs>),
}
#[tokio::main]
@@ -56,5 +61,6 @@ async fn main() -> Result<()> {
Command::Ab(args) => ab::run(*args).await,
Command::Projection(args) => projection::run(args),
Command::Retrieval(args) => retrieval::run(*args).await,
+ Command::Abstention(args) => abstention::run(*args).await,
}
}
diff --git a/evals/src/retrieval/mod.rs b/evals/src/retrieval/mod.rs
index 94804e35..fd9080c4 100644
--- a/evals/src/retrieval/mod.rs
+++ b/evals/src/retrieval/mod.rs
@@ -30,11 +30,11 @@
//! through the real page write path instead of replaying hooks;
//! - `--date-prefix turn|session-start|none` moves the per-turn date.
-mod dataset;
+pub(crate) mod dataset;
mod grep;
mod ingest;
-mod qa;
-mod query;
+pub(crate) mod qa;
+pub(crate) mod query;
mod reachability;
pub(crate) mod report;
mod score;
@@ -195,66 +195,8 @@ pub struct RetrievalArgs {
candidate_date_prefix: Option<DatePrefix>,
// ---- QA-accuracy mode (R2b, opt-in, live LLM) ----
- /// Enable end-to-end QA-accuracy grading (real LLM API calls). OFF by
- /// default. Per scored question the harness synthesizes an answer from
- /// the retrieved snippets (or uses a server-provided `answer` field when
- /// present) and grades it against the gold answer with an LLM judge.
- /// Requires `--qa-provider` + `--qa-model` and a resolvable key; if the
- /// key/provider cannot be resolved the run SKIPS QA (retrieval metrics
- /// are unaffected) instead of failing.
- #[arg(long)]
- qa: bool,
-
- /// QA answerer/judge provider
- /// (`anthropic|openai|openai-compat|openai-oauth|codex|copilot|gemini`).
- #[arg(long)]
- qa_provider: Option<String>,
-
- /// QA answerer model id.
- #[arg(long)]
- qa_model: Option<String>,
-
- /// QA answerer base URL (omit for native vendors).
- #[arg(long)]
- qa_base_url: Option<String>,
-
- /// QA answerer API key (raw value). Prefer `--qa-api-key-env` or the
- /// provider's default env var.
- #[arg(long)]
- qa_api_key: Option<String>,
-
- /// Env var to read the QA answerer API key from (defaults to the
- /// provider's canonical env var, e.g. `GEMINI_API_KEY`).
- #[arg(long)]
- qa_api_key_env: Option<String>,
-
- /// Auth file for a QA answerer using `openai-oauth`/`codex`/`copilot`.
- #[arg(long)]
- qa_token_file: Option<PathBuf>,
-
- /// QA grader provider; defaults to `--qa-provider`.
- #[arg(long)]
- qa_grader_provider: Option<String>,
-
- /// QA grader model; defaults to `--qa-model`.
- #[arg(long)]
- qa_grader_model: Option<String>,
-
- /// QA grader base URL; defaults to `--qa-base-url`.
- #[arg(long)]
- qa_grader_base_url: Option<String>,
-
- /// QA grader API key (raw); defaults to `--qa-api-key`.
- #[arg(long)]
- qa_grader_api_key: Option<String>,
-
- /// Env var for the QA grader API key; defaults to `--qa-api-key-env`.
- #[arg(long)]
- qa_grader_api_key_env: Option<String>,
-
- /// Auth file for the QA grader; defaults to `--qa-token-file`.
- #[arg(long)]
- qa_grader_token_file: Option<PathBuf>,
+ #[command(flatten)]
+ qa: qa::QaArgs,
}
impl RetrievalArgs {
@@ -471,7 +413,7 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
let run_dir = args.out.join(format!("{stamp}-retrieval"));
if args.engine == Engine::Grep {
- if args.ab_mode() || args.reachability || args.qa {
+ if args.ab_mode() || args.reachability || args.qa.enabled {
bail!("--engine grep runs alone: no candidate, --reachability, or --qa");
}
let report = run_grep(&args, variant, &questions, abstention.len(), &commit);
@@ -538,8 +480,8 @@ pub async fn run(args: RetrievalArgs) -> Result<()> {
// grader once and share them across both configs. If QA is requested but
// a provider/key can't be resolved, SKIP cleanly (retrieval metrics are
// unaffected) rather than failing the run — mirroring live-LLM tests.
- let qa_engine = if args.qa {
- match qa::QaEngine::resolve(&args) {
+ let qa_engine = if args.qa.enabled {
+ match qa::QaEngine::resolve(&args.qa) {
Ok(engine) => {
println!("QA-accuracy mode ON (live LLM): {}", engine.summary());
tracing::info!(qa = engine.summary(), "QA-accuracy mode enabled");
diff --git a/evals/src/retrieval/qa.rs b/evals/src/retrieval/qa.rs
index e70eeb53..26980e9c 100644
--- a/evals/src/retrieval/qa.rs
+++ b/evals/src/retrieval/qa.rs
@@ -35,10 +35,55 @@ use ai_memory_llm::{
ProviderConfig, Role, complete_structured,
};
-use super::RetrievalArgs;
use super::query::{HitContext, QueryOutcome};
use super::report::QaReportMeta;
+/// Shared opt-in QA provider arguments. Both retrieval QA and the abstention
+/// benchmark use this exact construction path so auth/provider behavior cannot
+/// drift between evaluators.
+#[derive(clap::Args, Debug, Default)]
+pub struct QaArgs {
+ /// Enable live-LLM QA. OFF by default; missing credentials produce a skip.
+ #[arg(long = "qa")]
+ pub enabled: bool,
+ /// QA answerer/judge provider.
+ #[arg(long)]
+ qa_provider: Option<String>,
+ /// QA answerer model id.
+ #[arg(long)]
+ qa_model: Option<String>,
+ /// QA answerer base URL (omit for native vendors).
+ #[arg(long)]
+ qa_base_url: Option<String>,
+ /// QA answerer API key. Prefer `--qa-api-key-env`.
+ #[arg(long)]
+ qa_api_key: Option<String>,
+ /// Env var containing the QA answerer API key.
+ #[arg(long)]
+ qa_api_key_env: Option<String>,
+ /// Auth file for OAuth/Codex/Copilot answerers.
+ #[arg(long)]
+ qa_token_file: Option<std::path::PathBuf>,
+ /// QA grader provider; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_provider: Option<String>,
+ /// QA grader model; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_model: Option<String>,
+ /// QA grader base URL; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_base_url: Option<String>,
+ /// QA grader API key; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_api_key: Option<String>,
+ /// Env var for the QA grader API key; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_api_key_env: Option<String>,
+ /// QA grader auth file; defaults to the answerer.
+ #[arg(long)]
+ qa_grader_token_file: Option<std::path::PathBuf>,
+}
+
/// chars/4 heuristic divisor, matching `query.rs`'s context-token estimate.
const CHARS_PER_TOKEN: usize = 4;
/// Cap on synthesized-answer output tokens (LongMemEval answers are short).
@@ -52,6 +97,9 @@ const GRADE_MAX_TOKENS: u32 = 256;
pub struct QaOutcome {
/// LLM-judge verdict: did the candidate correctly answer the question?
pub correct: bool,
+ /// Structured evidence-sufficiency verdict from synthesis (or from the
+ /// server answer's validated citation presence).
+ pub answerability: Answerability,
/// Where the candidate came from: `synthesized` (harness LLM over the
/// retrieved snippets) or `memory_query` (a server-provided `answer`).
pub answer_source: &'static str,
@@ -70,12 +118,27 @@ pub struct QaOutcome {
/// Answer synthesized from retrieved snippets (structured output).
#[derive(Debug, Clone, Deserialize, JsonSchema)]
-struct SynthesizedAnswer {
- /// The answer to the question, grounded strictly in the snippets, or
- /// "I don't know." when the snippets do not contain it.
+pub(crate) struct SynthesizedAnswer {
+ /// Typed verdict; evaluators never infer abstention from answer prose.
+ answerability: Answerability,
+ /// The answer to the question, grounded strictly in the snippets.
answer: String,
}
+/// Structured answerability classification used by both QA evaluators.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
+#[serde(rename_all = "snake_case")]
+pub enum Answerability {
+ /// The snippets contain enough evidence for the requested answer.
+ Answerable,
+ /// The snippets do not contain enough evidence.
+ Insufficient,
+ /// The snippets contain incompatible live claims.
+ Conflicting,
+ /// The snippets support only part of the requested answer.
+ Partial,
+}
+
/// LLM-as-judge verdict (structured output), LongMemEval-style.
#[derive(Debug, Clone, Deserialize, JsonSchema)]
struct Grade {
@@ -106,7 +169,7 @@ impl QaEngine {
/// provider's key is used when the operator does not name one, so
/// `--qa --qa-provider gemini --qa-model <m>` works with `GEMINI_API_KEY`
/// already in the environment.
- pub fn resolve(args: &RetrievalArgs) -> Result<Self> {
+ pub fn resolve(args: &QaArgs) -> Result<Self> {
let Some(answer_provider_str) = args.qa_provider.as_deref() else {
bail!("QA mode needs --qa-provider (e.g. anthropic|gemini|openai|openai-compat)");
};
@@ -178,18 +241,36 @@ impl QaEngine {
gold_answer: &str,
outcome: &QueryOutcome,
) -> Result<QaOutcome> {
- let (answer, answer_source, answer_latency_ms, answer_tokens) = match &outcome.answer {
- // Future `answer=true` feature: the server already produced the
- // answer, so grade it directly (no synthesis call).
- Some(server_answer) => (server_answer.clone(), "memory_query", 0u128, 0usize),
- None => {
- let synth = self.synthesize(question, &outcome.contexts).await?;
- (synth.answer, "synthesized", synth.latency_ms, synth.tokens)
- }
- };
+ let (answer, answerability, answer_source, answer_latency_ms, answer_tokens) =
+ match &outcome.answer {
+ // Future `answer=true` feature: the server already produced the
+ // answer, so grade it directly (no synthesis call).
+ Some(server_answer) => (
+ server_answer.text.clone(),
+ if server_answer.citations.is_empty() {
+ Answerability::Insufficient
+ } else {
+ Answerability::Answerable
+ },
+ "memory_query",
+ 0u128,
+ 0usize,
+ ),
+ None => {
+ let synth = self.synthesize(question, &outcome.contexts).await?;
+ (
+ synth.answer,
+ synth.answerability,
+ "synthesized",
+ synth.latency_ms,
+ synth.tokens,
+ )
+ }
+ };
let grade = self.grade(question, gold_answer, &answer).await?;
Ok(QaOutcome {
correct: grade.correct,
+ answerability,
answer_source,
answer,
grade_reason: grade.reason,
@@ -200,7 +281,11 @@ impl QaEngine {
/// Synthesize an answer from the retrieved snippets (the context an agent
/// would actually get), measuring latency and a chars/4 token estimate.
- async fn synthesize(&self, question: &str, contexts: &[HitContext]) -> Result<SynthResult> {
+ pub(crate) async fn synthesize(
+ &self,
+ question: &str,
+ contexts: &[HitContext],
+ ) -> Result<SynthResult> {
let user = synthesis_user_prompt(question, contexts);
let prompt_chars = SYNTHESIS_SYSTEM.chars().count() + user.chars().count();
let request = ChatRequest {
@@ -218,6 +303,7 @@ impl QaEngine {
let tokens = (prompt_chars + out.answer.chars().count()).div_ceil(CHARS_PER_TOKEN);
Ok(SynthResult {
answer: out.answer,
+ answerability: out.answerability,
latency_ms,
tokens,
})
@@ -240,10 +326,11 @@ impl QaEngine {
}
/// Internal synthesis result carrier.
-struct SynthResult {
- answer: String,
- latency_ms: u128,
- tokens: usize,
+pub(crate) struct SynthResult {
+ pub answer: String,
+ pub answerability: Answerability,
+ pub latency_ms: u128,
+ pub tokens: usize,
}
/// Convenience: borrow an inner `Option<String>` as `Option<&str>`, falling
@@ -259,8 +346,9 @@ impl OrRef for Option<String> {
const SYNTHESIS_SYSTEM: &str = "You answer a user's question using ONLY the memory snippets \
retrieved for them. Base your answer strictly on those snippets — do not use outside \
- knowledge or guess. Be concise and factual. If the snippets do not contain the answer, \
- reply exactly: I don't know.";
+ knowledge or guess. Classify `answerability` as `answerable`, `insufficient`, `conflicting`, \
+ or `partial`, then give a concise factual `answer`. For anything except `answerable`, explain \
+ the evidence limitation without filling the gap from outside knowledge.";
const GRADE_SYSTEM: &str = "You are a strict grader for a long-term-memory QA benchmark. Given a \
question, the gold (reference) answer, and a candidate answer, decide whether the candidate \
@@ -429,6 +517,22 @@ mod tests {
assert!(p.contains("Candidate answer:\ncand-y"));
}
+ #[test]
+ fn synthesis_requires_structured_answerability() {
+ let parsed: SynthesizedAnswer = serde_json::from_value(serde_json::json!({
+ "answerability": "insufficient",
+ "answer": "The snippets do not contain the port."
+ }))
+ .unwrap();
+ assert_eq!(parsed.answerability, Answerability::Insufficient);
+ assert!(
+ serde_json::from_value::<SynthesizedAnswer>(serde_json::json!({
+ "answer": "I don't know."
+ }))
+ .is_err()
+ );
+ }
+
#[test]
fn unknown_provider_is_rejected() {
let err = build_llm_provider("answerer", "nope", "m", None, None, None, None)
@@ -495,6 +599,7 @@ mod tests {
.await
.expect("QA evaluate succeeds with a live provider");
assert_eq!(qa.answer_source, "synthesized");
+ assert_eq!(qa.answerability, Answerability::Answerable);
assert!(qa.correct, "expected the judge to accept: {qa:?}");
}
}
diff --git a/evals/src/retrieval/query.rs b/evals/src/retrieval/query.rs
index 89c7358a..fe85820e 100644
--- a/evals/src/retrieval/query.rs
+++ b/evals/src/retrieval/query.rs
@@ -47,7 +47,7 @@ pub struct QueryOutcome {
/// `None` today; the QA path falls back to harness synthesis when it is
/// absent or blank. Detecting it now future-proofs R2b for the B4
/// `answer`/`reasoning` MCP features without a harness change.
- pub answer: Option<String>,
+ pub answer: Option<ServerAnswer>,
/// Wall-clock of the `memory_query` MCP round trip (request send →
/// response parsed).
pub latency: Duration,
@@ -74,7 +74,17 @@ struct QueryResponse {
/// Unknown fields are ignored by serde, so this is forward-compatible:
/// today's server simply never sets it.
#[serde(default)]
- answer: Option<String>,
+ answer: Option<ServerAnswer>,
+}
+
+/// Production `memory_query(answer=true)` response shape.
+#[derive(Debug, Clone, Deserialize)]
+pub struct ServerAnswer {
+ /// Synthesized answer text.
+ pub text: String,
+ /// Server-validated evidence paths. Empty means the server abstained.
+ #[serde(default)]
+ pub citations: Vec<String>,
}
#[derive(Debug, Deserialize)]
@@ -133,17 +143,15 @@ pub async fn memory_query(
)
.await?;
let latency = started.elapsed();
- let parsed: QueryResponse = serde_json::from_str(&text)
+ let mut parsed: QueryResponse = serde_json::from_str(&text)
.with_context(|| format!("parsing memory_query JSON: {text}"))?;
let raw_hits = parsed.raw_hits.len();
// A blank/whitespace `answer` is treated as absent so the QA path falls
// back to harness synthesis rather than grading an empty string.
let answer = parsed
.answer
- .as_deref()
- .map(str::trim)
- .filter(|s| !s.is_empty())
- .map(str::to_owned);
+ .take()
+ .filter(|answer| !answer.text.trim().is_empty());
let (retrieved, contexts, context_tokens) = flatten(parsed);
Ok(QueryOutcome {
retrieved,
@@ -342,4 +350,16 @@ mod tests {
assert_eq!(estimate_tokens(0), 0);
assert_eq!(estimate_tokens(8), 2);
}
+
+ #[test]
+ fn production_answer_object_is_parsed_with_citations() {
+ let parsed: QueryResponse = serde_json::from_value(serde_json::json!({
+ "hits": [],
+ "answer": {"text": "Use port 49374.", "citations": ["rules/server.md"]}
+ }))
+ .unwrap();
+ let answer = parsed.answer.unwrap();
+ assert_eq!(answer.text, "Use port 49374.");
+ assert_eq!(answer.citations, ["rules/server.md"]);
+ }
}
diff --git a/evals/src/retrieval/score.rs b/evals/src/retrieval/score.rs
index 1d9273a6..91ab5450 100644
--- a/evals/src/retrieval/score.rs
+++ b/evals/src/retrieval/score.rs
@@ -440,6 +440,7 @@ mod tests {
fn qa(correct: bool, latency_ms: u128, tokens: usize) -> QaOutcome {
QaOutcome {
correct,
+ answerability: crate::retrieval::qa::Answerability::Answerable,
answer_source: "synthesized",
answer: "a".into(),
grade_reason: "r".into(),