Skip to content

DeepTutor: Lifelong Personalized Tutoring. https://deeptutor.info/.

feat: add kb eval command for RAG retrieval quality scoring

Kung9808ca42ee5c
diff --git a/README.md b/README.md
index 5693726e4..5ddfdc206 100644
--- a/README.md
+++ b/README.md
@@ -796,6 +796,12 @@ The built-in LightRAG engine is installed with `pip install 'deeptutor[rag-light
Native LightRAG queries and incremental indexing require the embedding configuration recorded by the published index, including the model, dimension, and endpoint identity. If it changes, restore the original configuration or rebuild with the current embedding; indexes without a recorded embedding identity require a rebuild. The knowledge-base detail and index-version views show recovery guidance, while files remain available for viewing and download.
+**Scoring a knowledge base.** With several engines available, "which one retrieves best here?" is an empirical question — so `deeptutor kb eval` answers it. Write a QA set (one JSON object per line: a `query` plus the `gold` passages an ideal retrieval should return), then score the KB with `deeptutor kb eval <name> --dataset qa.jsonl --top-k 5 [--mode hybrid] [--save baseline.json]`. Each case is retrieved through the same path a chat turn uses, its ranked citations are matched against the gold passages (verbatim, truncated, or paraphrased — thresholds are tunable with `--min-ratio`), and the run reports **Recall@k / Precision@k / nDCG@k / MRR / MAP / Hit@k**. Metrics are pure functions over the ranking, so no model judges the output: scores are reproducible and diffable, and a saved baseline turns the set into a regression gate for an embedding switch, a reranker change, or a chunk-size experiment. Cases whose search failed (a missing index, bad credentials) carry no score and are reported separately rather than dragging the means down. PageIndex uses reasoning as retrieval and returns no ranked chunks, so it is excluded.
+
+```jsonl
+{"query": "Why do transformers scale attention?", "gold": ["Scaling keeps the dot products from growing with the dimension."]}
+```
+
</details>
<details>
@@ -957,7 +963,7 @@ The repo ships a root [`SKILL.md`](SKILL.md) — a ~200-line handover doc that t
| `deeptutor run <capability> <message>` | Run a single capability turn (`chat`, `ask_questions`, `deep_solve`, `deep_question`, `deep_research`, `visualize`, `math_animator`, `mastery_path`, `immersive_reading`, `course_study`, `immersive_watching`, `audio_overview`); add `--format json` for NDJSON output |
| `deeptutor chat` | Interactive REPL with capability, tool, KB, notebook, and history controls |
| `deeptutor partner list/create/start/stop` | Manage IM-connected partners |
-| `deeptutor kb list/info/create/add/search/set-default/delete/list-sources/sync` | Manage knowledge bases and synchronize registered GitHub/web sources (with source add/remove commands) |
+| `deeptutor kb list/info/create/add/search/eval/set-default/delete/list-sources/sync` | Manage knowledge bases, score retrieval quality against a QA set, and synchronize registered GitHub/web sources (with source add/remove commands) |
| `deeptutor skill search/install/list/remove/login/logout/publish/update` | Manage skills, install from hubs, and publish your own (`eduhub:<slug>` by default, see Ecosystem) |
| `deeptutor memory show/clear` | Inspect L2/L3 memory docs or clear L1/all memory |
| `deeptutor session list/show/open/rename/delete` | Manage shared sessions |
diff --git a/SKILL.md b/SKILL.md
index 92e29b468..8ef9e846a 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -70,6 +70,9 @@ deeptutor kb create <name> --doc file.pdf # Create from documents (--d
deeptutor kb create <name> --docs-dir ./papers # ...or from a directory of documents
deeptutor kb add <name> --doc more.pdf # Add documents incrementally
deeptutor kb search <name> "query text" [--mode hybrid] [--format rich|json]
+deeptutor kb eval <name> --dataset qa.jsonl [--top-k 5] [--mode hybrid] [--save report.json]
+# Scores retrieval quality against a QA set (JSONL: {"query": ..., "gold": [...]});
+# reports Recall@k / Precision@k / nDCG@k / MRR / MAP / Hit@k per case and averaged.
deeptutor kb set-default <name> # Set as default KB
deeptutor kb delete <name> [--force] # Delete a knowledge base
```
diff --git a/deeptutor/services/rag/eval/__init__.py b/deeptutor/services/rag/eval/__init__.py
new file mode 100644
index 000000000..785743977
--- /dev/null
+++ b/deeptutor/services/rag/eval/__init__.py
@@ -0,0 +1,87 @@
+"""Retrieval-quality evaluation for knowledge bases.
+
+``deeptutor kb eval`` scores a knowledge base against a QA set: each case pairs a
+query with the gold passages an ideal retrieval should return, and the run reports
+Recall@k, Precision@k, nDCG@k, MRR, MAP and Hit@k. The metrics are pure functions
+over the ranking — no model judges the output — so a score is reproducible, cheap
+to recompute, and comparable across engines, embedding models and retrieval
+settings. That is what makes the set usable as a regression gate: record a
+baseline with ``--save``, change a setting, and diff the metrics.
+
+Nothing here writes to a knowledge base or mutates its index; evaluation only
+reads through the same search path a chat turn uses.
+
+Example:
+ >>> from deeptutor.services.rag.eval import EvalDataset, evaluate_dataset
+ >>> dataset = EvalDataset.from_cases(
+ ... [{"query": "What is attention?", "gold": ["Attention weighs each token."]}]
+ ... )
+ >>> report = await evaluate_dataset(dataset, kb_name="ml-notes", k=5)
+ >>> report.aggregate()["recall@5"]
+"""
+
+from .dataset import EvalCase, EvalDataset, EvalDatasetError, load_dataset
+from .matching import (
+ DEFAULT_MIN_RATIO,
+ DEFAULT_MIN_TOKENS,
+ MatchPolicy,
+ MatchResult,
+ RelevanceMatcher,
+ normalize_text,
+ source_text,
+)
+from .metrics import (
+ QueryMetrics,
+ aggregate_query_metrics,
+ average_precision,
+ evaluate_hits,
+ hit_at_k,
+ hit_flags,
+ metric_keys,
+ ndcg_at_k,
+ precision_at_k,
+ recall_at_k,
+ reciprocal_rank,
+)
+from .report import EvalReport, format_metric
+from .runner import (
+ SOURCE_MODE_CITATIONS,
+ SOURCE_MODE_CONTEXT,
+ ProgressFn,
+ QueryEvaluation,
+ RetrievalEvaluator,
+ evaluate_dataset,
+)
+
+__all__ = [
+ "DEFAULT_MIN_RATIO",
+ "DEFAULT_MIN_TOKENS",
+ "EvalCase",
+ "EvalDataset",
+ "EvalDatasetError",
+ "EvalReport",
+ "MatchPolicy",
+ "MatchResult",
+ "ProgressFn",
+ "QueryEvaluation",
+ "QueryMetrics",
+ "RelevanceMatcher",
+ "RetrievalEvaluator",
+ "SOURCE_MODE_CITATIONS",
+ "SOURCE_MODE_CONTEXT",
+ "aggregate_query_metrics",
+ "average_precision",
+ "evaluate_dataset",
+ "evaluate_hits",
+ "format_metric",
+ "hit_at_k",
+ "hit_flags",
+ "load_dataset",
+ "metric_keys",
+ "ndcg_at_k",
+ "normalize_text",
+ "precision_at_k",
+ "recall_at_k",
+ "reciprocal_rank",
+ "source_text",
+]
diff --git a/deeptutor/services/rag/eval/dataset.py b/deeptutor/services/rag/eval/dataset.py
new file mode 100644
index 000000000..f22d04c93
--- /dev/null
+++ b/deeptutor/services/rag/eval/dataset.py
@@ -0,0 +1,244 @@
+"""QA-set schema for knowledge-base retrieval evaluation.
+
+An evaluation set is a list of cases; each case pairs one natural-language
+query with the source passages an ideal retrieval should return for it. The
+passages ("gold") are verbatim excerpts of the indexed documents, so a case
+stays meaningful across re-indexing, re-embedding, and engine changes — a chunk
+identifier does not survive an index rebuild, but document text does.
+
+Two on-disk shapes are supported:
+
+* ``.jsonl`` / ``.ndjson`` — one case object per line (recommended: a long set
+ stays reviewable in a diff);
+* ``.json`` — either a bare list of cases or ``{"name": ..., "cases": [...]}``.
+
+Case fields:
+
+* ``query`` (required) — the question or phrase handed to retrieval.
+* ``gold`` (required) — one excerpt or a list of excerpts that should be
+ retrieved. A single string is accepted as a one-passage list.
+* ``id`` (optional) — a stable label used in reports; defaults to ``q1``, ``q2``…
+* ``notes`` (optional) — free text carried through to the report.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Iterator, Sequence
+
+from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator
+
+JSONL_SUFFIXES = frozenset({".jsonl", ".ndjson"})
+
+
+class EvalDatasetError(ValueError):
+ """Raised when an evaluation set cannot be read or is structurally invalid."""
+
+
+class EvalCase(BaseModel):
+ """One query with the gold passages that should be retrieved for it."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ query: str
+ gold: list[str] = Field(min_length=1)
+ id: str = ""
+ notes: str = ""
+
+ @field_validator("query")
+ @classmethod
+ def _require_query(cls, value: str) -> str:
+ """Reject a blank query — there is nothing to retrieve for it."""
+ text = (value or "").strip()
+ if not text:
+ raise ValueError("query must be a non-empty string")
+ return text
+
+ @field_validator("gold", mode="before")
+ @classmethod
+ def _coerce_gold(cls, value: Any) -> Any:
+ """Accept a single excerpt string as a one-passage gold list."""
+ if isinstance(value, str):
+ return [value]
+ return value
+
+ @field_validator("gold")
+ @classmethod
+ def _require_gold(cls, value: list[str]) -> list[str]:
+ """Reject an empty or blank gold list — matching would be meaningless."""
+ passages = [(item or "").strip() for item in value]
+ if not passages or any(not passage for passage in passages):
+ raise ValueError("gold must hold at least one non-empty passage")
+ return passages
+
+ @field_validator("id", "notes", mode="before")
+ @classmethod
+ def _coerce_text(cls, value: Any) -> str:
+ """Treat a missing/null label or note as an empty string."""
+ return "" if value is None else str(value).strip()
+
+
+class EvalDataset(BaseModel):
+ """A named collection of evaluation cases."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ name: str = "eval-set"
+ cases: list[EvalCase] = Field(min_length=1)
+ source: str = ""
+
+ def __len__(self) -> int:
+ """Number of cases in the set."""
+ return len(self.cases)
+
+ def iter_cases(self) -> Iterator[EvalCase]:
+ """Iterate the cases in file order."""
+ return iter(self.cases)
+
+ def limited(self, limit: int | None) -> EvalDataset:
+ """Return a copy holding at most ``limit`` cases (``None`` keeps all)."""
+ if limit is None or limit >= len(self.cases):
+ return self
+ return self.model_copy(update={"cases": self.cases[: max(0, limit)]})
+
+ @classmethod
+ def from_cases(cls, cases: Sequence[EvalCase | dict], *, name: str = "eval-set") -> EvalDataset:
+ """Build a dataset programmatically (used by tests and scripts)."""
+ return cls(name=name, cases=[_coerce_case(case, index) for index, case in enumerate(cases)])
+
+
+def _coerce_case(case: EvalCase | dict, index: int) -> EvalCase:
+ """Validate one in-memory case, filling in a default id."""
+ if isinstance(case, EvalCase):
+ return case if case.id else case.model_copy(update={"id": f"q{index + 1}"})
+ validated = EvalCase.model_validate(case)
+ return validated if validated.id else validated.model_copy(update={"id": f"q{index + 1}"})
+
+
+def _first_error(exc: ValidationError) -> str:
+ """Flatten a pydantic error into one readable line."""
+ error = exc.errors()[0]
+ location = ".".join(str(part) for part in error.get("loc", ())) or "case"
+ return f"{location}: {error.get('msg', 'invalid value')}"
+
+
+def _case_context(raw: Any) -> str:
+ """A short excerpt of the offending case, to make errors actionable."""
+ if isinstance(raw, dict):
+ query = raw.get("query")
+ if isinstance(query, str) and query.strip():
+ return f" (query: {query.strip()[:48]!r})"
+ return ""
+
+
+def _build_case(raw: Any, index: int, path: Path) -> EvalCase:
+ """Validate one raw case object, raising with file/line context."""
+ if not isinstance(raw, dict):
+ raise EvalDatasetError(
+ f"{path}: case #{index + 1} must be a JSON object, got {type(raw).__name__}."
+ )
+ try:
+ case = EvalCase.model_validate(raw)
+ except ValidationError as exc:
+ raise EvalDatasetError(
+ f"{path}: case #{index + 1}{_case_context(raw)} is invalid — {_first_error(exc)}"
+ ) from exc
+ return case if case.id else case.model_copy(update={"id": f"q{index + 1}"})
+
+
+def _parse_jsonl(text: str, path: Path) -> list[Any]:
+ """Parse one case object per non-blank line, reporting the failing line."""
+ cases: list[Any] = []
+ for line_number, line in enumerate(text.splitlines(), start=1):
+ if not line.strip():
+ continue
+ try:
+ cases.append(json.loads(line))
+ except json.JSONDecodeError as exc:
+ raise EvalDatasetError(
+ f"{path}: line {line_number} is not valid JSON — {exc.msg}"
+ ) from exc
+ return cases
+
+
+def _parse_json(text: str, path: Path) -> tuple[list[Any], str]:
+ """Parse either a bare case list or an object with ``name`` and ``cases``."""
+ try:
+ payload = json.loads(text)
+ except json.JSONDecodeError as exc:
+ raise EvalDatasetError(f"{path}: is not valid JSON — {exc.msg}") from exc
+ if isinstance(payload, list):
+ return payload, ""
+ if isinstance(payload, dict):
+ cases = payload.get("cases")
+ if not isinstance(cases, list):
+ raise EvalDatasetError(f"{path}: object form must hold a 'cases' list.")
+ name = payload.get("name")
+ return cases, str(name).strip() if isinstance(name, str) else ""
+ raise EvalDatasetError(f"{path}: must hold a list of cases or an object with 'cases'.")
+
+
+def _reject_duplicate_ids(cases: Sequence[EvalCase], path: Path) -> None:
+ """Fail on repeated case ids so report rows cannot be ambiguous."""
+ seen: set[str] = set()
+ for case in cases:
+ if case.id in seen:
+ raise EvalDatasetError(f"{path}: duplicate case id {case.id!r}.")
+ seen.add(case.id)
+
+
+def load_dataset(path: str | Path, *, name: str = "") -> EvalDataset:
+ """Read an evaluation set from ``path``.
+
+ Args:
+ path: A ``.jsonl``, ``.ndjson`` or ``.json`` file.
+ name: Overrides the set name; defaults to the file's ``name`` field,
+ then to the file stem.
+
+ Returns:
+ The parsed, validated dataset.
+
+ Raises:
+ EvalDatasetError: The file is missing, unreadable, of an unsupported
+ format, empty, or holds an invalid or duplicated case.
+ """
+ target = Path(path).expanduser()
+ if not target.exists():
+ raise EvalDatasetError(f"Evaluation set not found: {target}")
+ if target.is_dir():
+ raise EvalDatasetError(f"Evaluation set must be a file, not a directory: {target}")
+
+ suffix = target.suffix.lower()
+ if suffix not in JSONL_SUFFIXES and suffix != ".json":
+ raise EvalDatasetError(
+ f"Unsupported evaluation set format {target.suffix!r}; use .jsonl or .json."
+ )
+
+ try:
+ text = target.read_text(encoding="utf-8")
+ except OSError as exc:
+ raise EvalDatasetError(f"Could not read {target}: {exc}") from exc
+ except UnicodeDecodeError as exc:
+ raise EvalDatasetError(f"{target} is not UTF-8 text: {exc}") from exc
+
+ if suffix in JSONL_SUFFIXES:
+ raw_cases, set_name = _parse_jsonl(text, target), ""
+ else:
+ raw_cases, set_name = _parse_json(text, target)
+
+ if not raw_cases:
+ raise EvalDatasetError(f"Evaluation set {target} holds no cases.")
+
+ cases = [_build_case(raw, index, target) for index, raw in enumerate(raw_cases)]
+ _reject_duplicate_ids(cases, target)
+ return EvalDataset(name=name or set_name or target.stem, cases=cases, source=str(target))
+
+
+__all__ = [
+ "EvalCase",
+ "EvalDataset",
+ "EvalDatasetError",
+ "JSONL_SUFFIXES",
+ "load_dataset",
+]
diff --git a/deeptutor/services/rag/eval/matching.py b/deeptutor/services/rag/eval/matching.py
new file mode 100644
index 000000000..cab037804
--- /dev/null
+++ b/deeptutor/services/rag/eval/matching.py
@@ -0,0 +1,218 @@
+"""Gold-passage matching for retrieval evaluation.
+
+An engine returns ranked chunks, not ground-truth identifiers, so relevance has
+to be decided from text. A chunk counts as covering a gold passage when either
+
+* one normalized text contains the other — the common cases are a chunk holding
+ the excerpt verbatim, and a *truncated* citation whose opening text sits inside
+ a longer gold passage (LlamaIndex and GraphRAG both cap ``content`` at 200
+ characters); or
+* the two share enough tokens — this catches a paraphrased or synthesized
+ citation (LightRAG reports, GraphRAG community summaries) that never repeats
+ the excerpt word for word.
+
+The token rule needs both a coverage ratio and a minimum shared-token count, so a
+short generic phrase ("the model") cannot claim a long passage it merely happens
+to appear in. Tokens are ASCII words plus individual CJK characters, which keeps
+a Chinese excerpt comparable without a segmenter dependency.
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+import re
+from typing import Any
+
+#: Share of the shorter side's tokens that must overlap for an overlap match.
+DEFAULT_MIN_RATIO = 0.5
+#: Shared tokens required before an overlap match is accepted.
+DEFAULT_MIN_TOKENS = 4
+
+_WHITESPACE = re.compile(r"\s+")
+# ASCII words and digits, or one token per CJK / Kana / Hangul character.
+_TOKEN = re.compile(
+ r"[0-9a-z]+"
+ r"|[\u3040-\u30ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff\uac00-\ud7af]"
+)
+
+#: Keys that hold a citation's text across the shipped engines.
+SOURCE_TEXT_KEYS = ("content", "text", "snippet", "excerpt", "summary")
+
+
+def normalize_text(value: str) -> str:
+ """Lower-case ``value`` and collapse whitespace for containment checks."""
+ return _WHITESPACE.sub(" ", (value or "").strip().lower())
+
+
+def tokenize(value: str) -> list[str]:
+ """Split ``value`` into comparison tokens (ASCII words + CJK characters)."""
+ return _TOKEN.findall((value or "").lower())
+
+
+def source_text(source: Any) -> str:
+ """Return the text of one citation record, across the engines' shapes.
+
+ Every pipeline normalizes what it retrieved into ``content``, but a
+ hand-written record or a future engine may use another key; an unrecognized
+ or non-mapping record yields an empty string, which never matches.
+ """
+ if not isinstance(source, Mapping):
+ return ""
+ for key in SOURCE_TEXT_KEYS:
+ value = source.get(key)
+ if isinstance(value, str) and value.strip():
+ return value
+ return ""
+
+
+@dataclass(frozen=True)
+class MatchPolicy:
+ """Thresholds for treating a retrieved chunk as covering a gold passage."""
+
+ min_ratio: float = DEFAULT_MIN_RATIO
+ min_tokens: int = DEFAULT_MIN_TOKENS
+
+ def __post_init__(self) -> None:
+ """Reject thresholds that would make matching either trivial or impossible."""
+ if not 0.0 < float(self.min_ratio) <= 1.0:
+ raise ValueError("min_ratio must be in (0, 1]")
+ if int(self.min_tokens) < 1:
+ raise ValueError("min_tokens must be >= 1")
+
+
+@dataclass(frozen=True)
+class MatchResult:
+ """Which gold passages a chunk covers, plus the strongest coverage ratio."""
+
+ indices: tuple[int, ...] = ()
+ ratio: float = 0.0
+
+ def __bool__(self) -> bool:
+ """True when the chunk covers at least one gold passage."""
+ return bool(self.indices)
+
+
+@dataclass(frozen=True)
+class GoldPassage:
+ """A gold passage prepared once, then matched against every retrieved chunk."""
+
+ index: int
+ text: str
+ normalized: str
+ tokens: Counter[str]
+ token_count: int
+
+ @classmethod
+ def from_text(cls, index: int, text: str) -> GoldPassage:
+ """Tokenize one gold passage for repeated comparison."""
+ token_list = tokenize(text)
+ return cls(
+ index=index,
+ text=text,
+ normalized=normalize_text(text),
+ tokens=Counter(token_list),
+ token_count=len(token_list),
+ )
+
+
+def _overlap(
+ chunk_tokens: Counter[str],
+ chunk_count: int,
+ passage: GoldPassage,
+) -> tuple[float, int]:
+ """Return ``(ratio, shared)`` for one chunk/passage pair.
+
+ The ratio divides by the shorter side's token count, so a truncated citation
+ whose tokens are all inside the passage still scores 1.0.
+ """
+ if chunk_count <= 0 or passage.token_count <= 0:
+ return 0.0, 0
+ shared = 0
+ for token, count in chunk_tokens.items():
+ passage_count = passage.tokens.get(token, 0)
+ if passage_count:
+ shared += min(count, passage_count)
+ shorter = min(chunk_count, passage.token_count)
+ return shared / shorter, shared
+
+
+class RelevanceMatcher:
+ """Decide whether retrieved chunks cover one case's gold passages.
+
+ The matcher is built per case: it prepares each gold passage once and then
+ scores every ranked chunk against them.
+
+ Args:
+ gold: The case's gold excerpts, in dataset order.
+ policy: Matching thresholds; defaults to :class:`MatchPolicy`.
+ """
+
+ def __init__(self, gold: Sequence[str], policy: MatchPolicy | None = None) -> None:
+ self.policy = policy or MatchPolicy()
+ self.gold: tuple[GoldPassage, ...] = tuple(
+ GoldPassage.from_text(index, passage)
+ for index, passage in enumerate(gold)
+ if isinstance(passage, str) and passage.strip()
+ )
+
+ @property
+ def gold_count(self) -> int:
+ """Number of non-blank gold passages this matcher scored against."""
+ return len(self.gold)
+
+ def match(self, text: str) -> MatchResult:
+ """Return the gold passages ``text`` covers and the best coverage ratio."""
+ normalized = normalize_text(text)
+ if not normalized or not self.gold:
+ return MatchResult()
+
+ chunk_tokens = Counter(tokenize(text))
+ chunk_count = sum(chunk_tokens.values())
+
+ indices: list[int] = []
+ best = 0.0
+ for passage in self.gold:
+ ratio = self._ratio(normalized, chunk_tokens, chunk_count, passage)
+ if ratio > 0.0:
+ indices.append(passage.index)
+ best = max(best, ratio)
+ return MatchResult(tuple(indices), round(best, 4))
+
+ def _ratio(
+ self,
+ normalized: str,
+ chunk_tokens: Counter[str],
+ chunk_count: int,
+ passage: GoldPassage,
+ ) -> float:
+ """Coverage ratio for one passage (0.0 when it is not a match)."""
+ if not passage.normalized:
+ return 0.0
+ if passage.normalized in normalized:
+ return 1.0
+ # A truncated citation is shorter than the passage it quotes, so an
+ # exact reverse containment is only meaningful once it carries enough
+ # tokens to be more than an incidental substring.
+ if chunk_count >= max(1, int(self.policy.min_tokens)) and normalized in passage.normalized:
+ return 1.0
+
+ ratio, shared = _overlap(chunk_tokens, chunk_count, passage)
+ if ratio < float(self.policy.min_ratio) or shared < int(self.policy.min_tokens):
+ return 0.0
+ return ratio
+
+
+__all__ = [
+ "DEFAULT_MIN_RATIO",
+ "DEFAULT_MIN_TOKENS",
+ "GoldPassage",
+ "MatchPolicy",
+ "MatchResult",
+ "RelevanceMatcher",
+ "SOURCE_TEXT_KEYS",
+ "normalize_text",
+ "source_text",
+ "tokenize",
+]
diff --git a/deeptutor/services/rag/eval/metrics.py b/deeptutor/services/rag/eval/metrics.py
new file mode 100644
index 000000000..69da0f4aa
--- /dev/null
+++ b/deeptutor/services/rag/eval/metrics.py
@@ -0,0 +1,207 @@
+"""Rank-aware retrieval metrics for knowledge-base evaluation.
+
+Every metric is computed at a rank cutoff ``k`` from the *hits* a case produced:
+``hits[i]`` is the set of gold-passage indices the chunk at rank ``i + 1`` covers
+(see :mod:`deeptutor.services.rag.eval.matching`). Working from a set per rank —
+rather than a single relevant/irrelevant flag — is what lets Recall count each
+gold passage once even when two chunks quote it, and lets MAP credit the rank at
+which each *new* passage first appears.
+
+No metric here calls a model: they are pure functions over rankings, so a score
+is reproducible from a stored report and cannot drift with a judge's sampling.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Collection, Sequence
+from dataclasses import dataclass
+import math
+
+#: A per-rank record of which gold passages that rank covered.
+Hits = Sequence[Collection[int]]
+
+#: QueryMetrics attributes in the order of :func:`metric_keys`.
+_METRIC_ATTRS = ("recall", "precision", "ndcg", "mrr", "hit", "average_precision")
+
+
+def metric_keys(k: int) -> tuple[str, ...]:
+ """Report keys for a cutoff ``k``, in display order."""
+ _require_k(k)
+ return (f"recall@{k}", f"precision@{k}", f"ndcg@{k}", "mrr", f"hit@{k}", f"map@{k}")
+
+
+def _require_k(k: int) -> int:
+ """Validate a rank cutoff, returning it as an int."""
+ value = int(k)
+ if value < 1:
+ raise ValueError("k must be >= 1")
+ return value
+
+
+def _window(hits: Hits, k: int | None) -> Sequence[Collection[int]]:
+ """Truncate a ranking to the cutoff (``None`` keeps every retrieved rank)."""
+ if k is None:
+ return hits
+ return hits[: _require_k(k)]
+
+
+def hit_flags(hits: Hits, k: int | None = None) -> list[bool]:
+ """Whether each rank (up to ``k``) covered at least one gold passage."""
+ return [bool(matched) for matched in _window(hits, k)]
+
+
+def recall_at_k(hits: Hits, gold_count: int, k: int) -> float:
+ """Share of the case's gold passages retrieved within the top ``k``."""
+ cutoff = _require_k(k)
+ if gold_count <= 0:
+ return 0.0
+ found: set[int] = set()
+ for matched in hits[:cutoff]:
+ found.update(matched)
+ return min(1.0, len(found) / gold_count)
+
+
+def precision_at_k(hits: Hits, k: int) -> float:
+ """Share of the retrieved chunks within ``k`` that covered a gold passage.
+
+ The denominator is the number of chunks actually retrieved, so a short
+ knowledge base is not punished for returning fewer than ``k`` chunks.
+ """
+ cutoff = _require_k(k)
+ window = hits[:cutoff]
+ if not window:
+ return 0.0
+ return sum(1 for matched in window if matched) / len(window)
+
+
+def hit_at_k(hits: Hits, k: int) -> float:
+ """``1.0`` when any gold passage reached the top ``k`` — the citation hit rate."""
+ return 1.0 if any(_window(hits, k)) else 0.0
+
+
+def reciprocal_rank(hits: Hits, k: int | None = None) -> float:
+ """Reciprocal rank of the first chunk that covered a gold passage (MRR@k)."""
+ for rank, matched in enumerate(_window(hits, k), start=1):
+ if matched:
+ return 1.0 / rank
+ return 0.0
+
+
+def average_precision(hits: Hits, gold_count: int, k: int | None = None) -> float:
+ """Mean of the precision measured each time a *new* gold passage appears."""
+ if gold_count <= 0:
+ return 0.0
+ seen: set[int] = set()
+ total = 0.0
+ for rank, matched in enumerate(_window(hits, k), start=1):
+ fresh = {index for index in matched if index not in seen}
+ if not fresh:
+ continue
+ seen.update(fresh)
+ total += len(seen) / rank
+ return min(1.0, total / gold_count)
+
+
+def _dcg(gains: Sequence[float]) -> float:
+ """Discounted cumulative gain with the standard ``log2(rank + 1)`` discount."""
+ return sum(gain / math.log2(rank + 1) for rank, gain in enumerate(gains, start=1))
+
+
+def ndcg_at_k(hits: Hits, gold_count: int, k: int) -> float:
+ """Binary nDCG@k — whether the gold passages are ranked near the top.
+
+ The ideal ranking holds the case's gold passages (capped at ``k``) ahead of
+ everything else, so a case whose gold is all retrieved but buried below
+ irrelevant chunks scores below 1.
+ """
+ cutoff = _require_k(k)
+ if gold_count <= 0:
+ return 0.0
+ gains = [1.0 if matched else 0.0 for matched in hits[:cutoff]]
+ ideal_dcg = _dcg([1.0] * min(gold_count, cutoff))
+ if ideal_dcg == 0:
+ return 0.0
+ return min(1.0, _dcg(gains) / ideal_dcg)
+
+
+@dataclass(frozen=True)
+class QueryMetrics:
+ """One case's retrieval scores, plus the counts a report should show."""
+
+ query_id: str
+ k: int
+ retrieved: int
+ relevant_retrieved: int
+ gold_count: int
+ recall: float
+ precision: float
+ ndcg: float
+ mrr: float
+ hit: float
+ average_precision: float
+
+ def values(self) -> dict[str, float]:
+ """Metric values keyed the way reports display them."""
+ keys = metric_keys(self.k)
+ return dict(zip(keys, (getattr(self, name) for name in _METRIC_ATTRS)))
+
+ def to_dict(self) -> dict[str, float | int | str]:
+ """Counts plus rounded metrics, ready for JSON output."""
+ return {
+ "query_id": self.query_id,
+ "k": self.k,
+ "retrieved": self.retrieved,
+ "relevant_retrieved": self.relevant_retrieved,
+ "gold_count": self.gold_count,
+ **{key: round(value, 4) for key, value in self.values().items()},
+ }
+
+
+def evaluate_hits(query_id: str, hits: Hits, gold_count: int, k: int) -> QueryMetrics:
+ """Score one case's ranking at cutoff ``k``."""
+ cutoff = _require_k(k)
+ window = hits[:cutoff]
+ return QueryMetrics(
+ query_id=query_id,
+ k=cutoff,
+ retrieved=len(hits),
+ relevant_retrieved=sum(1 for matched in window if matched),
+ gold_count=max(0, int(gold_count)),
+ recall=recall_at_k(hits, gold_count, cutoff),
+ precision=precision_at_k(hits, cutoff),
+ ndcg=ndcg_at_k(hits, gold_count, cutoff),
+ mrr=reciprocal_rank(hits, cutoff),
+ hit=hit_at_k(hits, cutoff),
+ average_precision=average_precision(hits, gold_count, cutoff),
+ )
+
+
+def aggregate_query_metrics(items: Sequence[QueryMetrics], k: int) -> dict[str, float | int]:
+ """Macro-average the metrics of ``items`` at cutoff ``k``.
+
+ An empty list yields zeros rather than an error, so a report always has a
+ metrics block to print.
+ """
+ keys = metric_keys(k)
+ averages = {key: 0.0 for key in keys}
+ if items:
+ for key, name in zip(keys, _METRIC_ATTRS):
+ total = sum(getattr(item, name) for item in items)
+ averages[key] = round(total / len(items), 4)
+ return {**averages, "queries": len(items)}
+
+
+__all__ = [
+ "Hits",
+ "QueryMetrics",
+ "aggregate_query_metrics",
+ "average_precision",
+ "evaluate_hits",
+ "hit_at_k",
+ "hit_flags",
+ "metric_keys",
+ "ndcg_at_k",
+ "precision_at_k",
+ "recall_at_k",
+ "reciprocal_rank",
+]
diff --git a/deeptutor/services/rag/eval/report.py b/deeptutor/services/rag/eval/report.py
new file mode 100644
index 000000000..02e04b7b8
--- /dev/null
+++ b/deeptutor/services/rag/eval/report.py
@@ -0,0 +1,111 @@
+"""Report shape for a retrieval-evaluation run.
+
+The report is plain data plus two renderings: :meth:`EvalReport.to_json` for a
+stored artifact (diffable between engine or settings changes) and
+:meth:`EvalReport.summary_lines` for a one-block CLI summary. Rich table building
+stays in the CLI, so this module carries no rendering dependency.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import json
+from typing import Any
+
+from .matching import MatchPolicy
+from .metrics import aggregate_query_metrics, metric_keys
+from .runner import SOURCE_MODE_CONTEXT, QueryEvaluation
+
+#: Decimals used when a metric is printed.
+METRIC_DECIMALS = 3
+
+
+def format_metric(value: float | int) -> str:
+ """Format one metric for a table cell or a summary line."""
+ return f"{float(value):.{METRIC_DECIMALS}f}"
+
+
+@dataclass(frozen=True)
+class EvalReport:
+ """Everything one evaluation run produced."""
+
+ dataset_name: str
+ kb_name: str
+ provider: str
+ k: int
+ mode: str | None
+ policy: MatchPolicy
+ evaluations: tuple[QueryEvaluation, ...]
+ duration_seconds: float = 0.0
+
+ @property
+ def scored(self) -> tuple[QueryEvaluation, ...]:
+ """Cases that produced a ranking and were scored."""
+ return tuple(item for item in self.evaluations if not item.failed)
+
+ @property
+ def failed(self) -> tuple[QueryEvaluation, ...]:
+ """Cases whose search failed, excluded from the aggregate."""
+ return tuple(item for item in self.evaluations if item.failed)
+
+ def metric_columns(self) -> tuple[str, ...]:
+ """Column headers for the per-case metrics table."""
+ return metric_keys(self.k)
+
+ def format_metrics(self, evaluation: QueryEvaluation) -> dict[str, str]:
+ """Formatted metric cells for one case, keyed by :meth:`metric_columns`."""
+ values = evaluation.metrics.values()
+ return {key: format_metric(values[key]) for key in self.metric_columns()}
+
+ def aggregate(self) -> dict[str, float | int]:
+ """Macro-averaged metrics over the scored cases, plus the case counts."""
+ summary = aggregate_query_metrics([item.metrics for item in self.scored], self.k)
+ summary["queries_scored"] = len(self.scored)
+ summary["queries_failed"] = len(self.failed)
+ return summary
+
+ def to_dict(self) -> dict[str, Any]:
+ """Full serializable report, including per-case detail."""
+ return {
+ "dataset": self.dataset_name,
+ "kb": self.kb_name,
+ "provider": self.provider,
+ "k": self.k,
+ "mode": self.mode or "",
+ "duration_seconds": round(self.duration_seconds, 2),
+ "match_policy": {
+ "min_ratio": self.policy.min_ratio,
+ "min_tokens": self.policy.min_tokens,
+ },
+ "metrics": self.aggregate(),
+ "queries": [
+ {
+ "query": item.query,
+ "error": item.error,
+ "context_only": item.source_mode == SOURCE_MODE_CONTEXT,
+ **item.to_dict(),
+ }
+ for item in self.evaluations
+ ],
+ }
+
+ def to_json(self, *, indent: int = 2) -> str:
+ """Serialize the report for ``--save`` or ``--format json``."""
+ return json.dumps(self.to_dict(), ensure_ascii=False, indent=indent)
+
+ def summary_lines(self) -> list[str]:
+ """The header lines a CLI prints under the per-case table."""
+ summary = self.aggregate()
+ lines = [
+ f"dataset: {self.dataset_name} kb: {self.kb_name} provider: {self.provider or 'unknown'}",
+ f"k: {self.k} mode: {self.mode or 'engine default'}",
+ f"cases: {len(self.evaluations)} scored: {len(self.scored)} failed: {len(self.failed)}",
+ ]
+ metrics = " ".join(f"{key}={format_metric(summary[key])}" for key in self.metric_columns())
+ lines.append(f"metrics (mean over scored cases): {metrics}")
+ if self.failed:
+ lines.append("failed cases carry no score and are excluded from the means")
+ return lines
+
+
+__all__ = ["EvalReport", "format_metric", "METRIC_DECIMALS"]
diff --git a/deeptutor/services/rag/eval/runner.py b/deeptutor/services/rag/eval/runner.py
new file mode 100644
index 000000000..2cde2da54
--- /dev/null
+++ b/deeptutor/services/rag/eval/runner.py
@@ -0,0 +1,290 @@
+"""Run an evaluation set against a knowledge base and score its retrieval.
+
+The runner drives the same search path a chat turn uses (``rag_search`` →
+:class:`~deeptutor.services.rag.service.RAGService` → the KB's bound pipeline),
+so a score reflects production retrieval rather than a re-implementation of it.
+One case is one search; the ranked citations it returns are matched against the
+case's gold passages and scored at the configured cutoff.
+
+A case that cannot be scored — the engine raised, or it answered with an error
+envelope (a missing index, a credentials failure) — is recorded as failed with
+its message. Failed cases are excluded from the aggregate so an unreachable
+engine reports "0 of 8 scored" instead of a misleadingly low quality score.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Awaitable, Callable, Mapping
+from dataclasses import dataclass
+import logging
+import time
+from typing import TYPE_CHECKING, Any
+
+from .dataset import EvalCase, EvalDataset
+from .matching import MatchPolicy, RelevanceMatcher, source_text
+from .metrics import QueryMetrics, evaluate_hits
+
+if TYPE_CHECKING: # ``report`` imports this module's result type.
+ from .report import EvalReport
+
+logger = logging.getLogger(__name__)
+
+#: Search callable shaped like :func:`deeptutor.tools.rag_tool.rag_search`.
+SearchFn = Callable[..., Awaitable[dict[str, Any]]]
+#: Progress hook: ``(index, total, evaluation)`` after each case.
+ProgressFn = Callable[[int, int, "QueryEvaluation"], None]
+
+#: The engine returned ranked citations.
+SOURCE_MODE_CITATIONS = "sources"
+#: The engine returned context but no citations; the context is scored as one span.
+SOURCE_MODE_CONTEXT = "content"
+
+
+@dataclass(frozen=True)
+class QueryEvaluation:
+ """One case's retrieval result, its ranked hits, and its metrics."""
+
+ case_id: str
+ query: str
+ k: int
+ gold_count: int
+ hits: tuple[frozenset[int], ...]
+ metrics: QueryMetrics
+ source_mode: str = SOURCE_MODE_CITATIONS
+ provider: str = ""
+ error: str = ""
+ sources: tuple[dict[str, Any], ...] = ()
+
+ @property
+ def failed(self) -> bool:
+ """True when the case could not be scored (error or failed engine)."""
+ return bool(self.error)
+
+ def to_dict(self) -> dict[str, Any]:
+ """Serializable form, including which gold passages were covered."""
+ return {
+ "case_id": self.case_id,
+ "query": self.query,
+ "error": self.error,
+ "provider": self.provider,
+ "source_mode": self.source_mode,
+ "matched_gold": [sorted(indices) for indices in self.hits],
+ **self.metrics.to_dict(),
+ }
+
+
+def _extract_sources(result: Mapping[str, Any]) -> tuple[list[dict[str, Any]], str]:
+ """Pull the ranked citations out of a search result.
+
+ Engines that offload retrieval (LightRAG server, WeKnora, IMA) and the local
+ ones all normalize to ``sources``. An engine that answers with context but no
+ citations is scored on that context as a single span, so the case still says
+ whether the material came back at all.
+ """
+ raw = result.get("sources")
+ if isinstance(raw, list):
+ records = [item for item in raw if isinstance(item, Mapping)]
+ if records:
+ return [dict(record) for record in records], SOURCE_MODE_CITATIONS
+
+ content = result.get("content") or result.get("answer") or ""
+ if isinstance(content, str) and content.strip():
+ return [{"content": content}], SOURCE_MODE_CONTEXT
+ return [], SOURCE_MODE_CITATIONS
+
+
+def _error_message(result: Mapping[str, Any]) -> str:
+ """The engine's own failure text, or an empty string when the search worked."""
+ if result.get("needs_reindex"):
+ message = (
+ result.get("answer") or result.get("content") or "Knowledge base needs re-indexing."
+ )
+ return str(message)
+ if result.get("error_type"):
+ message = result.get("answer") or result.get("content") or result["error_type"]
+ return str(message)
+ return ""
+
+
+async def _default_search(
+ query: str,
+ *,
+ kb_name: str,
+ kb_base_dir: str | None,
+ mode: str | None,
+ top_k: int,
+) -> dict[str, Any]:
+ """Retrieve through the production path (``rag_search`` → the bound pipeline)."""
+ from deeptutor.tools.rag_tool import rag_search
+
+ kwargs: dict[str, Any] = {"top_k": top_k}
+ if mode:
+ kwargs["mode"] = mode
+ return await rag_search(query=query, kb_name=kb_name, kb_base_dir=kb_base_dir, **kwargs)
+
+
+class RetrievalEvaluator:
+ """Score a knowledge base against an evaluation set.
+
+ Args:
+ kb_name: The knowledge base to search.
+ kb_base_dir: KB root to search; ``None`` resolves the caller's workspace.
+ k: Rank cutoff for every metric.
+ mode: Search mode for engines that support one (e.g. ``hybrid``);
+ ``None`` leaves each engine on its own default.
+ query_limit: Evaluate only the first N cases.
+ policy: Gold-matching thresholds; defaults to :class:`MatchPolicy`.
+ search_fn: Search callable override, for tests and embedders.
+ provider: Provider label for the report; a search result's own
+ ``provider`` still wins per case.
+
+ Raises:
+ ValueError: ``kb_name`` is empty or ``k`` is below 1.
+ """
+
+ def __init__(
+ self,
+ *,
+ kb_name: str,
+ kb_base_dir: str | None = None,
+ k: int = 5,
+ mode: str | None = None,
+ query_limit: int | None = None,
+ policy: MatchPolicy | None = None,
+ search_fn: SearchFn | None = None,
+ provider: str = "",
+ ) -> None:
+ self.kb_name = (kb_name or "").strip()
+ if not self.kb_name:
+ raise ValueError("kb_name must be a non-empty string")
+ self.k = int(k)
+ if self.k < 1:
+ raise ValueError("k must be >= 1")
+ self.kb_base_dir = kb_base_dir
+ self.mode = mode or None
+ self.query_limit = query_limit if query_limit and query_limit > 0 else None
+ self.policy = policy or MatchPolicy()
+ self.search_fn: SearchFn = search_fn or _default_search
+ self.provider = provider
+
+ async def evaluate(
+ self,
+ dataset: EvalDataset,
+ *,
+ progress: ProgressFn | None = None,
+ ) -> EvalReport:
+ """Score every case and return the report defined in ``...eval.report``."""
+ from .report import EvalReport
+
+ cases = dataset.limited(self.query_limit).cases
+ started = time.monotonic()
+ evaluations: list[QueryEvaluation] = []
+ for index, case in enumerate(cases, start=1):
+ evaluation = await self.evaluate_case(case)
+ evaluations.append(evaluation)
+ if evaluation.failed:
+ logger.warning(
+ "Evaluation case %s could not be scored: %s", case.id, evaluation.error
+ )
+ if progress is not None:
+ progress(index, len(cases), evaluation)
+
+ return EvalReport(
+ dataset_name=dataset.name,
+ kb_name=self.kb_name,
+ provider=self.provider,
+ k=self.k,
+ mode=self.mode,
+ policy=self.policy,
+ evaluations=tuple(evaluations),
+ duration_seconds=time.monotonic() - started,
+ )
+
+ async def evaluate_case(self, case: EvalCase) -> QueryEvaluation:
+ """Search one case and score the citations it came back with."""
+ matcher = RelevanceMatcher(case.gold, self.policy)
+ try:
+ result = await self.search_fn(
+ query=case.query,
+ kb_name=self.kb_name,
+ kb_base_dir=self.kb_base_dir,
+ mode=self.mode,
+ top_k=self.k,
+ )
+ except Exception as exc:
+ return self._failure(case, matcher, f"{type(exc).__name__}: {exc}")
+
+ if not isinstance(result, Mapping):
+ kind = type(result).__name__
+ return self._failure(case, matcher, f"unexpected search result {kind}")
+
+ error = _error_message(result)
+ if error:
+ return self._failure(case, matcher, error)
+
+ sources, source_mode = _extract_sources(result)
+ hits = tuple(
+ frozenset(matcher.match(source_text(source)).indices) for source in sources[: self.k]
+ )
+ return QueryEvaluation(
+ case_id=case.id,
+ query=case.query,
+ k=self.k,
+ gold_count=matcher.gold_count,
+ hits=hits,
+ metrics=evaluate_hits(case.id, hits, matcher.gold_count, self.k),
+ source_mode=source_mode,
+ provider=str(result.get("provider") or self.provider),
+ sources=tuple(sources),
+ )
+
+ def _failure(self, case: EvalCase, matcher: RelevanceMatcher, message: str) -> QueryEvaluation:
+ """Record a case that could not be scored, with zeroed metrics."""
+ return QueryEvaluation(
+ case_id=case.id,
+ query=case.query,
+ k=self.k,
+ gold_count=matcher.gold_count,
+ hits=(),
+ metrics=evaluate_hits(case.id, (), matcher.gold_count, self.k),
+ provider=self.provider,
+ error=message,
+ )
+
+
+async def evaluate_dataset(
+ dataset: EvalDataset,
+ *,
+ kb_name: str,
+ kb_base_dir: str | None = None,
+ k: int = 5,
+ mode: str | None = None,
+ query_limit: int | None = None,
+ policy: MatchPolicy | None = None,
+ search_fn: SearchFn | None = None,
+ provider: str = "",
+ progress: ProgressFn | None = None,
+) -> EvalReport:
+ """Score one dataset against one knowledge base (see :class:`RetrievalEvaluator`)."""
+ evaluator = RetrievalEvaluator(
+ kb_name=kb_name,
+ kb_base_dir=kb_base_dir,
+ k=k,
+ mode=mode,
+ query_limit=query_limit,
+ policy=policy,
+ search_fn=search_fn,
+ provider=provider,
+ )
+ return await evaluator.evaluate(dataset, progress=progress)
+
+
+__all__ = [
+ "ProgressFn",
+ "QueryEvaluation",
+ "RetrievalEvaluator",
+ "SearchFn",
+ "SOURCE_MODE_CITATIONS",
+ "SOURCE_MODE_CONTEXT",
+ "evaluate_dataset",
+]
diff --git a/deeptutor_cli/kb.py b/deeptutor_cli/kb.py
index bbc4a6c7c..030ffabdf 100644
--- a/deeptutor_cli/kb.py
+++ b/deeptutor_cli/kb.py
@@ -10,7 +10,7 @@ from __future__ import annotations
import asyncio
import json
from pathlib import Path
-from typing import Optional
+from typing import TYPE_CHECKING, Optional
from rich.console import Console
from rich.table import Table
@@ -22,6 +22,9 @@ from deeptutor.services.path_service import get_path_service
from deeptutor.services.rag.factory import DEFAULT_PROVIDER
from deeptutor.services.rag.file_routing import FileTypeRouter
+if TYPE_CHECKING: # pragma: no cover - imported for type annotations only
+ from deeptutor.services.rag.eval import EvalDataset, EvalReport, ProgressFn, QueryEvaluation
+
console = Console()
@@ -58,6 +61,85 @@ def _collect_documents(docs: list[str], docs_dir: Optional[str]) -> list[str]:
return unique
+def _load_eval_dataset(dataset: str) -> EvalDataset:
+ """Load the QA set for ``kb eval``, exiting with a readable message when it is invalid."""
+ from deeptutor.services.rag.eval import EvalDatasetError, load_dataset
+
+ try:
+ return load_dataset(dataset)
+ except EvalDatasetError as exc:
+ console.print(f"[red]{exc}[/]")
+ raise typer.Exit(code=1) from exc
+
+
+def _resolve_eval_provider(mgr: KnowledgeBaseManager, name: str) -> str:
+ """Return the provider bound to ``name``, rejecting engines that return no ranked chunks."""
+ from deeptutor.services.rag.factory import PAGEINDEX_OSS_PROVIDER, PAGEINDEX_PROVIDER
+ from deeptutor.services.rag.provider_binding import resolve_bound_provider
+
+ provider = resolve_bound_provider(str(mgr.base_dir), name)
+ if provider in {PAGEINDEX_PROVIDER, PAGEINDEX_OSS_PROVIDER}:
+ console.print(
+ "[red]PageIndex answers by reasoning over pages inside an agent loop, so it "
+ "returns no ranked chunks to score. Evaluate a KB bound to llamaindex, "
+ "graphrag, lightrag, a LightRAG server, WeKnora or IMA instead.[/]"
+ )
+ raise typer.Exit(code=1)
+ return provider
+
+
+def _build_eval_progress(top_k: int, fmt: str) -> ProgressFn:
+ """Return a per-case progress printer that stays silent while emitting JSON."""
+ from deeptutor.services.rag.eval import format_metric
+
+ recall_key = f"recall@{top_k}"
+
+ def _report(index: int, total: int, item: QueryEvaluation) -> None:
+ if fmt == "json":
+ return
+ if item.failed:
+ console.print(f" {index}/{total} [red]failed[/] {item.case_id}: {item.error}")
+ return
+ value = format_metric(item.metrics.values()[recall_key])
+ console.print(f" {index}/{total} [green]ok[/] {item.case_id} {recall_key}={value}")
+
+ return _report
+
+
+def _save_eval_report(save: str, payload: str) -> None:
+ """Write a serialized evaluation report to ``save``, creating parent directories."""
+ target = Path(save).expanduser()
+ try:
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_text(payload, encoding="utf-8")
+ except OSError as exc:
+ console.print(f"[red]Could not write report: {exc}[/]")
+ raise typer.Exit(code=1) from exc
+ console.print(f"[green]Report written to {target}[/]")
+
+
+def _render_eval_report(report: EvalReport, name: str, provider: str) -> None:
+ """Print the per-case metric table followed by the aggregate summary lines."""
+ columns = report.metric_columns()
+ table = Table(title=f"Retrieval evaluation - {name} ({provider})")
+ table.add_column("Case", style="bold")
+ table.add_column("Status")
+ table.add_column("Query", overflow="fold", max_width=48)
+ for column in columns:
+ table.add_column(column, justify="right")
+ for item in report.evaluations:
+ cells = report.format_metrics(item)
+ table.add_row(
+ item.case_id,
+ "[red]failed[/]" if item.failed else "[green]ok[/]",
+ item.query,
+ *(cells[column] for column in columns),
+ )
+ console.print(table)
+ for line in report.summary_lines():
+ console.print(line)
+
+
def register(app: typer.Typer) -> None:
@app.command("list")
def kb_list(
@@ -314,6 +396,71 @@ def register(app: typer.Typer) -> None:
console.print(f"[bold]Provider:[/] {provider}")
console.print(f"[bold]Answer:[/]\n{answer}")
+ @app.command("eval")
+ def kb_eval(
+ name: str = typer.Argument(..., help="KB name."),
+ dataset: str = typer.Option(..., "--dataset", "-d", help="QA set file (.jsonl or .json)."),
+ top_k: int = typer.Option(5, "--top-k", "-k", help="Rank cutoff for the metrics."),
+ mode: Optional[str] = typer.Option(
+ None, "--mode", help="Search mode for engines that support one (e.g. hybrid)."
+ ),
+ limit: Optional[int] = typer.Option(
+ None, "--limit", help="Evaluate only the first N cases of the set."
+ ),
+ min_ratio: float = typer.Option(
+ 0.5, "--min-ratio", help="Token coverage a retrieved chunk needs to count as a match."
+ ),
+ save: Optional[str] = typer.Option(None, "--save", help="Write the JSON report to a file."),
+ fmt: str = typer.Option("rich", "--format", "-f", help="Output format: rich | json."),
+ ) -> None:
+ """Score a knowledge base's retrieval quality against a QA set."""
+ from deeptutor.services.rag.eval import MatchPolicy, RetrievalEvaluator
+
+ mgr = _get_kb_manager()
+ if name not in mgr.list_knowledge_bases():
+ console.print(f"[red]Knowledge base '{name}' not found.[/]")
+ raise typer.Exit(code=1)
+
+ eval_set = _load_eval_dataset(dataset)
+ provider = _resolve_eval_provider(mgr, name)
+
+ try:
+ evaluator = RetrievalEvaluator(
+ kb_name=name,
+ kb_base_dir=str(mgr.base_dir),
+ k=top_k,
+ mode=mode,
+ query_limit=limit,
+ policy=MatchPolicy(min_ratio=min_ratio),
+ provider=provider,
+ )
+ except ValueError as exc:
+ console.print(f"[red]{exc}[/]")
+ raise typer.Exit(code=1) from exc
+
+ cases = eval_set.limited(limit)
+ if fmt != "json":
+ console.print(
+ f"[bold]Evaluating[/] '{name}' ({provider}) on {len(cases)} case(s), k={top_k} ..."
+ )
+
+ progress = _build_eval_progress(top_k, fmt)
+ try:
+ report = asyncio.run(evaluator.evaluate(eval_set, progress=progress))
+ except Exception as exc:
+ console.print(f"[red]Evaluation failed: {exc}[/]")
+ raise typer.Exit(code=1) from exc
+
+ payload = report.to_json()
+ if save:
+ _save_eval_report(save, payload)
+
+ if fmt == "json":
+ console.print_json(payload)
+ return
+
+ _render_eval_report(report, name, provider)
+
# ── GitHub source commands ──────────────────────────────────────
@app.command("add-github-source")
diff --git a/tests/cli/test_kb_eval_cli.py b/tests/cli/test_kb_eval_cli.py
new file mode 100644
index 000000000..5430090db
--- /dev/null
+++ b/tests/cli/test_kb_eval_cli.py
@@ -0,0 +1,188 @@
+"""Tests for `deeptutor kb eval`."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+from typer.testing import CliRunner
+
+from deeptutor.services.rag.eval import EvalReport, QueryEvaluation, evaluate_hits
+from deeptutor_cli.main import app
+
+runner = CliRunner()
+
+GOLD = "alpha beta gamma delta epsilon zeta"
+
+
+def eval_env(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> SimpleNamespace:
+ """A one-case KB, a one-case QA set, and a stubbed evaluator."""
+ import deeptutor.services.rag.eval as eval_module
+ import deeptutor.services.rag.provider_binding as binding_module
+ from deeptutor_cli import kb as kb_module
+
+ set_path = tmp_path / "set.jsonl"
+ base_dir = tmp_path / "kbs"
+ base_dir.mkdir()
+ set_path.write_text(
+ json.dumps({"query": "what is alpha?", "gold": [GOLD]}) + "\n", encoding="utf-8"
+ )
+
+ manager = SimpleNamespace(base_dir=base_dir, list_knowledge_bases=lambda: ["kb1"])
+ monkeypatch.setattr(kb_module, "_get_kb_manager", lambda: manager)
+ monkeypatch.setattr(binding_module, "resolve_bound_provider", lambda *_args: "llamaindex")
+
+ calls: list[dict[str, Any]] = []
+
+ class _FakeEvaluator:
+ def __init__(self, **kwargs: Any) -> None:
+ calls.append(kwargs)
+ self.kwargs = kwargs
+
+ async def evaluate(self, dataset, *, progress=None):
+ k = self.kwargs["k"]
+ case = dataset.cases[0]
+ item = QueryEvaluation(
+ case_id=case.id,
+ query=case.query,
+ k=k,
+ gold_count=1,
+ hits=({0},),
+ metrics=evaluate_hits(case.id, ({0},), 1, k),
+ provider=self.kwargs.get("provider", ""),
+ )
+ if progress is not None:
+ progress(1, len(dataset.cases), item)
+ return EvalReport(
+ dataset_name=dataset.name,
+ kb_name=self.kwargs["kb_name"],
+ provider=self.kwargs.get("provider", ""),
+ k=k,
+ mode=self.kwargs.get("mode"),
+ policy=self.kwargs["policy"],
+ evaluations=(item,),
+ duration_seconds=0.01,
+ )
+
+ monkeypatch.setattr(eval_module, "RetrievalEvaluator", _FakeEvaluator)
+
+ return SimpleNamespace(set_path=set_path, base_dir=base_dir, calls=calls)
+
+
+def test_eval_reports_metrics_for_a_scored_case(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(app, ["kb", "eval", "kb1", "--dataset", str(eval_env.set_path)])
+
+ assert result.exit_code == 0, result.output
+ assert "recall@5" in result.output
+ assert "kb: kb1" in result.output
+ assert len(eval_env.calls) == 1
+ call = eval_env.calls[0]
+ assert call["kb_name"] == "kb1"
+ assert call["kb_base_dir"] == str(eval_env.base_dir)
+ assert call["k"] == 5
+ assert call["mode"] is None
+ assert call["query_limit"] is None
+
+
+def test_eval_forwards_cutoff_mode_and_limit(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(
+ app,
+ [
+ "kb",
+ "eval",
+ "kb1",
+ "--dataset",
+ str(eval_env.set_path),
+ "--top-k",
+ "3",
+ "--mode",
+ "hybrid",
+ "--limit",
+ "1",
+ ],
+ )
+
+ assert result.exit_code == 0, result.output
+ assert "recall@3" in result.output
+ call = eval_env.calls[0]
+ assert call["k"] == 3
+ assert call["mode"] == "hybrid"
+ assert call["query_limit"] == 1
+
+
+def test_eval_json_output_carries_the_metrics_block(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(
+ app,
+ ["kb", "eval", "kb1", "--dataset", str(eval_env.set_path), "--format", "json"],
+ )
+
+ assert result.exit_code == 0, result.output
+ assert "metrics" in result.output
+ assert "recall@5" in result.output
+
+
+def test_eval_save_writes_a_parseable_report(eval_env: SimpleNamespace, tmp_path: Path) -> None:
+ target = tmp_path / "reports" / "baseline.json"
+
+ result = runner.invoke(
+ app,
+ ["kb", "eval", "kb1", "--dataset", str(eval_env.set_path), "--save", str(target)],
+ )
+
+ assert result.exit_code == 0, result.output
+ payload = json.loads(target.read_text(encoding="utf-8"))
+ assert payload["kb"] == "kb1"
+ assert payload["metrics"]["recall@5"] == 1.0
+
+
+def test_eval_rejects_an_unknown_knowledge_base(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(app, ["kb", "eval", "missing", "--dataset", str(eval_env.set_path)])
+
+ assert result.exit_code == 1
+ assert "not found" in result.output
+
+
+def test_eval_reports_a_missing_dataset(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(
+ app, ["kb", "eval", "kb1", "--dataset", str(eval_env.base_dir / "absent.jsonl")]
+ )
+
+ assert result.exit_code == 1
+ assert "Evaluation set not found" in result.output
+
+
+def test_eval_reports_an_invalid_dataset(eval_env: SimpleNamespace, tmp_path: Path) -> None:
+ broken = tmp_path / "broken.jsonl"
+ broken.write_text('{"query": "q"}\n', encoding="utf-8")
+
+ result = runner.invoke(app, ["kb", "eval", "kb1", "--dataset", str(broken)])
+
+ assert result.exit_code == 1
+ assert "case #1" in result.output
+
+
+def test_eval_rejects_a_reasoning_as_retrieval_engine(
+ eval_env: SimpleNamespace, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import deeptutor.services.rag.provider_binding as binding_module
+
+ monkeypatch.setattr(binding_module, "resolve_bound_provider", lambda *_args: "pageindex")
+
+ result = runner.invoke(app, ["kb", "eval", "kb1", "--dataset", str(eval_env.set_path)])
+
+ assert result.exit_code == 1
+ assert "PageIndex" in result.output
+ assert eval_env.calls == []
+
+
+def test_eval_rejects_an_impossible_match_ratio(eval_env: SimpleNamespace) -> None:
+ result = runner.invoke(
+ app, ["kb", "eval", "kb1", "--dataset", str(eval_env.set_path), "--min-ratio", "1.5"]
+ )
+
+ assert result.exit_code == 1
+ assert "min_ratio must be in (0, 1]" in result.output
diff --git a/tests/services/rag/eval/test_eval_dataset.py b/tests/services/rag/eval/test_eval_dataset.py
new file mode 100644
index 000000000..68680ff44
--- /dev/null
+++ b/tests/services/rag/eval/test_eval_dataset.py
@@ -0,0 +1,223 @@
+"""Tests for the retrieval-evaluation QA-set schema."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from deeptutor.services.rag.eval import EvalCase, EvalDataset, EvalDatasetError, load_dataset
+
+
+def _write_jsonl(path: Path, rows: list[dict]) -> Path:
+ path.write_text("\n".join(json.dumps(row) for row in rows) + "\n", encoding="utf-8")
+ return path
+
+
+def test_load_jsonl_assigns_default_ids_in_file_order(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "physics.jsonl",
+ [
+ {"query": "What is inertia?", "gold": "Inertia resists a change in motion."},
+ {"query": "What is momentum?", "gold": ["Momentum is mass times velocity."]},
+ ],
+ )
+
+ dataset = load_dataset(path)
+
+ assert dataset.name == "physics"
+ assert dataset.source == str(path)
+ assert [case.id for case in dataset.cases] == ["q1", "q2"]
+ assert dataset.cases[0].gold == ["Inertia resists a change in motion."]
+ assert len(dataset) == 2
+ assert [case.query for case in dataset.iter_cases()] == [
+ "What is inertia?",
+ "What is momentum?",
+ ]
+
+
+def test_explicit_id_and_notes_survive_a_round_trip(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "set.jsonl",
+ [
+ {
+ "id": "attention-1",
+ "query": "Why scale attention?",
+ "gold": ["Dot products grow."],
+ "notes": "ch.3",
+ }
+ ],
+ )
+
+ case = load_dataset(path).cases[0]
+
+ assert case.id == "attention-1"
+ assert case.notes == "ch.3"
+
+
+def test_query_and_gold_are_trimmed(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "set.jsonl",
+ [{"query": " spaced query ", "gold": [" padded passage "]}],
+ )
+
+ case = load_dataset(path).cases[0]
+
+ assert case.query == "spaced query"
+ assert case.gold == ["padded passage"]
+
+
+def test_blank_query_is_rejected_with_case_context(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "set.jsonl",
+ [{"query": "ok", "gold": ["passage"]}, {"query": " ", "gold": ["passage"]}],
+ )
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ message = str(excinfo.value)
+ assert "case #2" in message
+ assert "query must be a non-empty string" in message
+
+
+def test_blank_gold_passage_is_rejected(tmp_path: Path) -> None:
+ path = _write_jsonl(tmp_path / "set.jsonl", [{"query": "q", "gold": [" "]}])
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "gold must hold at least one non-empty passage" in str(excinfo.value)
+
+
+def test_unknown_case_field_is_rejected(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "set.jsonl",
+ [{"query": "q", "gold": ["passage"], "ansr": "typo"}],
+ )
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "case #1" in str(excinfo.value)
+
+
+def test_non_object_line_is_rejected(tmp_path: Path) -> None:
+ path = tmp_path / "set.jsonl"
+ path.write_text('["not an object"]\n', encoding="utf-8")
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "must be a JSON object" in str(excinfo.value)
+
+
+def test_invalid_json_reports_the_line_number(tmp_path: Path) -> None:
+ path = tmp_path / "set.jsonl"
+ path.write_text('{"query": "ok", "gold": ["p"]}\n{"query": broken}\n', encoding="utf-8")
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "line 2" in str(excinfo.value)
+
+
+def test_missing_file_is_reported(tmp_path: Path) -> None:
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(tmp_path / "absent.jsonl")
+
+ assert "not found" in str(excinfo.value)
+
+
+def test_unsupported_suffix_is_reported(tmp_path: Path) -> None:
+ path = tmp_path / "set.txt"
+ path.write_text("{}", encoding="utf-8")
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "Unsupported evaluation set format" in str(excinfo.value)
+
+
+def test_empty_set_is_reported(tmp_path: Path) -> None:
+ path = tmp_path / "set.jsonl"
+ path.write_text("\n\n", encoding="utf-8")
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "holds no cases" in str(excinfo.value)
+
+
+def test_duplicate_case_ids_are_rejected(tmp_path: Path) -> None:
+ path = _write_jsonl(
+ tmp_path / "set.jsonl",
+ [
+ {"id": "dup", "query": "a", "gold": ["p1"]},
+ {"id": "dup", "query": "b", "gold": ["p2"]},
+ ],
+ )
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "duplicate case id 'dup'" in str(excinfo.value)
+
+
+def test_json_object_form_carries_its_name(tmp_path: Path) -> None:
+ path = tmp_path / "set.json"
+ path.write_text(
+ json.dumps({"name": "textbook-ch4", "cases": [{"query": "q", "gold": ["passage"]}]}),
+ encoding="utf-8",
+ )
+
+ assert load_dataset(path).name == "textbook-ch4"
+
+
+def test_explicit_name_overrides_the_file(tmp_path: Path) -> None:
+ path = tmp_path / "set.jsonl"
+ _write_jsonl(path, [{"query": "q", "gold": ["passage"]}])
+
+ assert load_dataset(path, name="override").name == "override"
+
+
+def test_json_bare_list_form_is_accepted(tmp_path: Path) -> None:
+ path = tmp_path / "set.json"
+ path.write_text(json.dumps([{"query": "q", "gold": ["passage"]}]), encoding="utf-8")
+
+ dataset = load_dataset(path)
+
+ assert len(dataset) == 1
+ assert dataset.cases[0].id == "q1"
+
+
+def test_json_without_cases_list_is_rejected(tmp_path: Path) -> None:
+ path = tmp_path / "set.json"
+ path.write_text(json.dumps({"name": "x"}), encoding="utf-8")
+
+ with pytest.raises(EvalDatasetError) as excinfo:
+ load_dataset(path)
+
+ assert "must hold a 'cases' list" in str(excinfo.value)
+
+
+def test_limited_truncates_and_none_keeps_everything() -> None:
+ dataset = EvalDataset.from_cases(
+ [
+ {"query": "a", "gold": ["p1"]},
+ {"query": "b", "gold": ["p2"]},
+ {"query": "c", "gold": ["p3"]},
+ ]
+ )
+
+ assert [case.query for case in dataset.limited(2).cases] == ["a", "b"]
+ assert len(dataset.limited(None)) == 3
+ assert len(dataset.limited(9)) == 3
+
+
+def test_from_cases_fills_ids_and_validates() -> None:
+ dataset = EvalDataset.from_cases([EvalCase(query="a", gold=["p1"])], name="inline")
+
+ assert dataset.name == "inline"
+ assert dataset.cases[0].id == "q1"
diff --git a/tests/services/rag/eval/test_eval_matching.py b/tests/services/rag/eval/test_eval_matching.py
new file mode 100644
index 000000000..79e37be7d
--- /dev/null
+++ b/tests/services/rag/eval/test_eval_matching.py
@@ -0,0 +1,145 @@
+"""Tests for gold-passage matching."""
+
+from __future__ import annotations
+
+import pytest
+
+from deeptutor.services.rag.eval import (
+ MatchPolicy,
+ MatchResult,
+ RelevanceMatcher,
+ normalize_text,
+ source_text,
+)
+from deeptutor.services.rag.eval.matching import tokenize
+
+GOLD = "Transformers stack self-attention layers to model long-range dependencies in sequences."
+GOLD_TOKENS = "alpha beta gamma delta epsilon zeta eta theta"
+
+
+def test_a_chunk_holding_the_gold_verbatim_matches() -> None:
+ chunk = f"Introduction. {GOLD} The section then moves on."
+ result = RelevanceMatcher([GOLD]).match(chunk)
+
+ assert result
+ assert result.indices == (0,)
+ assert result.ratio == 1.0
+
+
+def test_case_and_whitespace_differences_do_not_break_containment() -> None:
+ chunk = (
+ "TRANSFORMERS STACK SELF-ATTENTION LAYERS TO MODEL LONG-RANGE DEPENDENCIES IN SEQUENCES."
+ )
+ assert RelevanceMatcher([GOLD]).match(chunk).indices == (0,)
+
+
+def test_a_truncated_citation_inside_a_longer_gold_still_matches() -> None:
+ """LlamaIndex and GraphRAG cap ``content`` at 200 characters."""
+ truncated = GOLD[:60]
+ assert len(tokenize(truncated)) >= 4
+
+ result = RelevanceMatcher([GOLD]).match(truncated)
+
+ assert result.indices == (0,)
+ assert result.ratio == 1.0
+
+
+def test_an_unrelated_chunk_does_not_match() -> None:
+ result = RelevanceMatcher([GOLD]).match("Boiling pasta needs salted water and fresh tomatoes.")
+
+ assert not result
+ assert result.indices == ()
+ assert result.ratio == 0.0
+
+
+def test_partial_token_overlap_above_the_ratio_matches() -> None:
+ chunk = "alpha beta gamma delta zeta eta theta iota"
+ result = RelevanceMatcher([GOLD_TOKENS]).match(chunk)
+
+ assert result.indices == (0,)
+ assert result.ratio == pytest.approx(0.875)
+
+
+def test_a_short_overlap_below_the_token_floor_is_rejected() -> None:
+ """``alpha beta gamma iota`` covers 3 of 4 own tokens but only 3 shared ones."""
+ matcher = RelevanceMatcher([GOLD_TOKENS])
+
+ assert not matcher.match("alpha beta gamma iota")
+
+
+def test_the_token_floor_can_be_lowered_for_a_smaller_corpus() -> None:
+ matcher = RelevanceMatcher([GOLD_TOKENS], MatchPolicy(min_ratio=0.5, min_tokens=3))
+
+ assert matcher.match("gamma delta epsilon").indices == (0,)
+
+
+def test_the_ratio_floor_can_reject_a_loose_overlap() -> None:
+ """Half of a four-token chunk overlapping is below a stricter ratio."""
+ strict = RelevanceMatcher([GOLD_TOKENS], MatchPolicy(min_ratio=0.9, min_tokens=2))
+
+ assert not strict.match("alpha beta iota kappa")
+
+
+def test_each_gold_passage_is_matched_independently() -> None:
+ matcher = RelevanceMatcher([GOLD, GOLD_TOKENS])
+
+ result = matcher.match("alpha beta gamma delta zeta eta theta iota")
+
+ assert result.indices == (1,)
+ assert matcher.gold_count == 2
+
+
+def test_one_chunk_can_cover_two_gold_passages() -> None:
+ matcher = RelevanceMatcher(["alpha beta gamma", "delta epsilon zeta"])
+
+ assert matcher.match("alpha beta gamma delta epsilon zeta").indices == (0, 1)
+
+
+def test_cjk_text_is_compared_character_by_character() -> None:
+ gold = "注意力机制让模型对每个词加权"
+ matcher = RelevanceMatcher([gold])
+
+ assert len(tokenize("注意力机制")) == 5
+ assert matcher.match(f"{gold},这是核心思想。").indices == (0,)
+
+
+def test_blank_input_never_matches() -> None:
+ matcher = RelevanceMatcher([GOLD])
+
+ assert not matcher.match("")
+ assert not matcher.match(" ")
+
+
+def test_a_matcher_without_usable_gold_never_matches() -> None:
+ matcher = RelevanceMatcher(["", " "])
+
+ assert matcher.gold_count == 0
+ assert not matcher.match(GOLD)
+
+
+def test_normalize_collapses_whitespace_and_case() -> None:
+ assert normalize_text(" Mixed\nCASE text ") == "mixed case text"
+
+
+def test_match_policy_rejects_impossible_thresholds() -> None:
+ with pytest.raises(ValueError):
+ MatchPolicy(min_ratio=0.0)
+ with pytest.raises(ValueError):
+ MatchPolicy(min_ratio=1.5)
+ with pytest.raises(ValueError):
+ MatchPolicy(min_tokens=0)
+
+
+def test_match_result_truthiness_follows_the_matched_indices() -> None:
+ assert MatchResult((0,), 1.0)
+ assert not MatchResult()
+
+
+def test_source_text_reads_the_engines_citation_shape() -> None:
+ assert source_text({"content": "excerpt"}) == "excerpt"
+ assert source_text({"content": " ", "text": "fallback"}) == "fallback"
+ assert source_text({"content": "", "snippet": "snippet wins"}) == "snippet wins"
+ assert source_text({"summary": "community report"}) == "community report"
+ assert source_text({}) == ""
+ assert source_text("not a mapping") == ""
+ assert source_text(None) == ""
diff --git a/tests/services/rag/eval/test_eval_metrics.py b/tests/services/rag/eval/test_eval_metrics.py
new file mode 100644
index 000000000..cc577e061
--- /dev/null
+++ b/tests/services/rag/eval/test_eval_metrics.py
@@ -0,0 +1,155 @@
+"""Tests for the retrieval metrics."""
+
+from __future__ import annotations
+
+import pytest
+
+from deeptutor.services.rag.eval import (
+ aggregate_query_metrics,
+ evaluate_hits,
+ hit_flags,
+ metric_keys,
+ ndcg_at_k,
+ precision_at_k,
+ recall_at_k,
+ reciprocal_rank,
+)
+
+
+def test_perfect_ranking_scores_one_across_the_board() -> None:
+ hits = ({0}, {1})
+
+ assert recall_at_k(hits, 2, 2) == 1.0
+ assert precision_at_k(hits, 2) == 1.0
+ assert ndcg_at_k(hits, 2, 2) == 1.0
+ assert reciprocal_rank(hits, 2) == 1.0
+
+ metrics = evaluate_hits("q1", hits, 2, 2)
+
+ assert metrics.hit == 1.0
+ assert metrics.average_precision == 1.0
+ assert metrics.relevant_retrieved == 2
+ assert metrics.retrieved == 2
+
+
+def test_first_gold_at_rank_three_discounts_every_ranked_metric() -> None:
+ hits = ({}, {}, {0})
+
+ assert recall_at_k(hits, 1, 5) == 1.0
+ assert precision_at_k(hits, 5) == pytest.approx(1 / 3)
+ assert reciprocal_rank(hits, 5) == pytest.approx(1 / 3)
+ assert ndcg_at_k(hits, 1, 5) == pytest.approx(0.5)
+ assert evaluate_hits("q1", hits, 1, 5).average_precision == pytest.approx(1 / 3)
+
+
+def test_ranking_without_any_gold_scores_zero() -> None:
+ hits = ({}, {})
+
+ assert recall_at_k(hits, 1, 5) == 0.0
+ assert precision_at_k(hits, 5) == 0.0
+ assert ndcg_at_k(hits, 1, 5) == 0.0
+ assert reciprocal_rank(hits, 5) == 0.0
+ assert evaluate_hits("q1", hits, 1, 5).hit == 0.0
+
+
+def test_metrics_ignore_gold_below_the_cutoff() -> None:
+ hits = ({}, {}, {}, {}, {0})
+ metrics = evaluate_hits("q1", hits, 1, 3)
+
+ assert metrics.retrieved == 5
+ assert metrics.recall == 0.0
+ assert metrics.ndcg == 0.0
+ assert metrics.mrr == 0.0
+ assert metrics.hit == 0.0
+
+
+def test_repeated_coverage_of_one_gold_cannot_exceed_one() -> None:
+ hits = ({0}, {0})
+ metrics = evaluate_hits("q1", hits, 1, 2)
+
+ assert metrics.recall == 1.0
+ assert metrics.ndcg == 1.0
+ assert metrics.average_precision == 1.0
+ assert metrics.precision == 1.0
+
+
+def test_partial_recall_averages_over_the_gold_pool() -> None:
+ hits = ({0}, {})
+ metrics = evaluate_hits("q1", hits, 2, 2)
+
+ assert metrics.recall == 0.5
+ assert metrics.precision == 0.5
+ assert metrics.average_precision == 0.5
+ assert metrics.ndcg == pytest.approx(0.6131, abs=1e-4)
+
+
+def test_precision_uses_the_retrieved_count_not_the_cutoff() -> None:
+ hits = ({0},)
+ metrics = evaluate_hits("q1", hits, 1, 5)
+
+ assert metrics.precision == 1.0
+ assert metrics.recall == 1.0
+ assert metrics.hit == 1.0
+
+
+def test_hit_flags_reports_matched_ranks() -> None:
+ assert hit_flags(({}, {2}, {0, 1})) == [False, True, True]
+
+
+def test_metric_keys_are_cutoff_labelled() -> None:
+ assert metric_keys(5) == (
+ "recall@5",
+ "precision@5",
+ "ndcg@5",
+ "mrr",
+ "hit@5",
+ "map@5",
+ )
+
+
[email protected]("k", [0, -1])
+def test_a_non_positive_cutoff_is_rejected(k: int) -> None:
+ hits = ({0},)
+
+ with pytest.raises(ValueError):
+ recall_at_k(hits, 1, k)
+ with pytest.raises(ValueError):
+ precision_at_k(hits, k)
+ with pytest.raises(ValueError):
+ ndcg_at_k(hits, 1, k)
+ with pytest.raises(ValueError):
+ evaluate_hits("q1", hits, 1, k)
+ with pytest.raises(ValueError):
+ metric_keys(k)
+
+
+def test_query_metrics_values_and_dict_share_report_keys() -> None:
+ metrics = evaluate_hits("q1", ({0}, {1}), 2, 5)
+
+ assert set(metrics.values()) == set(metric_keys(5))
+ payload = metrics.to_dict()
+
+ assert payload["query_id"] == "q1"
+ assert payload["k"] == 5
+ assert payload["gold_count"] == 2
+ assert payload["recall@5"] == 1.0
+
+
+def test_aggregate_macro_averages_scored_cases() -> None:
+ first = evaluate_hits("q1", ({0},), 1, 5)
+ second = evaluate_hits("q2", (), 1, 5)
+
+ summary = aggregate_query_metrics([first, second], 5)
+
+ assert summary["queries"] == 2
+ assert summary["recall@5"] == 0.5
+ assert summary["hit@5"] == 0.5
+ assert summary["mrr"] == 0.5
+
+
+def test_aggregate_of_nothing_is_zeroed_not_an_error() -> None:
+ summary = aggregate_query_metrics([], 5)
+
+ assert summary["queries"] == 0
+ assert summary["recall@5"] == 0.0
+ assert summary["ndcg@5"] == 0.0
diff --git a/tests/services/rag/eval/test_eval_report.py b/tests/services/rag/eval/test_eval_report.py
new file mode 100644
index 000000000..ba918a604
--- /dev/null
+++ b/tests/services/rag/eval/test_eval_report.py
@@ -0,0 +1,151 @@
+"""Tests for the retrieval-evaluation report."""
+
+from __future__ import annotations
+
+import json
+
+from deeptutor.services.rag.eval import (
+ EvalReport,
+ MatchPolicy,
+ QueryEvaluation,
+ evaluate_hits,
+ format_metric,
+)
+
+K = 5
+
+
+def _scored(case_id: str = "q1", query: str = "what is alpha", hits=({0},), gold_count: int = 1):
+ return QueryEvaluation(
+ case_id=case_id,
+ query=query,
+ k=K,
+ gold_count=gold_count,
+ hits=hits,
+ metrics=evaluate_hits(case_id, hits, gold_count, K),
+ provider="llamaindex",
+ )
+
+
+def _failed(case_id: str = "q2", error: str = "RuntimeError: connection reset"):
+ return QueryEvaluation(
+ case_id=case_id,
+ query="what is beta",
+ k=K,
+ gold_count=1,
+ hits=(),
+ metrics=evaluate_hits(case_id, (), 1, K),
+ error=error,
+ )
+
+
+def _report(*evaluations: QueryEvaluation) -> EvalReport:
+ return EvalReport(
+ dataset_name="physics-ch1",
+ kb_name="textbook",
+ provider="llamaindex",
+ k=K,
+ mode="hybrid",
+ policy=MatchPolicy(),
+ evaluations=evaluations,
+ duration_seconds=1.234,
+ )
+
+
+def test_aggregate_means_the_scored_cases_only() -> None:
+ summary = _report(_scored(), _failed()).aggregate()
+
+ assert summary["queries"] == 1
+ assert summary["queries_scored"] == 1
+ assert summary["queries_failed"] == 1
+ assert summary["recall@5"] == 1.0
+ assert summary["mrr"] == 1.0
+
+
+def test_a_report_of_only_failures_still_aggregates() -> None:
+ summary = _report(_failed()).aggregate()
+
+ assert summary["queries_scored"] == 0
+ assert summary["queries_failed"] == 1
+ assert summary["recall@5"] == 0.0
+
+
+def test_to_dict_carries_the_run_context_and_per_case_detail() -> None:
+ payload = _report(_scored(query="什么是注意力"), _failed()).to_dict()
+
+ assert payload["dataset"] == "physics-ch1"
+ assert payload["kb"] == "textbook"
+ assert payload["provider"] == "llamaindex"
+ assert payload["k"] == K
+ assert payload["mode"] == "hybrid"
+ assert payload["duration_seconds"] == 1.23
+ assert payload["match_policy"] == {"min_ratio": 0.5, "min_tokens": 4}
+
+ scored, failed = payload["queries"]
+ assert scored["matched_gold"] == [[0]]
+ assert scored["context_only"] is False
+ assert scored["query"] == "什么是注意力"
+ assert failed["error"] == "RuntimeError: connection reset"
+ assert failed["matched_gold"] == []
+
+
+def test_to_json_round_trips_and_keeps_unicode_readable() -> None:
+ report = _report(_scored(query="什么是注意力"))
+
+ text = report.to_json()
+
+ assert "什么是注意力" in text
+ assert json.loads(text) == report.to_dict()
+
+
+def test_metric_columns_and_cells_match_the_cutoff() -> None:
+ report = _report(_scored(hits=({}, {0}), gold_count=2))
+
+ assert report.metric_columns() == ("recall@5", "precision@5", "ndcg@5", "mrr", "hit@5", "map@5")
+ cells = report.format_metrics(report.evaluations[0])
+
+ assert cells["recall@5"] == "0.500"
+ assert cells["mrr"] == "0.500"
+ assert cells["hit@5"] == "1.000"
+
+
+def test_summary_lines_describe_the_run_and_the_metrics() -> None:
+ lines = _report(_scored(), _failed()).summary_lines()
+
+ joined = "\n".join(lines)
+ assert "kb: textbook" in joined
+ assert "provider: llamaindex" in joined
+ assert "k: 5 mode: hybrid" in joined
+ assert "cases: 2 scored: 1 failed: 1" in joined
+ assert "recall@5=1.000" in joined
+ assert "excluded from the means" in joined
+
+
+def test_summary_lines_omit_the_failure_note_when_every_case_scored() -> None:
+ joined = "\n".join(_report(_scored()).summary_lines())
+
+ assert "excluded from the means" not in joined
+ assert "failed: 0" in joined
+
+
+def test_mode_falls_back_to_the_engine_default_in_the_summary() -> None:
+ report = EvalReport(
+ dataset_name="set",
+ kb_name="kb",
+ provider="",
+ k=K,
+ mode=None,
+ policy=MatchPolicy(),
+ evaluations=(),
+ )
+
+ joined = "\n".join(report.summary_lines())
+
+ assert "mode: engine default" in joined
+ assert "provider: unknown" in joined
+
+
+def test_format_metric_uses_three_decimals() -> None:
+ assert format_metric(1) == "1.000"
+ assert format_metric(0.5) == "0.500"
+ assert format_metric(0.12345) == "0.123"
diff --git a/tests/services/rag/eval/test_eval_runner.py b/tests/services/rag/eval/test_eval_runner.py
new file mode 100644
index 000000000..192512ed1
--- /dev/null
+++ b/tests/services/rag/eval/test_eval_runner.py
@@ -0,0 +1,256 @@
+"""Tests for the retrieval-evaluation runner."""
+
+from __future__ import annotations
+
+import asyncio
+import sys
+import types
+from typing import Any
+
+import pytest
+
+from deeptutor.services.rag.eval import (
+ EvalDataset,
+ MatchPolicy,
+ RetrievalEvaluator,
+ evaluate_dataset,
+)
+from deeptutor.services.rag.eval import runner as runner_module
+
+GOLD = "alpha beta gamma delta epsilon zeta"
+UNRELATED = "boiling pasta needs salted water"
+
+
+def _dataset(*queries: str, gold: str = GOLD, **kwargs: Any) -> EvalDataset:
+ return EvalDataset.from_cases(
+ [{"query": query, "gold": [gold]} for query in queries],
+ **kwargs,
+ )
+
+
+def _sources(*texts: str) -> dict[str, Any]:
+ return {"sources": [{"content": text} for text in texts], "provider": "fake"}
+
+
+class _Recorder:
+ """Async search stand-in that records its calls and replays canned results."""
+
+ def __init__(self, *results: Any) -> None:
+ self.results = list(results)
+ self.calls: list[dict[str, Any]] = []
+
+ async def __call__(self, **kwargs: Any) -> Any:
+ self.calls.append(kwargs)
+ result = self.results[min(len(self.calls) - 1, len(self.results) - 1)]
+ if isinstance(result, Exception):
+ raise result
+ return result
+
+
+async def test_matching_citations_score_a_full_recall() -> None:
+ search = _Recorder(_sources(f"{GOLD} and more text", UNRELATED))
+
+ report = await evaluate_dataset(_dataset("what is alpha"), kb_name="kb", k=5, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert evaluation.hits[0] == frozenset({0})
+ assert evaluation.metrics.recall == 1.0
+ assert evaluation.metrics.mrr == 1.0
+ assert evaluation.provider == "fake"
+ assert evaluation.source_mode == "sources"
+ assert not evaluation.failed
+
+
+async def test_search_receives_the_kb_mode_and_cutoff() -> None:
+ search = _Recorder(_sources(GOLD))
+
+ await evaluate_dataset(
+ _dataset("q"),
+ kb_name="physics",
+ kb_base_dir="/tmp/kbs",
+ k=3,
+ mode="hybrid",
+ search_fn=search,
+ )
+
+ assert search.calls == [
+ {
+ "query": "q",
+ "kb_name": "physics",
+ "kb_base_dir": "/tmp/kbs",
+ "mode": "hybrid",
+ "top_k": 3,
+ }
+ ]
+
+
+def test_default_search_only_forwards_a_mode_when_one_is_set(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+
+ captured: list[dict[str, Any]] = []
+
+ async def fake_search(**kwargs: Any) -> dict[str, Any]:
+ captured.append(kwargs)
+ return {}
+
+ module = types.ModuleType("deeptutor.tools.rag_tool")
+ module.rag_search = fake_search # type: ignore[attr-defined]
+ monkeypatch.setitem(sys.modules, "deeptutor.tools.rag_tool", module)
+
+ asyncio.run(
+ runner_module._default_search("q", kb_name="kb", kb_base_dir=None, mode=None, top_k=2)
+ )
+ asyncio.run(
+ runner_module._default_search("q", kb_name="kb", kb_base_dir=None, mode="hybrid", top_k=2)
+ )
+
+ assert captured[0] == {"query": "q", "kb_name": "kb", "kb_base_dir": None, "top_k": 2}
+ assert captured[1]["mode"] == "hybrid"
+
+
+async def test_missing_citations_score_zero_but_stay_scored() -> None:
+ search = _Recorder(_sources(UNRELATED))
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=5, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert not evaluation.failed
+ assert evaluation.metrics.recall == 0.0
+ assert report.aggregate()["queries_scored"] == 1
+
+
+async def test_an_engine_error_envelope_is_recorded_and_excluded_from_the_means() -> None:
+ search = _Recorder(
+ {"error_type": "embedding_error", "answer": "Embedding API key not set"},
+ _sources(GOLD),
+ )
+
+ report = await evaluate_dataset(_dataset("q1", "q2"), kb_name="kb", k=5, search_fn=search)
+
+ failed = report.evaluations[0]
+ assert failed.failed
+ assert "Embedding API key not set" in failed.error
+ assert failed.metrics.recall == 0.0
+
+ summary = report.aggregate()
+ assert summary["queries_scored"] == 1
+ assert summary["queries_failed"] == 1
+ assert summary["recall@5"] == 1.0
+
+
+async def test_a_missing_index_is_reported_as_a_failed_case() -> None:
+ search = _Recorder({"needs_reindex": True, "answer": "This knowledge base has no index."})
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=5, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert evaluation.failed
+ assert "no index" in evaluation.error
+
+
+async def test_a_raised_search_error_is_captured_with_its_type() -> None:
+ search = _Recorder(RuntimeError("connection reset"))
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=5, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert evaluation.failed
+ assert evaluation.error == "RuntimeError: connection reset"
+
+
+async def test_a_non_mapping_result_is_reported_rather_than_crashing() -> None:
+ search = _Recorder("not a result")
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=5, search_fn=search)
+
+ assert report.evaluations[0].failed
+ assert "unexpected search result str" in report.evaluations[0].error
+
+
+async def test_context_without_citations_is_scored_as_one_span() -> None:
+ search = _Recorder({"content": f"prefix {GOLD} suffix", "provider": "graphrag"})
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=5, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert evaluation.source_mode == "content"
+ assert evaluation.hits[0] == frozenset({0})
+ assert evaluation.metrics.recall == 1.0
+
+
+async def test_hits_are_capped_at_the_cutoff() -> None:
+ search = _Recorder(_sources(UNRELATED, GOLD))
+
+ report = await evaluate_dataset(_dataset("q"), kb_name="kb", k=1, search_fn=search)
+
+ evaluation = report.evaluations[0]
+ assert len(evaluation.hits) == 1
+ assert evaluation.metrics.recall == 0.0
+ assert evaluation.metrics.hit == 0.0
+
+
+async def test_query_limit_shortens_the_run_and_drives_progress() -> None:
+ search = _Recorder(_sources(GOLD))
+ seen: list[tuple[int, int, str]] = []
+
+ def progress(index: int, total: int, evaluation: Any) -> None:
+ seen.append((index, total, evaluation.case_id))
+
+ report = await evaluate_dataset(
+ _dataset("q1", "q2", "q3"),
+ kb_name="kb",
+ k=5,
+ query_limit=2,
+ search_fn=search,
+ progress=progress,
+ )
+
+ assert len(report.evaluations) == 2
+ assert len(search.calls) == 2
+ assert seen == [(1, 2, "q1"), (2, 2, "q2")]
+
+
+async def test_case_ids_and_queries_are_carried_into_the_report() -> None:
+ search = _Recorder(_sources(GOLD))
+
+ report = await evaluate_dataset(
+ _dataset("first question", "second question"), kb_name="kb", k=5, search_fn=search
+ )
+
+ assert [item.case_id for item in report.evaluations] == ["q1", "q2"]
+ assert [item.query for item in report.evaluations] == ["first question", "second question"]
+
+
+def test_evaluator_rejects_an_unusable_cutoff_or_kb_name() -> None:
+ with pytest.raises(ValueError):
+ RetrievalEvaluator(kb_name="kb", k=0)
+ with pytest.raises(ValueError):
+ RetrievalEvaluator(kb_name=" ")
+
+
+async def test_a_custom_match_policy_flows_through_the_run() -> None:
+ search = _Recorder(_sources("alpha beta gamma iota"))
+
+ report = await evaluate_dataset(
+ _dataset("q", gold="alpha beta gamma delta"),
+ kb_name="kb",
+ k=5,
+ policy=MatchPolicy(min_ratio=0.5, min_tokens=3),
+ search_fn=search,
+ )
+
+ assert report.policy.min_tokens == 3
+ assert report.evaluations[0].metrics.recall == 1.0