Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

fix(answer): grade confidence on whether the top hit covers the question (#3716)

Swati Ahuja18893b373422
diff --git a/README.md b/README.md
index 3ff55d7fe..e8a9c1f69 100644
--- a/README.md
+++ b/README.md
@@ -598,7 +598,7 @@ knows how much to trust what it just read.
| Tool | What only this tool answers |
|---|---|
| `get_overview()` | Architecture summary, module map, entry points, git health. The first call on an unfamiliar codebase. |
-| `get_answer(question)` | Hybrid retrieval (full-text plus vector), graph expansion and one cited answer with a calibrated `retrieval_quality`. Search, read and reason in a single round-trip. |
+| `get_answer(question)` | Hybrid retrieval (full-text plus vector), graph expansion and one cited answer with a single calibrated `confidence`. Search, read and reason in a single round-trip. |
| `get_context(targets, include?)` | Triage card for files, modules or symbols: summary, signatures, hotspot flag, governing decisions, symbol ids. `include` opens callers, callees, ownership and metrics. Batch many targets in one call. |
| `get_symbol("file.py::Name")` | One indexed symbol's source with exact line bounds. |
| `search_codebase(query)` | Hybrid search over code and docs, by symbol, path or concept. |
diff --git a/docs/agent/MCP_TOOLS.md b/docs/agent/MCP_TOOLS.md
index 1ff10f915..1da9a132f 100644
--- a/docs/agent/MCP_TOOLS.md
+++ b/docs/agent/MCP_TOOLS.md
@@ -94,7 +94,7 @@ Every tool returns a JSON object with a `_meta` envelope. Most fields appear onl
| `omitted` | Refs to content cut for size. Restore with `repowise expand <ref>` or `get_symbol("repowise#<ref>")`. |
| `response_budget` | The character ceiling that applied and the size delivered. Present when the budget cut something. |
-`get_answer` adds a top-level `confidence` (rates the prose) and `retrieval_quality` (rates the evidence). Diagnostics such as `timing_ms` and `contract_version` ride only on `get_overview`; `REPOWISE_MCP_DEBUG_META=1` restores them everywhere. Full envelope: [MCP_RESPONSE_FIELDS.md](../reference/MCP_RESPONSE_FIELDS.md#the-_meta-envelope).
+`get_answer` adds a top-level `confidence`, its one trust signal, graded from the retrieval and from whether the top hit covers what the question asks; `include=["diagnostics"]` adds `retrieval_quality`, the rating of the retrieval alone. Diagnostics such as `timing_ms` and `contract_version` ride only on `get_overview`; `REPOWISE_MCP_DEBUG_META=1` restores them everywhere. Full envelope: [MCP_RESPONSE_FIELDS.md](../reference/MCP_RESPONSE_FIELDS.md#the-_meta-envelope).
### Reversible truncation: `_meta.omitted`
@@ -147,7 +147,7 @@ Answers a how, where or why question in one call: it runs hybrid retrieval over
| `repo` | string | default repo | Workspace repo alias. `"all"` is not supported |
| `include` | list[string] | none | `["evidence"]` returns the full evidence projection with a larger budget |
-**Key return fields:** `answer`, `confidence` (`high` / `medium` / `low`, rates the prose), `retrieval_quality` (`high` / `partial` / `weak`, rates the evidence), `citations`, `symbol_bodies` (live bodies of symbols the answer names), `retrieval`, `best_guesses` and `fallback_targets` (on low confidence; a low `best_guesses` row carries `functions`, `lines` and `size_bytes` instead of a page excerpt), `candidate_files` (ranked file paths the citations do not already name: up to 3 at `high`, 5 otherwise), `episodes` (dated facts bearing on the question), `degraded` (synthesis could not run), `_meta.scope_hint` (areas the answer did not touch).
+**Key return fields:** `answer`, `confidence` (`high` / `medium` / `low`), `citations`, `symbol_bodies` (live bodies of symbols the answer names), `retrieval`, `best_guesses` and `fallback_targets` (on low confidence; a low `best_guesses` row carries `functions`, `lines` and `size_bytes` instead of a page excerpt), `candidate_files` (ranked file paths the citations do not already name: up to 3 at `high`, 5 otherwise), `episodes` (dated facts bearing on the question), `degraded` (synthesis could not run), `_meta.scope_hint` (areas the answer did not touch).
A `high` answer can be cited directly. On `low`, read the rows the reply names, then `candidate_files`, before searching again. Without an LLM provider the tool still answers from retrieval, marked `degraded`. When retrieval is also `weak`, the reply serves `best_guesses` and `candidate_files` with one guidance line in `answer`, and no `citations`.
diff --git a/docs/reference/MCP_RESPONSE_FIELDS.md b/docs/reference/MCP_RESPONSE_FIELDS.md
index c80bc1787..cf72f1cd8 100644
--- a/docs/reference/MCP_RESPONSE_FIELDS.md
+++ b/docs/reference/MCP_RESPONSE_FIELDS.md
@@ -124,15 +124,15 @@ Every suggested follow-up (`get_health`'s `fix_first.lead.next_call`, the refact
| Field | Meaning |
|-------|---------|
| `answer` | The synthesised answer, with file and symbol citations |
-| `confidence` | `high`, `medium` or `low`: rates the prose. `high` is content-grounded and can be cited directly. A why-answer whose named mechanism is absent from the retrieved source is downgraded to `medium` |
-| `retrieval_quality` | `high`, `partial` or `weak`: rates the evidence under the prose |
+| `confidence` | `high`, `medium` or `low`: the one trust signal. `high` is content-grounded, on a top hit that covers what the question asks, and can be cited directly. A why-answer whose named mechanism is absent from the retrieved source is downgraded to `medium` |
+| `retrieval_quality` | Only under `include=["diagnostics"]`. `high`, `partial` or `weak`: rates the retrieval alone, and is `weak` when the top hit leaves out most of what the question asks |
| `symbol_bodies` | Live bodies of the symbols the answer names. Read these before calling `get_symbol` |
| `retrieval` | Evidence rows (summary, snippet, key symbols). Shrinks as confidence rises |
| `candidate_files` | Ranked file paths retrieval resolved, minus those already in `citations`: up to 3 at `high`, 5 at `medium`, `low` or `degraded`. Navigation, not evidence. Served at every confidence; absent when retrieval resolved no file |
| `candidates` | The top 5 of those files as `{path, lines?, defines?}` rows. Only with `include=["evidence"]` |
| `best_guesses`, `fallback_targets` | On low confidence: where to look, each with a one-line reason. At `low` or `degraded` a `best_guesses` row drops its page `excerpt` and carries `functions` (up to 3 `{name, line}` symbols relevant to the question) and the live file's `lines` and `size_bytes`; `include=["evidence"]` brings the excerpt back |
| `episodes` | A dated fact recorded about this checkout that bears on the question; `still_true` says how current it is |
-| `degraded` | Synthesis could not run (no provider, or the call failed). The answer is assembled from retrieval and mined rationale with no LLM, and `confidence` is graded from the retrieval: `medium` unless `retrieval_quality` is `weak`, never `high` |
+| `degraded` | Synthesis could not run (no provider, or the call failed). The answer is assembled from retrieval and mined rationale with no LLM, and `confidence` is graded from the retrieval: `medium` on a dominant top hit that covers the question, else `low`, never `high` |
How much evidence rides along depends on the grade. `high` keeps one quote, and one symbol body when the answer is not already grounded. `medium` keeps one body, two quotes and the top two evidence rows. `low` keeps two bodies and the top three rows. `include=["evidence"]` skips this trimming.
diff --git a/docs/reference/TELEMETRY.md b/docs/reference/TELEMETRY.md
index 82c4da65f..c0a369228 100644
--- a/docs/reference/TELEMETRY.md
+++ b/docs/reference/TELEMETRY.md
@@ -74,7 +74,7 @@ response, **never the question, query, results, paths, or repo/symbol names**:
| `properties.status` | `ok` / `error` | Tool error rate |
| `properties.duration_ms` | `820` | Tool latency in the wild |
| `properties.confidence` | `high` / `medium` / `low` | Answer confidence distribution |
-| `properties.retrieval_quality` | `strong` / `weak` | Retrieval quality distribution |
+| `properties.retrieval_quality` | `high` / `partial` / `weak` | Retrieval quality distribution; sent only when the call asked for `include=["diagnostics"]` |
| `properties.results_bucket` | `1-3`, `4-10` | Result count for searches (a range) |
| `properties.index_behind` | `true` / `false` | How often served content is stale |
| `properties.embedder_degraded` | `true` / `false` | How often search runs on mock vectors |
diff --git a/packages/cli/src/repowise/cli/commands/ask_cmd.py b/packages/cli/src/repowise/cli/commands/ask_cmd.py
index 027007a2d..e205536d9 100644
--- a/packages/cli/src/repowise/cli/commands/ask_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/ask_cmd.py
@@ -31,7 +31,7 @@ def project(payload: dict, question: str) -> dict:
================== ===========================================
kept answer, confidence, grounding,
- retrieval_quality, citations, quotes,
+ citations, quotes,
fallback_targets, candidate_files,
best_guesses (without their
excerpts), next_action_hint, note,
@@ -52,7 +52,6 @@ def project(payload: dict, question: str) -> dict:
"question": question,
"answer": payload.get("answer", ""),
"confidence": payload.get("confidence", ""),
- "retrieval_quality": payload.get("retrieval_quality", ""),
"citations": list(payload.get("citations") or []),
}
quotes = payload.get("quotes") or []
@@ -181,13 +180,11 @@ def _render(projected: dict) -> None:
console.print("[yellow]No answer synthesised for that question.[/yellow]")
confidence = projected.get("confidence") or "?"
- quality = projected.get("retrieval_quality") or "?"
colour = {"high": "green", "medium": "yellow"}.get(confidence, "red")
grounding = projected.get("grounding")
console.print(
- f"\n[{colour}]confidence: {confidence}[/{colour}] [dim]retrieval: {quality}"
- + (f" grounding: {grounding}" if grounding else "")
- + "[/dim]"
+ f"\n[{colour}]confidence: {confidence}[/{colour}]"
+ + (f" [dim]grounding: {grounding}[/dim]" if grounding else "")
)
if projected.get("note"):
# The value-grounding gate ("the numbers may be synthesised, read X
diff --git a/packages/core/src/repowise/core/generation/editor_files/tool_table.py b/packages/core/src/repowise/core/generation/editor_files/tool_table.py
index e347840eb..907d552ad 100644
--- a/packages/core/src/repowise/core/generation/editor_files/tool_table.py
+++ b/packages/core/src/repowise/core/generation/editor_files/tool_table.py
@@ -65,11 +65,10 @@ TOOL_TABLE_ROWS: dict[str, tuple[str, str]] = {
# question with `confidence: "low"`, since confidence rates prose that
# was never synthesised, and this row told the agent to distrust the
# payload on the strength of it, so it re-searched after every call.
- # `retrieval_quality` is the field that rates what such a payload does
- # carry. Reworded, not lengthened: 186 chars against the 179 it replaced.
+ # `confidence` there is graded from the evidence such a payload does carry.
"First call for any how/where/why question. "
+ _KEYED_ONLY["get_answer"]
- + "`degraded` means judge by `retrieval_quality`. `symbol_bodies` has live bodies.",
+ + "`degraded` means no prose; `confidence` rates the evidence. `symbol_bodies` has live bodies.",
),
"get_context": (
"get_context(targets=[...])",
diff --git a/packages/server/src/repowise/server/mcp_server/_code_rationale.py b/packages/server/src/repowise/server/mcp_server/_code_rationale.py
index 05364f974..91ff11c05 100644
--- a/packages/server/src/repowise/server/mcp_server/_code_rationale.py
+++ b/packages/server/src/repowise/server/mcp_server/_code_rationale.py
@@ -206,6 +206,17 @@ def _keep(
return len(matched) >= 2
+# A mined row may stand in for the answer only on this many distinct matches
+# (question terms or numbers). A marker plus one shared word clears `_keep`
+# ("FIXME ... unused by cobra" for a question naming cobra) but answers nothing.
+_LEAD_MIN_MATCHES = 2
+
+
+def can_lead_answer(row: dict) -> bool:
+ """Whether a mined rationale row matches enough of the question to be the answer."""
+ return len(set(row.get("matched_terms") or ())) >= _LEAD_MIN_MATCHES
+
+
def _read_text(repo_root: Path, file_path: str) -> str | None:
"""Read a repo file's live text, refusing paths outside the root."""
try:
diff --git a/packages/server/src/repowise/server/mcp_server/_meta.py b/packages/server/src/repowise/server/mcp_server/_meta.py
index 95fc20a2c..c7887b259 100644
--- a/packages/server/src/repowise/server/mcp_server/_meta.py
+++ b/packages/server/src/repowise/server/mcp_server/_meta.py
@@ -938,7 +938,7 @@ def answer_hint(
doubt its files on every question is what made a keyless agent call
search_codebase after every get_answer: one question, two calls, measured on
11 of 26 questions whose rank-1 file was right. So say which half is missing
- and let `retrieval_quality` rate the other half.
+ and let `confidence` rate the other half.
"""
if degraded:
if retrieval_quality == "weak":
@@ -953,11 +953,11 @@ def answer_hint(
if has_bodies:
return (
"Synthesis is what is missing here, not retrieval. Answer from "
- "symbol_bodies; retrieval_quality rates what was served."
+ "symbol_bodies."
)
return (
- "Synthesis is what is missing here, not retrieval. retrieval_quality "
- "rates the ranked hits; start from the first one."
+ "Synthesis is what is missing here, not retrieval. Start from the "
+ "first ranked hit."
)
if confidence == "low":
return (
diff --git a/packages/server/src/repowise/server/mcp_server/_retrieval_rank.py b/packages/server/src/repowise/server/mcp_server/_retrieval_rank.py
index a67832af3..186b9647e 100644
--- a/packages/server/src/repowise/server/mcp_server/_retrieval_rank.py
+++ b/packages/server/src/repowise/server/mcp_server/_retrieval_rank.py
@@ -23,6 +23,12 @@ _INFLECTIONS = {
}
+def window_term_weight(df: int, n: int) -> float:
+ """Weight of a term that *df* of *n* retrieved hits carry: 1 when every hit
+ carries it, more the fewer do, so the word that tells hits apart leads."""
+ return 1.0 + math.log((n + 1) / (df + 1))
+
+
def _ranking_terms(text: str, *, max_terms: int = 8) -> set[str]:
"""Content terms with a tiny retrieval-only inflection normalisation."""
return {_INFLECTIONS.get(term, term) for term in content_terms(text, max_terms=max_terms)}
@@ -103,8 +109,7 @@ def rerank_by_context_coverage(
term: sum(1 for tokens in tokens_by_hit if term in tokens) for term in terms
}
weights = {
- term: 1.0 + math.log((len(hits) + 1) / (document_frequency[term] + 1))
- for term in terms
+ term: window_term_weight(document_frequency[term], len(hits)) for term in terms
}
total_weight = sum(weights.values())
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py b/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
index a287295b6..7994a8c0d 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
@@ -138,6 +138,7 @@ from repowise.server.mcp_server.tool_answer.confidence import (
_is_value_question,
_retrieval_quality,
dominance_reason,
+ uncovered_terms,
)
from repowise.server.mcp_server.tool_answer.config import ( # noqa: F401
_GATED_EXCERPT_CHARS,
@@ -388,9 +389,9 @@ async def get_answer(
whose named mechanism is absent from the retrieved source is downgraded
to medium (the rationale may be conflated). Low confidence returns
best_guesses with one-line justifications instead of an empty answer.
- retrieval_quality separately rates the retrieval that fed synthesis; when
- it reads "weak" beside confidence=high, the note says what the confidence
- rests on instead of the ranking, and that is the claim to trust.
+ confidence is the one trust signal: it also asks whether the top hit
+ covers what was asked. retrieval_quality, the rating of the retrieval
+ alone, is served only under ``include=["diagnostics"]``.
When the answer names a function/method/class, ``symbol_bodies`` carries
its full live body — read that instead of a follow-up get_symbol.
``episodes``, when present, is a dated fact recorded about this checkout
@@ -405,7 +406,8 @@ async def get_answer(
question: developer question.
scope: optional path-prefix filter (e.g. "src/pkg/").
repo: usually omitted.
- include: optional ``["evidence"]`` expanded projection.
+ include: optional ``["evidence"]`` expanded projection;
+ ``["diagnostics"]`` adds ``retrieval_quality``.
"""
if repo == "all":
return _unsupported_repo_all("get_answer")
@@ -534,7 +536,7 @@ async def get_answer(
ctx,
repository,
t0,
- _retrieval_quality(hits, agreement_dominant),
+ _retrieval_quality(hits, agreement_dominant, question),
)
if union_payload is not None:
return _with_candidates(
@@ -708,6 +710,8 @@ async def get_answer(
_gather_body_candidates(hits, answer_text), repo_root
)
+ # One coverage measurement for the grade and the retrieval rating.
+ uncovered = uncovered_terms(question, hits)
grade = _grade_answer(
question=question,
question_ids=question_ids,
@@ -717,9 +721,10 @@ async def get_answer(
symbol_bodies=symbol_bodies,
served_named_body=served_named_body,
dominance=dominance,
+ uncovered=uncovered,
)
confidence = grade.confidence
- retrieval_quality = _retrieval_quality(hits, agreement_dominant)
+ retrieval_quality = _retrieval_quality(hits, agreement_dominant, question, uncovered)
payload = await build_synthesized_payload(
question=question,
@@ -830,7 +835,8 @@ async def get_answer(
question: Developer question.
scope: Optional repository-relative path prefix.
repo: Usually omitted; a workspace alias when needed.
- include: Optional ``["evidence"]`` expanded projection.
+ include: Optional ``["evidence"]`` expanded projection;
+ ``["diagnostics"]`` adds ``retrieval_quality``.
"""
return await _projected_get_answer(
question=question,
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/confidence.py b/packages/server/src/repowise/server/mcp_server/tool_answer/confidence.py
index 57c9139c5..ad752e9de 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/confidence.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/confidence.py
@@ -1,9 +1,10 @@
"""Confidence and retrieval_quality grading for get_answer.
-Two ratings that answer different questions. ``confidence`` says how much to
-trust the synthesised text; ``retrieval_quality`` says how good the retrieval
-that fed it was. The agent reads the first to decide whether to re-read the
-source, the second to decide whether to search again.
+``confidence`` is the one trust signal an agent reads: how much to trust the
+answer, from how the retrieval went and whether its top hit covers what was
+asked. ``retrieval_quality`` rates the retrieval alone; it feeds the degraded
+grade and is served only under ``include=["diagnostics"]``, so two ratings of
+one answer never disagree in front of the agent.
:func:`_grade_answer` runs the gate cascade — one starting grade from retrieval
dominance, then a run of gates that can only demote it — and the predicates each
@@ -22,9 +23,11 @@ from repowise.server.mcp_server._answer_context import (
is_why_question as _is_why_question,
)
from repowise.server.mcp_server._meta import _normalize_target_path
+from repowise.server.mcp_server._retrieval_rank import window_term_weight
from repowise.server.mcp_server.tool_answer.config import (
_AGREEMENT_RANK_GAP,
_AGREEMENT_TOP_RANK_MAX,
+ _ANSWER_COVERAGE_FLOOR,
_CLAIM_SUPPORT_GATE_ENV,
_DOMINANCE_ABS_GAP,
_DOMINANCE_ABS_SCORE_FLOOR,
@@ -40,6 +43,12 @@ from repowise.server.mcp_server.tool_answer.config import (
_flag_on,
_opt_in,
)
+from repowise.server.mcp_server.tool_answer.hydration import (
+ _stem,
+ _stem_hit,
+ _text_stems,
+ _text_words,
+)
from repowise.server.mcp_server.tool_answer.symbols import is_symbol_lookup_question
@@ -482,15 +491,67 @@ def is_dominant(hits: list[dict], *, agreement_dominant: bool = False) -> bool:
return dominance_reason(hits, agreement_dominant=agreement_dominant) is not None
-def _retrieval_quality(hits: list[dict], agreement_dominant: bool) -> str:
+# Kept apart from `_query_terms.STOPWORDS` on purpose: these are dropped for
+# coverage only, retrieval still searches them.
+# Words that frame what is asked ("how does X work", "where is X implemented")
+# rather than name it. Code rarely says them, so a hit that answers the question
+# would otherwise read as missing its rarest word.
+_QUESTION_FRAME_STEMS = frozenset(
+ _stem(w)
+ for w in (
+ "work", "works", "working", "implement", "implemented", "implements",
+ "implementation", "handle", "handled", "handles", "handling", "happen",
+ "happens", "use", "used", "uses", "using", "way", "done", "code", "logic",
+ )
+)
+
+
+def uncovered_terms(question: str, hits: list[dict]) -> list[str]:
+ """The question's words the top hit leaves out, when they outweigh the rest.
+
+ Dominance compares hits with each other, so a page about something else
+ still dominates a worse page about something else. This asks whether the
+ hit mentions what was asked, in the text the answer was built from. Each
+ word weighs by how few retrieved hits carry it: one every hit shares (the
+ project's own name) counts little, the one naming the topic counts most.
+ Returns ``[]`` when the hit covers at least :data:`_ANSWER_COVERAGE_FLOOR`
+ of that weight.
+ """
+ words = {
+ t: w for t, w in _text_words(question).items() if t not in _QUESTION_FRAME_STEMS
+ }
+ if not words or not hits:
+ return []
+ served = [_text_stems(_retrieval_corpus([h], include_paths=True)) for h in hits]
+ weight = {
+ t: window_term_weight(sum(_stem_hit(t, s) for s in served), len(served)) for t in words
+ }
+ missing = [t for t in words if not _stem_hit(t, served[0])]
+ left_out = sum(weight[t] for t in missing)
+ if left_out <= (1.0 - _ANSWER_COVERAGE_FLOOR) * sum(weight.values()):
+ return []
+ return [words[t] for t in missing]
+
+
+def _retrieval_quality(
+ hits: list[dict],
+ agreement_dominant: bool,
+ question: str = "",
+ uncovered: list[str] | None = None,
+) -> str:
"""Rate the retrieval, independently of the text it fed.
Shared with the degraded path so "high" means the same to keyless and keyed
- callers, and :func:`_degraded_confidence` grades from it. "weak" means
- exactly "not dominant" per :func:`is_dominant`.
+ callers, and :func:`_degraded_confidence` grades from it. "weak" means not
+ dominant per :func:`is_dominant`, or a top hit that leaves out most of what
+ *question* asks (:func:`uncovered_terms`; pass *uncovered* when already computed).
"""
top_score = _confidence_score(hits[0]) if hits else 0.0
dominant_grade = is_dominant(hits, agreement_dominant=agreement_dominant)
+ if uncovered is None:
+ uncovered = uncovered_terms(question, hits) if dominant_grade else []
+ if dominant_grade and uncovered:
+ return "weak"
if dominant_grade and top_score >= _HIGH_CONFIDENCE_SCORE_FLOOR:
return "high"
return "partial" if dominant_grade else "weak"
@@ -606,6 +667,8 @@ class _Grade(NamedTuple):
#: "symbol_body" may tell the agent not to re-read.
high_reason: str | None
ungrounded_values: list[str]
+ #: Question words the top hit leaves out, when they outweigh what it covers.
+ uncovered: list[str]
frame_unsupported: list[str]
exclusivity_over_truncated: bool
withheld_implicated: list[str]
@@ -623,6 +686,7 @@ def _grade_answer(
symbol_bodies: list[dict],
served_named_body: bool,
dominance: str | None,
+ uncovered: list[str] | None = None,
) -> _Grade:
"""Grade the synthesised answer through the gate cascade, in order.
@@ -632,6 +696,7 @@ def _grade_answer(
``dominance`` is :func:`dominance_reason`'s verdict, computed once by the
caller and used for the starting grade, the ceiling, and the note's reason.
+ ``uncovered`` is :func:`uncovered_terms`, when the caller already has it.
"""
dominant = dominance is not None
_ratio = _top_two_score_ratio(hits)
@@ -752,6 +817,15 @@ def _grade_answer(
if lookup_body_truncated:
confidence = "medium"
+ # Coverage gate, last so a more specific gate writes the note: dominance
+ # says the top hit beat the others, not that it is about the question. A
+ # served body does not exempt it: a question can name a symbol in passing.
+ gap: list[str] = []
+ if confidence == "high":
+ gap = uncovered_terms(question, hits) if uncovered is None else uncovered
+ if gap:
+ confidence = "medium"
+
# Non-dominant ceiling: with no single dominant page the answer must not
# read "high" (cite without verifying) unless it EARNED high. Keyed on
# `earn_high`, not `earned_body`, so the grounding opt-in really restores
@@ -772,6 +846,7 @@ def _grade_answer(
ratio=_ratio,
top_score=_top_score,
ungrounded_values=ungrounded_values,
+ uncovered=gap,
frame_unsupported=frame_unsupported,
exclusivity_over_truncated=exclusivity_over_truncated,
withheld_implicated=withheld_implicated,
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/config.py b/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
index d298e6b89..1a77795fc 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
@@ -270,6 +270,14 @@ _DOMAIN_PENALTY = 0.5
# weak retrieval, so it grades "medium". Useful wiki hits routinely score >1.5.
_HIGH_CONFIDENCE_SCORE_FLOOR = 1.5
+# Share of the question's words, weighted by rarity across the retrieved hits,
+# the answering hit must mention before confidence may rest on it. Dominance
+# only compares hits with each other, so an off-topic page beating a worse one
+# still dominates. Chosen on one repo's questions, where it kept every top hit
+# that answered and dropped most off-topic and wrong-file ones, then checked on
+# repos in other languages.
+_ANSWER_COVERAGE_FLOOR = 0.5
+
# Schema version stamped on every cached payload. Bump whenever the synthesised
# response shape or the meaning of a field (including `confidence` and `note`)
# changes: a cached row at a lower version is a miss and re-synthesises, since
@@ -281,7 +289,8 @@ _HIGH_CONFIDENCE_SCORE_FLOOR = 1.5
# 19: rows carry ``_candidate_file_facts``; low answers serve slim best_guesses rows.
# 20: caller questions carry ``graph_callers`` and synthesise with them.
# 21: impact questions carry ``graph_neighbors``, the users of the lead file.
-_ANSWER_SCHEMA_VERSION = 21
+# 22: ``confidence`` also asks whether the top hit covers the question's words.
+_ANSWER_SCHEMA_VERSION = 22
# Above this a whole-file Read costs more than the answer, so the low-confidence
# hint names a ranged Read or a skeleton instead.
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/degraded.py b/packages/server/src/repowise/server/mcp_server/tool_answer/degraded.py
index d80acb999..1f94e510c 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/degraded.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/degraded.py
@@ -9,6 +9,7 @@ from __future__ import annotations
import time
+from repowise.server.mcp_server._code_rationale import can_lead_answer
from repowise.server.mcp_server._meta import _normalize_target_path
from repowise.server.mcp_server._meta import answer_hint as _answer_hint
from repowise.server.mcp_server._meta import build_meta as _build_meta
@@ -119,7 +120,7 @@ async def _degraded_payload(
signals. ``confidence`` is graded by :func:`_degraded_confidence` from
``retrieval_quality`` and whether retrieval backs the file ``answer`` leads with.
"""
- retrieval_quality = _retrieval_quality(hits, agreement_dominant)
+ retrieval_quality = _retrieval_quality(hits, agreement_dominant, question)
repo_root = _repo_root(ctx)
symbol_bodies, _served_named_body = _build_symbol_bodies(
_gather_body_candidates(hits, "", anchor_names=question_ids or set()),
@@ -158,7 +159,7 @@ async def _degraded_payload(
elif best_guesses:
lead = best_guesses[0]["file"]
else:
- lead = code_rationale[0].get("path") if code_rationale else None
+ lead = next((r.get("path") for r in code_rationale[:1] if can_lead_answer(r)), None)
confidence = _degraded_confidence(
reason,
retrieval_quality,
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/hydration.py b/packages/server/src/repowise/server/mcp_server/tool_answer/hydration.py
index fad1fb133..b83e5055d 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/hydration.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/hydration.py
@@ -50,13 +50,20 @@ def _stem(token: str) -> str:
return token
+def _text_words(text: str) -> dict[str, str]:
+ """Stemmed content tokens of *text*, hump- and separator-split, each mapped
+ to the first word it came from (for quoting a stem back to a reader)."""
+ words: dict[str, str] = {}
+ for tok in re.split(r"[^A-Za-z0-9]+", split_humps(text)):
+ low = tok.lower()
+ if len(tok) >= 3 and low not in _STOPWORDS:
+ words.setdefault(_stem(low), low)
+ return words
+
+
def _text_stems(text: str) -> set[str]:
"""Stemmed content tokens of *text*, hump- and separator-split."""
- return {
- _stem(tok.lower())
- for tok in re.split(r"[^A-Za-z0-9]+", split_humps(text))
- if len(tok) >= 3 and tok.lower() not in _STOPWORDS
- }
+ return set(_text_words(text))
def _stem_hit(term: str, tokens: set[str]) -> bool:
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/payload.py b/packages/server/src/repowise/server/mcp_server/tool_answer/payload.py
index 345b3b2c6..2c1024b31 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/payload.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/payload.py
@@ -347,7 +347,9 @@ def build_value_payload(
The verbatim assignment line (read live by the hydrator) IS the answer.
Not cached: extraction is cheap and must reflect the current source.
Only a constant the question names exactly is ``high``: one whose name the
- question only spells out may not be the one meant.
+ question only spells out may not be the one meant. The coverage gate does
+ not apply: an exact name is the match, and the words around it ("default",
+ "value") are the question's frame, so it would only demote right answers.
"""
top_score_fp = hits[0].get("score", 0.0) if hits else 0.0
answer_text = extraction["answer"]
@@ -702,6 +704,12 @@ async def _graded_payload(
payload["next_action_hint"] = (
f"call get_symbol id='{cut['continuation']}' for the rest of {cut['name']}"
)
+ elif grade.uncovered:
+ payload["note"] = (
+ f"Coverage gate: the top source does not mention {', '.join(grade.uncovered[:4])}, "
+ "so it may answer a different question. Held at medium; confirm against "
+ "the next evidence row or candidate_files."
+ )
elif confidence == "high":
# "No hedging" is not cited as evidence: the prompt forbids hedging, so
# reading its absence back as a signal would be circular.
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/projection.py b/packages/server/src/repowise/server/mcp_server/tool_answer/projection.py
index abeebd1bc..38db94b89 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/projection.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/projection.py
@@ -9,6 +9,7 @@ from functools import wraps
from pathlib import Path
from typing import Any
+from repowise.server.mcp_server._code_rationale import can_lead_answer
from repowise.server.mcp_server.tool_answer.config import (
_CANDIDATE_FILES_HIGH,
_CANDIDATE_FILES_MAX,
@@ -28,6 +29,11 @@ _COLLECTIONS = (
"fallback_targets",
)
+# ``evidence`` restores the expanded projection; ``diagnostics`` adds
+# ``retrieval_quality``, which feeds ``confidence`` and is otherwise not served
+# so the agent reads one trust signal.
+_INCLUDES = frozenset({"evidence", "diagnostics"})
+
def _path(row: Any) -> str | None:
if isinstance(row, str):
@@ -243,9 +249,14 @@ def _same_file(a: str | None, b: str | None) -> bool:
def _lead_rationale(payload: dict[str, Any]) -> dict[str, Any] | None:
"""The rationale row ``answer`` may lead with: one from the top-ranked file.
- A row from any other file would name a file retrieval did not put first.
+ A row from any other file would name a file retrieval did not put first,
+ and one matching a single question word is not an answer (`can_lead_answer`).
"""
- rows = [row for row in payload.get("code_rationale") or [] if isinstance(row, dict)]
+ rows = [
+ row
+ for row in payload.get("code_rationale") or []
+ if isinstance(row, dict) and can_lead_answer(row)
+ ]
if not rows:
return None
guess = _first_guess(payload)
@@ -734,7 +745,9 @@ def _project(
payload, totals, scope=scope, repo=repo, expanded=expanded, ranked=ranked,
bodies_cut=bodies_cut,
)
- unknown = sorted(set(include or []) - {"evidence"})
+ if "diagnostics" not in set(include or []):
+ payload.pop("retrieval_quality", None)
+ unknown = sorted(set(include or []) - _INCLUDES)
if unknown:
payload.setdefault("_meta", {})["ignored_arguments"] = {"include": unknown}
return payload, slimmed, ranked
diff --git a/plugins/claude-code/skills/codebase-exploration/SKILL.md b/plugins/claude-code/skills/codebase-exploration/SKILL.md
index 961b00161..9c8e40bcf 100644
--- a/plugins/claude-code/skills/codebase-exploration/SKILL.md
+++ b/plugins/claude-code/skills/codebase-exploration/SKILL.md
@@ -20,7 +20,7 @@ plain file reads don't, usually in one round-trip instead of many.
| You want… | Call |
|---|---|
| First orientation in an unfamiliar repo | `get_overview()` — architecture summary, key modules, entry points, git health, knowledge map. Skip it once you have the map. |
-| A direct answer to "how/where/why does X work" | `get_answer(question="…")` — synthesised answer with citations + a `retrieval_quality` signal. Collapses the search → read → reason loop. |
+| A direct answer to "how/where/why does X work" | `get_answer(question="…")`: synthesised answer with citations + one `confidence` trust signal. Collapses the search → read → reason loop. |
| Find a symbol, file, or fuzzy concept | `search_codebase(query="…")` — hybrid search. `mode="auto"` routes an identifier to indexed symbol hits (`symbol_id`/line bounds → pipe into `get_symbol`), a path to file pages (→ `get_context`), and prose to semantic wiki search (each hit reports `search_method`: `embedding` vs `bm25`). Force a branch with `mode=symbol\|path\|concept\|hybrid`; narrow symbols with `symbol_kind`. |
| A triage card for specific files/symbols | `get_context(targets=[…])` — title, summary, signatures, hotspot bit, top callers, decision titles, symbol_ids (a row without one is `path::name`). Batch many targets in one call. |
| The actual source of one symbol | `get_symbol("path/to/file.py::Name")` — exact bytes with line bounds. Cheaper than Read + offset math. Use a `symbol_id` from `get_context`. |
@@ -44,7 +44,7 @@ context doesn't cover the specific detail the user asked about.
## Trust signals — verify when
- `_meta.stale_warning` is present (the index has diverged from HEAD), or
-- `retrieval_quality` is `partial`/`weak`, or
+- `confidence` is `medium`/`low`, or
- a result's `search_method` is `bm25`.
Otherwise the response is current — act on it.
diff --git a/plugins/codex/skills/codebase-exploration/SKILL.md b/plugins/codex/skills/codebase-exploration/SKILL.md
index b32139603..e909a4a37 100644
--- a/plugins/codex/skills/codebase-exploration/SKILL.md
+++ b/plugins/codex/skills/codebase-exploration/SKILL.md
@@ -15,7 +15,7 @@ plain file reads don't, usually in one round-trip instead of many.
| You want… | Call |
|---|---|
| First orientation in an unfamiliar repo | `get_overview()` — architecture summary, key modules, entry points, git health, knowledge map. Skip it once you have the map. |
-| A direct answer to "how/where/why does X work" | `get_answer(question="…")` — synthesised answer with citations + a `retrieval_quality` signal. Collapses the search → read → reason loop. |
+| A direct answer to "how/where/why does X work" | `get_answer(question="…")`: synthesised answer with citations + one `confidence` trust signal. Collapses the search → read → reason loop. |
| Find a symbol, file, or fuzzy concept | `search_codebase(query="…")` — hybrid search. `mode="auto"` routes an identifier to indexed symbol hits (`symbol_id`/line bounds → pipe into `get_symbol`), a path to file pages (→ `get_context`), and prose to semantic wiki search (each hit reports `search_method`: `embedding` vs `bm25`). Force a branch with `mode=symbol\|path\|concept\|hybrid`; narrow symbols with `symbol_kind`. |
| A triage card for specific files/symbols | `get_context(targets=[…])` — title, summary, signatures, hotspot bit, top callers, decision titles, symbol_ids (a row without one is `path::name`). Batch many targets in one call. |
| The actual source of one symbol | `get_symbol("path/to/file.py::Name")` — exact bytes with line bounds. Cheaper than Read + offset math. Use a `symbol_id` from `get_context`. |
@@ -39,7 +39,7 @@ context doesn't cover the specific detail the user asked about.
## Trust signals — verify when
- `_meta.stale_warning` is present (the index has diverged from HEAD), or
-- `retrieval_quality` is `partial`/`weak`, or
+- `confidence` is `medium`/`low`, or
- a result's `search_method` is `bm25`.
Otherwise the response is current — act on it.
diff --git a/plugins/shared/skills/codebase-exploration.md b/plugins/shared/skills/codebase-exploration.md
index ac24979f0..ab319d770 100644
--- a/plugins/shared/skills/codebase-exploration.md
+++ b/plugins/shared/skills/codebase-exploration.md
@@ -28,7 +28,7 @@ plain file reads don't, usually in one round-trip instead of many.
| You want… | Call |
|---|---|
| First orientation in an unfamiliar repo | `get_overview()` — architecture summary, key modules, entry points, git health, knowledge map. Skip it once you have the map. |
-| A direct answer to "how/where/why does X work" | `get_answer(question="…")` — synthesised answer with citations + a `retrieval_quality` signal. Collapses the search → read → reason loop. |
+| A direct answer to "how/where/why does X work" | `get_answer(question="…")`: synthesised answer with citations + one `confidence` trust signal. Collapses the search → read → reason loop. |
| Find a symbol, file, or fuzzy concept | `search_codebase(query="…")` — hybrid search. `mode="auto"` routes an identifier to indexed symbol hits (`symbol_id`/line bounds → pipe into `get_symbol`), a path to file pages (→ `get_context`), and prose to semantic wiki search (each hit reports `search_method`: `embedding` vs `bm25`). Force a branch with `mode=symbol\|path\|concept\|hybrid`; narrow symbols with `symbol_kind`. |
| A triage card for specific files/symbols | `get_context(targets=[…])` — title, summary, signatures, hotspot bit, top callers, decision titles, symbol_ids (a row without one is `path::name`). Batch many targets in one call. |
| The actual source of one symbol | `get_symbol("path/to/file.py::Name")` — exact bytes with line bounds. Cheaper than Read + offset math. Use a `symbol_id` from `get_context`. |
@@ -52,7 +52,7 @@ context doesn't cover the specific detail the user asked about.
## Trust signals — verify when
- `_meta.stale_warning` is present (the index has diverged from HEAD), or
-- `retrieval_quality` is `partial`/`weak`, or
+- `confidence` is `medium`/`low`, or
- a result's `search_method` is `bm25`.
Otherwise the response is current — act on it.
diff --git a/tests/fixtures/mcp/tool_response_shapes.json b/tests/fixtures/mcp/tool_response_shapes.json
index 25a21e86e..987aa0310 100644
--- a/tests/fixtures/mcp/tool_response_shapes.json
+++ b/tests/fixtures/mcp/tool_response_shapes.json
@@ -46,8 +46,7 @@
],
"quotes_emitted": null,
"quotes_reduced_reason": null,
- "quotes_total": null,
- "retrieval_quality": null
+ "quotes_total": null
},
"get_architecture": {
"_meta": {
diff --git a/tests/unit/generation/test_editor_file_base.py b/tests/unit/generation/test_editor_file_base.py
index 0c6760ac2..f287d2610 100644
--- a/tests/unit/generation/test_editor_file_base.py
+++ b/tests/unit/generation/test_editor_file_base.py
@@ -133,7 +133,7 @@ def test_keyless_index_gets_keyless_steering(any_gen):
assert "repowise update --full" not in result
assert "[fts]` only has no semantic agreement" not in result
assert 'Cite `confidence: "high"`' not in result
- assert "`degraded` means judge by `retrieval_quality`" in result
+ assert "`degraded` means no prose; `confidence` rates the evidence" in result
def test_keyed_index_keeps_model_guidance(any_gen):
diff --git a/tests/unit/server/mcp/test_answer_agreement.py b/tests/unit/server/mcp/test_answer_agreement.py
index 0d0534958..04fe35325 100644
--- a/tests/unit/server/mcp/test_answer_agreement.py
+++ b/tests/unit/server/mcp/test_answer_agreement.py
@@ -127,7 +127,7 @@ def _patch_agreement_pipeline(monkeypatch, answer_mod, *, top_both: bool):
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Upload module summary."
+ h["summary"] = "Upload module: streams large uploads in chunks."
h["snippet"] = ""
h["page_type"] = "file_page"
if i == 0:
@@ -154,7 +154,7 @@ async def test_agreement_lifts_compressed_retrieval_to_high(setup_mcp, monkeypat
_patch_agreement_pipeline(monkeypatch, answer_mod, top_both=True)
_patch_provider(monkeypatch, answer_mod, _GOOD_ANSWER)
- result = await get_answer("how does upload chunking work")
+ result = await get_answer("how does upload chunking work", include=["diagnostics"])
assert result["confidence"] == "high"
assert result["retrieval_quality"] == "high"
@@ -217,7 +217,7 @@ async def test_grounding_earns_high_only_when_opted_in(setup_mcp, monkeypatch):
_patch_agreement_pipeline(monkeypatch, answer_mod, top_both=False)
_patch_provider(monkeypatch, answer_mod, _GOOD_ANSWER)
- result = await get_answer("how are large uploads handled")
+ result = await get_answer("how are large uploads handled", include=["diagnostics"])
assert result["retrieval_quality"] == "weak"
assert result["confidence"] == "medium", "grounded prose does not out-vote weak retrieval"
@@ -371,7 +371,7 @@ def _patch_keyless_agreement_pipeline(monkeypatch, answer_mod, pipeline_mod, *,
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Upload module summary."
+ h["summary"] = "Upload module: streams large uploads in chunks."
h["snippet"] = ""
h["page_type"] = "file_page"
if i == 0:
@@ -440,4 +440,4 @@ async def test_symbol_agreement_flag_is_independently_reversible(setup_mcp, monk
# ...and the vector pair still lifts with the symbol flag off.
_patch_agreement_pipeline(monkeypatch, answer_mod, top_both=True)
_patch_provider(monkeypatch, answer_mod, _GOOD_ANSWER)
- assert (await get_answer("how does upload chunking work again"))["confidence"] == "high"
+ assert (await get_answer("how does upload chunking work in alpha"))["confidence"] == "high"
diff --git a/tests/unit/server/mcp/test_answer_always_synthesize.py b/tests/unit/server/mcp/test_answer_always_synthesize.py
index af332af41..6399cb18f 100644
--- a/tests/unit/server/mcp/test_answer_always_synthesize.py
+++ b/tests/unit/server/mcp/test_answer_always_synthesize.py
@@ -33,7 +33,7 @@ def _patch_pipeline(monkeypatch, answer_mod, scores=(2.0, 1.9), *, with_symbols=
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Module summary for the page. " * 4
+ h["summary"] = "The alpha module go function. " * 4
h["snippet"] = ""
h["page_type"] = "file_page"
if with_symbols and i == 0:
diff --git a/tests/unit/server/mcp/test_answer_calibration.py b/tests/unit/server/mcp/test_answer_calibration.py
index 2d69e7bf4..4daf2060e 100644
--- a/tests/unit/server/mcp/test_answer_calibration.py
+++ b/tests/unit/server/mcp/test_answer_calibration.py
@@ -309,7 +309,7 @@ def _patch_pipeline(monkeypatch, answer_mod, *, with_symbols: bool, symbol: dict
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Auth service summary."
+ h["summary"] = "Auth service summary. Callers get a bounded, limited list."
h["snippet"] = ""
h["page_type"] = "file_page"
if with_symbols and i == 0:
diff --git a/tests/unit/server/mcp/test_answer_confidence_reasons.py b/tests/unit/server/mcp/test_answer_confidence_reasons.py
index 6e4f4ac09..d11637376 100644
--- a/tests/unit/server/mcp/test_answer_confidence_reasons.py
+++ b/tests/unit/server/mcp/test_answer_confidence_reasons.py
@@ -35,7 +35,10 @@ import pytest
_SYMBOL = "run_sweep"
-def _patch_pipeline(monkeypatch, answer_mod, scores, *, with_symbols=True, matched=False):
+def _patch_pipeline(
+ monkeypatch, answer_mod, scores, *, with_symbols=True, matched=False,
+ summary="How the alpha module clears out stale rows. " * 4,
+):
"""Two file-page hits at *scores*, the top one carrying a hydrated symbol."""
async def _fake_retrieve(question, ctx):
@@ -48,7 +51,7 @@ def _patch_pipeline(monkeypatch, answer_mod, scores, *, with_symbols=True, match
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Module summary for the page. " * 4
+ h["summary"] = summary
h["snippet"] = ""
h["page_type"] = "file_page"
if with_symbols and i == 0:
@@ -106,7 +109,7 @@ async def test_ambiguous_retrieval_never_reads_as_dominant(setup_mcp, monkeypatc
_patch_pipeline(monkeypatch, answer_mod, scores=(2.91, 2.91))
_patch_provider(monkeypatch, answer_mod, _GROUNDED_ANSWER)
- result = await get_answer(_QUESTION)
+ result = await get_answer(_QUESTION, include=["diagnostics"])
note = result.get("note") or ""
assert result["confidence"] == "high", "the opt-in must actually reach the high branch"
assert result["retrieval_quality"] == "weak", "a tied top pair is ambiguous retrieval"
@@ -131,7 +134,7 @@ async def test_grounded_prose_does_not_earn_high_on_weak_retrieval(setup_mcp, mo
_patch_pipeline(monkeypatch, answer_mod, scores=(2.91, 2.91))
_patch_provider(monkeypatch, answer_mod, _GROUNDED_ANSWER)
- result = await get_answer(_QUESTION)
+ result = await get_answer(_QUESTION, include=["diagnostics"])
assert result["retrieval_quality"] == "weak"
assert result["confidence"] == "medium", (
"answer-grounding alone must not out-vote an ambiguous retrieval"
@@ -164,6 +167,25 @@ async def test_dominance_grade_honours_the_absolute_gap_tier(setup_mcp, monkeypa
assert "clearly dominates" in result["note"]
+async def test_a_dominant_top_hit_that_misses_the_question_is_not_high(setup_mcp, monkeypatch):
+ """Dominance compares hits with each other: an off-topic page beating a worse
+ one still dominates. The coverage gate holds it at medium and says why."""
+ import repowise.server.mcp_server.tool_answer.answer as answer_mod
+ from repowise.server.mcp_server import get_answer
+
+ _patch_pipeline(
+ monkeypatch, answer_mod, scores=(8.0, 2.0), summary="Parses yaml config files."
+ )
+ _patch_provider(monkeypatch, answer_mod, _PLAIN_ANSWER)
+
+ result = await get_answer(_QUESTION, include=["diagnostics"])
+ assert result["confidence"] == "medium"
+ assert result["retrieval_quality"] == "weak"
+ assert "Coverage gate" in result["note"]
+ assert "stale" in result["note"]
+
+
@pytest.mark.asyncio
async def test_the_gap_tier_does_not_quote_the_ratio_it_did_not_use(setup_mcp, monkeypatch):
"""The same self-refuting number, one tier down.
@@ -270,7 +292,7 @@ async def test_a_served_named_body_still_earns_high_on_weak_retrieval(
)
_patch_provider(monkeypatch, answer_mod, _GROUNDED_ANSWER)
- result = await get_answer(f"How does {_SYMBOL} work?")
+ result = await get_answer(f"How does {_SYMBOL} work?", include=["diagnostics"])
assert result["retrieval_quality"] == "weak"
assert result["symbol_bodies"], "the named body is what the grade rests on"
assert result["confidence"] == "high"
diff --git a/tests/unit/server/mcp/test_answer_degraded_evidence.py b/tests/unit/server/mcp/test_answer_degraded_evidence.py
index f15725b56..40827d7ed 100644
--- a/tests/unit/server/mcp/test_answer_degraded_evidence.py
+++ b/tests/unit/server/mcp/test_answer_degraded_evidence.py
@@ -222,9 +222,9 @@ async def test_degraded_with_no_anchor_match_still_describes_a_ranking(tmp_path)
# derived from it is pinned in the confidence block further down.
-def _scored(*scores: float) -> list[dict]:
+def _scored(*scores: float, summary: str = "how the module works") -> list[dict]:
return [
- {"target_path": f"pkg/m{i}.py", "title": f"m{i}", "summary": "s", "score": s}
+ {"target_path": f"pkg/m{i}.py", "title": f"m{i}", "summary": summary, "score": s}
for i, s in enumerate(scores)
]
@@ -253,6 +253,13 @@ async def test_degraded_rates_a_dominant_but_weak_retrieval_partial():
assert await _quality(_scored(1.0, 0.1)) == "partial"
+async def test_degraded_rates_a_dominant_hit_that_misses_the_question_weak():
+ """Dominance compares hits with each other; a winner that never mentions what
+ was asked is an off-topic page beating worse ones, so it rates weak."""
+ assert await _quality(_scored(6.0, 1.0, summary="parses yaml files")) == "weak"
+ assert await _confidence(_scored(6.0, 1.0, summary="parses yaml files")) == "low"
+
+
async def test_degraded_does_not_lift_a_genuinely_weak_retrieval():
"""The control that must not move.
@@ -572,13 +579,13 @@ async def test_degraded_cites_the_retrieval_top_file_before_a_rationale_hub(
]
-def _rationale_rows(monkeypatch, *paths: str) -> None:
+def _rationale_rows(monkeypatch, *paths: str, matched: tuple[str, ...] = ("flask", "app")) -> None:
"""Stub the rationale miner to return one row per path, in the given order."""
import repowise.server.mcp_server.tool_answer.degraded as degraded_mod
async def _rows(ctx, hits, fallback_targets, question):
return [
- {"path": p, "lines": [100 + i, 101 + i], "comment": "c", "matched_terms": []}
+ {"path": p, "lines": [100 + i, 101 + i], "comment": "c", "matched_terms": list(matched)}
for i, p in enumerate(paths)
]
@@ -819,7 +826,7 @@ def test_the_hint_is_the_same_whatever_the_payload_graded():
def _ranked(*paths: str, ranks: tuple[int | None, ...] | None = None) -> list[dict]:
"""Strong, dominant hits in this order, each stamped with its hybrid rank."""
hits = [
- {"target_path": p, "title": p, "summary": "s", "score": 6.0 if i == 0 else 2.0}
+ {"target_path": p, "title": p, "summary": "the Flask app", "score": 6.0 if i == 0 else 2.0}
for i, p in enumerate(paths)
]
for h, rank in zip(hits, ranks if ranks is not None else range(len(hits)), strict=True):
@@ -851,6 +858,16 @@ async def test_the_top_file_rationale_leads_over_a_higher_scoring_hub(tmp_path,
assert payload["confidence"] == "medium"
+async def test_a_rationale_matching_one_question_word_does_not_lead(tmp_path, monkeypatch):
+ """A marker plus one shared word clears the miner's gate but answers nothing."""
+ _rationale_rows(monkeypatch, "src/agents/poll-backoff.ts", matched=("flask",))
+ payload = await _keyless(tmp_path, _ranked("src/agents/poll-backoff.ts", "src/config/types.ts"))
+
+ answer = _answer(payload)
+ assert "Source rationale" not in answer
+ assert "Local retrieval points first to src/agents/poll-backoff.ts" in answer
+
+
async def test_without_a_top_file_rationale_the_answer_names_the_ranking(tmp_path, monkeypatch):
_rationale_rows(monkeypatch, "cmd/completions.go")
payload = await _keyless(tmp_path, _ranked("cmd/command.go", "cmd/completions.go"))
@@ -975,11 +992,11 @@ async def test_an_unranked_duplicate_does_not_hide_its_files_rank(tmp_path, monk
"""An injected symbol page of a ranked file can precede that file's own hit."""
_rationale_rows(monkeypatch, "src/hub.ts", "src/other.ts", "src/poll.ts")
hits = [
- {"target_path": "src/poll.ts::backoff", "title": "t", "summary": "s", "score": 6.0},
- {"target_path": "src/hub.ts", "title": "t", "summary": "s", "score": 2.0},
- {"target_path": "src/other.ts::Run", "title": "t", "summary": "s", "score": 1.5},
- {"target_path": "src/other.ts", "title": "t", "summary": "s", "score": 1.0},
- {"target_path": "src/poll.ts", "title": "t", "summary": "s", "score": 1.0},
+ {"target_path": "src/poll.ts::backoff", "title": "t", "summary": "the Flask app", "score": 6.0},
+ {"target_path": "src/hub.ts", "title": "t", "summary": "the Flask app", "score": 2.0},
+ {"target_path": "src/other.ts::Run", "title": "t", "summary": "the Flask app", "score": 1.5},
+ {"target_path": "src/other.ts", "title": "t", "summary": "the Flask app", "score": 1.0},
+ {"target_path": "src/poll.ts", "title": "t", "summary": "the Flask app", "score": 1.0},
]
for h, rank in zip(hits, (None, 5, None, 1, 0), strict=True):
if rank is not None:
diff --git a/tests/unit/server/mcp/test_answer_lookup_truncation.py b/tests/unit/server/mcp/test_answer_lookup_truncation.py
index 66b30cd15..25670536c 100644
--- a/tests/unit/server/mcp/test_answer_lookup_truncation.py
+++ b/tests/unit/server/mcp/test_answer_lookup_truncation.py
@@ -62,7 +62,7 @@ def _patch_pipeline(monkeypatch, answer_mod, *, symbol: dict, path: str | None =
for i, h in enumerate(hits):
h["target_path"] = path or h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Policy module summary."
+ h["summary"] = "Policy module: the retry loop decides its floor at startup."
h["snippet"] = ""
h["page_type"] = "file_page"
if i == 0:
diff --git a/tests/unit/server/mcp/test_answer_payload.py b/tests/unit/server/mcp/test_answer_payload.py
index 58a6cbf6e..e4c7bd757 100644
--- a/tests/unit/server/mcp/test_answer_payload.py
+++ b/tests/unit/server/mcp/test_answer_payload.py
@@ -209,7 +209,7 @@ def _patch_pipeline(monkeypatch, answer_mod, scores=(5.0, 4.0)):
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Summary text for the page. " * 12
+ h["summary"] = "Summary of the beta module function. " * 12
h["snippet"] = ""
h["page_type"] = "file_page"
if i == 0:
diff --git a/tests/unit/server/mcp/test_answer_projection.py b/tests/unit/server/mcp/test_answer_projection.py
index 8f590118f..0ddc355d7 100644
--- a/tests/unit/server/mcp/test_answer_projection.py
+++ b/tests/unit/server/mcp/test_answer_projection.py
@@ -270,6 +270,43 @@ def test_expanded_degraded_answer_names_only_retained_blocks():
assert "candidates" not in expanded["answer"]
[email protected]("confidence", ["high", "medium", "low"])
+def test_retrieval_quality_is_served_only_under_diagnostics(confidence):
+ """One trust signal by default; the retrieval rating is an opt-in diagnostic."""
+ default = project_answer_payload(_raw(confidence), question="how does auth work")
+ diagnostics = project_answer_payload(
+ _raw(confidence), question="how does auth work", include=["diagnostics"]
+ )
+
+ assert "retrieval_quality" not in default
+ assert diagnostics["retrieval_quality"] == _raw(confidence)["retrieval_quality"]
+ assert "ignored_arguments" not in diagnostics.get("_meta", {})
+
+
+def test_a_rationale_matching_one_question_word_does_not_lead():
+ raw = {
+ "answer": "generic fallback",
+ "citations": ["cobra.go"],
+ "confidence": "medium",
+ "degraded": "no-llm-provider",
+ "code_rationale": [
+ {
+ "path": "cobra.go",
+ "lines": [4, 5],
+ "comment": "FIXME Gt is unused by cobra and should be removed.",
+ "matched_terms": ["cobra"],
+ }
+ ],
+ "best_guesses": [{"file": "cobra.go", "why_relevant": "ranked first"}],
+ "_meta": {"contract_version": 1},
+ }
+
+ result = project_answer_payload(raw, question="how does cobra choose a subcommand")
+
+ assert "FIXME" not in result["answer"]
+ assert "points first to cobra.go" in result["answer"]
+
+
def test_rationale_only_degraded_answer_leads_with_the_local_conclusion():
raw = {
"answer": "generic fallback",
@@ -281,6 +318,7 @@ def test_rationale_only_degraded_answer_leads_with_the_local_conclusion():
"path": "src/cache.py",
"lines": [4, 5],
"comment": "The TTL matches the upstream feed's five-minute refresh.",
+ "matched_terms": ["ttl", "five", "minutes"],
}
],
"_meta": {"contract_version": 1},
@@ -318,7 +356,8 @@ async def test_no_llm_and_failed_legs_remain_compact_and_actionable(
)
result = await tool_middleware(get_answer)(
- f"how does auth work in {expected_degraded} {legs['vector']} mode"
+ "how does auth middleware work",
+ include=["diagnostics"],
)
assert result["degraded"] == expected_degraded
diff --git a/tests/unit/server/mcp/test_answer_shortlist.py b/tests/unit/server/mcp/test_answer_shortlist.py
index bd32dde07..48c14b939 100644
--- a/tests/unit/server/mcp/test_answer_shortlist.py
+++ b/tests/unit/server/mcp/test_answer_shortlist.py
@@ -276,7 +276,7 @@ def test_keyless_weak_reply_serves_the_ranked_list_and_one_guidance_line():
)
assert set(out) == {
- "answer", "confidence", "retrieval_quality", "degraded", "best_guesses",
+ "answer", "confidence", "degraded", "best_guesses",
"candidate_files", "code_rationale", "_meta",
}
assert "src/pkg/f0.py first" in out["answer"] and "search_codebase" in out["answer"]
diff --git a/tests/unit/server/mcp/test_answer_withheld_symbols.py b/tests/unit/server/mcp/test_answer_withheld_symbols.py
index 2fca7b8f2..f07f9dcb3 100644
--- a/tests/unit/server/mcp/test_answer_withheld_symbols.py
+++ b/tests/unit/server/mcp/test_answer_withheld_symbols.py
@@ -252,7 +252,7 @@ def _patch_pipeline(monkeypatch, answer_mod, *, symbol: dict):
for i, h in enumerate(hits):
h["target_path"] = h["page_id"].removeprefix("file_page:")
h["title"] = h["target_path"]
- h["summary"] = "Store summary."
+ h["summary"] = "Store keeps todos."
h["snippet"] = ""
h["page_type"] = "file_page"
if i == 0: