Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

feat(mcp): reach a file page by the words its own file uses, and serve paths a caller can open (#1259)

Raghav Chamadiya629da7e1672b
diff --git a/docs/agent/MCP_TOOLS.md b/docs/agent/MCP_TOOLS.md
index 42d03a173..ba6e55245 100644
--- a/docs/agent/MCP_TOOLS.md
+++ b/docs/agent/MCP_TOOLS.md
@@ -159,6 +159,17 @@ One-call RAG: retrieves over the wiki, gates synthesis on confidence, and return
**Returns:** A synthesized answer with file/symbol citations and a confidence label (`high`, `medium`, `low`). High-confidence answers can be cited directly. Low-confidence answers return ranked wiki excerpts instead.
+Two path-bearing blocks, with different jobs:
+
+| Field | Job | Confidence-gated? |
+|-------|-----|-------------------|
+| `retrieval` | **Evidence.** Enriched hits (summary, snippet, key symbols) to re-read when the prose needs checking. Shrinks as confidence rises, because a trustworthy answer needs less of it. | Yes |
+| `candidates` | **Navigation.** The ranked shortlist of files retrieval resolved, one `{path, lines?}` entry each, up to 20. | No |
+
+`candidates` is present whenever retrieval resolved anything, including on high-confidence answers where `retrieval` is deliberately empty. It is where to look next; it is not evidence that the answer is right.
+
+**Retrieval legs:** three, fused by Reciprocal Rank Fusion: full-text and vector search over wiki pages, plus the structural symbol index. The symbol leg is keyed on the content words of the question rather than on whether it happens to carry an identifier-shaped token, so "how does an incremental update persist symbols" reaches the same rows as `_persist_symbols`. It exists because a generated file page renders only the *public* symbol table: a private helper or a local name is not in the text the other two legs index.
+
**When to use:** First call on any code question. Collapses search, read, and reason into one round-trip. If confidence is low, follow up with `search_codebase` to discover candidate pages.
**Example call:**
@@ -281,7 +292,22 @@ the **wiki**, instead of forcing a fallback to Grep for identifiers.
- *Symbol hits*: `{type: "symbol", symbol_id, name, kind, file, start_line, end_line, signature, next: "get_symbol"}`. Ranked by exact-name/qualified-name match, query-token coverage, then graph centrality (PageRank / betweenness / entry-point); non-test before test unless `kind="test"`.
- *File hits*: `{type: "file", page_id, file, title, next: "get_context"}`.
-- *Concept hits*: ranked wiki pages with `relevance_score`, `snippet`, `target_path`, and a `search_method` (`embedding` vs `bm25` fallback).
+- *Concept hits*: ranked wiki pages with `relevance_score`, `snippet`, `target_path`, and a `search_method` (`embedding` vs `bm25` fallback). A `symbol_spotlight` page's `target_path` is a page identifier of the form `file.py::Symbol`; those hits also carry `file` with the openable path. **Read `file` when present.** `target_path` is for piping into `get_symbol`, not for opening.
+
+Alongside `results`, the response carries **`candidates`**: up to `limit`
+distinct files worth opening next, one `{path}` entry each, best first.
+
+Every entry is a real file path, and that is the difference between the two
+blocks. `results` ranks *pages*, and a page is not always a file: a
+`module_page` is named by a structural group key that reads exactly like a
+directory, an `scc_page` by `scc-<hash>`, an `onboarding` page by a slot name.
+Ranking those is correct; opening them is not. `candidates` resolves symbol
+pages to their file, collapses several symbols of one file to a single entry,
+skips every page that names no file, and backfills from below the result
+window so a slot spent on a module page does not also cost you a file.
+
+**If your next move is a Read, read `candidates`.** If you are enumerating
+matches or resolving a `symbol_id`, read `results`.
Tombstoned and `exclude_patterns`-excluded results are filtered. In workspace
mode, structural and concept searches both federate across repos and merge
diff --git a/packages/core/src/repowise/core/generation/context/assembler.py b/packages/core/src/repowise/core/generation/context/assembler.py
index 33c54f5f3..bbb2ef561 100644
--- a/packages/core/src/repowise/core/generation/context/assembler.py
+++ b/packages/core/src/repowise/core/generation/context/assembler.py
@@ -14,6 +14,7 @@ from repowise.core.ingestion.models import ParsedFile, RepoStructure, Symbol
from ..categories import file_category
from ..models import GenerationConfig
+from .code_vocabulary import code_vocabulary
from .contexts import (
ApiContractContext,
ArchitectureDiagramContext,
@@ -393,6 +394,13 @@ class ContextAssembler:
kg_tour_step=kg_context.tour_step if kg_context else None,
kg_tags=kg_context.tags if kg_context else [],
kg_node_summary=kg_context.node_summary if kg_context else "",
+ # Computed from the whole decoded source, not from ``snippet``.
+ # The snippet is budget-trimmed and on a large file is replaced by
+ # a structural summary, so reading it would make the vocabulary of
+ # the biggest files the thinnest, which is backwards. This section
+ # carries its own cap and is not charged against the prompt budget
+ # because it is page content rather than model input.
+ code_vocabulary=code_vocabulary(source_text),
)
# ------------------------------------------------------------------
diff --git a/packages/core/src/repowise/core/generation/context/code_vocabulary.py b/packages/core/src/repowise/core/generation/context/code_vocabulary.py
new file mode 100644
index 000000000..a386b49ef
--- /dev/null
+++ b/packages/core/src/repowise/core/generation/context/code_vocabulary.py
@@ -0,0 +1,127 @@
+"""The file's own vocabulary: the words a question about this file would use.
+
+A file page renders an overview, a table of symbols and a list of dependency
+paths. None of those carry the words a reader actually searches with: a flag
+name, an error message, a struct field, the phrasing of a doc comment. So a
+page can describe a file accurately and still be unreachable by any question
+about what the file does.
+
+Measured, on `pkg/cmd/release/list/list.go` in cli/cli. The source declares
+``Order string``, registers the flag as ``"order"`` with the help text
+``"Order of releases returned"``, and builds a GraphQL query naming
+``CREATED_AT``. Its page carried a path header, two symbol names and 29
+dependency paths, and not one of those tokens. A question asking how to order
+a release list by creation date had nothing to match.
+
+This module returns a bounded bag of that vocabulary, for the page to render
+and the index to embed. Three properties are deliberate:
+
+* **Ordered by how distinguishing it is, because it is capped.** Declared
+ names first, then literals, then comment prose, then everything else. A cap
+ that truncates the tail should truncate the least specific material.
+* **Deduplicated and lowercased into words.** ``ExcludePreReleases`` is stored
+ as itself and as ``exclude``, ``pre``, ``releases``, because a question is
+ asked in words, not in camel case.
+* **Vocabulary, not a code listing.** No structure, no bodies, no ordering
+ that implies control flow. Anyone wanting the code should open the file, and
+ the page says where it is.
+"""
+
+from __future__ import annotations
+
+import re
+
+# A cap, not a budget. Roughly the size of the rest of a file page, so this
+# section can matter without dominating what a reader sees.
+_MAX_CHARS = 1200
+
+# Two or more characters so single-letter receivers and loop variables do not
+# fill the bag; they are never what a question is about.
+_IDENT = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")
+
+# Quoted text, both quote styles, bounded at both ends. The lower bound drops
+# punctuation and separators; the upper bound drops embedded blobs, SQL and
+# minified data, which are long, unsearchable and would eat the whole cap.
+_STRING = re.compile(r'"([^"\\\n]{3,60})"' r"|'([^'\\\n]{3,60})'")
+
+# An indented, capitalised name followed by a type: a Go struct field, and
+# close enough to a TypeScript interface member to be worth the same pattern.
+_GO_FIELD = re.compile(r"^[ \t]+([A-Z][A-Za-z0-9_]*)\s+[\w\*\[\]\.\{\}]+", re.MULTILINE)
+
+# An indented assignment or annotation: a Python class attribute or dataclass
+# field. Matches some locals too, which is acceptable; a local name is still
+# vocabulary from this file.
+_PY_ATTR = re.compile(r"^[ \t]{4,}(?:self\.)?([a-z_][a-z0-9_]*)\s*[:=][^=]", re.MULTILINE)
+
+# A whole-line comment. Trailing comments are skipped on purpose: they annotate
+# a statement and tend to be fragments, where a leading comment is usually a
+# sentence about the thing below it.
+_COMMENT = re.compile(r"^[ \t]*(?://|#)[ \t]?(.{4,120})$", re.MULTILINE)
+
+
+def _words(token: str) -> list[str]:
+ """``ExcludePreReleases`` and ``exclude_pre_releases`` into their words."""
+ out: list[str] = []
+ for part in re.split(r"_+", token):
+ out.extend(re.findall(r"[A-Z]+(?![a-z])|[A-Z][a-z0-9]*|[a-z0-9]+", part))
+ return [w.lower() for w in out if len(w) > 2]
+
+
+def code_vocabulary(source: str, *, max_chars: int = _MAX_CHARS) -> str:
+ """The bounded vocabulary of one source file, most distinguishing first.
+
+ Returns a plain space-joined string. Empty when *source* yields nothing,
+ so the caller can drop the whole section rather than render a heading with
+ nothing under it.
+ """
+ if not source:
+ return ""
+
+ seen: set[str] = set()
+ bag: list[str] = []
+ budget = max_chars
+
+ def add(term: str) -> bool:
+ """Append if new and affordable. False once the cap is reached."""
+ nonlocal budget
+ term = term.strip()
+ if not term or term.lower() in seen:
+ return True
+ cost = len(term) + 1
+ if cost > budget:
+ return False
+ seen.add(term.lower())
+ bag.append(term)
+ budget -= cost
+ return True
+
+ # 1. Declared member names. The most specific thing a file contains that
+ # its symbol table does not already name.
+ for match in _GO_FIELD.findall(source) + _PY_ATTR.findall(source):
+ if not add(match):
+ return " ".join(bag)
+ for word in _words(match):
+ if not add(word):
+ return " ".join(bag)
+
+ # 2. String literals: flag names, error text, help strings, route
+ # templates. This is where user-facing wording lives, which is the
+ # wording a bug report repeats back.
+ for double, single in _STRING.findall(source):
+ if not add(double or single):
+ return " ".join(bag)
+
+ # 3. Comment prose, the file's own sentences about itself.
+ for comment in _COMMENT.findall(source):
+ if not add(comment):
+ return " ".join(bag)
+
+ # 4. Everything else, as words. Deliberately last: an identifier that
+ # matters is usually already above, and this tail is the part a cap
+ # should be free to cut.
+ for token in _IDENT.findall(source):
+ for word in _words(token):
+ if not add(word):
+ return " ".join(bag)
+
+ return " ".join(bag)
diff --git a/packages/core/src/repowise/core/generation/context/contexts.py b/packages/core/src/repowise/core/generation/context/contexts.py
index 5835b3785..93e01ef1c 100644
--- a/packages/core/src/repowise/core/generation/context/contexts.py
+++ b/packages/core/src/repowise/core/generation/context/contexts.py
@@ -62,6 +62,11 @@ class FilePageContext:
kg_tour_step: dict | None = None
kg_tags: list[str] = field(default_factory=list)
kg_node_summary: str = ""
+ # The words the file's own source uses, bounded and ordered
+ # most-distinguishing first. See ``context/code_vocabulary.py`` for what
+ # goes in it and why. Empty when the file yields nothing, in which case the
+ # template drops the section rather than rendering an empty heading.
+ code_vocabulary: str = ""
@dataclass
diff --git a/packages/core/src/repowise/core/generation/templates/file_page.j2 b/packages/core/src/repowise/core/generation/templates/file_page.j2
index 7882b13c5..d0c3d0bf9 100644
--- a/packages/core/src/repowise/core/generation/templates/file_page.j2
+++ b/packages/core/src/repowise/core/generation/templates/file_page.j2
@@ -105,5 +105,29 @@ Part of the **{{ ctx.community_label }}** module cluster.
- {{ q }}
{%- endfor %}
{%- endif %}
+{#- The words the file itself uses.
+
+ Everything above is a description of the file. This is the file's own
+ vocabulary: field names, string literals, help text, comment prose. The
+ page could describe a file accurately and still be unfindable by any
+ question about what it does, because a question is asked in the words of
+ the source and none of those words were on the page.
+
+ Per-file, and that is the whole point. Repo-level vocabulary rendered here
+ would be byte-identical across every file page in the corpus, which is the
+ failure this template's own header warns about: it would match every query
+ and distinguish none of them.
+
+ Last on the page, below the questions block, for the same reason that
+ block is low: ``_extract_summary`` reads back from the top, and a bag of
+ words is not a summary. Conditional, like every section below the
+ Overview, so a file that yields no vocabulary renders no heading.
+-#}
+{%- if ctx.code_vocabulary %}
+
+## In the code
+
+{{ ctx.code_vocabulary }}
+{%- endif %}
{% include "_footer.j2" %}
diff --git a/packages/server/src/repowise/server/mcp_server/_answer_pipeline.py b/packages/server/src/repowise/server/mcp_server/_answer_pipeline.py
index fa80bf057..917a89f9d 100644
--- a/packages/server/src/repowise/server/mcp_server/_answer_pipeline.py
+++ b/packages/server/src/repowise/server/mcp_server/_answer_pipeline.py
@@ -8,13 +8,18 @@ vice versa.
Pipeline (each stage is a pure function over hit dicts):
- 1. ``hybrid_retrieve`` — FTS + vector store in parallel, merged via
- Reciprocal Rank Fusion. Single retrieval
- modes systematically miss either token
- matches (vectors drift) or conceptual
- matches (FTS is literal). Two modes catch
- both classes of failure for the cost of one
- extra coroutine.
+ 1. ``hybrid_retrieve`` : FTS, vector store and the structural symbol
+ index in parallel, merged via Reciprocal
+ Rank Fusion. Single retrieval modes
+ systematically miss either token matches
+ (vectors drift) or conceptual matches (FTS
+ is literal), and both page-shaped modes miss
+ anything a generated file page does not
+ spell out: its public-symbol table is all
+ they can index, so a private helper or a
+ local name is invisible to them. The symbol
+ leg covers that third class, for the cost of
+ one more coroutine.
2. ``hydrate_hits`` — attach target_path, summary, page_type from
the Page table to each hit.
3. ``apply_pagerank_bias`` — multiply scores by a damped PageRank factor
@@ -37,17 +42,19 @@ from __future__ import annotations
import asyncio
import contextlib
+import contextvars
import logging
import re
from collections import OrderedDict
from typing import Any
-from sqlalchemy import select
+from sqlalchemy import func, select
from repowise.core.analysis.decisions.semantic_match import DECISION_VECTOR_PREFIX
from repowise.core.persistence.database import get_session
from repowise.core.persistence.models import GraphEdge, GraphNode, Page
from repowise.core.test_paths import is_test_path
+from repowise.server.mcp_server._prose_symbols import symbol_backed_pages
_log = logging.getLogger("repowise.mcp.answer")
@@ -90,6 +97,15 @@ _RRF_SCORE_SCALE = 180.0
_GRAPH_EXPAND_TOP_N = 2
_GRAPH_EXPAND_MAX_NEW = 3
+# Degree above which a neighbour is treated as a hub and dropped from the
+# expansion set. A file that half the repo imports is a near-neighbour of
+# everything, so it says nothing about *this* question; it is also exactly the
+# file PageRank ranks first, which is why the ordering below needs a guard
+# rather than trusting centrality on its own. Floor plus percentile, so a small
+# repo where every file has a handful of edges excludes nothing.
+_HUB_DEGREE_FLOOR = 50
+_HUB_DEGREE_PERCENTILE = 0.99
+
# PageRank bias is multiplicative and capped. We don't want a marginally
# more central file to outrank a strong text match — only to break ties.
# Empirically PageRank values on this corpus span ~0 to ~0.01; we normalise
@@ -108,6 +124,82 @@ _GRAPH_EXPAND_DAMPING = 0.7
# that it happens on its own.
_EMBED_TIMEOUT_S = 8.0
+# Which retrieval legs actually ran for the current question (finding A18).
+#
+# Every leg here is best-effort by design: a slow vector store must not be able
+# to block an answer. The defect was that the fallback was *silent*. An embed
+# that times out returns no vector, retrieval quietly continues lexical-only,
+# and every health signal still reports green — ``embedder_live`` included,
+# because a configured embedder is live whether or not this particular call
+# beat the clock. Five queries in one bake-off run were answered without their
+# vector leg and nothing in the response, the logs' structured fields, or the
+# per-cell record said so.
+#
+# So the leg outcome travels with the answer. ``embedder_live`` says the
+# embedder exists; this says whether it was used. Those are different claims
+# and only one of them is about the answer the caller is holding.
+_LEG_RECORD: contextvars.ContextVar[dict[str, str] | None] = contextvars.ContextVar(
+ "repowise_retrieval_legs", default=None
+)
+
+
+def _record_leg(leg: str, outcome: str) -> None:
+ """Note how one retrieval leg ended, if anyone is collecting."""
+ record = _LEG_RECORD.get()
+ if record is not None:
+ record[leg] = outcome
+
+
+def begin_leg_record() -> dict[str, str]:
+ """Start collecting leg outcomes for one question. Returns the record.
+
+ The legs run under ``asyncio.gather``, which copies the context into each
+ task. Copying rebinds names, not objects, so the tasks all mutate this one
+ dict and the caller sees what they wrote.
+ """
+ record: dict[str, str] = {}
+ _LEG_RECORD.set(record)
+ return record
+
+
+def retrieval_legs() -> dict[str, str]:
+ """How each retrieval leg ended for the question just answered.
+
+ ``{"fts": "ok", "vector": "timeout", "symbol": "ok"}`` and so on, plus an
+ ``embed`` key when embedding the question is what failed. Empty before any
+ retrieval has run.
+ """
+ return dict(_LEG_RECORD.get() or {})
+
+
+def degraded_legs(legs: dict[str, str]) -> list[str]:
+ """The legs that did not run, named. Empty when retrieval was whole."""
+ return sorted(leg for leg, outcome in legs.items() if outcome != "ok")
+
+# Third retrieval leg: the structural symbol index. FTS and the vector store
+# both read the generated wiki page, which by construction carries an overview
+# sentence, the *public* symbol table and dependency paths, with no function
+# bodies, no private helpers, no local names. A question about something a page
+# does not spell out therefore has nothing to match on, however well it is
+# phrased. The symbol index does carry those names, and it is keyed on the
+# words a symbol is built from, so it recovers exactly that class of miss.
+#
+# How many symbols the leg ranks, and how many distinct files they may
+# contribute. Deliberately smaller than the other two legs' 15: this leg is
+# there to add pool members the page legs cannot see, not to outvote them.
+_SYMBOL_LEG_FETCH_LIMIT = 20
+_SYMBOL_LEG_MAX_PAGES = 8
+
+# The symbol leg's own RRF constant, larger than the page legs' so its
+# contribution is smaller at every rank. Fusing it at k=60 like the others gave
+# one lexical symbol match the same weight as a page two independent retrievers
+# ranked first, which is not what a name match is worth: measured on the
+# 99-question eval it pushed the correct ``distill/skeleton.py`` out of the
+# served five in favour of a same-named React component. At k=180 the leg can
+# lift a file the page retrievers already liked and can add one they never saw,
+# but it cannot outvote them.
+_SYMBOL_LEG_RRF_K = 180
+
# ---------------------------------------------------------------------------
# The question's embedding, computed once per call
@@ -149,7 +241,20 @@ async def question_vector(ctx: Any, question: str) -> list[float] | None:
try:
vectors = await asyncio.wait_for(store.embed_texts([question]), timeout=_EMBED_TIMEOUT_S)
+ except TimeoutError:
+ # The A18 case, and the one worth naming separately: the embedder is
+ # configured, reachable and healthy, and simply did not answer inside
+ # the budget. Nothing downstream fails, so without this the answer is
+ # lexical-only and indistinguishable from one that was not.
+ _record_leg("embed", "timeout")
+ _log.warning(
+ "get_answer could not embed the question within %.1fs; retrieval "
+ "continues without a question vector",
+ _EMBED_TIMEOUT_S,
+ )
+ return None
except Exception:
+ _record_leg("embed", "error")
_log.warning(
"get_answer could not embed the question up front; each retrieval "
"stage will embed it again",
@@ -191,7 +296,7 @@ async def vector_search(
async def hybrid_retrieve(question: str, ctx: Any) -> list[dict]:
- """Run FTS and vector retrieval in parallel and merge via RRF.
+ """Run FTS, vector and symbol retrieval in parallel and merge via RRF.
Returns a list of dicts shaped ``{page_id, title, score, snippet,
page_type, _sources: set[str]}``. ``_sources`` names which retrievers
@@ -203,9 +308,13 @@ async def hybrid_retrieve(question: str, ctx: Any) -> list[dict]:
block the call. An empty result from one mode just means the other mode
fully drives ranking, which matches the pre-hybrid behaviour.
"""
+ # Reset before the legs run so the record describes this question and not
+ # a previous one that happened to share the task context.
+ begin_leg_record()
fts_task = _safe_fts_search(ctx, question)
vec_task = _safe_vector_search(ctx, question)
- fts_results, vec_results = await asyncio.gather(fts_task, vec_task)
+ sym_task = _safe_symbol_search(ctx, question)
+ fts_results, vec_results, sym_results = await asyncio.gather(fts_task, vec_task, sym_task)
# RRF merge. Each hit's contribution from a source is 1/(rank + k);
# hits appearing in both sources sum their contributions naturally.
@@ -225,6 +334,11 @@ async def hybrid_retrieve(question: str, ctx: Any) -> list[dict]:
entry["score"] = entry.get("score", 0.0) + 1.0 / (rank + _RRF_K)
entry["_sources"].add("vector")
entry["_vec_rank"] = rank
+ for rank, h in enumerate(sym_results):
+ entry = fused.setdefault(h.page_id, _hit_dict_from_result(h))
+ entry["score"] = entry.get("score", 0.0) + 1.0 / (rank + _SYMBOL_LEG_RRF_K)
+ entry["_sources"].add("symbol")
+ entry["_sym_rank"] = rank
# Scale to BM25-range so downstream confidence/dominance gates (tuned
# against the prior single-mode BM25 retrieval) keep behaving sanely.
@@ -241,13 +355,20 @@ async def hybrid_retrieve(question: str, ctx: Any) -> list[dict]:
async def _safe_fts_search(ctx: Any, question: str) -> list[Any]:
"""FTS search wrapped in timeout + suppression. Returns [] on any failure."""
if ctx.fts is None:
+ _record_leg("fts", "absent")
return []
try:
- return await asyncio.wait_for(
+ results = await asyncio.wait_for(
ctx.fts.search(question, limit=_RETRIEVAL_FETCH_LIMIT), timeout=5.0
)
+ except TimeoutError:
+ _record_leg("fts", "timeout")
+ return []
except Exception:
+ _record_leg("fts", "error")
return []
+ _record_leg("fts", "ok")
+ return results
def _pages_only(results: list[Any], limit: int) -> list[Any]:
@@ -294,6 +415,7 @@ async def _safe_vector_search(ctx: Any, question: str) -> list[Any]:
skipping the wait would race a background-loading store on cold start.
"""
if ctx.vector_store is None:
+ _record_leg("vector", "absent")
return []
ready = getattr(ctx, "vector_store_ready", None)
if ready is not None:
@@ -312,11 +434,71 @@ async def _safe_vector_search(ctx: Any, question: str) -> list[Any]:
),
timeout=8.0,
)
+ except TimeoutError:
+ _record_leg("vector", "timeout")
+ return []
except Exception:
+ _record_leg("vector", "error")
return []
+ _record_leg("vector", "ok")
return _pages_only(results, _RETRIEVAL_FETCH_LIMIT)
+class _SymbolLegResult:
+ """A file page reached through the symbol index, in retriever result shape.
+
+ The RRF merge is keyed on page ids and reads four attributes off whatever
+ the retrievers hand it, so the symbol leg presents its file pages the same
+ way FTS and the vector store present theirs.
+ """
+
+ __slots__ = ("page_id", "page_type", "snippet", "title")
+
+ def __init__(self, page_id: str, title: str, snippet: str, page_type: str) -> None:
+ self.page_id = page_id
+ self.title = title
+ self.snippet = snippet
+ self.page_type = page_type
+
+
+async def _safe_symbol_search(ctx: Any, question: str) -> list[_SymbolLegResult]:
+ """File pages whose symbols the question's words name. [] on any failure.
+
+ Ungated: it runs on every question, not only on ones that happen to carry
+ an identifier-shaped token. Which indexes get read is not something the
+ grammar of the sentence should decide. "How does an incremental update
+ persist symbols" and ``_persist_symbols`` are after the same file, and
+ before this leg only the second one reached the symbol index.
+
+ Best-effort with a timeout, like the other two legs: a slow or missing
+ symbol index degrades ``get_answer`` to its previous behaviour rather than
+ failing the call.
+ """
+ try:
+ pages = await asyncio.wait_for(
+ symbol_backed_pages(
+ ctx,
+ question,
+ max_files=_SYMBOL_LEG_MAX_PAGES,
+ symbol_limit=_SYMBOL_LEG_FETCH_LIMIT,
+ ),
+ timeout=5.0,
+ )
+ except TimeoutError:
+ _record_leg("symbol", "timeout")
+ _log.debug("get_answer symbol leg timed out; page retrieval stands")
+ return []
+ except Exception:
+ _record_leg("symbol", "error")
+ _log.debug("get_answer symbol leg failed; page retrieval stands", exc_info=True)
+ return []
+ _record_leg("symbol", "ok")
+ return [
+ _SymbolLegResult(p["page_id"], p["title"], (p.get("summary") or "")[:200], p["page_type"])
+ for p in pages
+ ]
+
+
def _hit_dict_from_result(result: Any) -> dict:
"""Convert a retriever result object to the pipeline's dict shape."""
return {
@@ -497,6 +679,39 @@ async def apply_pagerank_bias(hits: list[dict], ctx: Any) -> None:
# ---------------------------------------------------------------------------
+async def _neighbor_degrees(session: Any, nodes: set[str]) -> dict[str, int]:
+ """Total graph degree (in + out) for each node in *nodes*.
+
+ One grouped count per direction over the same indexed edge table the
+ expansion already reads, scoped to the candidate set rather than the whole
+ graph.
+ """
+ if not nodes:
+ return {}
+ degree: dict[str, int] = {}
+ for column in (GraphEdge.source_node_id, GraphEdge.target_node_id):
+ res = await session.execute(
+ select(column, func.count()).where(column.in_(nodes)).group_by(column)
+ )
+ for node_id, count in res.all():
+ degree[node_id] = degree.get(node_id, 0) + int(count or 0)
+ return degree
+
+
+def _hub_degree_cutoff(degree: dict[str, int]) -> int:
+ """Degree at which a candidate counts as a hub, for this candidate set.
+
+ ``max(floor, p99)``. The floor keeps a small or sparsely-linked repo from
+ excluding ordinary files, and the percentile keeps a densely-linked one
+ from excluding nothing.
+ """
+ if not degree:
+ return 1 << 30
+ values = sorted(degree.values())
+ idx = min(len(values) - 1, int(len(values) * _HUB_DEGREE_PERCENTILE))
+ return max(_HUB_DEGREE_FLOOR, values[idx])
+
+
async def expand_via_graph(hits: list[dict], ctx: Any) -> list[dict]:
"""Add up to ``_GRAPH_EXPAND_MAX_NEW`` graph-neighbor files to ``hits``.
@@ -562,10 +777,20 @@ async def expand_via_graph(hits: list[dict], ctx: Any) -> list[dict]:
)
)
pr_by_path = {row[0]: float(row[1] or 0.0) for row in pr_res.all()}
+ degree = await _neighbor_degrees(session, neighbors)
if not page_rows:
return hits
+ # Drop hubs before ranking. Ranking by PageRank alone actively prefers
+ # them, which is the wrong instinct here: expansion is trying to name the
+ # specific file the question is about, and the most-imported file in the
+ # repo is the least specific candidate available.
+ cutoff = _hub_degree_cutoff(degree)
+ non_hub = [row for row in page_rows if degree.get(row[0], 0) <= cutoff]
+ if non_hub:
+ page_rows = non_hub
+
# Damp parent score by _GRAPH_EXPAND_DAMPING for child candidates; pick
# the strongest parent each child connects to (taking the max parent
# score is conservative — favors well-connected neighbors).
diff --git a/packages/server/src/repowise/server/mcp_server/_neighbor_rerank.py b/packages/server/src/repowise/server/mcp_server/_neighbor_rerank.py
index d95f06aed..f7b712962 100644
--- a/packages/server/src/repowise/server/mcp_server/_neighbor_rerank.py
+++ b/packages/server/src/repowise/server/mcp_server/_neighbor_rerank.py
@@ -13,6 +13,19 @@ embedding and lexical relevance to the question. Within that ~200-file
neighbourhood a far endpoint that lost the corpus-wide retrieval wins, so it
rises; the top few contest the bottom served slot. The head is never reordered,
so a gold already surfaced can't be demoted.
+
+**The motion-cue gate was removed and put back, 2026-08-02.** Running the walk
+on every question is the obviously-right-looking change: phrasing is not a
+property of the graph, and this module's own note argues the precision controls
+are the walk and the contested slot rather than the gate. Measured on the
+99-question eval it cost 6.4 points of recall@5. The mechanism is specific: the
+stage re-ranks everything below ``_KEEP_TOP`` by fused relevance and injects
+**file** pages, and on a subsystem-shaped question ("what does the health
+subsystem cover") the answer is a *module* page sitting in one of those lower
+slots. Fabricated file neighbours evict it. The gate was accidentally load-
+bearing, not because flow questions are the only ones with far endpoints, but
+because they are the ones whose answer is never a module page. Reaching the
+other questions needs the injection to respect page type, not a wider gate.
"""
from __future__ import annotations
diff --git a/packages/server/src/repowise/server/mcp_server/_page_paths.py b/packages/server/src/repowise/server/mcp_server/_page_paths.py
new file mode 100644
index 000000000..eab8d2594
--- /dev/null
+++ b/packages/server/src/repowise/server/mcp_server/_page_paths.py
@@ -0,0 +1,97 @@
+"""Which page identifiers name a file a caller can actually open.
+
+A page id is ``{page_type}:{target_path}``, and ``target_path`` means a
+different thing for each type. For a file page it is the file. For a module
+page it is a structural group key, which reads like a directory. For an SCC
+page it is ``scc-<hash>``. For an onboarding page it is a slot name. For the
+repository overview it is the repository's name.
+
+Every one of those is a legitimate thing to *rank*, and none of them is a
+legitimate thing to hand back in a field named ``path`` or ``file``. An agent
+told to read ``onboarding/guided_tour`` or ``pkg/cmd/release`` gets an error,
+and nothing in the response says the string was never a path. This module is
+the one place that distinction is written down, so the tools cannot drift on
+it.
+"""
+
+from __future__ import annotations
+
+# Page types whose ``target_path`` is a real repository-relative file path.
+#
+# ``symbol_spotlight`` is included because its target_path is ``file.py::Sym``:
+# a page id built from a file path, so the file is recoverable by splitting.
+# Everything absent from this set is named by its members or by a curated id
+# (module_page by a structural group key, scc_page by ``scc-<hash>``,
+# layer_page by a layer name, repo_overview by the repo name,
+# architecture_diagram and onboarding by a slot), and no amount of string
+# manipulation turns those into something openable.
+FILE_BACKED_PAGE_TYPES: frozenset[str] = frozenset(
+ {
+ "file_page",
+ "symbol_spotlight",
+ "api_contract",
+ "infra_page",
+ }
+)
+
+
+def file_path_of(page_type: str | None, target_path: str | None) -> str | None:
+ """The file *page_type*/*target_path* names, or ``None`` if it names none.
+
+ Splits a ``symbol_spotlight``'s ``file.py::Symbol`` back to ``file.py``.
+ Returns ``None`` rather than the raw string for every page type that is not
+ file-backed, so a caller cannot accidentally serve a page id as a path by
+ forgetting to check the type.
+ """
+ if page_type not in FILE_BACKED_PAGE_TYPES:
+ return None
+ path = (target_path or "").split("::", 1)[0].strip()
+ return path or None
+
+
+def hit_file_path(hit: dict) -> str | None:
+ """The file a single search hit resolves to, whatever kind of hit it is.
+
+ Two hit shapes reach this. A page hit carries ``page_type`` and
+ ``target_path`` and is resolved by :func:`file_path_of`, so a module page
+ or an onboarding slot resolves to nothing. A symbol-index hit carries no
+ page type at all and names its file directly in ``file``.
+
+ The page-type branch is checked first and is authoritative: a page hit that
+ resolves to no file must not fall through to a looser key and smuggle a
+ page id out as a path.
+
+ A hit with **no** page type is a different case from a hit with an
+ unrecognised one, and they are deliberately not treated alike. An absent
+ type means the hit was never hydrated against the Page table, and dropping
+ it would silently lose a real file over a bookkeeping gap; so those fall
+ back to whatever path they carry. An unrecognised type means the page was
+ hydrated and classified as naming no file, which is an answer, not a gap.
+ """
+ if hit.get("page_type"):
+ return file_path_of(hit.get("page_type"), hit.get("target_path"))
+ raw = hit.get("file") or hit.get("target_path") or ""
+ return raw.split("::", 1)[0].strip() or None
+
+
+def file_candidates(hits: list[dict], *, limit: int) -> list[dict]:
+ """The distinct files *hits* resolve to, best first, one ``{path}`` each.
+
+ Ranked-order dedup: two symbol pages in one file are one file to open, and
+ a symbol page whose own file page also ranked is not a second place to
+ look. Pages that name no file are skipped rather than emitted as a path,
+ which is what keeps the block's promise that every entry can be opened.
+
+ Nothing is fetched to build this. It reads the hits a search already has.
+ """
+ out: list[dict] = []
+ seen: set[str] = set()
+ for hit in hits:
+ path = hit_file_path(hit)
+ if not path or path in seen:
+ continue
+ seen.add(path)
+ out.append({"path": path})
+ if len(out) >= limit:
+ break
+ return out
diff --git a/packages/server/src/repowise/server/mcp_server/_prose_symbols.py b/packages/server/src/repowise/server/mcp_server/_prose_symbols.py
new file mode 100644
index 000000000..cf1bbc6d5
--- /dev/null
+++ b/packages/server/src/repowise/server/mcp_server/_prose_symbols.py
@@ -0,0 +1,288 @@
+"""Symbol retrieval keyed on the words a question is asked in.
+
+RepoWise has always been able to search its structural symbol index, but only
+when the caller happened to *type* a symbol: ``search_codebase`` routed to the
+symbol leg on a bare identifier or on a snake_case / CamelCase token embedded
+in prose, and ``get_answer`` anchored symbols through the same shape test. Ask
+the identical question in English and the symbol index is never consulted:
+
+ "_persist_symbols" -> symbol leg runs
+ "how does an incremental update persist symbols" -> symbol leg silent
+
+Those two callers want the same file. The shape of the sentence is not a
+property of the codebase, so it should not decide which indexes get read.
+
+This module removes the shape test. It takes the content words of a question
+and searches ``wiki_symbols`` on them directly, because a symbol name is
+*built out of* the same words a developer uses to describe it
+(``persist`` + ``symbols`` -> ``_persist_symbols``).
+
+Two guards keep that from turning into noise, both borrowed from how lexical
+code search engines handle the same problem:
+
+* **Saturation down-weighting**. A term matching a large share of the
+ candidate window carries almost no information about which symbol is meant
+ (``get``, ``config``, ``file``). Rather than hand-maintaining a blocklist of
+ "generic" programming words, we measure it: a term whose per-term window
+ came back full is worth a quarter of a term whose window did not. This is a
+ cheap stand-in for IDF that needs no corpus statistics on disk.
+* **Corroboration**. A single common word is not enough to seed a hit. A
+ symbol has to cover either two distinct query terms, or one term that *is*
+ its leaf name. So "how does the update command work" cannot drag in every
+ symbol named ``update``, but "how does an incremental update persist
+ symbols" can reach ``_persist_symbols``.
+
+Per-term candidate windows (rather than one OR-clause with a global cap) are
+load-bearing: with a single window, one common term's matches evict every
+other term's before scoring ever runs, which is the failure that makes naive
+prose-to-symbol search useless.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sqlalchemy import func, select
+
+from repowise.core.persistence.database import get_session
+from repowise.core.persistence.models import GraphNode, Page, WikiSymbol
+from repowise.core.test_paths import is_test_path, is_test_related_path
+from repowise.server.mcp_server._helpers import (
+ LIKE_ESCAPE,
+ _get_exclude_spec,
+ _get_repo,
+ escape_like,
+ is_excluded,
+)
+from repowise.server.mcp_server._query_terms import content_terms as content_terms
+from repowise.server.mcp_server.tool_search_symbols import (
+ _symbol_kind_for_request_kind,
+ _symbol_result,
+ _tokens,
+ _tombstoned_paths,
+)
+
+# Rows pulled per term. Bounded per term so a common word cannot evict the
+# rare word that actually identifies the symbol; small enough that a
+# many-term question stays a handful of cheap scans.
+_PER_TERM_CANDIDATES = 80
+
+# A term whose window came back full matched at least this many rows, so we
+# learn nothing from it about *which* symbol is meant. Kept as a multiplier
+# rather than a drop: a saturated term is weak evidence, not none.
+_SATURATED_TERM_WEIGHT = 0.25
+
+# Score awarded per distinct query term a symbol's tokens cover, scaled by
+# that term's weight. The absolute number is arbitrary, since these scores are
+# only ever compared against each other, then fused by rank.
+_TERM_MATCH_SCORE = 10.0
+
+# Bonus when a term IS the symbol's whole name (``persist`` -> ``persist``),
+# as opposed to being one token inside it (``persist`` -> ``_persist_symbols``).
+_LEAF_NAME_BONUS = 15.0
+
+
+async def _candidates_for_term(session, repo_id: str, term: str) -> tuple[list[WikiSymbol], bool]:
+ """Up to :data:`_PER_TERM_CANDIDATES` symbols whose name mentions *term*.
+
+ Returns ``(rows, saturated)``. Shorter names first: a term is a larger
+ share of a short name, so ``persist`` identifies ``persist_pages`` far
+ more strongly than it identifies ``_persist_symbols_for_changed_files``,
+ and the window should not be spent on the latter.
+ """
+ esc = escape_like(term)
+ stmt = (
+ select(WikiSymbol)
+ .where(
+ WikiSymbol.repository_id == repo_id,
+ WikiSymbol.name.ilike(f"%{esc}%", escape=LIKE_ESCAPE),
+ )
+ .order_by(func.length(WikiSymbol.name))
+ .limit(_PER_TERM_CANDIDATES)
+ )
+ rows = list((await session.execute(stmt)).scalars().all())
+ return rows, len(rows) >= _PER_TERM_CANDIDATES
+
+
+def _score(
+ row: WikiSymbol,
+ gnode: GraphNode | None,
+ covered: dict[str, float],
+) -> float:
+ """Rank one candidate by how much of the question its name accounts for.
+
+ ``covered`` maps each matched term to its weight. Graph centrality breaks
+ ties the same way it does in :mod:`tool_search_symbols`, bounded so a
+ central file cannot outrank a real name match.
+ """
+ name = (row.name or "").lower()
+ score = sum(_TERM_MATCH_SCORE * w for w in covered.values())
+ if name in covered:
+ score += _LEAF_NAME_BONUS
+ if gnode is not None:
+ score += min(gnode.pagerank or 0.0, 0.1) * 50.0
+ score += min(gnode.betweenness or 0.0, 0.1) * 20.0
+ if gnode.is_entry_point:
+ score += 3.0
+ if (gnode is not None and gnode.is_test) or is_test_path(row.file_path or "", row.language):
+ score -= 5.0
+ return score
+
+
+def _corroborated(row: WikiSymbol, covered: dict[str, float], saturated: set[str]) -> bool:
+ """Whether *row* has enough independent evidence to enter the pool.
+
+ One term is enough when it is the symbol's whole name and it is not a
+ saturated one: that is a caller naming the thing, just without the
+ underscores. Otherwise **two informative terms** must land. Saturated terms
+ do not count toward that pair: two common words agreeing is not corroboration,
+ it is the same non-signal twice, and letting them through is what put
+ ``ui/lib/cn.ts`` in the top five for a question about building skeletons.
+ Directly mirrors the guard lexical code-search engines apply to bare English
+ query words.
+ """
+ if not covered:
+ return False
+ name = (row.name or "").lower()
+ if name in covered and name not in saturated:
+ return True
+ return sum(1 for t in covered if t not in saturated) >= 2
+
+
+async def search_symbols_by_terms(
+ ctx: Any,
+ terms: list[str],
+ limit: int,
+ *,
+ kind: str | None = None,
+) -> list[dict]:
+ """Rank indexed symbols against the content words of a question.
+
+ The prose-side counterpart to
+ :func:`tool_search_symbols.search_symbols_single`, which needs the caller
+ to have typed an identifier. Returns the same result shape, so a caller
+ can interleave the two without special-casing either.
+ """
+ terms = [t for t in terms if t]
+ if not terms:
+ return []
+
+ async with get_session(ctx.session_factory) as session:
+ repository = await _get_repo(session)
+ by_id: dict[str, WikiSymbol] = {}
+ matched: dict[str, set[str]] = {}
+ saturated: set[str] = set()
+ for term in terms:
+ rows, is_saturated = await _candidates_for_term(session, repository.id, term)
+ if is_saturated:
+ saturated.add(term)
+ for row in rows:
+ by_id[row.symbol_id] = row
+ matched.setdefault(row.symbol_id, set()).add(term)
+ if not by_id:
+ return []
+
+ gres = await session.execute(
+ select(GraphNode).where(
+ GraphNode.repository_id == repository.id,
+ GraphNode.node_id.in_(list(by_id)),
+ )
+ )
+ gmap = {g.node_id: g for g in gres.scalars().all()}
+ tombstoned = await _tombstoned_paths(
+ session, repository.id, {r.file_path for r in by_id.values()}
+ )
+
+ spec = _get_exclude_spec(ctx.path)
+ scored: list[tuple[float, WikiSymbol]] = []
+ for symbol_id, row in by_id.items():
+ if is_excluded(row.file_path, spec) or row.file_path in tombstoned:
+ continue
+ gnode = gmap.get(symbol_id)
+ is_test = (gnode is not None and gnode.is_test) or is_test_related_path(
+ row.file_path or "", row.language
+ )
+ if not _symbol_kind_for_request_kind(kind, is_test):
+ continue
+ # A term counts as covered when it survives tokenisation of the symbol
+ # name, not merely as a substring: ``update`` should match
+ # ``update_index``, not ``groupdater``.
+ stoks = _tokens(row.name) | _tokens(row.qualified_name)
+ covered = {
+ t: (_SATURATED_TERM_WEIGHT if t in saturated else 1.0)
+ for t in matched.get(symbol_id, ())
+ if t in stoks
+ }
+ if not _corroborated(row, covered, saturated):
+ continue
+ scored.append((_score(row, gnode, covered), row))
+
+ scored.sort(key=lambda pair: (-pair[0], pair[1].symbol_id or ""))
+ return [_symbol_result(row, score) for score, row in scored[:limit]]
+
+
+async def symbol_backed_pages(
+ ctx: Any,
+ question: str,
+ *,
+ max_files: int,
+ symbol_limit: int = 20,
+) -> list[dict]:
+ """File pages whose indexed symbols the words of *question* name.
+
+ The shared entry point for both retrieval surfaces. Returns page rows in
+ symbol-rank order, at most one per file: several symbol hits inside one
+ file are one candidate to read, not several.
+
+ Why this exists at all: a generated file page renders an overview
+ sentence, the **public** symbol table and dependency paths. Function
+ bodies, private helpers and local names never appear, so they are absent
+ from both the full-text index and the embedding. The symbol index is the
+ only place those names are searchable, and until now it was consulted only
+ when the caller happened to type one.
+
+ Never raises: a missing or slow symbol index leaves the page retrievers to
+ answer on their own, which is the behaviour that shipped before this.
+ """
+ terms = content_terms(question)
+ if not terms or max_files <= 0:
+ return []
+ try:
+ symbols = await search_symbols_by_terms(ctx, terms, symbol_limit)
+ except Exception:
+ return []
+ if not symbols:
+ return []
+
+ paths: list[str] = []
+ seen: set[str] = set()
+ for hit in symbols:
+ path = hit.get("file")
+ if path and path not in seen:
+ seen.add(path)
+ paths.append(path)
+ paths = paths[:max_files]
+ if not paths:
+ return []
+
+ async with get_session(ctx.session_factory) as session:
+ res = await session.execute(
+ select(Page.id, Page.target_path, Page.title, Page.summary, Page.page_type).where(
+ Page.target_path.in_(paths),
+ Page.page_type == "file_page",
+ Page.freshness_status != "tombstone",
+ )
+ )
+ by_path = {
+ row[1]: {
+ "page_id": row[0],
+ "target_path": row[1],
+ "title": row[2] or f"File: {row[1]}",
+ "summary": row[3] or "",
+ "page_type": row[4] or "file_page",
+ }
+ for row in res.all()
+ }
+ # A symbol whose file has no page cannot be read, so it contributes
+ # nothing and is dropped rather than ranked.
+ return [by_path[p] for p in paths if p in by_path]
diff --git a/packages/server/src/repowise/server/mcp_server/_query_terms.py b/packages/server/src/repowise/server/mcp_server/_query_terms.py
new file mode 100644
index 000000000..a9e368973
--- /dev/null
+++ b/packages/server/src/repowise/server/mcp_server/_query_terms.py
@@ -0,0 +1,147 @@
+"""The words of a question that could plausibly name code.
+
+Stdlib-only and dependency-free on purpose: the stopword set is shared by
+get_answer's coverage re-ranker (via ``tool_answer.config``) and by the
+prose-keyed symbol leg, and neither should have to import the other to reach
+it.
+"""
+
+from __future__ import annotations
+
+import re
+
+# English stopwords. Minimal list, just enough to keep "what is the" from
+# dominating coverage. Not language-specific, not repo-specific.
+STOPWORDS = frozenset(
+ {
+ "a",
+ "an",
+ "the",
+ "is",
+ "are",
+ "was",
+ "were",
+ "be",
+ "been",
+ "being",
+ "of",
+ "to",
+ "in",
+ "on",
+ "at",
+ "by",
+ "for",
+ "with",
+ "from",
+ "as",
+ "that",
+ "this",
+ "these",
+ "those",
+ "it",
+ "its",
+ "and",
+ "or",
+ "but",
+ "not",
+ "no",
+ "do",
+ "does",
+ "did",
+ "done",
+ "have",
+ "has",
+ "had",
+ "what",
+ "which",
+ "who",
+ "whom",
+ "whose",
+ "when",
+ "where",
+ "why",
+ "how",
+ "can",
+ "could",
+ "should",
+ "would",
+ "may",
+ "might",
+ "will",
+ "shall",
+ "i",
+ "you",
+ "he",
+ "she",
+ "we",
+ "they",
+ "me",
+ "him",
+ "her",
+ "us",
+ "them",
+ "my",
+ "your",
+ "his",
+ "their",
+ "our",
+ "if",
+ "then",
+ "than",
+ "so",
+ "such",
+ "there",
+ "here",
+ "about",
+ "into",
+ "through",
+ "between",
+ "across",
+ "over",
+ "under",
+ "up",
+ "down",
+ "out",
+ "off",
+ "via",
+ }
+)
+
+# Terms shorter than this match too much of any identifier to be worth a
+# lookup; it is also the floor the rest of the answer pipeline already uses.
+_MIN_TERM_LEN = 3
+
+# Cap so a pasted stack trace or a whole bug report cannot turn one question
+# into fifty table scans. Longest-first: a longer word is more specific, the
+# same intuition an IDF weight encodes.
+_MAX_TERMS = 8
+
+_WORD_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
+
+
+def split_humps(text: str) -> str:
+ """``HTTPServerPool`` -> ``HTTP Server Pool``; ``a_b`` -> ``a_b`` (unchanged)."""
+ out = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", text)
+ return re.sub(r"(?<=[A-Z])(?=[A-Z][a-z])", " ", out)
+
+
+def content_terms(question: str, *, max_terms: int = _MAX_TERMS) -> list[str]:
+ """Content words of *question*, deduplicated, longest first.
+
+ Deliberately inclusive: plain lowercase English words are kept, unlike the
+ identifier extractors elsewhere, which drop them because they are weak
+ signals *on their own*. Callers are expected to corroborate: one common
+ word must not be allowed to decide anything by itself.
+
+ A camelCase or snake_case token contributes its parts as well as itself, so
+ someone who writes ``persist_symbols`` and someone who writes "persist
+ symbols" reach the same rows.
+ """
+ seen: dict[str, None] = {}
+ for raw in _WORD_RE.findall(question or ""):
+ for tok in (raw, *re.split(r"[^A-Za-z0-9]+", split_humps(raw))):
+ t = tok.lower()
+ if len(t) < _MIN_TERM_LEN or t in STOPWORDS:
+ continue
+ seen.setdefault(t, None)
+ return sorted(seen, key=lambda t: (-len(t), t))[:max_terms]
diff --git a/packages/server/src/repowise/server/mcp_server/_server.py b/packages/server/src/repowise/server/mcp_server/_server.py
index 102a8a4c4..33b6c9d15 100644
--- a/packages/server/src/repowise/server/mcp_server/_server.py
+++ b/packages/server/src/repowise/server/mcp_server/_server.py
@@ -567,6 +567,12 @@ def run_mcp(
mcp.settings.port = port
mcp.run(transport="streamable-http")
else:
+ # stdout is the JSON-RPC channel on stdio, so every log line written
+ # there arrives at the client as a malformed protocol frame. Move the
+ # log sinks to stderr before anything can log.
+ from repowise.server.mcp_server._stdio_logging import route_logging_to_stderr
+
+ route_logging_to_stderr()
# stdio servers are spawned per-session by the MCP client; when the
# client dies abnormally the stdio loop doesn't exit (and Windows
# never kills children), leaking servers that hold wiki.db handles.
diff --git a/packages/server/src/repowise/server/mcp_server/_stdio_logging.py b/packages/server/src/repowise/server/mcp_server/_stdio_logging.py
new file mode 100644
index 000000000..1b60d1caf
--- /dev/null
+++ b/packages/server/src/repowise/server/mcp_server/_stdio_logging.py
@@ -0,0 +1,73 @@
+"""Keep log output off the stdio transport's protocol channel.
+
+On a stdio transport, **stdout is the JSON-RPC channel**. Anything else
+written there is framed as a protocol message by the client, which then fails
+to parse it:
+
+ Failed to parse JSONRPC message from server ...
+ Invalid JSON: trailing characters ... input_value='2026-08-01 20:12:13 [deb...'
+
+structlog's default logger factory prints to ``sys.stdout``, so every debug
+line the answer path emits during synthesis lands in the middle of the
+protocol stream: three malformed frames per ``get_answer`` call, reproducibly,
+on both repos of the 2026-08 bake-off.
+
+The Python ``mcp`` client logs those frames and drops them, so from the outside
+nothing looks broken. That is what makes this worth fixing rather than
+tolerating: a stricter JSON-RPC parser is entitled to close the connection on a
+malformed frame, and a dropped frame that happened to carry a real response
+leaves the call hanging with no error. stdio is the default transport every
+editor uses.
+
+Rich's console output already goes to stderr; this closes the structlog and
+stdlib-``logging`` halves.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+
+
+def route_logging_to_stderr() -> None:
+ """Point every log sink at stderr, leaving stdout to the protocol.
+
+ Idempotent and safe to call before any logging happens. Never raises: a
+ server that cannot reconfigure its logging should still start.
+ """
+ _redirect_stdlib_handlers()
+ _redirect_structlog()
+
+
+def _redirect_stdlib_handlers() -> None:
+ """Repoint any ``StreamHandler`` currently writing to stdout at stderr.
+
+ Handlers are rebound rather than removed so a caller that deliberately
+ installed one keeps its formatter and level; only the destination moves.
+ Loggers with no handler at all already fall back to ``logging.lastResort``,
+ which writes to stderr.
+ """
+ manager_loggers = list(logging.Logger.manager.loggerDict.values())
+ for obj in [logging.getLogger(), *manager_loggers]:
+ for handler in list(getattr(obj, "handlers", ()) or ()):
+ if getattr(handler, "stream", None) is sys.stdout:
+ handler.setStream(sys.stderr)
+
+
+def _redirect_structlog() -> None:
+ """Make structlog print to stderr.
+
+ ``cache_logger_on_first_use=False`` is load-bearing: modules that called
+ ``structlog.get_logger(__name__)`` at import time hold a logger bound
+ before this runs, and with caching left on their lines would keep going to
+ stdout. The same flag is set for the same reason in the CLI's
+ ``configure_cli_logging``.
+ """
+ try:
+ import structlog
+ except ImportError: # pragma: no cover - structlog is a hard dependency
+ return
+ structlog.configure(
+ logger_factory=structlog.PrintLoggerFactory(file=sys.stderr),
+ cache_logger_on_first_use=False,
+ )
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py b/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
index 82cacd430..15685b66a 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/answer.py
@@ -74,6 +74,9 @@ from repowise.server.mcp_server._answer_context import (
from repowise.server.mcp_server._answer_pipeline import (
apply_pagerank_bias as _apply_pagerank_bias,
)
+from repowise.server.mcp_server._answer_pipeline import (
+ degraded_legs as _degraded_legs,
+)
from repowise.server.mcp_server._answer_pipeline import (
demote_noise_hits as _demote_noise_hits,
)
@@ -87,6 +90,9 @@ from repowise.server.mcp_server._answer_pipeline import (
hybrid_retrieve as _hybrid_retrieve,
)
from repowise.server.mcp_server._answer_pipeline import hydrate_hits as _hydrate_hits
+from repowise.server.mcp_server._answer_pipeline import (
+ retrieval_legs as _retrieval_legs,
+)
from repowise.server.mcp_server._code_rationale import mine_rationale as _mine_rationale
from repowise.server.mcp_server._flow_path import expand_via_flow_path as _expand_via_flow_path
from repowise.server.mcp_server._helpers import (
@@ -135,6 +141,9 @@ from repowise.server.mcp_server.tool_answer.retrieval import (
_intersection_boost,
_rerank_by_coverage,
)
+from repowise.server.mcp_server.tool_answer.retrieval import (
+ serialize_candidates as _serialize_candidates,
+)
from repowise.server.mcp_server.tool_answer.retrieval import (
serialize_hits as _serialize_hits,
)
@@ -882,6 +891,12 @@ async def get_answer(
# all anchoring/expansion (which inject file/symbol pages, never noise) so it
# only reorders what those stages left in place.
hits = _demote_noise_hits(hits, question, is_why=_is_why_question(question))
+ # Everything retrieval resolved, in rank order, before the cap. Synthesis
+ # keeps its 5-hit budget, which is a context-window decision and the
+ # right one, but the files below the cut are still the best answer to
+ # "where do I look next", and they used to be discarded. ``candidates``
+ # (built after synthesis) hands them over at one line each.
+ resolved_pool = list(hits)
# Always cap retrieval hits at 5 for the response payload.
hits = hits[:5]
@@ -1660,6 +1675,16 @@ async def get_answer(
if flow_paths:
payload["flow_path"] = [" -> ".join(p) for p in flow_paths[:2]]
+ # Where to look next, always. ``retrieval`` shrinks as confidence rises
+ # (correctly: it is re-read evidence, and a trustworthy answer needs less
+ # of it), but that left the highest-confidence answers naming no file at
+ # all, which is the one thing an agent always has a use for. This block is
+ # navigation rather than evidence: the ranked shortlist, one path per line,
+ # ungated. It costs ~800 characters against a ~10k response.
+ candidates = _serialize_candidates(resolved_pool)
+ if candidates:
+ payload["candidates"] = candidates
+
# Persist to cache (upsert). Best-effort: cache failures must never block
# the response — but they must be LOGGED, not suppressed. A plain INSERT
# under a blanket suppress violated uq_answer_cache_q on every
@@ -1700,5 +1725,14 @@ async def get_answer(
repository=repository,
targets=[*citations, *fallback_targets],
)
+ # Finding A18. Each retrieval leg is best-effort so one slow backend cannot
+ # block an answer, which is right, but it made a lexical-only answer
+ # indistinguishable from a whole one: nothing failed, nothing was logged
+ # where a caller could see it, and ``embedder_live`` stayed true because a
+ # configured embedder is live whether or not this call beat its 8s budget.
+ # Named only when a leg actually fell over, so a healthy response pays
+ # nothing for it.
+ if degraded := _degraded_legs(_retrieval_legs()):
+ payload["_meta"]["retrieval_degraded"] = degraded
_apply_lean_high(payload, question)
return payload
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/config.py b/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
index 9aeb438f1..8417e3915 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/config.py
@@ -8,6 +8,8 @@ with a coverage re-ranker, not of any particular codebase.
from __future__ import annotations
+from repowise.server.mcp_server._query_terms import STOPWORDS
+
# How many top retrieval hits to enrich with WikiSymbol context. Enriching
# every hit produces large responses that bloat the cached prompt prefix on
# multi-turn agent sessions without changing the answer — the agent typically
@@ -360,102 +362,10 @@ _RELATIONAL_CONNECTIVES = (
# of their raw BM25 (vs 1.0 for a hit covering 3/3). Conjunctive coverage
# becomes a tie-breaker rather than a hard filter.
_COVERAGE_FLOOR = 0.5
-# English stopwords — minimal list, just enough to keep "what is the" from
-# dominating coverage. Not language-specific, not repo-specific.
-_STOPWORDS = frozenset(
- {
- "a",
- "an",
- "the",
- "is",
- "are",
- "was",
- "were",
- "be",
- "been",
- "being",
- "of",
- "to",
- "in",
- "on",
- "at",
- "by",
- "for",
- "with",
- "from",
- "as",
- "that",
- "this",
- "these",
- "those",
- "it",
- "its",
- "and",
- "or",
- "but",
- "not",
- "no",
- "do",
- "does",
- "did",
- "done",
- "have",
- "has",
- "had",
- "what",
- "which",
- "who",
- "whom",
- "whose",
- "when",
- "where",
- "why",
- "how",
- "can",
- "could",
- "should",
- "would",
- "may",
- "might",
- "will",
- "shall",
- "i",
- "you",
- "he",
- "she",
- "we",
- "they",
- "me",
- "him",
- "her",
- "us",
- "them",
- "my",
- "your",
- "his",
- "their",
- "our",
- "if",
- "then",
- "than",
- "so",
- "such",
- "there",
- "here",
- "about",
- "into",
- "through",
- "between",
- "across",
- "over",
- "under",
- "up",
- "down",
- "out",
- "off",
- "via",
- }
-)
+# English stopwords. Defined in ``_query_terms`` (stdlib-only, imported by
+# nothing in this package) so the coverage re-ranker here and the prose-keyed
+# symbol leg in ``_prose_symbols`` share one list instead of drifting apart.
+_STOPWORDS = STOPWORDS
# Cap on bytes read from source per symbol when we recover a real signature
# from disk (multi-line def with type annotations). Anything longer than this
# gets truncated; the agent can call get_symbol for the full body.
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
index f330d84c1..f0980625d 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
@@ -16,6 +16,7 @@ from sqlalchemy import select
from repowise.core.persistence.database import get_session
from repowise.core.persistence.models import Page
+from repowise.server.mcp_server._page_paths import hit_file_path
from repowise.server.mcp_server.tool_answer.config import (
_BACKEND_PATH_PREFIXES,
_BACKEND_QUESTION_TOKENS,
@@ -32,6 +33,58 @@ from repowise.server.mcp_server.tool_answer.config import (
_log = logging.getLogger("repowise.mcp.answer")
+# How many files ``candidates`` names. Twenty path lines cost roughly 800
+# characters against the ~10k a get_answer response already spends, so the
+# block makes the tool *more* token-efficient per candidate offered, not less.
+_CANDIDATE_LIMIT = 20
+
+
+def serialize_candidates(hits: list[dict], *, limit: int = _CANDIDATE_LIMIT) -> list[dict]:
+ """The files retrieval ranked, one line each, ordered best first.
+
+ Separate from ``retrieval`` on purpose, and deliberately not
+ confidence-gated. ``retrieval`` is *evidence*: enriched hits an agent reads
+ to check the prose, so it is right for it to shrink as the prose gets more
+ trustworthy. This block is *navigation*: the shortlist of files worth
+ opening next. Under the old shape a confident answer named zero files and a
+ medium one named two, which is backwards. The more sure we are of a
+ subsystem, the better placed we are to say which files it lives in.
+
+ One entry per distinct path, ``{path, lines?}``. Line bounds are attached
+ only where a hit already carries hydrated symbols; nothing is fetched to
+ build this.
+
+ ``path`` is always a **file** path, resolved through ``hit_file_path``. A
+ ``symbol_spotlight`` hit's ``target_path`` is ``file.py::Symbol``, which is
+ a page identifier, not something a consumer can open; two distinct symbols
+ in one file are also one file to read, so they collapse to one entry here.
+
+ Pages naming no file at all are skipped rather than emitted (finding A15).
+ A module page's target_path is a structural group key that reads like a
+ directory and an onboarding page's is a slot name, so every "does this look
+ like a path" heuristic says yes and the agent that opens it gets an error.
+ Scoring impact is about zero, measured; it is wrong on the same argument
+ A14 was, which is that this field has one meaning and it is not "page id".
+ """
+ out: list[dict] = []
+ seen: set[str] = set()
+ for h in hits:
+ path = hit_file_path(h)
+ if not path or path in seen:
+ continue
+ seen.add(path)
+ entry: dict[str, Any] = {"path": path}
+ symbols = h.get("symbols") or []
+ starts = [s["start_line"] for s in symbols if s.get("start_line")]
+ ends = [s["end_line"] for s in symbols if s.get("end_line")]
+ if starts and ends:
+ entry["lines"] = f"{min(starts)}-{max(ends)}"
+ out.append(entry)
+ if len(out) >= limit:
+ break
+ return out
+
+
def serialize_hits(
hits: list[dict],
*,
@@ -59,7 +112,14 @@ def serialize_hits(
"""
out: list[dict] = []
for h in hits[: limit if limit is not None else len(hits)]:
- entry: dict[str, Any] = {"path": h.get("target_path")}
+ target = h.get("target_path")
+ entry: dict[str, Any] = {"path": target}
+ # A symbol_spotlight page's target_path is ``file.py::Symbol``: a page
+ # id, not a path a consumer can open. Keep it (callers pipe it into
+ # get_symbol) and name the file too, so ``path`` never has to be
+ # guessed at by anything downstream.
+ if target and "::" in target:
+ entry["file"] = target.split("::", 1)[0]
if h.get("title"):
entry["title"] = h["title"]
summary = h.get("summary") or ""
diff --git a/packages/server/src/repowise/server/mcp_server/tool_search.py b/packages/server/src/repowise/server/mcp_server/tool_search.py
index ac56fc506..46fdcc856 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_search.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_search.py
@@ -25,6 +25,8 @@ from repowise.server.mcp_server._helpers import (
filter_dicts_by_key,
)
from repowise.server.mcp_server._meta import build_meta as _build_meta
+from repowise.server.mcp_server._page_paths import file_candidates, hit_file_path
+from repowise.server.mcp_server._prose_symbols import symbol_backed_pages
from repowise.server.mcp_server.tool_search_symbols import (
_qual_norm,
search_paths_single,
@@ -318,6 +320,64 @@ async def _rescue_all_decision_window(
return output
+# Slots at the tail of a concept window reserved for files reached through the
+# symbol index rather than through their page. A generated file page renders
+# only public symbols, so a question about a private helper or a local name has
+# nothing to match in either the full-text index or the embedding. The symbol
+# index is where those names live. One slot in three, taken from the weakest
+# end of the window, is a cheap price for reaching a class of file the page
+# retrievers structurally cannot see. No-op when the symbol leg finds nothing.
+_SYMBOL_TAIL_DIVISOR = 3
+
+
+def _symbol_tail_reserve(limit: int) -> int:
+ """How many tail slots the symbol leg may take from a window of *limit*."""
+ return limit // _SYMBOL_TAIL_DIVISOR
+
+
+async def _append_symbol_backed(
+ ctx, query: str, output: list[dict], limit: int, kind: str | None
+) -> list[dict]:
+ """Give the weakest tail slots to files the symbol index reached.
+
+ Concept hits keep the head. Symbol-backed files not already in the window
+ take up to :func:`_symbol_tail_reserve` slots, and the concept hits they
+ displaced fall in behind them (nothing is dropped before the caller's
+ ``limit`` cut). Returns ``output`` unchanged when the symbol leg is empty
+ or every file it named is already present.
+ """
+ reserve = _symbol_tail_reserve(limit)
+ if reserve <= 0 or kind in ("config", "doc"):
+ # config/doc windows are asking for pages with no symbols in them.
+ return output
+ pages = await symbol_backed_pages(ctx, query, max_files=reserve * 2)
+ if not pages:
+ return output
+
+ present = {item.get("target_path") for item in output}
+ extra = [p for p in pages if p["target_path"] not in present][:reserve]
+ if not extra:
+ return output
+ # Score them just under the last concept hit so the ordering the caller
+ # sees stays monotone; they are additions to the window, not a re-ranking
+ # of it.
+ floor = min((item.get("relevance_score") or 0.0) for item in output) if output else 0.0
+ additions = [
+ {
+ "page_id": p["page_id"],
+ "title": p["title"],
+ "page_type": p["page_type"],
+ "target_path": p["target_path"],
+ "snippet": (p.get("summary") or "")[:200],
+ "relevance_score": round(max(floor - 0.001 * (i + 1), 0.0), 4),
+ "sources": ["symbol"],
+ }
+ for i, p in enumerate(extra)
+ ]
+ head = output[: max(0, limit - len(additions))]
+ return head + additions + output[len(head) :]
+
+
def _fetch_limit_for(limit: int, kind: str | None) -> int:
"""Over-fetch headroom for post-filters and decision down-weighting.
@@ -401,6 +461,28 @@ def _filter_by_kind(output: list[dict], kind: str | None) -> list[dict]:
]
+def _attach_paths(output: list[dict], page_info: dict) -> None:
+ """Stamp ``target_path``, and ``file`` wherever the page id is not one.
+
+ A ``symbol_spotlight``'s target_path is ``file.py::Symbol``: a page id, and
+ not something a consumer can open. ``target_path`` is left exactly as it is
+ because callers pipe it into get_symbol, and ``file`` is added beside it so
+ nothing downstream has to know to split on ``::``.
+
+ This lives in one function because it did not used to. The concept branch
+ of ``search_codebase`` set ``file``; ``_search_single_repo``, which is what
+ the hybrid and federated branches call, did not. A query carrying a
+ CamelCase token routes to hybrid, so the tool's headline path kept serving
+ page ids in a field read as a path. Measured on the dev-fix1 predictions:
+ 45.4% of the python served paths still carried ``::``, and resolving them
+ takes gold containment from 19 to 39 of 50 instances.
+ """
+ for item in output:
+ item["target_path"] = page_info.get(item["page_id"], "")
+ if "::" in item["target_path"]:
+ item["file"] = item["target_path"].split("::", 1)[0]
+
+
async def _load_page_info(
session, output: list[dict], *, with_git: bool = False
) -> tuple[dict, set, dict]:
@@ -584,8 +666,7 @@ async def _search_single_repo(
async with get_session(ctx.session_factory) as session:
page_info, tombstoned, _ = await _load_page_info(session, output)
output = [item for item in output if item["page_id"] not in tombstoned]
- for item in output:
- item["target_path"] = page_info.get(item["page_id"], "")
+ _attach_paths(output, page_info)
output = filter_dicts_by_key(output, "target_path", _get_exclude_spec(ctx.path))
_downweight_test_pages(output, query)
@@ -617,20 +698,31 @@ async def _federated_search(
# Derive confidence from RRF position
_assign_confidence(output, "rrf_score", "confidence_score")
- return {"results": output, "_meta": _build_meta()}
+ response: dict = {"results": output, "_meta": _build_meta()}
+ # Drawn from every repo's ranked list, not the merged cut: a workspace
+ # search that spends its window on module pages should still be able to
+ # name files.
+ if candidates := file_candidates(all_results, limit=limit):
+ response["candidates"] = candidates
+ return response
def _result_paths(results: list[dict]) -> list[str]:
"""File paths a result set serves, for target-scoped freshness.
- Symbol hits carry ``file``; concept/page hits carry ``target_path``.
+ Symbol hits carry ``file``; page hits are resolved through their page type,
+ so a module page's group key and an onboarding slot resolve to nothing
+ rather than being handed on as if they were files. Freshness is per-file
+ git metadata, and a directory has none, so a page id reaching here could
+ only ever fail to match.
+
An empty result set returns ``[]`` (meaning "no file content served",
which suppresses the repo-level stale warning by design).
"""
paths: list[str] = []
for item in results:
- p = item.get("file") or item.get("target_path")
- if isinstance(p, str) and p:
+ p = hit_file_path(item)
+ if p:
paths.append(p)
return paths
@@ -786,6 +878,12 @@ async def _structured_search(
"mode": mode,
"_meta": _build_meta(repository=repository, targets=_result_paths(results)),
}
+ # Symbols first, then everything else the window holds: in symbol and
+ # hybrid modes the ranked pool leads with symbol hits, and those are the
+ # entries most likely to collapse onto one another (several symbols of one
+ # file). Deduping them is the point.
+ if candidates := file_candidates(results, limit=limit):
+ response["candidates"] = candidates
# Exact-match honesty: an identifier-shaped query whose target names no
# indexed symbol still returns fuzzy neighbours. Say so, or the agent
# anchors on a wrong hit that looks authoritative (their Alamofire
@@ -831,9 +929,11 @@ async def search_codebase(
the indexed symbols (returns symbol_id/file/line bounds — pipe into
get_symbol), path-shaped queries resolve files (pipe into get_context),
and conceptual queries run wiki-semantic search. Mixed queries run hybrid,
- symbol hits first. Concept hits carry a sources list (fts, vector, or
- both); decision records rank below file pages unless the query is
- why-shaped. See docs/agent/MCP_TOOLS.md for detail.
+ symbol hits first. Decision records rank below file pages unless the query
+ is why-shaped.
+
+ `candidates` lists up to `limit` distinct openable file paths, best first.
+ Some results are pages, not files; this is what to Read.
Args:
query: identifier, path, or natural-language query.
@@ -889,9 +989,9 @@ async def search_codebase(
output = [item for item in output if item["page_id"] not in tombstoned]
# Attach target_path to each item so the kind filter (path-prefix
- # heuristic) and downstream get_context callers can act on it.
- for item in output:
- item["target_path"] = page_info.get(item["page_id"], "")
+ # heuristic) and downstream get_context callers can act on it, and
+ # ``file`` wherever the page id is symbol-qualified.
+ _attach_paths(output, page_info)
output = filter_dicts_by_key(output, "target_path", _get_exclude_spec(ctx.path))
@@ -909,6 +1009,14 @@ async def search_codebase(
output = _dedup_decisions(output)
output = _filter_by_kind(output, kind)
+ # Files the page retrievers structurally cannot see (a private helper, a
+ # local name, anything a file page's public-symbol table omits) get the
+ # weakest tail slots. No-op when the symbol leg names nothing new.
+ with contextlib.suppress(Exception):
+ output = await _append_symbol_backed(ctx, query, output, limit, kind)
+ # The full ranked pool, kept before the caller's cut so ``candidates``
+ # below can reach past it. See its comment for why that matters.
+ ranked = list(output)
output = output[:limit]
# Derive confidence_score from relative position in the result set.
@@ -918,6 +1026,8 @@ async def search_codebase(
"results": output,
"_meta": _build_meta(repository=repository, targets=_result_paths(output)),
}
+ if candidates := file_candidates(ranked, limit=limit):
+ response["candidates"] = candidates
if grep_hint and not output:
response["grep_hint"] = grep_hint
return response
diff --git a/tests/unit/generation/test_code_vocabulary.py b/tests/unit/generation/test_code_vocabulary.py
new file mode 100644
index 000000000..fb9afea4f
--- /dev/null
+++ b/tests/unit/generation/test_code_vocabulary.py
@@ -0,0 +1,325 @@
+"""The file's own vocabulary, and the round trip that makes it worth having.
+
+A file page renders an overview, a symbol table and a dependency list. None of
+those carry the words a question is actually asked in: a flag name, an error
+string, a struct field, the phrasing of a doc comment. So a page can describe
+a file accurately and remain unreachable by any question about what it does.
+
+Measured on the cli/cli and django canary indexes, gold file in the top ten of
+a full-text search over page text: **Go 4 to 12 of 20 instances, python 34 to
+38 of 50**. The same probe against an index shaped like a symbol-only
+competitor scored 8 of 20 on Go and *worse than the current pages* on python,
+which is why this ships as page content and not as a second symbol index.
+
+Two properties carry the whole result and each has a test below:
+
+* it is **per-file**, so it discriminates. Repo-level vocabulary rendered here
+ would be byte-identical on every page in the corpus.
+* it reaches the **full-text row**, not only the rendered page. The write path
+ hands ``page.content`` to the indexer verbatim, so nothing would raise if the
+ section were dropped or moved into metadata.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import jinja2
+import pytest
+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
+from sqlalchemy.pool import StaticPool
+
+from repowise.core.generation.context.code_vocabulary import code_vocabulary
+from repowise.core.generation.context_assembler import FilePageContext
+from repowise.core.generation.page_generator.structural import (
+ as_markdown,
+ oneline,
+ signature,
+)
+from repowise.core.persistence.crud import upsert_page, upsert_repository
+from repowise.core.persistence.database import init_db
+from repowise.core.persistence.search import FullTextSearch
+
+VOCAB_HEADING = "## In the code"
+
+# The real thing, trimmed. This is the file behind the ContextBench instance
+# the diagnosis was built on: an `--order` flag on `gh release list`. Its
+# generated page carried a path header, two symbol names and 29 dependency
+# paths, and not one of the tokens below.
+GO_SOURCE = '''
+package list
+
+// ListOptions carries the flags for the release list command.
+type ListOptions struct {
+ Limit int
+ ExcludePreReleases bool
+ Order string
+}
+
+func NewCmdList(f *cmdutil.Factory) *cobra.Command {
+ cmd.Flags().StringVar(&opts.Order, "order", "desc", "Order of releases returned")
+ return cmd
+}
+
+func listRun(opts *ListOptions) error {
+ query := `orderBy: {field: CREATED_AT, direction: DESC}`
+ return nil
+}
+'''
+
+
+class TestWhatItLifts:
+ def test_declared_field_names_survive(self):
+ out = code_vocabulary(GO_SOURCE)
+ assert "Order" in out
+ assert "ExcludePreReleases" in out
+
+ def test_string_literals_survive(self):
+ """Flag names and help text: the wording a bug report repeats back."""
+ out = code_vocabulary(GO_SOURCE)
+ assert "Order of releases returned" in out
+ assert "desc" in out
+
+ def test_a_literal_that_repeats_an_identifier_is_not_stored_twice(self):
+ """Dedup is case-folded, and that is the right call for an index.
+
+ The flag string ``"order"`` and the field name ``Order`` are one token
+ to a full-text index, so carrying both would spend the cap to buy
+ nothing. The bag keeps whichever came first.
+
+ Deduplication is per entry, not per word: the help text
+ ``"Order of releases returned"`` is its own entry and keeps its own
+ copy of the word, which is correct, because the phrase is what a bug
+ report quotes back.
+ """
+ out = code_vocabulary(' Order string\nvar f = "order"\n')
+ assert out.lower().split().count("order") == 1
+
+ def test_camel_case_is_also_offered_as_words(self):
+ """A question is asked in words, not in camel case.
+
+ ``ExcludePreReleases`` is kept whole *and* split, because a reader asks
+ "how do I exclude pre-releases" and never types the identifier.
+ """
+ out = code_vocabulary(GO_SOURCE).lower()
+ for word in ("exclude", "pre", "releases"):
+ assert word in out.split() or word in out
+
+ def test_comment_prose_survives(self):
+ assert "ListOptions carries the flags" in code_vocabulary(GO_SOURCE)
+
+ def test_python_attributes_are_lifted(self):
+ source = "class Config:\n def __init__(self):\n self.retry_budget = 3\n"
+ out = code_vocabulary(source)
+ assert "retry_budget" in out
+
+ def test_the_tokens_the_gold_instance_needed_all_arrive(self):
+ """The end-to-end claim, stated as one assertion.
+
+ Every one of these appears in the gold source and none of them reached
+ the page before this section existed. Compared case-insensitively
+ because that is how the full-text index compares them: ``CREATED_AT``
+ arrives as ``created``, which answers "ordered by creation date"
+ exactly as well.
+ """
+ out = code_vocabulary(GO_SOURCE).lower()
+ for token in ("order", "desc", "created", "releases"):
+ assert token in out, f"{token!r} missing"
+
+
+class TestBounds:
+ def test_capped(self):
+ source = "\n".join(f' Field{i} string // comment number {i}' for i in range(4000))
+ assert len(code_vocabulary(source)) <= 1200
+
+ def test_the_cap_cuts_the_least_specific_material(self):
+ """Ordering is the reason the cap is affordable.
+
+ Declared names and literals go in before loose identifiers, so a
+ truncated bag keeps the distinguishing half.
+ """
+ source = GO_SOURCE + "\n" + "\n".join(f"var filler{i} int" for i in range(3000))
+ out = code_vocabulary(source)
+ assert "Order of releases returned" in out
+ assert "filler2999" not in out
+
+ def test_empty_source_yields_nothing_rather_than_a_heading(self):
+ assert code_vocabulary("") == ""
+
+ def test_single_and_double_character_names_are_not_collected(self):
+ """Receivers and loop variables are never what a question is about."""
+ out = code_vocabulary("func (c *Client) do(i int) { x := 1; _ = x }")
+ assert " i " not in f" {out} "
+ assert " x " not in f" {out} "
+
+ def test_long_blobs_do_not_eat_the_cap(self):
+ """Minified data and embedded SQL are long and unsearchable."""
+ blob = "A" * 900
+ out = code_vocabulary(f' Name string\nvar data = "{blob}"')
+ assert blob not in out
+ assert "Name" in out
+
+
+def jinja_env() -> jinja2.Environment:
+ templates_dir = (
+ Path(__file__).parents[3]
+ / "packages"
+ / "core"
+ / "src"
+ / "repowise"
+ / "core"
+ / "generation"
+ / "templates"
+ )
+ env = jinja2.Environment(
+ loader=jinja2.FileSystemLoader(str(templates_dir)),
+ undefined=jinja2.StrictUndefined,
+ autoescape=False,
+ )
+ env.filters.setdefault("oneline", oneline)
+ env.filters.setdefault("as_markdown", as_markdown)
+ env.filters.setdefault("signature", signature)
+ return env
+
+
+def _file_page(**overrides) -> FilePageContext:
+ base = dict(
+ file_path="pkg/cmd/release/list/list.go",
+ language="go",
+ docstring=None,
+ symbols=[
+ {
+ "name": "NewCmdList",
+ "kind": "function",
+ "signature": "func NewCmdList(f *cmdutil.Factory) *cobra.Command",
+ "docstring": None,
+ "visibility": "public",
+ "is_async": False,
+ "complexity_estimate": 1,
+ "decorators": [],
+ "parent_name": None,
+ "start_line": 1,
+ "end_line": 10,
+ }
+ ],
+ imports=[],
+ exports=["NewCmdList"],
+ file_source_snippet="",
+ pagerank_score=0.5,
+ betweenness_score=0.1,
+ community_id=0,
+ dependents=[],
+ dependencies=["api/client.go"],
+ is_api_contract=False,
+ is_entry_point=False,
+ is_test=False,
+ parse_errors=[],
+ estimated_tokens=50,
+ )
+ base.update(overrides)
+ return FilePageContext(**base)
+
+
+class TestRendering:
+ def test_the_section_renders_when_there_is_vocabulary(self, jinja_env):
+ ctx = _file_page(code_vocabulary=code_vocabulary(GO_SOURCE))
+ page = jinja_env.get_template("file_page.j2").render(ctx=ctx)
+ assert VOCAB_HEADING in page
+ assert "Order of releases returned" in page
+
+ def test_no_heading_when_the_file_yields_nothing(self, jinja_env):
+ """Every section below the Overview is conditional, and this is why.
+
+ An empty heading puts the same stock line into the index on thousands
+ of pages, where it matches every query and distinguishes none.
+ """
+ page = jinja_env.get_template("file_page.j2").render(ctx=_file_page(code_vocabulary=""))
+ assert VOCAB_HEADING not in page
+
+ def test_it_sits_below_the_questions_block(self, jinja_env):
+ """``_extract_summary`` reads back from the top. A bag of words is not
+ a summary, so it must not be the first thing the page says."""
+ ctx = _file_page(code_vocabulary=code_vocabulary(GO_SOURCE))
+ page = jinja_env.get_template("file_page.j2").render(ctx=ctx)
+ assert page.index("## Overview") < page.index(VOCAB_HEADING)
+ assert page.index("## Questions this page answers") < page.index(VOCAB_HEADING)
+
+ def test_two_files_get_different_vocabulary(self, jinja_env):
+ """The property the whole result rests on.
+
+ Repo-level vocabulary would render byte-identically on all ~700 file
+ pages of cli/cli and carry zero discriminative power. This is the
+ opposite granularity, deliberately.
+ """
+ tmpl = jinja_env.get_template("file_page.j2")
+ a = tmpl.render(ctx=_file_page(code_vocabulary=code_vocabulary(GO_SOURCE)))
+ b = tmpl.render(
+ ctx=_file_page(
+ file_path="api/client.go",
+ code_vocabulary=code_vocabulary('type Client struct {\n\tHTTP string\n}'),
+ )
+ )
+ assert "Order of releases returned" in a
+ assert "Order of releases returned" not in b
+
+
+async def fts():
+ engine = create_async_engine(
+ "sqlite+aiosqlite:///:memory:",
+ connect_args={"check_same_thread": False},
+ poolclass=StaticPool,
+ )
+ await init_db(engine)
+ search = FullTextSearch(engine)
+ await search.ensure_index()
+ yield engine, search
+ await engine.dispose()
+
+
+async def _index(engine, search, *, page_id, target_path, content):
+ factory = async_sessionmaker(engine, expire_on_commit=False)
+ async with factory() as session:
+ repo = await upsert_repository(session, name="r", local_path="/tmp/r")
+ await session.commit()
+ await upsert_page(
+ session,
+ page_id=page_id,
+ repository_id=repo.id,
+ page_type="file_page",
+ title=target_path,
+ content=content,
+ summary="",
+ target_path=target_path,
+ source_hash="h",
+ model_name="template",
+ provider_name="template",
+ )
+ await session.commit()
+ await search.index(page_id, target_path, content, summary="", target_path=target_path)
+
+
+async def test_the_words_reach_the_full_text_row(fts, jinja_env):
+ """The round trip, not the render.
+
+ ``FullTextSearch.index`` is handed ``page.content`` verbatim, so the
+ section could be dropped, moved into metadata, or added after the row was
+ written and nothing would raise. This is the assertion that the page is
+ actually findable by the words its own file contains, which is the entire
+ claim.
+ """
+ engine, search = fts
+ ctx = _file_page(code_vocabulary=code_vocabulary(GO_SOURCE))
+ content = jinja_env.get_template("file_page.j2").render(ctx=ctx)
+ await _index(
+ engine,
+ search,
+ page_id="file_page:pkg/cmd/release/list/list.go",
+ target_path="pkg/cmd/release/list/list.go",
+ content=content,
+ )
+
+ hits = await search.search("order of releases returned", limit=10)
+ assert [h.target_path for h in hits] == ["pkg/cmd/release/list/list.go"]
diff --git a/tests/unit/server/mcp/test_answer_candidates.py b/tests/unit/server/mcp/test_answer_candidates.py
new file mode 100644
index 000000000..45ff85568
--- /dev/null
+++ b/tests/unit/server/mcp/test_answer_candidates.py
@@ -0,0 +1,94 @@
+"""``candidates``: the navigation block get_answer returns unconditionally.
+
+``retrieval`` is evidence and shrinks as confidence rises, which left a
+high-confidence answer naming no file at all (finding A10). ``candidates`` is
+the other job (where to look next), so it is not confidence-gated and it is
+built from the pool before the 5-hit synthesis cap.
+"""
+
+from __future__ import annotations
+
+from repowise.server.mcp_server.tool_answer.retrieval import serialize_candidates
+
+
+def _hit(path, symbols=None):
+ h = {"target_path": path}
+ if symbols is not None:
+ h["symbols"] = symbols
+ return h
+
+
+def test_one_entry_per_file_in_rank_order():
+ out = serialize_candidates([_hit("a.py"), _hit("b.py"), _hit("c.py")])
+ assert [e["path"] for e in out] == ["a.py", "b.py", "c.py"]
+
+
+def test_line_bounds_come_from_hydrated_symbols_only():
+ hits = [
+ _hit("a.py", [{"start_line": 10, "end_line": 40}, {"start_line": 60, "end_line": 88}]),
+ _hit("b.py"),
+ ]
+ out = serialize_candidates(hits)
+ assert out[0] == {"path": "a.py", "lines": "10-88"}
+ assert out[1] == {"path": "b.py"} # nothing is fetched to fill this in
+
+
+def test_duplicate_paths_collapse():
+ out = serialize_candidates([_hit("a.py"), _hit("a.py"), _hit("b.py")])
+ assert [e["path"] for e in out] == ["a.py", "b.py"]
+
+
+def test_pathless_hits_are_skipped():
+ out = serialize_candidates([{"target_path": ""}, {"title": "no path"}, _hit("a.py")])
+ assert [e["path"] for e in out] == ["a.py"]
+
+
+def test_capped():
+ out = serialize_candidates([_hit(f"f{i}.py") for i in range(50)], limit=20)
+ assert len(out) == 20
+
+
+def test_empty_pool_yields_no_block():
+ assert serialize_candidates([]) == []
+
+
+def test_a_symbol_page_contributes_its_file_not_its_page_id():
+ """``file.py::Symbol`` is a page identifier; a candidate must be openable.
+
+ Measured cost of getting this wrong: on the rung 8 ContextBench dev split,
+ 39% of the `repowise-search` arm's served paths carried `::`, and 22 of 70
+ instances surfaced a gold file only once the suffix was stripped.
+ """
+ out = serialize_candidates([_hit("django/db/models/base.py::Model")])
+ assert out == [{"path": "django/db/models/base.py"}]
+
+
+def test_two_symbols_in_one_file_are_one_candidate():
+ out = serialize_candidates([_hit("a.py::One"), _hit("a.py::Two"), _hit("a.py")])
+ assert [e["path"] for e in out] == ["a.py"]
+
+
+def test_a_page_that_names_no_file_is_not_offered_as_one():
+ """Finding A15, on the answer arm.
+
+ A module page's target_path is a structural group key and reads exactly
+ like a directory, so an agent told to open it fails and nothing in the
+ response says the string was never a path.
+ """
+ hits = [
+ {"page_type": "module_page", "target_path": "pkg/cmd/release"},
+ {"page_type": "onboarding", "target_path": "onboarding/guided_tour"},
+ {"page_type": "scc_page", "target_path": "scc-8f21ab"},
+ {"page_type": "file_page", "target_path": "pkg/cmd/release/list.go"},
+ ]
+ assert serialize_candidates(hits) == [{"path": "pkg/cmd/release/list.go"}]
+
+
+def test_an_unhydrated_hit_is_kept_rather_than_dropped():
+ """A missing page type is a bookkeeping gap, not a verdict.
+
+ ``hydrate_hits`` fills page_type from the Page table; a hit whose row is
+ missing gets an empty one. Treating that like a classified non-file page
+ would lose a real file over a join that did not land.
+ """
+ assert serialize_candidates([_hit("a.py")]) == [{"path": "a.py"}]
diff --git a/tests/unit/server/mcp/test_neighbor_rerank.py b/tests/unit/server/mcp/test_neighbor_rerank.py
index 3f488b81c..61b3861c6 100644
--- a/tests/unit/server/mcp/test_neighbor_rerank.py
+++ b/tests/unit/server/mcp/test_neighbor_rerank.py
@@ -172,6 +172,13 @@ async def test_head_is_never_reordered(session, repo_id):
async def test_non_flow_question_is_noop(session, repo_id):
+ """The gate stays until injection respects page type (see module note).
+
+ Removing it was measured on 2026-08-02 and cost 6.4 points of recall@5:
+ fabricated **file** neighbours evict the **module** page that is the answer
+ to a subsystem-shaped question, because the stage re-ranks every slot below
+ the protected head.
+ """
hits = _hits([(p, 5.0 - i) for i, p in enumerate(_SEEDS)])
ctx = _Ctx([_FAR, *_SEEDS])
out = await expand_via_neighbor_rerank(session, repo_id, hits, "where is s0 defined", ctx)
diff --git a/tests/unit/server/mcp/test_prose_symbols.py b/tests/unit/server/mcp/test_prose_symbols.py
new file mode 100644
index 000000000..af73e823c
--- /dev/null
+++ b/tests/unit/server/mcp/test_prose_symbols.py
@@ -0,0 +1,86 @@
+"""Unit tests for the prose-keyed symbol leg (``_prose_symbols``).
+
+The behaviour under test is that a question asked in English reaches the same
+symbol rows as the same question typed as an identifier. These are pure-function
+tests over term extraction and the corroboration guard; the SQL side is covered
+by the retrieval eval rather than here.
+"""
+
+from __future__ import annotations
+
+from repowise.server.mcp_server._prose_symbols import (
+ _corroborated,
+ content_terms,
+)
+
+
+class _Sym:
+ """The two ``WikiSymbol`` fields the guard reads."""
+
+ def __init__(self, name, file_path="src/pipeline/incremental.py", language="python"):
+ self.name = name
+ self.file_path = file_path
+ self.language = language
+
+
+def test_prose_question_yields_the_words_a_symbol_is_built_from():
+ terms = content_terms("how does an incremental update persist symbols for changed files")
+ assert "persist" in terms
+ assert "symbols" in terms
+ assert "incremental" in terms
+
+
+def test_stopwords_and_short_tokens_are_dropped():
+ terms = content_terms("how does the a of it be")
+ assert terms == []
+
+
+def test_snake_case_contributes_both_the_whole_token_and_its_parts():
+ terms = content_terms("where is _persist_symbols called")
+ assert "_persist_symbols" in terms
+ assert "persist" in terms
+ assert "symbols" in terms
+
+
+def test_camel_case_is_split_into_its_humps():
+ terms = content_terms("what does HTTPServerPool do")
+ assert "server" in terms
+ assert "pool" in terms
+
+
+def test_terms_are_capped_so_a_pasted_bug_report_is_not_fifty_scans():
+ question = " ".join(f"distinctword{i}" for i in range(40))
+ assert len(content_terms(question)) <= 8
+
+
+def test_one_common_word_does_not_seed_a_match():
+ # "update" alone must not drag in every symbol named update-something.
+ assert not _corroborated(_Sym("update_index"), {"update": 1.0}, saturated={"update"})
+
+
+def test_two_distinct_terms_corroborate_each_other():
+ assert _corroborated(
+ _Sym("_persist_symbols"), {"persist": 1.0, "symbols": 1.0}, saturated=set()
+ )
+
+
+def test_two_saturated_terms_are_the_same_non_signal_twice():
+ # Both words match most of the corpus, so their agreeing says nothing.
+ assert not _corroborated(_Sym("cn"), {"get": 0.25, "file": 0.25}, saturated={"get", "file"})
+
+
+def test_one_informative_term_plus_one_saturated_is_not_enough():
+ assert not _corroborated(_Sym("build_config"), {"config": 1.0, "get": 0.25}, saturated={"get"})
+
+
+def test_a_single_term_that_is_the_whole_name_is_enough():
+ # The caller named the thing; the only difference is the underscores.
+ assert _corroborated(_Sym("persist"), {"persist": 1.0}, saturated=set())
+
+
+def test_a_saturated_term_cannot_carry_a_leaf_name_match_alone():
+ assert not _corroborated(_Sym("get"), {"get": 0.25}, saturated={"get"})
+
+
+def test_no_matched_terms_is_never_corroborated():
+ assert not _corroborated(_Sym("_persist_symbols"), {}, saturated=set())
diff --git a/tests/unit/server/mcp/test_retrieval_leg_visibility.py b/tests/unit/server/mcp/test_retrieval_leg_visibility.py
new file mode 100644
index 000000000..68bdc4667
--- /dev/null
+++ b/tests/unit/server/mcp/test_retrieval_leg_visibility.py
@@ -0,0 +1,129 @@
+"""A retrieval leg that did not run has to say so (finding A18).
+
+Every leg is best-effort with a timeout, deliberately: a slow vector store must
+not be able to block an answer. The defect was that the fallback was silent.
+An embed that misses its 8s budget returns no vector, retrieval continues
+lexical-only, and every health signal still reports green. ``embedder_live``
+in particular stays true, correctly, because a configured embedder *is* live
+whether or not one call beat the clock. Those are different claims and only
+one of them is about the answer the caller is holding.
+
+Measured: five queries in one bake-off run were answered without their vector
+leg. Nothing in the response, the logs, or the per-cell record said so, and the
+cells were written as clean.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+
+from repowise.server.mcp_server import _answer_pipeline as pipeline
+
+
+class _Store:
+ """A vector store whose embed call never returns in time."""
+
+ def __init__(self, *, embed_hangs=False, search_hangs=False):
+ self._embed_hangs = embed_hangs
+ self._search_hangs = search_hangs
+
+ async def embed_texts(self, texts):
+ if self._embed_hangs:
+ await asyncio.sleep(60)
+ return [[0.1, 0.2, 0.3]]
+
+ async def search_by_vector(self, vector, limit):
+ return None
+
+ async def search(self, query, limit):
+ if self._search_hangs:
+ await asyncio.sleep(60)
+ return []
+
+
+class _Fts:
+ def __init__(self, *, hangs=False):
+ self._hangs = hangs
+
+ async def search(self, query, limit):
+ if self._hangs:
+ await asyncio.sleep(60)
+ return []
+
+
+class _Ctx:
+ def __init__(self, *, store=None, fts=None):
+ self.vector_store = store
+ self.fts = fts
+ self.vector_store_ready = None
+ self.session_factory = None
+ self.path = "."
+
+
[email protected](autouse=True)
+def fresh_record():
+ pipeline.begin_leg_record()
+ yield
+
+
+async def test_a_healthy_leg_records_ok(monkeypatch):
+ monkeypatch.setattr(pipeline, "_EMBED_TIMEOUT_S", 0.5)
+ await pipeline._safe_vector_search(_Ctx(store=_Store()), "how does retrieval work")
+ assert pipeline.retrieval_legs()["vector"] == "ok"
+ assert pipeline.degraded_legs(pipeline.retrieval_legs()) == []
+
+
+async def test_an_embed_timeout_is_named_and_does_not_fail_the_call(monkeypatch):
+ """The exact A18 shape: the answer still comes back, lexical-only."""
+ monkeypatch.setattr(pipeline, "_EMBED_TIMEOUT_S", 0.05)
+ ctx = _Ctx(store=_Store(embed_hangs=True))
+
+ results = await pipeline._safe_vector_search(ctx, "how does retrieval work")
+
+ assert results == [] # the leg degraded rather than raising
+ legs = pipeline.retrieval_legs()
+ assert legs["embed"] == "timeout"
+ assert "embed" in pipeline.degraded_legs(legs)
+
+
+async def test_a_timeout_is_distinguished_from_an_error():
+ """They want different responses. A timeout is a budget problem and may
+ pass on a retry; an error is a broken backend and will not."""
+ ctx = _Ctx(store=_Store(search_hangs=True))
+ await pipeline._safe_vector_search(ctx, "q")
+ assert pipeline.retrieval_legs()["vector"] == "timeout"
+
+
+async def test_an_absent_backend_is_recorded_rather_than_passed_over():
+ await pipeline._safe_vector_search(_Ctx(store=None), "q")
+ await pipeline._safe_fts_search(_Ctx(fts=None), "q")
+ legs = pipeline.retrieval_legs()
+ assert legs["vector"] == "absent"
+ assert legs["fts"] == "absent"
+
+
+async def test_a_slow_full_text_leg_is_named_too():
+ await pipeline._safe_fts_search(_Ctx(fts=_Fts(hangs=True)), "q")
+ assert pipeline.retrieval_legs()["fts"] == "timeout"
+
+
+def test_degraded_legs_names_only_what_broke():
+ legs = {"fts": "ok", "vector": "timeout", "symbol": "ok", "embed": "error"}
+ assert pipeline.degraded_legs(legs) == ["embed", "vector"]
+
+
+async def test_the_record_does_not_leak_between_questions():
+ """A stale record would report a previous question's failure against this
+ one, which is worse than reporting nothing."""
+ await pipeline._safe_vector_search(_Ctx(store=None), "q")
+ assert pipeline.retrieval_legs()["vector"] == "absent"
+ pipeline.begin_leg_record()
+ assert pipeline.retrieval_legs() == {}
diff --git a/tests/unit/server/mcp/test_search.py b/tests/unit/server/mcp/test_search.py
index b2513160f..7919b9fec 100644
--- a/tests/unit/server/mcp/test_search.py
+++ b/tests/unit/server/mcp/test_search.py
@@ -986,3 +986,112 @@ class TestFusion:
result = await search_codebase("session cache layer", limit=10)
by_path = {r["target_path"]: r for r in result["results"]}
assert by_path["src/fts_rescue.py"]["sources"] == ["fts"]
+
+
+class TestSearchCandidates:
+ """`candidates`: every slot the caller paid for resolves to an openable file.
+
+ The unit-level rules live in ``test_search_candidates.py``. These drive the
+ real tool, because the defect this fixes was not in the path logic — it was
+ in which branch of ``search_codebase`` ever called it.
+ """
+
+ @pytest.mark.asyncio
+ async def test_pages_that_are_not_files_do_not_cost_the_caller_a_slot(self, setup_mcp):
+ """A module page and an onboarding page rank, and candidates still names
+ two files, reached from below the caller's cut."""
+ import repowise.server.mcp_server as mcp_mod
+ from repowise.server.mcp_server import search_codebase
+
+ await _seed_page("module_page:pkg/cmd/release", "pkg/cmd/release", "module_page")
+ await _seed_page(
+ "onboarding:onboarding/guided_tour", "onboarding/guided_tour", "onboarding"
+ )
+ await _seed_page("file_page:pkg/cmd/release/list.go", "pkg/cmd/release/list.go")
+ await _seed_page("file_page:pkg/cmd/release/http.go", "pkg/cmd/release/http.go")
+
+ async def fake_search(query, limit=10):
+ return [
+ _mk_result(
+ "module_page:pkg/cmd/release", "Release", "module_page", "pkg/cmd/release", 0.90
+ ),
+ _mk_result(
+ "onboarding:onboarding/guided_tour",
+ "Guided Tour",
+ "onboarding",
+ "onboarding/guided_tour",
+ 0.80,
+ ),
+ _mk_result(
+ "file_page:pkg/cmd/release/list.go",
+ "list.go",
+ "file_page",
+ "pkg/cmd/release/list.go",
+ 0.70,
+ ),
+ _mk_result(
+ "file_page:pkg/cmd/release/http.go",
+ "http.go",
+ "file_page",
+ "pkg/cmd/release/http.go",
+ 0.60,
+ ),
+ ]
+
+ mcp_mod._vector_store.search = fake_search
+ result = await search_codebase("listing releases in order", limit=10)
+
+ # The module page is still a legitimate ranked result.
+ assert "module_page" in [r["page_type"] for r in result["results"]]
+ # ...and candidates names only things that can be opened.
+ assert result["candidates"] == [
+ {"path": "pkg/cmd/release/list.go"},
+ {"path": "pkg/cmd/release/http.go"},
+ ]
+
+ @pytest.mark.asyncio
+ async def test_a_symbol_page_is_served_as_its_file(self, setup_mcp):
+ """The half of A14 that never reached this branch.
+
+ ``target_path`` keeps the page id, because callers pipe it into
+ get_symbol. ``file`` and ``candidates`` carry the openable path.
+ """
+ import repowise.server.mcp_server as mcp_mod
+ from repowise.server.mcp_server import search_codebase
+
+ await _seed_page(
+ "symbol_spotlight:api/client.go::HTTP", "api/client.go::HTTP", "symbol_spotlight"
+ )
+
+ async def fake_search(query, limit=10):
+ return [
+ _mk_result(
+ "symbol_spotlight:api/client.go::HTTP",
+ "HTTP",
+ "symbol_spotlight",
+ "api/client.go::HTTP",
+ 0.90,
+ )
+ ]
+
+ mcp_mod._vector_store.search = fake_search
+ result = await search_codebase("how are requests issued", limit=10)
+
+ hit = result["results"][0]
+ assert hit["target_path"] == "api/client.go::HTTP"
+ assert hit["file"] == "api/client.go"
+ assert result["candidates"] == [{"path": "api/client.go"}]
+
+ @pytest.mark.asyncio
+ async def test_no_block_when_nothing_openable_was_reached(self, setup_mcp):
+ import repowise.server.mcp_server as mcp_mod
+ from repowise.server.mcp_server import search_codebase
+
+ await _seed_page("repo_overview:cli", "cli", "repo_overview")
+
+ async def fake_search(query, limit=10):
+ return [_mk_result("repo_overview:cli", "Overview", "repo_overview", "cli", 0.90)]
+
+ mcp_mod._vector_store.search = fake_search
+ result = await search_codebase("what is this repository", limit=10)
+ assert "candidates" not in result
diff --git a/tests/unit/server/mcp/test_search_candidates.py b/tests/unit/server/mcp/test_search_candidates.py
new file mode 100644
index 000000000..0392768dc
--- /dev/null
+++ b/tests/unit/server/mcp/test_search_candidates.py
@@ -0,0 +1,140 @@
+"""``candidates``: the files a search_codebase call actually reached.
+
+Same defect as get_answer's A10, on the other tool. A caller asking for ten
+results gets ten *pages*, and a page is not always a file: module pages are
+named by a structural group key that reads like a directory, SCC pages by
+``scc-<hash>``, onboarding pages by a slot. Serving those in a field a consumer
+opens is finding A15, and it is what made half the slots the caller paid for
+unopenable.
+
+``results`` keeps its page semantics, because ranking a module page is
+correct. ``candidates`` is the navigation half: distinct, openable, drawn from
+the full ranked pool rather than the visible window so junk in the window does
+not cost the caller a file.
+"""
+
+from __future__ import annotations
+
+from repowise.server.mcp_server._page_paths import (
+ file_candidates,
+ file_path_of,
+ hit_file_path,
+)
+
+
+def _page(page_type, target_path):
+ return {"page_type": page_type, "target_path": target_path}
+
+
+class TestFilePathOf:
+ def test_a_file_page_names_its_file(self):
+ assert file_path_of("file_page", "pkg/cmd/release/list.go") == "pkg/cmd/release/list.go"
+
+ def test_a_symbol_page_resolves_to_the_file_it_is_cut_from(self):
+ assert file_path_of("symbol_spotlight", "api/client.go::HTTP") == "api/client.go"
+
+ def test_api_contract_and_infra_pages_are_file_backed(self):
+ assert file_path_of("api_contract", "api/queries.go") == "api/queries.go"
+ assert file_path_of("infra_page", "Dockerfile") == "Dockerfile"
+
+ def test_a_module_page_names_no_file_even_though_its_key_looks_like_one(self):
+ """The trap this module exists for: ``pkg/cmd/release`` is a group key.
+
+ It is path-shaped, so every heuristic that asks "does this look like a
+ path" says yes, and an agent told to read it gets a directory.
+ """
+ assert file_path_of("module_page", "pkg/cmd/release") is None
+
+ def test_scc_overview_and_onboarding_pages_name_no_file(self):
+ assert file_path_of("scc_page", "scc-8f21ab") is None
+ assert file_path_of("repo_overview", "cli") is None
+ assert file_path_of("onboarding", "onboarding/guided_tour") is None
+ assert file_path_of("layer_page", "presentation") is None
+
+ def test_an_empty_target_path_resolves_to_nothing(self):
+ assert file_path_of("file_page", "") is None
+ assert file_path_of("file_page", None) is None
+
+ def test_an_unknown_page_type_is_refused_rather_than_guessed(self):
+ """Default deny. A page type nobody has classified is not a path."""
+ assert file_path_of("some_future_page", "looks/like/a/path.go") is None
+
+
+class TestHitFilePath:
+ def test_a_symbol_index_hit_names_its_file_directly(self):
+ assert hit_file_path({"file": "api/client.go", "symbol_id": "s1"}) == "api/client.go"
+
+ def test_page_type_wins_over_a_looser_key(self):
+ """A page hit that resolves to no file must not fall through.
+
+ Otherwise the one field that is allowed to be a page id leaks back out
+ through the field that is not.
+ """
+ hit = {"page_type": "module_page", "target_path": "pkg/cmd", "file": "pkg/cmd"}
+ assert hit_file_path(hit) is None
+
+
+class TestFileCandidates:
+ def test_pages_that_name_no_file_do_not_consume_a_slot(self):
+ hits = [
+ _page("onboarding", "onboarding/guided_tour"),
+ _page("file_page", "a.go"),
+ _page("module_page", "pkg/cmd"),
+ _page("file_page", "b.go"),
+ ]
+ assert file_candidates(hits, limit=10) == [{"path": "a.go"}, {"path": "b.go"}]
+
+ def test_symbol_pages_in_one_file_collapse_to_one_entry(self):
+ hits = [
+ _page("symbol_spotlight", "api/client.go::HTTP"),
+ _page("symbol_spotlight", "api/client.go::Do"),
+ _page("file_page", "api/client.go"),
+ _page("file_page", "api/queries.go"),
+ ]
+ assert file_candidates(hits, limit=10) == [
+ {"path": "api/client.go"},
+ {"path": "api/queries.go"},
+ ]
+
+ def test_rank_order_is_preserved(self):
+ hits = [_page("file_page", f"f{i}.go") for i in range(5)]
+ assert [e["path"] for e in file_candidates(hits, limit=10)] == [
+ "f0.go",
+ "f1.go",
+ "f2.go",
+ "f3.go",
+ "f4.go",
+ ]
+
+ def test_the_pool_backfills_what_the_window_wasted(self):
+ """The whole point of drawing from the pool and not the cut window.
+
+ Four of the first five hits name no file. Asked for three candidates,
+ the caller gets three files, reached from below where the visible
+ window ends.
+ """
+ hits = [
+ _page("module_page", "pkg/cmd"),
+ _page("file_page", "a.go"),
+ _page("scc_page", "scc-11"),
+ _page("repo_overview", "cli"),
+ _page("onboarding", "onboarding/guided_tour"),
+ _page("file_page", "b.go"),
+ _page("file_page", "c.go"),
+ ]
+ assert file_candidates(hits, limit=3) == [
+ {"path": "a.go"},
+ {"path": "b.go"},
+ {"path": "c.go"},
+ ]
+
+ def test_capped_at_the_limit_asked_for(self):
+ hits = [_page("file_page", f"f{i}.go") for i in range(40)]
+ assert len(file_candidates(hits, limit=10)) == 10
+
+ def test_no_files_reached_yields_no_block(self):
+ hits = [_page("repo_overview", "cli"), _page("onboarding", "onboarding/guided_tour")]
+ assert file_candidates(hits, limit=10) == []
+
+ def test_empty_pool_yields_no_block(self):
+ assert file_candidates([], limit=10) == []
diff --git a/tests/unit/server/mcp/test_stdio_logging.py b/tests/unit/server/mcp/test_stdio_logging.py
new file mode 100644
index 000000000..63efe319b
--- /dev/null
+++ b/tests/unit/server/mcp/test_stdio_logging.py
@@ -0,0 +1,55 @@
+"""Unit tests for A13: the stdio server must not log onto its protocol channel.
+
+On stdio, stdout carries JSON-RPC. structlog's default factory prints there,
+so every debug line emitted during ``get_answer`` synthesis arrived at the
+client as a malformed frame. These tests assert the two sinks now point at
+stderr.
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+
+import structlog
+
+from repowise.server.mcp_server._stdio_logging import route_logging_to_stderr
+
+
+def test_a_stdout_stream_handler_is_repointed_at_stderr():
+ logger = logging.getLogger("repowise.test.stdio_logging")
+ handler = logging.StreamHandler(sys.stdout)
+ logger.addHandler(handler)
+ try:
+ route_logging_to_stderr()
+ assert handler.stream is sys.stderr
+ finally:
+ logger.removeHandler(handler)
+
+
+def test_a_handler_already_on_stderr_is_left_alone():
+ logger = logging.getLogger("repowise.test.stdio_logging_stderr")
+ handler = logging.StreamHandler(sys.stderr)
+ logger.addHandler(handler)
+ try:
+ route_logging_to_stderr()
+ assert handler.stream is sys.stderr
+ finally:
+ logger.removeHandler(handler)
+
+
+def test_structlog_writes_to_stderr_not_stdout(capsys):
+ route_logging_to_stderr()
+ # A module-level logger bound before the call must still be redirected,
+ # which is what cache_logger_on_first_use=False buys.
+ structlog.get_logger("repowise.test.stdio").warning("protocol channel check")
+ captured = capsys.readouterr()
+ assert "protocol channel check" not in captured.out
+ assert "protocol channel check" in captured.err
+
+
+def test_calling_twice_is_idempotent():
+ route_logging_to_stderr()
+ route_logging_to_stderr()
+ factory = structlog.get_config()["logger_factory"]
+ assert isinstance(factory, structlog.PrintLoggerFactory)