Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
feat(wiki): keep agent material in a page digest beside the body (#2739)
Raghav Chamadiya2f76dbc3d607
diff --git a/docs/agent/MCP_TOOLS.md b/docs/agent/MCP_TOOLS.mdindex c05c211eb..0702fc5c5 100644--- a/docs/agent/MCP_TOOLS.md+++ b/docs/agent/MCP_TOOLS.md@@ -170,7 +170,7 @@ A triage card for files, modules or symbols: summary, symbols with signatures anA file's compact symbol list holds its top 15 symbols: types (classes, interfaces, structs, traits, enums, type aliases, impls, modules) first, then functions and methods, then the rest, each group by centrality. `symbols_truncated` gives the total, and `include=["symbols"]` lists them all. When the response budget trims the list further, it keeps symbols by kind and name match, not centrality. A row without `symbol_id` is `path::name`; methods and overload variants carry theirs, so pass a row's id to `get_symbol` when it has one.-`skeleton` renders a file with bodies elided: every signature, the imports, and the bodies of its most central symbols, with line ranges on every elision. `doc_drift` lists the documents that name the file and whether they carry drift. An empty `callers` or `callees` list comes with a `*_basis` saying how much of that language's calls the graph resolved; read it before concluding nothing calls a symbol.+`full_doc` returns the page as `content_md`, plus `digest_md` on pages that carry an agent digest: the questions the page answers, its concept index, public API and git signals. Both are dropped first when the response is over budget. `skeleton` renders a file with bodies elided: every signature, the imports, and the bodies of its most central symbols, with line ranges on every elision. `doc_drift` lists the documents that name the file and whether they carry drift. An empty `callers` or `callees` list comes with a `*_basis` saying how much of that language's calls the graph resolved; read it before concluding nothing calls a symbol.```get_context(targets=["src/auth/middleware.ts", "src/api/routes.ts"], include=["callers"])diff --git a/docs/layers/WIKI.md b/docs/layers/WIKI.mdindex 57c9a3ce5..c1a7b71e5 100644--- a/docs/layers/WIKI.md+++ b/docs/layers/WIKI.md@@ -262,7 +262,10 @@ Changing the language later does not retranslate existing pages. Re-runSQLite full-text and vector results through Reciprocal Rank Fusion, biased byPageRank, then expands one to two hops along the imports and projected-callsgraph for flow-shaped questions. Full-text works with no embedder; semantic- search needs one configured.+ search needs one configured. A module page's agent digest (the questions it+ answers, its concept index, public API and git signals) is stored beside the+ page body, indexed with it, and returned by `get_context` as `digest_md`+ with `include=["full_doc"]`.- **CLI**: `repowise ask`, `repowise search`, `repowise context`.- **Dashboard**: `repowise serve`, then the Docs tab.- **Files**: `repowise export --format markdown|html|json`, written todiff --git a/docs/start/DASHBOARD.md b/docs/start/DASHBOARD.mdindex 034b1bb50..340fe8535 100644--- a/docs/start/DASHBOARD.md+++ b/docs/start/DASHBOARD.md@@ -61,7 +61,11 @@ The generated wiki: a page tree on the left, the rendered page in the middlewith mermaid diagrams, code blocks that link into the file views, a confidencebadge, a freshness indicator, backlinks, and a regenerate action for a singlepage. A command palette (`⌘K` / `Ctrl+K`) jumps between pages. Reader personas-re-filter prose for the audience you pick. You can attach human notes to a page,+re-filter prose for the audience you pick. Module pages keep the material+written for search and agents (the questions a page answers, its identifiers,+public API and git signals) out of the body; the **Reference** tab above the+page shows it, and a small History card beside the page summarises owners,+hotspots and fix history. You can attach human notes to a page,and those notes survive regeneration.The sidebar auto-collapses to icons here so the page gets the width, anddiff --git a/packages/api-client/src/types/pages.ts b/packages/api-client/src/types/pages.tsindex 81bb4236f..8b6215938 100644--- a/packages/api-client/src/types/pages.ts+++ b/packages/api-client/src/types/pages.ts@@ -52,6 +52,8 @@ export interface PageSummary {export interface PageResponse extends PageSummary {content: string;+ /** Agent material kept off the page body; the reader shows it on request. */+ digest?: string;metadata: Record<string, unknown>;}diff --git a/packages/cli/src/repowise/cli/commands/context_cmd.py b/packages/cli/src/repowise/cli/commands/context_cmd.pyindex 37b40c771..fc62ad265 100644--- a/packages/cli/src/repowise/cli/commands/context_cmd.py+++ b/packages/cli/src/repowise/cli/commands/context_cmd.py@@ -54,7 +54,7 @@ _REPLACED_KEYS = frozenset(#: ``docs`` stays: for a **symbol** target the whole card lives there#: (``signature``, ``docstring``, ``used_by``, ``candidates``), and for a file#: target it holds ``symbols`` — the list a caller pipes into ``repowise-#: symbol`` — plus ``content_md``/``documentation`` when ``--include full_doc``+#: symbol``, plus ``content_md``/``digest_md``/``documentation`` when ``--include full_doc``#: asked for them. Denylisting ``docs`` wholesale projected a symbol target to#: an empty card and made ``--include full_doc`` an inert flag._LIFTED_DOC_KEYS = ("title", "summary")diff --git a/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py b/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.pyindex f45bef3f2..05c9265ed 100644--- a/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py+++ b/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py@@ -142,7 +142,7 @@ async def _all_pages_for_reconciliation(session: object, repo_id: str) -> list:was the live index.Four columns rather than whole rows: the id to match against the stores,- the content for the information floor, metadata_json for the stub+ the content and digest for the information floor, metadata_json for the stubpredicate, and target_path to tell whether the file is excluded. Nothing downstream reads any other field, and hydrating fullORM objects for every page only to discard them is what made a cap looknecessary in the first place.@@ -152,7 +152,7 @@ async def _all_pages_for_reconciliation(session: object, repo_id: str) -> list:from repowise.core.persistence.models import Pageresult = await session.execute( # type: ignore[attr-defined]- select(Page.id, Page.content, Page.metadata_json, Page.target_path).where(+ select(Page.id, Page.content, Page.digest, Page.metadata_json, Page.target_path).where(Page.repository_id == repo_id))@@ -511,7 +511,7 @@ def _run_repo_checks(indexable_ids = {p.idfor p in pages- if meets_information_floor(p.content or "")+ if meets_information_floor(p.content or "", digest=p.digest or "")and not is_excluded((p.target_path or "").split("::", 1)[0], exclude_spec)@@ -838,7 +838,7 @@ def _run_repo_checks(# ORM rows key on ``id``, not ``page_id``, so they go# to index_many as tuples rather than to index_pages.batch = [- (p.id, p.title, p.content, p.summary, p.target_path)+ (p.id, p.title, p.content, p.summary, p.target_path, p.digest)for p in rows.scalars().all()]await fts.index_many(batch)@@ -921,6 +921,7 @@ def _run_repo_checks(summary=page.summary or "",content=page.content or "",page_metadata=page.metadata_json,+ digest=page.digest or "",)if item is None:# Below the information floor, so its absencediff --git a/packages/cli/src/repowise/cli/commands/export_cmd.py b/packages/cli/src/repowise/cli/commands/export_cmd.pyindex fbd7ad272..6e4f17fc2 100644--- a/packages/cli/src/repowise/cli/commands/export_cmd.py+++ b/packages/cli/src/repowise/cli/commands/export_cmd.py@@ -299,6 +299,8 @@ def export_command("page_type": page.page_type,"title": page.title,"content": page.content,+ # Agent material kept off the body; machine consumers want it.+ "digest": page.digest,"target_path": page.target_path,# Hierarchy columns, so a JSON consumer can rebuild the# stored tree instead of re-deriving one from path strings.diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py b/packages/cli/src/repowise/cli/commands/init_cmd/persistence.pyindex fbb36a81f..7321295e3 100644--- a/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py+++ b/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py@@ -97,13 +97,21 @@ async def _index_preserved_pages(sf: Any, fts: Any, preserved_page_ids: set[str]Page.content,Page.summary,Page.target_path,+ Page.digest,).where(Page.id.in_(batch)))).all()await fts.index_many([- (page_id, title or "", content or "", summary or "", target_path or "")- for page_id, title, content, summary, target_path in rows+ (+ page_id,+ title or "",+ content or "",+ summary or "",+ target_path or "",+ digest or "",+ )+ for page_id, title, content, summary, target_path, digest in rows])except Exception as exc: # pragma: no cover - defensivediff --git a/packages/cli/src/repowise/cli/commands/reindex_cmd.py b/packages/cli/src/repowise/cli/commands/reindex_cmd.pyindex e1dda66d7..9f4313fa5 100644--- a/packages/cli/src/repowise/cli/commands/reindex_cmd.py+++ b/packages/cli/src/repowise/cli/commands/reindex_cmd.py@@ -215,6 +215,7 @@ async def _reindex(repo_path, embedder_name: str, batch_size: int) -> None:summary=page.summary or "",content=page.content or "",page_metadata=page.metadata_json,+ digest=page.digest or "",)if item is None:# Below the information floor. Not a failure — the page isdiff --git a/packages/cli/src/repowise/cli/commands/update_cmd/command.py b/packages/cli/src/repowise/cli/commands/update_cmd/command.pyindex 22d1b380c..05eb71647 100644--- a/packages/cli/src/repowise/cli/commands/update_cmd/command.py+++ b/packages/cli/src/repowise/cli/commands/update_cmd/command.py@@ -1691,6 +1691,7 @@ def run_update(summary=page.summary,content=page.content,page_metadata=page.metadata,+ digest=page.digest,))is not Nonediff --git a/packages/core/alembic/versions/0044_fts_summary_and_target_path.py b/packages/core/alembic/versions/0044_fts_summary_and_target_path.pyindex 5a7777807..5cc047969 100644--- a/packages/core/alembic/versions/0044_fts_summary_and_target_path.py+++ b/packages/core/alembic/versions/0044_fts_summary_and_target_path.py@@ -34,7 +34,13 @@ from collections.abc import Sequencefrom alembic import op-from repowise.core.persistence.search import PG_FTS_EXPRESSION+# Frozen at this revision's shape. A migration that imports the live search+# constants replays a later column set on a database that does not have it.+PG_FTS_EXPRESSION = (+ "to_tsvector('english', "+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"+)# revision identifiersrevision: str = "0044"diff --git a/packages/core/alembic/versions/0095_page_digest.py b/packages/core/alembic/versions/0095_page_digest.pynew file mode 100644index 000000000..7ec570706--- /dev/null+++ b/packages/core/alembic/versions/0095_page_digest.py@@ -0,0 +1,103 @@+"""wiki_pages: an agent digest beside the page body.++``digest`` holds what a page carries for search and agents but not for a+reader: the questions it answers, its identifiers, public API and git signals.+It used to be appended to ``content``, which the reader, full-text search,+embeddings and MCP all read as one string.++Full-text search indexes the new column, so the PostgreSQL GIN expression is+rebuilt over it and the SQLite FTS5 table is dropped and refilled. Local SQLite+stores get the column from ``init_db``'s additive reconciler and the FTS table+from ``FullTextSearch.ensure_index``; this migration covers PostgreSQL.++Revision ID: 0095+Revises: 0094+Create Date: 2026-09-30+"""++from __future__ import annotations++from collections.abc import Sequence++import sqlalchemy as sa+from alembic import op++# Frozen at this revision's shape, never imported from ``search.py``: the next+# widening must not change what this revision creates.+PG_FTS_EXPRESSION = (+ "to_tsvector('english', "+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,'') || ' ' "+ "|| COALESCE(digest,''))"+)+# SQLite: the file vocabulary column predates this revision (added at runtime+# by ``FullTextSearch.ensure_index``), so both shapes keep it.+_VOCABULARY_SQL = (+ "CASE WHEN json_valid(metadata_json) "+ "THEN COALESCE(json_extract(metadata_json, '$.file_vocabulary'), '') "+ "ELSE '' END"+)+PAGE_FTS_DDL = (+ "CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary, digest)"+)++# revision identifiers+revision: str = "0095"+down_revision: str | None = "0094"+branch_labels: str | Sequence[str] | None = None+depends_on: str | Sequence[str] | None = None++_OLD_PG_EXPRESSION = (+ "to_tsvector('english', "+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"+)++_REFILL_SQL = (+ "INSERT INTO page_fts"+ "(page_id, title, content, summary, target_path, vocabulary, digest) "+ "SELECT id, COALESCE(title,''), COALESCE(content,''), "+ " COALESCE(summary,''), COALESCE(target_path,''), "+ f" {_VOCABULARY_SQL}, COALESCE(digest,'') "+ "FROM wiki_pages"+)++_OLD_SQLITE_DDL = (+ "CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary)"+)++_OLD_REFILL_SQL = (+ "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary) "+ "SELECT id, COALESCE(title,''), COALESCE(content,''), "+ " COALESCE(summary,''), COALESCE(target_path,''), "+ f" {_VOCABULARY_SQL} "+ "FROM wiki_pages"+)++def upgrade() -> None:+ op.add_column(+ "wiki_pages",+ sa.Column("digest", sa.Text(), nullable=False, server_default=""),+ )+ dialect = op.get_bind().dialect.name+ if dialect == "postgresql":+ op.execute("DROP INDEX IF EXISTS idx_wiki_pages_fts")+ op.execute(f"CREATE INDEX idx_wiki_pages_fts ON wiki_pages USING GIN({PG_FTS_EXPRESSION})")+ elif dialect == "sqlite":+ op.execute("DROP TABLE IF EXISTS page_fts")+ op.execute(PAGE_FTS_DDL)+ op.execute(_REFILL_SQL)+++def downgrade() -> None:+ dialect = op.get_bind().dialect.name+ if dialect == "postgresql":+ op.execute("DROP INDEX IF EXISTS idx_wiki_pages_fts")+ op.execute(f"CREATE INDEX idx_wiki_pages_fts ON wiki_pages USING GIN({_OLD_PG_EXPRESSION})")+ elif dialect == "sqlite":+ op.execute("DROP TABLE IF EXISTS page_fts")+ op.execute(_OLD_SQLITE_DDL)+ op.execute(_OLD_REFILL_SQL)+ op.drop_column("wiki_pages", "digest")diff --git a/packages/core/src/repowise/core/generation/agent_digest.py b/packages/core/src/repowise/core/generation/agent_digest.pynew file mode 100644index 000000000..c5e99d9e5--- /dev/null+++ b/packages/core/src/repowise/core/generation/agent_digest.py@@ -0,0 +1,112 @@+"""The agent digest: what a page carries for search and agents, not for a reader.++A module page used to end with the questions it answers, a concept index of+its identifiers, its public API and paragraphs of git signals. All of that+helps retrieval and an agent reading ``get_context``; none of it helps a person+reading the page, who had to scroll past it. The digest keeps that material in+``wiki_pages.digest``: indexed for full-text and vector search beside the body,+returned by MCP beside ``content``, and shown in the reader only on request.++The questions are the one model-written part. The model still ends its page+with them, because they are written in a reader's words and a template cannot+produce that, and :func:`split_questions` moves them out of the body before the+page is stored. Everything else in the digest is rendered from the index.+"""++from __future__ import annotations++import re+from collections.abc import Mapping+from typing import Any++from .structural_labels import ENGLISH_LABELS, LOCALIZED_LABELS++# The questions heading in every language a page can be written in. The model+# is asked for the English heading, and a localized run may translate it.+_QUESTION_HEADINGS = frozenset(+ label.lower()+ for label in (+ ENGLISH_LABELS["questions_heading"],+ *(c["questions_heading"] for c in LOCALIZED_LABELS.values() if "questions_heading" in c),+ )+)++#: Page-metadata key for :func:`module_signals`, read by the reader's signals card.+MODULE_SIGNALS_KEY = "module_signals"++_H2 = re.compile(r"^##\s+(.+?)\s*#*\s*$")+_FENCE = re.compile(r"^\s*(```|~~~)")+++def _h2_sections(content: str) -> list[tuple[str | None, list[str]]]:+ """Split *content* into ``(heading, lines)`` runs at level-2 headings.++ The first run has no heading. A ``##`` inside a fenced block is text.+ """+ sections: list[tuple[str | None, list[str]]] = [(None, [])]+ in_fence = False+ for line in content.splitlines():+ if _FENCE.match(line):+ in_fence = not in_fence+ match = None if in_fence else _H2.match(line)+ if match:+ sections.append((match.group(1), [line]))+ else:+ sections[-1][1].append(line)+ return sections+++def split_questions(content: str) -> tuple[str, str]:+ """Return ``(body, questions)`` with the questions section cut out of *content*.++ ``questions`` is the section as written, heading included, or ``""`` when+ the page has none. Every other section stays in the body in its order.+ """+ body: list[str] = []+ questions: list[str] = []+ for heading, lines in _h2_sections(content or ""):+ if heading is not None and heading.strip().lower() in _QUESTION_HEADINGS:+ questions.extend(lines)+ else:+ body.extend(lines)+ if not questions:+ return content, ""+ return "\n".join(body).strip() + "\n", "\n".join(questions).strip()+++def rejoin_questions(body: str, digest: str) -> str:+ """The model's response a stored page was split from: *body* plus its questions.++ A page reused from a prior run is rebuilt from the stored body, so without+ this the next split would find no questions and the digest would lose them.+ """+ _, questions = split_questions(digest or "")+ if not questions:+ return body+ return f"{(body or '').rstrip()}\n\n{questions}\n"+++def module_signals(ctx: Any, git_summary: Mapping[str, Any] | None) -> dict[str, Any]:+ """The git signals a module page reports, as plain data.++ One dict feeds both the digest text an agent reads and the small signals+ card beside the page in the reader, so the two cannot disagree. Absent or+ zero signals are left out: a missing key means "nothing to report", and a+ module with no history at all gets an empty dict. ``files`` is the+ denominator the counts are read against.+ """+ signals: dict[str, Any] = {+ "hotspots": ctx.hotspot_count,+ "bus_factor_one": ctx.single_owner_files,+ "bug_fixes": ctx.bugfix_total,+ "most_fixed_file": (ctx.most_fixed_file or {}).get("path", ""),+ "stable": ctx.stable_count,+ "owners": [{"name": o["name"], "files": o["file_count"]} for o in ctx.top_owners],+ "co_changes": [{"path": c["path"], "files": c["count"]} for c in ctx.coupled_modules],+ }+ if git_summary and git_summary.get("most_active_file"):+ signals["most_active_file"] = git_summary["most_active_file"]+ signals["most_active_commits_90d"] = git_summary.get("most_active_commits_90d", 0)+ found = {key: value for key, value in signals.items() if value}+ return {"files": len(ctx.files), **found} if found else {}+diff --git a/packages/core/src/repowise/core/generation/models.py b/packages/core/src/repowise/core/generation/models.pyindex 47245a4f8..39d267798 100644--- a/packages/core/src/repowise/core/generation/models.py+++ b/packages/core/src/repowise/core/generation/models.py@@ -529,6 +529,9 @@ class GeneratedPage:# MCP get_context as the default narrative payload (content is gated behind# include=["full_doc"]).summary: str = ""+ # Agent digest: material indexed and served to agents but kept out of the+ # reader's page body. See :mod:`repowise.core.generation.agent_digest`.+ digest: str = ""# Where this page sits in the wiki. Left unset by generators that do not# place their pages; the tree builder fills them in before persistence.# See the matching columns on the Page model for what each one means.diff --git a/packages/core/src/repowise/core/generation/page_generator/core.py b/packages/core/src/repowise/core/generation/page_generator/core.pyindex ee1bf59f1..186cbf86e 100644--- a/packages/core/src/repowise/core/generation/page_generator/core.py+++ b/packages/core/src/repowise/core/generation/page_generator/core.py@@ -25,6 +25,7 @@ import structlogfrom repowise.core.ingestion.models import ParsedFile, RepoStructurefrom repowise.core.providers.llm.base import BaseProvider, CacheHint, GeneratedResponse+from ..agent_digest import rejoin_questionsfrom ..context.evidence import (EvidenceItem,EvidenceSelection,@@ -130,6 +131,7 @@ class PriorPage:source_hash: strmodel_name: strcontent: str+ digest: str = ""input_tokens: int = 0output_tokens: int = 0cached_tokens: int = 0@@ -393,7 +395,9 @@ class PageGenerator(PerTypeGenerationMixin, StructuralRenderMixin):target_path=target_path,)return GeneratedResponse(- content=prior.content,+ # The stored body lost its questions to the digest; the+ # response it was split from carried them.+ content=rejoin_questions(prior.content, prior.digest),input_tokens=0,output_tokens=0,cached_tokens=0,diff --git a/packages/core/src/repowise/core/generation/page_generator/orchestrate.py b/packages/core/src/repowise/core/generation/page_generator/orchestrate.pyindex c1c6cc896..15efe0747 100644--- a/packages/core/src/repowise/core/generation/page_generator/orchestrate.py+++ b/packages/core/src/repowise/core/generation/page_generator/orchestrate.py@@ -1076,6 +1076,7 @@ def _embed_item(page: GeneratedPage) -> tuple[str, str, dict] | None:summary=page.summary,content=page.content,page_metadata=page.metadata,+ digest=page.digest,)diff --git a/packages/core/src/repowise/core/generation/page_generator/pertype.py b/packages/core/src/repowise/core/generation/page_generator/pertype.pyindex e9e0f665f..2c348fd73 100644--- a/packages/core/src/repowise/core/generation/page_generator/pertype.py+++ b/packages/core/src/repowise/core/generation/page_generator/pertype.py@@ -8,6 +8,7 @@ each module under the project's 400-line ceiling.from __future__ import annotations+import reimport uuidfrom collections.abc import Sequencefrom dataclasses import replace@@ -18,6 +19,7 @@ import structlogfrom repowise.core.ingestion.models import ParsedFile, RepoStructurefrom .. import onboarding as _onboarding+from ..agent_digest import MODULE_SIGNALS_KEY, module_signals, split_questionsfrom ..architecture_mermaid import embed_mermaidfrom ..context.assembler import build_concept_indexfrom ..context_assembler import FilePageContext@@ -315,47 +317,51 @@ class PerTypeGenerationMixin:page.structural_key = structural_key or page.structural_keyreturn page- def _with_concept_index(page: GeneratedPage) -> GeneratedPage:- """Append the module's own identifiers to whatever wrote the page.+ def _with_digest(page: GeneratedPage) -> GeneratedPage:+ """Split what the page carries for agents off its body.- After generation rather than inside the prompt, and that placement- is the point. ``module_page.j2`` is a prompt: a table put in it is- material the model may reformat, abbreviate or drop, and a page- whose identifiers came out of a provider is exactly as trustworthy- as the prose around them. Appended here, every name and path is the- symbol index's and no response can change it.+ The identifiers, public API and git signals are rendered here from+ the index, not asked of the model: material in a prompt is+ material a model may reformat, abbreviate or drop, and appended+ after generation every name and path is the index's own. After+ ``_build_generated_page`` too, so the page summary is still drawn+ from the model's opening.- After ``_build_generated_page`` too, so the page summary is still- drawn from the model's opening rather than from a table row.+ The package table of a roll-up stays in the body: it is how a+ reader finds the packages the page covers. Replaced, not appended,+ so a page reused from a prior run keeps one table."""- tables = []if ctx.packages:- # One row per package of a roll-up, for the same reason.- tables.append(- self._render("_package_table.j2", style_prefix=False, rows=ctx.packages)- )+ table = self._render("_package_table.j2", style_prefix=False, rows=ctx.packages)+ page.content = embed_package_table(page.content or "", table.strip())+ page.content, questions = split_questions(page.content)rows, omitted = build_concept_index(file_contexts)- if rows:- tables.append(- self._render(- "_concept_index_table.j2", style_prefix=False, rows=rows, omitted=omitted- )- )- for table in tables:- page.content = f"{(page.content or '').rstrip()}\n\n{table.strip()}\n"+ signals = module_signals(ctx, module_git_summary)+ if signals:+ page.metadata[MODULE_SIGNALS_KEY] = signals+ digest = self._render(+ "_agent_digest.j2",+ style_prefix=False,+ questions=questions,+ public_api=ctx.public_api,+ concept_rows=rows,+ concept_omitted=omitted,+ signals=signals,+ )+ page.digest = re.sub(r"\n{3,}", "\n\n", digest).strip()return pageif self._config.deterministic:- page = self._stub_module_page(ctx, page_target, title, module_git_summary)- return _stamp_concept(_with_concept_index(page))- user_prompt = self._render("module_page.j2", ctx=ctx, module_git_summary=module_git_summary)+ page = self._stub_module_page(ctx, page_target, title)+ return _stamp_concept(_with_digest(page))+ user_prompt = self._render("module_page.j2", ctx=ctx)try:response = await self._call_provider("module_page", user_prompt, str(uuid.uuid4()), target_path=page_target)except Exception as exc:- stub = self._stub_module_page(ctx, page_target, title, module_git_summary)- return _stamp_concept(_with_concept_index(_stub_fallback(stub, "module_page", exc)))+ stub = self._stub_module_page(ctx, page_target, title)+ return _stamp_concept(_with_digest(_stub_fallback(stub, "module_page", exc)))page = self._build_generated_page("module_page",page_target,@@ -364,7 +370,7 @@ class PerTypeGenerationMixin:compute_source_hash(user_prompt),GENERATION_LEVELS["module_page"],)- return _stamp_concept(_with_concept_index(page))+ return _stamp_concept(_with_digest(page))async def generate_scc_page(self,diff --git a/packages/core/src/repowise/core/generation/page_generator/structural.py b/packages/core/src/repowise/core/generation/page_generator/structural.pyindex 0801ca865..0e1ab984f 100644--- a/packages/core/src/repowise/core/generation/page_generator/structural.py+++ b/packages/core/src/repowise/core/generation/page_generator/structural.py@@ -930,7 +930,6 @@ class StructuralRenderMixin:ctx: Any,target_path: str,title: str,- module_git_summary: dict | None,) -> GeneratedPage:return self._stub_page(page_type="module_page",@@ -938,7 +937,6 @@ class StructuralRenderMixin:title=title,template="module_page.j2",ctx=ctx,- module_git_summary=module_git_summary,)def _stub_repo_overview(diff --git a/packages/core/src/repowise/core/generation/report.py b/packages/core/src/repowise/core/generation/report.pyindex 51d41dbab..f53c64d4c 100644--- a/packages/core/src/repowise/core/generation/report.py+++ b/packages/core/src/repowise/core/generation/report.py@@ -127,8 +127,8 @@ def measure_opening_frames(pages: list[GeneratedPage]) -> OpeningFrameReport:)-# The heading over the identifiers appended to a module page after the model-# has written it. Counted for the same reason the questions block is: the table+# The heading over the identifiers rendered into a module page's digest after+# the model has written it. Counted for the same reason the questions block is: the table# is conditional (a module exporting nothing renders none), so only the ratio# is readable, and nothing else in a run would report it going to zero.CONCEPT_INDEX_HEADING = "## Concept index"@@ -145,7 +145,7 @@ def _count_concept_indexes(pages: list[GeneratedPage]) -> dict[str, int]:eligible = [p for p in pages if p.page_type == "module_page"]return {"eligible_pages": len(eligible),- "with_index": sum(1 for p in eligible if CONCEPT_INDEX_HEADING in (p.content or "")),+ "with_index": sum(1 for p in eligible if CONCEPT_INDEX_HEADING in (p.digest or "")),}diff --git a/packages/core/src/repowise/core/generation/templates/_agent_digest.j2 b/packages/core/src/repowise/core/generation/templates/_agent_digest.j2new file mode 100644index 000000000..59843c1dd--- /dev/null+++ b/packages/core/src/repowise/core/generation/templates/_agent_digest.j2@@ -0,0 +1,57 @@+{#- A module page's agent digest (``wiki_pages.digest``): indexed and served to+ agents beside the page body, shown in the reader only on request. Rendered+ from the index after the model has written the page, so every name, path+ and number here is the index's own. The questions are the exception: the+ model writes them and ``agent_digest.split_questions`` moves them here.++ Each section renders only when it has rows: a heading with nothing under it+ claims the module has none, in the same shape as a real answer. -#}+{% if questions %}+{{ questions }}++{% endif %}+{% if public_api %}+## Public API++What a consumer imports, read off the package manifests and re-exports.++{% for e in public_api -%}+- `{{ e.name }}` ({{ e.kind }}, `{{ e.file }}`){% if e.alias_of %}, another name for `{{ e.alias_of }}`{% endif %}{% if e.signature %}: `` {{ e.signature | oneline }} ``{% endif %}+{% endfor %}++{% endif %}+{% if concept_rows %}+## Concept index++What the page calls things, the identifier to search for, and where it lives.++| Concept | Symbol | File |+| --- | --- | --- |+{% for row in concept_rows -%}+| {{ row.concept }} | `{{ row.symbol }}` | `{{ row.file }}` |+{% endfor -%}+{%- if concept_omitted %}+_{{ concept_omitted }} further public symbol{{ "s" if concept_omitted != 1 else "" }} in this module are not listed._+{% endif %}++{% endif %}+{% if signals %}+## Signals++From git history over the {{ signals.get("files", 0) }} files this page covers.++{% if signals.get("hotspots") %}- Change hotspots: {{ signals.get("hotspots") }}+{% endif %}+{%- if signals.get("bus_factor_one") %}- Files with a bus factor of one: {{ signals.get("bus_factor_one") }}+{% endif %}+{%- if signals.get("bug_fixes") %}- Bug-fix commits: {{ signals.get("bug_fixes") }}{% if signals.get("most_fixed_file") %}, most in `{{ signals.get("most_fixed_file") }}`{% endif %}+{% endif %}+{%- if signals.get("stable") %}- Stable files: {{ signals.get("stable") }}+{% endif %}+{%- if signals.get("owners") %}- Primary maintainers: {% for o in signals.get("owners") %}{{ o.name }} ({{ o.files }} file{{ "s" if o.files != 1 else "" }}){% if not loop.last %}, {% endif %}{% endfor %}+{% endif %}+{%- if signals.get("most_active_file") %}- Most active file: `{{ signals.get("most_active_file") }}` ({{ signals.get("most_active_commits_90d") }} commits in 90 days)+{% endif %}+{%- if signals.get("co_changes") %}- Changes together with (no import edge explains it): {% for c in signals.get("co_changes") %}`{{ c.path }}` ({{ c.files }} file{{ "s" if c.files != 1 else "" }}){% if not loop.last %}, {% endif %}{% endfor %}+{% endif %}+{% endif %}diff --git a/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2 b/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2deleted file mode 100644index 9b2c0ef49..000000000--- a/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2+++ /dev/null@@ -1,18 +0,0 @@-{#- The identifiers behind the page's prose, rendered from the assembled symbol- data and appended after the model has written the page. Nothing here passes- through a provider, so the names and paths are the index's own.-- Rendered only when there is at least one row: a header with nothing under- it claims the module exports nothing, in the same shape as a real answer. -#}-## Concept index--What the prose above calls things, the identifier to search for, and where it lives.--| Concept | Symbol | File |-| --- | --- | --- |-{% for row in rows -%}-| {{ row.concept }} | `{{ row.symbol }}` | `{{ row.file }}` |-{% endfor -%}-{%- if omitted %}-_{{ omitted }} further public symbol{{ "s" if omitted != 1 else "" }} in this module are not listed._-{% endif -%}diff --git a/packages/core/src/repowise/core/generation/templates/_package_table.j2 b/packages/core/src/repowise/core/generation/templates/_package_table.j2index 4e05a953c..8ff176620 100644--- a/packages/core/src/repowise/core/generation/templates/_package_table.j2+++ b/packages/core/src/repowise/core/generation/templates/_package_table.j2@@ -1,8 +1,7 @@{#- One row per package a package roll-up covers, rendered from the grouping andappended after the model has written the page, so no response can drop or- merge a package. -#}-## Packages-+ merge a package. The heading comes from ``embed_package_table``, which+ replaces the section instead of adding a second one when a page is reused. -#}This page covers {{ rows | length }} sibling packages, each with its own manifest.| Package | Files |diff --git a/packages/core/src/repowise/core/generation/templates/module_page.j2 b/packages/core/src/repowise/core/generation/templates/module_page.j2index 3ca0144f1..ba57ee895 100644--- a/packages/core/src/repowise/core/generation/templates/module_page.j2+++ b/packages/core/src/repowise/core/generation/templates/module_page.j2@@ -88,37 +88,6 @@ Use these to explain *why* the subsystem is shaped the way it is. Do not invent{% endfor %}{% endif %}-{# Health and history signals a reader cannot get from the code alone. Only- surfaced when git metadata supplied them; all zero/empty degrades to nothing- rather than a fabricated number. This is repowise's differentiation — weave- the true ones into the narrative, do not tabulate them mechanically. #}-{%- set _health = [] -%}-{%- if ctx.hotspot_count > 0 -%}{%- set _ = _health.append(ctx.hotspot_count ~ " file" ~ ("s" if ctx.hotspot_count != 1 else "") ~ " flagged as change hotspots") -%}{%- endif -%}-{%- if ctx.single_owner_files > 0 -%}{%- set _ = _health.append(ctx.single_owner_files ~ " file" ~ ("s" if ctx.single_owner_files != 1 else "") ~ " with a bus factor of one (single-maintainer risk)") -%}{%- endif -%}-{%- if ctx.bugfix_total > 0 -%}{%- set _ = _health.append(ctx.bugfix_total ~ " bug-fix commit" ~ ("s" if ctx.bugfix_total != 1 else "") ~ " in recent history" ~ (", most in `" ~ ctx.most_fixed_file.path ~ "`" if ctx.most_fixed_file else "")) -%}{%- endif -%}-{%- if ctx.stable_count > 0 -%}{%- set _ = _health.append(ctx.stable_count ~ " file" ~ ("s" if ctx.stable_count != 1 else "") ~ " that have been stable") -%}{%- endif -%}-{% if _health %}-### Subsystem health (from git history — state the notable ones plainly)-{% for h in _health %}-- {{ h }}-{% endfor %}-{% endif %}--{% if ctx.coupled_modules %}-### Change-coupled with (co-changes in history, not import edges)-These modules change together with this one in history and nothing in the dependency graph accounts for it — hidden coupling worth naming.-{% for cm in ctx.coupled_modules %}-- `{{ cm.path }}` ({{ cm.count }} co-changing file{{ "s" if cm.count != 1 else "" }})-{% endfor %}-{% endif %}--{% if ctx.top_owners %}-### Ownership-{% for o in ctx.top_owners %}-- **{{ o.name }}** — primary maintainer of {{ o.file_count }} file(s)-{% endfor %}-{% endif %}-{% if ctx.key_classes %}### Class hierarchy{% for h in ctx.key_classes %}@@ -126,11 +95,6 @@ These modules change together with this one in history and nothing in the depend{% endfor %}{% endif %}-{% if module_git_summary is defined and module_git_summary %}-### Recent activity-Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_summary.most_active_commits_90d }} commits in 90 days).-{% endif %}-{% if ctx.declared_files %}### What other files declare (signatures only, no code shown){% for line in ctx.declared_files %}@@ -150,7 +114,6 @@ Write the page now from the material above. Requirements:- Synthesise across the whole file set. Draw on the material as a body of evidence; a file-by-file walkthrough is a failure even when accurate. Draw on many files, not one.- Put any enumerable list (key files, dependencies, decisions) in a markdown table where it reads better than prose.- Name the subsystem's key files and the neighbouring modules it consumes or serves in backticks exactly as written above so they resolve to wiki links. Mention a neighbour only where it explains this subsystem's role — no links for density, no reciprocal mentions for symmetry.-- Weave the true health and history signals (hotspots, bus factor, change coupling, fix history) into the narrative where they explain the subsystem's character. Skip any that are absent; never invent one.- State behaviour only when an excerpt shows it; otherwise say what the file declares.- A commit subject is history, not current behaviour: if you cite one, label it (historical, <date>).- Third person throughout. No concluding or summary section.diff --git a/packages/core/src/repowise/core/generation/templates/stub/module_page.j2 b/packages/core/src/repowise/core/generation/templates/stub/module_page.j2index 556e28858..32f1be6ca 100644--- a/packages/core/src/repowise/core/generation/templates/stub/module_page.j2+++ b/packages/core/src/repowise/core/generation/templates/stub/module_page.j2@@ -85,13 +85,6 @@ Highest-PageRank first: the order to read them in.{% endfor %}{% endif %}-{% if ctx.top_owners %}-## Ownership-{% for o in ctx.top_owners %}-- **{{ o.name }}**: primary maintainer of {{ o.file_count }} file{{ "s" if o.file_count != 1 else "" }}-{% endfor %}-{% endif %}-{% if ctx.dead_code_findings %}## Potentially unused code{% for finding in ctx.dead_code_findings[:10] %}@@ -99,30 +92,7 @@ Highest-PageRank first: the order to read them in.{% endfor %}{% endif %}-{%- set _health = [] -%}-{%- if ctx.hotspot_count > 0 -%}{%- set _ = _health.append(ctx.hotspot_count ~ " file" ~ ("s" if ctx.hotspot_count != 1 else "") ~ " flagged as change hotspots") -%}{%- endif -%}-{%- if ctx.single_owner_files > 0 -%}{%- set _ = _health.append(ctx.single_owner_files ~ " file" ~ ("s" if ctx.single_owner_files != 1 else "") ~ " with a bus factor of one (single-maintainer risk)") -%}{%- endif -%}-{%- if ctx.bugfix_total > 0 -%}{%- set _ = _health.append(ctx.bugfix_total ~ " bug-fix commit" ~ ("s" if ctx.bugfix_total != 1 else "") ~ " in recent history" ~ (", most in " ~ ctx.most_fixed_file.path if ctx.most_fixed_file else "")) -%}{%- endif -%}-{%- if ctx.stable_count > 0 -%}{%- set _ = _health.append(ctx.stable_count ~ " file" ~ ("s" if ctx.stable_count != 1 else "") ~ " that have been stable") -%}{%- endif -%}-{% if _health %}-## Subsystem health-{% for h in _health %}-- {{ h }}-{% endfor %}-{% endif %}--{% if ctx.coupled_modules %}-## Change-coupled with-Modules that change together with this one in history without importing it.-{% for cm in ctx.coupled_modules %}-- `{{ cm.path }}` (co-changed {{ cm.count }}×)-{% endfor %}-{% endif %}--{% if module_git_summary %}-## Recent activity-Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_summary.most_active_commits_90d }} commits in the last 90 days).-{% endif %}+{% include "stub/_footer.j2" %}## Questions this page answers- What does {{ ctx.title }} do, and where does it sit in the system?@@ -130,4 +100,4 @@ Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_sum{% if ctx.entry_points %}- What are the entry points into {{ ctx.title }}?{% endif %}{% if ctx.dependencies %}- What does {{ ctx.title }} depend on?{% endif %}{% if ctx.dependents %}- What relies on {{ ctx.title }}?-{% endif %}{% include "stub/_footer.j2" %}+{% endif %}diff --git a/packages/core/src/repowise/core/persistence/crud/pages.py b/packages/core/src/repowise/core/persistence/crud/pages.pyindex 40fb79244..35fc57c1d 100644--- a/packages/core/src/repowise/core/persistence/crud/pages.py+++ b/packages/core/src/repowise/core/persistence/crud/pages.py@@ -62,6 +62,7 @@ def _apply_page_upsert(title: str,content: str,summary: str,+ digest: str,target_path: str,source_hash: str,model_name: str,@@ -140,6 +141,8 @@ def _apply_page_upsert():existing.title = titleexisting.summary = summary+ # Git signals and identifiers in the digest move without the prose.+ existing.digest = digestexisting.target_path = target_pathexisting.freshness_status = freshness_statusexisting.confidence = confidence@@ -174,6 +177,7 @@ def _apply_page_upsert(existing.title = titleexisting.content = contentexisting.summary = summary+ existing.digest = digestexisting.target_path = target_pathexisting.source_hash = source_hashexisting.model_name = model_name@@ -200,6 +204,7 @@ def _apply_page_upsert(title=title,content=content,summary=summary,+ digest=digest,target_path=target_path,source_hash=source_hash,model_name=model_name,@@ -232,6 +237,7 @@ async def upsert_page(title: str,content: str,summary: str = "",+ digest: str = "",target_path: str,source_hash: str,model_name: str,@@ -272,6 +278,7 @@ async def upsert_page(title=title,content=content,summary=summary,+ digest=digest,target_path=target_path,source_hash=source_hash,model_name=model_name,@@ -327,6 +334,7 @@ async def load_prior_pages(source_hash=row.source_hash,model_name=row.model_name,content=row.content,+ digest=row.digest or "",input_tokens=row.input_tokens,output_tokens=row.output_tokens,cached_tokens=row.cached_tokens,@@ -354,6 +362,7 @@ async def upsert_page_from_generated(title=gp.title, # type: ignore[attr-defined]content=gp.content, # type: ignore[attr-defined]summary=getattr(gp, "summary", "") or "",+ digest=getattr(gp, "digest", "") or "",target_path=gp.target_path, # type: ignore[attr-defined]source_hash=gp.source_hash, # type: ignore[attr-defined]model_name=gp.model_name, # type: ignore[attr-defined]@@ -436,6 +445,7 @@ async def upsert_pages_from_generated(title=gp.title,content=gp.content,summary=getattr(gp, "summary", "") or "",+ digest=getattr(gp, "digest", "") or "",target_path=gp.target_path,source_hash=gp.source_hash,model_name=gp.model_name,diff --git a/packages/core/src/repowise/core/persistence/information_floor.py b/packages/core/src/repowise/core/persistence/information_floor.pyindex 64bccb556..65490807d 100644--- a/packages/core/src/repowise/core/persistence/information_floor.py+++ b/packages/core/src/repowise/core/persistence/information_floor.py@@ -165,12 +165,14 @@ def reset_pages_denied_a_vector() -> None:_pages_denied_a_vector = 0-def meets_information_floor(content: str, floor: int | None = None) -> bool:- """Whether *content* says enough to be worth a slot in the index.+def meets_information_floor(content: str, floor: int | None = None, digest: str = "") -> bool:+ """Whether *content* and the page's *digest* say enough to be worth an index slot.- A floor of 0 admits everything, which is how the feature is turned off.+ Both count: the digest is indexed with the body, and a short page whose+ words moved into its digest still answers questions. A floor of 0 admits+ everything, which is how the feature is turned off."""limit = information_floor() if floor is None else floorif limit <= 0:return True- return len(substantive_text(content)) >= limit+ return len(substantive_text(f"{content}\n{digest}")) >= limitdiff --git a/packages/core/src/repowise/core/persistence/models.py b/packages/core/src/repowise/core/persistence/models.pyindex f1c15029e..b0b02e024 100644--- a/packages/core/src/repowise/core/persistence/models.py+++ b/packages/core/src/repowise/core/persistence/models.py@@ -201,6 +201,11 @@ class Page(Base):# Surfaced by get_context as the default narrative; content is gated# behind include=["full_doc"] to keep MCP responses small.summary: Mapped[str] = mapped_column(Text, nullable=False, default="")+ # Agent digest (markdown): what a page carries for search and agents but+ # not for a reader, e.g. the questions it answers, its identifiers, public+ # API and git signals. Indexed and served by MCP beside ``content``; the+ # reader shows it only on request. See ``generation/agent_digest.py``.+ digest: Mapped[str] = mapped_column(Text, nullable=False, default="", server_default="")target_path: Mapped[str] = mapped_column(Text, nullable=False)source_hash: Mapped[str] = mapped_column(String(64), nullable=False)model_name: Mapped[str] = mapped_column(String(128), nullable=False)diff --git a/packages/core/src/repowise/core/persistence/search.py b/packages/core/src/repowise/core/persistence/search.pyindex cb0d24d1d..d8a0cfe01 100644--- a/packages/core/src/repowise/core/persistence/search.py+++ b/packages/core/src/repowise/core/persistence/search.py@@ -66,16 +66,28 @@ _SNIPPET_LEN = 200# ``vocabulary`` is a file page's own identifiers and comment prose. It is kept# off the rendered page (it reads as noise), so without its own column a# keyless store cannot match a page on any of those words.-PAGE_FTS_COLUMNS = ("page_id", "title", "content", "summary", "target_path", "vocabulary")+#+# ``digest`` is the page's agent material (questions, identifiers, signals),+# kept out of ``content`` so the reader does not scroll past it and indexed here+# so retrieval still finds the page by those words.+PAGE_FTS_COLUMNS = (+ "page_id",+ "title",+ "content",+ "summary",+ "target_path",+ "vocabulary",+ "digest",+)PAGE_FTS_DDL = ("CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "- "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary)"+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary, digest)")_PAGE_FTS_INSERT_SQL = (- "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary) "- "VALUES (:pid, :title, :content, :summary, :target_path, :vocabulary)"+ "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary, digest) "+ "VALUES (:pid, :title, :content, :summary, :target_path, :vocabulary, :digest)")# Page-metadata key for a file page's vocabulary (field names, string@@ -150,7 +162,8 @@ _ORPHAN_DELETE_SQL = f"DELETE FROM page_fts WHERE {_ORPHAN_PREDICATE}"PG_FTS_EXPRESSION = ("to_tsvector('english', ""COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "- "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,'') || ' ' "+ "|| COALESCE(digest,''))")# How many distinct terms' document frequencies one FullTextSearch keeps. Term@@ -164,12 +177,13 @@ _SCORE_EPSILON = 1e-6# Per-column bm25 weights, in ``PAGE_FTS_COLUMNS`` order: title and# target_path name the file a page is about, content and summary only mention-# it. ``page_id`` is UNINDEXED and contributes nothing, but bm25() takes one-# weight per column, so it still needs one. bm25() ignores a weight past the+# it, and ``digest`` weighs like the prose it indexes. ``page_id`` is+# UNINDEXED and contributes nothing, but bm25() takes one weight per column,+# so it still needs one. bm25() ignores a weight past the# last column and defaults a missing one to 1.0, so the arity is checked by# test_search_fts_columns rather than by anything raising here. The vocabulary# is a bag of words, so it counts for less than prose written about the file.-_BM25_COLUMN_WEIGHTS = (0.0, 4.0, 1.0, 1.0, 3.0, 0.5)+_BM25_COLUMN_WEIGHTS = (0.0, 4.0, 1.0, 1.0, 3.0, 0.5, 1.0)_BM25_SCORE = "bm25(page_fts, " + ", ".join(str(w) for w in _BM25_COLUMN_WEIGHTS) + ")"_log = logging.getLogger(__name__)@@ -348,10 +362,10 @@ class FullTextSearch:await conn.execute(text("INSERT INTO page_fts"- "(page_id, title, content, summary, target_path, vocabulary) "+ "(page_id, title, content, summary, target_path, vocabulary, digest) ""SELECT id, COALESCE(title,''), COALESCE(content,''), "" COALESCE(summary,''), COALESCE(target_path,''), "- f" {_VOCABULARY_SQL} "+ f" {_VOCABULARY_SQL}, COALESCE(digest,'') ""FROM wiki_pages"))@@ -430,6 +444,7 @@ class FullTextSearch:content: str,summary: str | None = None,target_path: str | None = None,+ digest: str = "",) -> None:"""Add or replace a page in the FTS index.@@ -450,13 +465,13 @@ class FullTextSearch:exclude, and nothing would report the disagreement. The page itself isuntouched: it stays in ``wiki_pages`` and stays a valid link target."""- await self.index_many([(page_id, title, content, summary, target_path)])+ await self.index_many([(page_id, title, content, summary, target_path, digest)])async def index_pages(self, pages: Iterable[Any]) -> None:"""Index generated pages, one :meth:`index_many` transaction per chunk.- Each entry needs ``page_id``, ``title``, ``content``, ``summary`` and- ``target_path``. Use this, not :meth:`index` in a loop: a write deletes+ Each entry needs ``page_id``, ``title``, ``content``, ``summary``,+ ``target_path`` and ``digest``. Use this, not :meth:`index` in a loop: a write deletesby ``page_id``, which FTS5 stores unindexed, so every delete scans thewhole index. Per page that is quadratic in wiki size (18 GB of readsfor a 2,300-page rebuild); chunked it is one scan per chunk, while a@@ -464,16 +479,19 @@ class FullTextSearch:"""for chunk in chunked(list(pages)):await self.index_many(- [(p.page_id, p.title, p.content, p.summary, p.target_path) for p in chunk]+ [+ (p.page_id, p.title, p.content, p.summary, p.target_path, p.digest)+ for p in chunk+ ])async def index_many(self,- pages: Sequence[tuple[str, str, str, str | None, str | None]],+ pages: Sequence[tuple[str, str, str, str | None, str | None, str]],) -> None:"""Add or replace many pages in a single transaction.- Each entry is ``(page_id, title, content, summary, target_path)`` and+ Each entry is ``(page_id, title, content, summary, target_path, digest)`` andgets exactly the semantics :meth:`index` documents, floor exclusionincluded — this is the one implementation and :meth:`index` is thesingle-page call into it. What changes is the transaction count:@@ -497,11 +515,11 @@ class FullTextSearch:# existed.deletions: dict[str, None] = {}insertions: dict[str, dict[str, str]] = {}- for page_id, title, content, summary, target_path in pages:+ for page_id, title, content, summary, target_path, digest in pages:deletions[page_id] = Noneif summary is None or target_path is None:self._warn_missing_index_fields(page_id, summary, target_path)- if meets_information_floor(content):+ if meets_information_floor(content, digest=digest or ""):insertions[page_id] = {"pid": page_id,"title": title,@@ -509,6 +527,7 @@ class FullTextSearch:"summary": summary or "","target_path": target_path or "","vocabulary": "",+ "digest": digest or "",}else:self._count_skipped_below_floor(page_id, content)diff --git a/packages/core/src/repowise/core/persistence/vector_store/_base.py b/packages/core/src/repowise/core/persistence/vector_store/_base.pyindex 4359d1d55..26ad3b7a0 100644--- a/packages/core/src/repowise/core/persistence/vector_store/_base.py+++ b/packages/core/src/repowise/core/persistence/vector_store/_base.py@@ -129,6 +129,7 @@ def embed_item(summary: str,content: str,page_metadata: Mapping[str, object] | str | None = None,+ digest: str = "",) -> tuple[str, str, dict] | None:"""Build the one ``(page_id, text, metadata)`` item every writer embeds.@@ -160,7 +161,7 @@ def embed_item(page held out of one arm and kept in the other is still fetched, stilloccupies one of the fixed number of rows retrieval takes before it filtersanything, and still displaces a page that could have answered. The test is- applied to ``content`` alone for that reason — the same input the+ applied to ``content`` and ``digest`` for that reason, the same input thefull-text side measures, so the two arms cannot disagree about a page.The page itself is untouched either way. It stays in ``wiki_pages``, still@@ -171,7 +172,10 @@ def embed_item(``page_metadata`` is the page's metadata, as a dict (a generated page) orthe stored JSON string (a ``wiki_pages`` row). Only:data:`FILE_VOCABULARY_KEY` is read from it, and appended after the- content; the floor still measures ``content`` alone.+ content; the floor does not measure it.++ ``digest`` is the page's agent material (``wiki_pages.digest``), embedded+ after the content for the same reason the vocabulary is."""if not title.strip():raise ValueError(@@ -179,10 +183,12 @@ def embed_item(f"vector that cannot be found by name and reports nothing wrong; "f"pass the page's real title.")- if not meets_information_floor(content):+ if not meets_information_floor(content, digest=digest):count_page_denied_a_vector()return None- parts = [p for p in (title, target_path, summary, content, _vocabulary(page_metadata)) if p]+ parts = [+ p for p in (title, target_path, summary, content, digest, _vocabulary(page_metadata)) if p+ ]return (page_id,"\n".join(parts),diff --git a/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py b/packages/server/src/repowise/server/mcp_server/_budget/budgeter.pyindex 22f464dfe..fe9aa4b92 100644--- a/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py+++ b/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py@@ -348,7 +348,12 @@ def _with_budget_reason(prior_reason: Any) -> str:# Heavy optional fields we can strip from a target's docs block without losing# its identity. Ordering matters: earlier entries are dropped first because they# carry the most bytes per unit of navigational value.-HEAVY_DOC_FIELDS: tuple[str, ...] = ("content_md", "documentation", "file_summary")+HEAVY_DOC_FIELDS: tuple[str, ...] = (+ "content_md",+ "digest_md",+ "documentation",+ "file_summary",+)def symbol_priority(sym: dict[str, Any], query_terms: set[str]) -> tuple[int, int, int]:diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.pyindex 126bd7053..6cbcc1b7d 100644--- a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py@@ -240,8 +240,14 @@ async def _attach_page_excerpts(hits: list[dict], ctx: Any = None) -> int:return len(top)try:async with get_session(ctx.session_factory) as session:- res = await session.execute(select(Page.id, Page.content).where(Page.id.in_(page_ids)))- content_by_id = {row[0]: (row[1] or "") for row in res.all()}+ res = await session.execute(+ select(Page.id, Page.content, Page.digest).where(Page.id.in_(page_ids))+ )+ # The digest carries what the page answers in an agent's words.+ content_by_id = {+ row[0]: "\n\n".join(part for part in (row[1], row[2]) if part)+ for row in res.all()+ }except Exception:# Never fail the answer over an excerpt fetch, but never hide it either._log.warning(diff --git a/packages/server/src/repowise/server/mcp_server/tool_context/context.py b/packages/server/src/repowise/server/mcp_server/tool_context/context.pyindex 6e0b0961e..f4a86a9ff 100644--- a/packages/server/src/repowise/server/mcp_server/tool_context/context.py+++ b/packages/server/src/repowise/server/mcp_server/tool_context/context.py@@ -10,7 +10,8 @@ The split keeps the cached prompt prefix small on multi-turn agent sessions:``get_context`` stays under ~2k tokens for common targets.Optional ``include`` parameter widens the response:- - include=["full_doc"] → full wiki markdown content+ - include=["full_doc"] -> full wiki markdown content, plus the page's agent+ digest (questions, identifiers, git signals)- include=["callers"] → who calls this symbol (symbol targets only)- include=["callees"] → what this symbol calls (symbol targets only)- include=["ownership"] → primary owner, bus factor, contributor countdiff --git a/packages/server/src/repowise/server/mcp_server/tool_context/targets.py b/packages/server/src/repowise/server/mcp_server/tool_context/targets.pyindex cf951790b..35a1887c9 100644--- a/packages/server/src/repowise/server/mcp_server/tool_context/targets.py+++ b/packages/server/src/repowise/server/mcp_server/tool_context/targets.py@@ -709,6 +709,8 @@ async def _resolve_one_target(docs["summary"] = page.summary or ""if want_full_doc:docs["content_md"] = page.content+ if page.digest:+ docs["digest_md"] = page.digestif page.human_notes:docs["human_notes"] = page.human_notes# Symbols in this file@@ -818,6 +820,10 @@ async def _resolve_one_target(docs["section"] = page.section_numberif want_full_doc:docs["content_md"] = page.content+ # Questions, identifiers, public API and git signals: kept off+ # the reader's page body, served to agents beside it.+ if page.digest:+ docs["digest_md"] = page.digest# Non-file children only; file children are in "files" below.res = await session.execute(select(Page)diff --git a/packages/server/src/repowise/server/schemas/pages.py b/packages/server/src/repowise/server/schemas/pages.pyindex e3545e36c..5c3bb2549 100644--- a/packages/server/src/repowise/server/schemas/pages.py+++ b/packages/server/src/repowise/server/schemas/pages.py@@ -158,6 +158,9 @@ class PageSummaryResponse(BaseModel):class PageResponse(PageSummaryResponse):content: str+ # Agent material kept off the page body (questions, identifiers, public+ # API, git signals). The reader shows it on request. Empty when none.+ digest: str = ""metadata: dict@classmethod@@ -168,6 +171,7 @@ class PageResponse(PageSummaryResponse):return cls(**_summary_fields(obj, metadata),content=obj.content, # type: ignore[attr-defined]+ digest=obj.digest or "", # type: ignore[attr-defined]metadata=metadata,)diff --git a/packages/types/src/docs.ts b/packages/types/src/docs.tsindex a525e5e83..af51b26ca 100644--- a/packages/types/src/docs.ts+++ b/packages/types/src/docs.ts@@ -84,6 +84,13 @@ export interface DocPageSummary {export interface DocPage extends DocPageSummary {content: string;+ /**+ * The page's agent digest: the questions it answers, its identifiers,+ * public API and git signals. Indexed and served to agents beside+ * `content`, kept off the body, shown by the reader on request. Absent or+ * empty on pages that carry none.+ */+ digest?: string;metadata: Record<string, unknown>;}diff --git a/packages/types/src/generated/http.ts b/packages/types/src/generated/http.tsindex 8bfb0668b..32299a096 100644--- a/packages/types/src/generated/http.ts+++ b/packages/types/src/generated/http.ts@@ -2392,6 +2392,7 @@ export interface PageResponse {created_at: string;updated_at: string;content: string;+ digest?: string;metadata: Record<string, unknown>;}diff --git a/packages/ui/__tests__/docs/docs-reader-reference.test.tsx b/packages/ui/__tests__/docs/docs-reader-reference.test.tsxnew file mode 100644index 000000000..208495eb6--- /dev/null+++ b/packages/ui/__tests__/docs/docs-reader-reference.test.tsx@@ -0,0 +1,94 @@+import { describe, it, expect, beforeAll } from "vitest";+import { fireEvent, render, screen } from "@testing-library/react";+import { DocsReader } from "../../src/docs/docs-reader.js";+import type { DocPage } from "@repowise-dev/types/docs";++/**+ * A page's agent digest is one click away, never in the reading path.+ *+ * The body is what a reader came for. The questions, identifiers and git+ * signals written for search and agents sit behind a Reference tab, which+ * exists only on pages that carry a digest. Git signals also show as a small+ * History card beside the page.+ */++beforeAll(() => {+ Element.prototype.scrollTo = () => {};+});++function makePage(overrides: Partial<DocPage> = {}): DocPage {+ return {+ id: "module_page:pkg/parser",+ repository_id: "r1",+ page_type: "module_page",+ title: "Parsing",+ content: "The parser turns files into records.",+ target_path: "pkg/parser",+ source_hash: "h",+ model_name: "m",+ provider_name: "openai",+ input_tokens: 0,+ output_tokens: 0,+ cached_tokens: 0,+ generation_level: 4,+ version: 1,+ confidence: 1,+ freshness_status: "fresh",+ metadata: {},+ created_at: "2026-08-01T00:00:00Z",+ updated_at: "2026-08-01T00:00:00Z",+ ...overrides,+ } as DocPage;+}++function renderReader(page: DocPage) {+ return render(+ <DocsReader+ page={page}+ repoId="r1"+ persona="contributor"+ sidebarOpen={false}+ buildPageHref={(id) => `?page=${id}`}+ LinkComponent={({ href, children, ...rest }) => (+ <a href={href} {...rest}>+ {children}+ </a>+ )}+ />,+ );+}++const DIGEST = "## Questions this page answers\n\n- How is a file parsed?";++describe("DocsReader reference view", () => {+ it("shows the body first and the digest behind the Reference tab", () => {+ renderReader(makePage({ digest: DIGEST }));++ expect(screen.getByText("The parser turns files into records.")).toBeTruthy();+ expect(screen.queryByText("How is a file parsed?")).toBeNull();++ fireEvent.click(screen.getByRole("tab", { name: "Reference" }));++ expect(screen.getByText("How is a file parsed?")).toBeTruthy();+ expect(screen.queryByText("The parser turns files into records.")).toBeNull();+ });++ it("offers no tabs on a page without a digest", () => {+ renderReader(makePage());++ expect(screen.queryByRole("tab")).toBeNull();+ });++ it("summarises the module's git history beside the page", () => {+ renderReader(+ makePage({+ metadata: {+ module_signals: { files: 12, hotspots: 2, owners: [{ name: "Ada", files: 9 }] },+ },+ }),+ );++ expect(screen.getByText("Ada · 9 of 12 files")).toBeTruthy();+ expect(screen.getAllByText("2 of 12 files").length).toBeGreaterThan(0);+ });+});diff --git a/packages/ui/__tests__/docs/reader-persona.test.ts b/packages/ui/__tests__/docs/reader-persona.test.tsindex bad8aac21..39f2ed537 100644--- a/packages/ui/__tests__/docs/reader-persona.test.ts+++ b/packages/ui/__tests__/docs/reader-persona.test.ts@@ -8,12 +8,10 @@ import {/*** The reader lens over a deterministic page.*- * A file page's `## In the code` and `## Questions this page answers` are- * written for the index, not for a reader: one is a bag of the file's own- * tokens, the other is question-shaped text so a query has something to match.- * They cannot be dropped upstream — `content` is a single string that FTS- * indexes, the vector store embeds and `get_context` returns verbatim — so the- * lens is where they stop being a reader's problem.+ * A file page's `## Questions this page answers` is written for the index,+ * not for a reader: question-shaped text so a query has something to match.+ * It lives in `content`, which FTS indexes and `get_context` returns, so the+ * lens is where it stops being a reader's problem.** What the rendered page actually emits is pinned on the Python side, against* the template rather than against a string typed here.@@ -35,10 +33,6 @@ const PAGE = ["## Questions this page answers","","- What does `src/walk.py` export?",- "",- "## In the code",- "",- "prune_dirs max_depth node_modules",].join("\n");describe("filterMarkdownByPersona", () => {@@ -46,8 +40,7 @@ describe("filterMarkdownByPersona", () => {const out = filterMarkdownByPersona(PAGE, "contributor");expect(out).not.toContain("## Questions this page answers");- expect(out).not.toContain("## In the code");- expect(out).not.toContain("prune_dirs max_depth node_modules");+ expect(out).not.toContain("What does `src/walk.py` export?");});it("keeps the reference material the default reader came for", () => {@@ -84,3 +77,23 @@ describe("filterMarkdownByPersona", () => {expect(personaFilteringApplies("# Title\n\n## Overview\n\nProse.\n")).toBe(false);});});++describe("whole-heading matching", () => {+ it("keeps a step-named section that starts like a hidden heading", () => {+ const page = [+ "# Ingestion",+ "",+ "## Source files become symbols",+ "",+ "Parsing turns bytes into records.",+ "",+ "## Source",+ "",+ "```py\nx = 1\n```",+ ].join("\n");+ const out = filterMarkdownByPersona(page, "overview");++ expect(out).toContain("## Source files become symbols");+ expect(out).not.toContain("## Source\n");+ });+});diff --git a/packages/ui/src/docs/docs-reader.tsx b/packages/ui/src/docs/docs-reader.tsxindex 7a4e2547c..babed4d4f 100644--- a/packages/ui/src/docs/docs-reader.tsx+++ b/packages/ui/src/docs/docs-reader.tsx@@ -1,6 +1,6 @@"use client";-import { useCallback, useEffect, useMemo, useRef } from "react";+import { useCallback, useEffect, useMemo, useRef, useState } from "react";import {FileText,ArrowRight,@@ -14,6 +14,7 @@ import { formatRelativeTime, formatTokens } from "../lib/format";import { getPageLabel, isStubFallbackPage } from "../lib/page-types";import { computeDocNav } from "./doc-nav";import { filterMarkdownByPersona, type ReaderPersona } from "./reader-persona";+import { ModuleHistory } from "./module-history";import { WikiMarkdown } from "../wiki/wiki-markdown";import { TableOfContents } from "../wiki/table-of-contents";import { BacklinksPanel } from "../wiki/backlinks-panel";@@ -24,6 +25,7 @@ import {type RelatedReason,} from "../wiki/wiki-links-types";import { Breadcrumb } from "../shared/breadcrumb";+import { ViewTabs } from "../shared/view-tabs";import { Skeleton } from "../ui/skeleton";/** Related entries shown before the "+ N more" line. Five, not eight: the list@@ -31,6 +33,11 @@ import { Skeleton } from "../ui/skeleton";* of everything the graph knows. */const RELATED_LIMIT = 5;+const READER_VIEWS = [+ { id: "page", label: "Page" },+ { id: "reference", label: "Reference" },+];+const RELATED_REASON_LABELS: Record<RelatedReason, string> = {imports: "imports","imported-by": "imported by",@@ -195,6 +202,8 @@ export function DocsReader({return (<DocsReaderBody+ // Keyed so the Page / Reference choice resets with the page.+ key={page.id}page={page}pages={pages}repoId={repoId}@@ -307,6 +316,12 @@ function DocsReaderBody({[bodyContent, persona],);+ // The agent digest: questions, identifiers, public API and git signals the+ // page carries for search and MCP, kept off the body and shown on request.+ const digest = page.digest?.trim() ?? "";+ const [view, setView] = useState<"page" | "reference">("page");+ const showingReference = view === "reference" && digest !== "";+// The nearest ancestor that is actually a module. Breadcrumbs now come from// the stored tree, whose ancestors can be a layer or the file a symbol was// spotted in; showing either as "in <name>" would repeat the layer chip@@ -429,6 +444,23 @@ function DocsReaderBody({return Comp;}, [goToPageId]);+ /* No `prose` wrapper: every element the renderer emits is already styled+ through our own tokens, so the plugin contributed exactly two things, a+ hardcoded `prose-invert` that fed dark variables to light mode and+ `code::before/::after { content: "`" }`, which printed literal backticks+ around every unresolved inline ref. */+ const article = (markdown: string) => (+ <article data-chat-selection="" className="max-w-none leading-relaxed overflow-hidden">+ <WikiMarkdown+ content={markdown}+ wikiLinks={wikiLinks}+ buildHref={(pid) => buildPageHref(pid)}+ LinkComponent={WikiInlineLink}+ pages={pages}+ />+ </article>+ );+return (<div className="flex h-full"><div className="flex flex-col flex-1 min-w-0">@@ -597,24 +629,27 @@ function DocsReaderBody({</div>)}- {/* Markdown content.- No `prose` wrapper: every element the renderer emits is already- styled through our own tokens, so the plugin contributed exactly- two things — a hardcoded `prose-invert` that fed dark variables- to light mode, and `code::before/::after { content: "`" }`, which- printed literal backticks around every unresolved inline ref. */}- <article- data-chat-selection=""- className="max-w-none leading-relaxed overflow-hidden"- >- <WikiMarkdown- content={visibleContent}- wikiLinks={wikiLinks}- buildHref={(pid) => buildPageHref(pid)}- LinkComponent={WikiInlineLink}- pages={pages}- />- </article>+ {/* Markdown content. A page with an agent digest gets a Page /+ Reference switch: the body is what a reader came for, the+ digest is what search and agents read, one click away. */}+ {digest ? (+ <ViewTabs+ tabs={READER_VIEWS}+ value={view}+ onValueChange={(id) => setView(id === "reference" ? "reference" : "page")}+ aria-label="Page or reference"+ >+ {showingReference && (+ <p className="mb-4 text-xs text-[var(--color-text-tertiary)]">+ What this page carries for search and agents: the questions it answers,+ the identifiers behind it and its git history.+ </p>+ )}+ {article(showingReference ? digest : visibleContent)}+ </ViewTabs>+ ) : (+ article(visibleContent)+ )}{/* The upgrade affordance, at the end of the content rather thanbeside the title. Someone who has read to here knows the page is@@ -714,6 +749,7 @@ function DocsReaderBody({the intelligence sections and the contents. The breakpoint is2xl, not lg — see the rail below for why. */}<div className="mt-10 grid gap-8 border-t border-[var(--color-border-default)] pt-6 sm:grid-cols-2 2xl:hidden">+ <ModuleHistory metadata={page.metadata} />{intelligenceSlot && (<div className="flex flex-col gap-4">{intelligenceSlot}</div>)}@@ -765,7 +801,9 @@ function DocsReaderBody({{sidebarOpen && (<div className="hidden 2xl:block shrink-0 w-[300px] overflow-auto"><div className="flex flex-col gap-7 py-8 pl-6 pr-7">- <TableOfContents content={bodyContent} />+ <TableOfContents content={showingReference ? digest : bodyContent} />++ <ModuleHistory metadata={page.metadata} />{intelligenceSlot}diff --git a/packages/ui/src/docs/module-history.tsx b/packages/ui/src/docs/module-history.tsxnew file mode 100644index 000000000..fec13cab0--- /dev/null+++ b/packages/ui/src/docs/module-history.tsx@@ -0,0 +1,109 @@+/**+ * Git history for the files a module page covers: who maintains them, where+ * change and fixes concentrate, and what moves with them. Read from the+ * `module_signals` the generator stamps on the page, the same data the page's+ * agent digest states in prose, so the two cannot disagree.+ */++import { cn } from "../lib/cn";++interface ModuleSignals {+ files?: number;+ hotspots?: number;+ bus_factor_one?: number;+ bug_fixes?: number;+ most_fixed_file?: string;+ owners?: { name: string; files: number }[];+ most_active_file?: string;+ most_active_commits_90d?: number;+ co_changes?: { path: string; files: number }[];+}++function readSignals(metadata: Record<string, unknown> | undefined): ModuleSignals | null {+ const raw = metadata?.module_signals;+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;+ return raw as ModuleSignals;+}++function basename(path: string): string {+ return path.split("/").filter(Boolean).pop() ?? path;+}++function Row({+ label,+ value,+ title,+ mono = true,+}: {+ label: string;+ value: string;+ title?: string | undefined;+ /** Counts and paths are machine facts; a person's name is not. */+ mono?: boolean;+}) {+ const full = title ?? value;+ return (+ <div className="flex items-baseline justify-between gap-3">+ <span className="shrink-0 text-[var(--color-text-tertiary)]">{label}</span>+ {/* The visible value truncates; the full one stays readable to assistive tech. */}+ <span+ className={cn("min-w-0 truncate text-right tabular-nums", mono && "font-mono")}+ title={full}+ aria-hidden+ >+ {value}+ </span>+ <span className="sr-only">{full}</span>+ </div>+ );+}++export function ModuleHistory({ metadata }: { metadata: Record<string, unknown> | undefined }) {+ const s = readSignals(metadata);+ if (!s) return null;+ const of = s.files ? ` of ${s.files}` : "";+ const owner = s.owners?.[0];+ const coChange = s.co_changes?.[0];+ return (+ <div>+ <p className="mb-2.5 font-mono text-[10px] uppercase tracking-[0.12em] text-[var(--color-text-tertiary)]">+ History+ </p>+ <div className="space-y-1 text-xs text-[var(--color-text-secondary)]">+ {owner && (+ <Row+ label="Owner"+ mono={false}+ value={`${owner.name} · ${owner.files}${of} files`}+ title={s.owners!.map((o) => `${o.name} (${o.files} files)`).join(", ")}+ />+ )}+ {s.hotspots ? <Row label="Hotspots" value={`${s.hotspots}${of} files`} /> : null}+ {s.bus_factor_one ? (+ <Row label="Single maintainer" value={`${s.bus_factor_one}${of} files`} />+ ) : null}+ {s.bug_fixes ? (+ <Row+ label="Bug fixes"+ value={s.most_fixed_file ? `${s.bug_fixes} · most in ${basename(s.most_fixed_file)}` : `${s.bug_fixes}`}+ title={s.most_fixed_file ? `${s.bug_fixes} bug-fix commits, most in ${s.most_fixed_file}` : undefined}+ />+ ) : null}+ {s.most_active_file && (+ <Row+ label="Most active"+ value={`${basename(s.most_active_file)} · ${s.most_active_commits_90d ?? 0} in 90d`}+ title={`${s.most_active_file}: ${s.most_active_commits_90d ?? 0} commits in 90 days`}+ />+ )}+ {coChange && (+ <Row+ label="Changes with"+ value={coChange.path}+ title={s.co_changes!.map((c) => c.path).join(", ")}+ />+ )}+ </div>+ </div>+ );+}diff --git a/packages/ui/src/docs/reader-persona.ts b/packages/ui/src/docs/reader-persona.tsindex ebc90fe11..c6cbd689d 100644--- a/packages/ui/src/docs/reader-persona.ts+++ b/packages/ui/src/docs/reader-persona.ts@@ -25,50 +25,29 @@ export function isReaderPersona(value: string | null | undefined): value is Readreturn value === "overview" || value === "contributor" || value === "deep";}-// Heading keywords (lowercased, matched as a prefix of the heading text) that-// each persona hides. Matching is intentionally generous so it works across-// the file/module/layer page templates without an exhaustive list.+// Whole section headings (lowercased) that each persona hides. Exact matches+// only: a model-written page names its sections after what the code does, and+// a prefix such as "source" or "symbol" would delete a section called+// "Source files become symbols". Each entry is a heading a structural template+// emits (see ``structural_labels.py`` and the stub templates).-// Contributor keeps almost everything; only the rawest machine-oriented dumps-// are hidden so the page stays readable.-//-// The last three are retrieval scaffolding rather than reference material. A-// deterministic page carries them so the index has the words a question is-// asked in — the vocabulary bag and the question block are the only parts of-// a file page written in a reader's words rather than in paths, and the third-// is the importer list a symbol page keeps once its resolved callers have-// answered the question. They earn their place in `content`, which is one-// string that FTS indexes, the vector store embeds and `get_context` returns-// verbatim; they do not earn a reader's scroll. Hidden here rather than-// dropped upstream so the index keeps them.-const CONTRIBUTOR_HIDE = [- "raw metrics",- "parse errors",- "source snippet",- "in the code",- "questions this page answers",- "files importing this module",-];+// Contributor hides only retrieval scaffolding a file or symbol page still+// carries in `content`: the question block, written so the index has the words+// a question is asked in, and the importer list a symbol page keeps once its+// resolved callers have answered the question. Pages with an agent digest+// carry that material outside `content`, and the reader offers it as the+// Reference view instead.+const CONTRIBUTOR_HIDE = ["questions this page answers", "files importing this module"];-// Overview is contributor plus the reference dumps. Spread rather than-// restated: a section the balanced lens hides must never reappear in the-// narrower one.+// Overview is contributor plus the reference dumps. Spread, not restated: a+// section the balanced lens hides must never reappear in the narrower one.const OVERVIEW_HIDE = [...CONTRIBUTOR_HIDE,- "symbols",- "symbol","public api",- "imports",- "exports",- "dependencies",- "dependents",- "call graph",- "class hierarchy",- "community",- "neighbors",- "dead code","source",- "metrics",+ "symbols defined in the cycle",+ "dependencies (modules this imports)",+ "dependents (modules that import this)",];const HIDE_BY_PERSONA: Record<ReaderPersona, string[]> = {@@ -78,8 +57,7 @@ const HIDE_BY_PERSONA: Record<ReaderPersona, string[]> = {};function headingHidden(headingText: string, hideList: string[]): boolean {- const h = headingText.trim().toLowerCase();- return hideList.some((kw) => h === kw || h.startsWith(kw));+ return hideList.includes(headingText.trim().toLowerCase());}/**diff --git a/packages/web/src/components/docs/docs-viewer.tsx b/packages/web/src/components/docs/docs-viewer.tsxindex 56ac2d455..29d9f0e8b 100644--- a/packages/web/src/components/docs/docs-viewer.tsx+++ b/packages/web/src/components/docs/docs-viewer.tsx@@ -304,11 +304,14 @@ export function DocsViewer({[repoId],);- const hasTargetPath = !!page?.target_path;const targetPath = page?.target_path ?? "";+ // Decided by page type, not path shape: a module page's target is a+ // directory, which every row below would answer with nothing, and the+ // reader already gives module pages their own history card.+ const showSignals = !!targetPath && page?.page_type !== "module_page";// Security findings only make sense for real source files.const isFilePath =- hasTargetPath &&+ showSignals &&!targetPath.includes("::") &&!targetPath.startsWith("onboarding/") &&!targetPath.startsWith("layer:");@@ -347,7 +350,7 @@ export function DocsViewer({) : undefined}intelligenceSlot={- hasTargetPath ? (+ showSignals ? (// One "Signals" block, not five. At a glance, Importance, Community,// Call graph and Security each announced themselves with their own// uppercase label over a handful of rows, so the rail was mostlydiff --git a/tests/unit/cli/test_doctor_page_cap.py b/tests/unit/cli/test_doctor_page_cap.pyindex 2266c612c..ed3b995ca 100644--- a/tests/unit/cli/test_doctor_page_cap.py+++ b/tests/unit/cli/test_doctor_page_cap.py@@ -113,7 +113,7 @@ async def _build_repo(tmp_path: Path) -> Path:fts = FullTextSearch(engine)await fts.ensure_index()await fts.index_many(- [(r["id"], r["title"], r["content"], "", r["target_path"]) for r in rows]+ [(r["id"], r["title"], r["content"], "", r["target_path"], "") for r in rows])await engine.dispose()return repo_pathdiff --git a/tests/unit/cli/test_embedder_resolution.py b/tests/unit/cli/test_embedder_resolution.pyindex 23e8c092d..0cb7f4b5a 100644--- a/tests/unit/cli/test_embedder_resolution.py+++ b/tests/unit/cli/test_embedder_resolution.py@@ -217,6 +217,7 @@ async def test_reindex_persists_its_resolved_embedder(# Non-nullable on the real model, so a stand-in that omits it is a# stand-in for a page that cannot exist.summary = "what a.py does"+ digest = ""metadata_json = "{}"class _Result:diff --git a/tests/unit/cli/test_full_upgrade_progress.py b/tests/unit/cli/test_full_upgrade_progress.pyindex bdf52dd36..902eb4cef 100644--- a/tests/unit/cli/test_full_upgrade_progress.py+++ b/tests/unit/cli/test_full_upgrade_progress.py@@ -99,6 +99,7 @@ def test_full_upgrade_fts_cleanup_retries_after_failure(content="# Current",summary="Current page",target_path="current.py",+ digest="",)with pytest.raises(RuntimeError, match="temporary FTS failure"):diff --git a/tests/unit/cli/test_reindex_cmd.py b/tests/unit/cli/test_reindex_cmd.pyindex 88c74f860..c6fbbc248 100644--- a/tests/unit/cli/test_reindex_cmd.py+++ b/tests/unit/cli/test_reindex_cmd.py@@ -128,6 +128,7 @@ async def test_reindex_aborts_when_every_item_failed(target_path = "src/main.py"summary = ""content = "body"+ digest = ""metadata_json = "{}"decision = "a decision"diff --git a/tests/unit/cli/test_reindex_embed_recipe.py b/tests/unit/cli/test_reindex_embed_recipe.pyindex f4ff02e4b..8bee8d1fd 100644--- a/tests/unit/cli/test_reindex_embed_recipe.py+++ b/tests/unit/cli/test_reindex_embed_recipe.py@@ -24,6 +24,7 @@ class _DummyEngine:class _Page:metadata_json = "{}" # non-nullable on the real model+ digest = "" # non-nullable on the real modeldef __init__(self, **kw: Any) -> None:self.__dict__.update(kw)diff --git a/tests/unit/generation/test_agent_digest.py b/tests/unit/generation/test_agent_digest.pynew file mode 100644index 000000000..0daadcb5a--- /dev/null+++ b/tests/unit/generation/test_agent_digest.py@@ -0,0 +1,204 @@+"""A module page keeps agent material in its digest, off the body a reader reads.++The questions a page answers, its identifiers and its git signals help search+and agents; a reader scrolled past them. They now live in ``page.digest``,+which is indexed and served beside the body, so the body can be prose.+"""++from __future__ import annotations++import networkx as nx++from repowise.core.generation.agent_digest import (+ MODULE_SIGNALS_KEY,+ rejoin_questions,+ split_questions,+)+from repowise.core.generation.context_assembler import ContextAssembler, FilePageContext+from repowise.core.generation.models import GenerationConfig, compute_page_id+from repowise.core.generation.page_generator import PageGenerator, PriorPage+from repowise.core.providers.llm.mock import MockProvider++_BODY = "# Parsing\n\nThe parser turns files into records.\n\n## How a file is read\n\nIt reads.\n"+_QUESTIONS = "## Questions this page answers\n\n- How is a file parsed?\n- Where do records go?"+++class _Provider(MockProvider):+ async def generate(self, *args, **kwargs):+ response = await super().generate(*args, **kwargs)+ response.content = f"{_BODY}\n{_QUESTIONS}\n"+ return response+++def _file_context(path: str) -> FilePageContext:+ return FilePageContext(+ file_path=path,+ language="python",+ docstring=None,+ symbols=[+ {+ "name": "Parser",+ "qualified_name": "Parser",+ "kind": "class",+ "signature": "class Parser",+ "docstring": "",+ "visibility": "public",+ "is_async": False,+ "complexity_estimate": 1,+ "decorators": [],+ "parent_name": "",+ "start_line": 1,+ "end_line": 9,+ }+ ],+ imports=[],+ exports=["Parser"],+ pagerank_score=0.3,+ betweenness_score=0.0,+ community_id=0,+ dependents=[],+ dependencies=[],+ is_api_contract=False,+ is_entry_point=False,+ is_test=False,+ parse_errors=[],+ estimated_tokens=10,+ )+++_GIT = {+ "pkg/parser.py": {+ "file_path": "pkg/parser.py",+ "is_hotspot": True,+ "bus_factor": 1,+ "prior_defect_count": 3,+ "primary_owner_name": "Ada",+ "commit_count_90d": 7,+ }+}+++def _generator(config: GenerationConfig, **kwargs) -> PageGenerator:+ return PageGenerator(_Provider(), ContextAssembler(config), config, **kwargs)+++async def _module_page(gen: PageGenerator, git_meta_map=None):+ return await gen.generate_module_page(+ "Parsing",+ "python",+ [_file_context("pkg/parser.py")],+ nx.DiGraph(),+ git_meta_map=git_meta_map,+ target_path="pkg",+ structural_key="k",+ )+++def test_split_moves_the_questions_and_keeps_the_body():+ body, questions = split_questions(f"{_BODY}\n{_QUESTIONS}\n\n## After\n\nMore prose.\n")++ assert questions == _QUESTIONS+ assert "Questions this page answers" not in body+ assert "## How a file is read" in body and "## After" in body+++def test_split_ignores_a_heading_inside_a_code_fence():+ content = "# T\n\n```md\n## Questions this page answers\n```\n"++ assert split_questions(content) == (content, "")+++def test_rejoin_restores_the_response_a_body_was_split_from():+ body, questions = split_questions(f"{_BODY}\n{_QUESTIONS}\n")++ assert split_questions(rejoin_questions(body, questions)) == (body, questions)+ assert rejoin_questions(body, "") == body+++async def test_the_model_questions_move_into_the_digest(sample_config):+ page = await _module_page(_generator(sample_config))++ assert "Questions this page answers" not in page.content+ assert page.digest.startswith(_QUESTIONS)+ assert "## Concept index" in page.digest+ assert page.summary.startswith("The parser turns files into records")+++async def test_git_signals_reach_the_digest_and_the_card_not_the_body(sample_config):+ page = await _module_page(_generator(sample_config), git_meta_map=_GIT)++ signals = page.metadata[MODULE_SIGNALS_KEY]+ assert signals["files"] == 1+ assert signals["hotspots"] == 1+ assert signals["bus_factor_one"] == 1+ assert signals["bug_fixes"] == 3+ assert signals["owners"] == [{"name": "Ada", "files": 1}]+ assert "## Signals" in page.digest+ assert "Bug-fix commits: 3, most in `pkg/parser.py`" in page.digest+ assert "hotspot" not in page.content.lower()+++async def test_a_module_with_no_history_reports_no_signals(sample_config):+ page = await _module_page(_generator(sample_config))++ assert MODULE_SIGNALS_KEY not in page.metadata+ assert "## Signals" not in page.digest+++async def test_the_keyless_page_splits_the_same_way():+ config = GenerationConfig(+ max_tokens=1024, token_budget=2000, max_concurrency=2, deterministic=True+ )+ page = await _module_page(_generator(config), git_meta_map=_GIT)++ assert "Questions this page answers" not in page.content+ assert page.digest.startswith("## Questions this page answers")+ assert "Subsystem health" not in page.content+ assert page.metadata[MODULE_SIGNALS_KEY]["hotspots"] == 1+++async def test_a_page_reused_from_a_prior_run_keeps_its_questions(sample_config):+ first = await _module_page(_generator(sample_config))+ prior = {+ compute_page_id("module_page", "pkg"): PriorPage(+ source_hash=first.source_hash,+ model_name=first.model_name,+ content=first.content,+ digest=first.digest,+ )+ }++ again = await _module_page(_generator(sample_config, prior_pages=prior))++ assert again.metadata.get("reused_from_prior_run") is True+ assert again.content == first.content+ assert again.digest == first.digest+++async def test_a_reused_rollup_keeps_one_package_table(sample_config):+ async def rollup(gen):+ return await gen.generate_module_page(+ "Parsing",+ "python",+ [_file_context("pkg/parser.py")],+ nx.DiGraph(),+ target_path="pkg",+ structural_key="k",+ packages=[{"path": "pkg/a", "files": 1}, {"path": "pkg/b", "files": 2}],+ )++ page = await rollup(_generator(sample_config))+ for _ in range(2):+ prior = {+ page.page_id: PriorPage(+ source_hash=page.source_hash,+ model_name=page.model_name,+ content=page.content,+ digest=page.digest,+ )+ }+ page = await rollup(_generator(sample_config, prior_pages=prior))++ assert page.metadata.get("reused_from_prior_run") is True+ assert page.content.count("## Packages") == 1+ assert "| `pkg/b` | 2 |" in page.contentdiff --git a/tests/unit/generation/test_concept_index_reaches_search.py b/tests/unit/generation/test_concept_index_reaches_search.pyindex d4e53c30b..df44d5e5d 100644--- a/tests/unit/generation/test_concept_index_reaches_search.py+++ b/tests/unit/generation/test_concept_index_reaches_search.py@@ -1,11 +1,11 @@-"""The concept index has to reach the index, not just the rendered page.+"""The concept index has to reach the index, not just the page's digest.The table exists to make an identifier-exact query match the identifier rather-than the model's description of it. That only happens if the table text lands-in the full-text row. Nothing raises if it does not: the write path is handed-``page.content`` verbatim, so a table appended after the row was written, or-stripped on the way to persistence, would show up as a page that reads well and-cannot be found by the one token a reader would search for.+than the model's description of it. It lives in the page's agent digest, off+the body a reader scrolls, so it only helps if the digest lands in the+full-text row. Nothing raises if it does not: a digest dropped on the way to+persistence would show up as a page that reads well and cannot be found by the+one token a reader would search for.Module pages are the page type with the least to match on — a real run produced89 of them carrying four fenced code blocks between them — so this round trip@@ -90,7 +90,7 @@ def _file_context(path: str, names: list[str]) -> FilePageContext:)-async def _index(engine, search, *, page_id, title, target_path, content):+async def _index(engine, search, *, page_id, title, target_path, content, digest):factory = async_sessionmaker(engine, expire_on_commit=False)async with factory() as session:repo = await upsert_repository(session, name="r", local_path="/tmp/r")@@ -103,13 +103,16 @@ async def _index(engine, search, *, page_id, title, target_path, content):title=title,content=content,summary="",+ digest=digest,target_path=target_path,source_hash="h",model_name="mock",provider_name="mock",)await session.commit()- await search.index(page_id, title, content, summary="", target_path=target_path)+ await search.index(+ page_id, title, content, summary="", target_path=target_path, digest=digest+ )async def _matches(search: FullTextSearch, query: str) -> list[str]:@@ -129,9 +132,10 @@ async def test_a_module_page_is_findable_by_a_symbol_it_contains(fts):target_path="core/resolvers",structural_key="k",)- # The prose the model wrote does not contain the identifier, so a hit below- # can only have come from the appended table.- assert "ResolverContext" not in page.content.split("## Concept index")[0]+ # The body does not contain the identifier, so a hit below can only have+ # come from the digest's concept index.+ assert "ResolverContext" not in page.content+ assert "## Concept index" in page.digestawait _index(engine,@@ -140,6 +144,7 @@ async def test_a_module_page_is_findable_by_a_symbol_it_contains(fts):title=page.title,target_path=page.target_path,content=page.content,+ digest=page.digest,)assert page.page_id in await _matches(search, "ResolverContext")@@ -167,6 +172,7 @@ async def test_the_prose_spelling_finds_the_page_too(fts):title=page.title,target_path=page.target_path,content=page.content,+ digest=page.digest,)assert page.page_id in await _matches(search, "Tsconfig resolver")diff --git a/tests/unit/generation/test_concept_index_report.py b/tests/unit/generation/test_concept_index_report.pyindex c05789020..f01a3a1bd 100644--- a/tests/unit/generation/test_concept_index_report.py+++ b/tests/unit/generation/test_concept_index_report.py@@ -1,7 +1,7 @@"""The run reports how far the module concept index reached.-The table is appended in code, after the provider call, which is a quieter-place to lose something than a template. An early return on an error path, or a+The table is rendered into the page's digest in code, after the provider+call, which is a quieter place to lose something than a template. An early return on an error path, or arefactor that hands back the page before the append, leaves every module pagepure prose again — and every template test in the suite still passes, becausenone of the templates changed. This counter is the only thing that would say so.@@ -17,20 +17,20 @@ from repowise.core.generation.models import GeneratedPagefrom repowise.core.generation.report import CONCEPT_INDEX_HEADING, GenerationReport_WITH = (- "# Resolution Layer\n\nProse.\n\n"f"{CONCEPT_INDEX_HEADING}\n\n""| Concept | Symbol | File |\n| --- | --- | --- |\n""| Resolver context | `ResolverContext` | `a.py` |\n")-_WITHOUT = "# Resolution Layer\n\nProse.\n"+_WITHOUT = ""-def _page(page_id: str, page_type: str, content: str) -> GeneratedPage:+def _page(page_id: str, page_type: str, digest: str) -> GeneratedPage:return GeneratedPage(page_id=page_id,page_type=page_type,title=page_id,- content=content,+ content="# Resolution Layer\n\nProse.\n",+ digest=digest,source_hash="",model_name="mock",provider_name="mock",diff --git a/tests/unit/generation/test_deterministic_templates.py b/tests/unit/generation/test_deterministic_templates.pyindex 362dcda3a..aa3855fba 100644--- a/tests/unit/generation/test_deterministic_templates.py+++ b/tests/unit/generation/test_deterministic_templates.py@@ -446,7 +446,7 @@ def test_module_page_leads_with_the_concept_title(generator):group is; the line beneath says where to go and look."""ctx = _module_ctx(generator._assembler, ["src/ingest/read.py", "src/parse/ast.py"])- page = generator._stub_module_page(ctx, "src/ingest", "Ingestion Pipeline", None)+ page = generator._stub_module_page(ctx, "src/ingest", "Ingestion Pipeline")assert page.title == "Ingestion Pipeline"assert page.content.startswith("# Ingestion Pipeline\n")@@ -463,7 +463,7 @@ def test_module_page_of_root_files_says_so(generator):top level."""ctx = _module_ctx(generator._assembler, ["setup.py", "main.py"])- page = generator._stub_module_page(ctx, "root", "Project Entry Points", None)+ page = generator._stub_module_page(ctx, "root", "Project Entry Points")assert "Repository root" in page.contentassert "`.`" not in page.contentdiff --git a/tests/unit/generation/test_module_page_concept_index.py b/tests/unit/generation/test_module_page_concept_index.pyindex d5116f184..c329203c7 100644--- a/tests/unit/generation/test_module_page_concept_index.py+++ b/tests/unit/generation/test_module_page_concept_index.py@@ -7,10 +7,11 @@ grep for, and an identifier-exact query matches the description of the coderather than the code.The concept index closes that. It is rendered from the assembled symbol data-and appended after the model has written the page, so the identifiers and the-paths in it are the index's, not the model's, and no provider response can-change them. That placement is the whole point: put the same table in the-prompt and the model may reformat it, abbreviate a path, or drop it.+into the page's agent digest after the model has written the page, so the+identifiers and the paths in it are the index's, not the model's, and no+provider response can change them. That placement is the whole point: put the+same table in the prompt and the model may reformat it, abbreviate a path, or+drop it.A page whose members export nothing renders no table rather than an empty one —a header row with no rows under it is a claim that the module has no public@@ -140,13 +141,13 @@ async def test_rendered_page_carries_real_symbol_names_and_paths("""The identifiers reach the page the model wrote, spelled exactly."""page = await _module_page(prose_only_generator, resolver_module.contexts)- assert CONCEPT_INDEX_HEADING in page.content- assert "| Concept | Symbol | File |" in page.content- assert "`ResolverContext`" in page.content- assert "`resolve_imports`" in page.content- assert "`TsconfigResolver`" in page.content- assert "`packages/core/src/repowise/core/ingestion/resolvers/context.py`" in page.content- assert "`packages/core/src/repowise/core/ingestion/resolvers/tsconfig.py`" in page.content+ assert CONCEPT_INDEX_HEADING in page.digest+ assert "| Concept | Symbol | File |" in page.digest+ assert "`ResolverContext`" in page.digest+ assert "`resolve_imports`" in page.digest+ assert "`TsconfigResolver`" in page.digest+ assert "`packages/core/src/repowise/core/ingestion/resolvers/context.py`" in page.digest+ assert "`packages/core/src/repowise/core/ingestion/resolvers/tsconfig.py`" in page.digestasync def test_concept_column_spells_the_identifier_as_prose(prose_only_generator, resolver_module):@@ -154,14 +155,14 @@ async def test_concept_column_spells_the_identifier_as_prose(prose_only_generatoon the right, both on one line so either wording finds the page."""page = await _module_page(prose_only_generator, resolver_module.contexts)- assert "| Resolver context | `ResolverContext` |" in page.content- assert "| Resolve imports | `resolve_imports` |" in page.content+ assert "| Resolver context | `ResolverContext` |" in page.digest+ assert "| Resolve imports | `resolve_imports` |" in page.digestasync def test_non_public_symbols_stay_off_the_table(prose_only_generator, resolver_module):page = await _module_page(prose_only_generator, resolver_module.contexts)- assert "_private_helper" not in page.content+ assert "_private_helper" not in page.digestasync def test_a_module_with_no_public_symbols_renders_no_table(prose_only_generator):@@ -173,18 +174,18 @@ async def test_a_module_with_no_public_symbols_renders_no_table(prose_only_generpage = await _module_page(prose_only_generator, contexts)- assert CONCEPT_INDEX_HEADING not in page.content- assert "| Concept | Symbol | File |" not in page.content+ assert CONCEPT_INDEX_HEADING not in page.digest+ assert "| Concept | Symbol | File |" not in page.digest-async def test_the_table_starts_on_its_own_line(prose_only_generator, resolver_module):- """Appending is string concatenation, and a heading glued to the last line- of the model's prose renders as body text. Assert the separator that has to- be there, not merely the absence of extra ones."""+async def test_the_table_stays_off_the_body(prose_only_generator, resolver_module):+ """The body is what a reader scrolls; the table is for search and agents."""page = await _module_page(prose_only_generator, resolver_module.contexts)- assert f"\n\n{CONCEPT_INDEX_HEADING}\n" in page.content- assert "\n\n\n" not in page.content+ assert CONCEPT_INDEX_HEADING not in page.content+ assert "`ResolverContext`" not in page.content+ assert page.digest.startswith(CONCEPT_INDEX_HEADING)+ assert "\n\n\n" not in page.digestdef test_concept_index_orders_by_the_page_rank_of_the_file():diff --git a/tests/unit/persistence/test_information_floor.py b/tests/unit/persistence/test_information_floor.pyindex cfcb9f7e8..f3da3735c 100644--- a/tests/unit/persistence/test_information_floor.py+++ b/tests/unit/persistence/test_information_floor.py@@ -290,8 +290,8 @@ def test_the_floor_is_off_unless_it_is_set(monkeypatch):async def test_a_batch_holds_out_only_the_thin_pages(fts):await fts.index_many([- ("file_page:esbuild.mjs", "File: esbuild.mjs", SKELETON, "", "esbuild.mjs"),- ("file_page:search.py", "File: search.py", SUBSTANTIAL, "", "search.py"),+ ("file_page:esbuild.mjs", "File: esbuild.mjs", SKELETON, "", "esbuild.mjs", ""),+ ("file_page:search.py", "File: search.py", SUBSTANTIAL, "", "search.py", ""),])@@ -303,7 +303,7 @@ async def test_a_batch_drops_the_row_of_a_page_that_thinned_out(fts):"""The delete runs for every id in the batch, insert only for those above."""await fts.index("file_page:a.py", "File: a.py", SUBSTANTIAL, summary="", target_path="a.py")- await fts.index_many([("file_page:a.py", "File: a.py", SKELETON, "", "a.py")])+ await fts.index_many([("file_page:a.py", "File: a.py", SKELETON, "", "a.py", "")])assert await fts.list_indexed_ids() == set()assert await fts.search("FTS5 virtual table") == []@@ -313,8 +313,8 @@ async def test_a_batch_whose_last_entry_is_thin_leaves_no_row(fts):"""Last-entry-wins has to hold when the last entry is the excluded one."""await fts.index_many([- ("file_page:a.py", "File: a.py", SUBSTANTIAL, "", "a.py"),- ("file_page:a.py", "File: a.py", SKELETON, "", "a.py"),+ ("file_page:a.py", "File: a.py", SUBSTANTIAL, "", "a.py", ""),+ ("file_page:a.py", "File: a.py", SKELETON, "", "a.py", ""),])diff --git a/tests/unit/persistence/test_migration_fts_frozen.py b/tests/unit/persistence/test_migration_fts_frozen.pynew file mode 100644index 000000000..01aae1b16--- /dev/null+++ b/tests/unit/persistence/test_migration_fts_frozen.py@@ -0,0 +1,59 @@+"""Migrations that touch full-text search carry their own SQL.++A migration that imports the live search constants replays today's column set+on a database migrated from an older revision, where those columns do not+exist yet. Rendered offline against PostgreSQL, each revision has to create+exactly the index its own schema supports.+"""++from __future__ import annotations++import io+from pathlib import Path++import pytest+from alembic import command+from alembic.config import Config++_CORE = Path(__file__).resolve().parents[3] / "packages" / "core"+_VERSIONS = _CORE / "alembic" / "versions"+++def _offline_sql(monkeypatch, revisions: str) -> str:+ monkeypatch.setenv("DATABASE_URL", "postgresql://user:pass@localhost/db")+ buffer = io.StringIO()+ # No ini file: env.py would hand it to logging.fileConfig, which disables+ # every logger already created and breaks log assertions in later tests.+ config = Config(output_buffer=buffer)+ config.set_main_option("script_location", str(_CORE / "alembic"))+ command.upgrade(config, revisions, sql=True)+ return buffer.getvalue()+++def test_no_migration_imports_the_live_search_constants():+ offenders = [+ p.name for p in _VERSIONS.glob("*.py") if "repowise.core.persistence.search" in p.read_text()+ ]+ assert offenders == []+++ ("revisions", "has_digest"), [("0043:0044", False), ("0094:0095", True)]+)+def test_each_revision_builds_the_index_its_schema_supports(monkeypatch, revisions, has_digest):+ sql = _offline_sql(monkeypatch, revisions)+ index = next(line for line in sql.splitlines() if "CREATE INDEX idx_wiki_pages_fts" in line)++ assert "COALESCE(target_path,'')" in index+ assert ("COALESCE(digest,'')" in index) is has_digest+++def test_the_sqlite_shapes_around_the_digest_keep_the_vocabulary():+ import importlib.util++ spec = importlib.util.spec_from_file_location("m0095", _VERSIONS / "0095_page_digest.py")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)++ assert "target_path, vocabulary, digest)" in module.PAGE_FTS_DDL+ assert "target_path, vocabulary)" in module._OLD_SQLITE_DDLdiff --git a/tests/unit/persistence/test_search.py b/tests/unit/persistence/test_search.pyindex bdc2916c6..315439d42 100644--- a/tests/unit/persistence/test_search.py+++ b/tests/unit/persistence/test_search.py@@ -157,7 +157,7 @@ async def _indexed_ids(fts) -> set[str]:async def test_index_many_writes_the_same_rows_as_indexing_one_at_a_time(async_engine):"""The batch path is the single-page path, so the two must not diverge."""pages = [- (f"p{i}", f"Title {i}", f"content about widgets number {i}", f"sum {i}", f"a/b{i}.py")+ (f"p{i}", f"Title {i}", f"content about widgets number {i}", f"sum {i}", f"a/b{i}.py", "")for i in range(20)]@@ -183,7 +183,7 @@ async def test_index_many_spans_more_ids_than_one_statement_can_bind(fts):A wiki is thousands of pages, so the first real corpus would have been thefirst test of this path."""- pages = [(f"p{i}", f"Title {i}", "a page about chunking", "", f"{i}.py") for i in range(1200)]+ pages = [(f"p{i}", f"Title {i}", "a page about chunking", "", f"{i}.py", "") for i in range(1200)]await fts.index_many(pages)assert len(await _indexed_ids(fts)) == 1200@@ -197,8 +197,8 @@ async def test_a_page_id_repeated_in_one_batch_keeps_its_last_entry(fts):"""Every id is deleted before any is inserted, so a duplicate would double."""await fts.index_many([- ("p1", "First", "a page about alpacas", "", "a.py"),- ("p1", "Second", "a page about zebras", "", "a.py"),+ ("p1", "First", "a page about alpacas", "", "a.py", ""),+ ("p1", "Second", "a page about zebras", "", "a.py", ""),])@@ -246,6 +246,7 @@ async def test_path_only_match_outranks_content_only_match(fts):"An unrelated module about parsing and graphs. " * 3,"Filler.",f"packages/core/other/mod{i}.py",+ "",)for i in range(10)]@@ -334,6 +335,7 @@ async def test_index_pages_scans_the_index_once_per_id_chunk(async_engine, fts):content=f"Module {i} documents widget number {i}.",summary=f"summary {i}",target_path=f"m{i}.py",+ digest="",)for i in range(1100)]diff --git a/tests/unit/persistence/test_search_fts_columns.py b/tests/unit/persistence/test_search_fts_columns.pyindex ce74407be..a1141e9d3 100644--- a/tests/unit/persistence/test_search_fts_columns.py+++ b/tests/unit/persistence/test_search_fts_columns.py@@ -66,7 +66,7 @@ async def repo(async_session):async def test_ensure_index_upgrades_old_schema_and_backfills(async_engine, async_session, repo):- """An index written by an older repowise gains both columns, populated.+ """An index written by an older repowise gains the new columns, populated.The rebuild reads ``wiki_pages``, which is the only place the two newfields exist — the old FTS rows never carried them. A silent no-op here@@ -390,3 +390,23 @@ async def test_unreadable_page_metadata_indexes_without_a_vocabulary()assert [r.page_id for r in await fts.search("entry point")] == ["file_page:src/main.py"]+++async def test_the_vocabulary_and_the_digest_are_both_searchable(+ async_engine, async_session, repo+):+ """Two columns kept off the rendered body, one per kind of reader-invisible text."""+ await _seed_page(async_session, repo.id, **_VOCABULARY_PAGE)+ fts = FullTextSearch(async_engine)+ await fts.ensure_index()+ await fts.index(+ "file_page:src/main.py",+ "File: src/main.py",+ _VOCABULARY_PAGE["content"],+ summary="",+ target_path="src/main.py",+ digest="## Questions this page answers\n\n- Where does the zeppelin start?",+ )++ assert [r.page_id for r in await fts.search("xylophone")] == ["file_page:src/main.py"]+ assert [r.page_id for r in await fts.search("zeppelin")] == ["file_page:src/main.py"]