Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

feat(wiki): keep agent material in a page digest beside the body (#2739)

Raghav Chamadiya2f76dbc3d607
diff --git a/docs/agent/MCP_TOOLS.md b/docs/agent/MCP_TOOLS.md
index c05c211eb..0702fc5c5 100644
--- a/docs/agent/MCP_TOOLS.md
+++ b/docs/agent/MCP_TOOLS.md
@@ -170,7 +170,7 @@ A triage card for files, modules or symbols: summary, symbols with signatures an
A file's compact symbol list holds its top 15 symbols: types (classes, interfaces, structs, traits, enums, type aliases, impls, modules) first, then functions and methods, then the rest, each group by centrality. `symbols_truncated` gives the total, and `include=["symbols"]` lists them all. When the response budget trims the list further, it keeps symbols by kind and name match, not centrality. A row without `symbol_id` is `path::name`; methods and overload variants carry theirs, so pass a row's id to `get_symbol` when it has one.
-`skeleton` renders a file with bodies elided: every signature, the imports, and the bodies of its most central symbols, with line ranges on every elision. `doc_drift` lists the documents that name the file and whether they carry drift. An empty `callers` or `callees` list comes with a `*_basis` saying how much of that language's calls the graph resolved; read it before concluding nothing calls a symbol.
+`full_doc` returns the page as `content_md`, plus `digest_md` on pages that carry an agent digest: the questions the page answers, its concept index, public API and git signals. Both are dropped first when the response is over budget. `skeleton` renders a file with bodies elided: every signature, the imports, and the bodies of its most central symbols, with line ranges on every elision. `doc_drift` lists the documents that name the file and whether they carry drift. An empty `callers` or `callees` list comes with a `*_basis` saying how much of that language's calls the graph resolved; read it before concluding nothing calls a symbol.
```
get_context(targets=["src/auth/middleware.ts", "src/api/routes.ts"], include=["callers"])
diff --git a/docs/layers/WIKI.md b/docs/layers/WIKI.md
index 57c9a3ce5..c1a7b71e5 100644
--- a/docs/layers/WIKI.md
+++ b/docs/layers/WIKI.md
@@ -262,7 +262,10 @@ Changing the language later does not retranslate existing pages. Re-run
SQLite full-text and vector results through Reciprocal Rank Fusion, biased by
PageRank, then expands one to two hops along the imports and projected-calls
graph for flow-shaped questions. Full-text works with no embedder; semantic
- search needs one configured.
+ search needs one configured. A module page's agent digest (the questions it
+ answers, its concept index, public API and git signals) is stored beside the
+ page body, indexed with it, and returned by `get_context` as `digest_md`
+ with `include=["full_doc"]`.
- **CLI**: `repowise ask`, `repowise search`, `repowise context`.
- **Dashboard**: `repowise serve`, then the Docs tab.
- **Files**: `repowise export --format markdown|html|json`, written to
diff --git a/docs/start/DASHBOARD.md b/docs/start/DASHBOARD.md
index 034b1bb50..340fe8535 100644
--- a/docs/start/DASHBOARD.md
+++ b/docs/start/DASHBOARD.md
@@ -61,7 +61,11 @@ The generated wiki: a page tree on the left, the rendered page in the middle
with mermaid diagrams, code blocks that link into the file views, a confidence
badge, a freshness indicator, backlinks, and a regenerate action for a single
page. A command palette (`⌘K` / `Ctrl+K`) jumps between pages. Reader personas
-re-filter prose for the audience you pick. You can attach human notes to a page,
+re-filter prose for the audience you pick. Module pages keep the material
+written for search and agents (the questions a page answers, its identifiers,
+public API and git signals) out of the body; the **Reference** tab above the
+page shows it, and a small History card beside the page summarises owners,
+hotspots and fix history. You can attach human notes to a page,
and those notes survive regeneration.
The sidebar auto-collapses to icons here so the page gets the width, and
diff --git a/packages/api-client/src/types/pages.ts b/packages/api-client/src/types/pages.ts
index 81bb4236f..8b6215938 100644
--- a/packages/api-client/src/types/pages.ts
+++ b/packages/api-client/src/types/pages.ts
@@ -52,6 +52,8 @@ export interface PageSummary {
export interface PageResponse extends PageSummary {
content: string;
+ /** Agent material kept off the page body; the reader shows it on request. */
+ digest?: string;
metadata: Record<string, unknown>;
}
diff --git a/packages/cli/src/repowise/cli/commands/context_cmd.py b/packages/cli/src/repowise/cli/commands/context_cmd.py
index 37b40c771..fc62ad265 100644
--- a/packages/cli/src/repowise/cli/commands/context_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/context_cmd.py
@@ -54,7 +54,7 @@ _REPLACED_KEYS = frozenset(
#: ``docs`` stays: for a **symbol** target the whole card lives there
#: (``signature``, ``docstring``, ``used_by``, ``candidates``), and for a file
#: target it holds ``symbols`` — the list a caller pipes into ``repowise
-#: symbol`` — plus ``content_md``/``documentation`` when ``--include full_doc``
+#: symbol``, plus ``content_md``/``digest_md``/``documentation`` when ``--include full_doc``
#: asked for them. Denylisting ``docs`` wholesale projected a symbol target to
#: an empty card and made ``--include full_doc`` an inert flag.
_LIFTED_DOC_KEYS = ("title", "summary")
diff --git a/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py b/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py
index f45bef3f2..05c9265ed 100644
--- a/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py
+++ b/packages/cli/src/repowise/cli/commands/doctor_cmd/repo_checks.py
@@ -142,7 +142,7 @@ async def _all_pages_for_reconciliation(session: object, repo_id: str) -> list:
was the live index.
Four columns rather than whole rows: the id to match against the stores,
- the content for the information floor, metadata_json for the stub
+ the content and digest for the information floor, metadata_json for the stub
predicate, and target_path to tell whether the file is excluded. Nothing downstream reads any other field, and hydrating full
ORM objects for every page only to discard them is what made a cap look
necessary in the first place.
@@ -152,7 +152,7 @@ async def _all_pages_for_reconciliation(session: object, repo_id: str) -> list:
from repowise.core.persistence.models import Page
result = await session.execute( # type: ignore[attr-defined]
- select(Page.id, Page.content, Page.metadata_json, Page.target_path).where(
+ select(Page.id, Page.content, Page.digest, Page.metadata_json, Page.target_path).where(
Page.repository_id == repo_id
)
)
@@ -511,7 +511,7 @@ def _run_repo_checks(
indexable_ids = {
p.id
for p in pages
- if meets_information_floor(p.content or "")
+ if meets_information_floor(p.content or "", digest=p.digest or "")
and not is_excluded(
(p.target_path or "").split("::", 1)[0], exclude_spec
)
@@ -838,7 +838,7 @@ def _run_repo_checks(
# ORM rows key on ``id``, not ``page_id``, so they go
# to index_many as tuples rather than to index_pages.
batch = [
- (p.id, p.title, p.content, p.summary, p.target_path)
+ (p.id, p.title, p.content, p.summary, p.target_path, p.digest)
for p in rows.scalars().all()
]
await fts.index_many(batch)
@@ -921,6 +921,7 @@ def _run_repo_checks(
summary=page.summary or "",
content=page.content or "",
page_metadata=page.metadata_json,
+ digest=page.digest or "",
)
if item is None:
# Below the information floor, so its absence
diff --git a/packages/cli/src/repowise/cli/commands/export_cmd.py b/packages/cli/src/repowise/cli/commands/export_cmd.py
index fbd7ad272..6e4f17fc2 100644
--- a/packages/cli/src/repowise/cli/commands/export_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/export_cmd.py
@@ -299,6 +299,8 @@ def export_command(
"page_type": page.page_type,
"title": page.title,
"content": page.content,
+ # Agent material kept off the body; machine consumers want it.
+ "digest": page.digest,
"target_path": page.target_path,
# Hierarchy columns, so a JSON consumer can rebuild the
# stored tree instead of re-deriving one from path strings.
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py b/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py
index fbb36a81f..7321295e3 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd/persistence.py
@@ -97,13 +97,21 @@ async def _index_preserved_pages(sf: Any, fts: Any, preserved_page_ids: set[str]
Page.content,
Page.summary,
Page.target_path,
+ Page.digest,
).where(Page.id.in_(batch))
)
).all()
await fts.index_many(
[
- (page_id, title or "", content or "", summary or "", target_path or "")
- for page_id, title, content, summary, target_path in rows
+ (
+ page_id,
+ title or "",
+ content or "",
+ summary or "",
+ target_path or "",
+ digest or "",
+ )
+ for page_id, title, content, summary, target_path, digest in rows
]
)
except Exception as exc: # pragma: no cover - defensive
diff --git a/packages/cli/src/repowise/cli/commands/reindex_cmd.py b/packages/cli/src/repowise/cli/commands/reindex_cmd.py
index e1dda66d7..9f4313fa5 100644
--- a/packages/cli/src/repowise/cli/commands/reindex_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/reindex_cmd.py
@@ -215,6 +215,7 @@ async def _reindex(repo_path, embedder_name: str, batch_size: int) -> None:
summary=page.summary or "",
content=page.content or "",
page_metadata=page.metadata_json,
+ digest=page.digest or "",
)
if item is None:
# Below the information floor. Not a failure — the page is
diff --git a/packages/cli/src/repowise/cli/commands/update_cmd/command.py b/packages/cli/src/repowise/cli/commands/update_cmd/command.py
index 22d1b380c..05eb71647 100644
--- a/packages/cli/src/repowise/cli/commands/update_cmd/command.py
+++ b/packages/cli/src/repowise/cli/commands/update_cmd/command.py
@@ -1691,6 +1691,7 @@ def run_update(
summary=page.summary,
content=page.content,
page_metadata=page.metadata,
+ digest=page.digest,
)
)
is not None
diff --git a/packages/core/alembic/versions/0044_fts_summary_and_target_path.py b/packages/core/alembic/versions/0044_fts_summary_and_target_path.py
index 5a7777807..5cc047969 100644
--- a/packages/core/alembic/versions/0044_fts_summary_and_target_path.py
+++ b/packages/core/alembic/versions/0044_fts_summary_and_target_path.py
@@ -34,7 +34,13 @@ from collections.abc import Sequence
from alembic import op
-from repowise.core.persistence.search import PG_FTS_EXPRESSION
+# Frozen at this revision's shape. A migration that imports the live search
+# constants replays a later column set on a database that does not have it.
+PG_FTS_EXPRESSION = (
+ "to_tsvector('english', "
+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "
+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"
+)
# revision identifiers
revision: str = "0044"
diff --git a/packages/core/alembic/versions/0095_page_digest.py b/packages/core/alembic/versions/0095_page_digest.py
new file mode 100644
index 000000000..7ec570706
--- /dev/null
+++ b/packages/core/alembic/versions/0095_page_digest.py
@@ -0,0 +1,103 @@
+"""wiki_pages: an agent digest beside the page body.
+
+``digest`` holds what a page carries for search and agents but not for a
+reader: the questions it answers, its identifiers, public API and git signals.
+It used to be appended to ``content``, which the reader, full-text search,
+embeddings and MCP all read as one string.
+
+Full-text search indexes the new column, so the PostgreSQL GIN expression is
+rebuilt over it and the SQLite FTS5 table is dropped and refilled. Local SQLite
+stores get the column from ``init_db``'s additive reconciler and the FTS table
+from ``FullTextSearch.ensure_index``; this migration covers PostgreSQL.
+
+Revision ID: 0095
+Revises: 0094
+Create Date: 2026-09-30
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+
+# Frozen at this revision's shape, never imported from ``search.py``: the next
+# widening must not change what this revision creates.
+PG_FTS_EXPRESSION = (
+ "to_tsvector('english', "
+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "
+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,'') || ' ' "
+ "|| COALESCE(digest,''))"
+)
+# SQLite: the file vocabulary column predates this revision (added at runtime
+# by ``FullTextSearch.ensure_index``), so both shapes keep it.
+_VOCABULARY_SQL = (
+ "CASE WHEN json_valid(metadata_json) "
+ "THEN COALESCE(json_extract(metadata_json, '$.file_vocabulary'), '') "
+ "ELSE '' END"
+)
+PAGE_FTS_DDL = (
+ "CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "
+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary, digest)"
+)
+
+# revision identifiers
+revision: str = "0095"
+down_revision: str | None = "0094"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+_OLD_PG_EXPRESSION = (
+ "to_tsvector('english', "
+ "COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "
+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"
+)
+
+_REFILL_SQL = (
+ "INSERT INTO page_fts"
+ "(page_id, title, content, summary, target_path, vocabulary, digest) "
+ "SELECT id, COALESCE(title,''), COALESCE(content,''), "
+ " COALESCE(summary,''), COALESCE(target_path,''), "
+ f" {_VOCABULARY_SQL}, COALESCE(digest,'') "
+ "FROM wiki_pages"
+)
+
+_OLD_SQLITE_DDL = (
+ "CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "
+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary)"
+)
+
+_OLD_REFILL_SQL = (
+ "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary) "
+ "SELECT id, COALESCE(title,''), COALESCE(content,''), "
+ " COALESCE(summary,''), COALESCE(target_path,''), "
+ f" {_VOCABULARY_SQL} "
+ "FROM wiki_pages"
+)
+
+def upgrade() -> None:
+ op.add_column(
+ "wiki_pages",
+ sa.Column("digest", sa.Text(), nullable=False, server_default=""),
+ )
+ dialect = op.get_bind().dialect.name
+ if dialect == "postgresql":
+ op.execute("DROP INDEX IF EXISTS idx_wiki_pages_fts")
+ op.execute(f"CREATE INDEX idx_wiki_pages_fts ON wiki_pages USING GIN({PG_FTS_EXPRESSION})")
+ elif dialect == "sqlite":
+ op.execute("DROP TABLE IF EXISTS page_fts")
+ op.execute(PAGE_FTS_DDL)
+ op.execute(_REFILL_SQL)
+
+
+def downgrade() -> None:
+ dialect = op.get_bind().dialect.name
+ if dialect == "postgresql":
+ op.execute("DROP INDEX IF EXISTS idx_wiki_pages_fts")
+ op.execute(f"CREATE INDEX idx_wiki_pages_fts ON wiki_pages USING GIN({_OLD_PG_EXPRESSION})")
+ elif dialect == "sqlite":
+ op.execute("DROP TABLE IF EXISTS page_fts")
+ op.execute(_OLD_SQLITE_DDL)
+ op.execute(_OLD_REFILL_SQL)
+ op.drop_column("wiki_pages", "digest")
diff --git a/packages/core/src/repowise/core/generation/agent_digest.py b/packages/core/src/repowise/core/generation/agent_digest.py
new file mode 100644
index 000000000..c5e99d9e5
--- /dev/null
+++ b/packages/core/src/repowise/core/generation/agent_digest.py
@@ -0,0 +1,112 @@
+"""The agent digest: what a page carries for search and agents, not for a reader.
+
+A module page used to end with the questions it answers, a concept index of
+its identifiers, its public API and paragraphs of git signals. All of that
+helps retrieval and an agent reading ``get_context``; none of it helps a person
+reading the page, who had to scroll past it. The digest keeps that material in
+``wiki_pages.digest``: indexed for full-text and vector search beside the body,
+returned by MCP beside ``content``, and shown in the reader only on request.
+
+The questions are the one model-written part. The model still ends its page
+with them, because they are written in a reader's words and a template cannot
+produce that, and :func:`split_questions` moves them out of the body before the
+page is stored. Everything else in the digest is rendered from the index.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from typing import Any
+
+from .structural_labels import ENGLISH_LABELS, LOCALIZED_LABELS
+
+# The questions heading in every language a page can be written in. The model
+# is asked for the English heading, and a localized run may translate it.
+_QUESTION_HEADINGS = frozenset(
+ label.lower()
+ for label in (
+ ENGLISH_LABELS["questions_heading"],
+ *(c["questions_heading"] for c in LOCALIZED_LABELS.values() if "questions_heading" in c),
+ )
+)
+
+#: Page-metadata key for :func:`module_signals`, read by the reader's signals card.
+MODULE_SIGNALS_KEY = "module_signals"
+
+_H2 = re.compile(r"^##\s+(.+?)\s*#*\s*$")
+_FENCE = re.compile(r"^\s*(```|~~~)")
+
+
+def _h2_sections(content: str) -> list[tuple[str | None, list[str]]]:
+ """Split *content* into ``(heading, lines)`` runs at level-2 headings.
+
+ The first run has no heading. A ``##`` inside a fenced block is text.
+ """
+ sections: list[tuple[str | None, list[str]]] = [(None, [])]
+ in_fence = False
+ for line in content.splitlines():
+ if _FENCE.match(line):
+ in_fence = not in_fence
+ match = None if in_fence else _H2.match(line)
+ if match:
+ sections.append((match.group(1), [line]))
+ else:
+ sections[-1][1].append(line)
+ return sections
+
+
+def split_questions(content: str) -> tuple[str, str]:
+ """Return ``(body, questions)`` with the questions section cut out of *content*.
+
+ ``questions`` is the section as written, heading included, or ``""`` when
+ the page has none. Every other section stays in the body in its order.
+ """
+ body: list[str] = []
+ questions: list[str] = []
+ for heading, lines in _h2_sections(content or ""):
+ if heading is not None and heading.strip().lower() in _QUESTION_HEADINGS:
+ questions.extend(lines)
+ else:
+ body.extend(lines)
+ if not questions:
+ return content, ""
+ return "\n".join(body).strip() + "\n", "\n".join(questions).strip()
+
+
+def rejoin_questions(body: str, digest: str) -> str:
+ """The model's response a stored page was split from: *body* plus its questions.
+
+ A page reused from a prior run is rebuilt from the stored body, so without
+ this the next split would find no questions and the digest would lose them.
+ """
+ _, questions = split_questions(digest or "")
+ if not questions:
+ return body
+ return f"{(body or '').rstrip()}\n\n{questions}\n"
+
+
+def module_signals(ctx: Any, git_summary: Mapping[str, Any] | None) -> dict[str, Any]:
+ """The git signals a module page reports, as plain data.
+
+ One dict feeds both the digest text an agent reads and the small signals
+ card beside the page in the reader, so the two cannot disagree. Absent or
+ zero signals are left out: a missing key means "nothing to report", and a
+ module with no history at all gets an empty dict. ``files`` is the
+ denominator the counts are read against.
+ """
+ signals: dict[str, Any] = {
+ "hotspots": ctx.hotspot_count,
+ "bus_factor_one": ctx.single_owner_files,
+ "bug_fixes": ctx.bugfix_total,
+ "most_fixed_file": (ctx.most_fixed_file or {}).get("path", ""),
+ "stable": ctx.stable_count,
+ "owners": [{"name": o["name"], "files": o["file_count"]} for o in ctx.top_owners],
+ "co_changes": [{"path": c["path"], "files": c["count"]} for c in ctx.coupled_modules],
+ }
+ if git_summary and git_summary.get("most_active_file"):
+ signals["most_active_file"] = git_summary["most_active_file"]
+ signals["most_active_commits_90d"] = git_summary.get("most_active_commits_90d", 0)
+ found = {key: value for key, value in signals.items() if value}
+ return {"files": len(ctx.files), **found} if found else {}
+
diff --git a/packages/core/src/repowise/core/generation/models.py b/packages/core/src/repowise/core/generation/models.py
index 47245a4f8..39d267798 100644
--- a/packages/core/src/repowise/core/generation/models.py
+++ b/packages/core/src/repowise/core/generation/models.py
@@ -529,6 +529,9 @@ class GeneratedPage:
# MCP get_context as the default narrative payload (content is gated behind
# include=["full_doc"]).
summary: str = ""
+ # Agent digest: material indexed and served to agents but kept out of the
+ # reader's page body. See :mod:`repowise.core.generation.agent_digest`.
+ digest: str = ""
# Where this page sits in the wiki. Left unset by generators that do not
# place their pages; the tree builder fills them in before persistence.
# See the matching columns on the Page model for what each one means.
diff --git a/packages/core/src/repowise/core/generation/page_generator/core.py b/packages/core/src/repowise/core/generation/page_generator/core.py
index ee1bf59f1..186cbf86e 100644
--- a/packages/core/src/repowise/core/generation/page_generator/core.py
+++ b/packages/core/src/repowise/core/generation/page_generator/core.py
@@ -25,6 +25,7 @@ import structlog
from repowise.core.ingestion.models import ParsedFile, RepoStructure
from repowise.core.providers.llm.base import BaseProvider, CacheHint, GeneratedResponse
+from ..agent_digest import rejoin_questions
from ..context.evidence import (
EvidenceItem,
EvidenceSelection,
@@ -130,6 +131,7 @@ class PriorPage:
source_hash: str
model_name: str
content: str
+ digest: str = ""
input_tokens: int = 0
output_tokens: int = 0
cached_tokens: int = 0
@@ -393,7 +395,9 @@ class PageGenerator(PerTypeGenerationMixin, StructuralRenderMixin):
target_path=target_path,
)
return GeneratedResponse(
- content=prior.content,
+ # The stored body lost its questions to the digest; the
+ # response it was split from carried them.
+ content=rejoin_questions(prior.content, prior.digest),
input_tokens=0,
output_tokens=0,
cached_tokens=0,
diff --git a/packages/core/src/repowise/core/generation/page_generator/orchestrate.py b/packages/core/src/repowise/core/generation/page_generator/orchestrate.py
index c1c6cc896..15efe0747 100644
--- a/packages/core/src/repowise/core/generation/page_generator/orchestrate.py
+++ b/packages/core/src/repowise/core/generation/page_generator/orchestrate.py
@@ -1076,6 +1076,7 @@ def _embed_item(page: GeneratedPage) -> tuple[str, str, dict] | None:
summary=page.summary,
content=page.content,
page_metadata=page.metadata,
+ digest=page.digest,
)
diff --git a/packages/core/src/repowise/core/generation/page_generator/pertype.py b/packages/core/src/repowise/core/generation/page_generator/pertype.py
index e9e0f665f..2c348fd73 100644
--- a/packages/core/src/repowise/core/generation/page_generator/pertype.py
+++ b/packages/core/src/repowise/core/generation/page_generator/pertype.py
@@ -8,6 +8,7 @@ each module under the project's 400-line ceiling.
from __future__ import annotations
+import re
import uuid
from collections.abc import Sequence
from dataclasses import replace
@@ -18,6 +19,7 @@ import structlog
from repowise.core.ingestion.models import ParsedFile, RepoStructure
from .. import onboarding as _onboarding
+from ..agent_digest import MODULE_SIGNALS_KEY, module_signals, split_questions
from ..architecture_mermaid import embed_mermaid
from ..context.assembler import build_concept_index
from ..context_assembler import FilePageContext
@@ -315,47 +317,51 @@ class PerTypeGenerationMixin:
page.structural_key = structural_key or page.structural_key
return page
- def _with_concept_index(page: GeneratedPage) -> GeneratedPage:
- """Append the module's own identifiers to whatever wrote the page.
+ def _with_digest(page: GeneratedPage) -> GeneratedPage:
+ """Split what the page carries for agents off its body.
- After generation rather than inside the prompt, and that placement
- is the point. ``module_page.j2`` is a prompt: a table put in it is
- material the model may reformat, abbreviate or drop, and a page
- whose identifiers came out of a provider is exactly as trustworthy
- as the prose around them. Appended here, every name and path is the
- symbol index's and no response can change it.
+ The identifiers, public API and git signals are rendered here from
+ the index, not asked of the model: material in a prompt is
+ material a model may reformat, abbreviate or drop, and appended
+ after generation every name and path is the index's own. After
+ ``_build_generated_page`` too, so the page summary is still drawn
+ from the model's opening.
- After ``_build_generated_page`` too, so the page summary is still
- drawn from the model's opening rather than from a table row.
+ The package table of a roll-up stays in the body: it is how a
+ reader finds the packages the page covers. Replaced, not appended,
+ so a page reused from a prior run keeps one table.
"""
- tables = []
if ctx.packages:
- # One row per package of a roll-up, for the same reason.
- tables.append(
- self._render("_package_table.j2", style_prefix=False, rows=ctx.packages)
- )
+ table = self._render("_package_table.j2", style_prefix=False, rows=ctx.packages)
+ page.content = embed_package_table(page.content or "", table.strip())
+ page.content, questions = split_questions(page.content)
rows, omitted = build_concept_index(file_contexts)
- if rows:
- tables.append(
- self._render(
- "_concept_index_table.j2", style_prefix=False, rows=rows, omitted=omitted
- )
- )
- for table in tables:
- page.content = f"{(page.content or '').rstrip()}\n\n{table.strip()}\n"
+ signals = module_signals(ctx, module_git_summary)
+ if signals:
+ page.metadata[MODULE_SIGNALS_KEY] = signals
+ digest = self._render(
+ "_agent_digest.j2",
+ style_prefix=False,
+ questions=questions,
+ public_api=ctx.public_api,
+ concept_rows=rows,
+ concept_omitted=omitted,
+ signals=signals,
+ )
+ page.digest = re.sub(r"\n{3,}", "\n\n", digest).strip()
return page
if self._config.deterministic:
- page = self._stub_module_page(ctx, page_target, title, module_git_summary)
- return _stamp_concept(_with_concept_index(page))
- user_prompt = self._render("module_page.j2", ctx=ctx, module_git_summary=module_git_summary)
+ page = self._stub_module_page(ctx, page_target, title)
+ return _stamp_concept(_with_digest(page))
+ user_prompt = self._render("module_page.j2", ctx=ctx)
try:
response = await self._call_provider(
"module_page", user_prompt, str(uuid.uuid4()), target_path=page_target
)
except Exception as exc:
- stub = self._stub_module_page(ctx, page_target, title, module_git_summary)
- return _stamp_concept(_with_concept_index(_stub_fallback(stub, "module_page", exc)))
+ stub = self._stub_module_page(ctx, page_target, title)
+ return _stamp_concept(_with_digest(_stub_fallback(stub, "module_page", exc)))
page = self._build_generated_page(
"module_page",
page_target,
@@ -364,7 +370,7 @@ class PerTypeGenerationMixin:
compute_source_hash(user_prompt),
GENERATION_LEVELS["module_page"],
)
- return _stamp_concept(_with_concept_index(page))
+ return _stamp_concept(_with_digest(page))
async def generate_scc_page(
self,
diff --git a/packages/core/src/repowise/core/generation/page_generator/structural.py b/packages/core/src/repowise/core/generation/page_generator/structural.py
index 0801ca865..0e1ab984f 100644
--- a/packages/core/src/repowise/core/generation/page_generator/structural.py
+++ b/packages/core/src/repowise/core/generation/page_generator/structural.py
@@ -930,7 +930,6 @@ class StructuralRenderMixin:
ctx: Any,
target_path: str,
title: str,
- module_git_summary: dict | None,
) -> GeneratedPage:
return self._stub_page(
page_type="module_page",
@@ -938,7 +937,6 @@ class StructuralRenderMixin:
title=title,
template="module_page.j2",
ctx=ctx,
- module_git_summary=module_git_summary,
)
def _stub_repo_overview(
diff --git a/packages/core/src/repowise/core/generation/report.py b/packages/core/src/repowise/core/generation/report.py
index 51d41dbab..f53c64d4c 100644
--- a/packages/core/src/repowise/core/generation/report.py
+++ b/packages/core/src/repowise/core/generation/report.py
@@ -127,8 +127,8 @@ def measure_opening_frames(pages: list[GeneratedPage]) -> OpeningFrameReport:
)
-# The heading over the identifiers appended to a module page after the model
-# has written it. Counted for the same reason the questions block is: the table
+# The heading over the identifiers rendered into a module page's digest after
+# the model has written it. Counted for the same reason the questions block is: the table
# is conditional (a module exporting nothing renders none), so only the ratio
# is readable, and nothing else in a run would report it going to zero.
CONCEPT_INDEX_HEADING = "## Concept index"
@@ -145,7 +145,7 @@ def _count_concept_indexes(pages: list[GeneratedPage]) -> dict[str, int]:
eligible = [p for p in pages if p.page_type == "module_page"]
return {
"eligible_pages": len(eligible),
- "with_index": sum(1 for p in eligible if CONCEPT_INDEX_HEADING in (p.content or "")),
+ "with_index": sum(1 for p in eligible if CONCEPT_INDEX_HEADING in (p.digest or "")),
}
diff --git a/packages/core/src/repowise/core/generation/templates/_agent_digest.j2 b/packages/core/src/repowise/core/generation/templates/_agent_digest.j2
new file mode 100644
index 000000000..59843c1dd
--- /dev/null
+++ b/packages/core/src/repowise/core/generation/templates/_agent_digest.j2
@@ -0,0 +1,57 @@
+{#- A module page's agent digest (``wiki_pages.digest``): indexed and served to
+ agents beside the page body, shown in the reader only on request. Rendered
+ from the index after the model has written the page, so every name, path
+ and number here is the index's own. The questions are the exception: the
+ model writes them and ``agent_digest.split_questions`` moves them here.
+
+ Each section renders only when it has rows: a heading with nothing under it
+ claims the module has none, in the same shape as a real answer. -#}
+{% if questions %}
+{{ questions }}
+
+{% endif %}
+{% if public_api %}
+## Public API
+
+What a consumer imports, read off the package manifests and re-exports.
+
+{% for e in public_api -%}
+- `{{ e.name }}` ({{ e.kind }}, `{{ e.file }}`){% if e.alias_of %}, another name for `{{ e.alias_of }}`{% endif %}{% if e.signature %}: `` {{ e.signature | oneline }} ``{% endif %}
+{% endfor %}
+
+{% endif %}
+{% if concept_rows %}
+## Concept index
+
+What the page calls things, the identifier to search for, and where it lives.
+
+| Concept | Symbol | File |
+| --- | --- | --- |
+{% for row in concept_rows -%}
+| {{ row.concept }} | `{{ row.symbol }}` | `{{ row.file }}` |
+{% endfor -%}
+{%- if concept_omitted %}
+_{{ concept_omitted }} further public symbol{{ "s" if concept_omitted != 1 else "" }} in this module are not listed._
+{% endif %}
+
+{% endif %}
+{% if signals %}
+## Signals
+
+From git history over the {{ signals.get("files", 0) }} files this page covers.
+
+{% if signals.get("hotspots") %}- Change hotspots: {{ signals.get("hotspots") }}
+{% endif %}
+{%- if signals.get("bus_factor_one") %}- Files with a bus factor of one: {{ signals.get("bus_factor_one") }}
+{% endif %}
+{%- if signals.get("bug_fixes") %}- Bug-fix commits: {{ signals.get("bug_fixes") }}{% if signals.get("most_fixed_file") %}, most in `{{ signals.get("most_fixed_file") }}`{% endif %}
+{% endif %}
+{%- if signals.get("stable") %}- Stable files: {{ signals.get("stable") }}
+{% endif %}
+{%- if signals.get("owners") %}- Primary maintainers: {% for o in signals.get("owners") %}{{ o.name }} ({{ o.files }} file{{ "s" if o.files != 1 else "" }}){% if not loop.last %}, {% endif %}{% endfor %}
+{% endif %}
+{%- if signals.get("most_active_file") %}- Most active file: `{{ signals.get("most_active_file") }}` ({{ signals.get("most_active_commits_90d") }} commits in 90 days)
+{% endif %}
+{%- if signals.get("co_changes") %}- Changes together with (no import edge explains it): {% for c in signals.get("co_changes") %}`{{ c.path }}` ({{ c.files }} file{{ "s" if c.files != 1 else "" }}){% if not loop.last %}, {% endif %}{% endfor %}
+{% endif %}
+{% endif %}
diff --git a/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2 b/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2
deleted file mode 100644
index 9b2c0ef49..000000000
--- a/packages/core/src/repowise/core/generation/templates/_concept_index_table.j2
+++ /dev/null
@@ -1,18 +0,0 @@
-{#- The identifiers behind the page's prose, rendered from the assembled symbol
- data and appended after the model has written the page. Nothing here passes
- through a provider, so the names and paths are the index's own.
-
- Rendered only when there is at least one row: a header with nothing under
- it claims the module exports nothing, in the same shape as a real answer. -#}
-## Concept index
-
-What the prose above calls things, the identifier to search for, and where it lives.
-
-| Concept | Symbol | File |
-| --- | --- | --- |
-{% for row in rows -%}
-| {{ row.concept }} | `{{ row.symbol }}` | `{{ row.file }}` |
-{% endfor -%}
-{%- if omitted %}
-_{{ omitted }} further public symbol{{ "s" if omitted != 1 else "" }} in this module are not listed._
-{% endif -%}
diff --git a/packages/core/src/repowise/core/generation/templates/_package_table.j2 b/packages/core/src/repowise/core/generation/templates/_package_table.j2
index 4e05a953c..8ff176620 100644
--- a/packages/core/src/repowise/core/generation/templates/_package_table.j2
+++ b/packages/core/src/repowise/core/generation/templates/_package_table.j2
@@ -1,8 +1,7 @@
{#- One row per package a package roll-up covers, rendered from the grouping and
appended after the model has written the page, so no response can drop or
- merge a package. -#}
-## Packages
-
+ merge a package. The heading comes from ``embed_package_table``, which
+ replaces the section instead of adding a second one when a page is reused. -#}
This page covers {{ rows | length }} sibling packages, each with its own manifest.
| Package | Files |
diff --git a/packages/core/src/repowise/core/generation/templates/module_page.j2 b/packages/core/src/repowise/core/generation/templates/module_page.j2
index 3ca0144f1..ba57ee895 100644
--- a/packages/core/src/repowise/core/generation/templates/module_page.j2
+++ b/packages/core/src/repowise/core/generation/templates/module_page.j2
@@ -88,37 +88,6 @@ Use these to explain *why* the subsystem is shaped the way it is. Do not invent
{% endfor %}
{% endif %}
-{# Health and history signals a reader cannot get from the code alone. Only
- surfaced when git metadata supplied them; all zero/empty degrades to nothing
- rather than a fabricated number. This is repowise's differentiation — weave
- the true ones into the narrative, do not tabulate them mechanically. #}
-{%- set _health = [] -%}
-{%- if ctx.hotspot_count > 0 -%}{%- set _ = _health.append(ctx.hotspot_count ~ " file" ~ ("s" if ctx.hotspot_count != 1 else "") ~ " flagged as change hotspots") -%}{%- endif -%}
-{%- if ctx.single_owner_files > 0 -%}{%- set _ = _health.append(ctx.single_owner_files ~ " file" ~ ("s" if ctx.single_owner_files != 1 else "") ~ " with a bus factor of one (single-maintainer risk)") -%}{%- endif -%}
-{%- if ctx.bugfix_total > 0 -%}{%- set _ = _health.append(ctx.bugfix_total ~ " bug-fix commit" ~ ("s" if ctx.bugfix_total != 1 else "") ~ " in recent history" ~ (", most in `" ~ ctx.most_fixed_file.path ~ "`" if ctx.most_fixed_file else "")) -%}{%- endif -%}
-{%- if ctx.stable_count > 0 -%}{%- set _ = _health.append(ctx.stable_count ~ " file" ~ ("s" if ctx.stable_count != 1 else "") ~ " that have been stable") -%}{%- endif -%}
-{% if _health %}
-### Subsystem health (from git history — state the notable ones plainly)
-{% for h in _health %}
-- {{ h }}
-{% endfor %}
-{% endif %}
-
-{% if ctx.coupled_modules %}
-### Change-coupled with (co-changes in history, not import edges)
-These modules change together with this one in history and nothing in the dependency graph accounts for it — hidden coupling worth naming.
-{% for cm in ctx.coupled_modules %}
-- `{{ cm.path }}` ({{ cm.count }} co-changing file{{ "s" if cm.count != 1 else "" }})
-{% endfor %}
-{% endif %}
-
-{% if ctx.top_owners %}
-### Ownership
-{% for o in ctx.top_owners %}
-- **{{ o.name }}** — primary maintainer of {{ o.file_count }} file(s)
-{% endfor %}
-{% endif %}
-
{% if ctx.key_classes %}
### Class hierarchy
{% for h in ctx.key_classes %}
@@ -126,11 +95,6 @@ These modules change together with this one in history and nothing in the depend
{% endfor %}
{% endif %}
-{% if module_git_summary is defined and module_git_summary %}
-### Recent activity
-Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_summary.most_active_commits_90d }} commits in 90 days).
-{% endif %}
-
{% if ctx.declared_files %}
### What other files declare (signatures only, no code shown)
{% for line in ctx.declared_files %}
@@ -150,7 +114,6 @@ Write the page now from the material above. Requirements:
- Synthesise across the whole file set. Draw on the material as a body of evidence; a file-by-file walkthrough is a failure even when accurate. Draw on many files, not one.
- Put any enumerable list (key files, dependencies, decisions) in a markdown table where it reads better than prose.
- Name the subsystem's key files and the neighbouring modules it consumes or serves in backticks exactly as written above so they resolve to wiki links. Mention a neighbour only where it explains this subsystem's role — no links for density, no reciprocal mentions for symmetry.
-- Weave the true health and history signals (hotspots, bus factor, change coupling, fix history) into the narrative where they explain the subsystem's character. Skip any that are absent; never invent one.
- State behaviour only when an excerpt shows it; otherwise say what the file declares.
- A commit subject is history, not current behaviour: if you cite one, label it (historical, <date>).
- Third person throughout. No concluding or summary section.
diff --git a/packages/core/src/repowise/core/generation/templates/stub/module_page.j2 b/packages/core/src/repowise/core/generation/templates/stub/module_page.j2
index 556e28858..32f1be6ca 100644
--- a/packages/core/src/repowise/core/generation/templates/stub/module_page.j2
+++ b/packages/core/src/repowise/core/generation/templates/stub/module_page.j2
@@ -85,13 +85,6 @@ Highest-PageRank first: the order to read them in.
{% endfor %}
{% endif %}
-{% if ctx.top_owners %}
-## Ownership
-{% for o in ctx.top_owners %}
-- **{{ o.name }}**: primary maintainer of {{ o.file_count }} file{{ "s" if o.file_count != 1 else "" }}
-{% endfor %}
-{% endif %}
-
{% if ctx.dead_code_findings %}
## Potentially unused code
{% for finding in ctx.dead_code_findings[:10] %}
@@ -99,30 +92,7 @@ Highest-PageRank first: the order to read them in.
{% endfor %}
{% endif %}
-{%- set _health = [] -%}
-{%- if ctx.hotspot_count > 0 -%}{%- set _ = _health.append(ctx.hotspot_count ~ " file" ~ ("s" if ctx.hotspot_count != 1 else "") ~ " flagged as change hotspots") -%}{%- endif -%}
-{%- if ctx.single_owner_files > 0 -%}{%- set _ = _health.append(ctx.single_owner_files ~ " file" ~ ("s" if ctx.single_owner_files != 1 else "") ~ " with a bus factor of one (single-maintainer risk)") -%}{%- endif -%}
-{%- if ctx.bugfix_total > 0 -%}{%- set _ = _health.append(ctx.bugfix_total ~ " bug-fix commit" ~ ("s" if ctx.bugfix_total != 1 else "") ~ " in recent history" ~ (", most in " ~ ctx.most_fixed_file.path if ctx.most_fixed_file else "")) -%}{%- endif -%}
-{%- if ctx.stable_count > 0 -%}{%- set _ = _health.append(ctx.stable_count ~ " file" ~ ("s" if ctx.stable_count != 1 else "") ~ " that have been stable") -%}{%- endif -%}
-{% if _health %}
-## Subsystem health
-{% for h in _health %}
-- {{ h }}
-{% endfor %}
-{% endif %}
-
-{% if ctx.coupled_modules %}
-## Change-coupled with
-Modules that change together with this one in history without importing it.
-{% for cm in ctx.coupled_modules %}
-- `{{ cm.path }}` (co-changed {{ cm.count }}×)
-{% endfor %}
-{% endif %}
-
-{% if module_git_summary %}
-## Recent activity
-Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_summary.most_active_commits_90d }} commits in the last 90 days).
-{% endif %}
+{% include "stub/_footer.j2" %}
## Questions this page answers
- What does {{ ctx.title }} do, and where does it sit in the system?
@@ -130,4 +100,4 @@ Most active file: `{{ module_git_summary.most_active_file }}` ({{ module_git_sum
{% if ctx.entry_points %}- What are the entry points into {{ ctx.title }}?
{% endif %}{% if ctx.dependencies %}- What does {{ ctx.title }} depend on?
{% endif %}{% if ctx.dependents %}- What relies on {{ ctx.title }}?
-{% endif %}{% include "stub/_footer.j2" %}
+{% endif %}
diff --git a/packages/core/src/repowise/core/persistence/crud/pages.py b/packages/core/src/repowise/core/persistence/crud/pages.py
index 40fb79244..35fc57c1d 100644
--- a/packages/core/src/repowise/core/persistence/crud/pages.py
+++ b/packages/core/src/repowise/core/persistence/crud/pages.py
@@ -62,6 +62,7 @@ def _apply_page_upsert(
title: str,
content: str,
summary: str,
+ digest: str,
target_path: str,
source_hash: str,
model_name: str,
@@ -140,6 +141,8 @@ def _apply_page_upsert(
):
existing.title = title
existing.summary = summary
+ # Git signals and identifiers in the digest move without the prose.
+ existing.digest = digest
existing.target_path = target_path
existing.freshness_status = freshness_status
existing.confidence = confidence
@@ -174,6 +177,7 @@ def _apply_page_upsert(
existing.title = title
existing.content = content
existing.summary = summary
+ existing.digest = digest
existing.target_path = target_path
existing.source_hash = source_hash
existing.model_name = model_name
@@ -200,6 +204,7 @@ def _apply_page_upsert(
title=title,
content=content,
summary=summary,
+ digest=digest,
target_path=target_path,
source_hash=source_hash,
model_name=model_name,
@@ -232,6 +237,7 @@ async def upsert_page(
title: str,
content: str,
summary: str = "",
+ digest: str = "",
target_path: str,
source_hash: str,
model_name: str,
@@ -272,6 +278,7 @@ async def upsert_page(
title=title,
content=content,
summary=summary,
+ digest=digest,
target_path=target_path,
source_hash=source_hash,
model_name=model_name,
@@ -327,6 +334,7 @@ async def load_prior_pages(
source_hash=row.source_hash,
model_name=row.model_name,
content=row.content,
+ digest=row.digest or "",
input_tokens=row.input_tokens,
output_tokens=row.output_tokens,
cached_tokens=row.cached_tokens,
@@ -354,6 +362,7 @@ async def upsert_page_from_generated(
title=gp.title, # type: ignore[attr-defined]
content=gp.content, # type: ignore[attr-defined]
summary=getattr(gp, "summary", "") or "",
+ digest=getattr(gp, "digest", "") or "",
target_path=gp.target_path, # type: ignore[attr-defined]
source_hash=gp.source_hash, # type: ignore[attr-defined]
model_name=gp.model_name, # type: ignore[attr-defined]
@@ -436,6 +445,7 @@ async def upsert_pages_from_generated(
title=gp.title,
content=gp.content,
summary=getattr(gp, "summary", "") or "",
+ digest=getattr(gp, "digest", "") or "",
target_path=gp.target_path,
source_hash=gp.source_hash,
model_name=gp.model_name,
diff --git a/packages/core/src/repowise/core/persistence/information_floor.py b/packages/core/src/repowise/core/persistence/information_floor.py
index 64bccb556..65490807d 100644
--- a/packages/core/src/repowise/core/persistence/information_floor.py
+++ b/packages/core/src/repowise/core/persistence/information_floor.py
@@ -165,12 +165,14 @@ def reset_pages_denied_a_vector() -> None:
_pages_denied_a_vector = 0
-def meets_information_floor(content: str, floor: int | None = None) -> bool:
- """Whether *content* says enough to be worth a slot in the index.
+def meets_information_floor(content: str, floor: int | None = None, digest: str = "") -> bool:
+ """Whether *content* and the page's *digest* say enough to be worth an index slot.
- A floor of 0 admits everything, which is how the feature is turned off.
+ Both count: the digest is indexed with the body, and a short page whose
+ words moved into its digest still answers questions. A floor of 0 admits
+ everything, which is how the feature is turned off.
"""
limit = information_floor() if floor is None else floor
if limit <= 0:
return True
- return len(substantive_text(content)) >= limit
+ return len(substantive_text(f"{content}\n{digest}")) >= limit
diff --git a/packages/core/src/repowise/core/persistence/models.py b/packages/core/src/repowise/core/persistence/models.py
index f1c15029e..b0b02e024 100644
--- a/packages/core/src/repowise/core/persistence/models.py
+++ b/packages/core/src/repowise/core/persistence/models.py
@@ -201,6 +201,11 @@ class Page(Base):
# Surfaced by get_context as the default narrative; content is gated
# behind include=["full_doc"] to keep MCP responses small.
summary: Mapped[str] = mapped_column(Text, nullable=False, default="")
+ # Agent digest (markdown): what a page carries for search and agents but
+ # not for a reader, e.g. the questions it answers, its identifiers, public
+ # API and git signals. Indexed and served by MCP beside ``content``; the
+ # reader shows it only on request. See ``generation/agent_digest.py``.
+ digest: Mapped[str] = mapped_column(Text, nullable=False, default="", server_default="")
target_path: Mapped[str] = mapped_column(Text, nullable=False)
source_hash: Mapped[str] = mapped_column(String(64), nullable=False)
model_name: Mapped[str] = mapped_column(String(128), nullable=False)
diff --git a/packages/core/src/repowise/core/persistence/search.py b/packages/core/src/repowise/core/persistence/search.py
index cb0d24d1d..d8a0cfe01 100644
--- a/packages/core/src/repowise/core/persistence/search.py
+++ b/packages/core/src/repowise/core/persistence/search.py
@@ -66,16 +66,28 @@ _SNIPPET_LEN = 200
# ``vocabulary`` is a file page's own identifiers and comment prose. It is kept
# off the rendered page (it reads as noise), so without its own column a
# keyless store cannot match a page on any of those words.
-PAGE_FTS_COLUMNS = ("page_id", "title", "content", "summary", "target_path", "vocabulary")
+#
+# ``digest`` is the page's agent material (questions, identifiers, signals),
+# kept out of ``content`` so the reader does not scroll past it and indexed here
+# so retrieval still finds the page by those words.
+PAGE_FTS_COLUMNS = (
+ "page_id",
+ "title",
+ "content",
+ "summary",
+ "target_path",
+ "vocabulary",
+ "digest",
+)
PAGE_FTS_DDL = (
"CREATE VIRTUAL TABLE IF NOT EXISTS page_fts "
- "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary)"
+ "USING fts5(page_id UNINDEXED, title, content, summary, target_path, vocabulary, digest)"
)
_PAGE_FTS_INSERT_SQL = (
- "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary) "
- "VALUES (:pid, :title, :content, :summary, :target_path, :vocabulary)"
+ "INSERT INTO page_fts(page_id, title, content, summary, target_path, vocabulary, digest) "
+ "VALUES (:pid, :title, :content, :summary, :target_path, :vocabulary, :digest)"
)
# Page-metadata key for a file page's vocabulary (field names, string
@@ -150,7 +162,8 @@ _ORPHAN_DELETE_SQL = f"DELETE FROM page_fts WHERE {_ORPHAN_PREDICATE}"
PG_FTS_EXPRESSION = (
"to_tsvector('english', "
"COALESCE(title,'') || ' ' || COALESCE(content,'') || ' ' "
- "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,''))"
+ "|| COALESCE(summary,'') || ' ' || COALESCE(target_path,'') || ' ' "
+ "|| COALESCE(digest,''))"
)
# How many distinct terms' document frequencies one FullTextSearch keeps. Term
@@ -164,12 +177,13 @@ _SCORE_EPSILON = 1e-6
# Per-column bm25 weights, in ``PAGE_FTS_COLUMNS`` order: title and
# target_path name the file a page is about, content and summary only mention
-# it. ``page_id`` is UNINDEXED and contributes nothing, but bm25() takes one
-# weight per column, so it still needs one. bm25() ignores a weight past the
+# it, and ``digest`` weighs like the prose it indexes. ``page_id`` is
+# UNINDEXED and contributes nothing, but bm25() takes one weight per column,
+# so it still needs one. bm25() ignores a weight past the
# last column and defaults a missing one to 1.0, so the arity is checked by
# test_search_fts_columns rather than by anything raising here. The vocabulary
# is a bag of words, so it counts for less than prose written about the file.
-_BM25_COLUMN_WEIGHTS = (0.0, 4.0, 1.0, 1.0, 3.0, 0.5)
+_BM25_COLUMN_WEIGHTS = (0.0, 4.0, 1.0, 1.0, 3.0, 0.5, 1.0)
_BM25_SCORE = "bm25(page_fts, " + ", ".join(str(w) for w in _BM25_COLUMN_WEIGHTS) + ")"
_log = logging.getLogger(__name__)
@@ -348,10 +362,10 @@ class FullTextSearch:
await conn.execute(
text(
"INSERT INTO page_fts"
- "(page_id, title, content, summary, target_path, vocabulary) "
+ "(page_id, title, content, summary, target_path, vocabulary, digest) "
"SELECT id, COALESCE(title,''), COALESCE(content,''), "
" COALESCE(summary,''), COALESCE(target_path,''), "
- f" {_VOCABULARY_SQL} "
+ f" {_VOCABULARY_SQL}, COALESCE(digest,'') "
"FROM wiki_pages"
)
)
@@ -430,6 +444,7 @@ class FullTextSearch:
content: str,
summary: str | None = None,
target_path: str | None = None,
+ digest: str = "",
) -> None:
"""Add or replace a page in the FTS index.
@@ -450,13 +465,13 @@ class FullTextSearch:
exclude, and nothing would report the disagreement. The page itself is
untouched: it stays in ``wiki_pages`` and stays a valid link target.
"""
- await self.index_many([(page_id, title, content, summary, target_path)])
+ await self.index_many([(page_id, title, content, summary, target_path, digest)])
async def index_pages(self, pages: Iterable[Any]) -> None:
"""Index generated pages, one :meth:`index_many` transaction per chunk.
- Each entry needs ``page_id``, ``title``, ``content``, ``summary`` and
- ``target_path``. Use this, not :meth:`index` in a loop: a write deletes
+ Each entry needs ``page_id``, ``title``, ``content``, ``summary``,
+ ``target_path`` and ``digest``. Use this, not :meth:`index` in a loop: a write deletes
by ``page_id``, which FTS5 stores unindexed, so every delete scans the
whole index. Per page that is quadratic in wiki size (18 GB of reads
for a 2,300-page rebuild); chunked it is one scan per chunk, while a
@@ -464,16 +479,19 @@ class FullTextSearch:
"""
for chunk in chunked(list(pages)):
await self.index_many(
- [(p.page_id, p.title, p.content, p.summary, p.target_path) for p in chunk]
+ [
+ (p.page_id, p.title, p.content, p.summary, p.target_path, p.digest)
+ for p in chunk
+ ]
)
async def index_many(
self,
- pages: Sequence[tuple[str, str, str, str | None, str | None]],
+ pages: Sequence[tuple[str, str, str, str | None, str | None, str]],
) -> None:
"""Add or replace many pages in a single transaction.
- Each entry is ``(page_id, title, content, summary, target_path)`` and
+ Each entry is ``(page_id, title, content, summary, target_path, digest)`` and
gets exactly the semantics :meth:`index` documents, floor exclusion
included — this is the one implementation and :meth:`index` is the
single-page call into it. What changes is the transaction count:
@@ -497,11 +515,11 @@ class FullTextSearch:
# existed.
deletions: dict[str, None] = {}
insertions: dict[str, dict[str, str]] = {}
- for page_id, title, content, summary, target_path in pages:
+ for page_id, title, content, summary, target_path, digest in pages:
deletions[page_id] = None
if summary is None or target_path is None:
self._warn_missing_index_fields(page_id, summary, target_path)
- if meets_information_floor(content):
+ if meets_information_floor(content, digest=digest or ""):
insertions[page_id] = {
"pid": page_id,
"title": title,
@@ -509,6 +527,7 @@ class FullTextSearch:
"summary": summary or "",
"target_path": target_path or "",
"vocabulary": "",
+ "digest": digest or "",
}
else:
self._count_skipped_below_floor(page_id, content)
diff --git a/packages/core/src/repowise/core/persistence/vector_store/_base.py b/packages/core/src/repowise/core/persistence/vector_store/_base.py
index 4359d1d55..26ad3b7a0 100644
--- a/packages/core/src/repowise/core/persistence/vector_store/_base.py
+++ b/packages/core/src/repowise/core/persistence/vector_store/_base.py
@@ -129,6 +129,7 @@ def embed_item(
summary: str,
content: str,
page_metadata: Mapping[str, object] | str | None = None,
+ digest: str = "",
) -> tuple[str, str, dict] | None:
"""Build the one ``(page_id, text, metadata)`` item every writer embeds.
@@ -160,7 +161,7 @@ def embed_item(
page held out of one arm and kept in the other is still fetched, still
occupies one of the fixed number of rows retrieval takes before it filters
anything, and still displaces a page that could have answered. The test is
- applied to ``content`` alone for that reason — the same input the
+ applied to ``content`` and ``digest`` for that reason, the same input the
full-text side measures, so the two arms cannot disagree about a page.
The page itself is untouched either way. It stays in ``wiki_pages``, still
@@ -171,7 +172,10 @@ def embed_item(
``page_metadata`` is the page's metadata, as a dict (a generated page) or
the stored JSON string (a ``wiki_pages`` row). Only
:data:`FILE_VOCABULARY_KEY` is read from it, and appended after the
- content; the floor still measures ``content`` alone.
+ content; the floor does not measure it.
+
+ ``digest`` is the page's agent material (``wiki_pages.digest``), embedded
+ after the content for the same reason the vocabulary is.
"""
if not title.strip():
raise ValueError(
@@ -179,10 +183,12 @@ def embed_item(
f"vector that cannot be found by name and reports nothing wrong; "
f"pass the page's real title."
)
- if not meets_information_floor(content):
+ if not meets_information_floor(content, digest=digest):
count_page_denied_a_vector()
return None
- parts = [p for p in (title, target_path, summary, content, _vocabulary(page_metadata)) if p]
+ parts = [
+ p for p in (title, target_path, summary, content, digest, _vocabulary(page_metadata)) if p
+ ]
return (
page_id,
"\n".join(parts),
diff --git a/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py b/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py
index 22f464dfe..fe9aa4b92 100644
--- a/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py
+++ b/packages/server/src/repowise/server/mcp_server/_budget/budgeter.py
@@ -348,7 +348,12 @@ def _with_budget_reason(prior_reason: Any) -> str:
# Heavy optional fields we can strip from a target's docs block without losing
# its identity. Ordering matters: earlier entries are dropped first because they
# carry the most bytes per unit of navigational value.
-HEAVY_DOC_FIELDS: tuple[str, ...] = ("content_md", "documentation", "file_summary")
+HEAVY_DOC_FIELDS: tuple[str, ...] = (
+ "content_md",
+ "digest_md",
+ "documentation",
+ "file_summary",
+)
def symbol_priority(sym: dict[str, Any], query_terms: set[str]) -> tuple[int, int, int]:
diff --git a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
index 126bd7053..6cbcc1b7d 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_answer/retrieval.py
@@ -240,8 +240,14 @@ async def _attach_page_excerpts(hits: list[dict], ctx: Any = None) -> int:
return len(top)
try:
async with get_session(ctx.session_factory) as session:
- res = await session.execute(select(Page.id, Page.content).where(Page.id.in_(page_ids)))
- content_by_id = {row[0]: (row[1] or "") for row in res.all()}
+ res = await session.execute(
+ select(Page.id, Page.content, Page.digest).where(Page.id.in_(page_ids))
+ )
+ # The digest carries what the page answers in an agent's words.
+ content_by_id = {
+ row[0]: "\n\n".join(part for part in (row[1], row[2]) if part)
+ for row in res.all()
+ }
except Exception:
# Never fail the answer over an excerpt fetch, but never hide it either.
_log.warning(
diff --git a/packages/server/src/repowise/server/mcp_server/tool_context/context.py b/packages/server/src/repowise/server/mcp_server/tool_context/context.py
index 6e0b0961e..f4a86a9ff 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_context/context.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_context/context.py
@@ -10,7 +10,8 @@ The split keeps the cached prompt prefix small on multi-turn agent sessions:
``get_context`` stays under ~2k tokens for common targets.
Optional ``include`` parameter widens the response:
- - include=["full_doc"] → full wiki markdown content
+ - include=["full_doc"] -> full wiki markdown content, plus the page's agent
+ digest (questions, identifiers, git signals)
- include=["callers"] → who calls this symbol (symbol targets only)
- include=["callees"] → what this symbol calls (symbol targets only)
- include=["ownership"] → primary owner, bus factor, contributor count
diff --git a/packages/server/src/repowise/server/mcp_server/tool_context/targets.py b/packages/server/src/repowise/server/mcp_server/tool_context/targets.py
index cf951790b..35a1887c9 100644
--- a/packages/server/src/repowise/server/mcp_server/tool_context/targets.py
+++ b/packages/server/src/repowise/server/mcp_server/tool_context/targets.py
@@ -709,6 +709,8 @@ async def _resolve_one_target(
docs["summary"] = page.summary or ""
if want_full_doc:
docs["content_md"] = page.content
+ if page.digest:
+ docs["digest_md"] = page.digest
if page.human_notes:
docs["human_notes"] = page.human_notes
# Symbols in this file
@@ -818,6 +820,10 @@ async def _resolve_one_target(
docs["section"] = page.section_number
if want_full_doc:
docs["content_md"] = page.content
+ # Questions, identifiers, public API and git signals: kept off
+ # the reader's page body, served to agents beside it.
+ if page.digest:
+ docs["digest_md"] = page.digest
# Non-file children only; file children are in "files" below.
res = await session.execute(
select(Page)
diff --git a/packages/server/src/repowise/server/schemas/pages.py b/packages/server/src/repowise/server/schemas/pages.py
index e3545e36c..5c3bb2549 100644
--- a/packages/server/src/repowise/server/schemas/pages.py
+++ b/packages/server/src/repowise/server/schemas/pages.py
@@ -158,6 +158,9 @@ class PageSummaryResponse(BaseModel):
class PageResponse(PageSummaryResponse):
content: str
+ # Agent material kept off the page body (questions, identifiers, public
+ # API, git signals). The reader shows it on request. Empty when none.
+ digest: str = ""
metadata: dict
@classmethod
@@ -168,6 +171,7 @@ class PageResponse(PageSummaryResponse):
return cls(
**_summary_fields(obj, metadata),
content=obj.content, # type: ignore[attr-defined]
+ digest=obj.digest or "", # type: ignore[attr-defined]
metadata=metadata,
)
diff --git a/packages/types/src/docs.ts b/packages/types/src/docs.ts
index a525e5e83..af51b26ca 100644
--- a/packages/types/src/docs.ts
+++ b/packages/types/src/docs.ts
@@ -84,6 +84,13 @@ export interface DocPageSummary {
export interface DocPage extends DocPageSummary {
content: string;
+ /**
+ * The page's agent digest: the questions it answers, its identifiers,
+ * public API and git signals. Indexed and served to agents beside
+ * `content`, kept off the body, shown by the reader on request. Absent or
+ * empty on pages that carry none.
+ */
+ digest?: string;
metadata: Record<string, unknown>;
}
diff --git a/packages/types/src/generated/http.ts b/packages/types/src/generated/http.ts
index 8bfb0668b..32299a096 100644
--- a/packages/types/src/generated/http.ts
+++ b/packages/types/src/generated/http.ts
@@ -2392,6 +2392,7 @@ export interface PageResponse {
created_at: string;
updated_at: string;
content: string;
+ digest?: string;
metadata: Record<string, unknown>;
}
diff --git a/packages/ui/__tests__/docs/docs-reader-reference.test.tsx b/packages/ui/__tests__/docs/docs-reader-reference.test.tsx
new file mode 100644
index 000000000..208495eb6
--- /dev/null
+++ b/packages/ui/__tests__/docs/docs-reader-reference.test.tsx
@@ -0,0 +1,94 @@
+import { describe, it, expect, beforeAll } from "vitest";
+import { fireEvent, render, screen } from "@testing-library/react";
+import { DocsReader } from "../../src/docs/docs-reader.js";
+import type { DocPage } from "@repowise-dev/types/docs";
+
+/**
+ * A page's agent digest is one click away, never in the reading path.
+ *
+ * The body is what a reader came for. The questions, identifiers and git
+ * signals written for search and agents sit behind a Reference tab, which
+ * exists only on pages that carry a digest. Git signals also show as a small
+ * History card beside the page.
+ */
+
+beforeAll(() => {
+ Element.prototype.scrollTo = () => {};
+});
+
+function makePage(overrides: Partial<DocPage> = {}): DocPage {
+ return {
+ id: "module_page:pkg/parser",
+ repository_id: "r1",
+ page_type: "module_page",
+ title: "Parsing",
+ content: "The parser turns files into records.",
+ target_path: "pkg/parser",
+ source_hash: "h",
+ model_name: "m",
+ provider_name: "openai",
+ input_tokens: 0,
+ output_tokens: 0,
+ cached_tokens: 0,
+ generation_level: 4,
+ version: 1,
+ confidence: 1,
+ freshness_status: "fresh",
+ metadata: {},
+ created_at: "2026-08-01T00:00:00Z",
+ updated_at: "2026-08-01T00:00:00Z",
+ ...overrides,
+ } as DocPage;
+}
+
+function renderReader(page: DocPage) {
+ return render(
+ <DocsReader
+ page={page}
+ repoId="r1"
+ persona="contributor"
+ sidebarOpen={false}
+ buildPageHref={(id) => `?page=${id}`}
+ LinkComponent={({ href, children, ...rest }) => (
+ <a href={href} {...rest}>
+ {children}
+ </a>
+ )}
+ />,
+ );
+}
+
+const DIGEST = "## Questions this page answers\n\n- How is a file parsed?";
+
+describe("DocsReader reference view", () => {
+ it("shows the body first and the digest behind the Reference tab", () => {
+ renderReader(makePage({ digest: DIGEST }));
+
+ expect(screen.getByText("The parser turns files into records.")).toBeTruthy();
+ expect(screen.queryByText("How is a file parsed?")).toBeNull();
+
+ fireEvent.click(screen.getByRole("tab", { name: "Reference" }));
+
+ expect(screen.getByText("How is a file parsed?")).toBeTruthy();
+ expect(screen.queryByText("The parser turns files into records.")).toBeNull();
+ });
+
+ it("offers no tabs on a page without a digest", () => {
+ renderReader(makePage());
+
+ expect(screen.queryByRole("tab")).toBeNull();
+ });
+
+ it("summarises the module's git history beside the page", () => {
+ renderReader(
+ makePage({
+ metadata: {
+ module_signals: { files: 12, hotspots: 2, owners: [{ name: "Ada", files: 9 }] },
+ },
+ }),
+ );
+
+ expect(screen.getByText("Ada · 9 of 12 files")).toBeTruthy();
+ expect(screen.getAllByText("2 of 12 files").length).toBeGreaterThan(0);
+ });
+});
diff --git a/packages/ui/__tests__/docs/reader-persona.test.ts b/packages/ui/__tests__/docs/reader-persona.test.ts
index bad8aac21..39f2ed537 100644
--- a/packages/ui/__tests__/docs/reader-persona.test.ts
+++ b/packages/ui/__tests__/docs/reader-persona.test.ts
@@ -8,12 +8,10 @@ import {
/**
* The reader lens over a deterministic page.
*
- * A file page's `## In the code` and `## Questions this page answers` are
- * written for the index, not for a reader: one is a bag of the file's own
- * tokens, the other is question-shaped text so a query has something to match.
- * They cannot be dropped upstream — `content` is a single string that FTS
- * indexes, the vector store embeds and `get_context` returns verbatim — so the
- * lens is where they stop being a reader's problem.
+ * A file page's `## Questions this page answers` is written for the index,
+ * not for a reader: question-shaped text so a query has something to match.
+ * It lives in `content`, which FTS indexes and `get_context` returns, so the
+ * lens is where it stops being a reader's problem.
*
* What the rendered page actually emits is pinned on the Python side, against
* the template rather than against a string typed here.
@@ -35,10 +33,6 @@ const PAGE = [
"## Questions this page answers",
"",
"- What does `src/walk.py` export?",
- "",
- "## In the code",
- "",
- "prune_dirs max_depth node_modules",
].join("\n");
describe("filterMarkdownByPersona", () => {
@@ -46,8 +40,7 @@ describe("filterMarkdownByPersona", () => {
const out = filterMarkdownByPersona(PAGE, "contributor");
expect(out).not.toContain("## Questions this page answers");
- expect(out).not.toContain("## In the code");
- expect(out).not.toContain("prune_dirs max_depth node_modules");
+ expect(out).not.toContain("What does `src/walk.py` export?");
});
it("keeps the reference material the default reader came for", () => {
@@ -84,3 +77,23 @@ describe("filterMarkdownByPersona", () => {
expect(personaFilteringApplies("# Title\n\n## Overview\n\nProse.\n")).toBe(false);
});
});
+
+describe("whole-heading matching", () => {
+ it("keeps a step-named section that starts like a hidden heading", () => {
+ const page = [
+ "# Ingestion",
+ "",
+ "## Source files become symbols",
+ "",
+ "Parsing turns bytes into records.",
+ "",
+ "## Source",
+ "",
+ "```py\nx = 1\n```",
+ ].join("\n");
+ const out = filterMarkdownByPersona(page, "overview");
+
+ expect(out).toContain("## Source files become symbols");
+ expect(out).not.toContain("## Source\n");
+ });
+});
diff --git a/packages/ui/src/docs/docs-reader.tsx b/packages/ui/src/docs/docs-reader.tsx
index 7a4e2547c..babed4d4f 100644
--- a/packages/ui/src/docs/docs-reader.tsx
+++ b/packages/ui/src/docs/docs-reader.tsx
@@ -1,6 +1,6 @@
"use client";
-import { useCallback, useEffect, useMemo, useRef } from "react";
+import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import {
FileText,
ArrowRight,
@@ -14,6 +14,7 @@ import { formatRelativeTime, formatTokens } from "../lib/format";
import { getPageLabel, isStubFallbackPage } from "../lib/page-types";
import { computeDocNav } from "./doc-nav";
import { filterMarkdownByPersona, type ReaderPersona } from "./reader-persona";
+import { ModuleHistory } from "./module-history";
import { WikiMarkdown } from "../wiki/wiki-markdown";
import { TableOfContents } from "../wiki/table-of-contents";
import { BacklinksPanel } from "../wiki/backlinks-panel";
@@ -24,6 +25,7 @@ import {
type RelatedReason,
} from "../wiki/wiki-links-types";
import { Breadcrumb } from "../shared/breadcrumb";
+import { ViewTabs } from "../shared/view-tabs";
import { Skeleton } from "../ui/skeleton";
/** Related entries shown before the "+ N more" line. Five, not eight: the list
@@ -31,6 +33,11 @@ import { Skeleton } from "../ui/skeleton";
* of everything the graph knows. */
const RELATED_LIMIT = 5;
+const READER_VIEWS = [
+ { id: "page", label: "Page" },
+ { id: "reference", label: "Reference" },
+];
+
const RELATED_REASON_LABELS: Record<RelatedReason, string> = {
imports: "imports",
"imported-by": "imported by",
@@ -195,6 +202,8 @@ export function DocsReader({
return (
<DocsReaderBody
+ // Keyed so the Page / Reference choice resets with the page.
+ key={page.id}
page={page}
pages={pages}
repoId={repoId}
@@ -307,6 +316,12 @@ function DocsReaderBody({
[bodyContent, persona],
);
+ // The agent digest: questions, identifiers, public API and git signals the
+ // page carries for search and MCP, kept off the body and shown on request.
+ const digest = page.digest?.trim() ?? "";
+ const [view, setView] = useState<"page" | "reference">("page");
+ const showingReference = view === "reference" && digest !== "";
+
// The nearest ancestor that is actually a module. Breadcrumbs now come from
// the stored tree, whose ancestors can be a layer or the file a symbol was
// spotted in; showing either as "in <name>" would repeat the layer chip
@@ -429,6 +444,23 @@ function DocsReaderBody({
return Comp;
}, [goToPageId]);
+ /* No `prose` wrapper: every element the renderer emits is already styled
+ through our own tokens, so the plugin contributed exactly two things, a
+ hardcoded `prose-invert` that fed dark variables to light mode and
+ `code::before/::after { content: "`" }`, which printed literal backticks
+ around every unresolved inline ref. */
+ const article = (markdown: string) => (
+ <article data-chat-selection="" className="max-w-none leading-relaxed overflow-hidden">
+ <WikiMarkdown
+ content={markdown}
+ wikiLinks={wikiLinks}
+ buildHref={(pid) => buildPageHref(pid)}
+ LinkComponent={WikiInlineLink}
+ pages={pages}
+ />
+ </article>
+ );
+
return (
<div className="flex h-full">
<div className="flex flex-col flex-1 min-w-0">
@@ -597,24 +629,27 @@ function DocsReaderBody({
</div>
)}
- {/* Markdown content.
- No `prose` wrapper: every element the renderer emits is already
- styled through our own tokens, so the plugin contributed exactly
- two things — a hardcoded `prose-invert` that fed dark variables
- to light mode, and `code::before/::after { content: "`" }`, which
- printed literal backticks around every unresolved inline ref. */}
- <article
- data-chat-selection=""
- className="max-w-none leading-relaxed overflow-hidden"
- >
- <WikiMarkdown
- content={visibleContent}
- wikiLinks={wikiLinks}
- buildHref={(pid) => buildPageHref(pid)}
- LinkComponent={WikiInlineLink}
- pages={pages}
- />
- </article>
+ {/* Markdown content. A page with an agent digest gets a Page /
+ Reference switch: the body is what a reader came for, the
+ digest is what search and agents read, one click away. */}
+ {digest ? (
+ <ViewTabs
+ tabs={READER_VIEWS}
+ value={view}
+ onValueChange={(id) => setView(id === "reference" ? "reference" : "page")}
+ aria-label="Page or reference"
+ >
+ {showingReference && (
+ <p className="mb-4 text-xs text-[var(--color-text-tertiary)]">
+ What this page carries for search and agents: the questions it answers,
+ the identifiers behind it and its git history.
+ </p>
+ )}
+ {article(showingReference ? digest : visibleContent)}
+ </ViewTabs>
+ ) : (
+ article(visibleContent)
+ )}
{/* The upgrade affordance, at the end of the content rather than
beside the title. Someone who has read to here knows the page is
@@ -714,6 +749,7 @@ function DocsReaderBody({
the intelligence sections and the contents. The breakpoint is
2xl, not lg — see the rail below for why. */}
<div className="mt-10 grid gap-8 border-t border-[var(--color-border-default)] pt-6 sm:grid-cols-2 2xl:hidden">
+ <ModuleHistory metadata={page.metadata} />
{intelligenceSlot && (
<div className="flex flex-col gap-4">{intelligenceSlot}</div>
)}
@@ -765,7 +801,9 @@ function DocsReaderBody({
{sidebarOpen && (
<div className="hidden 2xl:block shrink-0 w-[300px] overflow-auto">
<div className="flex flex-col gap-7 py-8 pl-6 pr-7">
- <TableOfContents content={bodyContent} />
+ <TableOfContents content={showingReference ? digest : bodyContent} />
+
+ <ModuleHistory metadata={page.metadata} />
{intelligenceSlot}
diff --git a/packages/ui/src/docs/module-history.tsx b/packages/ui/src/docs/module-history.tsx
new file mode 100644
index 000000000..fec13cab0
--- /dev/null
+++ b/packages/ui/src/docs/module-history.tsx
@@ -0,0 +1,109 @@
+/**
+ * Git history for the files a module page covers: who maintains them, where
+ * change and fixes concentrate, and what moves with them. Read from the
+ * `module_signals` the generator stamps on the page, the same data the page's
+ * agent digest states in prose, so the two cannot disagree.
+ */
+
+import { cn } from "../lib/cn";
+
+interface ModuleSignals {
+ files?: number;
+ hotspots?: number;
+ bus_factor_one?: number;
+ bug_fixes?: number;
+ most_fixed_file?: string;
+ owners?: { name: string; files: number }[];
+ most_active_file?: string;
+ most_active_commits_90d?: number;
+ co_changes?: { path: string; files: number }[];
+}
+
+function readSignals(metadata: Record<string, unknown> | undefined): ModuleSignals | null {
+ const raw = metadata?.module_signals;
+ if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;
+ return raw as ModuleSignals;
+}
+
+function basename(path: string): string {
+ return path.split("/").filter(Boolean).pop() ?? path;
+}
+
+function Row({
+ label,
+ value,
+ title,
+ mono = true,
+}: {
+ label: string;
+ value: string;
+ title?: string | undefined;
+ /** Counts and paths are machine facts; a person's name is not. */
+ mono?: boolean;
+}) {
+ const full = title ?? value;
+ return (
+ <div className="flex items-baseline justify-between gap-3">
+ <span className="shrink-0 text-[var(--color-text-tertiary)]">{label}</span>
+ {/* The visible value truncates; the full one stays readable to assistive tech. */}
+ <span
+ className={cn("min-w-0 truncate text-right tabular-nums", mono && "font-mono")}
+ title={full}
+ aria-hidden
+ >
+ {value}
+ </span>
+ <span className="sr-only">{full}</span>
+ </div>
+ );
+}
+
+export function ModuleHistory({ metadata }: { metadata: Record<string, unknown> | undefined }) {
+ const s = readSignals(metadata);
+ if (!s) return null;
+ const of = s.files ? ` of ${s.files}` : "";
+ const owner = s.owners?.[0];
+ const coChange = s.co_changes?.[0];
+ return (
+ <div>
+ <p className="mb-2.5 font-mono text-[10px] uppercase tracking-[0.12em] text-[var(--color-text-tertiary)]">
+ History
+ </p>
+ <div className="space-y-1 text-xs text-[var(--color-text-secondary)]">
+ {owner && (
+ <Row
+ label="Owner"
+ mono={false}
+ value={`${owner.name} · ${owner.files}${of} files`}
+ title={s.owners!.map((o) => `${o.name} (${o.files} files)`).join(", ")}
+ />
+ )}
+ {s.hotspots ? <Row label="Hotspots" value={`${s.hotspots}${of} files`} /> : null}
+ {s.bus_factor_one ? (
+ <Row label="Single maintainer" value={`${s.bus_factor_one}${of} files`} />
+ ) : null}
+ {s.bug_fixes ? (
+ <Row
+ label="Bug fixes"
+ value={s.most_fixed_file ? `${s.bug_fixes} · most in ${basename(s.most_fixed_file)}` : `${s.bug_fixes}`}
+ title={s.most_fixed_file ? `${s.bug_fixes} bug-fix commits, most in ${s.most_fixed_file}` : undefined}
+ />
+ ) : null}
+ {s.most_active_file && (
+ <Row
+ label="Most active"
+ value={`${basename(s.most_active_file)} · ${s.most_active_commits_90d ?? 0} in 90d`}
+ title={`${s.most_active_file}: ${s.most_active_commits_90d ?? 0} commits in 90 days`}
+ />
+ )}
+ {coChange && (
+ <Row
+ label="Changes with"
+ value={coChange.path}
+ title={s.co_changes!.map((c) => c.path).join(", ")}
+ />
+ )}
+ </div>
+ </div>
+ );
+}
diff --git a/packages/ui/src/docs/reader-persona.ts b/packages/ui/src/docs/reader-persona.ts
index ebc90fe11..c6cbd689d 100644
--- a/packages/ui/src/docs/reader-persona.ts
+++ b/packages/ui/src/docs/reader-persona.ts
@@ -25,50 +25,29 @@ export function isReaderPersona(value: string | null | undefined): value is Read
return value === "overview" || value === "contributor" || value === "deep";
}
-// Heading keywords (lowercased, matched as a prefix of the heading text) that
-// each persona hides. Matching is intentionally generous so it works across
-// the file/module/layer page templates without an exhaustive list.
+// Whole section headings (lowercased) that each persona hides. Exact matches
+// only: a model-written page names its sections after what the code does, and
+// a prefix such as "source" or "symbol" would delete a section called
+// "Source files become symbols". Each entry is a heading a structural template
+// emits (see ``structural_labels.py`` and the stub templates).
-// Contributor keeps almost everything; only the rawest machine-oriented dumps
-// are hidden so the page stays readable.
-//
-// The last three are retrieval scaffolding rather than reference material. A
-// deterministic page carries them so the index has the words a question is
-// asked in — the vocabulary bag and the question block are the only parts of
-// a file page written in a reader's words rather than in paths, and the third
-// is the importer list a symbol page keeps once its resolved callers have
-// answered the question. They earn their place in `content`, which is one
-// string that FTS indexes, the vector store embeds and `get_context` returns
-// verbatim; they do not earn a reader's scroll. Hidden here rather than
-// dropped upstream so the index keeps them.
-const CONTRIBUTOR_HIDE = [
- "raw metrics",
- "parse errors",
- "source snippet",
- "in the code",
- "questions this page answers",
- "files importing this module",
-];
+// Contributor hides only retrieval scaffolding a file or symbol page still
+// carries in `content`: the question block, written so the index has the words
+// a question is asked in, and the importer list a symbol page keeps once its
+// resolved callers have answered the question. Pages with an agent digest
+// carry that material outside `content`, and the reader offers it as the
+// Reference view instead.
+const CONTRIBUTOR_HIDE = ["questions this page answers", "files importing this module"];
-// Overview is contributor plus the reference dumps. Spread rather than
-// restated: a section the balanced lens hides must never reappear in the
-// narrower one.
+// Overview is contributor plus the reference dumps. Spread, not restated: a
+// section the balanced lens hides must never reappear in the narrower one.
const OVERVIEW_HIDE = [
...CONTRIBUTOR_HIDE,
- "symbols",
- "symbol",
"public api",
- "imports",
- "exports",
- "dependencies",
- "dependents",
- "call graph",
- "class hierarchy",
- "community",
- "neighbors",
- "dead code",
"source",
- "metrics",
+ "symbols defined in the cycle",
+ "dependencies (modules this imports)",
+ "dependents (modules that import this)",
];
const HIDE_BY_PERSONA: Record<ReaderPersona, string[]> = {
@@ -78,8 +57,7 @@ const HIDE_BY_PERSONA: Record<ReaderPersona, string[]> = {
};
function headingHidden(headingText: string, hideList: string[]): boolean {
- const h = headingText.trim().toLowerCase();
- return hideList.some((kw) => h === kw || h.startsWith(kw));
+ return hideList.includes(headingText.trim().toLowerCase());
}
/**
diff --git a/packages/web/src/components/docs/docs-viewer.tsx b/packages/web/src/components/docs/docs-viewer.tsx
index 56ac2d455..29d9f0e8b 100644
--- a/packages/web/src/components/docs/docs-viewer.tsx
+++ b/packages/web/src/components/docs/docs-viewer.tsx
@@ -304,11 +304,14 @@ export function DocsViewer({
[repoId],
);
- const hasTargetPath = !!page?.target_path;
const targetPath = page?.target_path ?? "";
+ // Decided by page type, not path shape: a module page's target is a
+ // directory, which every row below would answer with nothing, and the
+ // reader already gives module pages their own history card.
+ const showSignals = !!targetPath && page?.page_type !== "module_page";
// Security findings only make sense for real source files.
const isFilePath =
- hasTargetPath &&
+ showSignals &&
!targetPath.includes("::") &&
!targetPath.startsWith("onboarding/") &&
!targetPath.startsWith("layer:");
@@ -347,7 +350,7 @@ export function DocsViewer({
) : undefined
}
intelligenceSlot={
- hasTargetPath ? (
+ showSignals ? (
// One "Signals" block, not five. At a glance, Importance, Community,
// Call graph and Security each announced themselves with their own
// uppercase label over a handful of rows, so the rail was mostly
diff --git a/tests/unit/cli/test_doctor_page_cap.py b/tests/unit/cli/test_doctor_page_cap.py
index 2266c612c..ed3b995ca 100644
--- a/tests/unit/cli/test_doctor_page_cap.py
+++ b/tests/unit/cli/test_doctor_page_cap.py
@@ -113,7 +113,7 @@ async def _build_repo(tmp_path: Path) -> Path:
fts = FullTextSearch(engine)
await fts.ensure_index()
await fts.index_many(
- [(r["id"], r["title"], r["content"], "", r["target_path"]) for r in rows]
+ [(r["id"], r["title"], r["content"], "", r["target_path"], "") for r in rows]
)
await engine.dispose()
return repo_path
diff --git a/tests/unit/cli/test_embedder_resolution.py b/tests/unit/cli/test_embedder_resolution.py
index 23e8c092d..0cb7f4b5a 100644
--- a/tests/unit/cli/test_embedder_resolution.py
+++ b/tests/unit/cli/test_embedder_resolution.py
@@ -217,6 +217,7 @@ async def test_reindex_persists_its_resolved_embedder(
# Non-nullable on the real model, so a stand-in that omits it is a
# stand-in for a page that cannot exist.
summary = "what a.py does"
+ digest = ""
metadata_json = "{}"
class _Result:
diff --git a/tests/unit/cli/test_full_upgrade_progress.py b/tests/unit/cli/test_full_upgrade_progress.py
index bdf52dd36..902eb4cef 100644
--- a/tests/unit/cli/test_full_upgrade_progress.py
+++ b/tests/unit/cli/test_full_upgrade_progress.py
@@ -99,6 +99,7 @@ def test_full_upgrade_fts_cleanup_retries_after_failure(
content="# Current",
summary="Current page",
target_path="current.py",
+ digest="",
)
with pytest.raises(RuntimeError, match="temporary FTS failure"):
diff --git a/tests/unit/cli/test_reindex_cmd.py b/tests/unit/cli/test_reindex_cmd.py
index 88c74f860..c6fbbc248 100644
--- a/tests/unit/cli/test_reindex_cmd.py
+++ b/tests/unit/cli/test_reindex_cmd.py
@@ -128,6 +128,7 @@ async def test_reindex_aborts_when_every_item_failed(
target_path = "src/main.py"
summary = ""
content = "body"
+ digest = ""
metadata_json = "{}"
decision = "a decision"
diff --git a/tests/unit/cli/test_reindex_embed_recipe.py b/tests/unit/cli/test_reindex_embed_recipe.py
index f4ff02e4b..8bee8d1fd 100644
--- a/tests/unit/cli/test_reindex_embed_recipe.py
+++ b/tests/unit/cli/test_reindex_embed_recipe.py
@@ -24,6 +24,7 @@ class _DummyEngine:
class _Page:
metadata_json = "{}" # non-nullable on the real model
+ digest = "" # non-nullable on the real model
def __init__(self, **kw: Any) -> None:
self.__dict__.update(kw)
diff --git a/tests/unit/generation/test_agent_digest.py b/tests/unit/generation/test_agent_digest.py
new file mode 100644
index 000000000..0daadcb5a
--- /dev/null
+++ b/tests/unit/generation/test_agent_digest.py
@@ -0,0 +1,204 @@
+"""A module page keeps agent material in its digest, off the body a reader reads.
+
+The questions a page answers, its identifiers and its git signals help search
+and agents; a reader scrolled past them. They now live in ``page.digest``,
+which is indexed and served beside the body, so the body can be prose.
+"""
+
+from __future__ import annotations
+
+import networkx as nx
+
+from repowise.core.generation.agent_digest import (
+ MODULE_SIGNALS_KEY,
+ rejoin_questions,
+ split_questions,
+)
+from repowise.core.generation.context_assembler import ContextAssembler, FilePageContext
+from repowise.core.generation.models import GenerationConfig, compute_page_id
+from repowise.core.generation.page_generator import PageGenerator, PriorPage
+from repowise.core.providers.llm.mock import MockProvider
+
+_BODY = "# Parsing\n\nThe parser turns files into records.\n\n## How a file is read\n\nIt reads.\n"
+_QUESTIONS = "## Questions this page answers\n\n- How is a file parsed?\n- Where do records go?"
+
+
+class _Provider(MockProvider):
+ async def generate(self, *args, **kwargs):
+ response = await super().generate(*args, **kwargs)
+ response.content = f"{_BODY}\n{_QUESTIONS}\n"
+ return response
+
+
+def _file_context(path: str) -> FilePageContext:
+ return FilePageContext(
+ file_path=path,
+ language="python",
+ docstring=None,
+ symbols=[
+ {
+ "name": "Parser",
+ "qualified_name": "Parser",
+ "kind": "class",
+ "signature": "class Parser",
+ "docstring": "",
+ "visibility": "public",
+ "is_async": False,
+ "complexity_estimate": 1,
+ "decorators": [],
+ "parent_name": "",
+ "start_line": 1,
+ "end_line": 9,
+ }
+ ],
+ imports=[],
+ exports=["Parser"],
+ pagerank_score=0.3,
+ betweenness_score=0.0,
+ community_id=0,
+ dependents=[],
+ dependencies=[],
+ is_api_contract=False,
+ is_entry_point=False,
+ is_test=False,
+ parse_errors=[],
+ estimated_tokens=10,
+ )
+
+
+_GIT = {
+ "pkg/parser.py": {
+ "file_path": "pkg/parser.py",
+ "is_hotspot": True,
+ "bus_factor": 1,
+ "prior_defect_count": 3,
+ "primary_owner_name": "Ada",
+ "commit_count_90d": 7,
+ }
+}
+
+
+def _generator(config: GenerationConfig, **kwargs) -> PageGenerator:
+ return PageGenerator(_Provider(), ContextAssembler(config), config, **kwargs)
+
+
+async def _module_page(gen: PageGenerator, git_meta_map=None):
+ return await gen.generate_module_page(
+ "Parsing",
+ "python",
+ [_file_context("pkg/parser.py")],
+ nx.DiGraph(),
+ git_meta_map=git_meta_map,
+ target_path="pkg",
+ structural_key="k",
+ )
+
+
+def test_split_moves_the_questions_and_keeps_the_body():
+ body, questions = split_questions(f"{_BODY}\n{_QUESTIONS}\n\n## After\n\nMore prose.\n")
+
+ assert questions == _QUESTIONS
+ assert "Questions this page answers" not in body
+ assert "## How a file is read" in body and "## After" in body
+
+
+def test_split_ignores_a_heading_inside_a_code_fence():
+ content = "# T\n\n```md\n## Questions this page answers\n```\n"
+
+ assert split_questions(content) == (content, "")
+
+
+def test_rejoin_restores_the_response_a_body_was_split_from():
+ body, questions = split_questions(f"{_BODY}\n{_QUESTIONS}\n")
+
+ assert split_questions(rejoin_questions(body, questions)) == (body, questions)
+ assert rejoin_questions(body, "") == body
+
+
+async def test_the_model_questions_move_into_the_digest(sample_config):
+ page = await _module_page(_generator(sample_config))
+
+ assert "Questions this page answers" not in page.content
+ assert page.digest.startswith(_QUESTIONS)
+ assert "## Concept index" in page.digest
+ assert page.summary.startswith("The parser turns files into records")
+
+
+async def test_git_signals_reach_the_digest_and_the_card_not_the_body(sample_config):
+ page = await _module_page(_generator(sample_config), git_meta_map=_GIT)
+
+ signals = page.metadata[MODULE_SIGNALS_KEY]
+ assert signals["files"] == 1
+ assert signals["hotspots"] == 1
+ assert signals["bus_factor_one"] == 1
+ assert signals["bug_fixes"] == 3
+ assert signals["owners"] == [{"name": "Ada", "files": 1}]
+ assert "## Signals" in page.digest
+ assert "Bug-fix commits: 3, most in `pkg/parser.py`" in page.digest
+ assert "hotspot" not in page.content.lower()
+
+
+async def test_a_module_with_no_history_reports_no_signals(sample_config):
+ page = await _module_page(_generator(sample_config))
+
+ assert MODULE_SIGNALS_KEY not in page.metadata
+ assert "## Signals" not in page.digest
+
+
+async def test_the_keyless_page_splits_the_same_way():
+ config = GenerationConfig(
+ max_tokens=1024, token_budget=2000, max_concurrency=2, deterministic=True
+ )
+ page = await _module_page(_generator(config), git_meta_map=_GIT)
+
+ assert "Questions this page answers" not in page.content
+ assert page.digest.startswith("## Questions this page answers")
+ assert "Subsystem health" not in page.content
+ assert page.metadata[MODULE_SIGNALS_KEY]["hotspots"] == 1
+
+
+async def test_a_page_reused_from_a_prior_run_keeps_its_questions(sample_config):
+ first = await _module_page(_generator(sample_config))
+ prior = {
+ compute_page_id("module_page", "pkg"): PriorPage(
+ source_hash=first.source_hash,
+ model_name=first.model_name,
+ content=first.content,
+ digest=first.digest,
+ )
+ }
+
+ again = await _module_page(_generator(sample_config, prior_pages=prior))
+
+ assert again.metadata.get("reused_from_prior_run") is True
+ assert again.content == first.content
+ assert again.digest == first.digest
+
+
+async def test_a_reused_rollup_keeps_one_package_table(sample_config):
+ async def rollup(gen):
+ return await gen.generate_module_page(
+ "Parsing",
+ "python",
+ [_file_context("pkg/parser.py")],
+ nx.DiGraph(),
+ target_path="pkg",
+ structural_key="k",
+ packages=[{"path": "pkg/a", "files": 1}, {"path": "pkg/b", "files": 2}],
+ )
+
+ page = await rollup(_generator(sample_config))
+ for _ in range(2):
+ prior = {
+ page.page_id: PriorPage(
+ source_hash=page.source_hash,
+ model_name=page.model_name,
+ content=page.content,
+ digest=page.digest,
+ )
+ }
+ page = await rollup(_generator(sample_config, prior_pages=prior))
+
+ assert page.metadata.get("reused_from_prior_run") is True
+ assert page.content.count("## Packages") == 1
+ assert "| `pkg/b` | 2 |" in page.content
diff --git a/tests/unit/generation/test_concept_index_reaches_search.py b/tests/unit/generation/test_concept_index_reaches_search.py
index d4e53c30b..df44d5e5d 100644
--- a/tests/unit/generation/test_concept_index_reaches_search.py
+++ b/tests/unit/generation/test_concept_index_reaches_search.py
@@ -1,11 +1,11 @@
-"""The concept index has to reach the index, not just the rendered page.
+"""The concept index has to reach the index, not just the page's digest.
The table exists to make an identifier-exact query match the identifier rather
-than the model's description of it. That only happens if the table text lands
-in the full-text row. Nothing raises if it does not: the write path is handed
-``page.content`` verbatim, so a table appended after the row was written, or
-stripped on the way to persistence, would show up as a page that reads well and
-cannot be found by the one token a reader would search for.
+than the model's description of it. It lives in the page's agent digest, off
+the body a reader scrolls, so it only helps if the digest lands in the
+full-text row. Nothing raises if it does not: a digest dropped on the way to
+persistence would show up as a page that reads well and cannot be found by the
+one token a reader would search for.
Module pages are the page type with the least to match on — a real run produced
89 of them carrying four fenced code blocks between them — so this round trip
@@ -90,7 +90,7 @@ def _file_context(path: str, names: list[str]) -> FilePageContext:
)
-async def _index(engine, search, *, page_id, title, target_path, content):
+async def _index(engine, search, *, page_id, title, target_path, content, digest):
factory = async_sessionmaker(engine, expire_on_commit=False)
async with factory() as session:
repo = await upsert_repository(session, name="r", local_path="/tmp/r")
@@ -103,13 +103,16 @@ async def _index(engine, search, *, page_id, title, target_path, content):
title=title,
content=content,
summary="",
+ digest=digest,
target_path=target_path,
source_hash="h",
model_name="mock",
provider_name="mock",
)
await session.commit()
- await search.index(page_id, title, content, summary="", target_path=target_path)
+ await search.index(
+ page_id, title, content, summary="", target_path=target_path, digest=digest
+ )
async def _matches(search: FullTextSearch, query: str) -> list[str]:
@@ -129,9 +132,10 @@ async def test_a_module_page_is_findable_by_a_symbol_it_contains(fts):
target_path="core/resolvers",
structural_key="k",
)
- # The prose the model wrote does not contain the identifier, so a hit below
- # can only have come from the appended table.
- assert "ResolverContext" not in page.content.split("## Concept index")[0]
+ # The body does not contain the identifier, so a hit below can only have
+ # come from the digest's concept index.
+ assert "ResolverContext" not in page.content
+ assert "## Concept index" in page.digest
await _index(
engine,
@@ -140,6 +144,7 @@ async def test_a_module_page_is_findable_by_a_symbol_it_contains(fts):
title=page.title,
target_path=page.target_path,
content=page.content,
+ digest=page.digest,
)
assert page.page_id in await _matches(search, "ResolverContext")
@@ -167,6 +172,7 @@ async def test_the_prose_spelling_finds_the_page_too(fts):
title=page.title,
target_path=page.target_path,
content=page.content,
+ digest=page.digest,
)
assert page.page_id in await _matches(search, "Tsconfig resolver")
diff --git a/tests/unit/generation/test_concept_index_report.py b/tests/unit/generation/test_concept_index_report.py
index c05789020..f01a3a1bd 100644
--- a/tests/unit/generation/test_concept_index_report.py
+++ b/tests/unit/generation/test_concept_index_report.py
@@ -1,7 +1,7 @@
"""The run reports how far the module concept index reached.
-The table is appended in code, after the provider call, which is a quieter
-place to lose something than a template. An early return on an error path, or a
+The table is rendered into the page's digest in code, after the provider
+call, which is a quieter place to lose something than a template. An early return on an error path, or a
refactor that hands back the page before the append, leaves every module page
pure prose again — and every template test in the suite still passes, because
none of the templates changed. This counter is the only thing that would say so.
@@ -17,20 +17,20 @@ from repowise.core.generation.models import GeneratedPage
from repowise.core.generation.report import CONCEPT_INDEX_HEADING, GenerationReport
_WITH = (
- "# Resolution Layer\n\nProse.\n\n"
f"{CONCEPT_INDEX_HEADING}\n\n"
"| Concept | Symbol | File |\n| --- | --- | --- |\n"
"| Resolver context | `ResolverContext` | `a.py` |\n"
)
-_WITHOUT = "# Resolution Layer\n\nProse.\n"
+_WITHOUT = ""
-def _page(page_id: str, page_type: str, content: str) -> GeneratedPage:
+def _page(page_id: str, page_type: str, digest: str) -> GeneratedPage:
return GeneratedPage(
page_id=page_id,
page_type=page_type,
title=page_id,
- content=content,
+ content="# Resolution Layer\n\nProse.\n",
+ digest=digest,
source_hash="",
model_name="mock",
provider_name="mock",
diff --git a/tests/unit/generation/test_deterministic_templates.py b/tests/unit/generation/test_deterministic_templates.py
index 362dcda3a..aa3855fba 100644
--- a/tests/unit/generation/test_deterministic_templates.py
+++ b/tests/unit/generation/test_deterministic_templates.py
@@ -446,7 +446,7 @@ def test_module_page_leads_with_the_concept_title(generator):
group is; the line beneath says where to go and look.
"""
ctx = _module_ctx(generator._assembler, ["src/ingest/read.py", "src/parse/ast.py"])
- page = generator._stub_module_page(ctx, "src/ingest", "Ingestion Pipeline", None)
+ page = generator._stub_module_page(ctx, "src/ingest", "Ingestion Pipeline")
assert page.title == "Ingestion Pipeline"
assert page.content.startswith("# Ingestion Pipeline\n")
@@ -463,7 +463,7 @@ def test_module_page_of_root_files_says_so(generator):
top level.
"""
ctx = _module_ctx(generator._assembler, ["setup.py", "main.py"])
- page = generator._stub_module_page(ctx, "root", "Project Entry Points", None)
+ page = generator._stub_module_page(ctx, "root", "Project Entry Points")
assert "Repository root" in page.content
assert "`.`" not in page.content
diff --git a/tests/unit/generation/test_module_page_concept_index.py b/tests/unit/generation/test_module_page_concept_index.py
index d5116f184..c329203c7 100644
--- a/tests/unit/generation/test_module_page_concept_index.py
+++ b/tests/unit/generation/test_module_page_concept_index.py
@@ -7,10 +7,11 @@ grep for, and an identifier-exact query matches the description of the code
rather than the code.
The concept index closes that. It is rendered from the assembled symbol data
-and appended after the model has written the page, so the identifiers and the
-paths in it are the index's, not the model's, and no provider response can
-change them. That placement is the whole point: put the same table in the
-prompt and the model may reformat it, abbreviate a path, or drop it.
+into the page's agent digest after the model has written the page, so the
+identifiers and the paths in it are the index's, not the model's, and no
+provider response can change them. That placement is the whole point: put the
+same table in the prompt and the model may reformat it, abbreviate a path, or
+drop it.
A page whose members export nothing renders no table rather than an empty one —
a header row with no rows under it is a claim that the module has no public
@@ -140,13 +141,13 @@ async def test_rendered_page_carries_real_symbol_names_and_paths(
"""The identifiers reach the page the model wrote, spelled exactly."""
page = await _module_page(prose_only_generator, resolver_module.contexts)
- assert CONCEPT_INDEX_HEADING in page.content
- assert "| Concept | Symbol | File |" in page.content
- assert "`ResolverContext`" in page.content
- assert "`resolve_imports`" in page.content
- assert "`TsconfigResolver`" in page.content
- assert "`packages/core/src/repowise/core/ingestion/resolvers/context.py`" in page.content
- assert "`packages/core/src/repowise/core/ingestion/resolvers/tsconfig.py`" in page.content
+ assert CONCEPT_INDEX_HEADING in page.digest
+ assert "| Concept | Symbol | File |" in page.digest
+ assert "`ResolverContext`" in page.digest
+ assert "`resolve_imports`" in page.digest
+ assert "`TsconfigResolver`" in page.digest
+ assert "`packages/core/src/repowise/core/ingestion/resolvers/context.py`" in page.digest
+ assert "`packages/core/src/repowise/core/ingestion/resolvers/tsconfig.py`" in page.digest
async def test_concept_column_spells_the_identifier_as_prose(prose_only_generator, resolver_module):
@@ -154,14 +155,14 @@ async def test_concept_column_spells_the_identifier_as_prose(prose_only_generato
on the right, both on one line so either wording finds the page."""
page = await _module_page(prose_only_generator, resolver_module.contexts)
- assert "| Resolver context | `ResolverContext` |" in page.content
- assert "| Resolve imports | `resolve_imports` |" in page.content
+ assert "| Resolver context | `ResolverContext` |" in page.digest
+ assert "| Resolve imports | `resolve_imports` |" in page.digest
async def test_non_public_symbols_stay_off_the_table(prose_only_generator, resolver_module):
page = await _module_page(prose_only_generator, resolver_module.contexts)
- assert "_private_helper" not in page.content
+ assert "_private_helper" not in page.digest
async def test_a_module_with_no_public_symbols_renders_no_table(prose_only_generator):
@@ -173,18 +174,18 @@ async def test_a_module_with_no_public_symbols_renders_no_table(prose_only_gener
page = await _module_page(prose_only_generator, contexts)
- assert CONCEPT_INDEX_HEADING not in page.content
- assert "| Concept | Symbol | File |" not in page.content
+ assert CONCEPT_INDEX_HEADING not in page.digest
+ assert "| Concept | Symbol | File |" not in page.digest
-async def test_the_table_starts_on_its_own_line(prose_only_generator, resolver_module):
- """Appending is string concatenation, and a heading glued to the last line
- of the model's prose renders as body text. Assert the separator that has to
- be there, not merely the absence of extra ones."""
+async def test_the_table_stays_off_the_body(prose_only_generator, resolver_module):
+ """The body is what a reader scrolls; the table is for search and agents."""
page = await _module_page(prose_only_generator, resolver_module.contexts)
- assert f"\n\n{CONCEPT_INDEX_HEADING}\n" in page.content
- assert "\n\n\n" not in page.content
+ assert CONCEPT_INDEX_HEADING not in page.content
+ assert "`ResolverContext`" not in page.content
+ assert page.digest.startswith(CONCEPT_INDEX_HEADING)
+ assert "\n\n\n" not in page.digest
def test_concept_index_orders_by_the_page_rank_of_the_file():
diff --git a/tests/unit/persistence/test_information_floor.py b/tests/unit/persistence/test_information_floor.py
index cfcb9f7e8..f3da3735c 100644
--- a/tests/unit/persistence/test_information_floor.py
+++ b/tests/unit/persistence/test_information_floor.py
@@ -290,8 +290,8 @@ def test_the_floor_is_off_unless_it_is_set(monkeypatch):
async def test_a_batch_holds_out_only_the_thin_pages(fts):
await fts.index_many(
[
- ("file_page:esbuild.mjs", "File: esbuild.mjs", SKELETON, "", "esbuild.mjs"),
- ("file_page:search.py", "File: search.py", SUBSTANTIAL, "", "search.py"),
+ ("file_page:esbuild.mjs", "File: esbuild.mjs", SKELETON, "", "esbuild.mjs", ""),
+ ("file_page:search.py", "File: search.py", SUBSTANTIAL, "", "search.py", ""),
]
)
@@ -303,7 +303,7 @@ async def test_a_batch_drops_the_row_of_a_page_that_thinned_out(fts):
"""The delete runs for every id in the batch, insert only for those above."""
await fts.index("file_page:a.py", "File: a.py", SUBSTANTIAL, summary="", target_path="a.py")
- await fts.index_many([("file_page:a.py", "File: a.py", SKELETON, "", "a.py")])
+ await fts.index_many([("file_page:a.py", "File: a.py", SKELETON, "", "a.py", "")])
assert await fts.list_indexed_ids() == set()
assert await fts.search("FTS5 virtual table") == []
@@ -313,8 +313,8 @@ async def test_a_batch_whose_last_entry_is_thin_leaves_no_row(fts):
"""Last-entry-wins has to hold when the last entry is the excluded one."""
await fts.index_many(
[
- ("file_page:a.py", "File: a.py", SUBSTANTIAL, "", "a.py"),
- ("file_page:a.py", "File: a.py", SKELETON, "", "a.py"),
+ ("file_page:a.py", "File: a.py", SUBSTANTIAL, "", "a.py", ""),
+ ("file_page:a.py", "File: a.py", SKELETON, "", "a.py", ""),
]
)
diff --git a/tests/unit/persistence/test_migration_fts_frozen.py b/tests/unit/persistence/test_migration_fts_frozen.py
new file mode 100644
index 000000000..01aae1b16
--- /dev/null
+++ b/tests/unit/persistence/test_migration_fts_frozen.py
@@ -0,0 +1,59 @@
+"""Migrations that touch full-text search carry their own SQL.
+
+A migration that imports the live search constants replays today's column set
+on a database migrated from an older revision, where those columns do not
+exist yet. Rendered offline against PostgreSQL, each revision has to create
+exactly the index its own schema supports.
+"""
+
+from __future__ import annotations
+
+import io
+from pathlib import Path
+
+import pytest
+from alembic import command
+from alembic.config import Config
+
+_CORE = Path(__file__).resolve().parents[3] / "packages" / "core"
+_VERSIONS = _CORE / "alembic" / "versions"
+
+
+def _offline_sql(monkeypatch, revisions: str) -> str:
+ monkeypatch.setenv("DATABASE_URL", "postgresql://user:pass@localhost/db")
+ buffer = io.StringIO()
+ # No ini file: env.py would hand it to logging.fileConfig, which disables
+ # every logger already created and breaks log assertions in later tests.
+ config = Config(output_buffer=buffer)
+ config.set_main_option("script_location", str(_CORE / "alembic"))
+ command.upgrade(config, revisions, sql=True)
+ return buffer.getvalue()
+
+
+def test_no_migration_imports_the_live_search_constants():
+ offenders = [
+ p.name for p in _VERSIONS.glob("*.py") if "repowise.core.persistence.search" in p.read_text()
+ ]
+ assert offenders == []
+
+
+ ("revisions", "has_digest"), [("0043:0044", False), ("0094:0095", True)]
+)
+def test_each_revision_builds_the_index_its_schema_supports(monkeypatch, revisions, has_digest):
+ sql = _offline_sql(monkeypatch, revisions)
+ index = next(line for line in sql.splitlines() if "CREATE INDEX idx_wiki_pages_fts" in line)
+
+ assert "COALESCE(target_path,'')" in index
+ assert ("COALESCE(digest,'')" in index) is has_digest
+
+
+def test_the_sqlite_shapes_around_the_digest_keep_the_vocabulary():
+ import importlib.util
+
+ spec = importlib.util.spec_from_file_location("m0095", _VERSIONS / "0095_page_digest.py")
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+
+ assert "target_path, vocabulary, digest)" in module.PAGE_FTS_DDL
+ assert "target_path, vocabulary)" in module._OLD_SQLITE_DDL
diff --git a/tests/unit/persistence/test_search.py b/tests/unit/persistence/test_search.py
index bdc2916c6..315439d42 100644
--- a/tests/unit/persistence/test_search.py
+++ b/tests/unit/persistence/test_search.py
@@ -157,7 +157,7 @@ async def _indexed_ids(fts) -> set[str]:
async def test_index_many_writes_the_same_rows_as_indexing_one_at_a_time(async_engine):
"""The batch path is the single-page path, so the two must not diverge."""
pages = [
- (f"p{i}", f"Title {i}", f"content about widgets number {i}", f"sum {i}", f"a/b{i}.py")
+ (f"p{i}", f"Title {i}", f"content about widgets number {i}", f"sum {i}", f"a/b{i}.py", "")
for i in range(20)
]
@@ -183,7 +183,7 @@ async def test_index_many_spans_more_ids_than_one_statement_can_bind(fts):
A wiki is thousands of pages, so the first real corpus would have been the
first test of this path.
"""
- pages = [(f"p{i}", f"Title {i}", "a page about chunking", "", f"{i}.py") for i in range(1200)]
+ pages = [(f"p{i}", f"Title {i}", "a page about chunking", "", f"{i}.py", "") for i in range(1200)]
await fts.index_many(pages)
assert len(await _indexed_ids(fts)) == 1200
@@ -197,8 +197,8 @@ async def test_a_page_id_repeated_in_one_batch_keeps_its_last_entry(fts):
"""Every id is deleted before any is inserted, so a duplicate would double."""
await fts.index_many(
[
- ("p1", "First", "a page about alpacas", "", "a.py"),
- ("p1", "Second", "a page about zebras", "", "a.py"),
+ ("p1", "First", "a page about alpacas", "", "a.py", ""),
+ ("p1", "Second", "a page about zebras", "", "a.py", ""),
]
)
@@ -246,6 +246,7 @@ async def test_path_only_match_outranks_content_only_match(fts):
"An unrelated module about parsing and graphs. " * 3,
"Filler.",
f"packages/core/other/mod{i}.py",
+ "",
)
for i in range(10)
]
@@ -334,6 +335,7 @@ async def test_index_pages_scans_the_index_once_per_id_chunk(async_engine, fts):
content=f"Module {i} documents widget number {i}.",
summary=f"summary {i}",
target_path=f"m{i}.py",
+ digest="",
)
for i in range(1100)
]
diff --git a/tests/unit/persistence/test_search_fts_columns.py b/tests/unit/persistence/test_search_fts_columns.py
index ce74407be..a1141e9d3 100644
--- a/tests/unit/persistence/test_search_fts_columns.py
+++ b/tests/unit/persistence/test_search_fts_columns.py
@@ -66,7 +66,7 @@ async def repo(async_session):
async def test_ensure_index_upgrades_old_schema_and_backfills(async_engine, async_session, repo):
- """An index written by an older repowise gains both columns, populated.
+ """An index written by an older repowise gains the new columns, populated.
The rebuild reads ``wiki_pages``, which is the only place the two new
fields exist — the old FTS rows never carried them. A silent no-op here
@@ -390,3 +390,23 @@ async def test_unreadable_page_metadata_indexes_without_a_vocabulary(
)
assert [r.page_id for r in await fts.search("entry point")] == ["file_page:src/main.py"]
+
+
+async def test_the_vocabulary_and_the_digest_are_both_searchable(
+ async_engine, async_session, repo
+):
+ """Two columns kept off the rendered body, one per kind of reader-invisible text."""
+ await _seed_page(async_session, repo.id, **_VOCABULARY_PAGE)
+ fts = FullTextSearch(async_engine)
+ await fts.ensure_index()
+ await fts.index(
+ "file_page:src/main.py",
+ "File: src/main.py",
+ _VOCABULARY_PAGE["content"],
+ summary="",
+ target_path="src/main.py",
+ digest="## Questions this page answers\n\n- Where does the zeppelin start?",
+ )
+
+ assert [r.page_id for r in await fts.search("xylophone")] == ["file_page:src/main.py"]
+ assert [r.page_id for r in await fts.search("zeppelin")] == ["file_page:src/main.py"]