Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

feat(generation): size-aware file-page volume policy, applied by default (#1076)

Raghav Chamadiyaf5687120102c
diff --git a/docs/reference/CLI_REFERENCE.md b/docs/reference/CLI_REFERENCE.md
index 6905337ec..9c468c8d7 100644
--- a/docs/reference/CLI_REFERENCE.md
+++ b/docs/reference/CLI_REFERENCE.md
@@ -56,7 +56,7 @@ In workspace mode, adds: repo scanning, per-repo indexing, cross-repo analysis (
2. **Index only**, the same layers plus a wiki rendered from structure; no LLM, no key, no cost. Answer **"Customize indexing?"** to set exclude patterns, commit limit, skip-tests/infra, submodules, and fast mode.
3. **Advanced**, full control. First choose **"Generate model-written wiki docs?"**; the prompts then split into an **Indexing** section (always) and a **Generation** section (provider, concurrency, embedder, wiki style, onboarding, decision harvesting, tiering, only when model-written docs are on).
-**Page volume (advanced mode, large repos only).** Above ~2,000 documentable files, advanced mode asks whether to bound the file-page bucket: the top 2,000 files by importance, or one page per eligible file. Below that the question is never asked and every eligible file gets a page. File pages are rendered from structure and cost no model tokens, so what the bound saves is wiki size, embedding calls and retrieval noise, and both answers are quoted in pages and megabytes before you pick. The answer is saved as `max_file_pages` in `config.yaml`, which `update --full` and `generate` honour; set it there directly (no terminal needed) for a scripted or hosted run.
+**Page volume.** File pages are rendered from structure and cost no model tokens, so what bounds them buys wiki size, embedding calls and less retrieval noise, never spend. Two numbers govern it, both derived from the size distribution of real indexed repos. Above **2,000** documentable files (p95), an interactive advanced-mode run offers the top 2,000 by importance instead of one page per file, quoted in pages and megabytes; nothing is capped automatically at that size. Above **4,500** file pages (about 40 MB of file layer, near p99), every run holds the bucket to 4,500 and says so in its output, because that is where the tail is the difference between a wiki that publishes and one that does not. `--max-file-pages N` sets a cap directly and `--max-file-pages 0` refuses one; either way the answer is saved as `max_file_pages` in `config.yaml`, which `update --full` and `generate` honour. A hosted or scripted caller sets the same field with no terminal involved.
All three reach the indexing knobs; the LLM-only knobs appear only when model-written docs are enabled. Passing any of the flags below (or `--yes`) skips the menu and runs non-interactively. If the first prompt reads EOF (common when an agent pipes stdin), init prints a notice and continues with defaults instead of aborting.
@@ -79,6 +79,7 @@ All three reach the indexing knobs; the LLM-only knobs appear only when model-wr
| `--include-submodules` | Include git submodule directories (excluded by default) |
| `--concurrency` | Max concurrent LLM calls (default: 10) |
| `--reasoning` | Reasoning mode for supported providers: `auto`, `off`/`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` (default: `auto`) |
+| `--max-file-pages` | Most file pages to emit, highest importance first. Omit to let the size policy decide (see Page volume above), `0` for one page per eligible file however many that is, or a positive number for a hard cap. Saved to `config.yaml`. |
| `--coverage-report` | Test-coverage report to ingest (LCOV / Cobertura / Clover). Repeatable. Auto-discovered when omitted. This is test coverage for code-health, not a documentation-breadth knob: every code file is documented either way. |
| `--onboarding` / `--no-onboarding` | Generate the curated Onboarding collection (up to 8 overview pages). Default: on; slots without enough signal are skipped. |
| `--harvest-decisions` / `--no-harvest-decisions` | Harvest architectural decisions during page generation (verified against source before storage). Default: on. |
diff --git a/docs/reference/CONFIG.md b/docs/reference/CONFIG.md
index cb899e03e..ca799d660 100644
--- a/docs/reference/CONFIG.md
+++ b/docs/reference/CONFIG.md
@@ -48,7 +48,7 @@ follow_renames: false # Track file renames in git history
wiki_style: comprehensive # comprehensive | caveman | reference | tutorial | custom
language: en # Output language for generated pages (en, zh, ru, hi, ...)
enable_onboarding: true # Show first-run onboarding prompts
-max_file_pages: 2000 # Cap the file-page bucket (omit for one page per file)
+max_file_pages: 2000 # Cap file pages (omit = size policy, 0 = one page per file)
exclude_patterns: # Gitignore-style patterns
- vendor/
- "*.generated.*"
@@ -80,7 +80,7 @@ You can edit this file directly. Changes take effect on the next `init`,
| `wiki_style` | `comprehensive` | `comprehensive`, `caveman`, `reference`, `tutorial`, `custom` |
| `language` | `en` | Output language for generated wiki pages: `en`, `ar`, `de`, `es`, `fr`, `hi`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `ru`, `tr`, `zh` |
| `enable_onboarding` | `true` | Show first-run onboarding prompts (CLI and web) |
-| `max_file_pages` | unset | Most file pages a run emits, highest importance first. Unset means one page per eligible file, which is the default on every repo. `repowise init` offers to set it in advanced mode above 2,000 documentable files; `update --full` and `generate` honour it after that |
+| `max_file_pages` | unset | Most file pages a run emits, highest importance first. Three states: **unset** lets the size policy decide (untouched below 4,500 documentable files, held to 4,500 above, which is about 1 repo in 100), **0** means one page per eligible file however many that is, and a **positive value** is a hard cap. `repowise init` offers a tighter cap in advanced mode above 2,000 documentable files, and `--max-file-pages N` sets it non-interactively. `update --full` and `generate` honour whatever is recorded. Capping file pages does not reduce model spend: file pages are rendered from structure |
| `distill` | see below | Output distillation config |
| `mcp` | see below | MCP tool surface config |
| `refactoring` | see below | Refactoring-intelligence config |
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/command.py b/packages/cli/src/repowise/cli/commands/init_cmd/command.py
index f61cf1ddd..b2af374c7 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd/command.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd/command.py
@@ -539,6 +539,19 @@ def _run_generation_phase(
"Slots with insufficient signal are skipped automatically."
),
)
+ "--max-file-pages",
+ "max_file_pages_opt",
+ type=int,
+ default=None,
+ metavar="N",
+ help=(
+ "Most file pages to emit, highest importance first. Omit to let the size "
+ "policy decide: untouched on a normal repo, held to a ceiling on a very "
+ "large one. Pass 0 for one page per eligible file however many that is. "
+ "Saved to config.yaml so later runs keep it."
+ ),
+)
@click.option(
"--coverage-report",
"coverage_report",
@@ -655,6 +668,7 @@ def init_command(
no_workspace: bool,
init_all: bool,
onboarding: bool,
+ max_file_pages_opt: int | None,
coverage_report: tuple[str, ...],
harvest_decisions: bool,
wiki_style: str | None,
@@ -846,9 +860,8 @@ def init_command(
language_choice: str | None = None
# File-page cap picked in the advanced-mode page-volume question; None until
- # chosen, and None also means "no cap", which is the default for every repo
- # small enough that the question is never asked. Resolved below:
- # advanced-mode answer > config.yaml > uncapped.
+ # chosen. Resolved below: --max-file-pages > advanced-mode answer >
+ # config.yaml > unset, and unset leaves the size policy in charge.
max_file_pages_choice: int | None = None
# The two orthogonal axes the interactive menu resolves: whether docs are
@@ -1003,10 +1016,11 @@ def init_command(
config = load_config(repo_path)
# Output language: CLI flag > advanced-mode choice > config.yaml > English.
language = language_opt or language_choice or config.get("language", "en")
- # File-page cap: advanced-mode answer > config.yaml > uncapped. There is no
- # flag; the question is the interactive surface and config.yaml is the one a
- # script or a hosted caller writes.
- max_file_pages = resolve_max_file_pages(max_file_pages_choice, config)
+ # File-page cap: flag > advanced-mode answer > config.yaml > unset (policy).
+ max_file_pages = resolve_max_file_pages(
+ max_file_pages_opt if max_file_pages_opt is not None else max_file_pages_choice,
+ config,
+ )
resolved_reasoning = resolve_reasoning(reasoning, config)
exclude_patterns: list[str] = list(config.get("exclude_patterns") or []) + list(exclude)
@@ -1375,12 +1389,15 @@ def init_command(
if not onboarding:
save_config_partial(repo_path, enable_onboarding=False)
- # Persist the file-page cap so `repowise update --full` and `repowise
- # generate` keep the wiki the size the user asked for instead of growing the
- # file layer back on the next run. Recorded whether or not a model wrote the
- # pages, because the cap applies to the template wiki just the same. No cap
- # stays unrecorded: it is the default.
+ # Persist the file-page decision so `repowise update --full` and `repowise
+ # generate` keep the wiki the size that was asked for instead of re-deciding.
+ # Recorded whether or not a model wrote the pages, since the cap applies to the
+ # template wiki just the same. A refusal (0) is recorded like any other answer:
+ # leaving it unrecorded would hand the next run back to the size policy the
+ # user just declined. Only an unset value stays unwritten.
if max_file_pages is not None and max_file_pages != config.get("max_file_pages"):
+ # save_config_partial drops None values, and 0 is not None, so an explicit
+ # refusal reaches the file.
save_config_partial(repo_path, max_file_pages=max_file_pages)
# Persist the wiki style so `repowise update` / `restyle` honor it without
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/generation.py b/packages/cli/src/repowise/cli/commands/init_cmd/generation.py
index 52ca736f8..dd974e423 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd/generation.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd/generation.py
@@ -40,6 +40,7 @@ from repowise.cli.ui import BRAND_STYLE, OWL_SPINNER, MaybeCountColumn, RichProg
__all__ = [
"COST_GATE_USD",
"CostGateDeclined",
+ "announce_file_page_cap",
"concept_page_count",
"confirm_cost_gate",
"cost_gate_blocks",
@@ -103,6 +104,33 @@ def concept_page_count(plans: list[Any]) -> int:
return next((p.count for p in plans if p.page_type == "module_page"), 0)
+def announce_file_page_cap(parsed_files: list[Any], gen_config: Any) -> None:
+ """Say so when the file bucket is being bounded, and how to undo it.
+
+ A cap the user chose needs no explanation, but the one the size policy applies
+ on its own would otherwise show up only as a page count nobody asked for. Both
+ are printed, since a run that emits fewer pages than there are files should say
+ which files it kept and how to ask for all of them.
+ """
+ from repowise.core.generation.selection import auto_file_page_cap, count_documentable_files
+
+ requested = getattr(gen_config, "max_file_pages", None)
+ if requested == 0:
+ return # explicitly uncapped: nothing is being held back
+ documentable = count_documentable_files(parsed_files)
+ cap = requested if requested else auto_file_page_cap(documentable)
+ if not cap or cap >= documentable:
+ return
+
+ reason = "this repo's size" if requested is None else "your setting"
+ console.print(
+ f" [dim]File pages: keeping the top [bold]{cap:,}[/bold] of about "
+ f"{documentable:,} by importance ({reason}). File pages cost no model "
+ "tokens, so this bounds wiki size and search noise, not spend. Pass "
+ "[bold]--max-file-pages 0[/bold] for one page per file.[/dim]"
+ )
+
+
def _enrich_knowledge_graph(
*,
result: Any,
@@ -180,6 +208,9 @@ def run_repo_generation(
"""
from ._generation_persist import run_generation_with_persistence
+ if verbose:
+ announce_file_page_cap(result.parsed_files, gen_config)
+
embedder_impl: Any = build_embedder(embedder_name_resolved)
vector_store: Any = build_vector_store(repo_path, embedder_impl)
result.vector_store = vector_store
diff --git a/packages/cli/src/repowise/cli/helpers.py b/packages/cli/src/repowise/cli/helpers.py
index 2efa48972..fc8cdf4e6 100644
--- a/packages/cli/src/repowise/cli/helpers.py
+++ b/packages/cli/src/repowise/cli/helpers.py
@@ -479,22 +479,25 @@ def resolve_max_file_pages(
chosen: int | None = None,
config: dict[str, Any] | None = None,
) -> int | None:
- """Resolve the file-page cap: explicit choice, then config.yaml, then none.
+ """Resolve the file-page cap: explicit choice, then config.yaml, then unset.
- ``None`` means unrationed, which is what every repo gets unless its owner
- asked otherwise (see ``GenerationConfig.max_file_pages``). A non-positive or
- unparseable ``max_file_pages`` in config.yaml is read as no cap rather than
- as zero pages: a typo must not silently delete the file layer.
+ Three states reach ``GenerationConfig.max_file_pages`` (see its comment and
+ ``selection/selector.py``): ``None`` leaves the volume policy in charge, ``0``
+ is an explicit refusal to cap, and a positive value is a cap.
+
+ A negative or unparseable ``max_file_pages`` in config.yaml resolves to
+ ``None`` rather than to zero pages, so a typo hands the decision back to the
+ policy instead of silently deleting the file layer.
"""
- if chosen is not None:
- return max(1, chosen)
- raw = (config or {}).get("max_file_pages")
+ raw = chosen if chosen is not None else (config or {}).get("max_file_pages")
if raw is None:
return None
try:
value = int(raw)
except (TypeError, ValueError):
return None
+ if value == 0:
+ return 0 # "all of them", and it survives the policy
return value if value > 0 else None
diff --git a/packages/cli/src/repowise/cli/ui/__init__.py b/packages/cli/src/repowise/cli/ui/__init__.py
index eb45fe2ec..78f2ed7d2 100644
--- a/packages/cli/src/repowise/cli/ui/__init__.py
+++ b/packages/cli/src/repowise/cli/ui/__init__.py
@@ -27,7 +27,6 @@ from repowise.cli.ui.mascot import (
mini,
)
from repowise.cli.ui.mode_selection import (
- FILE_PAGE_VOLUME_THRESHOLD,
LARGE_REPO_FILE_THRESHOLD,
interactive_advanced_config,
interactive_customize_offer,
@@ -66,7 +65,6 @@ __all__ = [
"BRAND_STYLE",
"DIM",
"ERR",
- "FILE_PAGE_VOLUME_THRESHOLD",
"LARGE_REPO_FILE_THRESHOLD",
"OK",
"OWL_SPINNER",
diff --git a/packages/cli/src/repowise/cli/ui/mode_selection.py b/packages/cli/src/repowise/cli/ui/mode_selection.py
index 851d5577e..a581dfc1d 100644
--- a/packages/cli/src/repowise/cli/ui/mode_selection.py
+++ b/packages/cli/src/repowise/cli/ui/mode_selection.py
@@ -16,6 +16,10 @@ from rich.text import Text
from repowise.cli.ui.brand import BRAND, BRAND_STYLE, DIM
from repowise.cli.ui.repo_scanner import RepoScanInfo, estimated_documentable_files
from repowise.core.generation.languages import SUPPORTED_LANGUAGES
+from repowise.core.generation.selection import (
+ FILE_PAGE_AUTO_CEILING,
+ recommended_file_page_cap,
+)
from repowise.core.generation.styles import DEFAULT_STYLE, list_styles
from repowise.core.reasoning import REASONING_MODES
@@ -23,20 +27,6 @@ from repowise.core.reasoning import REASONING_MODES
# fast-mode first index (graph + essential git, no LLM docs) is worth offering.
LARGE_REPO_FILE_THRESHOLD = 5000
-# Above this many documentable files, advanced mode asks whether to bound the
-# file-page bucket; at or below it the question is never asked and every eligible
-# file gets a page.
-#
-# The number comes from the size distribution of the 2,309 repositories indexed
-# on repowise.dev (latest ready snapshot each): median 64 files, p75 235, p90
-# 907, p95 2,047, p98 3,290, largest 14,943. 2,000 sits at roughly the 95th
-# percentile, so 19 in 20 repos never see the question, and the ones that do are
-# the ones where the file tail is measured in tens of megabytes.
-#
-# It doubles as the recommended cap, which keeps the choice continuous: a repo
-# just over the line loses almost nothing by taking the recommendation.
-FILE_PAGE_VOLUME_THRESHOLD = 2000
-
# Bytes a file page occupies with its metadata, measured over 1,961 file pages in
# a local index (mean 8.8 KB). Used to quote wiki size while asking; an order of
# magnitude is the point, not a byte count.
@@ -232,26 +222,34 @@ def _format_mb(pages: int) -> str:
def prompt_file_page_volume(
console: Console,
scan: RepoScanInfo | None,
- *,
- threshold: int = FILE_PAGE_VOLUME_THRESHOLD,
) -> int | None:
"""Ask a large repo whether to bound the file-page bucket.
- Returns the cap to write to ``GenerationConfig.max_file_pages``: an int to
- take the top slice by importance, or ``None`` for every eligible file (the
- unrationed default, and the answer on any repo at or below *threshold*).
+ Returns the value for ``GenerationConfig.max_file_pages``: a positive cap to
+ take the top slice by importance, ``0`` to refuse any cap (one page per
+ eligible file, however many that is), or ``None`` when the repo is small
+ enough that there is nothing to ask and the policy leaves it alone.
One page per source file is what makes small and mid-size repos good, so the
- question is only asked above *threshold*, and only in advanced mode. It is
- asked before ingestion, so the page count quoted is the pre-scan estimate.
+ question is only asked on repos where a cap is worth considering, and only in
+ advanced mode. It is asked before ingestion, so the page count is the pre-scan
+ estimate.
+
+ The recommended answer is whatever :func:`recommended_file_page_cap` says for
+ this size, so the question can never recommend a number the volume policy
+ would then override. Above the automatic ceiling, refusing writes an explicit
+ ``0`` rather than an unset value, because "every eligible file" has to mean
+ that even though the policy would otherwise step in.
Never blocks: a terminal that cannot answer takes the recommendation and the
run carries on, same as the other optional questions in an init flow.
"""
estimate = estimated_documentable_files(scan)
- if estimate <= threshold:
+ recommended = recommended_file_page_cap(estimate)
+ if recommended is None:
return None
+ would_be_capped_anyway = estimate > FILE_PAGE_AUTO_CEILING
console.print()
console.print(f" [{BRAND}]Page volume[/]")
console.print(
@@ -261,10 +259,15 @@ def prompt_file_page_volume(
" the tail costs is wiki size, one embedding call each, and search results\n"
" that restate what a subsystem page already says.[/dim]"
)
+ if would_be_capped_anyway:
+ console.print(
+ f" [dim]A repo this size is held to {FILE_PAGE_AUTO_CEILING:,} file pages "
+ "unless you say otherwise here.[/dim]"
+ )
console.print()
console.print(
- f" [{BRAND}][1][/] Top [bold]{threshold:,}[/bold] by importance [dim](recommended) — "
- f"~{threshold:,} pages, {_format_mb(threshold)}[/dim]"
+ f" [{BRAND}][1][/] Top [bold]{recommended:,}[/bold] by importance [dim](recommended) — "
+ f"~{recommended:,} pages, {_format_mb(recommended)}[/dim]"
)
console.print(
f" [{BRAND}][2][/] Every eligible file [dim]— ~{estimate:,} pages, "
@@ -279,8 +282,13 @@ def prompt_file_page_volume(
# pty wrappers, `docker run -t` without -i). Take the recommendation and
# keep going: an agent or CI job must never hang here.
console.print(" [dim]No answer available — taking the recommendation.[/dim]")
- return threshold
- return threshold if choice == "1" else None
+ return recommended
+ if choice == "1":
+ return recommended
+ # Refusing has to survive the policy on a repo the policy would cap, and
+ # recording 0 rather than nothing is what says the answer was "all of them"
+ # instead of "never asked".
+ return 0 if would_be_capped_anyway else None
def _prompt_run_mode(
@@ -490,8 +498,11 @@ def _build_summary_table(
# Bullet-list when many patterns — comma-joined wraps unreadably.
summary.add_row("Exclude", "\n".join(f"• {p}" for p in patterns))
- if result.get("max_file_pages"):
- summary.add_row("File pages", f"top {result['max_file_pages']:,} by importance")
+ cap = result.get("max_file_pages")
+ if cap:
+ summary.add_row("File pages", f"top {cap:,} by importance")
+ elif cap == 0:
+ summary.add_row("File pages", "one per eligible file (uncapped)")
if not generate_docs and result.get("embedder"):
summary.add_row("Embedder", result["embedder"])
diff --git a/packages/core/src/repowise/core/generation/models.py b/packages/core/src/repowise/core/generation/models.py
index 8bdc5f383..720601894 100644
--- a/packages/core/src/repowise/core/generation/models.py
+++ b/packages/core/src/repowise/core/generation/models.py
@@ -114,19 +114,22 @@ class GenerationConfig:
# public symbols still gets a spotlight.
top_symbol_percentile: float = 0.10
# The most ``file_page`` pages a run will emit, highest importance score
- # first. ``None`` (the default) means unrationed: every file that clears the
- # importance floor gets a page, which is what makes small and mid-size repos
- # good and is the behaviour every repo had before this field existed.
+ # first. Three states, resolved in selection/selector.py:
#
- # It exists for the other end of the distribution. A 10.8k-file monorepo
+ # None (default) -> the volume policy decides. Untouched below
+ # FILE_PAGE_AUTO_CEILING, held at it above, which is
+ # about one repo in a hundred.
+ # 0 -> explicitly unlimited: one page per eligible file,
+ # however many that is.
+ # N > 0 -> cap at N.
+ #
+ # It exists for the top of the size distribution. A 10.8k-file monorepo
# produced 8,756 file pages inside a 14,027-page wiki, and a file page
# averages ~8.8 KB with its metadata, so that tail alone is ~77 MB of wiki
- # whose entries mostly restate what their concept page already says. Bounding
- # it is a choice the owner of the repo makes: `repowise init` asks in advanced
- # mode above a threshold, and any programmatic caller (the hosted pipeline
- # included) sets it here. Capping file pages does not reduce model spend --
- # file pages are rendered from structure and cost no tokens -- it reduces
- # pages, bytes, embedding calls and retrieval dilution.
+ # whose entries mostly restate what their concept page already says. Capping
+ # file pages does not reduce model spend -- file pages are rendered from
+ # structure and cost no tokens -- it reduces pages, bytes, embedding calls
+ # and retrieval dilution.
max_file_pages: int | None = None
file_page_min_symbols: int = 1
skip_trivial_files: bool = True
diff --git a/packages/core/src/repowise/core/generation/selection/__init__.py b/packages/core/src/repowise/core/generation/selection/__init__.py
index 7aedb57fe..1aabf5f8b 100644
--- a/packages/core/src/repowise/core/generation/selection/__init__.py
+++ b/packages/core/src/repowise/core/generation/selection/__init__.py
@@ -21,17 +21,27 @@ from .scoring import (
score_symbol,
)
from .selector import (
+ FILE_PAGE_ASK_THRESHOLD,
+ FILE_PAGE_AUTO_CEILING,
ModuleGroup,
Selection,
SelectionInputs,
+ auto_file_page_cap,
+ count_documentable_files,
+ recommended_file_page_cap,
select_pages,
summarize_selection,
)
__all__ = [
+ "FILE_PAGE_ASK_THRESHOLD",
+ "FILE_PAGE_AUTO_CEILING",
"ModuleGroup",
"Selection",
"SelectionInputs",
+ "auto_file_page_cap",
+ "count_documentable_files",
+ "recommended_file_page_cap",
"score_api_contract",
"score_file",
"score_infra",
diff --git a/packages/core/src/repowise/core/generation/selection/selector.py b/packages/core/src/repowise/core/generation/selection/selector.py
index 0cb8007de..a52d8d4f5 100644
--- a/packages/core/src/repowise/core/generation/selection/selector.py
+++ b/packages/core/src/repowise/core/generation/selection/selector.py
@@ -195,6 +195,75 @@ def _is_code_file(parsed: Any) -> bool:
return not fi.is_api_contract and not _is_infra_file(parsed) and fi.language in _CODE_LANGUAGES
+# ---------------------------------------------------------------------------
+# File-page volume policy
+#
+# Two numbers, each answering a different question, both derived rather than
+# picked. The basis is the size distribution of the 2,309 repositories indexed on
+# repowise.dev (latest ready snapshot each): median 64 files, p75 235, p90 907,
+# p95 2,047, p98 3,290, p99 4,719, largest 14,943.
+# ---------------------------------------------------------------------------
+
+# Above this many documentable files, an interactive advanced-mode run offers a
+# leaner wiki. It is p95, so 19 repos in 20 are never asked. Nothing is capped
+# automatically at this size: a 2,500-file repo would lose 500 pages and gain
+# nothing anyone measured, which is not a trade to make on someone's behalf.
+FILE_PAGE_ASK_THRESHOLD = 2000
+
+# The ceiling every run holds the file bucket to, asked or not. A file page
+# averages 8.8 KB with its metadata (measured over 1,961 pages in a real index),
+# so this is about 40 MB of file layer: inside the hosted 48 MiB artifact ceiling
+# before compression, with room to spare after it. It sits near p99, so it fires
+# on roughly one repo in a hundred, and those are the repos where the tail is the
+# difference between a wiki that publishes and one that does not.
+#
+# A ceiling rather than a fraction, because bounding bytes is the whole point: a
+# percentage of the repo grows with the repo and bounds nothing. It is also the
+# least aggressive number that solves that problem, which is what makes it
+# defensible to apply without asking. Anything tighter is taste, and taste stays
+# a question.
+FILE_PAGE_AUTO_CEILING = 4500
+
+
+def auto_file_page_cap(documentable_files: int) -> int | None:
+ """The cap a run applies on its own, or ``None`` to leave the bucket alone.
+
+ This is what an unset ``max_file_pages`` resolves to. It only ever engages
+ above :data:`FILE_PAGE_AUTO_CEILING`, and it caps to exactly that, so a repo
+ just over the line loses almost nothing and a 15,000-file monorepo keeps the
+ 4,500 file pages its readers are most likely to want.
+ """
+ return FILE_PAGE_AUTO_CEILING if documentable_files > FILE_PAGE_AUTO_CEILING else None
+
+
+def recommended_file_page_cap(documentable_files: int) -> int | None:
+ """The cap to offer as the recommended answer, or ``None`` if none applies.
+
+ Above the automatic ceiling this is that ceiling, so the question can never
+ recommend something the policy would then override. Between the ask
+ threshold and the ceiling it is the threshold itself, which keeps the choice
+ continuous: a repo just over the line loses almost nothing by taking it.
+ """
+ if documentable_files > FILE_PAGE_AUTO_CEILING:
+ return FILE_PAGE_AUTO_CEILING
+ if documentable_files > FILE_PAGE_ASK_THRESHOLD:
+ return FILE_PAGE_ASK_THRESHOLD
+ return None
+
+
+def count_documentable_files(parsed_files: list[Any]) -> int:
+ """How many files would get a ``file_page``, from parsed output.
+
+ The exact predicate file-page selection uses, minus the score floor (which
+ needs graph metrics), so it reads slightly high on files the graph knows
+ nothing about. Exists so a caller can report what the volume policy is about
+ to do before generation starts, in the same terms the policy uses.
+ """
+ return sum(
+ 1 for p in parsed_files if _is_code_file(p) and _passes_importance_floor(p.file_info.path)
+ )
+
+
def _passes_importance_floor(path: str) -> bool:
"""Whether *path* is worth a file page at all.
@@ -249,14 +318,23 @@ def _build_file_candidates(
if s > 0.0:
scored.append((s, path))
scored.sort(key=lambda x: (-x[0], x[1]))
- # Unrationed by default: every file above the floor gets a page. A repo whose
- # owner asked for a bound gets the strongest ``max_file_pages`` of them, by
- # the score already computed above rather than by a second ranking. The sort
- # is total (score, then path), so the cut is deterministic.
+ # Three states, because "how many file pages" has three real answers.
+ #
+ # unset (None) -> the volume policy decides: untouched below the automatic
+ # ceiling, held at it above (see auto_file_page_cap)
+ # 0 -> explicitly unlimited, one page per eligible file, however
+ # many that is. This is a refusal, and it is honoured
+ # N > 0 -> cap at N
+ #
+ # Whatever the cap, it slices the ranking already computed above rather than a
+ # second one, and the sort is total (score, then path), so the cut is
+ # deterministic.
cap = getattr(inputs.config, "max_file_pages", None)
- if cap is not None and cap >= 0:
- return scored[:cap]
- return scored
+ if cap is None:
+ cap = auto_file_page_cap(len(scored))
+ if not cap: # None (nothing to do) or 0 (explicitly unlimited)
+ return scored
+ return scored[: max(1, cap)]
def _build_symbol_candidates(
diff --git a/tests/unit/cli/test_file_page_volume_prompt.py b/tests/unit/cli/test_file_page_volume_prompt.py
index 67cf2f403..3a27a7800 100644
--- a/tests/unit/cli/test_file_page_volume_prompt.py
+++ b/tests/unit/cli/test_file_page_volume_prompt.py
@@ -11,6 +11,7 @@ from __future__ import annotations
import io
import sys
+from unittest import mock
import pytest
from click.testing import CliRunner
@@ -20,11 +21,12 @@ from rich.prompt import Prompt
from repowise.cli.helpers import resolve_max_file_pages
from repowise.cli.main import cli
from repowise.cli.ui import mode_selection
-from repowise.cli.ui.mode_selection import (
- FILE_PAGE_VOLUME_THRESHOLD,
- prompt_file_page_volume,
-)
+from repowise.cli.ui.mode_selection import prompt_file_page_volume
from repowise.cli.ui.repo_scanner import RepoScanInfo, estimated_documentable_files
+from repowise.core.generation.selection import (
+ FILE_PAGE_ASK_THRESHOLD,
+ FILE_PAGE_AUTO_CEILING,
+)
def _scan(source_files: int, *, tests: int = 0) -> RepoScanInfo:
@@ -52,7 +54,7 @@ class TestWhenItIsAsked:
def test_repo_at_the_threshold_is_never_asked(self, monkeypatch):
monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")
- scan = _scan(FILE_PAGE_VOLUME_THRESHOLD)
+ scan = _scan(FILE_PAGE_ASK_THRESHOLD)
assert prompt_file_page_volume(_console(), scan) is None
def test_no_scan_is_never_asked(self, monkeypatch):
@@ -71,13 +73,32 @@ class TestWhenItIsAsked:
class TestTheAnswer:
- def test_recommendation_caps_at_the_threshold(self, monkeypatch):
+ """The recommendation tracks the repo's size, because 2,500 files and 15,000
+ files are not the same problem. Between the ask threshold and the automatic
+ ceiling the recommendation is the threshold (a leaner wiki, entirely the
+ user's call). Above the ceiling it is the ceiling, so the question can never
+ recommend a number the policy would then override."""
+
+ def test_midsize_repo_is_offered_the_ask_threshold(self, monkeypatch):
monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")
- assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD
+ assert prompt_file_page_volume(_console(), _scan(3_000)) == FILE_PAGE_ASK_THRESHOLD
+
+ def test_huge_repo_is_offered_the_automatic_ceiling(self, monkeypatch):
+ """Not the tighter ask threshold: capping a 15k-file repo to 2,000 is a
+ taste call, and taste is not what gets applied on someone's behalf."""
+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")
+ assert prompt_file_page_volume(_console(), _scan(15_000)) == FILE_PAGE_AUTO_CEILING
+
+ def test_everything_on_a_midsize_repo_leaves_it_unset(self, monkeypatch):
+ """Nothing would cap this repo anyway, so there is nothing to refuse."""
+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "2")
+ assert prompt_file_page_volume(_console(), _scan(3_000)) is None
- def test_everything_means_no_cap(self, monkeypatch):
+ def test_everything_above_the_ceiling_is_an_explicit_refusal(self, monkeypatch):
+ """0, not None: the policy would otherwise cap this repo, and the user
+ just said they want every page."""
monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "2")
- assert prompt_file_page_volume(_console(), _scan(9_000)) is None
+ assert prompt_file_page_volume(_console(), _scan(15_000)) == 0
def test_the_recommendation_is_the_default(self, monkeypatch):
"""Enter-through takes the bounded wiki."""
@@ -98,10 +119,18 @@ class TestTheAnswer:
prompt_file_page_volume(Console(file=buffer, width=100), _scan(9_000))
out = buffer.getvalue()
assert "9,000" in out # what every eligible file would cost
- assert f"{FILE_PAGE_VOLUME_THRESHOLD:,}" in out # what the recommendation costs
+ assert f"{FILE_PAGE_AUTO_CEILING:,}" in out # what the recommendation costs
assert "MB" in out
assert "no model tokens" in out # file pages are rendered, not written
+ def test_a_capped_repo_is_told_the_ceiling_applies_anyway(self, monkeypatch):
+ """Above the ceiling the question is a chance to refuse, not the only
+ thing standing between the repo and 15,000 pages. Say so."""
+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")
+ buffer = io.StringIO()
+ prompt_file_page_volume(Console(file=buffer, width=100), _scan(15_000))
+ assert "unless you say otherwise" in buffer.getvalue()
+
class TestItNeverBlocks:
def test_eof_takes_the_recommendation(self, monkeypatch):
@@ -111,13 +140,13 @@ class TestItNeverBlocks:
raise EOFError
monkeypatch.setattr(Prompt, "ask", _eof)
- assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD
+ assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_AUTO_CEILING
def test_closed_stdin_takes_the_recommendation(self, monkeypatch):
"""The real prompt against a stdin with nothing on it. Must not raise."""
monkeypatch.setattr(sys, "stdin", io.StringIO(""))
result = prompt_file_page_volume(_console(), _scan(9_000))
- assert result == FILE_PAGE_VOLUME_THRESHOLD
+ assert result == FILE_PAGE_AUTO_CEILING
def test_abort_takes_the_recommendation(self, monkeypatch):
"""click.Abort is what a pty wrapper produces on an unanswerable read."""
@@ -127,7 +156,7 @@ class TestItNeverBlocks:
raise click.Abort
monkeypatch.setattr(Prompt, "ask", _abort)
- assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD
+ assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_AUTO_CEILING
class TestYesNeverAsks:
@@ -174,10 +203,70 @@ class TestResolution:
def test_an_answer_beats_config(self):
assert resolve_max_file_pages(2000, {"max_file_pages": 500}) == 2000
- def test_zero_or_negative_reads_as_no_cap(self):
- """A typo must not silently delete the file layer."""
- assert resolve_max_file_pages(None, {"max_file_pages": 0}) is None
+ def test_zero_is_an_explicit_refusal_and_survives(self):
+ """0 has to reach the selector, or the policy would re-cap a repo whose
+ owner already said they want every page."""
+ assert resolve_max_file_pages(None, {"max_file_pages": 0}) == 0
+ assert resolve_max_file_pages(0, {}) == 0
+
+ def test_negative_reads_as_unset(self):
+ """A typo hands the decision back to the policy rather than deleting the
+ file layer."""
assert resolve_max_file_pages(None, {"max_file_pages": -5}) is None
def test_garbage_reads_as_no_cap(self):
assert resolve_max_file_pages(None, {"max_file_pages": "lots"}) is None
+
+
+class TestTheRunSaysSo:
+ """A cap the run applied on its own must never be silent: the only other
+ evidence would be a page count nobody can explain."""
+
+ @staticmethod
+ def _parsed(n: int):
+ from types import SimpleNamespace
+
+ return [
+ SimpleNamespace(
+ file_info=SimpleNamespace(
+ path=f"pkg/mod_{i}.py",
+ language="python",
+ is_test=False,
+ is_api_contract=False,
+ ),
+ symbols=[],
+ )
+ for i in range(n)
+ ]
+
+ def _announce(self, files: int, cap):
+ from types import SimpleNamespace
+
+ from repowise.cli.commands.init_cmd.generation import announce_file_page_cap
+
+ buffer = io.StringIO()
+ with mock.patch(
+ "repowise.cli.commands.init_cmd.generation.console", Console(file=buffer, width=100)
+ ):
+ announce_file_page_cap(self._parsed(files), SimpleNamespace(max_file_pages=cap))
+ return buffer.getvalue()
+
+ def test_policy_cap_is_announced_with_the_undo(self):
+ out = self._announce(FILE_PAGE_AUTO_CEILING + 500, None)
+ assert f"{FILE_PAGE_AUTO_CEILING:,}" in out
+ assert "this repo's size" in out
+ assert "--max-file-pages 0" in out
+ assert "not spend" in out # file pages are free, say so where it matters
+
+ def test_a_chosen_cap_says_it_was_chosen(self):
+ out = self._announce(1_000, 200)
+ assert "your setting" in out
+
+ def test_nothing_is_said_when_nothing_is_capped(self):
+ assert self._announce(1_000, None) == ""
+
+ def test_nothing_is_said_when_the_cap_exceeds_the_repo(self):
+ assert self._announce(50, 5_000) == ""
+
+ def test_nothing_is_said_on_an_explicit_refusal(self):
+ assert self._announce(FILE_PAGE_AUTO_CEILING + 500, 0) == ""
diff --git a/tests/unit/generation/test_selection_contract.py b/tests/unit/generation/test_selection_contract.py
index 89f10fc41..1fe1d99e2 100644
--- a/tests/unit/generation/test_selection_contract.py
+++ b/tests/unit/generation/test_selection_contract.py
@@ -13,7 +13,12 @@ from dataclasses import dataclass, field
from repowise.core.generation.models import GenerationConfig
from repowise.core.generation.selection import (
+ FILE_PAGE_ASK_THRESHOLD,
+ FILE_PAGE_AUTO_CEILING,
SelectionInputs,
+ auto_file_page_cap,
+ count_documentable_files,
+ recommended_file_page_cap,
select_pages,
)
@@ -235,8 +240,8 @@ def test_empty_repo_emits_no_content_pages():
# ---------------------------------------------------------------------------
-def test_file_bucket_is_unrationed_by_default():
- """No cap unless the repo's owner asked for one."""
+def test_file_bucket_is_unset_by_default():
+ """Unset means the size policy decides, not that a cap was chosen."""
assert GenerationConfig().max_file_pages is None
@@ -281,3 +286,82 @@ def test_cap_is_deterministic():
first = select_pages(_inputs(parsed, pagerank, betweenness, community, cfg))
second = select_pages(_inputs(list(reversed(parsed)), pagerank, betweenness, community, cfg))
assert first.file_page_paths == second.file_page_paths
+
+
+# ---------------------------------------------------------------------------
+# The size policy: what an unset max_file_pages resolves to
+# ---------------------------------------------------------------------------
+
+
+def test_policy_leaves_normal_repos_alone():
+ """A 2,500-file repo would lose 500 pages and gain nothing measured."""
+ assert auto_file_page_cap(0) is None
+ assert auto_file_page_cap(FILE_PAGE_ASK_THRESHOLD + 500) is None
+ assert auto_file_page_cap(FILE_PAGE_AUTO_CEILING) is None
+
+
+def test_policy_holds_huge_repos_at_the_ceiling():
+ assert auto_file_page_cap(FILE_PAGE_AUTO_CEILING + 1) == FILE_PAGE_AUTO_CEILING
+ assert auto_file_page_cap(15_000) == FILE_PAGE_AUTO_CEILING
+
+
+def test_recommendation_never_contradicts_the_policy():
+ """Whatever the question recommends, the policy must not override it."""
+ for n in (0, 500, 2_000, 2_001, 4_000, 4_500, 4_501, 9_000, 50_000):
+ recommended = recommended_file_page_cap(n)
+ policy = auto_file_page_cap(n)
+ if policy is not None:
+ assert recommended == policy, n
+ elif recommended is not None:
+ assert recommended <= n, n
+
+
+def test_unset_config_applies_the_policy_to_a_huge_repo():
+ parsed, pagerank, betweenness, community = _build_synthetic_repo(FILE_PAGE_AUTO_CEILING + 200)
+ sel = select_pages(_inputs(parsed, pagerank, betweenness, community, GenerationConfig()))
+ assert len(sel.file_page_paths) == FILE_PAGE_AUTO_CEILING
+
+
+def test_unset_config_leaves_a_midsize_repo_whole():
+ """Just under the ceiling: every eligible file still gets a page."""
+ parsed, pagerank, betweenness, community = _build_synthetic_repo(FILE_PAGE_AUTO_CEILING - 100)
+ sel = select_pages(_inputs(parsed, pagerank, betweenness, community, GenerationConfig()))
+ assert len(sel.file_page_paths) == FILE_PAGE_AUTO_CEILING - 100
+
+
+def test_zero_refuses_the_policy():
+ """An explicit 0 means every eligible file, on a repo the policy would cap."""
+ parsed, pagerank, betweenness, community = _build_synthetic_repo(FILE_PAGE_AUTO_CEILING + 200)
+ sel = select_pages(
+ _inputs(parsed, pagerank, betweenness, community, GenerationConfig(max_file_pages=0))
+ )
+ assert len(sel.file_page_paths) == FILE_PAGE_AUTO_CEILING + 200
+
+
+def test_explicit_cap_beats_the_policy_in_both_directions():
+ parsed, pagerank, betweenness, community = _build_synthetic_repo(FILE_PAGE_AUTO_CEILING + 200)
+ tighter = select_pages(
+ _inputs(parsed, pagerank, betweenness, community, GenerationConfig(max_file_pages=100))
+ )
+ assert len(tighter.file_page_paths) == 100
+ looser = select_pages(
+ _inputs(
+ parsed,
+ pagerank,
+ betweenness,
+ community,
+ GenerationConfig(max_file_pages=FILE_PAGE_AUTO_CEILING + 100),
+ )
+ )
+ assert len(looser.file_page_paths) == FILE_PAGE_AUTO_CEILING + 100
+
+
+def test_count_documentable_files_matches_the_floor():
+ """The count a caller reports with is the count selection acts on."""
+ parsed, pagerank, betweenness, community = _build_synthetic_repo(20)
+ for extra in ("tests/test_thing.py", "pkg0/__init__.py"):
+ parsed.append(FakeParsedFile(file_info=FakeFileInfo(path=extra), symbols=[]))
+ pagerank[extra] = 1.0
+ betweenness[extra] = 0.0
+ community[extra] = 0
+ assert count_documentable_files(parsed) == 20