Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.
Python aicode-complexitycode-healthcode-intelligencecode-qualitydead-codedeveloper-toolsdocumentationgit-analyticsmcpopen-sourcepythonrefactoringstatic-analysistechnical-debt
feat(generation): bound the file-page bucket on very large repos (#1074)
Raghav Chamadiya2877c5722316
diff --git a/docs/reference/CLI_REFERENCE.md b/docs/reference/CLI_REFERENCE.mdindex de8daa7fe..2aaf09d02 100644--- a/docs/reference/CLI_REFERENCE.md+++ b/docs/reference/CLI_REFERENCE.md@@ -56,6 +56,8 @@ In workspace mode, adds: repo scanning, per-repo indexing, cross-repo analysis (2. **Index only**, the same layers plus a wiki rendered from structure; no LLM, no key, no cost. Answer **"Customize indexing?"** to set exclude patterns, commit limit, skip-tests/infra, submodules, and fast mode.3. **Advanced**, full control. First choose **"Generate model-written wiki docs?"**; the prompts then split into an **Indexing** section (always) and a **Generation** section (provider, concurrency, embedder, wiki style, onboarding, decision harvesting, tiering, only when model-written docs are on).+**Page volume (advanced mode, large repos only).** Above ~2,000 documentable files, advanced mode asks whether to bound the file-page bucket: the top 2,000 files by importance, or one page per eligible file. Below that the question is never asked and every eligible file gets a page. File pages are rendered from structure and cost no model tokens, so what the bound saves is wiki size, embedding calls and retrieval noise, and both answers are quoted in pages and megabytes before you pick. The answer is saved as `max_file_pages` in `config.yaml`, which `update --full` and `generate` honour; set it there directly (no terminal needed) for a scripted or hosted run.+All three reach the indexing knobs; the LLM-only knobs appear only when model-written docs are enabled. Passing any of the flags below (or `--yes`) skips the menu and runs non-interactively. If the first prompt reads EOF (common when an agent pipes stdin), init prints a notice and continues with defaults instead of aborting.**Cost gate.** Before any tokens are spent, init shows the estimate and asks for confirmation. It never prompts when stdin is not a TTY: it auto-declines, keeps the index it already built, and renders the structural wiki instead. The default answer is Yes up to $25 and flips to No above that.diff --git a/docs/reference/CONFIG.md b/docs/reference/CONFIG.mdindex 010b0fe35..cb899e03e 100644--- a/docs/reference/CONFIG.md+++ b/docs/reference/CONFIG.md@@ -48,6 +48,7 @@ follow_renames: false # Track file renames in git historywiki_style: comprehensive # comprehensive | caveman | reference | tutorial | customlanguage: en # Output language for generated pages (en, zh, ru, hi, ...)enable_onboarding: true # Show first-run onboarding prompts+max_file_pages: 2000 # Cap the file-page bucket (omit for one page per file)exclude_patterns: # Gitignore-style patterns- vendor/- "*.generated.*"@@ -79,6 +80,7 @@ You can edit this file directly. Changes take effect on the next `init`,| `wiki_style` | `comprehensive` | `comprehensive`, `caveman`, `reference`, `tutorial`, `custom` || `language` | `en` | Output language for generated wiki pages: `en`, `ar`, `de`, `es`, `fr`, `hi`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `ru`, `tr`, `zh` || `enable_onboarding` | `true` | Show first-run onboarding prompts (CLI and web) |+| `max_file_pages` | unset | Most file pages a run emits, highest importance first. Unset means one page per eligible file, which is the default on every repo. `repowise init` offers to set it in advanced mode above 2,000 documentable files; `update --full` and `generate` honour it after that || `distill` | see below | Output distillation config || `mcp` | see below | MCP tool surface config || `refactoring` | see below | Refactoring-intelligence config |diff --git a/packages/cli/src/repowise/cli/commands/generate_cmd/command.py b/packages/cli/src/repowise/cli/commands/generate_cmd/command.pyindex 4ff194440..f6019cec9 100644--- a/packages/cli/src/repowise/cli/commands/generate_cmd/command.py+++ b/packages/cli/src/repowise/cli/commands/generate_cmd/command.py@@ -24,6 +24,7 @@ from repowise.cli.helpers import (get_head_commit,load_config,load_state,+ resolve_max_file_pages,resolve_provider_or_prompt,resolve_reasoning,resolve_repo_path,@@ -261,6 +262,9 @@ def generate_command(reasoning=resolve_reasoning(reasoning, cfg),enable_onboarding=bool(cfg.get("enable_onboarding", True)),wiki_style=cfg.get("wiki_style", "comprehensive"),+ # A whole-repo selection, so the file-page cap chosen at init applies+ # here; without it this run would grow the file layer back.+ max_file_pages=resolve_max_file_pages(config=cfg),)exclude_patterns = list(cfg.get("exclude_patterns") or [])diff --git a/packages/cli/src/repowise/cli/commands/init_cmd/command.py b/packages/cli/src/repowise/cli/commands/init_cmd/command.pyindex 0b6563dbe..f61cf1ddd 100644--- a/packages/cli/src/repowise/cli/commands/init_cmd/command.py+++ b/packages/cli/src/repowise/cli/commands/init_cmd/command.py@@ -36,6 +36,7 @@ from repowise.cli.helpers import (get_head_commit,load_config,load_state,+ resolve_max_file_pages,resolve_provider,resolve_reasoning,resolve_repo_path,@@ -154,6 +155,7 @@ def _run_deterministic_generation_phase(language: str,onboarding: bool,wiki_style: str,+ max_file_pages: int | None,embedder_name_resolved: str,embedder_was_requested: bool,resume: bool,@@ -220,6 +222,7 @@ def _run_deterministic_generation_phase(language=language,enable_onboarding=onboarding,wiki_style=wiki_style,+ max_file_pages=max_file_pages,)run_repo_generation(repo_path=repo_path,@@ -246,6 +249,7 @@ def _run_generation_phase(onboarding: bool,harvest_decisions: bool,wiki_style: str,+ max_file_pages: int | None,yes: bool,dry_run: bool,skip_tests: bool,@@ -277,6 +281,7 @@ def _run_generation_phase(enable_onboarding=onboarding,harvest_decisions=harvest_decisions,wiki_style=wiki_style,+ max_file_pages=max_file_pages,)plans, est = estimate_generation(result=result,@@ -840,6 +845,12 @@ def init_command(# until chosen. Resolved below: flag > this > config.yaml > English.language_choice: str | None = None+ # File-page cap picked in the advanced-mode page-volume question; None until+ # chosen, and None also means "no cap", which is the default for every repo+ # small enough that the question is never asked. Resolved below:+ # advanced-mode answer > config.yaml > uncapped.+ max_file_pages_choice: int | None = None+# The two orthogonal axes the interactive menu resolves: whether docs are# generated and whether we entered the advanced-config prompts. Initialized# for the non-interactive path (where the menu never runs); the menu@@ -924,6 +935,9 @@ def init_command(exclude = adv["exclude"]include_submodules = adv.get("include_submodules", include_submodules)run_mode = adv.get("run_mode", run_mode)+ # Asked in both branches for the same reason the embedder is: an+ # index-only run renders file pages too.+ max_file_pages_choice = adv.get("max_file_pages")# Asked in both branches: an index-only run renders a wiki too,# and those pages embed like any other, so the answer applies# either way. Read outside the docs-only block or the index-only@@ -989,6 +1003,10 @@ def init_command(config = load_config(repo_path)# Output language: CLI flag > advanced-mode choice > config.yaml > English.language = language_opt or language_choice or config.get("language", "en")+ # File-page cap: advanced-mode answer > config.yaml > uncapped. There is no+ # flag; the question is the interactive surface and config.yaml is the one a+ # script or a hosted caller writes.+ max_file_pages = resolve_max_file_pages(max_file_pages_choice, config)resolved_reasoning = resolve_reasoning(reasoning, config)exclude_patterns: list[str] = list(config.get("exclude_patterns") or []) + list(exclude)@@ -1269,6 +1287,7 @@ def init_command(language=language,onboarding=onboarding,wiki_style=wiki_style,+ max_file_pages=max_file_pages,embedder_name_resolved=embedder_name_resolved,embedder_was_requested=embedder_was_requested,resume=resume,@@ -1285,6 +1304,7 @@ def init_command(onboarding=onboarding,harvest_decisions=harvest_decisions,wiki_style=wiki_style,+ max_file_pages=max_file_pages,yes=yes,dry_run=dry_run,skip_tests=skip_tests,@@ -1324,6 +1344,7 @@ def init_command(language=language,onboarding=onboarding,wiki_style=wiki_style,+ max_file_pages=max_file_pages,# The user has a provider and just declined a bill, not the# embedder they configured, so honour it either way.embedder_was_requested=True,@@ -1354,6 +1375,14 @@ def init_command(if not onboarding:save_config_partial(repo_path, enable_onboarding=False)+ # Persist the file-page cap so `repowise update --full` and `repowise+ # generate` keep the wiki the size the user asked for instead of growing the+ # file layer back on the next run. Recorded whether or not a model wrote the+ # pages, because the cap applies to the template wiki just the same. No cap+ # stays unrecorded: it is the default.+ if max_file_pages is not None and max_file_pages != config.get("max_file_pages"):+ save_config_partial(repo_path, max_file_pages=max_file_pages)+# Persist the wiki style so `repowise update` / `restyle` honor it without# re-passing the flag. Written before any config_fingerprint is computed# below so the first update doesn't false-positive on a config change. Thediff --git a/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py b/packages/cli/src/repowise/cli/commands/init_cmd/workspace.pyindex d3bbaa635..831fd1016 100644--- a/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py+++ b/packages/cli/src/repowise/cli/commands/init_cmd/workspace.py@@ -36,6 +36,7 @@ from repowise.cli.helpers import (get_head_commit,load_config,load_state,+ resolve_max_file_pages,resolve_provider,resolve_reasoning,run_async,@@ -109,6 +110,9 @@ def _run_workspace_generation(harvest_decisions=harvest_decisions,wiki_style=wiki_style,language=language,+ # No question is asked in the workspace flow, so this only picks up a cap+ # already recorded for this repo (by a single-repo init, or by hand).+ max_file_pages=resolve_max_file_pages(config=load_config(repo_path)),)plans, est = estimate_generation(result=result,@@ -202,6 +206,9 @@ def _run_workspace_deterministic_generation(language=language,enable_onboarding=onboarding,wiki_style=wiki_style,+ # No question is asked in the workspace flow, so this only picks up a cap+ # already recorded for this repo (by a single-repo init, or by hand).+ max_file_pages=resolve_max_file_pages(config=load_config(repo_path)),)generated_pages = run_repo_generation(repo_path=repo_path,diff --git a/packages/cli/src/repowise/cli/commands/upgrade_flow.py b/packages/cli/src/repowise/cli/commands/upgrade_flow.pyindex 84bd1d53b..46b4d7146 100644--- a/packages/cli/src/repowise/cli/commands/upgrade_flow.py+++ b/packages/cli/src/repowise/cli/commands/upgrade_flow.py@@ -36,6 +36,7 @@ from repowise.cli.helpers import (get_head_commit,load_config,load_state,+ resolve_max_file_pages,resolve_provider,resolve_reasoning,run_async,@@ -434,6 +435,9 @@ def upgrade_to_full(language=cfg.get("language", "en"),reasoning=resolve_reasoning(reasoning, cfg),enable_onboarding=bool(cfg.get("enable_onboarding", True)),+ # A whole-repo selection, so the file-page cap chosen at init applies+ # here; without it this run would grow the file layer back.+ max_file_pages=resolve_max_file_pages(config=cfg),)exclude_patterns = list(cfg.get("exclude_patterns") or [])diff --git a/packages/cli/src/repowise/cli/commands/workspace_cmd.py b/packages/cli/src/repowise/cli/commands/workspace_cmd.pyindex 62b460e5b..fb11d6819 100644--- a/packages/cli/src/repowise/cli/commands/workspace_cmd.py+++ b/packages/cli/src/repowise/cli/commands/workspace_cmd.py@@ -13,6 +13,7 @@ from repowise.cli._setup import configure_cli_loggingfrom repowise.cli.helpers import (console,find_workspace_root,+ resolve_max_file_pages,resolve_reasoning,resolve_repo_path,run_async,@@ -692,6 +693,8 @@ def _generate_docs_for_added_repo(reasoning=reasoning,wiki_style=repo_cfg.get("wiki_style", "comprehensive"),language=repo_cfg.get("language", "en"),+ # Whole-repo selection, so honour the per-repo file-page cap.+ max_file_pages=resolve_max_file_pages(config=repo_cfg),)assembler = ContextAssembler(config)generator = PageGenerator(diff --git a/packages/cli/src/repowise/cli/helpers.py b/packages/cli/src/repowise/cli/helpers.pyindex 781d8c6e3..2efa48972 100644--- a/packages/cli/src/repowise/cli/helpers.py+++ b/packages/cli/src/repowise/cli/helpers.py@@ -475,6 +475,29 @@ def resolve_reasoning(raise click.ClickException(str(exc)) from exc+def resolve_max_file_pages(+ chosen: int | None = None,+ config: dict[str, Any] | None = None,+) -> int | None:+ """Resolve the file-page cap: explicit choice, then config.yaml, then none.++ ``None`` means unrationed, which is what every repo gets unless its owner+ asked otherwise (see ``GenerationConfig.max_file_pages``). A non-positive or+ unparseable ``max_file_pages`` in config.yaml is read as no cap rather than+ as zero pages: a typo must not silently delete the file layer.+ """+ if chosen is not None:+ return max(1, chosen)+ raw = (config or {}).get("max_file_pages")+ if raw is None:+ return None+ try:+ value = int(raw)+ except (TypeError, ValueError):+ return None+ return value if value > 0 else None++def save_config(repo_path: Path,provider: str,diff --git a/packages/cli/src/repowise/cli/ui/__init__.py b/packages/cli/src/repowise/cli/ui/__init__.pyindex f8205312b..eb45fe2ec 100644--- a/packages/cli/src/repowise/cli/ui/__init__.py+++ b/packages/cli/src/repowise/cli/ui/__init__.py@@ -27,6 +27,7 @@ from repowise.cli.ui.mascot import (mini,)from repowise.cli.ui.mode_selection import (+ FILE_PAGE_VOLUME_THRESHOLD,LARGE_REPO_FILE_THRESHOLD,interactive_advanced_config,interactive_customize_offer,@@ -34,6 +35,7 @@ from repowise.cli.ui.mode_selection import (interactive_generate_docs_toggle,interactive_mode_select,print_index_only_intro,+ prompt_file_page_volume,prompt_wiki_style,should_offer_fast_mode,)@@ -64,6 +66,7 @@ __all__ = ["BRAND_STYLE","DIM","ERR",+ "FILE_PAGE_VOLUME_THRESHOLD","LARGE_REPO_FILE_THRESHOLD","OK","OWL_SPINNER",@@ -94,6 +97,7 @@ __all__ = ["print_index_only_intro","print_phase_header","print_scan_summary",+ "prompt_file_page_volume","prompt_wiki_style","quick_repo_scan","should_offer_fast_mode",diff --git a/packages/cli/src/repowise/cli/ui/mode_selection.py b/packages/cli/src/repowise/cli/ui/mode_selection.pyindex 6a4ff7d64..851d5577e 100644--- a/packages/cli/src/repowise/cli/ui/mode_selection.py+++ b/packages/cli/src/repowise/cli/ui/mode_selection.py@@ -14,7 +14,7 @@ from rich.table import Tablefrom rich.text import Textfrom repowise.cli.ui.brand import BRAND, BRAND_STYLE, DIM-from repowise.cli.ui.repo_scanner import RepoScanInfo+from repowise.cli.ui.repo_scanner import RepoScanInfo, estimated_documentable_filesfrom repowise.core.generation.languages import SUPPORTED_LANGUAGESfrom repowise.core.generation.styles import DEFAULT_STYLE, list_stylesfrom repowise.core.reasoning import REASONING_MODES@@ -23,6 +23,25 @@ from repowise.core.reasoning import REASONING_MODES# fast-mode first index (graph + essential git, no LLM docs) is worth offering.LARGE_REPO_FILE_THRESHOLD = 5000+# Above this many documentable files, advanced mode asks whether to bound the+# file-page bucket; at or below it the question is never asked and every eligible+# file gets a page.+#+# The number comes from the size distribution of the 2,309 repositories indexed+# on repowise.dev (latest ready snapshot each): median 64 files, p75 235, p90+# 907, p95 2,047, p98 3,290, largest 14,943. 2,000 sits at roughly the 95th+# percentile, so 19 in 20 repos never see the question, and the ones that do are+# the ones where the file tail is measured in tens of megabytes.+#+# It doubles as the recommended cap, which keeps the choice continuous: a repo+# just over the line loses almost nothing by taking the recommendation.+FILE_PAGE_VOLUME_THRESHOLD = 2000++# Bytes a file page occupies with its metadata, measured over 1,961 file pages in+# a local index (mean 8.8 KB). Used to quote wiki size while asking; an order of+# magnitude is the point, not a byte count.+_FILE_PAGE_BYTES = 8_800+def should_offer_fast_mode(scan: RepoScanInfo | None) -> bool:"""Whether to surface the fast-mode offer for this repo.@@ -204,6 +223,66 @@ def _prompt_scope(console: Console, scan: RepoScanInfo | None, result: dict[str,result["include_submodules"] = False+def _format_mb(pages: int) -> str:+ """Wiki bytes for *pages* file pages, as a human size."""+ mb = pages * _FILE_PAGE_BYTES / 1_000_000+ return f"{mb:.0f} MB" if mb >= 10 else f"{mb:.1f} MB"+++def prompt_file_page_volume(+ console: Console,+ scan: RepoScanInfo | None,+ *,+ threshold: int = FILE_PAGE_VOLUME_THRESHOLD,+) -> int | None:+ """Ask a large repo whether to bound the file-page bucket.++ Returns the cap to write to ``GenerationConfig.max_file_pages``: an int to+ take the top slice by importance, or ``None`` for every eligible file (the+ unrationed default, and the answer on any repo at or below *threshold*).++ One page per source file is what makes small and mid-size repos good, so the+ question is only asked above *threshold*, and only in advanced mode. It is+ asked before ingestion, so the page count quoted is the pre-scan estimate.++ Never blocks: a terminal that cannot answer takes the recommendation and the+ run carries on, same as the other optional questions in an init flow.+ """+ estimate = estimated_documentable_files(scan)+ if estimate <= threshold:+ return None++ console.print()+ console.print(f" [{BRAND}]Page volume[/]")+ console.print(+ f" [dim]About [bold]{estimate:,}[/bold] files here would each get a file page "+ f"({_format_mb(estimate)} of wiki).\n"+ " File pages are rendered from structure, so they cost no model tokens — what\n"+ " the tail costs is wiki size, one embedding call each, and search results\n"+ " that restate what a subsystem page already says.[/dim]"+ )+ console.print()+ console.print(+ f" [{BRAND}][1][/] Top [bold]{threshold:,}[/bold] by importance [dim](recommended) — "+ f"~{threshold:,} pages, {_format_mb(threshold)}[/dim]"+ )+ console.print(+ f" [{BRAND}][2][/] Every eligible file [dim]— ~{estimate:,} pages, "+ f"{_format_mb(estimate)}, roughly "+ f"{max(1, round(estimate / 1000)):,}-{max(2, round(2 * estimate / 1000)):,} min "+ "longer to render and embed[/dim]"+ )+ try:+ choice = Prompt.ask(" File pages", choices=["1", "2"], default="1", console=console)+ except (click.Abort, EOFError):+ # isatty() lied about being answerable (Windows Git Bash `< /dev/null`,+ # pty wrappers, `docker run -t` without -i). Take the recommendation and+ # keep going: an agent or CI job must never hang here.+ console.print(" [dim]No answer available — taking the recommendation.[/dim]")+ return threshold+ return threshold if choice == "1" else None++def _prompt_run_mode(console: Console,result: dict[str, Any],@@ -411,6 +490,9 @@ def _build_summary_table(# Bullet-list when many patterns — comma-joined wraps unreadably.summary.add_row("Exclude", "\n".join(f"• {p}" for p in patterns))+ if result.get("max_file_pages"):+ summary.add_row("File pages", f"top {result['max_file_pages']:,} by importance")+if not generate_docs and result.get("embedder"):summary.add_row("Embedder", result["embedder"])@@ -458,9 +540,9 @@ def interactive_advanced_config(Returns a dict with keys matching init_command kwargs:``commit_limit``, ``follow_renames``, ``skip_tests``, ``skip_infra``,- ``exclude``, ``include_submodules``, ``run_mode``, ``generate_docs`` (always),- plus ``concurrency``, ``reasoning``, ``embedder``, ``test_run``,- ``onboarding``, ``harvest_decisions``, ``wiki_style``,+ ``exclude``, ``include_submodules``, ``run_mode``, ``max_file_pages``,+ ``generate_docs`` (always), plus ``concurrency``, ``reasoning``, ``embedder``,+ ``test_run``, ``onboarding``, ``harvest_decisions``, ``wiki_style``,``language`` (docs only).Editor integration prompts are intentionally not asked here so that full and@@ -482,6 +564,11 @@ def interactive_advanced_config(_prompt_run_mode(console, result, allow_fast=allow_fast, is_large=is_large)patterns = _prompt_exclude(console, scan, result)_prompt_git(console, scan, result)+ # Asked in both branches: an index-only run renders file pages too (they need+ # no model), so page volume is a real question either way. Fast mode renders+ # no wiki at all, so there is nothing to bound.+ if result.get("run_mode") != "fast":+ result["max_file_pages"] = prompt_file_page_volume(console, scan)if generate_docs:_prompt_generation(console,diff --git a/packages/cli/src/repowise/cli/ui/repo_scanner.py b/packages/cli/src/repowise/cli/ui/repo_scanner.pyindex 0cf13655a..320f93b7c 100644--- a/packages/cli/src/repowise/cli/ui/repo_scanner.py+++ b/packages/cli/src/repowise/cli/ui/repo_scanner.py@@ -131,16 +131,42 @@ def quick_repo_scan(repo_path: Path) -> RepoScanInfo:return info+_NON_SOURCE_LANGS = frozenset({"JSON", "YAML", "Markdown", "HTML", "CSS"})+++def source_file_counts(scan: RepoScanInfo) -> dict[str, int]:+ """Per-language counts for the source languages only.++ Data, markup and stylesheet files are counted by the scan (they are files)+ but are not what "how much code is here" means, and they get no file page.+ """+ return {+ lang: count for lang, count in scan.language_counts.items() if lang not in _NON_SOURCE_LANGS+ }+++def estimated_documentable_files(scan: RepoScanInfo | None) -> int:+ """Roughly how many files would get a ``file_page``, from the pre-scan.++ An estimate, not the count: the real allow-set comes out of ingestion, which+ has not run when the interactive questions are asked. It is the source-file+ count less the test files, because test files and pure re-export modules are+ the two classes the importance floor drops+ (``generation.selection.selector._passes_importance_floor``). Used only to+ decide whether a repo is big enough to be asked about page volume, and to+ quote an order of magnitude while asking.+ """+ if scan is None:+ return 0+ src = sum(source_file_counts(scan).values()) or scan.total_files+ return max(0, src - scan.test_file_count)++def print_scan_summary(console: Console, scan: RepoScanInfo) -> None:"""Print a compact pre-scan summary below the banner."""# File count + language count- lang_count = len(- [- name- for name, c in scan.language_counts.items()- if c > 0 and name not in ("JSON", "YAML", "Markdown", "HTML", "CSS")- ]- )+ source_langs = {lang: c for lang, c in source_file_counts(scan).items() if c > 0}+ lang_count = len(source_langs)parts = [f"[bold]{scan.total_files:,}[/bold] files"]if lang_count:@@ -151,11 +177,6 @@ def print_scan_summary(console: Console, scan: RepoScanInfo) -> None:header_line = " · ".join(parts)# Top languages (source code only, top 4)- source_langs = {- lang: count- for lang, count in scan.language_counts.items()- if lang not in ("JSON", "YAML", "Markdown", "HTML", "CSS")- }total_source = sum(source_langs.values()) or 1top_langs = sorted(source_langs.items(), key=lambda x: -x[1])[:4]lang_parts = [f"{lang} {count / total_source:.0%}" for lang, count in top_langs]diff --git a/packages/core/src/repowise/core/generation/models.py b/packages/core/src/repowise/core/generation/models.pyindex 33b740748..8bdc5f383 100644--- a/packages/core/src/repowise/core/generation/models.py+++ b/packages/core/src/repowise/core/generation/models.py@@ -113,12 +113,21 @@ class GenerationConfig:# practice: the selector floors the bucket at one, so a repo with few# public symbols still gets a spotlight.top_symbol_percentile: float = 0.10- # NOTE: read by nothing today. File pages are deliberately not rationed- # (see selection/selector.py::select_pages -- the concept partition is a- # total cover of the production files). Kept because bounding the file- # bucket is a live question on very large repos, but bounding it is a- # design decision, not a default change.- file_page_top_percentile: float = 0.10+ # The most ``file_page`` pages a run will emit, highest importance score+ # first. ``None`` (the default) means unrationed: every file that clears the+ # importance floor gets a page, which is what makes small and mid-size repos+ # good and is the behaviour every repo had before this field existed.+ #+ # It exists for the other end of the distribution. A 10.8k-file monorepo+ # produced 8,756 file pages inside a 14,027-page wiki, and a file page+ # averages ~8.8 KB with its metadata, so that tail alone is ~77 MB of wiki+ # whose entries mostly restate what their concept page already says. Bounding+ # it is a choice the owner of the repo makes: `repowise init` asks in advanced+ # mode above a threshold, and any programmatic caller (the hosted pipeline+ # included) sets it here. Capping file pages does not reduce model spend --+ # file pages are rendered from structure and cost no tokens -- it reduces+ # pages, bytes, embedding calls and retrieval dilution.+ max_file_pages: int | None = Nonefile_page_min_symbols: int = 1skip_trivial_files: bool = Truededupe_near_clones: bool = Truediff --git a/packages/core/src/repowise/core/generation/selection/selector.py b/packages/core/src/repowise/core/generation/selection/selector.pyindex 58c357613..0cb8007de 100644--- a/packages/core/src/repowise/core/generation/selection/selector.py+++ b/packages/core/src/repowise/core/generation/selection/selector.py@@ -249,6 +249,13 @@ def _build_file_candidates(if s > 0.0:scored.append((s, path))scored.sort(key=lambda x: (-x[0], x[1]))+ # Unrationed by default: every file above the floor gets a page. A repo whose+ # owner asked for a bound gets the strongest ``max_file_pages`` of them, by+ # the score already computed above rather than by a second ranking. The sort+ # is total (score, then path), so the cut is deterministic.+ cap = getattr(inputs.config, "max_file_pages", None)+ if cap is not None and cap >= 0:+ return scored[:cap]return scored@@ -585,8 +592,11 @@ def select_pages(inputs: SelectionInputs) -> Selection:One path, whatever the caller. Every page type below the concept tree isrendered from structure and costs no tokens, and the concept partition is atotal cover of the production files that would stop being a map of the- repository if it were rationed. So there is nothing left to ration: each- bucket takes every candidate that clears its floor.+ repository if it were rationed. So by default there is nothing left to+ ration: each bucket takes every candidate that clears its floor. The one+ exception is opt-in and owner-chosen -- ``config.max_file_pages`` bounds the+ file bucket for repos large enough that its tail costs more in bytes and+ retrieval noise than it pays back.That this does not depend on whether an API key is present is the point.Selection used to fork on it, which meant a keyed and a keyless index of thediff --git a/tests/unit/cli/test_file_page_volume_prompt.py b/tests/unit/cli/test_file_page_volume_prompt.pynew file mode 100644index 000000000..67cf2f403--- /dev/null+++ b/tests/unit/cli/test_file_page_volume_prompt.py@@ -0,0 +1,183 @@+"""The page-volume question, and the ways it must not block.++The question only fires on a repo big enough for the file-page tail to matter,+and only in advanced mode. Everything else about it is agent safety: `--yes`+must never reach it, and a terminal that reports itself answerable and then+reads EOF must take the recommendation and let the run finish. A hang here+strands every agent and CI job that sets repowise up unattended.+"""++from __future__ import annotations++import io+import sys++import pytest+from click.testing import CliRunner+from rich.console import Console+from rich.prompt import Prompt++from repowise.cli.helpers import resolve_max_file_pages+from repowise.cli.main import cli+from repowise.cli.ui import mode_selection+from repowise.cli.ui.mode_selection import (+ FILE_PAGE_VOLUME_THRESHOLD,+ prompt_file_page_volume,+)+from repowise.cli.ui.repo_scanner import RepoScanInfo, estimated_documentable_files+++def _scan(source_files: int, *, tests: int = 0) -> RepoScanInfo:+ return RepoScanInfo(+ total_files=source_files + tests,+ language_counts={"Python": source_files + tests},+ test_file_count=tests,+ )+++def _console() -> Console:+ """A console that renders to a throwaway buffer, not the test output."""+ return Console(file=io.StringIO(), width=100)+++class TestWhenItIsAsked:+ def test_small_repo_is_never_asked(self, monkeypatch):+ """One page per file is what makes small repos good. No question."""++ def _boom(*_a, **_k):+ raise AssertionError("prompted a repo below the threshold")++ monkeypatch.setattr(Prompt, "ask", _boom)+ assert prompt_file_page_volume(_console(), _scan(300)) is None++ def test_repo_at_the_threshold_is_never_asked(self, monkeypatch):+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")+ scan = _scan(FILE_PAGE_VOLUME_THRESHOLD)+ assert prompt_file_page_volume(_console(), scan) is None++ def test_no_scan_is_never_asked(self, monkeypatch):+ """Without a pre-scan there is no evidence the repo is large."""++ def _boom(*_a, **_k):+ raise AssertionError("prompted with no scan")++ monkeypatch.setattr(Prompt, "ask", _boom)+ assert prompt_file_page_volume(_console(), None) is None++ def test_tests_do_not_count_towards_the_threshold(self):+ """Test files get no file page, so they cannot push a repo over."""+ scan = _scan(500, tests=5_000)+ assert estimated_documentable_files(scan) == 500+++class TestTheAnswer:+ def test_recommendation_caps_at_the_threshold(self, monkeypatch):+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")+ assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD++ def test_everything_means_no_cap(self, monkeypatch):+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "2")+ assert prompt_file_page_volume(_console(), _scan(9_000)) is None++ def test_the_recommendation_is_the_default(self, monkeypatch):+ """Enter-through takes the bounded wiki."""+ seen: dict = {}++ def _ask(_msg, **kwargs):+ seen["default"] = kwargs.get("default")+ return kwargs["default"]++ monkeypatch.setattr(Prompt, "ask", _ask)+ prompt_file_page_volume(_console(), _scan(9_000))+ assert seen["default"] == "1"++ def test_it_says_what_each_choice_costs(self, monkeypatch):+ """The page count and wiki size are the point of asking at all."""+ monkeypatch.setattr(Prompt, "ask", lambda *_a, **_k: "1")+ buffer = io.StringIO()+ prompt_file_page_volume(Console(file=buffer, width=100), _scan(9_000))+ out = buffer.getvalue()+ assert "9,000" in out # what every eligible file would cost+ assert f"{FILE_PAGE_VOLUME_THRESHOLD:,}" in out # what the recommendation costs+ assert "MB" in out+ assert "no model tokens" in out # file pages are rendered, not written+++class TestItNeverBlocks:+ def test_eof_takes_the_recommendation(self, monkeypatch):+ """isatty() lied. Continue with the default rather than dying."""++ def _eof(*_a, **_k):+ raise EOFError++ monkeypatch.setattr(Prompt, "ask", _eof)+ assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD++ def test_closed_stdin_takes_the_recommendation(self, monkeypatch):+ """The real prompt against a stdin with nothing on it. Must not raise."""+ monkeypatch.setattr(sys, "stdin", io.StringIO(""))+ result = prompt_file_page_volume(_console(), _scan(9_000))+ assert result == FILE_PAGE_VOLUME_THRESHOLD++ def test_abort_takes_the_recommendation(self, monkeypatch):+ """click.Abort is what a pty wrapper produces on an unanswerable read."""+ import click++ def _abort(*_a, **_k):+ raise click.Abort++ monkeypatch.setattr(Prompt, "ask", _abort)+ assert prompt_file_page_volume(_console(), _scan(9_000)) == FILE_PAGE_VOLUME_THRESHOLD+++class TestYesNeverAsks:+ @pytest.fixture+ def keyless_env(self, monkeypatch):+ for var in (+ "ANTHROPIC_API_KEY",+ "OPENAI_API_KEY",+ "OPENROUTER_API_KEY",+ "DEEPSEEK_API_KEY",+ "KIMI_API_KEY",+ "GOOGLE_API_KEY",+ "GEMINI_API_KEY",+ "OLLAMA_BASE_URL",+ "LITELLM_API_KEY",+ "REPOWISE_PROVIDER",+ ):+ monkeypatch.delenv(var, raising=False)++ def test_init_yes_never_reaches_the_question(self, tmp_path, monkeypatch, keyless_env):+ """`--yes` forces the non-interactive path, so no question is asked.++ Asserted on a real run with a terminal claimed and nothing on stdin, the+ exact shape an agent produces.+ """++ def _boom(*_a, **_k):+ raise AssertionError("asked about page volume under --yes")++ monkeypatch.setattr(mode_selection, "prompt_file_page_volume", _boom)+ monkeypatch.setattr(sys.stdin, "isatty", lambda: True, raising=False)++ result = CliRunner().invoke(cli, ["init", str(tmp_path), "--no-prose", "--yes"])+ assert result.exit_code == 0, result.output+++class TestResolution:+ def test_no_config_means_no_cap(self):+ assert resolve_max_file_pages(None, {}) is None++ def test_config_value_is_honoured(self):+ assert resolve_max_file_pages(None, {"max_file_pages": 1500}) == 1500++ def test_an_answer_beats_config(self):+ assert resolve_max_file_pages(2000, {"max_file_pages": 500}) == 2000++ def test_zero_or_negative_reads_as_no_cap(self):+ """A typo must not silently delete the file layer."""+ assert resolve_max_file_pages(None, {"max_file_pages": 0}) is None+ assert resolve_max_file_pages(None, {"max_file_pages": -5}) is None++ def test_garbage_reads_as_no_cap(self):+ assert resolve_max_file_pages(None, {"max_file_pages": "lots"}) is Nonediff --git a/tests/unit/cli/test_ui_package.py b/tests/unit/cli/test_ui_package.pyindex 7585c0d9c..38db96a43 100644--- a/tests/unit/cli/test_ui_package.py+++ b/tests/unit/cli/test_ui_package.py@@ -111,6 +111,8 @@ def test_advanced_config_default_keys_no_fast(monkeypatch: pytest.MonkeyPatch) -"exclude": (),"commit_limit": 500,"follow_renames": False,+ # No scan, so the repo cannot be shown to be large: no cap, no question.+ "max_file_pages": None,"concurrency": 10,"reasoning": "auto","embedder": "mock",@@ -141,6 +143,8 @@ def test_advanced_config_index_only_omits_generation_keys("exclude": (),"commit_limit": 500,"follow_renames": False,+ # Asked in this branch too: an index-only run renders file pages as well.+ "max_file_pages": None,"embedder": "mock",}# The knobs that only shape a model's writing must not appear.diff --git a/tests/unit/generation/test_selection_contract.py b/tests/unit/generation/test_selection_contract.pyindex db7c5eaa8..89f10fc41 100644--- a/tests/unit/generation/test_selection_contract.py+++ b/tests/unit/generation/test_selection_contract.py@@ -228,3 +228,56 @@ def test_empty_repo_emits_no_content_pages():assert sel.file_page_paths == []assert sel.symbol_spotlights == []assert sel.module_groups == []+++# ---------------------------------------------------------------------------+# The one opt-in bound: max_file_pages+# ---------------------------------------------------------------------------+++def test_file_bucket_is_unrationed_by_default():+ """No cap unless the repo's owner asked for one."""+ assert GenerationConfig().max_file_pages is None+++def test_cap_takes_the_strongest_file_pages():+ """The cap slices the ranking the selector already computed."""+ parsed, pagerank, betweenness, community = _build_synthetic_repo(400)+ cfg = GenerationConfig(max_file_pages=50)++ sel = select_pages(_inputs(parsed, pagerank, betweenness, community, cfg))+ uncapped = select_pages(_inputs(parsed, pagerank, betweenness, community, GenerationConfig()))++ assert len(sel.file_page_paths) == 50+ # Same ranking, just cut short: the kept pages are the uncapped run's first 50.+ assert sel.file_page_paths == uncapped.file_page_paths[:50]+++def test_cap_above_the_candidate_count_changes_nothing():+ """A repo smaller than its cap is untouched by it."""+ parsed, pagerank, betweenness, community = _build_synthetic_repo(30)+ capped = select_pages(+ _inputs(parsed, pagerank, betweenness, community, GenerationConfig(max_file_pages=2000))+ )+ assert len(capped.file_page_paths) == 30+++def test_cap_leaves_the_other_buckets_alone():+ """Bounding the file layer is not a request to shrink the concept tree."""+ parsed, pagerank, betweenness, community = _build_synthetic_repo(300)+ capped = select_pages(+ _inputs(parsed, pagerank, betweenness, community, GenerationConfig(max_file_pages=25))+ )+ uncapped = select_pages(_inputs(parsed, pagerank, betweenness, community, GenerationConfig()))+ assert capped.counts()["file_page"] == 25+ assert capped.counts()["module_page"] == uncapped.counts()["module_page"]+ assert capped.counts()["symbol_spotlight"] == uncapped.counts()["symbol_spotlight"]+++def test_cap_is_deterministic():+ """Same inputs, same cut — the sort is total, so the tie-break is the path."""+ parsed, pagerank, betweenness, community = _build_synthetic_repo(200)+ cfg = GenerationConfig(max_file_pages=40)+ first = select_pages(_inputs(parsed, pagerank, betweenness, community, cfg))+ second = select_pages(_inputs(list(reversed(parsed)), pagerank, betweenness, community, cfg))+ assert first.file_page_paths == second.file_page_paths