Skip to content

Codebase intelligence for AI and humans: code health scores, auto-generated docs, git analytics, dead code detection, and architectural decisions via MCP.

fix: ingestion UX, submodule handling, and docs cleanup (#57) (#70)

Raghav Chamadiyac92cb809f4f2
diff --git a/README.md b/README.md
index ec56a38f7..409283f41 100644
--- a/README.md
+++ b/README.md
@@ -358,6 +358,9 @@ Hosted adds what only makes sense in a managed, multi-user environment:
```bash
# Core
repowise init [PATH] # index codebase (one-time)
+repowise init --index-only # graph + git + dead code, no LLM, no cost
+repowise init -x vendor/ -x proto/ # exclude patterns (gitignore syntax)
+repowise init --include-submodules # include git submodule directories
repowise update [PATH] # incremental update (<30 seconds)
repowise serve [PATH] # MCP server + local dashboard
repowise watch [PATH] # auto-update on file save
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 7e8723c1b..dcca50e0c 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -9,6 +9,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
+## [0.2.2] — 2026-04-11
+
+### Added
+- **Traversal stats** — `FileTraverser` now tracks skip reasons (`.gitignore`, blocked extension, binary, oversized, generated, `--exclude`, `.repowiseIgnore`, unknown language) via a new `TraversalStats` dataclass. Stats are surfaced after traversal as a filtering summary showing how many files were included vs excluded and why.
+- **Submodule handling** — git submodule directories (parsed from `.gitmodules`) are now excluded by default during traversal. Added `--include-submodules` flag to `repowise init` to opt in.
+- **Language breakdown** — generation plan table now shows language distribution (e.g. "Languages: python 79%, typescript 14%"). Completion panel shows top languages with percentages instead of just a count.
+- **Multi-line exclude input** — interactive advanced mode now prompts for exclude patterns one per line instead of comma-separated on a single line.
+- 10 new unit tests covering `TraversalStats` counters, language counts, and submodule handling.
+
+### Changed
+- Traverse progress bar uses spinner mode instead of showing misleading pre-filter totals (e.g. "2132/83601").
+- Traverse phase label changed from "Traversing files..." to "Scanning & filtering files...".
+
+### Docs
+- Updated README CLI reference with `--index-only`, `-x`, and `--include-submodules` examples.
+- Updated website docs (`cli-reference.md`, `configuration.md`, `getting-started.md`) with submodule handling, `.gitignore` documentation, and new output examples.
+- Reorganized `docs/` directory: architecture docs into `docs/architecture/`, internals into `docs/internals/`.
+- Removed stale one-time documents (PHASE_5_5_IMPLEMENTATION, GIT_INTELLIGENCE_AUDIT, MCP_AND_STATE_REVIEW, MCP_TOOLS_TEST_REPORT).
+
+---
+
## [0.2.1] — 2026-04-10
### Added
diff --git a/docs/GIT_INTELLIGENCE_AUDIT.md b/docs/GIT_INTELLIGENCE_AUDIT.md
deleted file mode 100644
index 9eecd4322..000000000
--- a/docs/GIT_INTELLIGENCE_AUDIT.md
+++ /dev/null
@@ -1,351 +0,0 @@
-# Git Intelligence Audit & Improvement Plan
-
-> Audit date: 2026-03-26
-> Scope: git_indexer.py, git metadata model, MCP tools, decision records
-
----
-
-## Current Architecture
-
-### Collection (git_indexer.py)
-
-| Operation | Method | Limit |
-|-----------|--------|-------|
-| Per-file commit history | `repo.iter_commits(paths=file, max_count=500)` | 500 commits/file |
-| Blame ownership | `repo.blame("HEAD", file)` | Files <= 100 KB only |
-| Co-change detection | `git log -500 --name-only --no-merges` | 500 repo-wide commits |
-| Tracked file listing | `git ls-files` | All files |
-| Significant commits | Filter by message quality, skip bots/noise | Top 10/file |
-| Churn percentile | Rank by `commit_count_90d` across all files | Top 25% = hotspot |
-
-### Storage (git_metadata table)
-
-```
-commit_count_total, commit_count_90d, commit_count_30d
-first_commit_at, last_commit_at, age_days
-primary_owner_name, primary_owner_email, primary_owner_commit_pct
-top_authors_json, significant_commits_json, co_change_partners_json
-is_hotspot, is_stable, churn_percentile
-```
-
-### Consumers (MCP tools)
-
-| Tool | Git Fields Used | Purpose |
-|------|----------------|---------|
-| `get_overview` | None | Pre-generated wiki pages only |
-| `get_context` | owner, last_commit, age, top_authors | Ownership & freshness |
-| `get_risk` | churn_percentile, commit_count_30d/90d, significant_commits, co_change_partners, is_hotspot | Risk trends, hotspot detection, risk classification |
-| `get_why` | top_authors, significant_commits, first/last_commit, age_days, commit_count_total | Origin story, commit-decision linking, archaeology fallback |
-| `search_codebase` | Indirect (page confidence) | - |
-| `get_dependency_path` | None | Pre-computed graph only |
-| `get_dead_code` | last_commit_at, primary_owner, age_days | Enriches dead code findings |
-| `get_architecture_diagram` | None | Pre-computed pagerank for sorting |
-
----
-
-## Phase 1 — Quick Wins (Low Effort, High Signal)
-
-### 1.1 Make the 500 commit limit configurable
-
-**Where:** `git_indexer.py:357` and `git_indexer.py:233`
-
-**Problem:** Hardcoded `max_count=500` for both per-file history and co-change analysis. For long-lived repos, this may cover only 1-2 years. `commit_count_total` is capped at 500 even if the file has 3000 commits, making the "stable" classification unreliable and hiding how deeply embedded a file is.
-
-**Fix:**
-- Add `commit_limit` to `.repowise/config.yaml` (default 500, max 5000)
-- Pass through from CLI: `repowise init --commit-limit 2000`
-- Store a `commit_count_capped: bool` column on `git_metadata` so downstream tools know when the count is truncated
-- Surface this in `get_risk`: "Note: commit history truncated at 500; actual history may be deeper"
-
-**Affected tools:** `get_risk` (total count accuracy), `get_context` (ownership accuracy), `get_why` (archaeology depth)
-
----
-
-### 1.2 Un-skip revert commits
-
-**Where:** `git_indexer.py:45` — `_SKIP_PREFIXES = ("Merge ", "Bump ", "chore:", "ci:", "style:", "build:", "release:", "revert:")`
-
-**Problem:** `revert:` commits are some of the most valuable signals in a codebase — they indicate something went wrong and was rolled back. Currently filtered as noise.
-
-**Fix:**
-- Remove `"revert:"` from `_SKIP_PREFIXES`
-- Add `"revert"` to `DECISION_SIGNAL_KEYWORDS` in `decision_extractor.py` so reverts are mined for decisions
-- In `_classify_risk_type`, count revert commits alongside fix commits for bug-prone classification
-
-**Affected tools:** `get_risk` (better bug-prone detection), `get_why` (richer archaeology)
-
----
-
-### 1.3 Lower minimum message length
-
-**Where:** `git_indexer.py:47` — `_MIN_MESSAGE_LEN = 20`
-
-**Problem:** Filters out meaningful short messages like `"fix auth race cond"` (18 chars), `"add JWT support"` (15 chars).
-
-**Fix:** Lower to `12`. Still filters `"wip"`, `"fix"`, `"update"` type noise.
-
----
-
-### 1.4 Conditionally keep `build:` commits with decision signals
-
-**Where:** `git_indexer.py:478-490` — `_is_significant_commit`
-
-**Problem:** `build:` commits like `"build: migrate from webpack to vite"` are architectural decisions but are currently filtered.
-
-**Fix:** Before checking skip prefixes, check if the message contains any `DECISION_SIGNAL_KEYWORDS`. If it does, keep it regardless of prefix (except `"Merge "`).
-
----
-
-## Phase 2 — Richer Signals (Moderate Effort)
-
-### 2.1 Track diff size (lines added/deleted)
-
-**Problem:** Commit count alone doesn't capture change magnitude. A file with 50 one-line typo fixes looks identical to one with 50 massive refactors. This is a major blind spot for `get_risk`.
-
-**What to add:**
-```python
-# New columns on git_metadata
-lines_added_90d: int
-lines_deleted_90d: int
-avg_commit_size: float # (added + deleted) / commit_count_90d
-```
-
-**How:** Use `git log --numstat` (one extra flag) during per-file indexing. Parse the `+N -N` output per commit. Aggregate into 90d and 30d windows.
-
-**Downstream impact:**
-- `get_risk`: Distinguish "churn-heavy" (many small changes) from "volatile" (large rewrites). Add `change_magnitude` to risk output.
-- `get_dead_code`: Files with 0 lines added but existing in repo are stronger dead-code candidates.
-
----
-
-### 2.2 Commit message classification
-
-**Problem:** `_classify_risk_type` only detects bug-fix commits via regex. No classification for feature, refactor, or dependency changes.
-
-**What to add:** Classify each significant commit into categories:
-```python
-COMMIT_CATEGORIES = {
- "feature": r"\b(add|implement|introduce|create|new|feat:)\b",
- "refactor": r"\b(refactor|restructure|cleanup|rename|reorganize|extract|simplify)\b",
- "fix": r"\b(fix|bug|patch|hotfix|revert|regression|broken|crash|error)\b",
- "dependency": r"\b(upgrade|bump|update dep|migrate to|switch to)\b",
-}
-```
-
-**Store as:** `commit_categories_json` on `git_metadata` — `{"feature": 12, "refactor": 8, "fix": 3, "dependency": 2}`
-
-**Downstream impact:**
-- `get_risk`: New `change_pattern` field: "primarily refactored", "bug-prone", "feature-active", "dependency-churn"
-- `get_why`: Richer intent signals without LLM calls — "60% of commits are refactors suggests ongoing architectural improvement"
-
----
-
-### 2.3 Recent ownership & bus factor
-
-**Problem:** Ownership is single-dimensional — one primary owner based on blame lines or total commit count. Doesn't answer "who owns this *now*?" or "how risky is the bus factor?"
-
-**New columns:**
-```python
-recent_owner_name: str # Most commits in last 90d
-recent_owner_commit_pct: float
-bus_factor: int # Authors needed to cover 80% of commits
-contributor_count: int # Total distinct authors
-```
-
-**Computation:**
-- `recent_owner`: Filter commits to 90d window, count by author, pick top
-- `bus_factor`: Sort authors by commit count desc, sum until >= 80% of total, count = bus factor
-
-**Downstream impact:**
-- `get_context`: Show both historical owner and recent owner. "Originally alice's (67%), recently maintained by bob (85% of last 90d changes)"
-- `get_risk`: New risk type `"bus-factor-risk"` when bus_factor == 1 and commit_count_total > 20. Surface in risk summary.
-
----
-
-### 2.4 Add temporal decay to co-change scoring
-
-**Problem:** A file pair that co-changed 5 times 2 years ago gets the same score as one co-changing 5 times last month. Stale coupling is noise.
-
-**Fix:** Capture commit timestamps during co-change analysis. Change `--format=%x00` to `--format=%x00%ct` (adds Unix timestamp). Apply exponential decay:
-
-```python
-# Instead of: pair_counts[(a, b)] += 1
-age_days = (now - commit_timestamp).days
-weight = math.exp(-age_days / 180) # half-life ~125 days
-pair_counts[(a, b)] += weight
-```
-
-**Store as:** `co_change_partners_json` already has `co_change_count` — change to float, add `last_co_change_date`.
-
-**Downstream impact:**
-- `get_risk`: Co-change partners reflect *current* coupling, not historical noise
-- Reduces false positives where files were once coupled but have since been decoupled
-
----
-
-## Phase 3 — Deeper Intelligence (Higher Effort)
-
-### 3.1 Rename/move tracking with --follow
-
-**Where:** `git_indexer.py:357` — `repo.iter_commits(paths=file_path, max_count=500)`
-
-**Problem:** If `auth/service.py` was renamed from `auth/handler.py`, commit history, age, and co-change relationships start at the rename. The file's true history is lost.
-
-**Fix:** Use `repo.iter_commits(paths=file_path, follow=True, max_count=N)` during initial indexing. GitPython supports `--follow`. Store `original_path` if different from current path.
-
-**Trade-off:** Slower — `--follow` is more expensive. Consider making this opt-in: `repowise init --follow-renames`.
-
-**Downstream impact:**
-- `get_context`: Accurate age and ownership for renamed files
-- `get_why`: Origin story traces back to true creation, not rename point
-- `get_risk`: Accurate commit_count_total for risk assessment
-
----
-
-### 3.2 Fix incremental update gaps
-
-**Where:** `git_indexer.py:267` — `index_changed_files()`
-
-**Problem:** Incremental updates (`repowise update`) re-index individual files but do NOT recompute:
-- Co-change partners (requires full 500-commit walk)
-- Churn percentiles (requires all files to rank against)
-
-A file that just became the hottest in the repo won't be flagged until the next full `repowise init`.
-
-**Fix options:**
-1. **Lightweight:** Store `last_full_reindex_at` timestamp. Surface in tools: "Percentile data from 2026-03-20; run `repowise init` to refresh."
-2. **Medium:** Recompute percentiles across all files after incremental git re-index (cheap — just sorting existing data)
-3. **Full:** Run co-change analysis on a smaller window (last 100 commits) during update, merge with existing data
-
-Recommendation: Option 2 (recompute percentiles) + option 1 (staleness indicator for co-changes).
-
----
-
-### 3.3 PR-level context for decision mining
-
-**Problem:** `mine_git_archaeology` processes individual commits, but architectural decisions are often captured at the PR level — PR descriptions, review discussions, grouped commits. Individual commits are often terse ("wip", "fix tests") while the PR says "Migrate auth from sessions to JWT."
-
-**Approach:**
-- Detect GitHub remote from `git remote -v`
-- For significant commits, resolve PR via `git log --merges --ancestry-path <sha>..HEAD` or GitHub API
-- Store `pr_number`, `pr_title` alongside significant commits
-- Feed PR titles into decision extraction (higher signal than commit messages)
-
-**Trade-off:** Requires GitHub API access (optional). Could use `gh` CLI if available, degrade gracefully if not.
-
-**Downstream impact:**
-- `get_why`: Much richer origin stories — "Introduced in PR #142: Migrate auth to JWT (merged by alice, 2024-03-15)"
-- Decision extraction: Higher-confidence decisions from PR descriptions
-
----
-
-### 3.4 Merge conflict frequency tracking
-
-**Problem:** Files that consistently have merge conflicts are risky — this signal isn't captured anywhere.
-
-**How:** `git log --merges --diff-filter=U` or parse merge commits for conflict markers. Alternatively, track files that appear in merge commits with multiple parents.
-
-**Store as:** `merge_conflict_count_90d` on `git_metadata`.
-
-**Downstream impact:**
-- `get_risk`: New signal for `"high-coupling"` risk type. Files with frequent merge conflicts are coordination bottlenecks.
-
----
-
-### 3.5 Git-aware decision staleness
-
-**Where:** `decision_extractor.py` — `compute_staleness()`
-
-**Problem:** Staleness is based on commit count and age, not on *what* changed. A decision about "use JWT for auth" isn't stale because someone reformatted the file. It's stale if recent commits conflict with the decision's content.
-
-**Fix:** When computing staleness, check if significant commits (post-decision-creation) contain keywords that conflict with the decision's tags or content. E.g.:
-- Decision says "use Redis for caching", recent commit says "migrate to memcached" → high staleness
-- Decision says "use JWT", recent commit says "reformat auth module" → low staleness
-
-Keyword overlap between decision text and recent commit messages, weighted by conflict signals ("replace", "remove", "deprecate", "switch from").
-
----
-
-## Phase 4 — Tool Enhancements
-
-Improvements to MCP tools that leverage the new data from Phases 1-3.
-
-### 4.1 Enrich `get_overview` with repo-wide git health
-
-Currently `get_overview` returns only wiki pages with no git data. Add a `git_health` section:
-
-```json
-{
- "git_health": {
- "total_files_indexed": 342,
- "hotspot_count": 28,
- "avg_bus_factor": 1.8,
- "files_with_bus_factor_1": 45,
- "churn_trend": "increasing",
- "top_churn_modules": ["src/auth", "src/api"],
- "stale_since_last_full_reindex": false
- }
-}
-```
-
----
-
-### 4.2 Boost `search_codebase` by git freshness
-
-Currently search ranks by semantic/FTS relevance only. Boost recently-active files:
-
-```python
-final_score = relevance_score * (1 + 0.2 * recency_factor)
-# recency_factor: 1.0 for files changed in 30d, 0.5 for 90d, 0.0 for older
-```
-
----
-
-### 4.3 Augment `get_dependency_path` with co-change signal
-
-When no direct import path exists between two files, check co-change data:
-
-```json
-{
- "path": [],
- "distance": -1,
- "co_change_signal": {
- "co_change_count": 7,
- "last_co_change": "2026-03-10",
- "note": "No import dependency, but these files co-change frequently — likely logical coupling"
- }
-}
-```
-
----
-
-### 4.4 Annotate `get_architecture_diagram` with change heat
-
-Add optional `show_heat: bool` parameter. When true, annotate Mermaid nodes with churn color:
-
-```mermaid
-graph TD
- auth["src/auth/service.py"]:::hot
- db["src/db/models.py"]:::warm
- utils["src/utils/helpers.py"]:::cold
- classDef hot fill:#ff6b6b
- classDef warm fill:#ffd93d
- classDef cold fill:#6bcb77
-```
-
----
-
-### 4.5 Smarter `get_dead_code` with "last meaningful change"
-
-Currently uses `last_commit_at` which includes formatting-only changes. Use commit classification (Phase 2.2) to find the last *meaningful* commit (feature or fix, not style/chore). Dead code that hasn't had a meaningful change in 12+ months is a much stronger signal.
-
----
-
-## Summary / Phase Map
-
-| Phase | Items | Theme | Depends On |
-|-------|-------|-------|------------|
-| **1** | 1.1, 1.2, 1.3, 1.4 | Quick wins — config, filter fixes | Nothing |
-| **2** | 2.1, 2.2, 2.3, 2.4 | Richer git signals — diff size, classification, ownership, decay | Phase 1 (for config) |
-| **3** | 3.1, 3.2, 3.3, 3.4, 3.5 | Deeper intelligence — renames, incremental fix, PR context, conflicts, staleness | Phase 2 (for classification data) |
-| **4** | 4.1, 4.2, 4.3, 4.4, 4.5 | Tool enhancements — consume new data in MCP tools | Phases 2-3 |
diff --git a/docs/MCP_AND_STATE_REVIEW.md b/docs/MCP_AND_STATE_REVIEW.md
deleted file mode 100644
index bf2cf8624..000000000
--- a/docs/MCP_AND_STATE_REVIEW.md
+++ /dev/null
@@ -1,712 +0,0 @@
-# repowise — MCP Server & Project State Review
-
-**Date:** 2026-03-25
-**Test repo:** `interview-coach` (3,557 files, polyglot monorepo)
-**Reviewer:** Claude Code (automated + manual)
-
----
-
-## Table of Contents
-
-1. [MCP Server Architecture](#1-mcp-server-architecture)
-2. [MCP Tools Inventory](#2-mcp-tools-inventory)
-3. [MCP Server Startup & Connection](#3-mcp-server-startup--connection)
-4. [Test Run Output Analysis](#4-test-run-output-analysis)
-5. [Database State Audit](#5-database-state-audit)
-6. [Generated Content Quality Assessment](#6-generated-content-quality-assessment)
-7. [MCP Tool-by-Tool Assessment](#7-mcp-tool-by-tool-assessment)
-8. [Issues Found — Prioritized](#8-issues-found--prioritized)
-9. [Core Engine Code Review](#9-core-engine-code-review)
-10. [Recommendations](#10-recommendations)
-
----
-
-## 1. MCP Server Architecture
-
-### Overview
-
-The MCP server (`packages/server/src/repowise/server/mcp_server.py`) exposes
-the full repowise wiki as queryable tools via the Model Context Protocol. It
-supports two transports:
-
-- **stdio** — for Claude Code, Cursor, Cline (the primary use case)
-- **SSE** — for web-based MCP clients (port 7338)
-
-### Lifecycle
-
-On startup (`_lifespan`), the server:
-
-1. Resolves the database URL from `_repo_path` → `.repowise/wiki.db`
-2. Creates an async SQLAlchemy engine + session factory
-3. Initializes FTS5 full-text search index
-4. Creates a `MockEmbedder` + `InMemoryVectorStore` (**Issue: always mock**)
-5. Attempts to load LanceDB if available (passes mock embedder to it)
-6. Sets up a decision store (LanceDB or InMemory fallback)
-
-### Key Dependencies
-
-| Dependency | Version | Purpose |
-|------------|---------|---------|
-| `mcp` (Python SDK) | >=1.0, <2 | MCP protocol implementation |
-| `FastMCP` | from mcp.server.fastmcp | High-level MCP server builder |
-| `SQLAlchemy` | >=2.0 | Async database access |
-| `aiosqlite` | >=0.20 | SQLite async driver |
-| `networkx` | >=3.3 | Graph operations (dependency path) |
-
-### Global State
-
-The server uses module-level globals set during lifespan:
-
-```python
-_session_factory # async_sessionmaker — DB sessions
-_vector_store # VectorStore — semantic search
-_decision_store # VectorStore — decision search
-_fts # FullTextSearch — FTS5 fallback
-_repo_path # str — repository root path
-```
-
-This design is functional but means only one repo can be served per MCP
-server process.
-
----
-
-## 2. MCP Tools Inventory
-
-The server implements **9 tools** (consolidated from 16 on 2026-03-25 to reduce
-sequential tool calls for common tasks).
-
-| # | Tool | Category | What It Answers |
-|---|------|----------|----------------|
-| 1 | `get_overview` | Navigation | Architecture summary, module map, entry points |
-| 2 | `get_context(targets, include?)` | Context | Docs, ownership, history, decisions, freshness for files/modules/symbols. Multi-target, single call. |
-| 3 | `get_risk(targets)` | Risk | Hotspot score, dependents, co-change partners, risk summary. Multi-target, single call. |
-| 4 | `get_why(query?)` | Decision Intelligence | 3 modes: NL search, path-based decisions, no-arg health dashboard |
-| 5 | `search_codebase` | Search | Semantic search over full wiki (natural language) |
-| 6 | `get_dependency_path` | Graph | How two files/modules are connected |
-| 7 | `get_dead_code` | Analysis | Dead/unused code findings |
-| 8 | `get_architecture_diagram` | Visualization | Mermaid diagram for repo/module/file |
-| 9 | `update_decision_records(action, ...)` | Decision Management | CRUD on decision records: create, update, update_status, delete, list, get |
-
-**Consolidation mapping** (old → new):
-- `get_module_docs`, `get_file_docs`, `get_symbol`, `get_file_history`, `get_codebase_ownership` (partial), `get_stale_pages` → **`get_context`**
-- `get_hotspots`, `get_co_changes`, `get_codebase_ownership` (partial) → **`get_risk`**
-- `get_decisions`, old `get_why`, `get_decision_health` → **`get_why`** (3-mode)
-
-### Tool Design Patterns
-
-All tools follow a consistent pattern:
-1. Accept optional `repo` parameter (path, name, or ID) — defaults to first repo
-2. Open an async session via `get_session(_session_factory)`
-3. Resolve the repository via `_get_repo()` (path → ID → name → first)
-4. Query SQL tables and return structured dicts
-
-Error handling: tools return `{"error": "message"}` dicts rather than raising
-exceptions, which is the correct pattern for MCP tools.
-
----
-
-## 3. MCP Server Startup & Connection
-
-### Prerequisites
-
-All three packages must be installed:
-
-```
-pip install -e packages/core
-pip install -e packages/cli
-pip install -e packages/server # <-- was missing, caused ModuleNotFoundError
-```
-
-### Startup Command
-
-```bash
-repowise mcp /path/to/repo --transport stdio
-```
-
-### Claude Code Configuration
-
-Project-level config at `~/.claude/projects/<project-hash>/.mcp.json`:
-
-```json
-{
- "mcpServers": {
- "repowise": {
- "command": "repowise",
- "args": ["mcp", "C:/Users/ragha/Desktop/interview-coach", "--transport", "stdio"]
- }
- }
-}
-```
-
-### Startup Issues Found
-
-| Issue | Severity | Description |
-|-------|----------|-------------|
-| Missing `repowise-server` install | **Blocker** | `repowise mcp` fails with `ModuleNotFoundError: No module named 'repowise.server'` if only cli+core are installed. The `Makefile` and README should document that all 3 packages must be installed. |
-| No startup validation | Minor | Server starts silently in stdio mode even if `.repowise/wiki.db` doesn't exist — tools will fail at query time instead of at startup |
-| Mock embedder always used | **Major** | `_lifespan()` always creates `MockEmbedder()` regardless of config |
-
----
-
-## 4. Test Run Output Analysis
-
-### Job History (interview-coach repo)
-
-| Job ID (short) | Date | Provider | Model | Status | Pages | Notes |
-|----------------|------|----------|-------|--------|-------|-------|
-| `0f25154f` | Mar 19 | litellm | gemini/gemini-2.0-flash | pending | 0/0 | Never completed |
-| `6f115548` | Mar 19 | litellm | gemini/gemini-2.0-flash | pending | 0/0 (1 failed) | `failed_page_ids: ["level-1"]` — level name not page ID |
-| `6b10db4b` | Mar 20 | litellm | gemini-3.5-flash-preview | completed | 0/0 (5 failed) | Failed levels 1-4, 6 |
-| `6cbc7ec1` | Mar 20 | gemini | gemini-3.1-flash-lite-preview | completed | 33/33 | First successful full run |
-| `2eb0b48f` | Mar 20 | gemini | gemini-3.1-flash-lite-preview | completed | 31/31 | Re-run with different files |
-| `b3f0b97c` | Mar 20 | gemini | gemini-3.1-flash-lite-preview | completed | 28/23 | `completed > total` bug |
-| `46e86452` | Mar 22 | gemini | gemini-3.1-flash-lite-preview | completed | 685/337 | Final big run, `completed > total` bug |
-
-### Key Observations
-
-- **Provider evolution:** Started with litellm→gemini-2.0-flash (failed), switched to native gemini provider with gemini-3.1-flash-lite-preview (succeeded)
-- **Early failures used level names as page IDs** — `"level-1"` instead of actual page identifiers
-- **`completed_pages > total_pages`** in multiple jobs — the `total_pages` estimate is computed before all levels are planned
-- **Total tokens used:** 4,223,332 (from state.json)
-
-### .repowise Directory Structure
-
-```
-.repowise/
-├── config.yaml (74 bytes — provider, model, embedder)
-├── state.json (191 bytes — last commit, total pages, provider)
-├── wiki.db (24 MB — all structured data)
-├── mcp.json (292 bytes — auto-generated MCP config)
-├── findings.md (11 KB — codebase intelligence report)
-├── jobs/ (7 job checkpoint files)
-└── export/ (721 markdown files)
-```
-
-### Export Files
-
-721 markdown files exported, matching the DB page count. Files are named using
-the target_path with `/` → `_`, `::` → `__` sanitization. File types exported:
-
-- 549 file pages (individual source file documentation)
-- 102 symbol spotlight pages (high-PageRank symbols)
-- 34 infrastructure pages (Dockerfile, shell scripts, etc.)
-- 22 SCC pages (circular dependency analysis)
-- 6 module pages (backend, frontend, hire-backend, etc.)
-- 6 cross-package relationship pages
-- 1 repository overview
-- 1 architecture diagram
-
----
-
-## 5. Database State Audit
-
-### Schema (15 tables)
-
-| Table | Rows | Purpose |
-|-------|------|---------|
-| `repositories` | 1 | Repo registration and sync state |
-| `wiki_pages` | 721 | Generated wiki page content |
-| `wiki_page_versions` | — | Version history for diff view |
-| `wiki_symbols` | 12,955 | Symbol index (functions, classes, methods) |
-| `graph_nodes` | 3,632 | Dependency graph nodes (file type only) |
-| `graph_edges` | 9,506 | Dependency graph edges |
-| `git_metadata` | 2,022 | Per-file git history |
-| `dead_code_findings` | 1,233 | Dead code findings |
-| `decision_records` | 23 | Architectural decision records |
-| `generation_jobs` | — | Job state machine |
-| `webhook_events` | — | Webhook audit log |
-| `page_fts` + related | — | SQLite FTS5 search index |
-
-### Data Integrity Checks
-
-| Check | Result | Notes |
-|-------|--------|-------|
-| `state.json total_pages` vs DB | **MISMATCH** | 685 vs 721 — state tracks last job only |
-| Page types distribution | OK | All expected types present |
-| Symbol coverage | OK | 12,955 symbols across all parsed files |
-| Graph completeness | **PARTIAL** | Only `file` node type (no symbol/package/external nodes) |
-| Graph edge types | **MISSING** | No `edge_type` column in `graph_edges` |
-| Git metadata coverage | OK | 2,022/3,557 files (git-tracked files only) |
-| Dead code by kind | OK | 1,230 unreachable_file + 3 zombie_package |
-| Decision records | OK | 23 proposed (from git_archaeology + readme_mining) |
-| FTS index | OK | Tables created and populated |
-
-### Graph Schema Gap
-
-**Architecture doc describes:**
-- Node types: `file`, `symbol`, `package`, `external`
-- Edge types: `imports`, `calls`, `inherits`, `implements`, `co_changes`, etc.
-
-**Actual DB schema:**
-- `graph_nodes`: has `node_type` column but only contains `file` values
-- `graph_edges`: has NO `edge_type` column — only `source_node_id`, `target_node_id`, `imported_names_json`
-
-This means the `get_dependency_path` MCP tool hardcodes `"relationship": "imports"` for every edge, losing the rich relationship type information described in the architecture.
-
----
-
-## 6. Generated Content Quality Assessment
-
-### Scoring (1-5)
-
-| Content Type | Quality | Notes |
-|-------------|---------|-------|
-| Repository overview | 5/5 | Accurate tech stack, entry points, architecture description |
-| Architecture diagram | 4/5 | Good Mermaid diagram, correct dependency flow, notes circular deps |
-| Module pages | 5/5 | Comprehensive API summaries, correct architecture notes |
-| File pages | 4/5 | Accurate symbol docs, good dependency analysis |
-| Symbol spotlights | 4/5 | Useful for key symbols, good parameter documentation |
-| SCC pages | 5/5 | Excellent cycle analysis with actionable refactoring suggestions |
-| Cross-package pages | 5/5 | Real coupling analysis with actual API types identified |
-| Infrastructure pages | 4/5 | Good script documentation |
-| Findings report | 5/5 | Outstanding codebase intelligence — hotspots, co-changes, milestones |
-
-### Example Quality: SCC-79 (Circular Dependency)
-
-The `scc-79.md` page correctly identifies:
-- The `auth-context.tsx` → `payment-context.tsx` → `tier-context.tsx` → `auth-context.tsx` cycle
-- WHY it exists (bidirectional state dependency)
-- 5 specific refactoring strategies (extract types, dependency injection, domain service layer, event-driven, hierarchical refactoring)
-
-This is genuinely useful documentation that would take a human engineer significant time to produce.
-
----
-
-## 7. MCP Tool-by-Tool Assessment (Live Test Results)
-
-All tools were tested live against the interview-coach wiki database on 2026-03-25
-(originally 16 tools; consolidated to 9 tools on 2026-03-25).
-
-### Result Summary
-
-| # | Tool | Status | Verdict |
-|---|------|--------|---------|
-**Post-consolidation (9 tools, tested 2026-03-25):**
-
-| # | Tool | Status | Verdict |
-|---|------|--------|---------|
-| 1 | `get_overview` | **PASS** | Rich response: overview, 6 modules, 45 entry points |
-| 2 | `get_context` | **PASS** | Multi-target (file+module+symbol) resolved in one call. Docs, ownership, decisions, freshness all returned. |
-| 3 | `get_risk` | **PASS** | DSAPatternTable at 100% hotspot, 43 co-change partners, 26 dependents. Global hotspots exclude targets. |
-| 4 | `get_why` (3 modes) | **PASS** | Search mode finds keyword-matched decisions; path mode returns governing decisions; health mode shows dashboard |
-| 5 | `search_codebase` | **PASS** | Semantic search via LanceDB + Gemini. "payment system" → payment_service.py (score 6.72). FTS fallback also works. |
-| 6 | `get_dependency_path` | **PASS** | Correctly finds 1-hop path tiro_router → tiro/service |
-| 7 | `get_dead_code` | **PASS** | 1,232 findings, 48,780 deletable lines, sorted by confidence desc |
-| 8 | `get_architecture_diagram` | **PASS** | Repo scope returns pre-gen Mermaid; module scope generates dynamic |
-
-**Original 16-tool results (pre-consolidation, for reference):**
-
-| # | Tool | Status | Verdict |
-|---|------|--------|---------|
-| 1 | `get_overview` | **PASS** | Rich response: overview, 6 modules, 45 entry points |
-| 2 | `get_module_docs` | **PASS** | Returns full module page + 200+ child file listings |
-| 3 | `get_file_docs` | **PASS** | Returns page content, 11 symbols, 3 importers, confidence 1.0 |
-| 4 | `get_symbol` | **PASS** | Exact match works, returns `candidates` array for disambiguation |
-| 5 | `search_codebase` | **PASS** (was FAIL) | Fixed: FTS fallback + LanceDB + stop word stripping |
-| 6 | `get_architecture_diagram` | **PASS** | Repo scope returns pre-gen Mermaid; module scope generates dynamic |
-| 7 | `get_dependency_path` | **PASS** | Correctly finds 1-hop path tiro_router → tiro/service |
-| 8 | `get_stale_pages` | **PASS** | Empty (all pages fresh at 1.0) — correct behavior. Now part of `get_context` freshness. |
-| 9 | `get_file_history` | **PARTIAL** | Ownership, hotspot, commits work; co_change counts all 0 (fixed via key mismatch fix) |
-| 10 | `get_hotspots` | **PASS** | Returns top 5 hotspots with churn percentiles + stable files. Now part of `get_risk`. |
-| 11 | `get_codebase_ownership` | **PASS** | Correctly identifies 3 knowledge silos. Now part of `get_context`/`get_risk`. |
-| 12 | `get_co_changes` | **PASS** (was FAIL) | Fixed via co_change_count key mismatch fix (M10). Now part of `get_risk`. |
-| 13 | `get_dead_code` | **PASS** | Summary correct; default min_confidence lowered to 0.5. |
-| 14 | `get_decisions` | **PASS** | Returns all 23 proposed decisions with full context. Now part of `get_why`. |
-| 15 | `get_why` | **PASS** (was FAIL) | Fixed: include_proposed=True + real embedder + FTS fallback. Now 3-mode tool. |
-| 16 | `get_decision_health` | **PASS** | Shows 0 active, 23 proposed, 505 ungoverned hotspots. Now `get_why()` no-arg mode. |
-
-### Detailed Test Results
-
-#### Tool 1: `get_overview` — PASS
-
-**Test:** `get_overview()` (no args)
-**Result:** Returned structured dict with:
-- `title`: "Repository Overview: repo"
-- `content_md`: 2,500+ word project summary with tech stack, entry points, architecture
-- `architecture_diagram_mermaid`: Full Mermaid diagram with dependency flow
-- `key_modules`: 6 modules (backend, blogs, frontend, hire-backend, hire-frontend, scripts)
-- `entry_points`: 45 files (backend apps, frontend index.ts barrels, hire modules)
-
-**Quality:** Excellent. This gives an AI assistant immediate understanding of the
-entire codebase structure. The architecture diagram includes technical debt notes.
-
-**Issue:** Module descriptions truncated at 200 chars without word-boundary
-awareness (line 231). Minor.
-
----
-
-#### Tool 2: `get_module_docs` — PASS
-
-**Test:** `get_module_docs("frontend")`
-**Result:** Full module page content (68.7 KB response) with:
-- Comprehensive overview of Next.js architecture
-- Public API summary (dashboards, code execution, auth, analytics)
-- Architecture notes (framework, state management, data fetching, real-time)
-- 200+ child file page listings with paths and confidence scores
-
-**Quality:** Excellent. Partial match fallback also tested — works.
-
----
-
-#### Tool 3: `get_file_docs` — PASS
-
-**Test:** `get_file_docs("backend/services/tiro/service.py")`
-**Result:**
-- `content_md`: Accurate documentation of TiroService RAG engine
-- `symbols`: 11 methods (TiroService, stream_chat, generate_suggestions, etc.)
-- `imported_by`: 3 files (tiro_router, ai_interview_analytics, code_practice_analytics)
-- `confidence_score`: 1.0, `freshness_status`: "fresh"
-
-**Quality:** Very good. Symbol signatures include full parameter lists.
-
----
-
-#### Tool 4: `get_symbol` — PASS
-
-**Tests:**
-- `get_symbol("SupabaseService")` → exact match, file path, 20 importers, `candidates` shows hire-backend duplicate
-- `get_symbol("stream_chat", kind="method")` → exact match with full async signature
-
-**Quality:** Excellent. The `candidates` array for disambiguation when multiple
-symbols share a name is very useful for AI assistants.
-
----
-
-#### Tool 5: `search_codebase` — PASS (after fixes)
-
-**Test:** `search_codebase("how does the payment system work")`
-**Result (after fix):** 5 results, topped by `backend/services/payment_service.py` (score 6.72)
-
-**Original failures and fixes (2026-03-25):**
-1. `MockEmbedder` produced random vectors → Fixed by `_resolve_embedder()` reading config.yaml
-2. FTS fallback never triggered (vector store returned empty list, not exception) → Fixed: now
- falls back to FTS when vector results are empty
-3. FTS used exact phrase match for multi-word queries → Fixed: `_build_fts5_query()` strips
- stop words and joins terms with OR + prefix matching
-4. No LanceDB directory existed (init never persisted embeddings) → Fixed: `init_cmd.py` now
- tries LanceDB before InMemoryVectorStore; added `repowise reindex` command to embed
- existing pages without re-generating
-5. `lancedb` package not installed → Installed; 721 pages + 23 decisions indexed via Gemini embedder
-
-**Current status:** Semantic search via LanceDB + Gemini embeddings fully operational.
-FTS fallback also works for environments without LanceDB.
-
----
-
-#### Tool 6: `get_architecture_diagram` — PASS
-
-**Tests:**
-- `scope="repo"` → Returns pre-generated Mermaid diagram with dependency flow
-- `scope="module", path="backend"` → Generates dynamic Mermaid with ~50 edges
-
-**Quality:** Repo scope is excellent (human-curated by LLM). Module scope is
-functional but dense — the 50-edge cap produces a readable but incomplete view.
-
----
-
-#### Tool 7: `get_dependency_path` — PASS
-
-**Test:** `source="backend/routers/tiro_router.py", target="backend/services/tiro/service.py"`
-**Result:** Path found, distance 1, correct direct import relationship
-
-**Issue:** All relationships labeled "imports" regardless of actual type (no
-`edge_type` in DB). Performance concern: loads all 9,506 edges per call.
-
----
-
-#### Tool 8: `get_stale_pages` — PASS
-
-**Test:** `threshold=0.6`
-**Result:** `{"stale_pages": []}` — correct, all pages are fresh (confidence 1.0)
-
-This tool will become useful after `repowise update` runs and confidence decays.
-
----
-
-#### Tool 9: `get_file_history` — PARTIAL PASS
-
-**Test:** `file_path="frontend/components/problems/DSAPatternTable.tsx"`
-**Result:**
-- `age_days`: 161, `is_hotspot`: true, `commit_count_total`: 74, `commit_count_90d`: 51
-- `primary_owner`: swati510 (78%)
-- `significant_commits`: 10 commits with meaningful messages
-- `co_change_partners`: 43 partners listed BUT **all have `co_change_count: 0`**
-
-**Bug:** The co-change count data is not being persisted into `git_metadata.co_change_partners_json`
-correctly. The `findings.md` report shows real co-change counts (e.g., DSAPatternTable ↔
-technical_coaching_router: 8 times), but the JSON stored in the DB has counts of 0 for all
-partners. The partner *list* is correct (the right files are identified), but the
-*counts* are lost during persistence.
-
-**Impact:** MEDIUM — the co-change partner names are still useful, but without counts
-there's no way to rank them by coupling strength.
-
----
-
-#### Tool 10: `get_hotspots` — PASS
-
-**Test:** `limit=5, include_stable=true`
-**Result:**
-- 5 hotspots: DSAPatternTable (51 commits/90d, 99.95th percentile), dashboard, landing, sidebar, PurchaseGate
-- 5 stable files: streamingInterviewService (30 total, 0 recent), ResumeAnalyzer, SystemDesignCanvas, etc.
-
-**Quality:** Excellent. The churn percentile scoring makes results actionable.
-
----
-
-#### Tool 11: `get_codebase_ownership` — PASS
-
-**Test:** `by="module"`
-**Result:** 6 modules with ownership data:
-- `blogs`: 100% RaghavChamadiya → `is_silo: true`
-- `hire-backend`: 89% RaghavChamadiya → `is_silo: true`
-- `hire-frontend`: 96% RaghavChamadiya → `is_silo: true`
-- `backend`, `frontend`: split ownership → `is_silo: false`
-- `scripts`: no owner data
-
-**Quality:** Good. The silo detection is accurate and actionable.
-
----
-
-#### Tool 12: `get_co_changes` — FAIL
-
-**Test:** `file_path="backend/routers/pipecat_ai_interviewer_router.py"`
-**Result:** `{"co_change_partners": []}` — empty
-
-**Root cause:** Same as Tool 9 — co-change counts stored as 0, and the default
-`min_count=3` filter removes all partners. Even with `min_count=0`, the data
-would lack meaningful ranking.
-
-**Impact:** HIGH — this tool is designed for one of repowise's most valuable
-features (revealing hidden coupling not visible in imports). Without real counts,
-it's useless.
-
----
-
-#### Tool 13: `get_dead_code` — PASS (with caveat)
-
-**Tests:**
-- `kind="zombie_package", min_confidence=0.6` → `findings: []` (filtered out)
-- `kind="zombie_package", min_confidence=0` → 3 findings (scripts, .github, monitoring)
-
-**Issue:** Zombie package findings have confidence 0.5, which is below the default
-`min_confidence=0.6`. Users will miss them unless they explicitly lower the threshold.
-
-**Fix suggested:** Either raise zombie_package confidence to 0.7 or lower the
-default min_confidence to 0.5.
-
----
-
-#### Tool 14: `get_decisions` — PASS
-
-**Test:** `include_proposed=true`
-**Result:** 23 decisions with full context, rationale, consequences, affected files.
-Sources: 17 from readme_mining (conf 0.6), 6 from git_archaeology (conf 0.7).
-
-**Quality:** Good. The git_archaeology decisions include specific affected files
-(e.g., Pipecat upgrade lists 7 affected files). The readme_mining decisions have
-useful architectural context.
-
----
-
-#### Tool 15 (now part of `get_why`): `get_why` — PASS (after fixes)
-
-**Original failure:** `include_proposed=False` excluded all 23 decisions; MockEmbedder
-returned nothing; FTS fallback never triggered.
-
-**Fixes applied (2026-03-25):**
-- Changed to `include_proposed=True` so keyword search finds proposed decisions
-- Real embedder (Gemini) now used; LanceDB decision_records table populated via `repowise reindex`
-- FTS fallback improved (stop word stripping, OR-based matching)
-
-**Current status (consolidated `get_why` tool — 3 modes):**
-- `get_why("why is JWT used")` → mode="search", returns keyword-matched decisions. **PASS**
-- `get_why("src/auth/service.py")` → mode="path", returns decisions governing that file. **PASS**
-- `get_why()` → mode="health", returns dashboard with 23 proposed, 505 ungoverned. **PASS**
-
----
-
-#### Tool 16: `get_decision_health` — PASS
-
-**Test:** `get_decision_health()` (no args)
-**Result:**
-- `summary`: "0 active · 0 stale · 23 proposed · 505 ungoverned hotspots"
-- `proposed_awaiting_review`: First 10 of 23 decisions
-- `ungoverned_hotspots`: 15 high-churn files with no decision coverage
-
-**Quality:** Good. The "ungoverned hotspots" concept is valuable — it identifies
-files that are actively being changed but have no documented architectural decisions.
-
----
-
-## 8. Issues Found — Prioritized
-
-### CRITICAL (must fix)
-
-| # | Issue | Location | Description |
-|---|-------|----------|-------------|
-| C1 | **DB filename mismatch** (FIXED) | `mcp_server.py:64` | MCP server looked for `.repowise/repowise.db` but CLI creates `.repowise/wiki.db`. Fixed 2026-03-25: changed to `wiki.db`. |
-| C2 | **LanceDB filter injection** (FIXED) | `vector_store.py:240,274` | `f"page_id = '{page_id}'"` — string interpolation in LanceDB delete filter. Fixed 2026-03-25: single quotes in page_id are now escaped before interpolation. |
-| C3 | **Tool count mismatch in docs** (FIXED) | Multiple files | Updated "13 tools" → "16 tools" → "9 tools" (after consolidation) in mcp_server.py, ARCHITECTURE.md, server README. |
-
-### MAJOR (significant impact)
-
-| # | Issue | Location | Description |
-|---|-------|----------|-------------|
-| M1 | **MockEmbedder in MCP server** (FIXED) | `mcp_server.py:82-83` | Always created `MockEmbedder()`. Fixed 2026-03-25: added `_resolve_embedder()` that checks `REPOWISE_EMBEDDER` env var and `.repowise/config.yaml` for real embedder (gemini/openai). |
-| M2 | **CLI semantic search broken** (FIXED) | `search_cmd.py:60-69` | Created empty `InMemoryVectorStore`. Fixed 2026-03-25: now tries LanceDB from `.repowise/lancedb/` first, falls back to FTS. |
-| M3 | **`state.json` total_pages wrong** (FIXED) | `init_cmd.py:744` | `state["total_pages"] = len(generated_pages)` only counts last job. Fixed 2026-03-25: now queries actual DB page count via `SELECT COUNT(*)` after persistence. |
-| M4 | **Missing `edge_type` in graph DB** (FIXED) | `persistence/models.py` | `graph_edges` table had no `edge_type` column. Fixed 2026-03-25: added `edge_type` column (Alembic migration 0004), CRUD layer, init_cmd persistence, and MCP server now uses real edge types in `get_dependency_path`. |
-| M5 | **Missing `repo_path` in update** (FIXED) | `update_cmd.py:87` | `resolve_provider(provider_name, model)` was missing `repo_path`. Fixed 2026-03-25: now passes `repo_path=repo_path`. |
-| M6 | ~~`get_repo_structure()` arg mismatch~~ (NOT A BUG) | `update_cmd.py:97` | The `files` parameter is optional (`files: list[FileInfo] | None = None`). Calling without args triggers a fresh traversal. |
-| M7 | **`repowise-server` not auto-installed** (FIXED) | `packages/cli/pyproject.toml` | Fixed 2026-03-25: added `repowise-server` as explicit dependency of `repowise-cli` with workspace source. |
-| M8 | **`completed_pages > total_pages`** (FIXED) | `job_system.py` | Fixed 2026-03-25: `complete_page()` now clamps `total_pages = max(total_pages, completed_pages)`. |
-| M9 | **`search_codebase` returns empty** (FIXED) | `mcp_server.py`, `search.py`, `init_cmd.py` | Fixed: (a) FTS fallback triggers on empty results not just exceptions, (b) FTS query builder strips stop words and uses OR + prefix matching, (c) `init_cmd.py` now uses LanceDB when available, (d) added `repowise reindex` command. LanceDB indexed with 721 pages + 23 decisions via Gemini embedder. |
-| M10 | **Co-change counts all zero** (FIXED) | `mcp_server.py` | Root cause: key mismatch — data stores `"co_change_count"` but MCP tools read `"count"`. Fixed 2026-03-25: tools now check both keys via `p.get("co_change_count", p.get("count", 0))`. |
-| M11 | **`get_why` excludes all decisions** (FIXED) | `mcp_server.py:1155-1156` | `include_proposed=False` returned 0 results. Fixed 2026-03-25: changed to `include_proposed=True`. |
-
-### MINOR (code quality / polish)
-
-| # | Issue | Location | Description |
-|---|-------|----------|-------------|
-| m1 | **HTML export doesn't render markdown** (FIXED) | `export_cmd.py:118-127` | Wrapped content in `<pre>`. Fixed 2026-03-25: now renders via `markdown` or `mistune` library, falls back to `<pre>`. |
-| m2 | Failed page IDs are level names | Job JSON files | `"level-1"` used as failed page ID instead of actual page identifier. Investigation shows `fail_page()` receives actual page IDs — the "level name" was a display callback, not a data bug. |
-| m3 | **Module description truncation** (FIXED) | `mcp_server.py:293` | Fixed 2026-03-25: added parentheses to fix operator precedence bug (was duplicating content), and added `rsplit(" ", 1)[0]` for word-boundary awareness. |
-| m4 | Graph nodes only `file` type | DB audit | Architecture describes symbol/package/external nodes but only file nodes are persisted. |
-| m5 | **No startup validation in MCP** (FIXED) | `mcp_server.py:50-108` | Server started silently. Fixed 2026-03-25: added warning logs when `.repowise/` or `wiki.db` is missing. |
-| m6 | **`click` import at bottom of file** (FIXED) | `helpers.py:257` | Fixed 2026-03-25: moved `import click` to top of file with other imports. |
-| m7 | Ownership grouping too coarse | `mcp_server.py:858` | Groups by first directory only — too coarse for nested monorepos. |
-| m8 | **Architecture diagram edge limit** (FIXED) | `mcp_server.py` | Fixed 2026-03-25: edges now sorted by source node PageRank before 50-edge cap, plus Mermaid node ID sanitization improved to use regex for all non-alphanumeric chars. |
-| m9 | All decisions are "proposed" | DB content | No inline markers found — all 23 are auto-extracted. Expected but worth noting. |
-| m10 | **`_decision_store` may be empty InMemory** (FIXED) | `mcp_server.py:103-104` | Fixed: `repowise reindex` now indexes 23 decision records into LanceDB `decision_records` table. MCP server loads this on startup when `.repowise/lancedb/` exists. |
-| m11 | **`get_dead_code` default hides zombies** (FIXED) | `mcp_server.py:982` | Default `min_confidence` was 0.6, hiding 0.5-confidence zombie findings. Fixed 2026-03-25: lowered default to 0.5. |
-| m12 | **`repowise-server` not installed by default** (FIXED) | `packages/cli/pyproject.toml` | Fixed 2026-03-25: same as M7 — `repowise-server` is now a declared dependency of `repowise-cli`. |
-
----
-
-## 9. Core Engine Code Review
-
-### Ingestion Pipeline
-
-| Component | Status | Notes |
-|-----------|--------|-------|
-| `FileTraverser` | Good | Multi-layer exclusion, binary detection, monorepo detection |
-| `ASTParser` | Good | Single class for all languages via tree-sitter queries |
-| `GraphBuilder` | Good | PageRank, betweenness, SCC, community detection |
-| `ChangeDetector` | Good | Symbol rename detection, cascade budget |
-| `GitIndexer` | Good | Parallel git log mining, co-change analysis |
-| `SpecialHandlers` | Good | OpenAPI, Dockerfile, CI YAML, Makefile, Protobuf, GraphQL |
-
-### Generation Engine
-
-| Component | Status | Notes |
-|-----------|--------|-------|
-| `PageGenerator` | Good | Hierarchical 8-level generation order |
-| `ContextAssembler` | Excellent | 9-priority context assembly with token budgeting |
-| `JobSystem` | Adequate | JSON-based checkpointing, not database-backed |
-
-### Persistence Layer
-
-| Component | Status | Notes |
-|-----------|--------|-------|
-| `models.py` (ORM) | Good | Complete schema for all entities |
-| `crud.py` | Good | Async CRUD with batch operations |
-| `database.py` | Good | SQLite + PostgreSQL abstraction |
-| `search.py` (FTS) | Good | SQLite FTS5 integration |
-| `vector_store.py` | **Issues** | LanceDB filter injection (C1), mock embedder concerns |
-| `embedder.py` | Good | Clean abstraction with Gemini + OpenAI implementations |
-
-### Provider Layer
-
-| Component | Status | Notes |
-|-----------|--------|-------|
-| `base.py` | Good | Clean abstract interface |
-| `registry.py` | Good | Dynamic provider registration |
-| `anthropic.py` | Good | Batch API + prompt caching support |
-| `openai.py` | Good | Standard implementation |
-| `gemini.py` | Good | Google GenAI SDK integration |
-| `ollama.py` | Good | Local model support |
-| `litellm.py` | Good | 100+ provider support via LiteLLM |
-| `mock.py` | Good | Test double |
-| `rate_limiter.py` | Adequate | Token-bucket with backoff, potential starvation risk |
-
-### Analysis
-
-| Component | Status | Notes |
-|-----------|--------|-------|
-| `dead_code.py` | Good | Graph-based detection, conservative confidence |
-| `decision_extractor.py` | Good | 4 extraction sources, staleness tracking |
-
----
-
-## 10. Recommendations
-
-### Immediate Fixes (before next release)
-
-1. ~~**Fix LanceDB filter injection**~~ — DONE (C2)
-2. ~~**Fix MCP embedder**~~ — DONE (M1)
-3. ~~**Fix `update_cmd.py`**~~ — DONE (M5)
-4. ~~**Install repowise-server in Makefile**~~ — DONE (M7): added as dependency in `packages/cli/pyproject.toml`
-5. ~~**Update tool count**~~ — DONE (C3): updated to 9 tools everywhere
-
-### Short-term Improvements
-
-6. ~~**Add `edge_type` to `graph_edges`**~~ — DONE (M4): Alembic migration 0004, CRUD, init_cmd, MCP server
-7. **Persist non-file graph nodes** — symbols, packages, external nodes (still open)
-8. ~~**Fix `total_pages` tracking**~~ — DONE (M3): now queries DB count
-9. ~~**Fix CLI semantic search**~~ — DONE (M2)
-10. ~~**Fix HTML export**~~ — DONE (m1)
-
-### Architecture Improvements (longer term)
-
-11. **Lazy graph loading in MCP** — don't load all edges per `get_dependency_path` call
-12. **Configurable ownership depth** — support nested module grouping (m7 still open)
-13. ~~**MCP startup validation**~~ — DONE (m5)
-14. **Database-backed job system** — replace JSON checkpointing with SQL
-15. **Rate limiter starvation guard** — add max-wait timeout
-
-### Additional fixes applied 2026-03-25 (second pass)
-
-- **`search_codebase` page_type filter** — over-fetch (3x limit) when filtering by page_type to avoid returning 0 results
-- **`search_codebase` confidence_score** — batch-lookup `Page.confidence` from DB instead of always returning None
-- **`get_dependency_path` error consistency** — all error cases now return `{"path": [], "distance": -1, "explanation": "..."}`
-- **`get_dead_code` docstring** — corrected default from 0.6 to 0.5
-- **`get_dead_code` safe_only** — moved filter to SQL WHERE clause for efficiency
-- **`get_context` module LIKE prefix** — added trailing `/` to prevent `src/auth` matching `src/auth_helpers`
-- **`get_overview` operator precedence** — fixed content duplication bug with parentheses + word-boundary truncation
-- **Architecture diagram sanitization** — regex-based `_sanitize_mermaid_id()` replaces all non-alphanumeric chars
-- **Architecture diagram prioritization** — edges sorted by source node PageRank before 50-edge cap
-- **`completed_pages > total_pages`** — `complete_page()` now clamps total to max(total, completed)
-- **6 pre-existing test failures fixed** — dead_code, git_indexer, parser, cost_estimator, registry, gemini_provider tests
-
----
-
-## Appendix: Test Data Summary
-
-### interview-coach Repo Statistics
-
-| Metric | Value |
-|--------|-------|
-| Total files indexed | 3,557 |
-| Dependency edges | 8,080 (graph) / 9,506 (DB) |
-| Symbols parsed | 12,807 (findings) / 12,955 (DB) |
-| Git commits analyzed | 500 |
-| Files with git metadata | 2,022 |
-| Contributors | 2 (RaghavChamadiya, swati510) |
-| Languages | TypeScript (1,144), Python (802), Markdown (682), JSON (168), SQL (114), JS (37) |
-| Hotspot files | ~15 files with >10 commits in 90 days |
-| Circular dependencies | 22 SCCs detected |
-| Dead code findings | 1,233 (1,230 unreachable + 3 zombie packages) |
-| Decision records | 23 (all proposed) |
-| Wiki pages generated | 721 |
-| Total tokens used | 4,223,332 |
-
-### Top Hotspot Files
-
-| File | 90d Commits | Owner |
-|------|------------|-------|
-| `frontend/components/problems/DSAPatternTable.tsx` | 51 | swati510 |
-| `frontend/app/dashboard/page.tsx` | 20 | swati510 |
-| `frontend/app/page.tsx` | 18 | swati510 |
-| `frontend/components/layout/TechnicalCoachingSidebar.tsx` | 17 | swati510 |
-| `frontend/app/layout.tsx` | 15 | RaghavChamadiya |
diff --git a/docs/MCP_TOOLS_TEST_REPORT.md b/docs/MCP_TOOLS_TEST_REPORT.md
deleted file mode 100644
index 1d1fe2e21..000000000
--- a/docs/MCP_TOOLS_TEST_REPORT.md
+++ /dev/null
@@ -1,332 +0,0 @@
-# MCP Tools Test Report
-
-**Date:** 2026-03-26
-**Tested on:** repowise repo (self-hosted, 315 files, 52k+ LOC)
-**Server transport:** stdio
-**Client:** Claude Code (Opus 4.6)
-
----
-
-## Executive Summary
-
-All 8 MCP tools are functional. 6 of 8 return useful, accurate results. `get_why` has a relevance problem — it returns decisions unrelated to the query. `search_codebase` returns results for nonsense queries with no indication they're low-quality matches. No tool crashed, but `get_why` experienced hangs in 2 of 4 test runs (file-path mode and health dashboard mode).
-
----
-
-## Tool-by-Tool Results
-
-### 1. `get_overview`
-
-**Status:** Working
-**Response time:** Fast (~2s)
-
-**What it returned:**
-- Project summary with accurate LOC count (52k) and file count (315)
-- Correct tech stack breakdown (Python 50.2%, TypeScript 32.7%)
-- Entry points (CLI main, server app, test fixtures)
-- Architecture description with module breakdown
-- Mermaid diagram (CLI -> Core -> Server dependency graph)
-- Git health: 285 indexed files, 71 hotspots, bus factor 0.8, churn trend increasing
-
-**Quality assessment:**
-- Summary is accurate and well-structured
-- Correctly identifies the monorepo structure (core, server, cli, web)
-- Mentions the circular dependency between crud/database/models/mcp_server — this is genuinely useful context
-- Top churn modules list is accurate
-
-**Issues:**
-- Entry points include test fixtures (`go_pkg/main.go`, `rust_pkg/src/lib.rs`) — these aren't real entry points, they're test data. A developer following this advice would start in the wrong place.
-- The `packages/web` module is listed as a top churn module but the architecture section barely mentions it. Mismatch between git signals and documentation depth.
-
-**Improvement ideas:**
-- Filter test fixtures from entry points. An entry point should be something a developer actually runs, not test data.
-- Weight the architecture summary toward high-churn areas — if `packages/web` is churning heavily, the overview should say more about it.
-
----
-
-### 2. `get_context`
-
-**Status:** Working
-**Response time:** Fast (~2s)
-
-**Tests run:**
-| Input | Result |
-|-------|--------|
-| Single file (`main.py`) | Returned docs, symbols, importers, last_change |
-| Multi-file (`models.py` + `parser.py`) | Both returned correctly in one call |
-| `include` filter (`docs`, `ownership`, `freshness`) | Correctly filtered response fields |
-| Nonexistent file (`NonExistentFile.py`) | Clean error: `"Target not found: 'NonExistentFile.py'"` |
-
-**Quality assessment:**
-- Documentation quality is high — the `parser.py` docs accurately describe the tree-sitter architecture, language config system, and even how to add a new language
-- Symbol extraction is thorough — `parser.py` returned 35 symbols with correct signatures
-- `imported_by` lists are accurate and useful for understanding coupling
-- Ownership data works (`models.py` shows RaghavChamadiya at 100%)
-- Freshness status works (`fresh` / `not stale`)
-
-**Issues:**
-- `parser.py` shows `primary_owner: null`, `contributor_count: 0`, `bus_factor: 0` — but the file clearly has commit history. Likely a git indexing gap where some files don't get ownership attributed.
-- `last_change` returned `date: null, author: null, days_ago: 0` for `main.py` — this should have real data.
-
-**Improvement ideas:**
-- Investigate why some files have null ownership despite having git history. This is a data quality issue that undermines the `get_risk` tool too (it showed `owner: unknown` for files that definitely have an owner).
-- `last_change` with all nulls is confusing — either return the actual data or omit the field. A null date with `days_ago: 0` is misleading (implies "changed today").
-
----
-
-### 3. `search_codebase`
-
-**Status:** Working
-**Response time:** Fast (~1s)
-
-**Tests run:**
-| Query | Results | Quality |
-|-------|---------|---------|
-| `"how does the MCP server work?"` | mcp_cmd.py, scc-38 circular dep, architecture diagram | Good — top result is exactly right |
-| `"authentication and API keys"` (filtered to file_page) | providers.py, openai.py, deps.py | Good — relevant files about provider/API key management |
-| `"xyznonexistent gibberish query"` | search.py, search_cmd.py, GitMetadata | Bad — returned results for nonsense with no low-confidence signal |
-
-**Quality assessment:**
-- Semantic search works well for real queries — results are relevant and properly ranked
-- `page_type` filter works correctly
-- Relevance scores are present and ordered correctly
-- Snippets are useful for quick scanning
-
-**Issues:**
-- **No empty results for nonsense queries.** The gibberish query returned 3 results with relevance scores of 2.4–3.4. There's no threshold or indicator telling the consumer "these results are probably not what you're looking for." An AI agent would treat these as real answers.
-- `confidence_score` is always `1.0` across every result in every query. This field appears to be hardcoded and provides no actual signal.
-
-**Improvement ideas:**
-- Add a minimum relevance threshold. If the top result scores below some baseline, return an empty result set or add a `low_confidence: true` flag.
-- Make `confidence_score` meaningful or remove it. A field that's always 1.0 is noise.
-- Consider returning a `"no_relevant_results": true` flag when the best match score is below threshold.
-
----
-
-### 4. `get_risk`
-
-**Status:** Working
-**Response time:** Fast (~2s)
-
-**Tests run:**
-| Input | Key findings |
-|-------|-------------|
-| `mcp_server.py` | 99% hotspot, churn-heavy, increasing, 2 co-change partners, 2414 lines added in 90d |
-| `ui.py` + `git_indexer.py` (multi-target) | ui.py: 5% stable; git_indexer: 12% high-coupling, 3 co-change partners |
-| `nonexistent/file.py` | Clean fallback: 0% hotspot, "no git metadata available" |
-
-**Quality assessment:**
-- **This is the strongest tool.** The data it returns is genuinely not derivable from reading source code.
-- Hotspot scores are well-calibrated — `mcp_server.py` at 99% makes sense given the heavy recent development.
-- Co-change partner data is actionable: knowing that `git_indexer.py` always changes with `init_cmd.py` and `models.py` helps predict blast radius.
-- `risk_type` classification (churn-heavy, high-coupling, stable) provides quick triage.
-- `impact_surface` showing the top downstream files by PageRank is useful.
-- Global hotspots list provides ambient awareness regardless of what you query.
-- Error handling for nonexistent files is clean.
-
-**Issues:**
-- `ui.py` shows `primary_owner: "unknown"` with `owner_pct: 0.0` — same null ownership issue as `get_context`. For risk assessment, knowing who owns a file is critical.
-- `change_magnitude` shows `lines_added_90d: 0` for `git_indexer.py` — but this file was modified recently (commit `f749ca9` from this week). The 90-day window calculation may be off, or git indexing didn't capture recent commits.
-
-**Improvement ideas:**
-- Fix the ownership attribution gap. This is the #1 data quality issue across multiple tools.
-- Verify that `change_magnitude` windows align with actual git history. A file committed this week showing 0 lines added in 90 days is wrong.
-
----
-
-### 5. `get_why`
-
-**Status:** Partially working (reliability + relevance issues)
-**Response time:** Variable — fast when it works, hangs in some modes
-
-**Tests run:**
-| Input | Result |
-|-------|--------|
-| No args (health dashboard) | **Hung — had to cancel** |
-| File path (`crud.py`) | **Hung — had to cancel** |
-| NL query: `"why is the persistence layer using SQLite?"` | Returned 2 decisions + 3 related docs |
-| NL query: `"why does the parser use tree-sitter?"` | Returned 8 decisions + 3 related docs |
-
-**Quality assessment (when it works):**
-- The `related_documentation` section works well — the tree-sitter query correctly surfaced `parser.py` as the top result with a high relevance score (16.9).
-- The concept is valuable — surfacing architectural decisions that govern a piece of code is something no other tool does.
-
-**However, the decisions themselves have a serious relevance problem:**
-- Asked "why does the parser use tree-sitter?" → got decisions about Jinja2 templates, SQLite, streaming protocol, API key resolution order, token estimation heuristics. **None of these are about tree-sitter or the parser.**
-- Asked "why is the persistence layer using SQLite?" → got a decision about Gemini's agentic loop. Not relevant.
-- All decisions show `confidence: 0.6` and `source: "readme_mining"` — suggests these are mined from docs/READMEs with no filtering by actual relevance to the query.
-- All decisions have `affected_files: []` — without file associations, the system can't filter decisions by what actually governs the queried code.
-
-**Issues:**
-1. **Hangs in file-path mode and health dashboard mode.** 2 of 4 calls had to be cancelled by the user. This is a blocker for production use.
-2. **Decision relevance is poor.** The decisions returned don't match the query. It appears to return all known decisions rather than filtering to relevant ones.
-3. **No `affected_files` populated.** This makes target-aware mode (`get_why` with both query and targets) likely non-functional since there's nothing to match against.
-4. **All decisions are `"proposed"` status with 0.6 confidence.** No differentiation between strong architectural decisions and speculative observations.
-
-**Improvement ideas:**
-- **P0: Fix the hangs.** The file-path and dashboard modes need to complete or timeout gracefully.
-- **P0: Filter decisions by relevance.** If the query is about tree-sitter, don't return decisions about Jinja2. Use the semantic search scores to threshold.
-- **P1: Populate `affected_files`.** Without this, the tool can't answer "what decisions govern this file?" — which is its core value proposition.
-- **P2: Differentiate decision confidence.** A decision extracted from an explicit ADR should score higher than one mined from a README sentence.
-
----
-
-### 6. `get_dependency_path`
-
-**Status:** Working
-**Response time:** Fast (~1s)
-
-**Tests run:**
-| Source | Target | Result |
-|--------|--------|--------|
-| `cli/main.py` | `persistence/crud.py` | Path found: main.py -> init_cmd.py -> crud.py (2 hops) |
-| `cli/main.py` | `web/sidebar.tsx` | No direct path — returned visual context |
-
-**Quality assessment:**
-- Direct path finding works correctly — the 2-hop chain through `init_cmd.py` is accurate.
-- **The "no path found" response is excellent.** Instead of just saying "no path," it returns:
- - Nearest common ancestors with distances
- - Community analysis (source in community 26, target in community 91)
- - Bridge suggestions with PageRank scores
- - Human-readable suggestion: "both nodes connect through reindex_cmd.py"
-- This makes the tool useful even when there's no direct dependency — it helps understand architectural silos.
-
-**Issues:**
-- The bridge suggestion points to `reindex_cmd.py` as the connection between CLI and web — this seems unlikely to be useful as an architectural insight. The CLI and web packages are intentionally separate.
-- `external:@/lib/utils/cn` appears as a bridge suggestion with a high PageRank. External/alias nodes in the graph could be confusing.
-
-**Improvement ideas:**
-- When source and target are in completely different packages (Python CLI vs TypeScript web), consider saying "these are separate packages with no intended dependency path" rather than suggesting bridges. Not every pair of files should be connected.
-- Filter or label external/alias nodes in suggestions.
-
----
-
-### 7. `get_dead_code`
-
-**Status:** Working
-**Response time:** Fast (~2s)
-
-**Tests run:**
-| Parameters | Results |
-|------------|---------|
-| `tier: "high"` | 0 findings (none at high confidence) |
-| `group_by: "directory"` | 21 medium-confidence findings across 11 directories |
-
-**Quality assessment:**
-- Summary stats are useful: 128 total findings, 420 deletable lines, 15 safe to delete.
-- `kind` breakdown shows categories: 122 unreachable files, 6 zombie packages.
-- Directory rollup is actionable: `tests/fixtures` has 4 findings (180 lines), `packages/core` has 8 (170 lines).
-- `safe_to_delete` flag is conservative — good for production use.
-- Individual findings include file path, confidence, reason, line count.
-
-**Issues:**
-- Test fixtures (`tests/fixtures/sample_repo/go_pkg/calculator/calculator.go`) are flagged as dead code with `reason: "File has no importers"`. These are test data files — they're not supposed to have importers. Same for `conftest.py` which is auto-loaded by pytest.
-- `alembic/env.py` is flagged as dead — it's loaded by the Alembic migration framework, not imported. Framework-loaded files need special handling.
-- All findings are medium confidence (0.7) with identical reasons. No differentiation.
-
-**Improvement ideas:**
-- Exclude known framework files: `conftest.py` (pytest), `alembic/env.py` (alembic), `__main__.py`, etc. These are entry points loaded by tools, not dead code.
-- Exclude test fixture directories — or at least add a `is_test_fixture: true` flag so consumers can filter.
-- Vary confidence scores based on additional signals (age, ownership, last commit date) rather than a flat 0.7.
-
----
-
-### 8. `get_architecture_diagram`
-
-**Status:** Working
-**Response time:** Fast (~2s)
-
-**Tests run:**
-| Scope | Parameters | Result |
-|-------|------------|--------|
-| `repo` | Default | Markdown architecture doc with embedded mermaid |
-| `module` | `packages/core`, `show_heat: true` | Detailed flowchart with churn heat annotations |
-
-**Quality assessment:**
-- Repo-level returns a well-structured architecture document (not just a diagram) — includes component descriptions, key modules, and the SCC note.
-- Module-level diagram correctly shows all files in `packages/core` with their import relationships.
-- Churn heat annotations work: `models.py` and `mcp_server.py` marked `hot`, `rust_pkg/src/models.rs` marked `warm`, most others `cold`. Matches `get_risk` data.
-- Mermaid syntax is valid and renderable.
-
-**Issues:**
-- Repo-level scope returns the pre-generated architecture page (markdown + embedded mermaid), not a standalone diagram. This is fine but inconsistent with the module-level scope which returns pure mermaid.
-- Module-level diagram for `packages/core` includes `packages/server/src/repowise/server/mcp_server.py` and `tests/fixtures/sample_repo/rust_pkg/src/models.rs` — these are outside `packages/core`. The diagram leaks across module boundaries because it follows imports.
-- Node IDs in mermaid are very long (`packages_core_src_repowise_core_persistence_models_py`) making the rendered diagram hard to read.
-
-**Improvement ideas:**
-- Use short labels in the diagram (e.g., `models.py` or `persistence/models.py`) while keeping full paths in tooltips or node metadata.
-- For module-scope diagrams, consider clipping edges at the module boundary with an "external" node rather than including the full external file.
-- Make repo-level and module-level return consistent formats.
-
----
-
-## Cross-Cutting Issues
-
-### 1. Ownership data gaps
-Multiple tools return `null` or `"unknown"` for file ownership on files that definitely have git history. This affects `get_context`, `get_risk`, and `get_dead_code`. **Root cause likely in git indexer** — some files may timeout during indexing or the ownership attribution logic has gaps.
-
-**Impact:** High. Ownership is a key signal for risk assessment, code review routing, and dead code cleanup prioritization.
-
-### 2. `confidence_score` is meaningless
-In `search_codebase`, every result has `confidence_score: 1.0`. In `get_why`, every decision has `confidence: 0.6`. In `get_dead_code`, every finding has `confidence: 0.7`. These flat values provide no signal for consumers to differentiate quality.
-
-**Impact:** Medium. AI agents consuming these tools have no way to judge result quality, leading to hallucination amplification — the agent treats a bad match the same as a good one.
-
-### 3. Test fixtures pollute results
-Test fixtures appear as entry points (`get_overview`), dead code (`get_dead_code`), and in dependency graphs (`get_architecture_diagram`). For a developer using these tools, test data showing up in architecture overviews is confusing.
-
-**Impact:** Low-medium. Mostly a noise issue, but for public repos with large test suites this could significantly pollute results.
-
-### 4. Error handling is solid
-Every tool handles nonexistent inputs gracefully — no crashes, clean error messages, fallback data where appropriate. `get_dependency_path` is particularly good, providing visual context even when no path exists.
-
----
-
-## Reliability Summary
-
-| Tool | Functional | Response Time | Data Quality | Reliability |
-|------|-----------|---------------|-------------|-------------|
-| `get_overview` | Yes | Fast | Good (entry point issue) | Stable |
-| `get_context` | Yes | Fast | Good (ownership gaps) | Stable |
-| `search_codebase` | Yes | Fast | Good (no empty results for garbage) | Stable |
-| `get_risk` | Yes | Fast | Good (ownership gaps, stale magnitude) | Stable |
-| `get_why` | Partial | Variable | Poor (irrelevant decisions) | **Unstable — hangs** |
-| `get_dependency_path` | Yes | Fast | Good | Stable |
-| `get_dead_code` | Yes | Fast | Okay (false positives on fixtures) | Stable |
-| `get_architecture_diagram` | Yes | Fast | Good (label readability) | Stable |
-
----
-
-## Priority Fixes
-
-### P0 (Blocking for production)
-1. **`get_why` hangs** — file-path mode and health dashboard mode hang indefinitely. Must either fix or add timeouts.
-2. **`get_why` decision relevance** — decisions returned are unrelated to queries. The tool's core promise is broken without this.
-
-### P1 (Important for quality)
-3. **Ownership attribution gaps** — files with git history showing `null` owner across multiple tools.
-4. **Search confidence scoring** — nonsense queries should return empty results or be flagged as low-confidence.
-5. **`affected_files` in decisions** — empty across all decisions, preventing target-aware decision lookup.
-
-### P2 (Polish)
-6. **Exclude test fixtures** from entry points, dead code findings, and architecture diagrams.
-7. **Exclude framework-loaded files** (conftest.py, alembic/env.py) from dead code.
-8. **Shorter node labels** in architecture diagrams for readability.
-9. **Consistent response formats** between repo-level and module-level diagrams.
-10. **Meaningful confidence scores** — vary based on actual signal strength rather than flat values.
-
----
-
-## For Public Repo Demos (React, FastAPI, etc.)
-
-The tools that will demo well today:
-- **`get_overview`** — instant architecture summary of a massive codebase
-- **`get_risk`** — "what are the riskiest files in React?" is a compelling demo
-- **`get_context`** — deep file documentation with ownership and import graphs
-- **`get_dependency_path`** — "how does the scheduler connect to the reconciler?"
-- **`get_architecture_diagram`** with heat — visual, immediately understandable
-
-The tools that need fixes before public demos:
-- **`get_why`** — hanging will kill a live demo. Decision relevance issues will confuse the audience.
-- **`search_codebase`** — works, but a nonsense query returning results in a demo would raise eyebrows.
-- **`get_dead_code`** — flagging React's test fixtures as dead code would undermine credibility. For a repo with thousands of test files, false positives would dominate.
diff --git a/docs/PHASE_5_5_IMPLEMENTATION.md b/docs/PHASE_5_5_IMPLEMENTATION.md
deleted file mode 100644
index f8b805cd7..000000000
--- a/docs/PHASE_5_5_IMPLEMENTATION.md
+++ /dev/null
@@ -1,944 +0,0 @@
-# Phase 5.5 — Git Intelligence + Dead Code Detection: Implementation Prompt
-
-This document contains everything needed to implement Phase 5.5 of repowise.
-Read the full document before writing any code.
-
-**Context:** Phases 1-5 are complete and passing (473 tests, 11.48s). Phase 5.5 is
-a standalone feature detour before Phase 6 (Server). All additions are additive —
-do not modify any existing Phase 1-5 code, only extend it.
-
-**Important files to read first:**
-- `docs/ARCHITECTURE.md` — sections 7 (Git Intelligence) and 8 (Dead Code Detection)
-- `BUILD_STATUS.md` — Phase 5.5 step breakdown
-- `PLAN.md` — full project plan with git/dead-code integrated
-
----
-
-## Codebase Layout (What Already Exists)
-
-```
-packages/core/src/repowise/core/
-├── generation/
-│ ├── context_assembler.py # ContextAssembler class
-│ ├── page_generator.py # PageGenerator class (8-level pipeline)
-│ ├── models.py # GeneratedPage, GenerationConfig, confidence logic
-│ ├── job_system.py # JobSystem + Checkpoint
-│ └── templates/ # 9 Jinja2 .j2 templates
-├── ingestion/
-│ ├── graph.py # GraphBuilder (NetworkX)
-│ ├── change_detector.py # ChangeDetector, FileDiff, AffectedPages
-│ ├── models.py # ParsedFile, Symbol, FileInfo, RepoStructure
-│ ├── parser.py # ASTParser
-│ └── traverser.py # FileTraverser (extra_exclude_patterns + per-dir .repowiseIgnore)
-├── persistence/
-│ ├── models.py # SQLAlchemy ORM: Repository, Page, GraphNode, etc.
-│ ├── database.py # Engine/session creation
-│ ├── crud.py # Async CRUD operations
-│ ├── search.py # Full-text search
-│ └── vector_store.py # Vector storage
-└── providers/ # LLM providers (unchanged)
-
-packages/cli/src/repowise/cli/
-├── commands/
-│ ├── init_cmd.py # repowise init pipeline
-│ ├── update_cmd.py # repowise update pipeline
-│ └── ...
-├── helpers.py
-├── cost_estimator.py
-└── main.py # Click group + command registrations
-```
-
----
-
-## Existing Key Interfaces (Exact Signatures)
-
-### persistence/models.py
-
-```python
-# Imports
-from datetime import datetime, timezone
-from uuid import uuid4
-from sqlalchemy import (Boolean, DateTime, Float, ForeignKey, Integer, String, Text, UniqueConstraint)
-from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
-
-# Helper functions used throughout
-def _new_uuid() -> str: return uuid4().hex
-def _now_utc() -> datetime: return datetime.now(timezone.utc)
-
-class Base(DeclarativeBase): pass
-
-# Existing tables: Repository, GenerationJob, Page, PageVersion, GraphNode,
-# GraphEdge, WebhookEvent, WikiSymbol (all use Mapped[] type hints)
-```
-
-### generation/models.py
-
-```python
-from dataclasses import dataclass, field
-from typing import Literal
-
-PageType = Literal["api_contract", "symbol_spotlight", "file_page", "scc_page",
- "module_page", "cross_package", "repo_overview",
- "architecture_diagram", "infra_page", "diff_summary"]
-FreshnessStatus = Literal["fresh", "stale", "expired", "unknown"]
-
-@dataclass(frozen=True)
-class GenerationConfig:
- max_tokens: int = 4096
- temperature: float = 0.3
- token_budget: int = 8000
- max_concurrency: int = 5
- cache_enabled: bool = True
- staleness_threshold_days: int = 7
- expiry_threshold_days: int = 30
- top_symbol_percentile: float = 0.10
- file_page_top_percentile: float = 0.20
- file_page_min_symbols: int = 1
- jobs_dir: str = ".repowise/jobs"
-
-# Confidence: linear decay based on days since update
-def compute_freshness(...) -> str: ...
-def decay_confidence(...) -> ConfidenceDecayResult: ...
-```
-
-### generation/context_assembler.py
-
-```python
-@dataclass
-class FilePageContext:
- file_path: str
- language: str
- docstring: str | None
- symbols: list[dict[str, Any]]
- imports: list[str]
- exports: list[str]
- file_source_snippet: str
- pagerank_score: float
- betweenness_score: float
- community_id: int
- dependents: list[str]
- dependencies: list[str]
- is_api_contract: bool
- is_entry_point: bool
- is_test: bool
- parse_errors: list[str]
- estimated_tokens: int
- rag_context: list[str] = field(default_factory=list)
-
-class ContextAssembler:
- def __init__(self, config: GenerationConfig) -> None: ...
- def assemble_file_page(self, parsed, graph, pagerank, betweenness, community, source_bytes) -> FilePageContext: ...
- def assemble_module_page(self, module_path, language, file_contexts, graph) -> ModulePageContext: ...
- def assemble_repo_overview(self, repo_structure, pagerank, sccs, community) -> RepoOverviewContext: ...
- def assemble_diff_summary(self, file_diffs, affected_pages, from_ref, to_ref) -> DiffSummaryContext: ...
- # ... other assemble_* methods
-```
-
-### generation/page_generator.py
-
-```python
-class PageGenerator:
- def __init__(self, provider, assembler, config, jinja_env=None): ...
-
- async def generate_all(self, parsed_files, source_map, graph_builder, repo_structure, repo_name, job_system=None) -> list[GeneratedPage]:
- # Levels 0-7 run sequentially. Within each level:
- # - Selects files/symbols for that level
- # - Uses asyncio.Semaphore(config.max_concurrency)
- # - Level 2 (file pages): filters by _is_significant_file(), groups by module
- ...
-```
-
-### ingestion/graph.py
-
-```python
-class GraphBuilder:
- def __init__(self) -> None: ...
- def add_file(self, parsed: ParsedFile) -> None: ...
- def build(self) -> nx.DiGraph: ...
- def graph(self) -> nx.DiGraph: ...
- def pagerank(self, alpha=0.85) -> dict[str, float]: ...
- def betweenness_centrality(self) -> dict[str, float]: ...
- def strongly_connected_components(self) -> list[frozenset[str]]: ...
- def community_detection(self) -> dict[str, int]: ...
- # Node attrs: language, symbol_count, has_error, is_test, is_entry_point, pagerank, betweenness, community_id
- # Edge attrs: imported_names (list[str])
-```
-
-### ingestion/change_detector.py
-
-```python
-@dataclass
-class FileDiff:
- path: str
- status: Literal["added", "deleted", "modified", "renamed"]
- old_path: str | None
- old_parsed: ParsedFile | None
- new_parsed: ParsedFile | None
- symbol_diff: SymbolDiff | None
-
-@dataclass
-class AffectedPages:
- regenerate: list[str]
- rename_patch: list[str]
- decay_only: list[str]
-
-class ChangeDetector:
- def __init__(self, repo_path: Path) -> None: ...
- def get_changed_files(self, base_ref="HEAD~1", until_ref="HEAD") -> list[FileDiff]: ...
- def get_affected_pages(self, file_diffs, graph, cascade_budget=30) -> AffectedPages: ...
-```
-
-### persistence/crud.py
-
-```python
-# Key existing functions:
-async def upsert_repository(session, *, name, local_path, ...) -> Repository
-async def upsert_page_from_generated(session, generated_page, repository_id) -> Page
-async def batch_upsert_graph_nodes(session, repository_id, nodes) -> None
-async def batch_upsert_graph_edges(session, repository_id, edges) -> None
-async def batch_upsert_symbols(session, repository_id, symbols) -> None
-async def get_page(session, page_id) -> Page | None
-async def list_pages(session, repository_id, ...) -> list[Page]
-
-_BATCH_SIZE = 500 # Used for all batch operations
-```
-
-### CLI: init_cmd.py pipeline (abbreviated)
-
-```python
-# Pipeline order:
-# 1. Resolve repo path
-# 2. Ensure .repowise dir
-# 2.5 Load config; merge exclude_patterns from config + --exclude/-x flags
-# 3. Resolve LLM provider
-# 4. FileTraverser(extra_exclude_patterns=...) → list[FileInfo]
-# 5. ASTParser → list[ParsedFile]
-# 6. GraphBuilder → build graph, compute metrics
-# 7. Cost estimate + confirmation
-# 8. PageGenerator.generate_all() → list[GeneratedPage]
-# 9. Persist pages, graph, symbols to DB
-# 10. FTS indexing
-# 11. Update state file
-```
-
-### CLI: update_cmd.py pipeline (abbreviated)
-
-```python
-# Pipeline order:
-# 1. Load state, get base_ref
-# 2. ChangeDetector.get_changed_files()
-# 3. Re-ingest full repo for graph
-# 4. ChangeDetector.get_affected_pages()
-# 5. PageGenerator.generate_all() for affected files
-# 6. Persist + update state
-```
-
----
-
-## What to Build
-
-### New Files to Create
-
-```
-packages/core/src/repowise/core/
-├── ingestion/
-│ └── git_indexer.py # GitIndexer class
-├── analysis/
-│ ├── __init__.py
-│ └── dead_code.py # DeadCodeAnalyzer class
-
-packages/cli/src/repowise/cli/
-└── commands/
- └── dead_code_cmd.py # repowise dead-code CLI command
-
-tests/
-├── unit/
-│ ├── test_git_indexer.py
-│ ├── test_dead_code.py
-│ └── test_confidence_with_git.py
-└── integration/
- ├── test_git_intelligence_integration.py
- └── test_dead_code_integration.py
-
-tests/fixtures/sample_repo/
-├── dead/
-│ └── unreachable_module.py # nothing imports this
-└── utils/
- └── helpers.py # has unused_helper() export
-```
-
-### Existing Files to Modify (additive only)
-
-| File | Change |
-|------|--------|
-| `persistence/models.py` | Add `GitMetadata` + `DeadCodeFinding` ORM classes |
-| `persistence/crud.py` | Add git metadata + dead code CRUD functions |
-| `generation/models.py` | Add `GitConfig`, `DeadCodeConfig` dataclasses; extend `GenerationConfig`; add `compute_confidence_decay_with_git()` |
-| `generation/context_assembler.py` | Add `git_metadata`, `co_change_pages`, `dead_code_findings` to `FilePageContext`; add `assemble_update_context()` method; add `_select_generation_depth()` method |
-| `generation/page_generator.py` | Add `_sort_level_files()` method for hotspot priority ordering |
-| `generation/templates/file_page.j2` | Add git context block + dead code block (all `{% if %}` gated) |
-| `generation/templates/module_page.j2` | Add ownership summary block |
-| `generation/templates/repo_overview.j2` | Add codebase health signals block |
-| `generation/templates/diff_summary.j2` | Add trigger commit + diff context |
-| `ingestion/graph.py` | Add `add_co_change_edges()` + `update_co_change_edges()` methods |
-| `ingestion/change_detector.py` | Extend `FileDiff` with trigger commit fields; include co-change partners in staleness |
-| `cli/commands/init_cmd.py` | Add Steps 3.5 (git indexing) and 3.6 (dead code analysis) |
-| `cli/commands/update_cmd.py` | Add git re-index + dead code re-analysis |
-| `cli/main.py` | Register `dead-code` command |
-
----
-
-## Detailed Specifications
-
-### 1. GitMetadata ORM Model (`persistence/models.py`)
-
-Add to existing file — do NOT modify existing models:
-
-```python
-class GitMetadata(Base):
- __tablename__ = "git_metadata"
- __table_args__ = (
- UniqueConstraint("repository_id", "file_path", name="uq_git_metadata"),
- )
-
- id: Mapped[str] = mapped_column(String(32), primary_key=True, default=_new_uuid)
- repository_id: Mapped[str] = mapped_column(
- String(32), ForeignKey("repositories.id", ondelete="CASCADE"), nullable=False
- )
- file_path: Mapped[str] = mapped_column(Text, nullable=False)
-
- # Commit volume
- commit_count_total: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
- commit_count_90d: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
- commit_count_30d: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
-
- # Timeline
- first_commit_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
- last_commit_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
-
- # Ownership
- primary_owner_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
- primary_owner_email: Mapped[str | None] = mapped_column(String(255), nullable=True)
- primary_owner_commit_pct: Mapped[float | None] = mapped_column(Float, nullable=True)
-
- # JSON fields (stored as Text, parsed/serialized in CRUD layer)
- top_authors_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
- significant_commits_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
- co_change_partners_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
-
- # Derived signals
- is_hotspot: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
- is_stable: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
- churn_percentile: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
- age_days: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
-
- created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now_utc)
- updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now_utc, onupdate=_now_utc)
-```
-
-**Important:** The existing codebase does NOT use SQLAlchemy JSON type — it stores JSON
-as Text and parses in the CRUD layer (see `config_json`, `imported_names_json`, etc.).
-Follow this same pattern for `top_authors_json`, `significant_commits_json`, and
-`co_change_partners_json`.
-
-### 2. DeadCodeFinding ORM Model (`persistence/models.py`)
-
-```python
-class DeadCodeFinding(Base):
- __tablename__ = "dead_code_findings"
-
- id: Mapped[str] = mapped_column(String(32), primary_key=True, default=_new_uuid)
- repository_id: Mapped[str] = mapped_column(
- String(32), ForeignKey("repositories.id", ondelete="CASCADE"), nullable=False
- )
- kind: Mapped[str] = mapped_column(String(32), nullable=False) # unreachable_file, unused_export, etc.
- file_path: Mapped[str] = mapped_column(Text, nullable=False)
- symbol_name: Mapped[str | None] = mapped_column(String(255), nullable=True)
- symbol_kind: Mapped[str | None] = mapped_column(String(32), nullable=True)
- confidence: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
- reason: Mapped[str] = mapped_column(Text, nullable=False, default="")
- last_commit_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
- commit_count_90d: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
- lines: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
- package: Mapped[str | None] = mapped_column(String(255), nullable=True)
- evidence_json: Mapped[str] = mapped_column(Text, nullable=False, default="[]")
- safe_to_delete: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
- primary_owner: Mapped[str | None] = mapped_column(String(255), nullable=True)
- age_days: Mapped[int | None] = mapped_column(Integer, nullable=True)
- status: Mapped[str] = mapped_column(String(32), nullable=False, default="open") # open, acknowledged, resolved, false_positive
- note: Mapped[str | None] = mapped_column(Text, nullable=True)
- analyzed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, default=_now_utc)
-```
-
-### 3. Alembic Migration
-
-Create a new Alembic migration file. Follow the pattern of the existing
-`0001_initial_schema.py` migration. The migration should:
-- Create `git_metadata` table with all columns
-- Create `dead_code_findings` table with all columns
-- Add indexes on `repository_id` for both tables
-- Add index on `(repository_id, file_path)` for git_metadata
-
-### 4. CRUD Functions (`persistence/crud.py`)
-
-Add these functions — follow existing patterns (async, use `_BATCH_SIZE = 500`):
-
-```python
-# Git metadata CRUD
-async def upsert_git_metadata(session, *, repository_id, file_path, **kwargs) -> GitMetadata
-async def get_git_metadata(session, repository_id, file_path) -> GitMetadata | None
-async def get_git_metadata_bulk(session, repository_id, file_paths: list[str]) -> dict[str, GitMetadata]
-async def get_all_git_metadata(session, repository_id) -> dict[str, GitMetadata]
-async def upsert_git_metadata_bulk(session, repository_id, metadata_list: list[dict]) -> None
-
-# Dead code CRUD
-async def save_dead_code_findings(session, repository_id, findings: list[dict]) -> None
-async def get_dead_code_findings(session, repository_id, *, kind=None, min_confidence=0.0, status="open") -> list[DeadCodeFinding]
-async def update_dead_code_status(session, finding_id, status, note=None) -> DeadCodeFinding | None
-async def get_dead_code_summary(session, repository_id) -> dict
-```
-
-### 5. GitIndexer (`ingestion/git_indexer.py`)
-
-```python
-import asyncio
-import json
-from dataclasses import dataclass, field
-from datetime import datetime, timezone, timedelta
-from pathlib import Path
-from typing import Any
-
-import structlog
-
-logger = structlog.get_logger()
-
-
-@dataclass
-class GitIndexSummary:
- files_indexed: int
- hotspots: int
- stable_files: int
- duration_seconds: float = 0.0
-
-
-class GitIndexer:
- """
- Mines git history into the git_metadata table.
-
- Uses gitpython (already a dependency) for git operations.
- Parallelizes per-file git log calls with asyncio.Semaphore(20).
-
- Non-blocking: if git is unavailable or repo has no history, log a warning
- and return an empty summary. All downstream features degrade gracefully.
- """
-
- def __init__(self, repo_path: str | Path):
- self.repo_path = Path(repo_path)
-
- async def index_repo(self, repo_id: str) -> tuple[GitIndexSummary, list[dict]]:
- """
- Full index of all tracked files. Returns summary + list of metadata dicts
- ready for bulk upsert.
-
- Steps:
- 1. Get list of tracked files from git
- 2. Per-file git log (parallelized with Semaphore(20))
- 3. Per-file git blame for ownership (if blame_enabled)
- 4. Co-change analysis (walk last N commits)
- 5. Compute repo-wide percentiles for hotspot detection
- """
-
- async def index_changed_files(self, changed_file_paths: list[str]) -> list[dict]:
- """
- Incremental update: re-index only changed files.
- Also re-index any file whose co_change_partners include a changed file.
- """
-
- # Internal methods:
- # _get_tracked_files() -> list[str]
- # _index_file(file_path, repo) -> dict (runs in executor to avoid blocking)
- # _get_blame_ownership(file_path, repo) -> tuple[name, email, pct]
- # _is_significant_commit(message) -> bool
- # _compute_co_changes(repo, all_files, commit_limit=500) -> dict[str, list[dict]]
- # _compute_percentiles(metadata_list) -> None (mutates in place)
-```
-
-**Key implementation notes:**
-- Use `git.Repo(self.repo_path)` from gitpython
-- Run `git log` and `git blame` in `asyncio.get_event_loop().run_in_executor()` since
- gitpython is synchronous
-- `_is_significant_commit()`: skip prefixes "Merge ", "Bump ", "chore:", "ci:", "style:",
- "build:", "release:", "revert:"; skip authors containing "dependabot", "renovate",
- "github-actions"; skip messages < 20 chars
-- Co-change analysis: walk commits, for each commit touching >= 2 tracked files,
- record co-occurrence pairs. Return pairs with count >= `co_change_min_count` (default 3).
- Limit to last `co_change_commit_limit` commits (default 500).
-- Percentiles: `churn_percentile = rank / total_files`. `is_hotspot = commit_count_90d > p75
- AND complexity_estimate > p75` (get complexity from GraphNode if available, else use file size).
- `is_stable = commit_count_total > 10 AND commit_count_90d == 0`.
-
-### 6. DeadCodeAnalyzer (`analysis/dead_code.py`)
-
-Pure graph traversal + SQL — no LLM calls. Must complete in < 10 seconds.
-
-```python
-from dataclasses import dataclass, field
-from datetime import datetime
-from enum import Enum
-from typing import Any, Literal
-
-import structlog
-
-
-class DeadCodeKind(str, Enum):
- UNREACHABLE_FILE = "unreachable_file"
- UNUSED_EXPORT = "unused_export"
- UNUSED_INTERNAL = "unused_internal"
- ZOMBIE_PACKAGE = "zombie_package"
-
-
-@dataclass
-class DeadCodeFindingData:
- kind: DeadCodeKind
- file_path: str
- symbol_name: str | None
- symbol_kind: str | None
- confidence: float
- reason: str
- last_commit_at: datetime | None
- commit_count_90d: int
- lines: int
- package: str | None
- evidence: list[str]
- safe_to_delete: bool
- primary_owner: str | None
- age_days: int | None
-
-
-@dataclass
-class DeadCodeReport:
- repo_id: str
- analyzed_at: datetime
- total_findings: int
- findings: list[DeadCodeFindingData]
- deletable_lines: int
- confidence_summary: dict # {"high": N, "medium": N, "low": N}
-
-
-class DeadCodeAnalyzer:
- """
- Detects unreachable files, unused exports, unused internals, and
- zombie packages using the dependency graph and git metadata.
-
- All analysis is graph traversal + SQL. No LLM calls.
- """
-
- def __init__(self, graph, git_meta_map: dict | None = None):
- self.graph = graph # nx.DiGraph
- self.git_meta_map = git_meta_map or {}
-
- def analyze(self, config: dict | None = None) -> DeadCodeReport:
- """Full analysis. Returns report with all findings."""
-
- def analyze_partial(self, affected_files: list[str], config: dict | None = None) -> DeadCodeReport:
- """Partial analysis for incremental updates."""
-```
-
-**Confidence rules (implement exactly):**
-
-Unreachable files (`graph.in_degree(node) == 0` AND not entry point AND not test AND not config):
-- 1.0 if in_degree==0 AND commit_count_90d==0 AND last_commit > 6 months ago
-- 0.7 if in_degree==0 AND commit_count_90d==0
-- 0.4 if in_degree==0 but recently touched
-- `safe_to_delete=True` only if confidence >= 0.7 AND NOT filename matches dynamic patterns
-
-Unused exports (symbol `visibility=="public"` AND no incoming edges):
-- 1.0 if symbol has no importers AND file IS imported elsewhere
-- 0.7 if symbol has no importers AND file also has no importers
-- 0.3 if name matches `*_DEPRECATED`, `*_LEGACY`, `*_COMPAT`
-- `safe_to_delete=True` only if confidence >= 0.7 AND complexity_estimate < 5
-
-**Never flag as dead:**
-- Files in `__init__.py` / files that are re-export barrels
-- `@pytest.fixture`, `@pytest.mark.*` symbols
-- Files matching `*migrations*`, `*schema*`, `*seed*`
-- `.d.ts` files
-- `is_api_contract == True` files
-- Files/symbols matching `config.dead_code.dynamic_patterns`
-- Files in whitelist
-
-### 7. Template Updates
-
-**file_page.j2** — add at the END of the existing template, BEFORE the closing
-instructions. All blocks gated on `{% if %}`:
-
-```jinja2
-{# ── Git context block ────────────────────────────────────────── #}
-{% if git_metadata %}
-
-{% if git_metadata.primary_owner_name %}
-## Ownership
-Primary maintainer: **{{ git_metadata.primary_owner_name }}**
-({{ (git_metadata.primary_owner_commit_pct * 100) | round | int }}% of lines)
-{% if git_metadata.top_authors | length > 1 %}
-Also contributed by: {% for a in git_metadata.top_authors[1:3] %}{{ a.name }}{% if not loop.last %}, {% endif %}{% endfor %}
-{% endif %}
-{% endif %}
-
-{% if git_metadata.significant_commit_messages %}
-## Evolution context
-This file has been shaped by {{ git_metadata.commit_count_total }} commits
-over {{ git_metadata.age_days }} days. Key changes:
-{% for commit in git_metadata.significant_commit_messages %}
-- **{{ commit.date[:10] }}** — {{ commit.message }} *({{ commit.author }})*
-{% endfor %}
-
-Use this history to explain *why* the code is structured as it is.
-{% endif %}
-
-{% if git_metadata.is_hotspot %}
-IMPORTANT: This file is a HOTSPOT — top 25% for both change frequency and
-complexity. Note this prominently. High-risk area for bugs.
-{% endif %}
-
-{% if git_metadata.is_stable %}
-NOTE: This file is STABLE — {{ git_metadata.commit_count_total }} commits
-but unchanged in 90+ days. Emphasize stability and reliability.
-{% endif %}
-
-{% if git_metadata.co_change_partners %}
-## Hidden coupling (co-change analysis)
-These files have no import relationship but frequently change together:
-{% for partner in git_metadata.co_change_partners[:3] %}
-- `{{ partner.file_path }}` (co-changed {{ partner.co_change_count }} times)
-{% endfor %}
-Mention this relationship explicitly.
-{% endif %}
-
-{% endif %}
-{# ── End git context block ────────────────────────────────────── #}
-
-{% if dead_code_findings %}
-## Potentially unused code
-{% for finding in dead_code_findings %}
-- `{{ finding.symbol_name }}` — {{ finding.reason }}
- (confidence: {{ (finding.confidence * 100) | int }}%{% if finding.safe_to_delete %}, safe to remove{% endif %})
-{% endfor %}
-{% endif %}
-```
-
-**module_page.j2** — add at end:
-```jinja2
-{% if module_git_summary %}
-## Team ownership
-{% for pkg_owner in module_git_summary.top_owners %}
-- **{{ pkg_owner.file_count }} files** primarily maintained by {{ pkg_owner.name }}
-{% endfor %}
-Most active recently: `{{ module_git_summary.most_active_file }}`
-({{ module_git_summary.most_active_commits_90d }} commits in 90 days)
-{% endif %}
-```
-
-**repo_overview.j2** — add at end:
-```jinja2
-{% if repo_git_summary %}
-## Codebase health signals
-- **Hotspots:** {{ repo_git_summary.hotspot_count }} files are high-churn AND high-complexity
-- **Stable core:** {{ repo_git_summary.stable_count }} files unchanged in 90+ days
-- **Most changed (90d):** {{ repo_git_summary.top_churn_files[:3] | join(', ') }}
-- **Oldest file:** `{{ repo_git_summary.oldest_file }}` ({{ repo_git_summary.oldest_file_age_days }} days)
-{% endif %}
-```
-
-### 8. Context Assembler Extensions
-
-**Extend `FilePageContext`** — add three new fields with defaults:
-
-```python
-@dataclass
-class FilePageContext:
- # ... all existing fields unchanged ...
- git_metadata: dict | None = None # NEW
- co_change_pages: list[dict] = field(default_factory=list) # NEW
- dead_code_findings: list[dict] = field(default_factory=list) # NEW
-```
-
-**Extend `assemble_file_page()`** — add optional `git_meta` and `dead_code_findings`
-parameters. If provided, include them in the returned context. If not, the fields
-default to None/[].
-
-**Add `_select_generation_depth()` method:**
-```python
-def _select_generation_depth(self, file, git_meta, config_depth) -> str:
- # Upgrade to "thorough" if: hotspot, >100 commits with >10 in 90d,
- # >=8 significant commits, or has co-change partners
- # Downgrade to "minimal" if: stable AND pagerank < 0.3 AND commit_count < 5
-```
-
-**Add `assemble_update_context()` method** for maintenance regeneration using
-the triggering commit + diff instead of full file source.
-
-### 9. Generation Ordering
-
-Add `_sort_level_files()` to `PageGenerator`:
-
-```python
-async def _sort_level_files(self, files: list[ParsedFile], git_meta_map: dict) -> list[ParsedFile]:
- """Priority (highest first): entry points, hotspots, high PageRank, high commit count, alphabetical."""
-```
-
-Call this in `generate_all()` when processing Level 2 (file pages).
-
-### 10. Confidence Decay Git Modifiers
-
-Add `compute_confidence_decay_with_git()` to `generation/models.py`:
-
-```python
-def compute_confidence_decay_with_git(
- base_decay: float, # e.g. 0.85 for direct
- relationship: str, # "direct", "1hop", "2hop"
- git_meta: dict | None,
- commit_message: str | None,
-) -> float:
- """Apply git modifiers multiplicatively on base decay."""
- # Hotspot: direct *= 0.94, 1hop *= 0.95
- # Stable: direct *= 1.03
- # "rewrite"/"refactor"/"migrate" in message: direct *= 0.71, 1hop *= 0.84
- # "typo"/"lint"/"format" in message: direct *= 1.12
-```
-
-### 11. Co-Change Edges in Graph
-
-Add to `GraphBuilder`:
-
-```python
-def add_co_change_edges(self, git_meta_map: dict, min_count: int = 3) -> int:
- """Add co_changes edges. Returns count of edges added.
- These DO NOT affect PageRank — filter them out before computing."""
-
-def update_co_change_edges(self, updated_meta: dict, min_count: int = 3) -> None:
- """Remove old co_changes edges for updated files, add new ones."""
-```
-
-### 12. Extended FileDiff
-
-Add optional fields to `FileDiff` in `change_detector.py`:
-
-```python
-@dataclass
-class FileDiff:
- # ... existing fields ...
- trigger_commit_sha: str | None = None # NEW
- trigger_commit_message: str | None = None # NEW
- trigger_commit_author: str | None = None # NEW
- diff_text: str | None = None # NEW (unified diff, capped at 4K chars)
-```
-
-### 13. Config Extensions
-
-Extend `GenerationConfig` in `models.py`. Since it's `frozen=True`, add new config
-as separate dataclasses:
-
-```python
-@dataclass(frozen=True)
-class GitConfig:
- enabled: bool = True
- co_change_commit_limit: int = 500
- co_change_min_count: int = 3
- blame_enabled: bool = True
- prompt_commit_count: int = 10
- depth_auto_upgrade: bool = True
-
-@dataclass(frozen=True)
-class DeadCodeConfig:
- enabled: bool = True
- detect_unreachable_files: bool = True
- detect_unused_exports: bool = True
- detect_unused_internals: bool = False
- detect_zombie_packages: bool = True
- min_confidence: float = 0.4
- safe_to_delete_threshold: float = 0.7
- dynamic_patterns: tuple[str, ...] = ("*Plugin", "*Handler", "*Adapter", "*Middleware", "register_*", "on_*")
- analyze_on_update: bool = True
-```
-
-Pass these as parameters where needed rather than modifying the frozen `GenerationConfig`.
-
-### 14. Init Pipeline Integration
-
-In `init_cmd.py`, add between graph computation and cost estimation:
-
-```python
-# Step 3.5: Git indexing
-git_indexer = GitIndexer(repo_path)
-git_summary, git_metadata_list = await git_indexer.index_repo(repo_id)
-# Persist git metadata to DB
-await upsert_git_metadata_bulk(session, repo_id, git_metadata_list)
-# Add co-change edges to graph
-git_meta_map = {m["file_path"]: m for m in git_metadata_list}
-graph_builder.add_co_change_edges(git_meta_map)
-
-# Step 3.6: Dead code analysis
-analyzer = DeadCodeAnalyzer(graph_builder.graph(), git_meta_map)
-dead_code_report = analyzer.analyze()
-await save_dead_code_findings(session, repo_id, dead_code_report.findings)
-```
-
-Show progress output:
-```
-[3.5] Indexing git history... done 3,247 files · 14 hotspots · 892 stable (2m 14s)
-[3.6] Detecting dead code... done 7 unreachable files · 23 unused exports (~2,847 lines)
-```
-
-### 15. Update Pipeline Integration
-
-In `update_cmd.py`, add after change detection:
-
-```python
-# Re-index git metadata for changed files
-git_indexer = GitIndexer(repo_path)
-updated_meta = await git_indexer.index_changed_files(changed_file_paths)
-await upsert_git_metadata_bulk(session, repo_id, updated_meta)
-# Update co-change edges
-git_meta_map = {m["file_path"]: m for m in updated_meta}
-graph_builder.update_co_change_edges(git_meta_map)
-```
-
-### 16. CLI: `repowise dead-code` Command
-
-```python
[email protected]("dead-code")
[email protected]("path", required=False, type=click.Path(exists=True))
[email protected]("--min-confidence", default=0.4, type=float)
[email protected]("--safe-only", is_flag=True)
[email protected]("--kind", type=click.Choice(["unreachable_file", "unused_export", "unused_internal", "zombie_package"]))
[email protected]("--format", "fmt", default="table", type=click.Choice(["table", "json", "md"]))
-def dead_code_command(path, min_confidence, safe_only, kind, fmt):
- """Detect dead and unused code."""
-```
-
-Register in `main.py` via `cli.add_command(dead_code_command)`.
-
-### 17. Tests
-
-**test_git_indexer.py** (7 tests):
-```python
-def test_significant_commit_filter() # merges, bumps, chore filtered out
-def test_co_change_detection() # files changed together >= 3 times detected
-def test_hotspot_classification() # top 25% churn + complexity = is_hotspot
-def test_stable_classification() # >10 commits, 0 in 90d = is_stable
-def test_git_unavailable_graceful() # no history → empty metadata, no crash
-def test_co_change_below_threshold_skipped() # pairs with count < min_count not stored
-def test_blame_ownership_computed() # primary_owner set to dominant author
-```
-
-**test_dead_code.py** (12 tests):
-```python
-def test_unreachable_file_detected()
-def test_entry_point_not_flagged()
-def test_test_files_excluded()
-def test_unused_export_detected()
-def test_framework_decorator_excluded()
-def test_dynamic_pattern_excluded()
-def test_confidence_low_for_recent_files()
-def test_confidence_high_for_stale_unreachable()
-def test_zombie_package_detected()
-def test_whitelist_respected()
-def test_safe_to_delete_conservative()
-def test_report_deletable_lines_sum()
-```
-
-**test_confidence_with_git.py** (4 tests):
-```python
-def test_hotspot_decays_faster()
-def test_stable_decays_slower()
-def test_rewrite_commit_hard_decay()
-def test_typo_commit_soft_decay()
-```
-
-**Integration tests** (use sample_repo with added fixtures):
-```python
-async def test_git_indexer_on_sample_repo()
-async def test_dead_code_detects_unreachable_fixture()
-async def test_co_change_edges_in_graph()
-async def test_hotspot_sorted_first_in_generation()
-```
-
-### 18. Test Fixtures
-
-Add to `tests/fixtures/sample_repo/`:
-
-**`dead/unreachable_module.py`** — a Python file with functions that nothing imports:
-```python
-"""This module is intentionally unreachable — used for dead code detection tests."""
-
-def orphaned_function():
- """No one calls this."""
- return 42
-
-class OrphanedClass:
- """No one imports this."""
- pass
-```
-
-**`utils/helpers.py`** — add an unused export (if this file already exists, add to it):
-```python
-def unused_helper():
- """This function is exported but never imported anywhere."""
- return "unused"
-```
-
----
-
-## Implementation Order
-
-Build in this exact order (each step should be testable independently):
-
-1. `GitMetadata` + `DeadCodeFinding` ORM models + migration
-2. CRUD functions for git metadata and dead code
-3. `GitIndexer` class + `test_git_indexer.py` tests
-4. `DeadCodeAnalyzer` class + `test_dead_code.py` tests
-5. `GitConfig` + `DeadCodeConfig` dataclasses
-6. Template updates (file_page.j2, module_page.j2, repo_overview.j2)
-7. `FilePageContext` extensions + `ContextAssembler` extensions
-8. `_sort_level_files()` in PageGenerator
-9. `_select_generation_depth()` in ContextAssembler
-10. Confidence decay git modifiers + `test_confidence_with_git.py`
-11. Co-change edges in GraphBuilder
-12. FileDiff trigger commit extensions
-13. diff_summary.j2 update + `assemble_update_context()`
-14. ChangeDetector co-change partner staleness
-15. Init pipeline integration (Steps 3.5 + 3.6)
-16. Update pipeline integration
-17. `repowise dead-code` CLI command
-18. Integration tests
-19. Run full test suite: `pytest tests/ -q` — all must pass
-
----
-
-## Critical Constraints
-
-1. **Additive only.** Do NOT rewrite existing methods — add new methods, extend dataclasses
- with defaulted fields, add `{% if %}` blocks to templates.
-
-2. **Graceful degradation.** Every use of `git_metadata` must handle `None`. If `GitIndexer`
- fails or git is unavailable, all features fall back silently to pre-feature behavior.
-
-3. **Follow existing patterns.** JSON stored as Text (not SQLAlchemy JSON type). UUIDs via
- `_new_uuid()`. Timestamps via `_now_utc()`. `Mapped[]` for all ORM columns. `_BATCH_SIZE = 500`
- for bulk operations. `structlog` for logging.
-
-4. **No new dependencies.** `gitpython` is already in the dependency tree. All analysis is
- pure Python + NetworkX + SQL.
-
-5. **Performance.** GitIndexer < 3 min for 3K files. DeadCodeAnalyzer < 10 seconds.
- Co-change computation < 30 seconds.
-
-6. **Conservative dead code.** When in doubt, do NOT flag. `safe_to_delete` is very conservative.
-
-7. **Test everything.** 23 new unit tests + integration tests. All existing 473 tests must
- continue passing.
-
-8. **MCP tools, REST API endpoints, and Web UI pages are NOT built in this phase.** They are
- deferred to Phases 6, 7, and 8 respectively. Only the core logic, templates, CLI, and
- pipeline integration are built here.
diff --git a/docs/ARCHITECTURE.md b/docs/architecture/ARCHITECTURE.md
similarity index 100%
rename from docs/ARCHITECTURE.md
rename to docs/architecture/ARCHITECTURE.md
diff --git a/docs/deep-dives-guide.md b/docs/architecture/deep-dives.md
similarity index 100%
rename from docs/deep-dives-guide.md
rename to docs/architecture/deep-dives.md
diff --git a/docs/graph-algorithms-guide.md b/docs/architecture/graph-algorithms.md
similarity index 100%
rename from docs/graph-algorithms-guide.md
rename to docs/architecture/graph-algorithms.md
diff --git a/docs/architecture-guide.md b/docs/architecture/implementation-deep-dive.md
similarity index 100%
rename from docs/architecture-guide.md
rename to docs/architecture/implementation-deep-dive.md
diff --git a/docs/CHAT.md b/docs/internals/chat.md
similarity index 100%
rename from docs/CHAT.md
rename to docs/internals/chat.md
diff --git a/docs/critical-analysis.md b/docs/internals/critical-analysis.md
similarity index 100%
rename from docs/critical-analysis.md
rename to docs/internals/critical-analysis.md
diff --git a/docs/EDITOR_FILES.md b/docs/internals/editor-files.md
similarity index 100%
rename from docs/EDITOR_FILES.md
rename to docs/internals/editor-files.md
diff --git a/docs/LLM_GENERATION_DEEP_DIVE.md b/docs/internals/llm-generation.md
similarity index 100%
rename from docs/LLM_GENERATION_DEEP_DIVE.md
rename to docs/internals/llm-generation.md
diff --git a/packages/cli/src/repowise/cli/__init__.py b/packages/cli/src/repowise/cli/__init__.py
index a0d551603..ccc1e3fe1 100644
--- a/packages/cli/src/repowise/cli/__init__.py
+++ b/packages/cli/src/repowise/cli/__init__.py
@@ -6,4 +6,4 @@ git signals, dead code detection, architectural decisions, and
AI-generated documentation.
"""
-__version__ = "0.2.1"
+__version__ = "0.2.2"
diff --git a/packages/cli/src/repowise/cli/commands/init_cmd.py b/packages/cli/src/repowise/cli/commands/init_cmd.py
index 1ba9becbc..2f66e85c4 100644
--- a/packages/cli/src/repowise/cli/commands/init_cmd.py
+++ b/packages/cli/src/repowise/cli/commands/init_cmd.py
@@ -239,6 +239,12 @@ async def _persist_result(
default=False,
help="Skip generating CLAUDE.md. Saves 'editor_files.claude_md: false' to config.",
)
+ "--include-submodules",
+ is_flag=True,
+ default=False,
+ help="Include git submodule directories (excluded by default).",
+)
def init_command(
path: str | None,
provider_name: str | None,
@@ -257,6 +263,7 @@ def init_command(
commit_limit: int | None,
follow_renames: bool,
no_claude_md: bool,
+ include_submodules: bool,
) -> None:
"""Generate wiki documentation for a codebase.
@@ -426,6 +433,7 @@ def init_command(
skip_tests=skip_tests,
skip_infra=skip_infra,
exclude_patterns=exclude_patterns if exclude_patterns else None,
+ include_submodules=include_submodules,
generate_docs=False,
llm_client=llm_client,
concurrency=concurrency,
@@ -463,6 +471,13 @@ def init_command(
table.add_row("[bold]Total[/bold]", f"[bold]{est.total_pages}[/bold]", "")
console.print(table)
+ # Language breakdown
+ lang_dist = result.repo_structure.root_language_distribution
+ if lang_dist:
+ lang_items = sorted(lang_dist.items(), key=lambda x: -x[1])[:6]
+ lang_parts = [f"{lang} {pct:.0%}" for lang, pct in lang_items]
+ console.print(f" Languages: {', '.join(lang_parts)}")
+
console.print(
f" Estimated tokens: ~{est.estimated_input_tokens + est.estimated_output_tokens:,} "
f"(${est.estimated_cost_usd:.2f} USD)"
@@ -624,6 +639,7 @@ def init_command(
# ---- State + config (full mode only) ----
if not index_only and provider:
+
async def _count_db_pages() -> int:
from sqlalchemy import func as sa_func
from sqlalchemy import select as sa_select
@@ -687,11 +703,21 @@ def init_command(
)
_n_decisions = sum(result.decision_report.by_source.values()) if result.decision_report else 0
+ # Build a compact language summary for the completion panel
+ _lang_dist = result.repo_structure.root_language_distribution
+ if _lang_dist:
+ _top = sorted(_lang_dist.items(), key=lambda x: -x[1])[:4]
+ _lang_summary = ", ".join(f"{lang} {pct:.0%}" for lang, pct in _top)
+ if len(_lang_dist) > 4:
+ _lang_summary += f" +{len(_lang_dist) - 4} more"
+ else:
+ _lang_summary = str(len(result.languages))
+
if index_only:
metrics: list[tuple[str, str]] = [
("Files indexed", str(result.file_count)),
("Symbols", f"{result.symbol_count:,}"),
- ("Languages", str(len(result.languages))),
+ ("Languages", _lang_summary),
("Elapsed", format_elapsed(elapsed)),
("", ""),
(
diff --git a/packages/cli/src/repowise/cli/ui.py b/packages/cli/src/repowise/cli/ui.py
index 633c052a8..087fc3303 100644
--- a/packages/cli/src/repowise/cli/ui.py
+++ b/packages/cli/src/repowise/cli/ui.py
@@ -407,9 +407,16 @@ def interactive_advanced_config(console: Console) -> dict[str, Any]:
)
# --- exclude patterns ---
- console.print(" [dim]Exclude patterns (gitignore-style, comma-separated, or empty):[/dim]")
- raw = click.prompt(" Exclude", default="", show_default=False)
- patterns = [p.strip() for p in raw.split(",") if p.strip()]
+ console.print(
+ " [dim]Exclude patterns (gitignore-style). Enter one per line, empty line to finish:[/dim]"
+ )
+ patterns: list[str] = []
+ while True:
+ raw = click.prompt(" Exclude pattern", default="", show_default=False)
+ raw = raw.strip()
+ if not raw:
+ break
+ patterns.append(raw)
result["exclude"] = tuple(patterns)
# --- test run ---
@@ -535,7 +542,7 @@ def format_elapsed(seconds: float) -> str:
# ---------------------------------------------------------------------------
_PHASE_LABELS: dict[str, str] = {
- "traverse": "Traversing files...",
+ "traverse": "Scanning & filtering files...",
"parse": "Parsing files...",
"graph": "Building dependency graph...",
"git": "Indexing file history...",
@@ -569,9 +576,7 @@ class RichProgressCallback:
if phase in self._tasks:
self._progress.update(self._tasks[phase], total=total, visible=True)
else:
- self._tasks[phase] = self._progress.add_task(
- label, total=total, visible=True, cost=0.0
- )
+ self._tasks[phase] = self._progress.add_task(label, total=total, visible=True, cost=0.0)
def on_item_done(self, phase: str) -> None:
if phase in self._tasks:
diff --git a/packages/core/src/repowise/core/__init__.py b/packages/core/src/repowise/core/__init__.py
index b2f6b5c0c..518e3b035 100644
--- a/packages/core/src/repowise/core/__init__.py
+++ b/packages/core/src/repowise/core/__init__.py
@@ -6,4 +6,4 @@ generation engine. This is the heart of repowise — all other packages depend o
Namespace package: repowise.core is part of the repowise namespace.
"""
-__version__ = "0.2.1"
+__version__ = "0.2.2"
diff --git a/packages/core/src/repowise/core/ingestion/__init__.py b/packages/core/src/repowise/core/ingestion/__init__.py
index ec79ea165..b6d8f716c 100644
--- a/packages/core/src/repowise/core/ingestion/__init__.py
+++ b/packages/core/src/repowise/core/ingestion/__init__.py
@@ -24,7 +24,7 @@ from .models import (
compute_content_hash,
)
from .parser import LANGUAGE_CONFIGS, ASTParser, LanguageConfig, parse_file
-from .traverser import FileTraverser
+from .traverser import FileTraverser, TraversalStats
__all__ = [
# Models
@@ -39,6 +39,7 @@ __all__ = [
"FileInfo",
# Traversal
"FileTraverser",
+ "TraversalStats",
# Graph
"GraphBuilder",
"Import",
diff --git a/packages/core/src/repowise/core/ingestion/traverser.py b/packages/core/src/repowise/core/ingestion/traverser.py
index fa878b056..79f15c4a8 100644
--- a/packages/core/src/repowise/core/ingestion/traverser.py
+++ b/packages/core/src/repowise/core/ingestion/traverser.py
@@ -14,9 +14,11 @@ It also detects monorepo structure and returns a RepoStructure.
from __future__ import annotations
+import configparser
import os
import threading
from collections.abc import Iterator
+from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
@@ -32,6 +34,32 @@ from .models import (
RepoStructure,
)
+
+# ---------------------------------------------------------------------------
+# Traversal statistics
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class TraversalStats:
+ """Counts collected during file traversal, broken down by skip reason."""
+
+ total_paths_walked: int = 0
+ included: int = 0
+ skipped_gitignore: int = 0
+ skipped_blocked_extension: int = 0
+ skipped_oversized: int = 0
+ skipped_binary: int = 0
+ skipped_generated: int = 0
+ skipped_extra_ignore: int = 0
+ skipped_extra_exclude: int = 0
+ skipped_blocked_pattern: int = 0
+ skipped_unknown_language: int = 0
+ skipped_dir_ignore: int = 0
+ skipped_submodule: int = 0
+ lang_counts: dict[str, int] = field(default_factory=dict)
+
+
log = structlog.get_logger(__name__)
# ---------------------------------------------------------------------------
@@ -185,6 +213,8 @@ class FileTraverser:
Defaults to ``.repowiseIgnore``.
extra_exclude_patterns: Additional gitignore-style patterns to exclude
(from CLI ``--exclude`` flags or ``repo.settings["exclude_patterns"]``).
+ include_submodules: When False (default), directories listed in
+ ``.gitmodules`` are skipped during traversal.
"""
def __init__(
@@ -194,6 +224,7 @@ class FileTraverser:
max_file_size_kb: int = 500,
extra_ignore_filename: str = ".repowiseIgnore",
extra_exclude_patterns: list[str] | None = None,
+ include_submodules: bool = False,
) -> None:
self.repo_root = repo_root.resolve()
self.max_file_size_bytes = max_file_size_kb * 1024
@@ -210,14 +241,17 @@ class FileTraverser:
self._dir_ignore_cache: dict[str, pathspec.PathSpec] = {
str(self.repo_root): self._extra_ignore,
}
- self._oversized_skip_count: int = 0
- self._oversized_files: list[tuple[str, int]] = [] # (path, size_bytes)
+ self._submodule_paths: frozenset[str] = frozenset()
+ if not include_submodules:
+ self._submodule_paths = _parse_gitmodules(self.repo_root)
+ self.stats = TraversalStats()
self._count_lock = threading.Lock()
log.info(
"FileTraverser initialised",
repo_root=str(self.repo_root),
max_file_size_kb=max_file_size_kb,
extra_exclude_patterns=len(patterns),
+ submodules_skipped=len(self._submodule_paths),
)
# ------------------------------------------------------------------
@@ -229,6 +263,11 @@ class FileTraverser:
for abs_path in self._walk():
info = self._build_file_info(abs_path)
if info is not None:
+ with self._count_lock:
+ self.stats.included += 1
+ self.stats.lang_counts[info.language] = (
+ self.stats.lang_counts.get(info.language, 0) + 1
+ )
yield info
def get_repo_structure(self, files: list[FileInfo] | None = None) -> RepoStructure:
@@ -286,6 +325,7 @@ class FileTraverser:
)
for filename in sorted(filenames):
+ self.stats.total_paths_walked += 1
yield dirpath_obj / filename
def _get_dir_ignore(self, dirpath: Path) -> pathspec.PathSpec:
@@ -309,6 +349,9 @@ class FileTraverser:
if dirname in _BLOCKED_DIRS:
return True
rel_str = rel_path.as_posix()
+ if rel_str in self._submodule_paths:
+ self.stats.skipped_submodule += 1
+ return True
if self._gitignore.match_file(rel_str + "/"):
return True
if self._extra_ignore.match_file(rel_str + "/"):
@@ -335,29 +378,40 @@ class FileTraverser:
# Size limit
if size_bytes > self.max_file_size_bytes:
with self._count_lock:
- self._oversized_skip_count += 1
- self._oversized_files.append((rel_str, size_bytes))
+ self.stats.skipped_oversized += 1
log.debug("Skipping oversized file", path=rel_str, size_kb=size_bytes // 1024)
return None
# Blocked extension
if abs_path.suffix.lower() in _BLOCKED_EXTENSIONS:
+ with self._count_lock:
+ self.stats.skipped_blocked_extension += 1
return None
# gitignore / extra ignore / extra exclude patterns
if self._gitignore.match_file(rel_str):
+ with self._count_lock:
+ self.stats.skipped_gitignore += 1
return None
if self._extra_ignore.match_file(rel_str):
+ with self._count_lock:
+ self.stats.skipped_extra_ignore += 1
return None
if self._extra_exclude.match_file(rel_str):
+ with self._count_lock:
+ self.stats.skipped_extra_exclude += 1
return None
# Per-directory .repowiseIgnore: check filename against the parent dir's spec.
dir_ignore = self._get_dir_ignore(abs_path.parent)
if dir_ignore.match_file(abs_path.name):
+ with self._count_lock:
+ self.stats.skipped_dir_ignore += 1
return None
# Blocklist filename patterns
if self._blocked_patterns.match_file(rel_str):
+ with self._count_lock:
+ self.stats.skipped_blocked_pattern += 1
return None
# Language detection — name/extension lookup is free (no I/O). Only
@@ -366,14 +420,20 @@ class FileTraverser:
language = _language_from_name_or_ext(abs_path)
if language is None:
if _is_binary(abs_path):
+ with self._count_lock:
+ self.stats.skipped_binary += 1
return None
language = _detect_by_shebang(abs_path)
if language == "unknown":
+ with self._count_lock:
+ self.stats.skipped_unknown_language += 1
return None
# Generated file detection: only meaningful for code files. Skipping
# for data/markup files avoids a 512-byte read per file with no benefit.
if language not in _SKIP_GENERATED_CHECK and _is_generated(abs_path):
+ with self._count_lock:
+ self.stats.skipped_generated += 1
log.debug("Skipping generated file", path=rel_str)
return None
@@ -548,6 +608,26 @@ def _find_entry_points_in(directory: Path, repo_root: Path) -> list[str]:
return sorted(result)
+def _parse_gitmodules(repo_root: Path) -> frozenset[str]:
+ """Parse ``.gitmodules`` and return the set of submodule paths (POSIX-style, relative)."""
+ gitmodules = repo_root / ".gitmodules"
+ if not gitmodules.exists():
+ return frozenset()
+ try:
+ parser = configparser.ConfigParser()
+ parser.read(str(gitmodules), encoding="utf-8")
+ paths: set[str] = set()
+ for section in parser.sections():
+ path = parser.get(section, "path", fallback=None)
+ if path:
+ # Normalize to POSIX-style relative path
+ paths.add(path.strip().replace("\\", "/"))
+ return frozenset(paths)
+ except Exception:
+ log.warning("Failed to parse .gitmodules", path=str(gitmodules))
+ return frozenset()
+
+
def _load_gitignore_spec(repo_root: Path) -> pathspec.PathSpec:
gitignore = repo_root / ".gitignore"
lines: list[str] = []
diff --git a/packages/core/src/repowise/core/pipeline/orchestrator.py b/packages/core/src/repowise/core/pipeline/orchestrator.py
index a7c390ce2..084ed8386 100644
--- a/packages/core/src/repowise/core/pipeline/orchestrator.py
+++ b/packages/core/src/repowise/core/pipeline/orchestrator.py
@@ -52,11 +52,13 @@ def _parse_one(path_and_fi_and_bytes: tuple) -> Any:
try:
if _WORKER_PARSER is None:
from repowise.core.ingestion import ASTParser
+
_WORKER_PARSER = ASTParser()
return _WORKER_PARSER.parse_file(fi, source)
except Exception as exc:
return (fi.abs_path, str(exc))
+
# Maximum seconds to spend on decision extraction before giving up.
# Large repos with tens of thousands of files can take arbitrarily long.
DECISION_EXTRACTION_TIMEOUT_SECS = 300
@@ -119,6 +121,10 @@ class PipelineResult:
languages: set[str] = field(default_factory=set)
elapsed_seconds: float = 0.0
+ # Traversal stats
+ traversal_stats: Any | None = None
+ """``TraversalStats`` from the file traverser, or None."""
+
# ---------------------------------------------------------------------------
# Pipeline
@@ -133,6 +139,7 @@ async def run_pipeline(
skip_tests: bool = False,
skip_infra: bool = False,
exclude_patterns: list[str] | None = None,
+ include_submodules: bool = False,
generate_docs: bool = False,
llm_client: Any | None = None,
embedder: Any | None = None,
@@ -212,21 +219,26 @@ async def run_pipeline(
return await _run_ingestion(
repo_path,
exclude_patterns=exclude_patterns,
+ include_submodules=include_submodules,
skip_tests=skip_tests,
skip_infra=skip_infra,
progress=progress,
)
(
- parsed_files,
- file_infos,
- repo_structure,
- source_map,
- graph_builder,
- ), (
- git_summary,
- git_metadata_list,
- git_meta_map,
+ (
+ parsed_files,
+ file_infos,
+ repo_structure,
+ source_map,
+ graph_builder,
+ traversal_stats,
+ ),
+ (
+ git_summary,
+ git_metadata_list,
+ git_meta_map,
+ ),
) = await asyncio.gather(_ingestion_stage(), _git_stage())
# Add co-change edges to the graph
@@ -264,9 +276,7 @@ async def run_pipeline(
if progress:
progress.on_message("info", "Phase 2: Analysis")
- dead_code_report = await _run_dead_code_analysis(
- graph_builder, git_meta_map, progress=progress
- )
+ dead_code_report = await _run_dead_code_analysis(graph_builder, git_meta_map, progress=progress)
decision_report = await _run_decision_extraction(
repo_path,
@@ -315,6 +325,7 @@ async def run_pipeline(
dead_code_report=dead_code_report,
decision_report=decision_report,
generated_pages=generated_pages,
+ traversal_stats=traversal_stats,
repo_name=repo_path.name,
file_count=len(parsed_files),
symbol_count=symbol_count,
@@ -332,22 +343,29 @@ async def _run_ingestion(
repo_path: Path,
*,
exclude_patterns: list[str] | None,
+ include_submodules: bool = False,
skip_tests: bool,
skip_infra: bool,
progress: ProgressCallback | None,
-) -> tuple[list[Any], list[Any], Any, dict[str, bytes], Any]:
+) -> tuple[list[Any], list[Any], Any, dict[str, bytes], Any, Any]:
"""Traverse, parse, and build the dependency graph.
- Returns (parsed_files, file_infos, repo_structure, source_map, graph_builder).
+ Returns (parsed_files, file_infos, repo_structure, source_map, graph_builder, traversal_stats).
"""
from repowise.core.ingestion import ASTParser, FileTraverser, GraphBuilder
- traverser = FileTraverser(repo_path, extra_exclude_patterns=exclude_patterns or None)
+ traverser = FileTraverser(
+ repo_path,
+ extra_exclude_patterns=exclude_patterns or None,
+ include_submodules=include_submodules,
+ )
# Walk directory tree
all_paths = list(traverser._walk())
if progress:
- progress.on_phase_start("traverse", len(all_paths))
+ # Use indeterminate progress (spinner) to avoid showing a misleading
+ # pre-filter total like "2132/83601".
+ progress.on_phase_start("traverse", None)
# Parallel stat + header reads (I/O bound).
# Use asyncio.wrap_future so the event loop stays responsive while waiting.
@@ -355,8 +373,7 @@ async def _run_ingestion(
io_pool = ThreadPoolExecutor(max_workers=8)
try:
aws = [
- asyncio.wrap_future(io_pool.submit(traverser._build_file_info, p))
- for p in all_paths
+ asyncio.wrap_future(io_pool.submit(traverser._build_file_info, p)) for p in all_paths
]
for coro in asyncio.as_completed(aws):
try:
@@ -412,10 +429,7 @@ async def _run_ingestion(
try:
with ProcessPoolExecutor(max_workers=workers) as pool:
- tasks = [
- loop.run_in_executor(pool, _parse_one, item)
- for item in fi_and_bytes
- ]
+ tasks = [loop.run_in_executor(pool, _parse_one, item) for item in fi_and_bytes]
# Use as_completed via asyncio.as_completed to report per-file progress.
# We need to preserve (task → fi_and_bytes index) for source_map so we
# wrap tasks in a list and drain with gather instead.
@@ -483,23 +497,51 @@ async def _run_ingestion(
if progress:
progress.on_item_done("graph")
- # Report oversized file skips
- if traverser._oversized_skip_count and progress:
+ # Emit filtering summary so users can see what was included/excluded
+ stats = traverser.stats
+ if progress:
+ parts: list[str] = []
+ if stats.skipped_gitignore:
+ parts.append(f"{stats.skipped_gitignore:,} by .gitignore")
+ if stats.skipped_blocked_extension:
+ parts.append(f"{stats.skipped_blocked_extension:,} by extension")
+ if stats.skipped_blocked_pattern:
+ parts.append(f"{stats.skipped_blocked_pattern:,} by filename pattern")
+ if stats.skipped_oversized:
+ parts.append(f"{stats.skipped_oversized:,} oversized")
+ if stats.skipped_binary:
+ parts.append(f"{stats.skipped_binary:,} binary")
+ if stats.skipped_generated:
+ parts.append(f"{stats.skipped_generated:,} generated")
+ if stats.skipped_extra_exclude:
+ parts.append(f"{stats.skipped_extra_exclude:,} by --exclude")
+ if stats.skipped_extra_ignore:
+ parts.append(f"{stats.skipped_extra_ignore:,} by .repowiseIgnore")
+ if stats.skipped_submodule:
+ parts.append(f"{stats.skipped_submodule:,} submodule dirs")
+ if stats.skipped_unknown_language:
+ parts.append(f"{stats.skipped_unknown_language:,} unknown type")
+
+ excluded_str = ", ".join(parts) if parts else "none"
progress.on_message(
- "warning",
- f"Skipped {traverser._oversized_skip_count} oversized files "
- f"(>{traverser.max_file_size_bytes // 1024} KB)",
+ "info",
+ f"Scanned {stats.total_paths_walked:,} files, {len(file_infos):,} included",
)
- for path, size_bytes in sorted(
- traverser._oversized_files, key=lambda x: x[1], reverse=True
- ):
- logger.info(
- "oversized_file_skipped",
- path=path,
- size_kb=size_bytes // 1024,
+ if parts:
+ progress.on_message("info", f" Excluded: {excluded_str}")
+
+ # Language breakdown
+ if stats.lang_counts:
+ top_langs = sorted(stats.lang_counts.items(), key=lambda x: -x[1])[:6]
+ lang_str = ", ".join(f"{lang} {count:,}" for lang, count in top_langs)
+ rest_count = sum(
+ c for _, c in sorted(stats.lang_counts.items(), key=lambda x: -x[1])[6:]
)
+ if rest_count:
+ lang_str += f", other {rest_count:,}"
+ progress.on_message("info", f" Languages: {lang_str}")
- return parsed_files, file_infos, repo_structure, source_map, graph_builder
+ return parsed_files, file_infos, repo_structure, source_map, graph_builder, stats
async def _run_git_indexing(
@@ -608,7 +650,9 @@ async def _run_decision_extraction(
git_meta_map=git_meta_map,
parsed_files=parsed_files,
)
- report = await asyncio.wait_for(extractor.extract_all(), timeout=DECISION_EXTRACTION_TIMEOUT_SECS)
+ report = await asyncio.wait_for(
+ extractor.extract_all(), timeout=DECISION_EXTRACTION_TIMEOUT_SECS
+ )
if progress:
inline = report.by_source.get("inline_marker", 0)
diff --git a/packages/server/src/repowise/server/__init__.py b/packages/server/src/repowise/server/__init__.py
index a2e3bb388..2a5513f2e 100644
--- a/packages/server/src/repowise/server/__init__.py
+++ b/packages/server/src/repowise/server/__init__.py
@@ -7,4 +7,4 @@ Provides:
- Background job scheduler (APScheduler)
"""
-__version__ = "0.2.1"
+__version__ = "0.2.2"
diff --git a/pyproject.toml b/pyproject.toml
index 4d116b742..d41c8364a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "repowise"
-version = "0.2.1"
+version = "0.2.2"
description = "Codebase intelligence for developers and AI — generates and maintains a structured wiki for any codebase"
readme = "README.md"
requires-python = ">=3.11"
diff --git a/tests/unit/ingestion/test_traverser.py b/tests/unit/ingestion/test_traverser.py
index 5618b0f87..ef5037bc8 100644
--- a/tests/unit/ingestion/test_traverser.py
+++ b/tests/unit/ingestion/test_traverser.py
@@ -301,3 +301,125 @@ class TestMonorepoDetection:
structure.root_language_distribution["python"]
> structure.root_language_distribution["typescript"]
)
+
+
+# ---------------------------------------------------------------------------
+# TraversalStats
+# ---------------------------------------------------------------------------
+
+
+class TestTraversalStats:
+ def test_stats_counts_included_files(self, tmp_path: Path) -> None:
+ (tmp_path / "a.py").write_text("pass")
+ (tmp_path / "b.py").write_text("pass")
+ (tmp_path / "c.ts").write_text("const x = 1;")
+ traverser = FileTraverser(tmp_path)
+ list(traverser.traverse())
+ assert traverser.stats.included == 3
+ assert traverser.stats.total_paths_walked >= 3
+
+ def test_stats_counts_gitignore_skips(self, tmp_path: Path) -> None:
+ (tmp_path / ".gitignore").write_text("*.log\n")
+ (tmp_path / "app.py").write_text("pass")
+ (tmp_path / "debug.log").write_text("log data")
+ traverser = FileTraverser(tmp_path)
+ list(traverser.traverse())
+ assert traverser.stats.skipped_gitignore >= 1
+ assert traverser.stats.included >= 1
+
+ def test_stats_counts_oversized_skips(self, tmp_path: Path) -> None:
+ big = tmp_path / "big.py"
+ big.write_bytes(b"x = 1\n" * 200_000)
+ (tmp_path / "small.py").write_text("pass")
+ traverser = FileTraverser(tmp_path, max_file_size_kb=500)
+ list(traverser.traverse())
+ assert traverser.stats.skipped_oversized == 1
+ assert traverser.stats.included == 1
+
+ def test_stats_counts_blocked_extension(self, tmp_path: Path) -> None:
+ (tmp_path / "lib.so").write_bytes(b"\x00" * 100)
+ (tmp_path / "app.py").write_text("pass")
+ traverser = FileTraverser(tmp_path)
+ list(traverser.traverse())
+ assert traverser.stats.skipped_blocked_extension >= 1
+
+ def test_stats_lang_counts(self, tmp_path: Path) -> None:
+ (tmp_path / "a.py").write_text("pass")
+ (tmp_path / "b.py").write_text("pass")
+ (tmp_path / "c.ts").write_text("const x = 1;")
+ traverser = FileTraverser(tmp_path)
+ list(traverser.traverse())
+ assert traverser.stats.lang_counts.get("python") == 2
+ assert traverser.stats.lang_counts.get("typescript") == 1
+
+ def test_stats_extra_exclude(self, tmp_path: Path) -> None:
+ (tmp_path / "src").mkdir()
+ (tmp_path / "src" / "main.py").write_text("pass")
+ (tmp_path / "vendor").mkdir()
+ (tmp_path / "vendor" / "lib.py").write_text("pass")
+ traverser = FileTraverser(tmp_path, extra_exclude_patterns=["vendor/"])
+ list(traverser.traverse())
+ # vendor/ is pruned at directory level, not file level
+ assert traverser.stats.included == 1
+
+
+# ---------------------------------------------------------------------------
+# Submodule handling
+# ---------------------------------------------------------------------------
+
+
+class TestSubmoduleHandling:
+ def test_skips_submodule_dirs(self, tmp_path: Path) -> None:
+ (tmp_path / ".gitmodules").write_text(
+ '[submodule "libs/foo"]\n'
+ " path = libs/foo\n"
+ " url = https://github.com/example/foo.git\n"
+ )
+ (tmp_path / "libs" / "foo").mkdir(parents=True)
+ (tmp_path / "libs" / "foo" / "main.py").write_text("pass")
+ (tmp_path / "src").mkdir()
+ (tmp_path / "src" / "app.py").write_text("pass")
+ traverser = FileTraverser(tmp_path)
+ paths = [f.path for f in traverser.traverse()]
+ assert any("app.py" in p for p in paths)
+ assert not any("libs/foo" in p for p in paths)
+ assert traverser.stats.skipped_submodule >= 1
+
+ def test_include_submodules_flag(self, tmp_path: Path) -> None:
+ (tmp_path / ".gitmodules").write_text(
+ '[submodule "libs/foo"]\n'
+ " path = libs/foo\n"
+ " url = https://github.com/example/foo.git\n"
+ )
+ (tmp_path / "libs" / "foo").mkdir(parents=True)
+ (tmp_path / "libs" / "foo" / "main.py").write_text("pass")
+ traverser = FileTraverser(tmp_path, include_submodules=True)
+ paths = [f.path for f in traverser.traverse()]
+ assert any("libs/foo" in p for p in paths)
+
+ def test_no_gitmodules_file(self, tmp_path: Path) -> None:
+ (tmp_path / "app.py").write_text("pass")
+ traverser = FileTraverser(tmp_path)
+ paths = [f.path for f in traverser.traverse()]
+ assert any("app.py" in p for p in paths)
+
+ def test_multiple_submodules(self, tmp_path: Path) -> None:
+ (tmp_path / ".gitmodules").write_text(
+ '[submodule "libs/foo"]\n'
+ " path = libs/foo\n"
+ " url = https://github.com/example/foo.git\n"
+ '[submodule "libs/bar"]\n'
+ " path = libs/bar\n"
+ " url = https://github.com/example/bar.git\n"
+ )
+ (tmp_path / "libs" / "foo").mkdir(parents=True)
+ (tmp_path / "libs" / "foo" / "main.py").write_text("pass")
+ (tmp_path / "libs" / "bar").mkdir(parents=True)
+ (tmp_path / "libs" / "bar" / "index.ts").write_text("export const x = 1;")
+ (tmp_path / "src").mkdir()
+ (tmp_path / "src" / "app.py").write_text("pass")
+ traverser = FileTraverser(tmp_path)
+ paths = [f.path for f in traverser.traverse()]
+ assert any("app.py" in p for p in paths)
+ assert not any("libs/foo" in p for p in paths)
+ assert not any("libs/bar" in p for p in paths)
diff --git a/tests/unit/server/conftest.py b/tests/unit/server/conftest.py
index b29f49f2b..7e80e599a 100644
--- a/tests/unit/server/conftest.py
+++ b/tests/unit/server/conftest.py
@@ -6,6 +6,8 @@ provides an httpx AsyncClient for making requests.
from __future__ import annotations
+from pathlib import Path
+
import pytest
from httpx import ASGITransport, AsyncClient
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
@@ -128,13 +130,25 @@ async def session(session_factory) -> AsyncSession:
await s.commit()
-async def create_test_repo(client: AsyncClient) -> dict:
- """Helper: create a test repository via the API and return its response."""
+async def create_test_repo(client: AsyncClient, tmp_path: Path | None = None) -> dict:
+ """Helper: create a test repository via the API and return its response.
+
+ Uses a real temp directory with a ``.git`` folder to pass server-side
+ path validation (``RepoCreate.validate_local_path``).
+ """
+ import tempfile
+
+ if tmp_path is None:
+ tmp_path = Path(tempfile.mkdtemp())
+ repo_dir = tmp_path / "test-repo"
+ repo_dir.mkdir(exist_ok=True)
+ (repo_dir / ".git").mkdir(exist_ok=True)
+
resp = await client.post(
"/api/repos",
json={
"name": "test-repo",
- "local_path": "/tmp/test-repo",
+ "local_path": str(repo_dir),
"url": "https://github.com/example/test-repo",
},
)
diff --git a/tests/unit/server/test_repos.py b/tests/unit/server/test_repos.py
index 700b1384d..8c861291a 100644
--- a/tests/unit/server/test_repos.py
+++ b/tests/unit/server/test_repos.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+import tempfile
+from pathlib import Path
from unittest.mock import AsyncMock, patch
import pytest
@@ -12,18 +14,21 @@ from tests.unit.server.conftest import create_test_repo
@pytest.mark.asyncio
async def test_create_repo(client: AsyncClient) -> None:
+ repo_dir = Path(tempfile.mkdtemp()) / "my-repo"
+ repo_dir.mkdir()
+ (repo_dir / ".git").mkdir()
+
resp = await client.post(
"/api/repos",
json={
"name": "my-repo",
- "local_path": "/tmp/my-repo",
+ "local_path": str(repo_dir),
"url": "https://github.com/example/my-repo",
},
)
assert resp.status_code == 201
data = resp.json()
assert data["name"] == "my-repo"
- assert data["local_path"] == "/tmp/my-repo"
assert data["url"] == "https://github.com/example/my-repo"
assert data["default_branch"] == "main"
assert "id" in data
diff --git a/website/cli-reference.md b/website/cli-reference.md
index 8d2f6759c..29ba7f8ae 100644
--- a/website/cli-reference.md
+++ b/website/cli-reference.md
@@ -53,6 +53,7 @@ repowise init [PATH] [OPTIONS]
| `--skip-tests` | flag | false | Exclude test files |
| `--skip-infra` | flag | false | Exclude Dockerfiles, Makefiles, Terraform, shell scripts |
| `--exclude` / `-x` | string | — | Gitignore-style exclude pattern (repeatable: `-x vendor/ -x "*.gen.*"`) |
+| `--include-submodules` | flag | false | Include git submodule directories (excluded by default) |
| `--concurrency` | int | 5 | Max concurrent LLM calls |
| `--resume` | flag | false | Resume from last checkpoint after an interruption |
| `--force` | flag | false | Regenerate all pages even if up to date |
@@ -79,6 +80,9 @@ repowise init --skip-tests --skip-infra --concurrency 3
# Exclude generated files and vendor directories
repowise init -x "*.generated.ts" -x vendor/ -x proto/
+# Repo with git submodules — include them in indexing
+repowise init --include-submodules
+
# Dry run to estimate cost before committing
repowise init --dry-run
diff --git a/website/configuration.md b/website/configuration.md
index f0ed70b33..e3edcefa3 100644
--- a/website/configuration.md
+++ b/website/configuration.md
@@ -248,21 +248,39 @@ repowise also writes a `.mcp.json` at the repository root for Claude Code auto-d
## Exclude patterns
-Control which files are indexed using gitignore-style patterns:
+repowise respects your `.gitignore` automatically. All patterns in your root `.gitignore` are honored during file traversal using the same `gitwildmatch` format that git uses.
+
+On top of `.gitignore`, you can add extra patterns via `--exclude` / `-x`:
```bash
repowise init -x vendor/ -x "*.generated.ts" -x proto/ -x "**/*.pb.go"
```
-Patterns are saved to `config.yaml` and applied on subsequent `update` runs.
+Patterns are saved to `config.yaml` and applied on subsequent `update` runs. In interactive advanced mode, you can enter exclude patterns one per line instead of comma-separated.
+
+You can also create a `.repowiseIgnore` file (same gitignore syntax) at the repo root or in any subdirectory for more granular control.
Built-in exclusions (always applied):
- `.git/`, `.repowise/`, `node_modules/`
- `__pycache__/`, `*.pyc`, `.venv/`
-- Binary files and media assets
+- Binary files, lockfiles, and minified assets
Use `--skip-tests` to exclude test files and `--skip-infra` to exclude Dockerfiles, Makefiles, and shell scripts.
+After traversal, repowise prints a filtering summary showing how many files were included and excluded by each rule (`.gitignore`, extensions, binary, oversized, etc.), along with a language breakdown.
+
+---
+
+## Submodules
+
+Git submodule directories are excluded by default. If your repo uses submodules and you want to include them in the index, pass `--include-submodules`:
+
+```bash
+repowise init --include-submodules
+```
+
+repowise reads `.gitmodules` to detect submodule paths. When excluded, submodule directories are skipped entirely during traversal.
+
---
## PostgreSQL
diff --git a/website/getting-started.md b/website/getting-started.md
index e79adcd3b..77e84c498 100644
--- a/website/getting-started.md
+++ b/website/getting-started.md
@@ -78,16 +78,20 @@ When it finishes, the MCP server is registered and Claude can query your codebas
Indexing your codebase...
Phase 1 of 4 — Ingestion
+ Scanning & filtering files... [████████████████████]
+ Scanned 1,247 files, 247 included
+ Excluded: 820 by .gitignore, 45 by extension, 12 binary, 123 unknown type
+ Languages: python 892, typescript 245, go 80, yaml 18, json 12
Parsing files... [████████████████████] 247/247
Building dependency graph... done
- Running git analysis... done
Phase 2 of 4 — Analysis
- Computing churn scores... done
+ Indexing file history... done
Detecting dead code... done
- Mining decisions... done
+ Extracting decisions... done
Phase 3 of 4 — Generation
+ Languages: python 58%, typescript 28%, go 10%, yaml 2%, json 1%
Generating wiki pages... [████████████████░░░░] 82%
Estimated tokens: ~180k